Government affair scene multi-mode voice interaction method based on lip movement
By combining multimodal analysis of lip movements and voice information, the problem of low accuracy of voice interaction recognition in complex noise environments in government affairs scenarios is solved, and efficient recognition effect is achieved in complex environments.
Patent Information
- Application Number
- CN202411082901.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-03
AI Technical Summary
The self-service voice interaction in existing government affairs scenarios has poor recognition effect in complex noise environments, and the accuracy of the recognition results is not high, especially in the case of surrounding people's noise.
Using multimodal voice interaction method based on lip action, collecting face information through binocular cameras, narrow beam algorithm enhances array microphone audio signals, high-sensitivity microphones collect voice, high-definition cameras capture lip action video, combining computer vision and Deep learning algorithms perform information fusion and use NLP large model to perform collaborative analysis and recognition of speech and lip movements.
Realize efficient identification in complex noise environments, improve identification accuracy, and provide better usage prospects and commercial value.
Smart Images

Figure CN119065499A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of government service, and specifically is a multimodal voice interaction method for government scenarios based on lip movements. Background Art
[0002] The government affairs "digital person" appears in front of the public in the form of an all-in-one machine. The Free Trade Zone Government Affairs Office integrates the high-frequency matters handled by enterprises and the public into the government affairs "digital person", realizing the seamless integration of "digital person" and application scenarios such as consultation, acceptance, and handling of government affairs services. It promotes government services from the traditional government affairs service hall model to a standardized, efficient, and stable digital and intelligent direction, and innovates the government services "digital person" full-process guidance and assistance models, providing enterprises and the public with "one-to-one" immersive consultation and guidance services.
[0003] In the existing self-service voice interaction process of government affairs scenarios, in complex noise environments, especially when the noise is the surrounding human voices, the recognition effect is usually poor and the accuracy of the recognition results is not high. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides a multimodal voice interaction method for government affairs scenarios based on lip movements, which has the advantages of efficient recognition in complex noisy environments and high accuracy.
[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multimodal voice interaction method for government affairs scenarios based on lip movements, the interaction method comprising: collecting facial information through a binocular camera, automatically waking up the device, and using a digital human to give voice prompts to allow users to interact; using a narrow beam algorithm to enhance and locate the array microphone audio signal, perform directional reception, and shield sounds from other directions; using a high-sensitivity microphone to collect the user's voice input in real time, and using a voice processing algorithm to extract voice features; using a high-definition camera to capture the user's lip movement video, and applying computer vision technology to analyze the morphological changes and mouth contours of the lips in real time; comprehensively analyzing the voice features and lip movement features to form comprehensive user interaction information, and using a deep learning algorithm to perform multimodal information fusion; based on multimodal information, performing collaborative analysis and recognition of voice and lip movements; based on the collected audio signal, transmitting it to the NLP big model, matching customer needs from massive data and knowledge; establishing a high-precision digital human model, using the NLP big model to understand and analyze the user's questions, and giving answers through digital human broadcasts.
[0006] Preferably, the step of S1 is:
[0007] Use the binocular camera to collect facial information and trigger the automatic wake-up function of the device to wake up the digital human on the main screen, the task list on the back screen and the operation interface on the touch screen. Start the operation interface on the touch screen and the user can interact. The operation options that the user can perform are displayed. The digital human uses the voice function to perform voice enhancement interaction with the user. The user interacts and is prompted to perform specific operations through the touch screen and voice commands.
[0008] Preferably, step S2 is: arranging 10-20 microphones in an array in a rectangular geometric shape, with each microphone in the array being positioned at a distance of 3 cm, and the microphones simultaneously collecting sound signals in the environment and preprocessing the collected audio signals;
[0009] Apply narrow beamforming algorithm to synthesize and process the signals of each microphone, calculate the delay and weight of each microphone signal, and the synthesized beam enhances the target signal in a specific direction and suppresses the signals in other directions;
[0010] Beamforming weight vector:
[0011] where w is the weight vector, a|θ| is the directional vector at an angle θ), and |·| represents the Euclidean norm of the vector;
[0012] Output the audio signal after beamforming processing, and reflect the characteristics of the main beam and the sub-beam in the output;
[0013] Use beamforming output to achieve directional reception of sound in a specific direction.
[0014] Preferably, the step of S3 is: using a high-sensitivity microphone to capture the user's continuous voice signal, eliminating background noise and other non-language sound interference, enhancing the high-frequency part, compensating for the high-frequency components lost during the recording process, dividing the continuous audio stream into time frames of 25ms-40ms, applying a window function to each frame of data to reduce the spectrum leakage effect, and performing a fast Fourier transform on each frame of data to convert it to the frequency domain to analyze its spectrum characteristics;
[0015] For specific acoustic parameters, the cepstrum method is used to extract the pitch period. The extraction steps are:
[0016] Fourier transform: First, perform fast Fourier transform FF on the original time domain signal x(t) to obtain its frequency domain representation X(f):
[0017] [X(f)=Fx(t)]
[0018] Calculate the logarithm of the magnitude: Then take X(f) to get the magnitude and calculate its natural logarithm:
[0019]
[0020] where T0,tf,k,f / 0 is a relevant parameter or constant;
[0021] Finally, the above results are inverse fast Fourier transformed back to the time domain space, and the "cepstrum" is obtained:
[0022]
[0023] The processed data is encoded and sent to the next receiving system and storage device.
[0024] Preferably, the step of S4 is:
[0025] Use a high-definition camera to capture a video of the user's face, perform denoising and contrast enhancement on the video to improve image quality, determine the direction and position of the face through feature points, and standardize the image to a uniform size and direction, identify important facial feature points in the lip area, including the corners of the mouth and the lip peak, and use threshold segmentation technology to separate the lips from the background;
[0026] The formula for threshold segmentation technique is:
[0027] [I0(x,y)={0ifI(x,y) <T1ifI(x,y)≥T}]
[0028] Where I(x, y) is the intensity of the original image at (x, y), and T is the preset global brightness or color limit;
[0029] The extracted lip regions are further processed to remove noise, and the changes in key points are tracked between consecutive frames to understand the movement patterns of the lips during speaking.
[0030] Preferably, the step S5 is: comprehensively analyzing the speech features and lip movement features to form comprehensive user interaction information, extracting the features of the sound, identifying and tracking the movements of the lips, and extracting the features of the key point positions and the changes in the lip shape;
[0031] Use convolutional neural networks and recurrent neural networks to learn and encode feature representations of audio and visual modalities respectively;
[0032] Early fusion merges the two modalities at the input stage, and late fusion combines the results of two independent models at the decision stage.
[0033] Preferably, the step of S6 is:
[0034] At the same time, the user's audio and video signals are collected to pre-process the audio data to extract acoustic features from the audio and visual features from the video, including the movement trajectory and shape changes of the lips;
[0035] Use timestamps to ensure temporal consistency between audio and video data, and identify and track key points of lips in the video;
[0036] Use a neural network structure to train the model to learn the mapping relationship from raw audio input to target output;
[0037] Establish a complete system process that can directly receive raw audio and video input and generate final recognition results or user interaction instructions.
[0038] Preferably, the step of S7 is:
[0039] Based on the parsed intent and key information, search for relevant service and product options in the database or knowledge base, and keep the user's historical interaction content in mind during the conversation;
[0040] Train deep learning models to optimize semantic understanding and continuously adjust parameters to adapt to changing requirements in different fields and use cases;
[0041] Generate a personalized service or product recommendation list for users based on the analysis results, and display it to users in order of relevance;
[0042] User feedback on the recommended services or products is used to further train and optimize the performance of the system.
[0043] Preferably, the step of S8 is:
[0044] Use NLP technology to analyze text content, extract key information, build a knowledge base containing a large amount of information, search for the most relevant information and answers in the knowledge base based on the intentions and keywords parsed from user input, design the non-verbal behavior of digital humans' facial expressions and body movements, and implement a system to manage the interaction process between users and digital humans, including answering questions and feedback loops, and generate answer scripts based on the analysis results of the large NLP model.
[0045] Preferably, verify that all components work together, deploy the tested system to the production environment, and continuously monitor its performance to respond to issues in a timely manner.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] By capturing the details of facial expressions, this application can achieve efficient recognition in complex environments with high recognition accuracy, bringing better usage prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a flow chart of the steps of the present invention. DETAILED DESCRIPTION
[0049] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] The present invention provides a technical solution: a multimodal voice interaction method for government affairs scenarios based on lip movements, the interaction method comprising:
[0052] S1: Collect facial information through binocular cameras, automatically wake up the device, and the digital human gives voice prompts to allow users to interact;
[0053] S2: Use narrow beam algorithm to enhance and locate the array microphone audio signal, perform directional reception, and shield sounds from other directions;
[0054] S3: Use a high-sensitivity microphone to collect user voice input in real time and use a voice processing algorithm to extract voice features;
[0055] S4: Use high-definition camera equipment to capture the user's lip movement video, apply computer vision technology, and analyze the lip morphology changes and mouth contours in real time;
[0056] S5: Comprehensively analyze speech features and lip movement features to form comprehensive user interaction information, and use deep learning algorithms to perform multimodal information fusion;
[0057] S6: Collaborative analysis and recognition of speech and lip movements based on multimodal information;
[0058] S7: Based on the collected audio signals, they are transmitted to the NLP big model to match customer needs from massive data and knowledge;
[0059] S8: Establish a high-precision digital human model, use the NLP big model to understand and analyze user questions, and give answers through digital human broadcast.
[0060] Among them, the steps of S1 are:
[0061] Use the binocular camera to collect facial information and trigger the automatic wake-up function of the device to wake up the digital human on the main screen, the task list on the back screen and the operation interface on the touch screen. Start the operation interface on the touch screen and the user can interact. The operation options that the user can perform are displayed. The digital human uses the voice function to perform voice enhancement interaction with the user. The user interacts and is prompted to perform specific operations through the touch screen and voice commands.
[0062] Further, the step S2 is: arranging 10-20 microphones in an array according to a rectangular geometric shape, each microphone in the array is positioned at a distance of 3 cm, the microphones simultaneously collect sound signals in the environment, and pre-processing the collected audio signals;
[0063] Apply narrow beamforming algorithm to synthesize and process the signals of each microphone, calculate the delay and weight of each microphone signal, and the synthesized beam enhances the target signal in a specific direction and suppresses the signals in other directions;
[0064] Beamforming weight vector:
[0065] where w is the weight vector, a|θ| is the directional vector at an angle θ), and |·| represents the Euclidean norm of the vector;
[0066] Output the audio signal after beamforming processing, and reflect the characteristics of the main beam and the sub-beam in the output;
[0067] Use beamforming output to achieve directional reception of sound in a specific direction.
[0068] Furthermore, the steps of S3 are: using a high-sensitivity microphone to capture the user's continuous voice signal, eliminating background noise and other non-language sound interference, enhancing the high-frequency part, compensating for the high-frequency components lost during the recording process, dividing the continuous audio stream into time frames of 25ms-40ms, applying a window function to each frame of data to reduce the spectrum leakage effect, and performing a fast Fourier transform on each frame of data to convert it to the frequency domain to analyze its spectrum characteristics;
[0069] For specific acoustic parameters, the cepstrum method is used to extract the pitch period. The extraction steps are:
[0070] Fourier transform: First, perform fast Fourier transform FF on the original time domain signal x(t) to obtain its frequency domain representation X(f):
[0071] [X(f)=Fx(t)]
[0072] Calculate the logarithm of the magnitude: Then take X(f) to get the magnitude and calculate its natural logarithm:
[0073]
[0074] where T0,tf,k,f / 0 is a relevant parameter or constant;
[0075] Finally, the above results are inverse fast Fourier transformed back to the time domain space, and the "cepstrum" is obtained:
[0076]
[0077] The processed data is encoded and sent to the next receiving system and storage device.
[0078] Furthermore, the steps of S4 are:
[0079] Use a high-definition camera to capture a video of the user's face, perform denoising and contrast enhancement on the video to improve image quality, determine the direction and position of the face through feature points, and standardize the image to a uniform size and direction, identify important facial feature points in the lip area, including the corners of the mouth and the lip peak, and use threshold segmentation technology to separate the lips from the background;
[0080] The formula for threshold segmentation technique is:
[0081] [I0(x,y)={0ifI(x,y) <T1ifI(x,y)≥T}]
[0082] Where I(x, y) is the intensity of the original image at (x, y), and T is the preset global brightness or color limit;
[0083] The extracted lip regions are further processed to remove noise, and the changes in key points are tracked between consecutive frames to understand the movement patterns of the lips during speaking.
[0084] Further, the step S5 is: comprehensively analyzing the speech features and lip movement features to form comprehensive user interaction information, extracting the features of the sound, identifying and tracking the movements of the lips, and extracting the features of the key point positions and the changes in the lip shape;
[0085] Use convolutional neural networks and recurrent neural networks to learn and encode feature representations of audio and visual modalities respectively;
[0086] Early fusion merges the two modalities at the input stage, and late fusion combines the results of two independent models at the decision stage.
[0087] Furthermore, the steps of S6 are:
[0088] At the same time, the user's audio and video signals are collected to pre-process the audio data to extract acoustic features from the audio and visual features from the video, including the movement trajectory and shape changes of the lips;
[0089] Use timestamps to ensure temporal consistency between audio and video data, and identify and track key points of lips in the video;
[0090] Use a neural network structure to train the model to learn the mapping relationship from raw audio input to target output;
[0091] Establish a complete system process that can directly receive raw audio and video input and generate final recognition results or user interaction instructions.
[0092] Furthermore, the steps of S7 are:
[0093] Based on the parsed intent and key information, search for relevant service and product options in the database or knowledge base, and keep the user's historical interaction content in mind during the conversation;
[0094] Train deep learning models to optimize semantic understanding and continuously adjust parameters to adapt to changing requirements in different fields and use cases;
[0095] Generate a personalized service or product recommendation list for users based on the analysis results, and display it to users in order of relevance;
[0096] User feedback on the recommended services or products is used to further train and optimize the performance of the system.
[0097] Furthermore, the steps of S8 are:
[0098] Use NLP technology to analyze text content, extract key information, build a knowledge base containing a large amount of information, search for the most relevant information and answers in the knowledge base based on the intentions and keywords parsed from user input, design the non-verbal behavior of digital humans' facial expressions and body movements, and implement a system to manage the interaction process between users and digital humans, including answering questions and feedback loops, and generate answer scripts based on the analysis results of the large NLP model.
[0099] Furthermore, verify the effectiveness of all components working together, deploy the tested system to the production environment, and continuously monitor its performance to respond to problems in a timely manner.
[0100] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0101] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal voice interaction method for government affairs scenarios based on lip movements, characterized in that: Interaction methods include: S1: Collect facial information through binocular cameras, automatically wake up the device, and the digital human gives voice prompts to allow users to interact; S2: Use narrow beam algorithm to enhance and locate the array microphone audio signal, perform directional reception, and shield sounds from other directions; S3: Use a high-sensitivity microphone to collect user voice input in real time and use a voice processing algorithm to extract voice features; S4: Use high-definition camera equipment to capture the user's lip movement video, apply computer vision technology, and analyze the lip morphology changes and mouth contours in real time; S5: Comprehensively analyze speech features and lip movement features to form comprehensive user interaction information, and use deep learning algorithms to perform multimodal information fusion; S6: Collaborative analysis and recognition of speech and lip movements based on multimodal information; S7: Based on the collected audio signals, they are transmitted to the NLP big model to match customer needs from massive data and knowledge; S8: Establish a high-precision digital human model, use the NLP big model to understand and analyze user questions, and give answers through digital human broadcasts.
2. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S1 are: Use the binocular camera to collect facial information, trigger the automatic wake-up function of the device, wake up the digital human on the main screen, the task list on the back screen and the operation interface on the touch screen, start the operation interface on the touch screen, and the user interacts to display the operation options performed by the user. The digital human uses the voice function to perform voice enhancement interaction with the user, and the user interacts to prompt the user that he can perform specific operations through the touch screen and voice commands.
3. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The step of S2 is: arranging 10-20 microphones in an array according to a rectangular geometric shape, each microphone in the array is positioned at a distance of 3 cm, the microphones simultaneously collect sound signals in the environment, and pre-process the collected audio signals; Apply narrow beamforming algorithm to synthesize and process the signals of each microphone, calculate the delay and weight of each microphone signal, and the synthesized beam enhances the target signal in a specific direction and suppresses the signals in other directions; Beamforming weight vector: where w is the weight vector, a|θ| is the directional vector when the pointing angle is θ, and |·| represents the Euclidean norm of the vector; Output the audio signal after beamforming processing, and reflect the characteristics of the main beam and the sub-beam in the output; Use beamforming output to achieve directional reception of sound in a specific direction.
4. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S3 are: using a high-sensitivity microphone to capture the user's continuous voice signal, eliminating background noise and other non-language sound interference, enhancing the high-frequency part, compensating for the high-frequency components lost during the recording process, dividing the continuous audio stream into time frames of 25ms-40ms, applying a window function to each frame of data to reduce the spectrum leakage effect, and performing a fast Fourier transform on each frame of data to convert it to the frequency domain to analyze its spectrum characteristics; For specific acoustic parameters, the cepstrum method is used to extract the pitch period. The extraction steps are: Fourier transform: First, perform fast Fourier transform FF on the original time domain signal x(t) to obtain its frequency domain representation X(f): [X(f)=Fx(t)] Calculate the logarithm of the magnitude: Then take X(f) to get the magnitude and calculate its natural logarithm: where T0,tf,k,f / 0 is a constant; Finally, the result is inverse fast Fourier transformed back to the time domain space and the "cepstrum" is obtained; The processed data is encoded and sent to the next receiving system and storage device.
5. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S4 are: Use a high-definition camera to capture a video of the user's face, perform denoising and contrast enhancement on the video to improve image quality, determine the direction and position of the face through feature points, and standardize the image to a uniform size and direction, identify important facial feature points in the lip area, including the corners of the mouth and the lip peak, and use threshold segmentation technology to separate the lips from the background; The formula for threshold segmentation technique is: [I0(x,y)={0ifI(x,y) <T1ifI(x,y)≥T}] Where I(x, y) is the intensity of the original image at (x, y), and T is the preset global brightness and color boundary; The extracted lip regions are further processed to remove noise, and the changes in key points are tracked between consecutive frames to understand the movement patterns of the lips during speaking.
6. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The step of S5 is: comprehensively analyzing the speech features and lip movement features to form comprehensive user interaction information, extracting the features of the sound, identifying and tracking the movements of the lips, and extracting the features of the key point positions and the changes in the lip shape; Use convolutional neural networks and recurrent neural networks to learn and encode feature representations of audio and visual modalities respectively; Early fusion merges the two modalities at the input stage, and late fusion combines the results of two independent models at the decision stage.
7. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S6 are: At the same time, the user's audio and video signals are collected to pre-process the audio data to extract acoustic features from the audio and visual features from the video, including the movement trajectory and shape changes of the lips; Use timestamps to ensure temporal consistency between audio and video data, and identify and track key points of lips in the video; The neural network structure is used to train the model to learn the mapping relationship from the original audio and video input to the target output.
8. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S7 are: Based on the parsed intent and key information, search for relevant services and product options in the database, and keep the user's historical interaction content in the conversation process; Generate a personalized list of service and product recommendations for users based on the analysis results, and display them to users in order of relevance; User feedback on recommended services and products is used to further train and optimize the system's performance.
9. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 1 is characterized in that: The steps of S8 are: Use NLP technology to analyze text content, extract key information, build a knowledge base containing a large amount of information, search for the most relevant information and answers in the knowledge base based on the intentions and keywords parsed from user input, design the non-verbal behavior of digital humans' facial expressions and body movements, and implement a system to manage the interaction process between users and digital humans, including answering questions and feedback loops, and generate answer scripts based on the analysis results of the large NLP model.
10. The multimodal voice interaction method for government affairs scenarios based on lip movements according to claim 9, characterized in that: Verify that all components work together, deploy the tested system to the production environment, and continuously monitor its performance to respond to issues in a timely manner.
Citation Information
Cited By
Digital human real-time interaction and video generation method based on large model
CN121617415A
