A system and method for matching speech, lip and action of a conversational robot

Abstract

The invention discloses a system and method for matching speech and mouth shape movements of a dialogue robot, and relates to the technical field of speech matching. The method adopts Bluetooth dual-channel technology, transmits a time point array to a dialogue robot Arduino mainboard through a first Bluetooth channel, and transmits a voice file to a dialogue robot audio device through a second Bluetooth channel, so as to achieve high synchronization of speech playback and mechanical movements, and synchronous transmission of voice files and time point arrays, thereby ensuring a close association between speech playback and mouth shape movements, and avoiding movement lags caused by transmission delays or untimely hardware responses. Each time point in the time point array can be accurately read and compared through a built-in timer of the Arduino mainboard, ensuring that at each set time point, a servo can respond immediately to drive the opening and closing movement of the dialogue robot's mouth. When the time point arrives, an instruction is sent to control the opening and closing of the mouth shape, so that the speech and mouth shape movements remain highly consistent.

Description

Technical Field

[0001] The present invention relates to the technical field of voice matching, and in particular to a system and method for voice and lip-action matching of a conversational robot. Background Art

[0002] With the continuous development of artificial intelligence technology, speech recognition and synthesis technology has been widely used in many fields. From the initial intelligent assistant to the voice dialogue robot, and then to today's dialogue robot and virtual anchor, voice technology has become an important bridge for human-computer interaction. As human-computer interaction becomes more natural, dialogue robots not only need to have the ability to understand and generate human language, but also need to simulate the natural way of human expression during voice interaction, that is, to match the voice through mechanical mouth movements to enhance the realism of interaction. Therefore, the matching of dialogue robot mouth movements and voice has become an important research topic. This technology is mainly used in dialogue robots, virtual reality VR, and augmented reality AR. Especially in the field of dialogue robots, how to make dialogue robots adjust their mouth movements in real time according to the content and rhythm of the voice has become one of the key technologies to improve the "emotional expression" and "interactive experience" of dialogue robots.

[0003] At present, although the dialogue dolls and dialogue robots can play human recorded or machine-synthesized voices and interact with body movements, there is no effective coupling between the voice content and the mouth movements. Even though some high-end dialogue robot dolls have the ability to synchronize mouth movements with voice, they usually rely on fixed movements preset for specific voices. The dialogue robot can only synchronize the mouth shape and voice when playing limited fixed voice content. This method has significant limitations because it cannot handle dynamic and changeable voice content, resulting in the coordination of mouth movements relying on pre-programmed static codes, lacking flexibility and adaptability. Therefore, real-time synchronization of voice and mouth shape movements is still a major challenge in current technology applications. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a system and method for speech and lip movement matching of a conversational robot, which solves the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention is implemented by the following technical scheme: comprising the following steps:

[0006] S1. Recording voice through a recording device to obtain a voice file, uploading the voice file to a cloud server, and preprocessing the voice file in the cloud server to obtain spectrum data of the voice file;

[0007] S2, extracting frequency domain features based on the spectrum data to obtain frequency domain feature vectors, and building a cloud database to store the frequency domain feature vectors in the cloud database;

[0008] S3, by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth and comparing them one by one with all the parameters in the frequency domain feature vector, extracting the time point array Ttimes of the voice file;

[0009] S4. Set up Bluetooth dual channels to transmit the time point array Ttimes and the voice file to the conversation robot Arduino mainboard and the conversation robot audio device respectively;

[0010] S5. Through the built-in timer of the conversation robot Arduino mainboard, read the time point parameters in the time point array Ttimes one by one. Every time a specified time point is reached, a command signal is sent to drive the servo to execute the conversation robot's mouth opening and closing action.

[0011] Preferably, said S1 includes S11, S12 and S13;

[0012] S11. Record the user's voice signal through the applet and APP of the recording device, convert the voice signal into digital voice data, obtain the voice file, and convert the voice file into a standard audio format, wherein the standard audio format is MP3 format and WAV format;

[0013] S12, uploading the acquired voice file to the cloud server via the communication network, and automatically generating a unique file ID for each uploaded voice file in the cloud server;

[0014] S13, pre-processing the received voice file on the cloud server to eliminate the interfering voice signal in the voice file, extracting the spectrum data in the voice file through Fourier transform, and then normalizing the spectrum data to eliminate the dimensional influence between the spectrum data;

[0015] The spectrum data includes an amplitude value A and a frequency component f.

[0016] Preferably, S2 includes S21 and S22;

[0017] S21, extracting frequency domain features based on the spectrum data, obtaining a speech periodicity index Psig and a frequency component change rate Rf, and then summarizing the speech periodicity index Psig and the frequency component change rate Rf with the amplitude value A to obtain a frequency domain feature vector;

[0018] The S21 includes S211 and S212;

[0019] S211, the speech periodicity index Psig is calculated based on the amplitude value A to obtain the autocorrelation function ACF, and the specific algorithm formula is: ; Among them, A (t i ) represents the amplitude value at the i-th time point t, ACF(t) represents the autocorrelation function at time t, N represents the total number of sampling points, A(t i+1 ) represents the amplitude value of the speech signal at the i+1th time point;

[0020] Then the calculation output is based on the autocorrelation function ACF. The specific algorithm formula is: , where t max Represents the upper limit of the autocorrelation function ACF at time t.

[0021] Preferably, in S212, the frequency component change rate Rf is calculated and output based on the frequency component f, and the specific algorithm formula is: , where f(t i ) represents the frequency component at the i-th time point t, f(t i+1 ) represents the frequency component at the i+1th time point;

[0022] S22. Build a cloud database, create a voice storage table and a data storage table, store the voice file in the voice storage table, and store the frequency domain feature vector in the data storage table.

[0023] Preferably, said S3 includes S31, S32, S33 and S34;

[0024] S31, based on the user setting the amplitude threshold Ath according to the content of the voice file, the amplitude value A of the frequency domain feature vector is extracted and compared with the set amplitude threshold Ath, and the time point sequence of the voice file is extracted according to the amplitude fluctuation to obtain the amplitude time point sequence Tamp;

[0025] The specific extraction contents are as follows;

[0026] ;

[0027] Where, t i represents the i-th time point t, A(t i ) represents the amplitude value at the i-th time point t.

[0028] Preferably, S32, based on the periodicity threshold Pth set by the user according to the content of the voice file, the voice periodicity index Psig of the extracted frequency domain feature vector is compared and analyzed with the periodicity threshold Pth, and the periodic time point sequence Tper is obtained according to the time point sequence of the periodically extracted voice file;

[0029] The specific extraction content is as follows:

[0030] ;

[0031] Where, Psig (t i ) represents the speech periodicity index at the i-th time point t.

[0032] Preferably, S33, based on the user setting the frequency change threshold Rth according to the content of the voice file, the frequency component change rate Rf of the frequency domain feature vector is extracted and compared with the frequency change threshold Rth, and the frequency change time point sequence Tfre is obtained according to the time point sequence of the periodically extracted voice file;

[0033] The specific extraction content is as follows:

[0034] ;

[0035] Where Rf(ti) represents the rate of change of the frequency component at the i-th time point t;

[0036] S34, based on the extracted amplitude time point sequence Tamp, periodic time point sequence Tper and frequency change time point sequence Tfre, merge and obtain a time point array Ttimes, wherein the time point array Ttimes=[t1, t2, t3, ..., t n ], where t n Represents the nth time point in the time point array Ttimes;

[0037] The time point array Ttimes is merged by the following formula:

[0038] ;

[0039] Preferably, the S4 includes S41;

[0040] S41, setting a Bluetooth dual channel, wherein the Bluetooth dual channel includes a first Bluetooth channel and a second Bluetooth channel;

[0041] The first Bluetooth channel transmits the acquired time point array Ttimes to the dialogue robot Arduino mainboard by connecting to the dialogue robot Arduino mainboard;

[0042] The second Bluetooth channel is connected to the audio device of the conversation robot to play the voice file through the audio device.

[0043] Preferably, the S5 includes S51;

[0044] S51, using the built-in timer of the conversation robot Arduino mainboard, compare the time point in the time point array Ttimes with the current time, and based on the comparison evaluation, send instructions to the servo to control the operation of the servo and drive the opening and closing of the conversation robot's mouth;

[0045] The specific comparison contents are as follows;

[0046] ;

[0047] Among them, t current Indicates the current time in the timer built into the conversational robot Arduino mainboard;

[0048] When the current time t in the timer built into the conversational robot Arduino mainboard current ≥ the nth time point t in the time point array Ttimes n At this time, the opening instruction 1 is sent to drive the servo to control the conversation robot to open its mouth;

[0049] When the current time t in the timer built into the conversational robot Arduino mainboard current <The nth time point t in the time point array Ttimes n At this time, the closing command 0 is sent to drive the servo to control the conversational robot's mouth to close.

[0050] A speech and lip-action matching system for a conversational robot, comprising a speech recording module, a feature extraction module, a time point extraction module, a Bluetooth transmission module and a lip-action triggering module;

[0051] The voice recording module records voice through a recording device to obtain a voice file, then uploads the voice file to a cloud server, and pre-processes the voice file in the cloud server to obtain spectrum data of the voice file;

[0052] The feature extraction module extracts frequency domain features through the spectrum data to obtain frequency domain feature vectors, and at the same time builds a cloud database to store the frequency domain feature vectors in the cloud database;

[0053] The time point extraction module extracts the time point array Ttimes of the voice file by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth to compare with all the parameters in the frequency domain feature vector one by one;

[0054] The Bluetooth transmission module transmits the time point array Ttimes and the voice file to the dialogue robot Arduino mainboard and the dialogue robot audio device respectively by setting up a Bluetooth dual channel;

[0055] The lip action trigger module reads the time point parameters in the time point array Ttimes one by one through the timer built into the dialogue robot Arduino mainboard, and sends a command signal to drive the servo to perform the dialogue robot's mouth opening and closing action every time a specified time point is reached.

[0056] The present invention provides a system and method for matching speech, lip shape and action of a conversational robot. It has the following beneficial effects:

[0057] (1) This method effectively improves the synchronization accuracy of the conversation robot's mouth shape and speech through preprocessing of spectral data and extraction of frequency domain features. First, the spectral data of the voice file is obtained through Fourier transform, and then the frequency domain feature vector is generated through frequency domain feature extraction, which further optimizes the conversation robot's perception of speech rhythm, volume, etc. Then, according to the set amplitude threshold Ath, periodicity threshold Pth and frequency change threshold Rth, the features are analyzed and the time point sequence is extracted, so that each change point of the speech can accurately trigger the corresponding mouth shape movement, avoiding the phenomenon of mismatch between speech and mouth shape in traditional methods. Through this optimization, the conversation robot can accurately perform the opening and closing of the mouth while playing the speech, achieving a more natural speech and lip synchronization effect.

[0058] (2) This method extracts the periodicity, emotional fluctuation and frequency change characteristics of speech by setting the amplitude threshold Ath, periodicity threshold Pth and frequency change threshold Rth, thereby achieving a more accurate match between speech rhythm and emotional expression. In step S5, the opening and closing frequency and amplitude of the mouth shape movement can be dynamically adjusted according to the speech content by combining the time point array Ttimes with parameters such as amplitude, periodicity and frequency component change rate Rf. When there are large emotional fluctuations or changes in intonation in the speech, the dialogue robot can adjust the mouth shape movement in time through the change of the frequency component change rate, so that the dialogue robot can better express the emotions in the speech and improve the naturalness and emotional richness of the speech interaction. Through this mechanism, the dialogue robot can adaptively adjust the mouth shape movement according to the speed, volume and emotional fluctuation of the speech, avoiding the shortcomings of lack of emotional expression or uncoordinated movements in traditional methods.

[0059] (3) This method uses Bluetooth dual-channel technology. The time point array Ttimes is transmitted to the conversation robot Arduino mainboard through the first Bluetooth channel, and the voice file is transmitted to the conversation robot audio device through the second Bluetooth channel, so as to achieve high synchronization between voice playback and mechanical action. The synchronous transmission of the voice file and the time point array ensures the close connection between voice playback and mouth shape action, and avoids the action lag caused by transmission delay or untimely hardware response. The built-in timer of the Arduino mainboard can accurately read and compare each time point in the time point array, ensuring that at each set time point, the servo can respond immediately and drive the opening and closing action of the conversation robot's mouth. When the time point arrives, a command will be sent to control the opening and closing of the mouth shape, so that the voice and mouth shape action are highly consistent. This synchronization mechanism of hardware and audio playback significantly improves the performance of the conversation robot in a real environment, eliminates the unnatural feeling caused by the asynchrony between mechanical action and voice playback, and thus enhances the user interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the steps of a method for matching speech and lip movements of a conversational robot according to the present invention;

[0061] Figure 2 The present invention is a flow chart of a system for matching speech, lip shape and action of a conversational robot. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example 1

[0064] See also Figure 1 The present invention provides a method for matching speech and lip movements of a conversational robot. To achieve the above object, the present invention is implemented by the following technical scheme: comprising the following steps:

[0065] S1. Recording voice through a recording device to obtain a voice file, uploading the voice file to a cloud server, and preprocessing the voice file in the cloud server to obtain spectrum data of the voice file;

[0066] S2, extracting frequency domain features based on the spectrum data, obtaining frequency domain feature vectors, and building a cloud database at the same time, storing the frequency domain feature vectors in the cloud database;

[0067] S3, by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth and comparing them with all the parameters in the frequency domain feature vector one by one, extracting the time point array Ttimes of the voice file;

[0068] S4. Set up Bluetooth dual channels to transmit the time point array Ttimes and the voice file to the conversation robot Arduino mainboard and the conversation robot audio device respectively;

[0069] S5. Through the built-in timer of the conversation robot Arduino mainboard, read the time point parameters in the time point array Ttimes one by one. Every time a specified time point is reached, a command signal is sent to drive the servo to execute the conversation robot's mouth opening and closing action.

[0070] In this embodiment, the method obtains a voice file through a recording device, uploads it to a cloud server for preprocessing, and then obtains spectrum data. This process not only ensures the quality of the voice signal, but also provides a clear frequency domain data basis for subsequent analysis. Next, by extracting frequency domain features from the spectrum data, a cloud database is constructed to store frequency domain feature vectors. This operation provides sufficient feature support for subsequent voice analysis and mouth shape motion control, and realizes effective management of voice data. By setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth, and comparing them one by one with all the parameters in the frequency domain feature vector, the time point array Ttimes of the voice file is successfully extracted. This step can accurately capture each key change moment in the voice, and provides an accurate time basis for the control of the subsequent mouth shape motion of the dialogue robot. By setting a Bluetooth dual channel, the time point array Ttimes and the voice file are successfully transmitted to the dialogue robot Arduino mainboard and the dialogue robot audio equipment, respectively, to ensure the synchronous playback of voice and action. Finally, the time point in the time point array Ttimes is accurately read by the built-in timer of the Arduino mainboard, and the servo is driven to perform the mouth shape opening and closing action. This process ensures that the mouth shape movement of the dialogue robot is synchronized with the changes in voice, and improves the user experience. Compared with the prior art, the present invention has achieved important improvements in many aspects: First, through the establishment of a cloud database and the storage of frequency domain feature vectors, voice data can be centrally managed, which improves data processing efficiency and accuracy. Secondly, by setting multiple thresholds for refined time point extraction, the synchronization of voice and mouth shape movement is more accurate, avoiding the phenomenon of lagging or unnatural mouth shape movement that may occur in traditional methods. Finally, the application of Bluetooth dual channels ensures the efficient transmission of voice and motion control, thereby avoiding the problems caused by hardware delays and improving the real-time response capability of the dialogue robot. In general, the method of the present invention has significant effects and improvements in improving the synchronization accuracy of voice and dialogue robot mouth shape movement and optimizing user interaction experience.

[0071] Example 2

[0072] Specifically: S1 includes S11, S12 and S13;

[0073] S11. Record the user's voice signal through the applet and APP of the recording device, convert the voice signal into digital voice data, obtain the voice file, and convert the voice file into a standard audio format, the standard audio format is MP3 format and WAV format;

[0074] S12, uploading the acquired voice file to the cloud server via the communication network, and automatically generating a unique file ID for each uploaded voice file in the cloud server;

[0075] S13, pre-processing the received voice file on the cloud server to eliminate the interfering voice signal in the voice file, extracting the spectrum data in the voice file through Fourier transform, and then normalizing the spectrum data to eliminate the dimensional influence between the spectrum data;

[0076] The spectrum data includes amplitude value A and frequency component f.

[0077] In this embodiment, the method uses a mini program and an APP to obtain the user's voice signal through a recording device, and converts it into a standard audio format such as MP3 or WAV to ensure the standardized processing and compatibility of voice data. It not only simplifies the recording process, but also enables voice data to be seamlessly transmitted between different platforms, improving the operability and flexibility of data processing. The voice file is uploaded to the cloud server and a unique file ID is automatically generated. This mechanism ensures the uniqueness of each voice file, facilitates subsequent storage, retrieval and management, provides an efficient file tracking system for the cloud database, and makes data processing more efficient and orderly. The preprocessing process, including eliminating interfering voice signals and extracting spectral data through Fourier transform, further improves the quality of voice data. The normalization of spectral data eliminates the influence between different dimensions and ensures the consistency and accuracy of the data in subsequent processing. This process optimizes the clarity and availability of voice signals and ensures the reliable extraction of frequency domain features.

[0078] Example 3

[0079] Specifically: S2 includes S21 and S22;

[0080] S21, extracting frequency domain features based on the spectrum data, obtaining a speech periodicity index Psig and a frequency component change rate Rf, and then summarizing the speech periodicity index Psig and the frequency component change rate Rf with the amplitude value A to obtain a frequency domain feature vector;

[0081] S21 includes S211 and S212;

[0082] S211, the speech periodicity index Psig is calculated based on the amplitude value A to obtain the autocorrelation function ACF, and the specific algorithm formula is: ; Among them, A (t i ) represents the amplitude value at the i-th time point t, ACF(t) represents the autocorrelation function at time t, N represents the total number of sampling points, A(t i+1 ) represents the amplitude value of the speech signal at the i+1th time point;

[0083] Then the autocorrelation function ACF is used for calculation and output to help determine the rhythm and naturalness of speech. The specific algorithm formula is: , where t max Represents the upper limit of the autocorrelation function ACF at time t.

[0084] S212, the frequency component change rate Rf is calculated and output based on the frequency component f. The frequency component change rate Rf reflects the rapid change of the frequency component in the audio signal, and can be used to identify emotional fluctuations or changes in intonation. For example, when the frequency component change rate Rf is high, it may indicate that the intonation of the voice fluctuates greatly, while when the frequency component change rate Rf is low, the voice may be relatively stable and monotonous. The specific algorithm formula is: , where f(t i ) represents the frequency component at the i-th time point t, f(t i+1 ) represents the frequency component at the i+1th time point;

[0085] S22. Build a cloud database, create a voice storage table and a data storage table, store the voice file in the voice storage table, and store the frequency domain feature vector in the data storage table.

[0086] In this embodiment, the method extracts the speech periodicity index Psig and the frequency component change rate Rf based on the spectrum data, and forms a frequency domain feature vector by summarizing the amplitude value A, the periodicity index Psig and the frequency component change rate Rf. Among them, the periodicity index Psig of the speech is calculated by the autocorrelation function ACF. The method captures the rhythm and naturalness of the speech signal through the change of the amplitude value A, so that the rhythmic characteristics of the speech can be better judged, and the timing characteristics of the speech data are optimized, so that the dialogue robot is more accurate and natural when performing lip movements. By calculating the frequency component change rate Rf, the rapid frequency changes in the audio signal are effectively identified, which provides a basis for capturing the emotional fluctuations and intonation changes in the speech, and improves the synchronization of the dialogue robot's mouth shape movement and speech emotion. These frequency domain feature extraction methods are not common in the prior art, filling the gap in efficient speech feature extraction and emotion matching. At the same time, a cloud database is constructed and a speech storage table and a data storage table are created. The separate storage of voice files and frequency domain feature vectors not only optimizes data management efficiency, but also improves the data retrieval speed, and provides reliable data support for subsequent lip movement matching. Compared with the prior art, the present invention further enhances the scalability and flexibility of the system through the use of cloud database and efficient storage of frequency domain features.

[0087] Example 4

[0088] Specifically: S3 includes S31, S32, S33 and S34;

[0089] S31, based on the user setting the amplitude threshold Ath according to the content of the voice file, the amplitude value A of the frequency domain feature vector is extracted and compared with the set amplitude threshold Ath, and the time point sequence of the voice file is extracted according to the amplitude fluctuation to obtain the amplitude time point sequence Tamp;

[0090] The specific extraction contents are as follows;

[0091] ;

[0092] Where, t i represents the i-th time point t, A(t i ) represents the amplitude value at the i-th time point t;

[0093] The significance of extracting the amplitude time point sequence Tamp is that only the time points greater than the amplitude threshold Ath are extracted, and the time points less than or equal to the amplitude threshold Ath are not extracted. This time point sequence is used to generate trigger points for mouth movements. When the voice is louder, it may correspond to the opening movement of the mouth; when the voice becomes smaller or softer, it may correspond to the closing movement of the mouth.

[0094] S32, based on the periodicity threshold Pth set by the user according to the content of the voice file, the voice periodicity index Psig of the extracted frequency domain feature vector is compared and analyzed with the periodicity threshold Pth, and the periodic time point sequence Tper is obtained according to the time point sequence of the periodically extracted voice file;

[0095] The specific extraction contents are as follows;

[0096] ;

[0097] Where, Psig (t i ) represents the speech periodicity index at the i-th time point t;

[0098] The significance of extracting the periodic time point sequence Tper is that only the time points greater than the periodic threshold Pth are extracted, and the time points less than or equal to the periodic threshold Pth are not extracted. Periodic analysis can help capture the rhythmic features of speech and adjust the timing of mouth movements accordingly. When the speaking speed is faster, the periodicity is shorter and the action time points may be more dense; when the speaking speed is slower, the periodicity is longer and the action time points may be more sparsely distributed.

[0099] S33, based on the user setting the frequency change threshold Rth according to the content of the voice file, the frequency component change rate Rf of the frequency domain feature vector is extracted and compared with the frequency change threshold Rth, and the frequency change time point sequence Tfre is obtained according to the time point sequence of the voice file extracted periodically;

[0100] The specific extraction contents are as follows;

[0101] ;

[0102] Where Rf(ti) represents the rate of change of the frequency component at the i-th time point t;

[0103] The significance of extracting the frequency change time point sequence Tfre is that only the time points with a frequency change greater than the frequency change threshold Rth are extracted, and the time points with a frequency change less than or equal to the frequency change threshold Rth are not extracted. Based on the frequency component change rate Rf, the time points with obvious emotional fluctuations are extracted to adjust the performance of the mouth shape movement to make it more consistent with the emotional changes in the speech;

[0104] S34, based on the extracted amplitude time point sequence Tamp, periodic time point sequence Tper and frequency change time point sequence Tfre, merge and obtain a time point array Ttimes, where the time point array Ttimes=[t1, t2, t3, ..., t n ], where t n Represents the nth time point in the time point array Ttimes;

[0105] The time point array Ttimes is merged using the following formula;

[0106] .

[0107] In this embodiment, the method sets the amplitude threshold Ath according to the content of the voice file by the user, and extracts the amplitude time point sequence Tamp that meets the conditions by comparing and analyzing with the amplitude value A in the frequency domain feature vector. This process can effectively capture the volume fluctuation in the voice signal, and then generate accurate mouth shape action trigger points. For example, when the amplitude of the voice is large, the mouth of the dialogue robot may open; when the amplitude is small, the mouth may be closed. Compared with the traditional method, the present invention sets an adjustable threshold Ath, so that the triggering of the mouth shape action is more matched with the actual volume fluctuation of the voice, thereby improving the naturalness and interactive experience. Based on the setting of the periodic threshold Pth, the periodic time point sequence Tper is extracted, which effectively captures the rhythmic features in the voice. The periodic analysis of the voice helps the dialogue robot accurately judge the fast and slow rhythm of the voice, thereby adjusting the timing of the mouth shape action: when the speech speed is fast, the action time points are more dense, and when the speech speed is slow, the action time points are sparsely distributed. This technical improvement enables the dialogue robot to better synchronize with the rhythm of the voice, avoid unnatural movements or delays, and improve the affinity and naturalness when talking with humans. By setting the frequency change threshold Rth, the frequency change time point sequence Tfre is extracted to identify the emotional fluctuations in the speech. This step can adjust the mouth shape movement of the dialogue robot when the emotional fluctuations are large by analyzing the changes in the frequency components, ensuring that the movement is consistent with the emotional changes, so that the dialogue robot can better fit the real emotional changes when expressing the voice emotions. Finally, by merging the amplitude time point sequence Tamp, the periodic time point sequence Tper and the frequency change time point sequence Tfre, the time point array Ttimes is generated. This array accurately provides the trigger time point of the mouth shape movement for the dialogue robot, ensuring that each movement can be highly synchronized with the various features of the voice signal, such as volume, rhythm, emotional fluctuations, etc., so as to achieve more natural and smooth lip shape movements.

[0108] Example 5

[0109] Specifically: S4 includes S41;

[0110] S41, setting a Bluetooth dual channel, where the Bluetooth dual channel includes a first Bluetooth channel and a second Bluetooth channel;

[0111] The first Bluetooth channel is connected to the dialogue robot Arduino mainboard to transmit the acquired time point array Ttimes to the dialogue robot Arduino mainboard;

[0112] The second Bluetooth channel is connected to the audio device of the dialogue robot to play the voice file through the audio device, thereby playing the voice file synchronously with the mechanical action.

[0113] In this embodiment, the method realizes a high degree of synchronization between the mouth shape movement and voice playback of the dialogue robot by setting a Bluetooth dual channel. The design of the Bluetooth dual channel includes a first Bluetooth channel and a second Bluetooth channel, which are respectively used for connecting to the dialogue robot Arduino mainboard and an audio device. Through this dual-channel transmission mode, the present invention ensures the real-time transmission and accurate execution of the time point array Ttimes, as well as the synchronous playback of the voice file. Specifically, the first Bluetooth channel is responsible for transmitting the acquired time point array Ttimes to the Arduino mainboard to ensure that the servo triggers the mouth shape movement according to the predetermined time point. The second Bluetooth channel plays the voice file through the audio device to ensure the accurate synchronization between the voice and the mouth shape movement. This design not only optimizes the coordination between the mouth shape movement and the voice playback of the dialogue robot, but also avoids the common voice delay or action lag phenomenon in traditional technology. The dual-channel Bluetooth transmission mode significantly improves the synchronization accuracy of the dialogue robot's voice and action. Through a dedicated channel to process control instructions and audio data respectively, the system can efficiently avoid signal congestion and ensure that the voice playback and mouth shape movement are smoother and more natural.

[0114] Example 6

[0115] Specifically: S5 includes S51;

[0116] S51, using the built-in timer of the conversation robot Arduino mainboard, compare the time point in the time point array Ttimes with the current time, and based on the comparison evaluation, send instructions to the servo to control the operation of the servo and drive the opening and closing of the conversation robot's mouth;

[0117] The specific comparison contents are as follows;

[0118] ;

[0119] Among them, t current Indicates the current time in the timer built into the conversational robot Arduino mainboard;

[0120] When the current time t in the timer built into the conversational robot Arduino mainboard current ≥ the nth time point t in the time point array Ttimes n At this time, the opening instruction 1 is sent to drive the servo to control the conversation robot to open its mouth;

[0121] When the current time t in the timer built into the conversational robot Arduino mainboard current<The nth time point t in the time point array Ttimes n At this time, the closing command 0 is sent to drive the servo to control the conversational robot's mouth to close.

[0122] In this embodiment, the method realizes precise control of the mouth movement of the conversation robot by using the timer built into the conversation robot Arduino mainboard. Specifically, the timer is based on the time point in the time point array Ttimes and the current time t in the timer built into the conversation robot Arduino mainboard. current Compare, evaluate and control the operation of the servo in real time. In this way, the system can ensure that the mouth of the conversational robot makes accurate opening and closing movements at the correct time node, enhancing the synchronization between voice and mechanical movements. When comparing the existing technical means, the present invention adopts a timer comparison mechanism, which provides a more accurate and automatic control solution without external intervention compared to the traditional manual control or external synchronization signal method. When the current time t in the timer built into the conversational robot Arduino motherboard is current Greater than or equal to the nth time point t in the time point array Ttimes n , the Arduino motherboard will send an opening command to drive the servo to control the conversation robot to open its mouth; conversely, when the current moment t in the timer built into the conversation robot Arduino motherboard is current Less than the nth time point t in the time point array Ttimes n When the time comes, a closing command is sent to drive the servo to control the mouth to close. This precise timing control ensures that the movement of the conversational robot's mouth is highly matched with the rhythm of the voice, greatly improving the user experience. Compared with the simple opening and closing action control system commonly used at present, the present invention combines a timer with a time point array, which not only improves the accuracy of the action, but also eliminates the need for an external clock or additional synchronization equipment, making the system more efficient and stable. In addition, since the control method can make fine adjustments based on the time point calculated in real time, it further optimizes the coordination of voice and action, avoiding delays or inaccuracies that may occur in traditional methods.

[0123] Example 7

[0124] See also Figure 1 and Figure 2 , a speech and lip-action matching system for a conversational robot, comprising a speech recording module, a feature extraction module, a time point extraction module, a Bluetooth transmission module and a lip-action triggering module;

[0125] The voice recording module records voice through a recording device to obtain a voice file, then uploads the voice file to a cloud server, and pre-processes the voice file in the cloud server to obtain spectrum data of the voice file;

[0126] The feature extraction module extracts frequency domain features through the spectrum data to obtain frequency domain feature vectors, and at the same time builds a cloud database to store the frequency domain feature vectors in the cloud database;

[0127] The time point extraction module extracts the time point array Ttimes of the voice file by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth to compare with all the parameters in the frequency domain feature vector one by one;

[0128] The Bluetooth transmission module transmits the time point array Ttimes and the voice file to the dialogue robot Arduino mainboard and the dialogue robot audio device respectively by setting up a Bluetooth dual channel;

[0129] The lip action trigger module reads the time point parameters in the time point array Ttimes one by one through the timer built into the dialogue robot Arduino mainboard, and sends a command signal to drive the servo to perform the dialogue robot's mouth opening and closing action every time a specified time point is reached.

[0130] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for matching speech and lip movements of a conversational robot, characterized in that: The following steps are involved: S1. Recording voice through a recording device to obtain a voice file, uploading the voice file to a cloud server, and preprocessing the voice file in the cloud server to obtain spectrum data of the voice file; S2, extracting frequency domain features based on the spectrum data to obtain frequency domain feature vectors, and building a cloud database to store the frequency domain feature vectors in the cloud database; S21, extracting frequency domain features based on the spectrum data, obtaining a speech periodicity index Psig and a frequency component change rate Rf, and then summarizing the speech periodicity index Psig and the frequency component change rate Rf with the amplitude value A to obtain a frequency domain feature vector; S211, the speech periodicity index Psig is calculated based on the amplitude value A to obtain the autocorrelation function ACF, and the specific algorithm formula is: ; Among them, A (t i ) represents the amplitude value at the i-th time point t, ACF(t) represents the autocorrelation function at time t, N represents the total number of sampling points, A(t i+1 ) represents the amplitude value of the speech signal at the i+1th time point; Then the calculation output is based on the autocorrelation function ACF. The specific algorithm formula is: , where t max Represents the upper limit of the autocorrelation function ACF at time t; S212, the frequency component change rate Rf is calculated and output based on the frequency component f, and the specific algorithm formula is: , where f(t i ) represents the frequency component at the i-th time point t, f(t i+1 ) represents the frequency component at the i+1th time point; S3, by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth and comparing them one by one with all the parameters in the frequency domain feature vector, extracting the time point array Ttimes of the voice file; S31, based on the user setting the amplitude threshold Ath according to the content of the voice file, the amplitude value A of the frequency domain feature vector is extracted and compared with the set amplitude threshold Ath, and the time point sequence of the voice file is extracted according to the amplitude fluctuation to obtain the amplitude time point sequence Tamp; The specific extraction contents are as follows; ； In the formula, t i represents the i-th time point t, A(t i ) represents the amplitude value at the i-th time point t; S32, based on the periodicity threshold Pth set by the user according to the content of the voice file, the voice periodicity index Psig of the extracted frequency domain feature vector is compared and analyzed with the periodicity threshold Pth, and the periodic time point sequence Tper is obtained according to the time point sequence of the periodically extracted voice file; The specific extraction content is as follows: ； Where, Psig (t i ) represents the speech periodicity index at the i-th time point t; S33, based on the user setting the frequency change threshold Rth according to the content of the voice file, the frequency component change rate Rf of the frequency domain feature vector is extracted and compared with the frequency change threshold Rth, and the frequency change time point sequence Tfre is obtained according to the time point sequence of the voice file extracted periodically; The specific extraction content is as follows: ； Where Rf(ti) represents the rate of change of the frequency component at the i-th time point t; S34, based on the extracted amplitude time point sequence Tamp, periodic time point sequence Tper and frequency change time point sequence Tfre, merge and obtain a time point array Ttimes, wherein the time point array Ttimes=[t1, t2, t3, ..., t n ], where t n Represents the nth time point in the time point array Ttimes; The time point array Ttimes is merged by the following formula; ； S4. Set up Bluetooth dual channels to transmit the time point array Ttimes and the voice file to the conversation robot Arduino mainboard and the conversation robot audio device respectively; S5. Through the built-in timer of the conversation robot Arduino mainboard, read the time point parameters in the time point array Ttimes one by one. Every time a specified time point is reached, a command signal is sent to drive the servo to execute the conversation robot's mouth opening and closing action.

2. A method for matching speech and lip movements of a conversational robot according to claim 1, characterized in that: Said S1 includes S11, S12 and S13; S11. Record the user's voice signal through the applet and APP of the recording device, convert the voice signal into digital voice data, obtain a voice file, and convert the voice file into a standard audio format, wherein the standard audio format is MP3 format and WAV format; S12, uploading the acquired voice file to the cloud server via the communication network, and automatically generating a unique file ID for each uploaded voice file in the cloud server; S13, pre-processing the received voice file on the cloud server to eliminate the interfering voice signal in the voice file, extracting the spectrum data in the voice file through Fourier transform, and then normalizing the spectrum data to eliminate the dimensional influence between the spectrum data; The spectrum data includes an amplitude value A and a frequency component f.

3. The method for matching speech and lip movements of a conversational robot according to claim 1, characterized in that: S22. Build a cloud database, create a voice storage table and a data storage table, store the voice file in the voice storage table, and store the frequency domain feature vector in the data storage table.

4. The method for matching speech and lip movements of a conversational robot according to claim 1, characterized in that: The S4 includes S41; S41, setting a Bluetooth dual channel, wherein the Bluetooth dual channel includes a first Bluetooth channel and a second Bluetooth channel; The first Bluetooth channel transmits the acquired time point array Ttimes to the dialogue robot Arduino mainboard by connecting to the dialogue robot Arduino mainboard; The second Bluetooth channel is connected to the audio device of the dialogue robot to play the voice file through the audio device.

5. The method for matching speech and lip movements of a conversational robot according to claim 1, characterized in that: The S5 includes S51; S51, using the built-in timer of the conversation robot Arduino mainboard, compare the time point in the time point array Ttimes with the current time, and based on the comparison evaluation, send instructions to the servo to control the operation of the servo and drive the opening and closing of the conversation robot's mouth; The specific comparison contents are as follows; ； Among them, t current Indicates the current time in the timer built into the conversational robot Arduino mainboard; When the current time t in the timer built into the conversational robot Arduino mainboard current ≥ the nth time point t in the time point array Ttimes n At this time, the opening instruction 1 is sent to drive the servo to control the conversation robot to open its mouth; When the current time t in the timer built into the conversational robot Arduino mainboard current <The nth time point t in the time point array Ttimes n At this time, the closing command 0 is sent to drive the servo to control the conversational robot's mouth to close.

6. A system for matching speech and lip movements of a conversational robot, applied to a method for matching speech and lip movements of a conversational robot as claimed in any one of claims 1 to 5, characterized in that: It includes a voice recording module, a feature extraction module, a time point extraction module, a Bluetooth transmission module and a lip-action triggering module; The voice recording module records voice through a recording device to obtain a voice file, then uploads the voice file to a cloud server, and pre-processes the voice file in the cloud server to obtain spectrum data of the voice file; The feature extraction module extracts frequency domain features through the spectrum data to obtain frequency domain feature vectors, and at the same time builds a cloud database to store the frequency domain feature vectors in the cloud database; The time point extraction module extracts the time point array Ttimes of the voice file by setting the amplitude threshold Ath, the periodicity threshold Pth and the frequency change threshold Rth to compare with all the parameters in the frequency domain feature vector one by one; The Bluetooth transmission module transmits the time point array Ttimes and the voice file to the dialogue robot Arduino mainboard and the dialogue robot audio device respectively by setting up a Bluetooth dual channel; The lip action trigger module reads the time point parameters in the time point array Ttimes one by one through the timer built into the dialogue robot Arduino mainboard, and sends a command signal to drive the servo to perform the dialogue robot's mouth opening and closing action every time a specified time point is reached.

Images