Multi-language speech recognition and interaction method of humanoid robot

Abstract

The invention discloses a multi-language speech recognition and interaction method of a humanoid robot, particularly relates to the technical field of speech recognition and interaction, and effectively separates background noise and target speech signals through a multi-channel high-sensitivity microphone array in combination with an adaptive noise elimination algorithm, thereby improving the speech signal recognition accuracy in a noisy environment and improving the speech recognition efficiency. A speech speed fluctuation abnormal coefficient, an intonation deviation coefficient and a background noise influence coefficient are calculated through deep analysis of speech signals and background noise feature data of a target interaction object, and a sentence segmentation model is adjusted according to habits of speech speeds, intonation, pauses and the like of different speakers, so that the problem of inaccurate sentence segmentation under different users or contexts is avoided; the sentence segmentation adaptation optimization index is generated according to the sentence segmentation adaptation optimization model, whether parameters of the sentence segmentation model need to be adjusted or not is judged by combining a threshold value, real-time response and optimization are achieved, the situation of misjudgment or missed judgment is reduced, and the smoothness and accuracy of overall interaction are improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition and interaction technology, and more specifically, to a multi-language speech recognition and interaction method for a humanoid robot. Background Art

[0002] With the rapid development of artificial intelligence technology, humanoid robots are increasingly used in the field of multilingual speech recognition and interaction. Humanoid robots collect speech signals from the objects of interaction and identify the interaction needs of the objects of interaction based on the speech signals. In the process of real-time transcription or simultaneous interpretation, how to accurately understand and punctuate in a very short time is a key challenge. The continuity of speech signals means that the sound waveform continues to flow without obvious pauses, and the punctuation points are the key positions for dividing these continuous sound waveforms into meaningful and understandable sentences or phrases. In a noisy environment, background noise may interfere with the accuracy of the punctuation model, resulting in misjudgment or omission of punctuation points. Different speakers may have different speaking speeds, intonations, and pause habits, which makes it difficult for the punctuation model to adapt to the styles of all speakers, which in turn may lead to inaccurate punctuation in real-time transcription and simultaneous interpretation, affecting the user's interactive experience. Summary of the invention

[0003] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a multi-language speech recognition and interaction method of a humanoid robot to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: A multi-language speech recognition and interaction method for a humanoid robot comprises the following steps: Step S1, collecting the target interactive object voice signal through a multi-channel high-sensitivity microphone array, and using an adaptive noise elimination algorithm to separate the target interactive object voice signal from background noise; Step S2, performing feature engineering on the speech signal of the target interactive object and the background noise of the environment in which the target interactive object is located, and extracting speech signal feature data of the target interactive object and background noise feature data of the environment in which the target interactive object is located; Step S3, constructing a sentence segmentation adaptation optimization model according to the speech signal feature data of the target interactive object and the background noise feature data of the environment in which the target interactive object is located, generating a sentence segmentation adaptation optimization index, and evaluating the degree of need for parameter adjustment of the preset sentence segmentation model; Step S4, comparing the sentence segmentation adaptation optimization index with a preset sentence segmentation adaptation optimization index threshold to determine whether to adjust parameters of the preset sentence segmentation model; Step S5: when it is necessary to adjust the parameters of the preset sentence segmentation model, the sentence segmentation adaptation optimization index is used as a parameter tuning influencing factor to perform parameter tuning on the sentence segmentation model based on the parameter adjustment optimization function.

[0005] In a preferred embodiment, the speech signal feature data of the target interactive object includes the speech rate fluctuation abnormality coefficient , Intonation Deviation Coefficient , the background noise characteristic data of the target interactive object's environment includes the background noise influence coefficient .

[0006] In a preferred embodiment, the logic for obtaining the speech rate fluctuation abnormality coefficient is as follows: The target interactive object’s speech activity interval is detected based on the deep learning model, and the target interactive object’s speech signal is analyzed based on the speech activity interval. Segment and obtain a set of segmented speech signals: ,in Indicates the sequence number of each segment of the speech signal after segmentation. , is the total number of segments after the speech signal is segmented; Calculate the speech rate of each speech signal , the expression is as follows ,in represents the number of speech words in the i-th speech signal, Indicates the duration of the i-th speech signal; Summarize the speech rate of each speech signal to get the instantaneous speech rate sequence: ; Calculate the mean instantaneous speech rate , the expression is as follows ; Calculate the standard deviation of instantaneous speech rate , the expression is as follows ; Calculate the relative rate of change of speech rate between adjacent segments , the expression is as follows ,in Indicates the speaking speed of the i+1th speech signal. , Be a minimum value to avoid the denominator being zero; Calculate the mean of the relative rate of change of speech rate , the expression is as follows ; Calculate the standard deviation of the relative rate of change of speech rate , the expression is as follows ; Calculate the abnormal coefficient of speech rate fluctuation , the expression is as follows ,in Represents an adjustment parameter used to balance the impact of instantaneous speech rate fluctuations and fluctuations in the relative rate of change of speech rate on the speech rate fluctuation anomaly coefficient.

[0007] In a preferred embodiment, the logic for obtaining the intonation deviation coefficient is as follows: Voice signal to the target interaction object Divide the frame into fixed frame lengths and perform frame division on each frame of speech signal. Applying a Hamming Window : , , ; To avoid interference from the DC component in the speech signal and ensure that the volatility of the speech signal is analyzed more accurately, the mean value of each frame is removed: ,in represents the speech signal after windowing of the nth frame, Indicates the total number of frames of the windowed speech signal; For each frame of speech signal Perform a fast Fourier transform: ,in The spectrum of each frame of speech signal represents the speech signal The kth frequency component in the frequency domain, , represents an imaginary unit; Calculate the power spectral density of each frame of speech signal , the expression is as follows ; Power Spectral Density Perform peak and valley detection: Peak value meets: ; The valley value satisfies: ; Calculate the peak and valley frequency distribution of the speech signal and the preset reference frequency distribution The degree of deviation between the peak value deviation coefficient is calculated , the expression is as follows ,in represents the frequency position of the mth speech signal peak, , Indicates the total number of speech signal peaks; calculates the valley deviation coefficient , the expression is as follows ,in represents the frequency position of the yth speech signal valley value, , Indicates the total number of valleys in the speech signal; Calculate the intonation deviation coefficient , the expression is as follows .

[0008] In a preferred embodiment, the logic for obtaining the background noise influence coefficient is as follows: According to the background noise signal of the target interaction object's environment Calculate short-time energy , the expression is as follows ,in represents the value of the background noise signal at time h, , is the size of the analysis window; According to the background noise signal of the target interaction object's environment Calculate the short-time zero-crossing rate , the expression is as follows ,in is the sign function of the background noise signal, defined as , the positive value of the background noise signal is mapped to 1, the negative value is mapped to -1, and the zero value is mapped to 0; The test is the sign change of the background noise signal between two consecutive points; if the sign of the current value is different from that of the next value, it means that a zero crossing has occurred, and 1 is returned, otherwise 0 is returned; Calculate the background noise influence coefficient , the expression is as follows .

[0009] In a preferred embodiment, a sentence segmentation adaptation optimization model is constructed according to the speech rate fluctuation abnormality coefficient, intonation deviation coefficient, and background noise influence coefficient, and a sentence segmentation adaptation optimization index is output. The model is based on the following formula , where represents the abnormal coefficient of speech rate fluctuation, represents the intonation deviation coefficient, represents the background noise influence coefficient, They represent the preset proportional coefficients of the speech rate fluctuation abnormality coefficient, the intonation deviation coefficient, and the background noise influence coefficient, respectively, and Both are greater than 0.

[0010] In a preferred implementation, the sentence segmentation adaptation optimization index is compared with a preset sentence segmentation adaptation optimization index threshold to determine whether to adjust the parameters of the preset sentence segmentation model, as follows: If the sentence segmentation adaptation optimization index is greater than the sentence segmentation adaptation optimization index threshold, it is necessary to adjust the parameters of the preset sentence segmentation model to improve the adaptation effect; if the sentence segmentation adaptation optimization index is less than or equal to the sentence segmentation adaptation optimization index threshold, no further parameter adjustment is required.

[0011] In a preferred embodiment, when it is necessary to adjust the parameters of the preset sentence segmentation model, the sentence segmentation adaptation optimization index is used as a parameter tuning influencing factor to perform parameter tuning on the sentence segmentation model based on the parameter adjustment optimization function, as follows: Step A1, the parameter adjustment optimization function It can be defined as: ,in Indicates the sentence segmentation adaptation optimization index. Indicates the threshold of sentence segmentation adaptation optimization index. Represents the segmentation accuracy error of the segmentation model, ,in represents the segmentation position predicted by the segmentation model, is the actual punctuation position, Represents the total duration of the speech signal, ; , express , The preset scaling factor of Step A2: Use gradient descent method to adjust the parameters of the optimization function To minimize: ,in represents the optimized sentence segmentation model parameter set, represents the parameter set of the sentence segmentation model before optimization, Represents the learning rate, which is used to determine the step size of each parameter adjustment. is the parameter adjustment optimization function The gradient of each parameter; The gradient of each parameter is obtained by taking the partial derivative of the parameter adjustment optimization function with respect to each parameter; Step A3, repeating steps A1 and A2 until the output value of the parameter adjustment optimization function is less than or equal to the convergence threshold, then stopping the iteration process and completing the parameter optimization of the sentence segmentation model.

[0012] Technical effects and advantages of the present invention: The present invention effectively separates background noise and target speech signals by combining a multi-channel high-sensitivity microphone array with an adaptive noise elimination algorithm, avoids interference of background noise on a sentence segmentation model, improves the accuracy of speech signal recognition in a noisy environment, and calculates speech rate fluctuation abnormality coefficient, intonation deviation coefficient, and background noise influence coefficient by in-depth analysis of speech signals and background noise feature data of target interactive objects. The sentence segmentation model can be adjusted according to the speech rates, intonations, pauses and other habits of different speakers, so as to better adapt to various interactive environments and avoid inaccurate sentence segmentation problems in different users or contexts. The sentence segmentation adaptation optimization model quickly generates a sentence segmentation adaptation optimization index according to the real-time collected speech signals and background noise features, and determines whether the parameters of the sentence segmentation model need to be adjusted in combination with a threshold, so as to achieve real-time response and optimization, and ensure that the robot can accurately understand the speech content and perform effective transcription or simultaneous interpretation in a short time. Since the optimized sentence segmentation model can more accurately identify and segment sentences and phrases in the speech signal, the user's interactive experience is greatly improved, especially in a multilingual or noisy environment, the robot can accurately understand the user's intention, reduce misjudgment or missed judgment, and improve the fluency and accuracy of the overall interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Example: Figure 1 A multi-language speech recognition and interaction method for a humanoid robot of the present invention is provided, comprising the following steps: Step S1, collecting the target interactive object voice signal through a multi-channel high-sensitivity microphone array, and using an adaptive noise elimination algorithm to separate the target interactive object voice signal from background noise; Step S2, performing feature engineering on the speech signal of the target interactive object and the background noise of the environment in which the target interactive object is located, and extracting speech signal feature data of the target interactive object and background noise feature data of the environment in which the target interactive object is located; Step S3, constructing a sentence segmentation adaptation optimization model according to the speech signal feature data of the target interactive object and the background noise feature data of the environment in which the target interactive object is located, generating a sentence segmentation adaptation optimization index, and evaluating the degree of need for parameter adjustment of the preset sentence segmentation model; Step S4, comparing the sentence segmentation adaptation optimization index with a preset sentence segmentation adaptation optimization index threshold to determine whether to adjust parameters of the preset sentence segmentation model; Step S5, when it is necessary to adjust the parameters of the preset sentence segmentation model, the sentence segmentation adaptation optimization index is used as a parameter tuning influencing factor to perform parameter tuning on the sentence segmentation model based on the parameter adjustment optimization function; Step S1, collecting the target interactive object voice signal through a multi-channel high-sensitivity microphone array, and using an adaptive noise elimination algorithm to separate the target interactive object voice signal from background noise; According to the target application scenario, multiple high-sensitivity microphones are equipped to form an array structure for collecting the voice signal of the target interactive object. The array uses beamforming technology to spatially enhance the sound and improve the collection effect of the voice signal. The difference between the voice signal of the target interactive object and the background noise in the spatial and temporal domains can be captured by the microphone array to achieve optimized collection before separation. The collected voice signal of the target interactive object is processed in real time using an adaptive filtering algorithm (such as the LMS algorithm or the RLS algorithm). The filtering parameters are dynamically adjusted by learning the noise characteristics to separate the voice signal of the target interactive object from the background noise, further optimize the voice quality of the target interactive object, and minimize noise interference. After the background noise is separated, the clean voice signal of the target interactive object is output, while retaining the background noise information for subsequent feature extraction.

[0016] The embodiment of the present invention realizes high-quality acquisition of the target interactive object's voice signal through a multi-channel high-sensitivity microphone array, and effectively separates the target voice from the background noise using an adaptive noise elimination algorithm, laying a reliable foundation for subsequent voice signal processing and sentence segmentation model optimization; Step S2, performing feature engineering on the speech signal of the target interactive object and the background noise of the environment in which the target interactive object is located, and extracting speech signal feature data of the target interactive object and background noise feature data of the environment in which the target interactive object is located; The speech signal feature data of the target interactive object includes a speech rate fluctuation abnormality coefficient , Intonation Deviation Coefficient , the background noise characteristic data of the target interactive object's environment includes the background noise influence coefficient ; The speech rate fluctuation anomaly coefficient is used to measure the fluctuation degree of the target interactive object's speech rate, reflecting the degree of influence of the target interactive object's speech rate fluctuation degree on the sentence segmentation model. By quantifying the fluctuation of the target interactive object's speech rate, it can effectively reveal the consistency of the target interactive object's language expression under different speech rate conditions, thereby providing a key reference for the adaptation of the sentence segmentation model. When the speech rate fluctuation anomaly coefficient is high, it means that the speaker's speech rate changes dramatically, and the sentence segmentation point may be difficult to locate stably. At this time, it is necessary to dynamically adjust the time window size and speech rate adaptation parameters of the sentence segmentation model to enhance the robustness of the model under speech rate changes; when the speech rate fluctuation anomaly coefficient is low, it means that the speech rate is relatively uniform, and the sentence segmentation model can maintain the preset parameters to obtain stable performance. Through this evaluation method based on the speech rate fluctuation anomaly coefficient, the adaptive optimization of the sentence segmentation model can be achieved, ensuring that in real-time speech transcription and interaction, the model can take into account the accuracy and stability under diverse speech rate conditions, and improve the overall performance of speech recognition and interaction.

[0017] The logic for obtaining the speech rate fluctuation abnormal coefficient is as follows: The target interactive object’s speech activity interval is detected based on the deep learning model, and the target interactive object’s speech signal is analyzed based on the speech activity interval. Segment and obtain a set of segmented speech signals: ,in Indicates the sequence number of each segment of the speech signal after segmentation. , is the total number of segments after the speech signal is segmented; It should be noted that the choice of deep learning model can be selected according to actual needs, such as convolutional neural network (CNN), recurrent neural network (RNN), etc.; each segmented speech signal represents a speech activity interval; Calculate the speech rate of each speech signal , the expression is as follows ,in represents the number of speech words in the i-th speech signal, Indicates the duration of the i-th speech signal; Summarize the speech rate of each speech signal to get the instantaneous speech rate sequence: ; Calculate the mean instantaneous speech rate , the expression is as follows ; Calculate the standard deviation of instantaneous speech rate , the expression is as follows ; Calculate the relative rate of change of speech rate between adjacent segments , the expression is as follows ,in Indicates the speaking speed of the i+1th speech signal. , Be a minimum value to avoid the denominator being zero; Calculate the mean of the relative rate of change of speech rate , the expression is as follows ; Calculate the standard deviation of the relative rate of change of speech rate , the expression is as follows ; Calculate the abnormal coefficient of speech rate fluctuation , the expression is as follows ,in represents an adjustment parameter used to balance the influence of instantaneous speech rate fluctuation and speech rate relative change rate fluctuation on the speech rate fluctuation abnormality coefficient; The intonation deviation coefficient is used to measure the degree of deviation between the intonation of the target interactive object and the recognition benchmark of the sentence segmentation model. By quantifying the deviation of the intonation of the target interactive object, the adaptability and accuracy of the sentence segmentation model in diversified voice interaction scenarios can be significantly improved. The intonation deviation coefficient quantifies the degree of deviation of the intonation characteristics of the target speech signal by analyzing the distribution characteristics of the speech signal of the target interactive object in the frequency domain and the degree of deviation of the intonation peak and valley, and compares it with the preset sentence segmentation model recognition benchmark. When the intonation deviation coefficient is large, it indicates that the intonation characteristics of the current speech signal may be significantly different from the intonation adaptation range of the sentence segmentation model. At this time, by taking the intonation deviation coefficient as an important influencing factor for parameter tuning, the parameter settings of the sentence segmentation model are dynamically adjusted, including the sentence segmentation probability threshold and the adaptability of the model sentence segmentation boundary, so as to ensure the accuracy and real-time performance of the model in different intonation environments, avoid sentence segmentation errors caused by differences in intonation characteristics, and finally effectively improve the user experience and interaction efficiency of humanoid robots in real-time speech recognition and interaction scenarios.

[0018] The logic for obtaining the intonation deviation coefficient is as follows: Voice signal to the target interaction object Divide the frame into fixed frame lengths and perform frame division on each frame of speech signal. Applying a Hamming Window : , , ; It should be noted that represents the original speech signal of the nth frame, Represents the window function applied to the nth frame. The function of the window function is to perform windowing on the signal, reduce the edge effect of the signal, and make the signal smoother to facilitate subsequent Fourier transform analysis. represents the speech signal after windowing of the nth frame; To avoid interference from the DC component in the speech signal and ensure that the volatility of the speech signal is analyzed more accurately, the mean value of each frame is removed: ,in represents the speech signal after windowing of the nth frame, Indicates the total number of frames of the windowed speech signal; For each frame of speech signal Perform a fast Fourier transform: ,in The spectrum of each frame of speech signal represents the speech signal The kth frequency component in the frequency domain, , represents an imaginary unit; Calculate the power spectral density of each frame of speech signal , the expression is as follows ; Power Spectral Density Perform peak and valley detection: Peak value meets: ; The valley value satisfies: ; Calculate the peak and valley frequency distribution of the speech signal and the preset reference frequency distribution The degree of deviation between the peak value deviation coefficient is calculated , the expression is as follows ,in represents the frequency position of the mth speech signal peak, , Indicates the total number of speech signal peaks; calculates the valley deviation coefficient , the expression is as follows ,in represents the frequency position of the yth speech signal valley value, , Indicates the total number of valleys in the speech signal; Calculate the intonation deviation coefficient , the expression is as follows ; The background noise impact coefficient is used to measure the degree of influence of environmental noise on the quality of the target interactive object's voice signal and thus affect the accuracy of the sentence segmentation model. Through quantitative analysis of background noise, the impact of noise on the accuracy of sentence segmentation is judged, and it is decided whether the parameters of the sentence segmentation model need to be adjusted to improve its performance. The impact of background noise in speech recognition and sentence segmentation processing is an important factor that cannot be ignored. Especially in noisy environments, the presence of noise often leads to distortion of speech signals, thus affecting the accurate judgment of sentence segmentation points. Evaluating the parameter adjustment requirements of the sentence segmentation model through the background noise impact coefficient is an effective method to optimize the model performance based on the actual impact of noise. The larger the noise impact coefficient, the more serious the interference of noise on the speech signal, and the accuracy of the sentence segmentation model may be more affected, so it is more necessary to adjust the model parameters to adapt to this environmental change. When the background noise is strong, the clarity of the speech signal is reduced, which not only affects the feature extraction of the speech signal (such as the spectrum and fundamental frequency of the audio), but also may cause the sentence segmentation model to misjudge when identifying pauses or sentence segments. Background noise can blur the pause features in speech signals, especially when there are no obvious pauses but changes in tone, noise is more likely to be misidentified as pauses or sentence separation points. These misjudgments directly lead to inaccurate sentence segmentation, which affects the effect of real-time transcription or simultaneous interpretation, causes understanding deviations, and reduces the interactive experience; on the contrary, if the noise has a smaller impact, the need for model adjustment is lower, and the system can maintain its original parameter settings more stably. This method can improve the accuracy of speech recognition in complex environments, ensure the accuracy of sentence segmentation, and thus improve the overall effect of real-time transcription and simultaneous interpretation.

[0019] The logic for obtaining the background noise influence coefficient is as follows: According to the background noise signal of the target interaction object's environment Calculate short-time energy , the expression is as follows ,in represents the value of the background noise signal at time h, , is the size of the analysis window; According to the background noise signal of the target interaction object's environment Calculate the short-time zero-crossing rate , the expression is as follows ,in is the sign function of the background noise signal, defined as , the positive value of the background noise signal is mapped to 1, the negative value is mapped to -1, and the zero value is mapped to 0; The test is the sign change of the background noise signal between two consecutive points; if the sign of the current value is different from that of the next value, it means that a zero crossing has occurred, and 1 is returned, otherwise 0 is returned; Calculate the background noise influence coefficient , the expression is as follows ; Step S3, constructing a sentence segmentation adaptation optimization model according to the speech signal feature data of the target interactive object and the background noise feature data of the environment in which the target interactive object is located, outputting a sentence segmentation adaptation optimization index, and evaluating the degree of need for parameter adjustment of the preset sentence segmentation model; According to the abnormal coefficient of speech rate fluctuation, the coefficient of intonation deviation and the influence coefficient of background noise, a sentence segmentation adaptation optimization model is constructed, and the sentence segmentation adaptation optimization index is output. The model is based on the following formula , where represents the abnormal coefficient of speech rate fluctuation, represents the intonation deviation coefficient, represents the background noise influence coefficient, They represent the preset proportional coefficients of the speech rate fluctuation abnormality coefficient, the intonation deviation coefficient, and the background noise influence coefficient, respectively, and All are greater than 0; It should be noted that before constructing the sentence segmentation adaptation optimization model, it is necessary to ensure that the speech rate fluctuation abnormality coefficient, intonation deviation coefficient, and background noise influence coefficient are all normalized; Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations; It can be seen from the above calculation expression that the larger the speech rate fluctuation abnormality coefficient, the larger the intonation deviation coefficient, and the larger the background noise influence coefficient, the larger the sentence segmentation adaptation optimization index, which means that the probability of adaptation optimization of the sentence segmentation model is greater. Conversely, the smaller the speech rate fluctuation abnormality coefficient, the smaller the intonation deviation coefficient, and the smaller the background noise influence coefficient, the smaller the sentence segmentation adaptation optimization index, which means that the probability of adaptation optimization of the sentence segmentation model is smaller. The sentence segmentation adaptation optimization model constructed based on the speech rate fluctuation abnormality coefficient, intonation deviation coefficient and background noise influence coefficient can effectively adjust the sentence segmentation position to ensure that the speech recognition system can adapt to the characteristics of various speech inputs. Through the optimization of the model, the accuracy of speech recognition can be improved, and the misjudgment and missed judgment can be reduced, thereby improving the overall speech processing performance; based on the calculated sentence segmentation adaptation optimization index, the system will rearrange the sentence segmentation points according to the adjusted points, making the pauses and sentence separations in the speech more natural. The optimized sentence segmentation position will be dynamically adjusted according to factors such as speech rate fluctuations, intonation changes and noise, thereby improving the accuracy of speech recognition; Step S4, comparing the sentence segmentation adaptation optimization index with a preset sentence segmentation adaptation optimization index threshold, and determining whether to adjust the parameters of the preset sentence segmentation model, as follows: If the sentence segmentation adaptation optimization index is greater than the sentence segmentation adaptation optimization index threshold, it means that the current sentence segmentation model adaptation optimization effect has not met expectations. The sentence segmentation accuracy may be unsatisfactory due to factors such as speaking speed, intonation, background noise, etc. At this time, the system needs to adjust the parameters of the preset sentence segmentation model to improve the adaptation effect; if the sentence segmentation adaptation optimization index is less than or equal to the sentence segmentation adaptation optimization index threshold, it means that the system performs well in the current voice input environment and can adapt to the language features of the target interactive object more effectively and the effect is better than expected. At this time, the system does not need to make further parameter adjustments.

[0020] Step S5, when it is necessary to adjust the parameters of the preset sentence segmentation model, the sentence segmentation adaptation optimization index is used as a parameter tuning influencing factor to perform parameter tuning on the sentence segmentation model based on the parameter adjustment optimization function, as follows: Adjust the optimization function based on parameters to minimize the segmentation accuracy error and maximize the adaptation optimization effect; Step A1, the parameter adjustment optimization function It can be defined as: ,in Indicates the sentence segmentation adaptation optimization index. Indicates the threshold of sentence segmentation adaptation optimization index. Represents the segmentation accuracy error of the segmentation model, ,in represents the segmentation position predicted by the segmentation model, is the actual punctuation position, Represents the total duration of the speech signal, ; , express , The preset scaling factor of It should be noted that before outputting the optimized value of the segmentation model parameter set, it is necessary to ensure that , All were normalized; , Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations; It should be noted that the sentence segmentation model parameter set includes but is not limited to sentence segmentation interval, speech feature tolerance, noise tolerance, and model response speed; Step A2: Use gradient descent method to adjust the parameters of the optimization function To minimize: ,in represents the optimized sentence segmentation model parameter set, represents the parameter set of the sentence segmentation model before optimization, Represents the learning rate, which is used to determine the step size of each parameter adjustment. is the parameter adjustment optimization function The gradient of each parameter; The gradient of each parameter is obtained by taking the partial derivative of the parameter adjustment optimization function with respect to each parameter; Step A3, repeating steps A1 and A2 until the output value of the parameter adjustment optimization function is less than or equal to the convergence threshold, then stopping the iteration process, and completing the parameter optimization of the sentence segmentation model; The present invention effectively separates background noise and target speech signals by combining a multi-channel high-sensitivity microphone array with an adaptive noise elimination algorithm, avoids interference of background noise on a sentence segmentation model, improves the accuracy of speech signal recognition in a noisy environment, and calculates speech rate fluctuation abnormality coefficient, intonation deviation coefficient, and background noise influence coefficient by in-depth analysis of speech signals and background noise feature data of target interactive objects. The sentence segmentation model can be adjusted according to the speech rates, intonations, pauses and other habits of different speakers, so as to better adapt to various interactive environments and avoid inaccurate sentence segmentation problems in different users or contexts. The sentence segmentation adaptation optimization model quickly generates a sentence segmentation adaptation optimization index according to the real-time collected speech signals and background noise features, and determines whether the parameters of the sentence segmentation model need to be adjusted in combination with a threshold, so as to achieve real-time response and optimization, and ensure that the robot can accurately understand the speech content and perform effective transcription or simultaneous interpretation in a short time. Since the optimized sentence segmentation model can more accurately identify and segment sentences and phrases in the speech signal, the user's interactive experience is greatly improved, especially in a multilingual or noisy environment, the robot can accurately understand the user's intention, reduce misjudgment or missed judgment, and improve the fluency and accuracy of the overall interaction.

[0021] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0022] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-language speech recognition and interaction method for a humanoid robot, characterized in that: The steps include: Step S1, collecting the target interactive object voice signal through a multi-channel high-sensitivity microphone array, and using an adaptive noise elimination algorithm to separate the target interactive object voice signal from background noise; Step S2, performing feature engineering on the speech signal of the target interactive object and the background noise of the environment in which the target interactive object is located, and extracting speech signal feature data of the target interactive object and background noise feature data of the environment in which the target interactive object is located; Step S3, constructing a sentence segmentation adaptation optimization model according to the speech signal feature data of the target interactive object and the background noise feature data of the target interactive object's environment, generating a sentence segmentation adaptation optimization index, and evaluating the degree of need for parameter adjustment of the preset sentence segmentation model; Step S4, comparing the sentence segmentation adaptation optimization index with a preset sentence segmentation adaptation optimization index threshold to determine whether to adjust parameters of the preset sentence segmentation model; Step S5: when it is necessary to adjust the parameters of the preset sentence segmentation model, the sentence segmentation adaptation optimization index is used as a parameter tuning influencing factor to perform parameter tuning on the sentence segmentation model based on the parameter adjustment optimization function.

2. The multi-language speech recognition and interaction method of a humanoid robot according to claim 1, characterized in that: The speech signal feature data of the target interactive object includes a speech rate fluctuation abnormality coefficient , Intonation Deviation Coefficient , the background noise characteristic data of the target interactive object's environment includes the background noise influence coefficient .

3. The multi-language speech recognition and interaction method of a humanoid robot according to claim 2, characterized in that: The logic for obtaining the speech rate fluctuation abnormal coefficient is as follows: The target interactive object’s speech activity interval is detected based on the deep learning model, and the target interactive object’s speech signal is analyzed based on the speech activity interval. Segment and obtain a set of segmented speech signals: ,in Indicates the sequence number of each segment of the speech signal after segmentation. , is the total number of segments after the speech signal is segmented; Calculate the speech rate of each speech signal , the expression is as follows ,in represents the number of speech words in the i-th speech signal, Represents the duration of the i-th speech signal; Summarize the speech rate of each speech signal to get the instantaneous speech rate sequence: ; Calculate the mean instantaneous speech rate , the expression is as follows ; Calculate the standard deviation of instantaneous speech rate , the expression is as follows ; Calculate the relative rate of change of speech rate between adjacent segments , the expression is as follows ,in Indicates the speaking speed of the i+1th speech signal. , Be a minimum value to avoid the denominator being zero; Calculate the mean of the relative rate of change of speech rate , the expression is as follows ; Calculate the standard deviation of the relative rate of change of speech rate , the expression is as follows ; Calculate the abnormal coefficient of speech rate fluctuation , the expression is as follows ,in Represents an adjustment parameter used to balance the impact of instantaneous speech rate fluctuations and speech rate relative change rate fluctuations on the speech rate fluctuation anomaly coefficient.

4. The multi-language speech recognition and interaction method of a humanoid robot according to claim 2, characterized in that: The logic for obtaining the intonation deviation coefficient is as follows: Voice signal to the target interaction object Divide the frame into fixed frame lengths and perform frame division on each frame of speech signal. Applying a Hamming Window : , , ; To avoid interference from the DC component in the speech signal and ensure that the volatility of the speech signal is analyzed more accurately, the mean value of each frame is removed: ,in represents the speech signal after windowing of the nth frame, Indicates the total number of frames of the windowed speech signal; For each frame of speech signal Perform a fast Fourier transform: ,in The spectrum of each frame of speech signal represents the speech signal The kth frequency component in the frequency domain, , represents an imaginary unit; Calculate the power spectral density of each frame of speech signal , the expression is as follows ; Power Spectral Density Perform peak and valley detection: Peak value meets: ; The valley value satisfies: ; Calculate the peak and valley frequency distribution of the speech signal and the preset reference frequency distribution The degree of deviation between the peak value deviation coefficient is calculated , the expression is as follows ,in represents the frequency position of the mth speech signal peak, , Indicates the total number of speech signal peaks; Calculate the valley deviation coefficient , the expression is as follows ,in represents the frequency position of the yth speech signal valley value, , Indicates the total number of valleys in the speech signal; Calculate the intonation deviation coefficient , the expression is as follows .

5. The multi-language speech recognition and interaction method of a humanoid robot according to claim 2, characterized in that: The logic for obtaining the background noise influence coefficient is as follows: According to the background noise signal of the target interaction object's environment Calculate short-time energy , the expression is as follows ,in represents the value of the background noise signal at time h, , is the size of the analysis window; According to the background noise signal of the target interaction object's environment Calculate the short-time zero-crossing rate , the expression is as follows ,in is the sign function of the background noise signal, defined as , the positive value of the background noise signal is mapped to 1, the negative value is mapped to -1, and the zero value is mapped to 0; The test is the sign change of the background noise signal between two consecutive points; if the sign of the current value is different from that of the next value, it means that a zero crossing has occurred, and 1 is returned, otherwise 0 is returned; Calculate the background noise influence coefficient , the expression is as follows .

6. The multi-language speech recognition and interaction method of a humanoid robot according to claim 2, characterized in that: According to the abnormal coefficient of speech rate fluctuation, the coefficient of intonation deviation and the influence coefficient of background noise, a sentence segmentation adaptation optimization model is constructed, and the sentence segmentation adaptation optimization index is output. The model is based on the following formula , where represents the abnormal coefficient of speech rate fluctuation, represents the intonation deviation coefficient, represents the background noise influence coefficient, They represent the preset proportional coefficients of the speech rate fluctuation abnormality coefficient, the intonation deviation coefficient, and the background noise influence coefficient, respectively, and Both are greater than 0.

7. The multi-language speech recognition and interaction method of a humanoid robot according to claim 6, characterized in that: The sentence segmentation adaptation optimization index is compared with the preset sentence segmentation adaptation optimization index threshold to determine whether to adjust the parameters of the preset sentence segmentation model, as follows: If the sentence segmentation adaptation optimization index is greater than the sentence segmentation adaptation optimization index threshold, it is necessary to adjust the parameters of the preset sentence segmentation model to improve the adaptation effect; if the sentence segmentation adaptation optimization index is less than or equal to the sentence segmentation adaptation optimization index threshold, no further parameter adjustment is required.

8. The multi-language speech recognition and interaction method of a humanoid robot according to claim 7, characterized in that: When it is necessary to adjust the parameters of the preset segmentation model, the segmentation adaptation optimization index is used as the parameter tuning influencing factor to tune the parameters of the segmentation model based on the parameter adjustment optimization function, as follows: Step A1, the parameter adjustment optimization function It can be defined as: ,in Indicates the sentence segmentation adaptation optimization index. Indicates the threshold of sentence segmentation adaptation optimization index. Represents the segmentation accuracy error of the segmentation model, ,in represents the segmentation position predicted by the segmentation model, is the actual punctuation position, Indicates the total duration of the speech signal, ; , express , The preset scaling factor of Step A2: Use gradient descent method to adjust the parameters of the optimization function To minimize: ,in represents the optimized sentence segmentation model parameter set, represents the parameter set of the sentence segmentation model before optimization, Represents the learning rate, which is used to determine the step size of each parameter adjustment. is the parameter adjustment optimization function The gradient of each parameter; The gradient of each parameter is obtained by taking the partial derivative of the parameter adjustment optimization function with respect to each parameter; Step A3, repeating steps A1 and A2 until the output value of the parameter adjustment optimization function is less than or equal to the convergence threshold, then stopping the iteration process and completing the parameter optimization of the sentence segmentation model.

Images