Language recognition method and system based on artificial intelligence auxiliary conversation scene

Abstract

The invention discloses a language recognition method and system based on an artificial intelligence auxiliary conversation scene, and belongs to the technical field of language recognition, and the method specifically comprises the steps: collecting an audio signal and video data through a sensor, carrying out the noise filtering of the audio signal and the video data in real time, carrying out the preprocessing of the audio signal after noise filtering, and carrying out the recognition of the voice. Separating voices in the preprocessed audio signals, segmenting talkers, extracting voice features of each talker, performing voice decoding according to the voice features, outputting a final text, and analyzing intentions of the talkers according to the final text and context output; facial expressions and body languages of the talker are captured through the camera, and the real semantics of the dialogue can be more comprehensively understood in combination with tone and emotion information in the voice signal.

Description

Technical Field

[0001] The present invention belongs to the technical field of language recognition, and in particular to a language recognition method and system based on artificial intelligence-assisted conversation scenarios. Background Art

[0002] With the rapid development of artificial intelligence (AI) and natural language processing (NLP) technologies, speech recognition and language understanding systems have been widely used in scenarios such as smart assistants, customer service, and smart device interaction. However, existing technologies still face the following problems and challenges in actual conversation scenarios: 1) Environmental noise interference. In complex real-world environments, such as conference rooms, open offices or public places, background noise (such as crowd noise and traffic noise) interferes greatly with voice signals. 2) Insufficient understanding of dynamic context. In conversation scenarios, semantics is often affected by context, situation and the intention of the speaker. For example, the same sentence may have completely different meanings in different contexts. 3) Weak ability to capture emotions and intentions. Language is not just an expression of text or voice, but also contains implicit emotional information and intentions. For example, changes in tone and speech speed can convey emotions, and complex intentions (such as requests, suggestions, and commands) may be implied behind text expressions. 4) Insufficient integration of multimodal information. In actual conversations, language information is often closely related to other modalities (such as facial expressions, body language, and environmental features), but existing technologies mostly focus on a single modality (such as voice or text) and ignore the synergy of multimodal information.

[0003] For example, a Chinese patent with authorization announcement number CN107704450B discloses a natural language recognition device and a natural language recognition method, including an input device, a processing device and a storage device. The input device is used to provide natural language data. The storage device is used to store multiple program modules. The multiple program modules include a grammatical analysis module. The processing device executes the grammatical analysis module to analyze the natural language data through a formal grammatical model and generate multiple string data. When the processing device determines that at least one of the multiple string data meets the preset valid grammatical conditions, the processing device determines that at least one of the string data is intention data, and the processing device outputs a corresponding response signal based on the intention data.

[0004] For example, a Chinese patent with the authorization announcement number CN113128216B discloses a language recognition method, system and device. In the language recognition method of the invention, a standard vocabulary library storing standard indicator names is established for the professional and technical fields. After obtaining the indicator class description of the sentence, it is first determined whether there is a completely corresponding standard indicator name in the standard vocabulary library. If so, the standard indicator name is used as the indicator class information of the sentence, so as to improve the language recognition in the professional field; if not, the word segmentation list is obtained after combining the standard vocabulary library and the conventional vocabulary library, and the word segmentation list is parsed or the indicator class information is obtained. Since the word segmentation is performed in combination with the standard vocabulary library and the conventional vocabulary library, compared with the word segmentation performed by relying solely on the conventional vocabulary library, the word segmentation of the invention is closer to the similar words in the field, so that the indicator class information obtained after the word segmentation is more accurate and the language recognition is more precise.

[0005] The defects of the above patents: insufficient understanding of dynamic context and weak ability to capture emotions and intentions. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention proposes a language recognition method and system based on artificial intelligence-assisted conversation scenarios, which improves the recognition accuracy of conversational speech through adaptive recognition of the environment, context and talker habits.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The language recognition method based on artificial intelligence assisted conversation scenario includes the following specific steps:

[0009] Step S1: using sensors to collect audio signals and video data, and filtering the audio signals and video data for noise in real time;

[0010] Step S2: preprocessing the audio signal after noise filtering;

[0011] Step S3: separating the speech in the preprocessed audio signal and segmenting the speaker;

[0012] Step S4: extracting the speech features of each speaker, performing speech decoding according to the speech features, and outputting the final text;

[0013] Step S5: Analyze the speaker's intention based on the final output text and context.

[0014] Specifically, the specific steps of step S1 are:

[0015] Step S101: Set the collected audio signal as x(t), the audio signal is divided into a target signal S(t) and a noise signal N(t), perform short-time Fourier transform on the audio signal x(t), and obtain an audio time-frequency domain signal X(k,n), where k represents a frequency index and n represents a time index;

[0016] Step S102: Calculate the signal-to-noise ratio (SNR) of the audio time-frequency domain signal, and dynamically adjust the noise reduction parameters according to the calculated SNR to enhance the audio time-frequency domain signal. The specific formula is:

[0017] |Y(k,n)| 2 =|X(k,n)| 2 -α×|N(k,n)| 2 ,

[0018] Among them, Y(k,n) represents the enhanced audio time-frequency domain signal, N(k,n) represents the time-frequency domain signal of the noise in the audio, α represents the noise reduction coefficient, and |·| represents the absolute value function.

[0019] Specifically, the specific steps of step S3 are:

[0020] Step S301: Generate a speech separation mask using a speech separation network trained by self-supervised learning. The specific formula is:

[0021] M i (k,n)=σ(f DPRNN (Y'(k,n),Θ)),

[0022] Among them, M i (k,n) represents the time-frequency mask of the ith speaker, σ(·) represents the activation function, and f DPRNN (·) represents the DPRNN network output, Θ represents the network model parameters, and Y'(k,n) represents the preprocessed audio signal;

[0023] Step S302: Use the speech separation mask to weight the time-frequency domain representation of the preprocessed audio signal to obtain the separated speech time-frequency domain signal of each speaker. The specific formula is: Represents the time-frequency domain signal of the separated speech of the i-th speaker. Through the inverse short-time Fourier transform, the time-domain signal of the separated speech of the i-th speaker is obtained

[0024] Step S303: Extract the time-frequency features Fp of the separated speech signals of each speaker i , Fp i ={fp1,fp2,...,fp m}, fp mrepresents the mth time-frequency feature of the speech signal separated by the ith speaker, and uses the pre-trained speaker embedding model to embed the time-frequency feature Fp i Mapped to an embedding vector e of fixed dimension i ;

[0025] Step S304: Use time series segmentation and clustering algorithms to embed the vector e i Perform temporal clustering to generate paragraph tags {dl i ,dl i+1 ,...,dl sl},dl sl Indicates the slth paragraph mark generated;

[0026] Step S305: Combine the separated speech signal and time series clustering to output the speech segment and corresponding label of the i-th speaker: Among them, C i represents the identifier of the i-th speaker, and Z represents the number of speakers.

[0027] Specifically, the specific steps of step S4 are:

[0028] Step S401: Analyze the conversation environment and determine the context category P (Lb|F env );

[0029] Step S402: Map the speech segment of the i-th speaker to the logarithmic scale f of human hearing through the Mel frequency scale mel , take the logarithmic scale f mel After logarithm, discrete cosine transform is performed to extract static features C i [n], and extract the first-order dynamic feature ΔC based on the static feature i [n] and the second-order dynamic characteristic Δ 2 C i [n], the specific formula is:

[0030]

[0031] Among them, C i [n+tc] represents the static features of the n+tcth frame, tc represents the time difference, that is, the offset relative to the current frame, and Nc represents the half width of the time window;

[0032] Step S403: Extract the static feature C i [n], first-order dynamic characteristic ΔC i [n] and the second-order dynamic characteristic Δ 2 C i [n] is input into the bidirectional RNN model for time series modeling to capture the dependencies between the previous and next contexts and obtain the final feature h sj ;

[0033] Step S404: Use the attention mechanism to weight the final features to obtain weighted speech features Highlight the temporal features most relevant to the current recognition task;

[0034] Step S405: Using a model based on sequence-to-sequence structure, combining the attention mechanism and language model to transform speech features Mapped to the corresponding text sequence to get the final text.

[0035] Specifically, the specific steps of step S401 are:

[0036] Step S4011: Perform multimodal analysis on the speech segment and video data of the i-th speaker to obtain environmental features, including: audio features f audio , video features vidio and semantic features f semantic , and construct the environmental feature vector F env , F env =[f audio ,f vidio ,f semantic ];

[0037] Step S4012: Use a deep neural network to analyze the environmental feature vector F env Classify and output the context category Lb. The specific classification formula is:

[0038] P(Lb|F env )=softmax(W env ×F env +b env ),

[0039] Among them, P(Lb|F env ) represents the context category of the speech segment of the i-th speaker, softmax represents the classification function, W env represents the weight coefficient, b env represents a constant term;

[0040] Step S4013: Construct an emotional feature vector F by analyzing the speaker's intonation, rhythm and semantics emotion , F emotion =[f frame ,ZCR,f pitch ,f tempo ], f frame represents the short-time energy of the speech segment of the i-th speaker, ZCR represents the zero crossing rate of the speech segment of the i-th speaker, and f pitch represents the fundamental frequency of the speech segment of the i-th speaker, f tempoRepresents the rhythm of the speech segment of the i-th speaker and judges the emotional state. The specific formula is:

[0041] P(E|F emotion )=softmax(W emotion ×F emotion +b emotion ),

[0042] Among them, P(E|F emotion ) represents the emotion category of the speech segment of the i-th speaker, softmax represents the classification function, and W eemotion represents the weight coefficient, b emotion Represents a constant term.

[0043] Specifically, the specific steps of step S5 are:

[0044] Step S501: Set the intention as a hidden state I y , the comprehensive feature of the i-th speaker is F comb , infer the intention sequence I1,I2,……,I js , the context is collected to calculate the intention probability, the specific formula is: Among them, P(I js |F comb ) represents the intention probability, that is, the intention I of js at the current time js In the comprehensive feature F comb The probability of js |I js-1 ) represents the intention state transition probability, that is, from the previous state I js-1 To current state I js The probability, P(F comb |I js ) represents the comprehensive feature F comb In the current intention I js The probability of the following;

[0045] Step S502: Based on the results of multiple rounds of dialogue, define a reward function R(I js ,A),R(I js ,A)=λ1×Prec(I js )+λ2×Rele(A)-λ3×Pena(A), where A represents the response action, Prec(I js ) represents the accuracy of the recognized intention, Rele(A) represents the response action A and the intention I js Pena(A) represents the negative impact of wrong actions on performance, and λ1, λ2 and λ3 represent weight parameters;

[0046] Step S503: Optimize the strategy through deep reinforcement learning to dynamically adjust the talker's intention in multiple rounds of dialogue. The specific formula is:

[0047]

[0048] Among them, P(I js |O 1:js ) represents the speaker's intention after dynamic adjustment of multiple rounds of dialogue, O 1:js represents the observation characteristics of the current round, P(O 1:js |I js ) represents the observed feature O of the current round 1:js In the intention js The probability of js |O 1:js-1 ) represents the speaker’s intention after the dynamic adjustment of the previous round of dialogue, P(O 1:js ) represents the normalization constant of the observed features of the current round.

[0049] Specifically, the preprocessing in step S2 includes: pre-emphasis, frame segmentation and normalization processing to obtain a preprocessed audio signal Y'(k,n);

[0050] The pre-emphasis enhances the high-frequency components through a high-pass filter and reduces the masking effect of the low-frequency components on the signal; the frame segmentation segments the speech signal into overlapping frames of length Ns; the normalization processing adjusts the signal amplitude to a fixed range.

[0051] A language recognition system based on artificial intelligence-assisted conversation scenarios, used to implement the language recognition method based on artificial intelligence-assisted conversation scenarios, including: a data acquisition module, a preprocessing module, a separation and segmentation module, a speech decoding module and an intention analysis module;

[0052] The data acquisition module is used to collect audio signals and video data using sensors and perform noise filtering on the audio signals and video data in real time;

[0053] The preprocessing module is used to preprocess the audio signal after noise filtering;

[0054] The separation and segmentation module is used to separate the speech in the preprocessed audio signal and segment the speaker;

[0055] The speech decoding module is used to extract the speech features of each speaker, perform speech decoding according to the speech features, and output the final text;

[0056] The intention analysis module is used to analyze the intention of the speaker based on the final output text and context.

[0057] Specifically, the separation and segmentation module includes a speech signal separation unit, a time-frequency feature extraction unit and a time series clustering unit;

[0058] The speech signal separation unit is used to weight the time-frequency domain representation of the preprocessed audio signal using a separation mask to extract a separated speech signal of each talker;

[0059] The time-frequency feature extraction unit is used to extract the time-frequency features of the separated speech signals of each talker;

[0060] The temporal clustering unit is used to perform temporal clustering on the embedded vectors using a temporal segmentation and clustering algorithm, and output the speech segments and corresponding labels of each speaker.

[0061] Specifically, the speech decoding module includes: an environment analysis unit, a feature extraction unit and a text output unit;

[0062] The environment analysis unit is used to analyze the conversation environment and determine the context category of the speaker;

[0063] The feature extraction unit is used for time series modeling, capturing the dependencies between the previous and next contexts, and extracting the final features;

[0064] The text output unit is used to output the text corresponding to the speaker by adopting a model based on a sequence-to-sequence structure in combination with an attention mechanism and a language model.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. The present invention proposes a language recognition method based on artificial intelligence-assisted conversation scenarios, which can separate the target speech and background noise in real time, dynamically adjust the noise reduction parameters to improve the signal-to-noise ratio. Even in complex noise environments (such as meetings or public places), the system can still accurately capture the talker's voice signal, greatly improving the robustness of speech recognition.

[0067] 2. The present invention proposes a language recognition method based on artificial intelligence-assisted conversation scenarios, which fully integrates voice, text, vision and environmental data. It captures the facial expressions and body language of the talker through a camera, and combines the tone and emotional information in the voice signal to more comprehensively understand the true semantics of the conversation.

[0068] 3. The present invention proposes a language recognition method based on artificial intelligence-assisted conversation scenarios, which integrates voice features (such as intonation and speaking speed), semantic features (such as syntactic analysis and keyword extraction) and emotional features (such as emotion classification and emotion intensity) for deep fusion. By introducing an intention analysis module optimized by reinforcement learning, the system can accurately identify complex emotions (such as anger, joy, sadness) and implicit intentions (such as suggestions, requests, and denials). BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flow chart of a language recognition method based on an artificial intelligence-assisted conversation scenario provided by the present invention;

[0070] Figure 2 This is an architecture diagram of the language recognition system based on artificial intelligence-assisted conversation scenarios provided by the present invention. DETAILED DESCRIPTION

[0071] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, in the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate the orientation or position relationship based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "No. 1", "No. 2" and "No. 3" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The present invention is further explained below in conjunction with specific embodiments.

[0072] Example 1

[0073] See also Figure 1 , an embodiment provided by the present invention: a language recognition method based on artificial intelligence assisted conversation scenario, comprising the following specific steps:

[0074] Step S1: using sensors to collect audio signals and video data, and performing noise filtering on the audio signals and video data in real time;

[0075] Sensors include: microphone array and camera;

[0076] The specific steps of step S1 are:

[0077] Step S101: Set the collected audio signal as x(t), the audio signal is divided into a target signal S(t) and a noise signal N(t), perform short-time Fourier transform on the audio signal x(t), and obtain an audio time-frequency domain signal X(k,n), where k represents a frequency index and n represents a time index;

[0078] Step S102: Calculate the signal-to-noise ratio (SNR) of the audio time-frequency domain signal, and dynamically adjust the noise reduction parameters according to the calculated SNR to enhance the audio time-frequency domain signal. The specific formula is:

[0079] |Y(k,n)| 2 =|X(k,n)| 2 -α×|N(k,n)| 2,

[0080] Among them, Y(k,n) represents the enhanced audio time-frequency domain signal, N(k,n) represents the time-frequency domain signal of the noise in the audio, α represents the noise reduction coefficient, and |·| represents the absolute value function.

[0081] By calculating the ratio of signal to noise power, the signal quality is reflected in real time, the noise is weakened within a specific frequency range, the target signal is kept intact, and the noise reduction coefficient α is adaptively adjusted according to the real-time environmental noise level to avoid speech distortion caused by excessive noise reduction.

[0082] Step S2: preprocessing the audio signal after noise filtering;

[0083] The preprocessing in step S2 includes: preprocessing the audio signal after noise filtering, including: pre-emphasis, frame segmentation and normalization processing, to obtain a preprocessed audio signal Y'(k,n);

[0084] In this embodiment, pre-emphasis enhances high-frequency components through a high-pass filter, reduces the masking effect of low-frequency components on the signal, and reduces the impact of environmental noise. The human ear is more sensitive to high-frequency signals. Pre-emphasis simulates this auditory perception characteristic and helps to enhance the significance of high-frequency features. The speech signal is a non-stationary signal. The signal is divided into small segments (frames) through short-time analysis. The signal can be regarded as stationary in each frame. The speech signal is divided into overlapping frames of length Ns (usually Ns is 20 to 40 milliseconds), which is conducive to short-time feature extraction. Normalization adjusts the signal amplitude to a fixed range, reduces the impact of different volumes, improves the robustness of the features, and is more robust to different recording conditions.

[0085] Step S3: separating the speech in the preprocessed audio signal and segmenting the speaker;

[0086] The specific steps of step S3 are:

[0087] Step S301: Generate a speech separation mask using a speech separation network trained by self-supervised learning. The specific formula is:

[0088] M i (k,n)=σ(f DPRNN (Y'(k,n),Θ)),

[0089] Among them, M i (k,n) represents the time-frequency mask of the ith speaker, σ(·) represents the activation function, usually the Sigmoid function, which is used to normalize the mask value, and f DPRNN (·) represents the DPRNN network output, Θ represents the network model parameters;

[0090] The key to speech separation is to extract the spectral information of the target speech signal while suppressing other interference (such as the voices of other speakers). The dual-path recurrent neural network (DPRNN) models the input time-frequency signal and dynamically generates a separation mask based on the speaker characteristics and context. The activation function σ(·) maps the network output to the interval 0 to 1.

[0091] Step S302: Use the speech separation mask to weight the time-frequency domain representation of the preprocessed audio signal to obtain the separated speech time-frequency domain signal of each speaker. The specific formula is: Represents the time-frequency domain signal of the separated speech of the i-th speaker. Through the inverse short-time Fourier transform, the time-domain signal of the separated speech of the i-th speaker is obtained

[0092] After the mask is generated, the target speech needs to be separated from other interference signals and the result converted back to the time domain signal;

[0093] Step S303: Extract the time-frequency features Fp of the separated speech signals of each speaker i , Fp i ={fp1,fp2,...,fp m}, fp m represents the mth time-frequency feature of the speech signal separated by the ith speaker, and uses the pre-trained speaker embedding model to embed the time-frequency feature Fp i Mapped to an embedding vector e of fixed dimension i ;

[0094] Types of time-frequency features include: Mel-frequency cepstral coefficients (MFCC), time domain features (such as energy, zero-crossing rate, etc., which mainly reflect the instantaneous characteristics of speech) and frequency domain features (such as spectral centroid, spectral bandwidth, etc., which capture the acoustic characteristics of the speaker);

[0095] Speech features are an important basis for distinguishing different speakers. The combination of time domain and frequency domain features can improve the accuracy of speaker segmentation.

[0096] Step S304: Use time series segmentation and clustering algorithms to embed the vector e i Perform temporal clustering to generate paragraph tags {dl i ,dl i+1 ,...,dl sl},dl sl Indicates the slth paragraph mark generated;

[0097] After separating the talker speech signals, it is necessary to cluster the speech segments of different talkers;

[0098] Step S305: Combine the separated speech signal and time series clustering to output the speech segment and corresponding label of the i-th speaker: Among them, C i represents the identifier of the i-th speaker, and Z represents the number of speakers.

[0099] The final output needs to combine the separated speech signals and timing information to generate a complete talker segmentation result.

[0100] Step S4: extracting the speech features of each speaker, performing speech decoding according to the speech features, and outputting the final text;

[0101] The specific steps of step S4 are:

[0102] Step S401: Analyze the conversation environment and determine the context category P (Lb|F env );

[0103] Step S402: Map the speech segment of the i-th speaker to the logarithmic scale f of human hearing through the Mel frequency scale mel , take the logarithmic scale f mel After logarithm, discrete cosine transform is performed to extract static features C i [n], and extract the first-order dynamic feature ΔC based on the static feature i [n] and the second-order dynamic characteristic Δ 2 C i [n], the specific formula is:

[0104]

[0105] Among them, C i [n+tc] represents the static features of the n+tcth frame, tc represents the time difference, that is, the offset relative to the current frame, and Nc represents the half width of the time window, which determines the number of frames used to calculate the dynamic features;

[0106] In this embodiment, the first-order dynamic characteristic ΔC i [n] For each static feature C i The time variation trend of [n] is calculated, which is equivalent to the first-order difference of the frame features. It describes the variation trend of the features over time and helps the system capture the dynamic information of the speech (such as changes in pitch and speech speed). It is very important for the temporal modeling of the speech. The second-order dynamic feature Δ 2 C i [n] represents the acceleration of the dynamic feature over time, which helps capture complex speech dynamic changes (such as acceleration or deceleration of pronunciation).

[0107] Step S403: Extract the static feature C i [n], first-order dynamic characteristic ΔCi [n] and the second-order dynamic characteristic Δ 2 C i [n] is input into the bidirectional RNN model for time series modeling to capture the dependencies between the previous and next contexts and obtain the final feature h sj ;

[0108] The final feature identification is: represents the backward hidden state, represents the forward hidden state;

[0109] Step S404: Use the attention mechanism to weight the final features to obtain weighted speech features Highlight the temporal features most relevant to the current recognition task;

[0110] Step S405: Using a model based on sequence-to-sequence structure, combining the attention mechanism and language model to transform speech features Mapped to the corresponding text sequence to get the final text.

[0111] The specific steps of step S401 are:

[0112] Step S4011: Perform multimodal analysis on the speech segment and video data of the i-th speaker to obtain environmental features, including: audio features f audio , video features vidio and semantic features f semantic , and construct the environmental feature vector F env , F env =[f audio ,f vidio ,f semantic ];

[0113] Audio features are used to analyze speech speed, pitch, and background sound type; video features are used to detect scene content (such as tables, whiteboards, and computer screens) through cameras; semantic features are used to extract keywords and determine the topic of conversation;

[0114] Step S4012: Use a deep neural network to analyze the environmental feature vector F env Classify and output the possible context category Lb. The specific classification formula is:

[0115] P(Lb|F env )=softmax(W env ×F env +b env ),

[0116] Among them, P(Lb|F env ) represents the context category of the speech segment of the i-th speaker, softmax represents the classification function, Wenv represents the weight coefficient, b env represents a constant term;

[0117] The weight coefficient and the constant term are set according to the actual situation or obtained by personnel in this field through a large number of simulation experiments. The possible values ​​of the context category Lb include formal meetings, informal meetings, casual chats, and teaching scenarios;

[0118] Step S4013: Construct an emotional feature vector F by analyzing the speaker's intonation, rhythm and semantics emotion , F emotion =[f frame ,ZCR,f pitch ,f tempo ], f frame represents the short-time energy of the speech segment of the i-th speaker, ZCR represents the zero crossing rate of the speech segment of the i-th speaker, and f pitch represents the fundamental frequency of the speech segment of the i-th speaker, f tempo Represents the rhythm of the speech segment of the i-th speaker and judges the emotional state. The specific formula is:

[0119] P(E|F emotion )=softmax(W emotion ×F emotion +b emotion ),

[0120] Among them, P(E|F emotion ) represents the emotion category of the speech segment of the i-th speaker, softmax represents the classification function, and W eemotion represents the weight coefficient, b emotion Represents a constant term.

[0121] In this embodiment, the weight coefficient and the constant term are set according to the actual situation or obtained by those skilled in the art through a large number of simulation experiments;

[0122] Examples of affective states include: 1) high frequency and fast speech: indicating tension or excitement, 2) low frequency and slow speech: indicating calmness or fatigue;

[0123] The short-term energy f of the speech segment of the i-th speaker frame Medium, high energy indicates strong sounds and is usually associated with emotions such as anger and excitement, while low energy is associated with emotions such as calmness and fatigue;

[0124] In the zero crossing rate ZCR of the speech segment of the i-th talker, a higher zero crossing rate indicates a rapidly changing signal, which is usually related to emotions such as tension and excitement; a lower zero crossing rate indicates a stable speech signal, which is related to calm and slow emotions;

[0125] The fundamental frequency f of the speech segment of the i-th speakerpitch In the middle, high fundamental frequencies are associated with emotions such as excitement and happiness, while low fundamental frequencies are common in sad or calm speech;

[0126] The rhythm f of the speech segment of the i-th speaker tempo Medium, fast tempo (higher f tempo ) is often associated with anxiety and excitement, while a slow pace is often associated with fatigue, sadness or calm emotions.

[0127] Step S5: Analyze the speaker's intention based on the final output text and context.

[0128] The specific steps of step S5 are:

[0129] Step S501: Set the intention as a hidden state I y , the comprehensive feature of the i-th speaker is F comb , infer the intention sequence I1,I2,……,I js , the context is collected to calculate the intention probability, the specific formula is: Among them, P(I js |F comb ) represents the intention probability, that is, the intention I of js at the current time js In the comprehensive feature F comb The probability under this condition describes how the current feature vector affects the intention inference, P(I js |I js-1 ) represents the intention state transition probability, which means the transition from the previous state I js-1 To current state I js The possibility of capturing the dynamic change of intention, such as the change from "asking a question" to "waiting for an answer", P(F comb |I js ) represents the comprehensive feature F comb In the current intention I js The probability under is used to measure the support degree of feature data for the current intention;

[0130] Comprehensive features include phonetic features, semantic features, and emotional features;

[0131] Step S502: Based on the results of multiple rounds of dialogue, define a reward function R(I js ,A),R(I js ,A)=λ1×Prec(I js )+λ2×Rele(A)-λ3×Pena(A), where A represents the response action, Prec(I js ) represents the accuracy of the recognized intent, measuring I js Is this correct? Rele(A) indicates the response action A and the intention I jsPena(A) represents the negative impact of wrong actions on performance, λ1, λ2 and λ3 represent weight parameters, which control the contribution of accuracy, relevance and penalty respectively;

[0132] The reward function comprehensively considers the accuracy of intention recognition and the relevance of the action, and the penalty term controls the tolerance for incorrect responses and optimizes the behavior;

[0133] Step S503: Optimize the strategy through deep reinforcement learning to dynamically adjust the talker's intention in multiple rounds of dialogue. The specific formula is:

[0134]

[0135] Among them, P(I js |O 1:js ) represents the speaker's intention after dynamic adjustment of multiple rounds of dialogue, O 1:js represents the observation characteristics of the current round, P(O 1:js |I js ) represents the observed feature O of the current round 1:js In the intention js The probability of js |O 1:js-1 ) represents the speaker’s intention after the dynamic adjustment of the previous round of dialogue, P(O 1:js ) represents the normalization constant of the observed features of the current round.

[0136] Combining historical observations with current features improves the accuracy of intent prediction.

[0137] Example 2

[0138] See also Figure 2 , another embodiment provided by the present invention: a language recognition system based on artificial intelligence assisted conversation scenario, comprising: a data acquisition module, a preprocessing module, a separation and segmentation module, a speech decoding module and an intention analysis module;

[0139] The data acquisition module is used to collect audio signals and video data using sensors and perform noise filtering on the audio signals and video data in real time;

[0140] The preprocessing module is used to preprocess the audio signal after noise filtering;

[0141] The separation and segmentation module is used to separate the speech in the preprocessed audio signal and segment the speaker;

[0142] The speech decoding module is used to extract the speech features of each speaker, perform speech decoding according to the speech features, and output the final text;

[0143] The intention analysis module is used to analyze the intention of the speaker based on the final output text and context.

[0144] A separation and segmentation module, including a speech signal separation unit, a time-frequency feature extraction unit, and a time series clustering unit;

[0145] The speech signal separation unit is used to weight the time-frequency domain representation of the preprocessed audio signal using a separation mask to extract a separated speech signal of each talker;

[0146] The time-frequency feature extraction unit is used to extract the time-frequency features of the separated speech signals of each talker;

[0147] The temporal clustering unit is used to perform temporal clustering on the embedded vectors using a temporal segmentation and clustering algorithm, and output the speech segments and corresponding labels of each speaker.

[0148] A speech decoding module, including: an environment analysis unit, a feature extraction unit and a text output unit;

[0149] The environment analysis unit is used to analyze the conversation environment and determine the context category of the speaker;

[0150] The feature extraction unit is used for time series modeling, capturing the dependencies between the previous and next contexts, and extracting the final features;

[0151] The text output unit is used to output the text corresponding to the speaker by adopting a model based on a sequence-to-sequence structure in combination with an attention mechanism and a language model.

[0152] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0153] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A language recognition method based on artificial intelligence-assisted conversation scenarios, characterized in that: The specific steps include: Step S1: using sensors to collect audio signals and video data, and performing noise filtering on the audio signals and video data in real time; Step S2: preprocessing the audio signal after noise filtering; Step S3: separating the speech in the preprocessed audio signal and segmenting the speaker; Step S4: extracting the speech features of each speaker, performing speech decoding according to the speech features, and outputting the final text; Step S5: Analyze the speaker's intention based on the final output text and context.

2. The language recognition method based on artificial intelligence assisted conversation scenario according to claim 1, characterized in that: The specific steps of step S1 are: Step S101: Set the collected audio signal as x(t), the audio signal is divided into a target signal S(t) and a noise signal N(t), perform short-time Fourier transform on the audio signal x(t), and obtain an audio time-frequency domain signal X(k,n), where k represents a frequency index and n represents a time index; Step S102: Calculate the signal-to-noise ratio (SNR) of the audio time-frequency domain signal, and dynamically adjust the noise reduction parameters according to the calculated SNR to enhance the audio time-frequency domain signal. The specific formula is: Y(k,n) 2 |X(k,n) 2 −α×|N(k,n) 2 , Among them, Y(k,n) represents the enhanced audio time-frequency domain signal, N(k,n) represents the time-frequency domain signal of the noise in the audio, α represents the noise reduction coefficient, and |·| represents the absolute value function.

3. The language recognition method based on artificial intelligence assisted conversation scenario according to claim 1, characterized in that: The specific steps of step S3 are: Step S301: Generate a speech separation mask using a speech separation network trained by self-supervised learning. The specific formula is: M i (k,n)=σ(f DPRNN (Y'(k,n),Θ)) Among them, M i (k,n) represents the time-frequency mask of the ith speaker, σ(·) represents the activation function, and f DPRNN (·) represents the DPRNN network output, Θ represents the network model parameters, and Y'(k,n) represents the preprocessed audio signal; Step S302: Use the speech separation mask to weight the time-frequency domain representation of the preprocessed audio signal to obtain the separated speech time-frequency domain signal of each speaker. The specific formula is: Represents the time-frequency domain signal of the separated speech of the i-th speaker. Through the inverse short-time Fourier transform, the time-domain signal of the separated speech of the i-th speaker is obtained Step S303: Extract the time-frequency features Fp of the separated speech signals of each speaker i , Fp i ={fp1,fp2,...,fp m }, fp m represents the mth time-frequency feature of the speech signal separated by the ith speaker, and uses the pre-trained speaker embedding model to embed the time-frequency feature Fp i Mapped to an embedding vector e of fixed dimension i ; Step S304: Use time series segmentation and clustering algorithms to embed the vector e i Perform temporal clustering to generate paragraph tags {dl i ,dl i+1 ,...,dl sl },dl sl Indicates the slth paragraph mark generated; Step S305: Combine the separated speech signal and time series clustering to output the speech segment and corresponding label of the i-th speaker: Among them, C i represents the identifier of the i-th speaker, and Z represents the number of speakers.

4. The language recognition method based on artificial intelligence assisted conversation scenario according to claim 1, characterized in that: The specific steps of step S4 are: Step S401: Analyze the conversation environment and determine the context category P (Lb|F env ); Step S402: Map the speech segment of the i-th speaker to the logarithmic scale f of human hearing through the Mel frequency scale mel , take the logarithmic scale f mel After logarithm, discrete cosine transform is performed to extract static features C i [n], and extract the first-order dynamic feature ΔC based on the static feature i [n] and the second-order dynamic characteristic Δ 2 C i [n], the specific formula is: Among them, C i [n+tc] represents the static features of the n+tcth frame, tc represents the time difference, that is, the offset relative to the current frame, and Nc represents the half width of the time window; Step S403: Extract the static feature C i [n], first-order dynamic characteristic ΔC i [n] and the second-order dynamic characteristic Δ 2 C i [n] is input into the bidirectional RNN model for time series modeling to capture the dependencies between the previous and next contexts and obtain the final feature h sj ; Step S404: Use the attention mechanism to weight the final features to obtain weighted speech features Highlight the temporal features most relevant to the current recognition task; Step S405: Using a model based on sequence-to-sequence structure, combining the attention mechanism and language model to transform speech features Mapped to the corresponding text sequence to get the final text.

5. The method for language recognition based on artificial intelligence-assisted conversation scenarios as claimed in claim 4, characterized in that: The specific steps of step S401 are: Step S4011: Perform multimodal analysis on the speech segment and video data of the i-th speaker to obtain environmental features, including: audio features f audio , video features vidio and semantic features f semantic , and construct the environmental feature vector F env , F env =[f audio ,f vidio ,f semantic ]; Step S4012: Use a deep neural network to analyze the environmental feature vector F env Classify and output the context category Lb. The specific classification formula is: P(Lb|F env )=softmax(W env ×F env +b env ), Among them, P(Lb|F env ) represents the context category of the speech segment of the i-th speaker, softmax represents the classification function, W env represents the weight coefficient, b env represents a constant term; Step S4013: Construct an emotional feature vector F by analyzing the speaker's intonation, rhythm and semantics emotion , F emotion =[f frame ,ZCR,f pitch ,f tempo ], f frame represents the short-time energy of the speech segment of the i-th speaker, ZCR represents the zero crossing rate of the speech segment of the i-th speaker, and f pitch represents the fundamental frequency of the speech segment of the i-th speaker, f tempo Represents the rhythm of the speech segment of the i-th speaker and judges the emotional state. The specific formula is: P(E|F emotion )=softmax(W emotion ×F emotion +b emotion ), Among them, P(E|F emotion ) represents the emotion category of the speech segment of the i-th speaker, softmax represents the classification function, and W eemotion represents the weight coefficient, b emotion Represents a constant term.

6. The method for language recognition based on artificial intelligence-assisted conversation scenarios according to claim 1, characterized in that: The specific steps of step S5 are: Step S501: Set the intention as a hidden state I y , the comprehensive feature of the i-th speaker is F comb , infer the intention sequence I1,I2,……,I js , the context is collected to calculate the intention probability, the specific formula is: Among them, P(I js |F comb ) represents the intention probability, that is, the intention I of js at the current time js In the comprehensive feature F comb The probability of js |I js-1 ) represents the intention state transition probability, that is, from the previous state I js-1 To current state I js The probability, P(F comb |I js ) represents the comprehensive feature F comb In the current intention I js The probability of the following; Step S502: Based on the results of multiple rounds of dialogue, define a reward function R(I js ,A),R(I js ,A)=λ1×Prec(I js )+λ2×Rele(A)-λ3×Pena(A), where A represents the response action, Prec(I js ) represents the accuracy of the recognized intention, Rele(A) represents the response action A and the intention I js Pena(A) represents the negative impact of wrong actions on performance, and λ1, λ2 and λ3 represent weight parameters; Step S503: Optimize the strategy through deep reinforcement learning to dynamically adjust the talker's intention in multiple rounds of dialogue. The specific formula is: Among them, P(I js |O 1:js ) represents the speaker's intention after dynamic adjustment of multiple rounds of dialogue, O 1:js represents the observation characteristics of the current round, P(O 1:js |I js ) represents the observed feature O of the current round 1:js In the intention js The probability of js |O 1:js-1 ) represents the speaker’s intention after the dynamic adjustment of the previous round of dialogue, P(O 1:js ) represents the normalization constant of the observed features of the current round.

7. The language recognition method based on artificial intelligence assisted conversation scenario according to claim 1, characterized in that: The preprocessing in step S2 includes: pre-emphasis, frame segmentation and normalization processing to obtain a preprocessed audio signal Y'(k,n); The pre-emphasis enhances the high-frequency components through a high-pass filter and reduces the masking effect of the low-frequency components on the signal; the frame segmentation segments the speech signal into overlapping frames of length Ns; the normalization processing adjusts the signal amplitude to a fixed range.

8. A language recognition system based on artificial intelligence-assisted conversation scenarios, used to implement the language recognition method based on artificial intelligence-assisted conversation scenarios according to any one of claims 1 to 7, characterized in that: include: Data acquisition module, preprocessing module, separation and segmentation module, speech decoding module and intention analysis module; The data acquisition module is used to collect audio signals and video data using sensors and perform noise filtering on the audio signals and video data in real time; The preprocessing module is used to preprocess the audio signal after noise filtering; The separation and segmentation module is used to separate the speech in the preprocessed audio signal and segment the speaker; The speech decoding module is used to extract the speech features of each speaker, perform speech decoding according to the speech features, and output the final text; The intention analysis module is used to analyze the intention of the speaker based on the final output text and context.

9. The language recognition system based on artificial intelligence-assisted conversation scenario as claimed in claim 8, characterized in that: The separation and segmentation module includes a speech signal separation unit, a time-frequency feature extraction unit and a time series clustering unit; The speech signal separation unit is used to weight the time-frequency domain representation of the preprocessed audio signal using a separation mask to extract a separated speech signal of each talker; The time-frequency feature extraction unit is used to extract the time-frequency features of the separated speech signals of each talker; The temporal clustering unit is used to perform temporal clustering on the embedded vectors using a temporal segmentation and clustering algorithm, and output the speech segments and corresponding labels of each speaker.

10. The language recognition system based on artificial intelligence-assisted conversation scenario as claimed in claim 9, characterized in that: The speech decoding module includes: an environment analysis unit, a feature extraction unit and a text output unit; The environment analysis unit is used to analyze the conversation environment and determine the context category of the speaker; The feature extraction unit is used for time series modeling, capturing the dependencies between the previous and next contexts, and extracting the final features; The text output unit is used to output the text corresponding to the speaker by adopting a model based on a sequence-to-sequence structure in combination with an attention mechanism and a language model.

Images