A voice dialogue interaction method and system for industrial equipment

Abstract

The application provides an industrial equipment-oriented voice dialogue interaction method and system, and relates to the technical field of human-computer interaction, and comprises the following steps: acquiring a voice signal of a user, and performing voice enhancement processing on the voice signal of the user through an incremental adaptive filtering algorithm to obtain an enhanced voice; according to the enhanced voice, using a transfer learning method of information gain, recognizing an interaction intention of the user, and generating a voice to be interacted based on the interaction intention of the user; based on the enhanced voice, using a time delay estimation method to recognize an interaction position of the user, and performing position prediction on the position of the user; and according to the position prediction result, constructing an industrial equipment loudspeaker output strategy; based on the industrial equipment loudspeaker output strategy, outputting the voice to be interacted, so as to realize voice dialogue interaction between the user and the industrial equipment. The application greatly improves the convenience, efficiency and accuracy of user interaction with the equipment in an industrial scene, and helps the intelligent development of industrial production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, in particular, to a voice dialogue interaction method and system for industrial equipment. BACKGROUND

[0002] In the field of industrial production, with the continuous development of intelligent technology, efficient and convenient human-computer interaction has become a key link to improve production efficiency and quality. Voice dialogue interaction, as a natural and efficient human-computer interaction method, has great application potential in the field of industrial equipment. It allows operators to communicate with equipment through voice commands, without the need for manual operation, greatly improving the convenience and flexibility of operation.

[0003] In terms of voice signal acquisition, there are often high-intensity environmental noises in industrial sites, such as machine running sound, tool knocking sound, etc. These noises can seriously interfere with the collection of user voice signals, leading to a decrease in voice quality, making subsequent voice recognition and intent understanding very difficult. Although existing voice enhancement techniques can suppress noise to some extent, their effectiveness is still not satisfactory in complex and variable industrial noise environments, making it difficult to meet the requirements of industrial equipment for voice interaction accuracy and reliability. In terms of interactive intent recognition, voice commands in the industrial field are highly professional and complex in semantics. Voice commands related to different industrial equipment and different production processes vary greatly, and users have different expression habits, making it difficult to accurately identify user interaction intent. Traditional voice recognition and intent understanding methods often fail to adapt to the particularity of industrial scenarios, and cannot effectively handle these complex and variable voice commands, resulting in low intent recognition accuracy and affecting the practicality and user experience of voice dialogue interaction.

[0004] In view of the problems in the related art, no effective solutions have been proposed so far. SUMMARY

[0005] Therefore, the present application provides a voice dialogue interaction method and system for industrial equipment to solve the above-mentioned problems.

[0006] In order to solve the above problems, the specific technical solutions adopted by the present application are as follows:

[0007] According to one aspect of the present application, a voice dialogue interaction method for industrial equipment is provided, comprising the following steps:

[0008] S1, based on a microphone array pre-configured on the industrial equipment, acquiring a user's voice signal, and performing voice enhancement processing on the user's voice signal through an incremental adaptive filtering algorithm to obtain an enhanced voice;

[0009] S2, identifying the interactive intention of the user based on the enhanced voice using the information gain transfer learning method, and generating the voice to be interacted based on the interactive intention of the user;

[0010] S3, identifying the interactive position of the user based on the enhanced voice using the time delay estimation method, and predicting the position of the user; and constructing an industrial device loudspeaker output strategy according to the position prediction result;

[0011] S4, outputting the voice to be interacted based on the industrial device loudspeaker output strategy, to realize voice dialogue interaction between the user and the industrial device.

[0012] Preferably, the microphone array pre-configured on the industrial device acquires the voice signal of the user, and the voice signal of the user is subjected to voice enhancement processing through the incremental adaptive filtering algorithm to obtain the enhanced voice, including the following steps:

[0013] S11, synchronously collecting the spatially distributed multi-channel voice signal using the microphone array pre-configured on the industrial device, and pre-processing the voice signal of each channel to obtain the pre-processed voice signal;

[0014] S12, constructing an initial error signal based on the pre-processed voice signal, and using the incremental adaptive filtering algorithm to preliminarily enhance the pre-processed voice signal to obtain a preliminarily enhanced voice signal;

[0015] S13, using a fixed-time disturbance observer to estimate and compensate transient noise of the preliminarily enhanced voice signal to obtain the final enhanced voice.

[0016] Preferably, the step of constructing an initial error signal based on the pre-processed voice signal, and using the incremental adaptive filtering algorithm to preliminarily enhance the pre-processed voice signal to obtain a preliminarily enhanced voice signal includes the following steps:

[0017] S121, for the pre-processed voice signal of each channel, randomly selecting a group of channels as reference channels, and constructing an initial error signal based on the pre-processed voice signal of the reference channels;

[0018] S122, initializing the parameters of the incremental adaptive filtering algorithm, including the filter order, the step factor and the initial filter coefficient;

[0019] S123, using the initial error signal to iteratively update the filter coefficient according to the initialized incremental adaptive filtering algorithm, and filtering the pre-processed voice signal of the reference channels to obtain the preliminarily enhanced voice signal.

[0020] Preferably, the step of constructing an initial error signal based on the pre-processed voice signal of the reference channels includes the following steps:

[0021] The preprocessed speech signal of the reference channel is frame processed, and a fast Fourier transform is performed on each frame of the speech signal to obtain a spectrum of the speech signal;

[0022] According to the spectrum of the speech signal, a power spectrum of each frequency point is calculated, and at each frequency point, a minimum value search is performed on each frequency point within a preset time sliding window to obtain a minimum power value of each frequency point within the time sliding window as an initial noise power spectrum estimate;

[0023] The initial estimate value of the noise power of each frequency point is smoothed, and a noise signal is synthesized based on the smoothed noise power estimate value;

[0024] The initial error signal is obtained by subtracting the synthesized noise signal from the reference channel speech signal.

[0025] Preferably, the method for identifying the interactive intent of the user according to the enhanced speech using the information gain transfer learning method, and generating the to-be-interacted speech based on the interactive intent of the user comprises the following steps:

[0026] S21, converting the enhanced speech into a speech signal using a speech recognition technology to obtain a speech text;

[0027] S22, extracting semantic features from the obtained speech text using an information gain method, and processing the extracted features using a transfer learning algorithm based on acoustic semantic distance to identify the interactive intent of the user;

[0028] S23, generating the to-be-interacted speech using a speech synthesis technology according to the interactive intent of the user in combination with a knowledge graph.

[0029] Preferably, the method for extracting semantic features from the obtained speech text using an information gain method, and processing the extracted features using a transfer learning algorithm based on acoustic semantic distance to identify the interactive intent of the user comprises the following steps:

[0030] S221, performing text processing on the obtained speech text, and performing category statistics on the text-processed speech text based on a preset industrial equipment category label to determine the frequency of occurrence of each category;

[0031] S222, determining the distribution of each vocabulary in different categories, and calculating the importance score of each vocabulary using an information gain algorithm according to the frequency of occurrence of each category, and extracting semantic features according to the importance score of each vocabulary;

[0032] S223, calculating the acoustic semantic distance between the vocabularies based on the semantic features of the vocabularies, and constructing a semantic similarity matrix in an industrial scene according to the acoustic semantic distance;

[0033] S224, dynamically adjust the classification weight by combining the semantic similarity matrix using the transfer learning algorithm, and output the user intent category.

[0034] Preferably, the semantic feature based on the vocabulary, the acoustic semantic distance between the words is calculated, and the semantic similarity matrix in the industrial scene is constructed according to the acoustic semantic distance includes the following steps:

[0035] S2331, for the semantic feature of the word, the semantic feature is divided into acoustic feature and text feature by feature decoupling technology;

[0036] S2332, according to the acoustic feature and the text feature, the acoustic feature vector and the text semantic feature vector of each word are extracted respectively, and the semantic embedding vector is generated by fusion mechanism;

[0037] S2333, based on the fused semantic embedding vector, the acoustic semantic distance between the words is calculated by cosine similarity, and the semantic similarity matrix is constructed according to the calculated acoustic semantic distance.

[0038] Preferably, the transfer learning algorithm is used to dynamically adjust the classification weight by combining the semantic similarity matrix, and the user intent category is outputted, which includes the following steps:

[0039] S2241, initialize the pre-trained intent classification model weight, and take the intent classification model weight as the starting point of transfer learning;

[0040] S2242, according to the semantic similarity matrix, construct the graph Laplace matrix, and use the graph Laplace matrix to construct the graph Laplace regularization term;

[0041] S2243, based on the maximum likelihood function, and combining the graph Laplace regularization term, the intent classification model weight is optimized to obtain the optimized intent classification model;

[0042] S2244, obtain new speech text, convert the new speech text into feature vector by using the optimized intent classification model, calculate the probability of each intent category, and select the category with the highest probability as the final user interaction intent.

[0043] Preferably, the enhanced speech is used to identify the user's interaction position by time delay estimation method, and the position of the user is predicted; and according to the position prediction result, the industrial equipment loudspeaker output strategy is constructed, which includes the following steps:

[0044] S31, based on the enhanced speech, the generalized cross-correlation-phase transform method is used to calculate the time delay estimation value between the microphone pairs;

[0045] S32, coordinates of the microphone array in space are acquired, based on the time delay estimation value between the microphone pairs, the position estimation coordinates of the user sound source are solved by minimizing the sound source positioning error function, and the current interactive position of the user is identified;

[0046] S33, the next interactive position of the user is predicted based on the historical interactive position and the current identification result, a target area is obtained, and a loudspeaker output strategy is constructed according to the position of the target area.

[0047] According to another aspect of the present application, an industrial equipment-oriented voice dialogue interaction system is provided, which comprises:

[0048] A voice enhancement processing module is configured to acquire the voice signal of the user based on the microphone array pre-configured on the industrial equipment, and perform voice enhancement processing on the voice signal of the user through an incremental adaptive filtering algorithm to obtain an enhanced voice.

[0049] An interactive intention recognition module is configured to recognize the interactive intention of the user by using a transfer learning method of information gain based on the enhanced voice, and generate a voice to be interacted based on the interactive intention of the user.

[0050] A strategy construction module is configured to recognize the interactive position of the user by using a time delay estimation method based on the enhanced voice, and predict the position of the user, and construct a loudspeaker output strategy of the industrial equipment according to the position prediction result.

[0051] A voice interaction module is configured to output the voice to be interacted based on the loudspeaker output strategy of the industrial equipment, so as to realize voice dialogue interaction between the user and the industrial equipment.

[0052] The present application has the following advantages:

[0053] The present application can effectively improve the voice quality and reduce the environmental noise interference by acquiring and enhancing the voice signal of the user through the microphone array and the incremental adaptive filtering algorithm, can more accurately understand the user demand and improve the interactive intention recognition accuracy by recognizing the interactive intention and generating the voice to be interacted by using the transfer learning method of information gain, can realize the pertinence and accuracy of voice output by recognizing the interactive position and predicting the position by using the time delay estimation method, and then constructing the loudspeaker output strategy, and finally outputs the voice to be interacted according to the strategy to achieve smooth and natural voice dialogue interaction, greatly improves the convenience, efficiency and accuracy of the user in the industrial scene and the device interaction, and helps the intelligent development of industrial production. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0055] Figure 1 is a flow chart of a voice dialogue interaction method for industrial equipment according to an embodiment of the present application;

[0056] Figure 2 is a principle block diagram of a voice dialogue interaction system for industrial equipment according to an embodiment of the present application.

[0057] In the drawings:

[0058] 1, voice enhancement processing module; 2, interactive intention recognition module; 3, strategy construction module; 4, voice interaction module. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0060] According to an embodiment of the present application, a voice dialogue interaction method and system for industrial equipment are provided.

[0061] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in the drawings, according to an embodiment of the present application, a voice dialogue interaction method for industrial equipment is provided, comprising the following steps: Figure 1

[0062] S1, based on a microphone array pre-configured on the industrial equipment, obtaining a voice signal of a user, and through an incremental adaptive filtering algorithm, performing voice enhancement processing on the voice signal of the user to obtain an enhanced voice;

[0063] As a preferred embodiment, the step of obtaining the voice signal of the user based on the microphone array pre-configured on the industrial equipment, and performing voice enhancement processing on the voice signal of the user through the incremental adaptive filtering algorithm to obtain the enhanced voice comprises the following steps:

[0064] ​S11, synchronously collect a plurality of channels of voice signals distributed in space by using a microphone array pre-configured on an industrial device, and pre-process the voice signals of each channel to obtain pre-processed voice signals;

[0065] It should be noted that the pre-processing of the voice signals of each channel includes removing direct current components and pre-emphasis processing, and the direct current components are removed by calculating the average value of the signal and subtracting it from each sampling point to eliminate the direct current offset in the signal and make the signal fluctuate around zero. The pre-emphasis is to use a first-order high-pass filter to pre-emphasize the signal, which improves the high frequency part and makes the frequency spectrum of the signal flat, facilitating subsequent spectral analysis and feature extraction. The pre-processing can improve the signal quality and provide more suitable input for subsequent processing.

[0066] S12, based on the pre-processed voice signals, constructing an initial error signal, and using an incremental adaptive filtering algorithm to preliminarily enhance the pre-processed voice signals to obtain a preliminarily enhanced voice signal;

[0067] As a preferred embodiment, the step of constructing an initial error signal based on the pre-processed voice signals and using an incremental adaptive filtering algorithm to preliminarily enhance the pre-processed voice signals to obtain a preliminarily enhanced voice signal includes the following steps:

[0068] S121, for the pre-processed voice signals of each channel, randomly selecting a group of channels as reference channels, and constructing an initial error signal according to the pre-processed voice signals of the reference channels;

[0069] Specifically, in the microphone array, the signal of any one channel contains user voice and noise components; by randomly selecting a group of channels as reference channels, the signal of the channel can be used to estimate the noise, and then the initial error signal is constructed.

[0070] As a preferred embodiment, the step of constructing an initial error signal according to the pre-processed voice signals of the reference channels includes the following steps:

[0071] Frame processing is performed on the pre-processed voice signals of the reference channels, and fast Fourier transform processing is performed on each frame of voice signals to obtain the frequency spectrum of the voice signals;

[0072] It should be noted that the representation of voice signals in the time domain cannot directly reflect the frequency components and energy distribution. Frequency domain analysis can more intuitively show the frequency characteristics of the signal, facilitating noise estimation and voice enhancement processing. Fast Fourier transform is an efficient algorithm for calculating discrete Fourier transform (DFT), which can quickly convert time domain signals to frequency domain signals.

[0073] Specifically, the Fast Fourier Transform treats each frame of the time-domain speech signal after framing as a discrete sequence, and converts it into a complex sequence in the frequency domain through the Fast Fourier Transform algorithm; the modulus of the complex sequence represents the amplitude of the signal at different frequencies, and the phase represents the phase information of the signal at different frequencies.

[0074] Based on the spectrum of the speech signal, calculate the power spectrum of each frequency point, and at each frequency point, perform a minimum value search within a preset time sliding window to obtain the minimum power value of each frequency point within the time sliding window, which is used as the initial noise power spectrum estimate.

[0075] It should be noted that for a frequency domain complex sequence after Fast Fourier Transform, its power spectrum can be obtained by calculating the square of the modulus of the complex number at each frequency point. That is, for the i-th frequency complex sequence... k Each frequency point has a power spectrum P ( k )for:

[0076] P ( k = | X ( k )∣ 2 ;

[0077] in, X ( k ) is the th after the Fast Fourier Transform. k A complex number of frequency points.

[0078] Furthermore, the minimum power value found can be approximated as the noise power at that frequency point, thus obtaining an initial noise power spectrum estimate.

[0079] The initial estimates of noise power at each frequency point are smoothed, and the noise signal is synthesized based on the smoothed noise power estimates.

[0080] Specifically, the initial estimate of noise power at each frequency point is smoothed using the moving average method. By averaging the noise power estimate at the current frequency point with the estimates at several previous and subsequent frequency points, a smoothed noise power estimate is obtained.

[0081] In noise signal synthesis, the noise power spectrum in the frequency domain can be converted into a noise signal in the time domain using the inverse fast Fourier transform (IFFT). This involves generating a complex noise sequence in the frequency domain based on the noise power estimate (usually assuming the noise phase is random), and then performing an IFFT transform on the complex noise sequence in the frequency domain to obtain the noise signal in the time domain.

[0082] The initial error signal is obtained by subtracting the synthesized noise signal from the reference channel speech signal.

[0083] S122, initialize parameters of the incremental adaptive filtering algorithm, including filter order, step size factor, and initial filter coefficients;

[0084] It should be noted that the filter order refers to the number of past input signal samples used in the adaptive filter; it determines the degree of utilization of historical information of the input signal by the filter; in the incremental adaptive filtering algorithm, the higher the filter order, the wider the historical range of the input signal that the filter can consider. The step size factor is a key parameter in the incremental adaptive filtering algorithm, which controls the adjustment amplitude of the filter coefficients at each iteration update; the initial filter coefficients are the coefficient values of the adaptive filter before starting the iteration update; they provide a starting point for the iteration update of the filter coefficients.

[0085] S123, according to the initialized incremental adaptive filtering algorithm, iteratively update the filter coefficients using the initial error signal, and filter the preprocessed speech signal of the reference channel to obtain a preliminary enhanced speech signal.

[0086] Specifically, the core idea of the incremental adaptive filtering algorithm is to continuously adjust the filter coefficients according to the input signal and the error signal, so that the output of the filter is as close to the expected signal as possible (in speech enhancement, the expected signal is the pure speech signal, but it cannot be directly obtained, so it is usually adjusted indirectly through the error signal). In each iteration, the algorithm updates the filter coefficients according to the current input signal, error signal, and step size factor, according to certain rules. Specifically, the following steps are included:

[0087] Convolve the preprocessed speech signal of the reference channel (input signal) with the current filter coefficients to obtain the output signal of the filter.

[0088] Compare the initial error signal (at the first iteration) or the error signal obtained in the last iteration with the filter output signal to obtain the error signal at the current time. In speech enhancement, the error signal can be represented as the difference between the expected signal and the filter output;

[0089] Update the filter coefficients according to the input signal and the step size factor according to the update formula of the incremental adaptive filtering algorithm, which is:

[0090] w i ( n +1)= w i ( n )+ μe ( n ) x ( n - i );

[0091] wherein, μ denotes a step factor, w i n denotes the k-th filter coefficient of the adaptive filter at time n i w i n denotes the k-th filter coefficient of the adaptive filter at time n i e n denotes the error signal at time n

[0092] Further, after updating the filter coefficients in each iteration, the pre-processed speech signal of the reference channel is filtered using the updated filter coefficients; the purpose of filtering is to remove the noise components in the speech signal and obtain a signal closer to pure speech.

[0093] Specifically, in each iteration, the filter output signal at the current time is obtained by convoluting the updated filter coefficients with the current input signal (the pre-processed speech signal of the reference channel). As the iteration proceeds, the filter continuously adjusts its coefficients to better adapt to the changes in the signal and noise, thereby gradually improving the filtering effect; after multiple iterations, the output signal of the filter can be regarded as a preliminarily enhanced speech signal.

[0094] S13, using a fixed-time disturbance observer, transient noise estimation and compensation are performed on the preliminarily enhanced speech signal to obtain a final enhanced speech.

[0095] It should be noted that the fixed-time disturbance observer is a device capable of quickly and accurately estimating the disturbance (transient noise in the speech signal processing scenario) in the system, which is based on the input and output information of the system and the preset model structure, and estimates the disturbance in a fixed time through a specific algorithm. Specifically, when using the fixed-time disturbance observer to estimate and compensate the transient noise of the preliminarily enhanced speech signal, the following steps are included:

[0096] The preliminarily enhanced speech signal is received, and the signal is collected in real time, and the time domain and frequency domain characteristics of the signal are analyzed; in the time domain, the amplitude change, waveform characteristics, etc. of the signal are observed; in the frequency domain, the signal is converted to the frequency domain through algorithms such as fast Fourier transform (FFT), and the frequency component distribution of the signal is analyzed.

[0097] ​​​​​​​​Based on the preset system model and the interference estimation algorithm, the fixed-time interference observer starts to estimate the transient noise. For example, the fixed-time interference observer designed by using the sliding mode control theory can make the state of the observer converge to the value of the actual interference in a fixed time by defining a suitable sliding surface and a sliding mode control law, so as to obtain the estimated value of the transient noise.

[0098] After obtaining the estimated value of the transient noise, the weighted compensation method is used for compensation. The weighted compensation is to assign a weight coefficient to the noise estimated value according to the accuracy of the noise estimation and the characteristics of the signal, and then the weighted compensation is performed by using the weight coefficient.

[0099] S2, according to the enhanced speech, the transfer learning method using information gain is used to identify the interactive intention of the user, and the interactive speech to be interacted is generated based on the interactive intention of the user;

[0100] As a preferred embodiment, the step of identifying the interactive intention of the user according to the enhanced speech by using the transfer learning method using information gain and generating the interactive speech to be interacted based on the interactive intention of the user comprises the following steps:

[0101] S21, the speech signal conversion is performed on the enhanced speech by using the speech recognition technology, and the speech text is obtained;

[0102] It should be noted that the speech recognition technology is to convert the word content in human speech into a computer-readable text format; its process involves multiple key components such as acoustic model and language model. First, the acoustic model is responsible for mapping the features of the speech signal (such as mel frequency cepstral coefficient, etc.) to phonemes or sub-word units, which is trained by a large amount of speech data to learn the probability relationship between different speech features and corresponding pronunciation. Then the language model combines and adjusts the phoneme or sub-word sequence output by the acoustic model based on grammar rules and a large amount of text corpus to generate text that conforms to language habits.

[0103] S22, the semantic feature extraction is performed on the obtained speech text by using the information gain method, and the extracted features are processed by using the transfer learning algorithm based on acoustic semantic distance to identify the interactive intention of the user;

[0104] As a preferred embodiment, the step of identifying the interactive intention of the user by using the information gain method to extract the semantic features of the obtained speech text and processing the extracted features by using the transfer learning algorithm based on acoustic semantic distance comprises the following steps:

[0105] S221, the text processing is performed on the obtained speech text, and the category statistics of the speech text after text processing is performed based on the preset industrial equipment category label to determine the frequency of occurrence of each category;

[0106] It should be noted that the text processing operation includes the operations of removing stop words, stem extraction, and morphological reduction, and can be specifically performed by a natural language processing tool library (such as NLTK, spaCy, etc.) to preprocess the voice text; first, the text is segmented into words or phrases, then the stop words are removed, and then the stem extraction or morphological reduction operation is performed to obtain the processed text.

[0107] In addition, the preset industrial equipment category label is defined according to the actual needs and equipment types in the industrial scene, such as machine tools, robots, conveyors, etc.

[0108] S222, determine the distribution of each vocabulary in different categories, and calculate the importance score of each vocabulary using the information gain algorithm according to the frequency of each category, and extract semantic features according to the importance score of each vocabulary;

[0109] It should be noted that information gain is an index used to describe the importance of features to the classification system, which is based on the concept of information entropy, and evaluates the importance of features by calculating the change of information entropy of the system when the feature exists and does not exist; the greater the information gain, the greater the contribution of the feature to the classification system.

[0110] For each vocabulary, the number of occurrences in each category is counted, and the occurrence frequency in each category is calculated, and the information entropy of the probability distribution of all categories is calculated according to the frequency of each category; then, for each vocabulary, the conditional information entropy when the vocabulary exists and does not exist is calculated; finally, the information gain value of each vocabulary is calculated according to the calculation formula of information gain as its importance score; the calculation formula of information gain: information gain = system information entropy - conditional information entropy.

[0111] S223, based on the semantic features of the vocabulary, calculate the acoustic semantic distance between the vocabularies, and construct a semantic similarity matrix in the industrial scene according to the acoustic semantic distance;

[0112] As a preferred embodiment, the acoustic semantic distance between the vocabularies is calculated based on the semantic features of the vocabularies, and a semantic similarity matrix in the industrial scene is constructed according to the acoustic semantic distance, which includes the following steps:

[0113] S2331, for the semantic features of the vocabulary, the semantic features are divided into acoustic features and text features by feature decoupling technology;

[0114] It should be noted that in the scenario of combining voice and text, the semantic features mix acoustic information (such as tone, speech rate, pronunciation method, etc.) from the user's voice and semantic information (such as the meaning of the vocabulary, the grammatical structure, etc.) from the text, and the feature decoupling technology can separate the features of the two different sources; Specifically, by inputting the semantic features of the vocabulary into the pre-trained feature decoupling model; The model will analyze and process the input features, gradually separate the acoustic features and text features through the internal network structure and parameter adjustment.

[0115] S2332, according to the acoustic features and the text features, the acoustic feature vectors and the text semantic feature vectors of each vocabulary are extracted respectively, and the semantic embedding vectors are generated through the fusion mechanism;

[0116] It should be noted that the acoustic feature vector is used to represent the voice characteristics of the vocabulary. Common acoustic feature extraction methods include Mel frequency cepstral coefficient (MFCC), linear predictive coding (LPC), etc. MFCC can effectively capture the spectral information in the speech signal by simulating the perception characteristics of the human ear;

[0117] The text semantic feature vector is used to represent the semantic meaning of the vocabulary; Common methods include word embedding technology, such as Word2Vec, GloVe, etc. By learning the context relationship of the vocabulary in a large amount of text corpus, each vocabulary is mapped to a low-dimensional vector space, so that semantically similar vocabularies are closer in the vector space.

[0118] Specifically, after the acoustic feature vector and the text semantic feature vector are extracted, the acoustic feature vector and the text semantic feature vector can be spliced and fused by direct splicing, and finally the semantic embedding vector is obtained.

[0119] S2333, based on the fused semantic embedding vector, the cosine similarity is used to calculate the acoustic semantic distance between the vocabularies, and a semantic similarity matrix is constructed according to the calculated acoustic semantic distance.

[0120] It should be noted that the cosine similarity is an index for measuring the similarity in direction between two vectors. The value range is between [-1, 1], the closer the value is to 1, the more similar the two vectors are in direction; the closer the value is to -1, the more opposite the two vectors are in direction; when the value is 0, the two vectors are orthogonal, that is, unrelated. When calculating the acoustic semantic distance between the vocabularies, the cosine similarity can effectively measure the similarity between the semantic embedding vectors of the two vocabularies.

[0121] S224, using a transfer learning algorithm, combining the semantic similarity matrix to dynamically adjust the classification weight, and outputting the user intent category.

[0122] As a preferred embodiment, the step of utilizing the transfer learning algorithm, combining the semantic similarity matrix to dynamically adjust the classification weight, and outputting the user intention category comprises the following steps:

[0123] S2241, initialize the pre-trained intention classification model weight, and take the intention classification model weight as the starting point of transfer learning;

[0124] It should be noted that in the industrial scene, the intention classification model can be selected as a model based on a recurrent neural network (RNN) and its variants (long short-term memory network LSTM, gated recurrent unit GRU), or a deep learning model pre-trained on industrial field voice text data based on a Transformer architecture model (such as BERT and its variants).

[0125] Among them, the core idea of transfer learning is to transfer the knowledge learned by the pre-trained model to the new task. Taking the pre-trained model weight as the starting point, the existing knowledge can be used on the new industrial scene intention classification task to reduce the demand for training data, improve the convergence speed and generalization ability of the model.

[0126] S2242, construct a graph Laplacian matrix according to the semantic similarity matrix, and construct a graph Laplacian regularization term using the graph Laplacian matrix;

[0127] It should be noted that when constructing the graph Laplacian matrix, the words are regarded as nodes in the graph, and the edges between the nodes are determined according to the semantic similarity matrix. If the semantic similarity of two words is higher than a certain threshold, an edge is established between them, and the weight of the edge can be set to their semantic similarity value.

[0128] Among them, the graph Laplacian matrix L can be calculated by the adjacency matrix A and the degree matrix D , and the formula is L = D - A . Among them, the degree matrix D is a diagonal matrix, and the diagonal elements D ii represent the degree of node i , that is, the sum of the weights of the edges connected to node i . The graph Laplacian matrix can capture the local structure and relationship between nodes in the graph.

[0129] In addition, the purpose of constructing the graph Laplacian regularization term is to introduce semantic similarity information in the model training process, so that the model can consider the semantic relationship between words when classifying. By adding the regularization term to the loss function, the parameter update of the model can be constrained, so that the model outputs corresponding to similar words are also similar; the graph Laplacian regularization termR θ ) can be expressed as R θ θ T Lθ wherein, θ T is the transpose of the parameter θ .

[0130] S2243, based on maximizing the likelihood function, and combining the graph Laplacian regularization term to optimize the intention classification model weight, obtain the optimized intention classification model;

[0131] It should be noted that by maximizing the likelihood function, the parameter that maximizes the probability of the model outputting the correct label on the given data can be found. By taking the logarithm of the likelihood function, the log-likelihood function is obtained, and then the log-likelihood function is maximized by using an optimization algorithm (such as stochastic gradient descent SGD, Adam, etc.).

[0132] wherein the graph Laplacian regularization term is added to the log-likelihood function to construct a new loss function J θ The formula of

[0133] J θ L θ X , Y + λR θ ;

[0134] wherein, λ is a hyperparameter for controlling the strength of the regularization term. λ The greater the regularization term, the stronger the constraint effect of the regularization term on the model parameter update; λ The smaller X is the input data, Y is the output label, and the model is more inclined to fit the training data; then use an optimization algorithm (such as Adam algorithm) to minimize the loss function J θ In each iteration, the gradient of the loss function with respect to the model parameter θ is calculated, and then the parameter is updated according to the gradient. By continuously iterating, the loss function is gradually reduced, and the model parameter is gradually converged to the optimal value, thereby obtaining the optimized intention classification model.

[0135] S2244, obtaining a new speech text, converting the new speech text into a feature vector using the optimized intention classification model, and calculating the probability of each intention category, selecting the category with the highest probability as the final user interaction intention.

[0136] ​​​​​​​​​​It should be noted that the feature vector is input into the classification layer of the optimized intent classification model. The classification layer is a fully connected layer followed by a softmax function. The softmax function converts the output of the classification layer into a probability distribution for each intent category.

[0137] S23, according to the user's interactive intention, combined with the knowledge graph, using speech synthesis technology to generate the interactive voice.

[0138] It should be noted that before generating the interactive voice according to the user's interactive intention, combined with the knowledge graph, using speech synthesis technology, a knowledge graph containing device information, operation process, fault knowledge, product description and other aspects of content needs to be constructed.

[0139] Then according to the user's intention, information retrieval is performed in the knowledge graph. Using graph database query language or graph algorithm-based retrieval method, find the nodes and edges related to the user's intention. For example, when the user's intention is to query the operation process of a certain device, find the device node in the knowledge graph, and then retrieve the specific operation information along the edge related to the operation process. Finally, integrate the retrieved multiple related information to form a complete and coherent reply content.

[0140] S3, based on the enhanced voice, using time delay estimation method to identify the user's interactive position, and position prediction is made on the user's position; and according to the position prediction result, the industrial equipment loudspeaker output strategy is constructed;

[0141] As a preferred embodiment, the method of identifying the user's interactive position based on the enhanced voice, using time delay estimation method, and making position prediction on the user's position; and constructing industrial equipment loudspeaker output strategy according to the position prediction result includes the following steps:

[0142] S31, based on the enhanced voice, using generalized cross-correlation-phase transform method to calculate the time delay estimation value between the microphone pairs;

[0143] It should be noted that the cross-correlation function is used to measure the similarity of two signals at different time offsets. For two microphone-acquired voice signals x 1( n ) and x 2( n ), their cross-correlation function R 12 ( τ ) is defined as:

[0144] ;

[0145] where, τ is the time offset, when R 12 ​τ ) is the time delay estimation value between the two signals. τ

[0146] In addition, in order to improve the accuracy and anti-interference ability of time delay estimation, generalized cross-correlation is introduced. It highlights the useful information in the signal and suppresses the influence of noise and reverberation by weighting the cross-correlation function. The phase transform is a commonly used weighting method for generalized cross-correlation. Its weighting function W ( ω ) is:

[0147] W ( ω )=1 / ∣ X 1( ω ) X 2( ω )∣;

[0148] wherein, X 1( ω ) and X 2( ω ) are the Fourier transforms of x 1( n ) and x 2( n ), respectively. The phase transform weighting can highlight the phase information of the signal and reduce the influence of amplitude information on time delay estimation, and can achieve better time delay estimation effect in a strong reverberation environment. By calculating the generalized cross-correlation function after phase transform weighting, the value corresponding to the maximum value is found, that is, the time delay estimation value between the microphone pairs can be obtained. τ

[0149] S32, obtain the coordinate positions of the microphone array in space based on the time delay estimation value between the microphone pairs; solve the position estimation coordinates of the user sound source by minimizing the sound source positioning error function, and identify the current interaction position of the user;

[0150] It should be noted that after the coordinate positions of each microphone in the microphone array and the time delay estimation value between the microphone pairs are known, the propagation speed of sound in air (usually 340 m / s ) can be used to establish the geometric relationship between the time delay estimation value and the coordinates.

[0151] ​​From a geometric point of view, taking the microphone at each known coordinate position as a reference point, the path length formed by the sound propagating from the user's sound source to different microphones is not the same, and the time delay estimation value actually reflects the difference between these path lengths. For example, there are two microphones, and the sound reaches one first and the other second. The time delay estimation value between them corresponds to the difference in the propagation path length of the sound at the two microphones.

[0152] When there are multiple microphones in the microphone array, multiple such microphone combination pairs are formed, and each combination pair can reflect a geometric constraint relationship of the user's sound source position relative to the microphones according to the time delay estimation value and the sound propagation speed; by synthesizing the geometric constraint information provided by all microphone pairs, a complete conditional framework can be constructed for determining the estimated coordinates of the user's sound source in space, and then by minimizing the sound source positioning error function, the estimated coordinates of the user's sound source can be solved, and the current interactive position of the user can be finally identified.

[0153] S33, predicting the next interactive position of the user based on the historical interactive position and the current recognition result, obtaining a target area, and constructing a loudspeaker output strategy according to the position of the target area.

[0154] It should be noted that when predicting the next interactive position of the user based on the historical interactive position and the current recognition result and constructing the loudspeaker output strategy, the user's interactive position information in the past period of time is continuously recorded, and then combined with the regularity exhibited by the historical interactive position and the current position, a specific analysis method is used to predict the interactive position range that the user is most likely to go to next, which is the target area; after the target area is determined, the loudspeaker output strategy needs to be constructed according to its position; first, the loudspeaker that can most effectively cover the target area is selected according to the relative position relationship between the target area and each loudspeaker; if the target area is close to a specific loudspeaker, this loudspeaker can be used for voice output first, and the volume of the loudspeaker output can be adjusted according to the size of the target area and the possible activity of the user, to ensure that the user can clearly hear the voice prompt in the target area, while not disturbing the normal work of other areas due to the volume being too large.

[0155] S4, based on the loudspeaker output strategy of the industrial equipment, outputting the interactive voice to realize voice dialogue interaction between the user and the industrial equipment.

[0156] It should be noted that according to the previously constructed loudspeaker output strategy, the interactive voice is broadcast through the loudspeaker on the industrial equipment in the most optimal way, realizing voice dialogue interaction between the user and the equipment; the voice information is accurately and clearly conveyed to the user's ear. This process not only ensures the effective transmission of information.

[0157] As Figure 2 shown in the drawings, according to one embodiment of the present application, there is provided an industrial equipment-oriented voice dialogue interaction system, which comprises:

[0158] a voice enhancement processing module 1, configured to acquire a user's voice signal based on a microphone array pre-configured on an industrial equipment, and perform voice enhancement processing on the user's voice signal through an incremental adaptive filtering algorithm to obtain an enhanced voice;

[0159] an interaction intention recognition module 2, configured to recognize the user's interaction intention through a transfer learning method of information gain based on the enhanced voice, and generate a voice to be interacted based on the user's interaction intention;

[0160] a strategy construction module 3, configured to recognize the user's interaction position through a time delay estimation method based on the enhanced voice, and perform position prediction on the user's position; and construct an industrial equipment loudspeaker output strategy according to the position prediction result;

[0161] a voice interaction module 4, configured to output the voice to be interacted based on the industrial equipment loudspeaker output strategy, so as to realize voice dialogue interaction between the user and the industrial equipment.

[0162] To sum up, by means of the above technical solutions of the present application, the user's voice signal is acquired and enhanced through a microphone array and an incremental adaptive filtering algorithm, which can effectively improve voice quality and reduce environmental noise interference; the transfer learning method of information gain is used to recognize the interaction intention and generate the voice to be interacted, which can more accurately understand user demand and improve interaction intention recognition accuracy; the time delay estimation method is used to recognize the interaction position and perform prediction, and then the loudspeaker output strategy is constructed, which can realize the pertinence and accuracy of voice output; finally, the voice to be interacted is output according to the strategy, so as to achieve smooth and natural voice dialogue interaction, greatly improving the convenience, efficiency and accuracy of user interaction with the equipment in an industrial scene, and helping the intelligent development of industrial production.

[0163] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer usable program code.

[0164] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above-described specific embodiments are merely examples of the present application and are not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A voice dialogue interaction method for industrial equipment, characterized by, The method comprises the following steps: S1, based on the microphone array pre-configured on the industrial equipment, the voice signal of the user is acquired, and the voice signal of the user is processed by the incremental adaptive filtering algorithm to obtain enhanced voice; S2, according to the enhanced voice, the interactive intention of the user is recognized by using the information gain transfer learning method, and the interactive voice to be interacted is generated based on the interactive intention of the user; S3, based on the enhanced voice, the interactive position of the user is recognized by using the time delay estimation method, and the position of the user is predicted; And according to the position prediction result, the industrial equipment loudspeaker output strategy is constructed; S4, based on the industrial equipment loudspeaker output strategy, the interactive voice to be interacted is outputted to realize the voice dialogue interaction between the user and the industrial equipment; Wherein, S2 includes: S21, the enhanced voice is converted into voice signal by using voice recognition technology to obtain voice text; S22, the semantic features of the obtained voice text are extracted by using the information gain method, and the extracted features are processed by using the transfer learning algorithm based on acoustic semantic distance to recognize the interactive intention of the user; S23, according to the interactive intention of the user, the knowledge graph is combined, and the interactive voice to be interacted is generated by using voice synthesis technology; Wherein, S22 includes: S221, the obtained voice text is processed, and the frequency of each category appearing is determined based on the pre-set industrial equipment category label after the voice text is processed; S222, the distribution of each vocabulary in different categories is determined, and the importance score of each vocabulary is calculated by using the information gain algorithm according to the frequency of each category, and the semantic features are extracted according to the importance score of each vocabulary; S223, based on the semantic features of the vocabulary, the acoustic semantic distance between the vocabulary is calculated, and the semantic similarity matrix in the industrial scene is constructed according to the acoustic semantic distance; S224, the classification weight is dynamically adjusted by using the transfer learning algorithm combined with the semantic similarity matrix, and the user intention category is outputted.

2. The voice dialogue interaction method for industrial equipment according to claim 1, characterized in that, The method comprises the following steps: S11, the spatially distributed multi-channel voice signal is synchronously collected by using the microphone array pre-configured on the industrial equipment, and the voice signal of each channel is preprocessed to obtain the preprocessed voice signal; S12, based on the preprocessed voice signal, the initial error signal is constructed, and the preprocessed voice signal is preliminarily enhanced by using the incremental adaptive filtering algorithm to obtain the preliminary enhanced voice signal; S13, the transient noise estimation and compensation of the preliminary enhanced voice signal is carried out by using the fixed time interference observer to obtain the final enhanced voice.

3. The voice dialogue interaction method for industrial equipment according to claim 2, characterized in that, The method comprises the following steps: S12, based on the preprocessed voice signal, the initial error signal is constructed, and the preprocessed voice signal is preliminarily enhanced by using the incremental adaptive filtering algorithm to obtain the preliminary enhanced voice signal; S121, for each channel of the preprocessed speech signal, a set of channels is randomly selected as a reference channel, and an initial error signal is constructed according to the preprocessed speech signal of the reference channel; S122, initialize the parameters of the incremental adaptive filtering algorithm, including the filter order, the step factor and the initial filter coefficient; S123, according to the initialized incremental adaptive filtering algorithm, the filter coefficient is iteratively updated using the initial error signal, and the reference channel preprocessed speech signal is filtered to obtain the preliminary enhanced speech signal.

4. The voice dialogue interaction method for industrial equipment according to claim 3, characterized in that, The initial error signal is constructed according to the preprocessed speech signal of the reference channel, including the following steps: frame processing is performed on the preprocessed speech signal of the reference channel, and fast Fourier transform processing is performed on each frame of speech signal to obtain the frequency spectrum of the speech signal; According to the frequency spectrum of the speech signal, the power spectrum of each frequency point is calculated, and on each frequency point, the minimum value search is performed on each frequency point in the preset time sliding window to obtain the minimum power value of each frequency point in the time sliding window as the initial noise power spectrum estimation; The initial estimation value of the noise power of each frequency point is smoothed, and the noise signal is synthesized based on the smoothed noise power estimation value; The initial error signal is obtained by subtracting the synthesized noise signal from the reference channel speech signal.

5. The voice dialogue interaction method for industrial equipment according to claim 1, characterized in that, The acoustic semantic distance between the words is calculated based on the semantic features of the words, and the semantic similarity matrix in the industrial scene is constructed according to the acoustic semantic distance, including the following steps: S2331, for the semantic features of the words, the semantic features are divided into acoustic features and text features by feature decoupling technology; S2332, according to the acoustic features and text features, the acoustic feature vector and text semantic feature vector of each word are extracted respectively, and the semantic embedding vector is generated through fusion mechanism; S2333, based on the fused semantic embedding vector, the acoustic semantic distance between the words is calculated by cosine similarity, and the semantic similarity matrix is constructed according to the calculated acoustic semantic distance.

6. The voice dialogue interaction method for industrial equipment according to claim 5, characterized in that, The classification weight is dynamically adjusted by combining the semantic similarity matrix using the transfer learning algorithm, and the user intent category is output, including the following steps: S2241, initialize the pre-trained intent classification model weight, and use the intent classification model weight as the starting point of transfer learning; S2242, according to the semantic similarity matrix, a graph Laplacian matrix is constructed, and a graph Laplacian regularization term is constructed using the graph Laplacian matrix; S2243, based on the maximum likelihood function, and combining the graph Laplacian regularization term, the intent classification model weight is optimized to obtain the optimized intent classification model; S2244, obtain a new speech text, convert the new speech text into a feature vector using the optimized intent classification model, calculate the probability of each intent category, and select the category with the highest probability as the final user interaction intent.

7. The voice dialogue interaction method for industrial equipment according to claim 1, wherein Based on the enhanced speech, the time delay estimation method is used to identify the user's interaction position, and the user's position is predicted; And according to the position prediction result, the industrial equipment loudspeaker output strategy is constructed, including the following steps: S31, based on the enhanced voice, using generalized cross-correlation-phase transform method to calculate the time delay estimation value between the microphone pairs; S32, obtaining the coordinate position of the microphone array in space, based on the time delay estimation value between the microphone pairs, solving the position estimation coordinates of the user sound source by minimizing the sound source positioning error function, and identifying the current interactive position of the user; S33, predicting the next interactive position of the user based on the historical interactive position and the current recognition result, obtaining the target area, and constructing the loudspeaker output strategy according to the position of the target area.

8. An industrial device-oriented voice dialogue interaction system for implementing the industrial device-oriented voice dialogue interaction method according to any one of claims 1 to 7, characterized by, The system comprises: A voice enhancement processing module is configured to obtain the user's voice signal based on the microphone array pre-configured on the industrial equipment, and perform voice enhancement processing on the user's voice signal through an incremental adaptive filtering algorithm to obtain enhanced voice. An interactive intention recognition module is configured to identify the user's interactive intention using the transfer learning method of information gain based on the enhanced voice, and generate the voice to be interacted based on the user's interactive intention. A strategy construction module is configured to identify the user's interactive position using the time delay estimation method based on the enhanced voice, and to predict the user's position; and to construct the industrial equipment loudspeaker output strategy according to the position prediction result. A voice interaction module is configured to output the voice to be interacted based on the industrial equipment loudspeaker output strategy, so as to realize the voice dialogue interaction between the user and the industrial equipment.

Images