Multilayer natural language interaction method based on deep learning and pattern recognition

Abstract

The invention relates to the technical field of data processing, in particular to a multilayer natural language interaction method based on deep learning and pattern recognition, which comprises the following steps: S1, collecting language interaction information in real time; s2, positioning a sound source, and performing real-time separation shielding; s3, performing parallel multi-user identification according to the visual perception information; s4, adjusting a real-time separation and shielding process of the real-time acquisition process of the language interaction information; step S5, the parallel multi-user identification result is corrected; s6, performing model fine tuning on parallel multi-user identification; s7, performing information fusion to obtain language interaction fusion data; step S8, performing coarse-grained semantic layering on the language interaction fusion data; and step S9, performing fine-grained semantic understanding on the language interaction fusion data. According to the invention, the natural language interaction efficiency in the digital human interaction process is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-layer natural language interaction method based on deep learning and pattern recognition. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technologies, the concept of digital employees has gradually become a reality. In order to achieve multi-layer natural language interaction, it is necessary to solve the challenges of multimodal perception and interaction, including ASR, TTS and visual recognition. Through self-developed multi-channel microphone arrays and acoustic algorithms, it is possible to accurately collect sound in a noisy environment, and achieve dynamic follow-up of multi-person communication through machine vision algorithms. The rise of AI big models provides digital employees with "knowledgeable" thinking ability, combined with industry data, to quickly learn and improve service efficiency. Digital employees need to interact with the physical world, which requires them to be able to control existing intelligent systems and reduce the company's R&D investment. The advancement of 3D visual technology has made it possible to realize the realistic image of digital employees. With the support of AI GC technology, AI-driven technologies for image and video generation and 3D models are maturing. By combining deep learning with pattern recognition and developing multi-level natural language interaction solutions, we can improve the adaptability and business value of digital employees and open up new opportunities for the application of AI in business.

[0003] Chinese Patent Publication No.: CN108009285A discloses a human-computer interaction method for forestry ecological environment based on natural language processing, which belongs to the field of neural networks. This method uses the knowledge graph method to infer the entities of natural language in the forestry ecological environment, so that the knowledge reasoning is converted into the problem of processing natural language questions by constructing a deep neural network, so as to find the corresponding relationship, represent the deep learning reasoning of the knowledge graph under learning, and obtain the corresponding conclusion. The present invention introduces the concept of knowledge graph in deep learning. On the basis of constructing the knowledge graph, the shallow semantic understanding results are injected into the knowledge graph, and a deeper semantic understanding is obtained through corresponding knowledge reasoning. However, this solution does not solve the problem of optimizing the collection of sound sources in the language interaction process, nor can it perform multi-layer semantic understanding and analysis based on accurate and efficient sound source data. Summary of the invention

[0004] To this end, the present invention provides a multi-layer natural language interaction method based on deep learning and pattern recognition to overcome the problem of low efficiency of natural language interaction in the digital human interaction process caused by poor sound source optimization and acquisition effect in the language interaction process and the failure to perform multi-layer semantic understanding and analysis based on accurate and efficient sound source data in the prior art.

[0005] To achieve the above objectives, the present invention provides a multi-layer natural language interaction method based on deep learning and pattern recognition, comprising:

[0006] Step S1, collecting language interaction information in real time through a multi-microphone array device;

[0007] Step S2, locating the sound source according to the language interaction information, performing real-time separation and shielding on the real-time collection process of the language interaction information, and optimizing the multi-microphone array device;

[0008] Step S3, collecting visual perception information in real time, and performing parallel multi-user recognition based on the visual perception information;

[0009] Step S4, adjusting the real-time separation and shielding process of the real-time collection process of the language interaction information according to the parallel multi-user recognition result;

[0010] Step S5, collecting environmental perception information in real time, performing user scene recognition based on the environmental perception information, and correcting the parallel multi-user recognition result based on the user scene recognition result;

[0011] Step S6, judging the visual fine-tuning situation according to the correction ratio of the parallel multi-user recognition result, and fine-tuning the model of the parallel multi-user recognition according to the visual fine-tuning situation;

[0012] Step S7, fusing the language interaction information, the visual perception information and the environmental perception information to obtain language interaction fusion data;

[0013] Step S8, performing coarse-grained semantic stratification on the language interaction fusion data according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information;

[0014] Step S9, after the coarse-grained semantic stratification, fine-grained semantic understanding is performed on the language interaction fusion data according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information.

[0015] Furthermore, in the step S2, the microphone signal generalized cross-correlation function Rx1 x2(τ) is calculated according to the first microphone signal x1(t) and the second microphone signal x2(t) in the language interaction information, and the setting Ψ x1x2(f) is a weighting function, x1(f) is the Fourier transform of x1(t), x2(f) is the Fourier transform of x2(t), x2*(f) is the conjugate complex number of x2(f), e is the base of the natural logarithm, j is the imaginary unit, f is the frequency, which represents the identification of different frequency components in the frequency domain when the signal is converted from the time domain to the frequency domain when analyzing the sound signal received by the microphone, τ is a variable related to the time delay, and the peak value τmax of the generalized cross-correlation function Rx1x2(τ) of the microphone signal is obtained, and the distance difference △r from the sound source to the first microphone and the second microphone is calculated according to the peak value τmax, and △r=τmax×c is set, c is the sound propagation speed, and the angle θ between the sound source and the plane of the microphone array is calculated, and set r is the distance from the sound source to the center of the microphone array, and the distance difference Δr and the angle θ are used as the positioning information of the sound source.

[0016] Furthermore, in step S2, after obtaining the localization information of the sound source, the effective coefficient D=α of the sound source is calculated. 1 ×1 / r+α 2 ×T+α 3 ×△F,α 1 is the first effective coefficient weight parameter, α 2 is the second effective coefficient weight parameter, α 3 is the third effective coefficient weight parameter, T is the duration of the current speech of the sound source, △F refers to the sum of the differences between the sound source and the reference value in each spectral feature, and △F=∑ n k＝1 |Fk-Fk'|, Fk' is the reference value of each spectral feature, the sound source corresponding to the maximum effective coefficient Dmax is taken as the target sound source, and the real-time collection process of the language interaction information of the target sound source is separated and shielded in real time according to the positioning information of the target sound source, and the multi-microphone array device is optimized until the positioning information of the target sound source is that the distance difference is the minimum value and the angle is 90°.

[0017] Furthermore, in step S3, the visual perception information is input into a user recognition deep learning model, and a multi-user recognition result output by the user recognition deep learning model is obtained.

[0018] Furthermore, in step S4, the adjustment of the real-time separation and shielding process of the real-time collection process of the language interaction information is judged according to the parallel multi-user recognition result, wherein:

[0019] When the parallel multi-user recognition result is a single user, determining not to adjust the real-time separation and shielding process of the real-time collection process of the language interaction information;

[0020] When the parallel multi-user recognition result is multiple users, it is determined that the real-time separation and shielding process of the real-time collection process of the language interaction information is adjusted, and the adjusted target sound source is the sound source of multiple users. The real-time collection process of the language interaction information of the adjusted target sound source is separated and shielded in real time according to the positioning information of the adjusted target sound source.

[0021] Further, in step S5, an environment perception feature map is generated according to the environment perception information, and the environment perception feature map is input into the user scene recognition model, a user scene recognition result output by the user scene recognition model is obtained, and the parallel multi-user recognition result is corrected according to the user scene recognition result, wherein:

[0022] When the user scene recognition result is a multi-person interaction scene, the parallel multi-user recognition result is not corrected;

[0023] When the user scene recognition result is a single-person interaction scene, the parallel multi-user recognition result is corrected to correct the parallel multi-user recognition results of multiple users to a single user.

[0024] Further, in the step S6, the parallel multi-user recognition result correction ratio A is calculated according to the number of user scene recognition times L0 and the number of parallel multi-user recognition result correction times L1, and A=L1 / L0 is set. The parallel multi-user recognition result correction ratio A is compared with the preset parallel multi-user recognition result correction ratio A0, and the visual fine-tuning situation is judged according to the comparison result, wherein:

[0025] When A≤A0, it is determined that the visual fine-tuning situation does not require fine-tuning;

[0026] When A>A0, it is determined that the visual fine-tuning situation requires fine-tuning, and the model fine-tuning is performed on the parallel multi-user recognition.

[0027] Furthermore, in step S7, the language interaction information, the visual perception information and the environmental perception information are time synchronized, format unified and normalized to obtain preprocessed language interaction information, preprocessed visual perception information and preprocessed environmental perception information; feature extraction is performed on the preprocessed language interaction information, the preprocessed visual perception information and the preprocessed environmental perception information to obtain language interaction information feature extraction data, visual perception information feature extraction data and environmental perception information feature extraction data; feature layer fusion is performed on the language interaction information feature extraction data, the visual perception information feature extraction data and the environmental perception information feature extraction data according to a multimodal convolutional neural network to obtain feature layer fusion data, which is used as language interaction fusion data.

[0028] Furthermore, in the step S8, the language interaction fusion data is subjected to coarse-grained semantic stratification according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information, and the coarse-grained semantic stratification method includes:

[0029] Step S100, constructing a neural network model including an LSTM layer and a fully connected layer;

[0030] Step S200, setting the input of the neural network model including the LSTM layer and the fully connected layer to the language interaction fusion data obtained through the previous information fusion step, and setting the output of the neural network model including the LSTM layer and the fully connected layer to two parts, one part for intent classification, and the other part for context perception;

[0031] Step S300, collecting language interaction fusion data samples;

[0032] Step S400, dividing the language interaction fusion data samples into a training set, a validation set and a test set;

[0033] Step S500, setting the loss function of the neural network model including the LSTM layer and the fully connected layer;

[0034] Step S600, updating the model parameters of the neural network model including the LSTM layer and the fully connected layer;

[0035] Step S700, outputting the neural network model including the LSTM layer and the fully connected layer;

[0036] Step S800: input the language interaction fusion data into the neural network model including the LSTM layer and the fully connected layer, and obtain the intent classification information and context perception information output by the neural network model including the LSTM layer and the fully connected layer.

[0037] Furthermore, in the step S9, after the coarse-grained semantic stratification, the language interaction fusion data is subjected to fine-grained semantic understanding according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information, and the fine-grained semantic understanding method includes:

[0038] Step S1000, constructing a sentiment dictionary, and performing fine-grained semantic understanding on the language interaction fusion data according to the sentiment dictionary to obtain sentiment analysis information;

[0039] Step S2000, performing fine-grained semantic understanding on the language interaction fusion data through feature engineering and machine learning models to obtain tone analysis information;

[0040] Step S3000, performing fine-grained semantic understanding on the language interaction fusion data by combining an attention mechanism with a deep learning model to obtain deep context analysis information;

[0041] Step S4000: Perform fine-grained semantic understanding on the language interaction fusion data through a polysemy representation learning model to obtain ambiguous resolution information.

[0042] Compared with the prior art, the beneficial effect of the present invention is that the method provides a basic data source for the entire interaction process through the real-time collection of language interaction information in step S1, and can capture sound wave signals from different directions through a multi-microphone array device, thereby enhancing spatial resolution, so that the digital human can obtain surrounding voice information more comprehensively, avoid missing important voice content, and lay the foundation for subsequent accurate understanding of user intentions. The method helps to accurately identify the direction and distance of the sound source in a complex environment through the sound source localization in step S2, and can further accurately estimate the sound source location information by combining the environmental model to estimate the physical position, so that the digital human can focus on the target sound source, distinguish different speakers in a multi-sound source scenario, and collect and process the target user's speech in a targeted manner. The method can effectively improve the quality of voice signals and reduce the impact of interfering sounds on target voices by separating and shielding target sound source signals and background noises in real time, so that the digital human can receive clearer voice commands or information. The sensitivity and directionality of the microphone array can be dynamically adjusted to automatically optimize the acquisition effect according to the sound source position and environmental changes, adapt to the voice interaction needs in different scenarios, and further improve the accuracy and stability of voice acquisition. The method collects visual perception information in real time and performs parallel multi-user recognition in step S3, which enriches the digital human's understanding of user behavior and intentions and makes the interaction more natural and intuitive. The method maintains the current acquisition state in a single-user scenario in step S4 to avoid unnecessary adjustments.In a multi-user scenario, timely adjustment is made to the collection and separation shielding of multiple user sound sources to ensure that each user's voice can be effectively processed, thereby improving the accuracy and adaptability of voice collection in a multi-user interaction scenario and avoiding confusion or omissions in sound collection. The method provides the digital human with contextual information about the interaction environment through step S5. The digital human can judge the type of the current environment by generating an environmental perception feature map through sensor data such as humidity, temperature, and light, and combining it with a user scenario recognition model. The parallel multi-user recognition results are corrected according to the scenario recognition results, thereby improving the accuracy of user recognition and avoiding misjudgment caused by environmental factors. The digital human can more accurately recognize user situations in different scenarios, optimize interaction strategies and The method calculates the correction ratio of parallel multi-user recognition results and compares it with a preset value in step S6 to judge the visual fine-tuning situation. This judgment method based on data statistics is scientific and reasonable, and can accurately evaluate the performance of the current visual model in practical applications. When fine-tuning is required, the visual model is optimized using transfer learning technology, which can make full use of the knowledge and experience of existing models, quickly adapt to the specific needs of digital human language interaction scenarios, and improve the accuracy of the visual model for parallel multi-user recognition, thereby improving the performance and stability of the entire system in complex scenarios. The method integrates language interaction information, visual perception information and environmental perception information into an organic whole through information fusion in step S7, forming a more comprehensive and rich language interaction The mutual fusion data, combined with voice commands and user's behavior in a specific environment, can more accurately understand the user's real needs, provide more powerful data support for subsequent semantic understanding, improve the accuracy and depth of semantic understanding, and thus enhance the digital human's ability to understand the user's intentions and the accuracy of interactive responses. The method performs preliminary processing on the fusion data through coarse-grained semantic stratification in step S8 to obtain intention classification information, so that the digital human can quickly identify the approximate purpose of the user's voice, such as querying information, issuing commands, expressing emotions, etc. The method performs further in-depth analysis on the basis of coarse-grained semantic understanding through step S9 to obtain emotional analysis information, which can enable the digital human to perceive the user's emotional state, such as happiness, sadness, anger, etc. When the user is in a bad mood, the digital human can respond in a gentler and more comforting tone, which helps the digital human understand the tone characteristics of the user's speech, such as statements, questions, imperatives, exclamations, etc., better grasp the user's intentions and attitudes, and make more appropriate responses. Deep context analysis information enables the digital human to deeply understand the contextual meaning behind the voice, accurately understand pronoun references, implicit information, etc., improve the ability to understand complex semantics, and avoid misunderstanding user intentions. Ambiguity resolution information solves possible ambiguous problems in the language, ensuring that the digital human's understanding of the user's voice is accurate, further improving the accuracy and fluency of the interaction, and improving the user experience. ; BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the process of the multi-layer natural language interaction method based on deep learning and pattern recognition in this embodiment. DETAILED DESCRIPTION

[0044] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0046] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0047] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] See also Figure 1 As shown, it is a flow chart of the multi-layer natural language interaction method based on deep learning and pattern recognition in this embodiment, and the method includes:

[0049] Step S1, collecting language interaction information in real time through a multi-microphone array device;

[0050] Step S2, locating the sound source according to the language interaction information, performing real-time separation and shielding on the real-time collection process of the language interaction information, and optimizing the multi-microphone array device;

[0051] Step S3, collecting visual perception information in real time, and performing parallel multi-user recognition based on the visual perception information;

[0052] Step S4, adjusting the real-time separation and shielding process of the real-time collection process of the language interaction information according to the parallel multi-user recognition result;

[0053] Step S5, collecting environmental perception information in real time, performing user scene recognition based on the environmental perception information, and correcting the parallel multi-user recognition result based on the user scene recognition result;

[0054] Step S6, judging the visual fine-tuning situation according to the correction ratio of the parallel multi-user recognition result, and fine-tuning the model of the parallel multi-user recognition according to the visual fine-tuning situation;

[0055] Step S7, fusing the language interaction information, the visual perception information and the environmental perception information to obtain language interaction fusion data;

[0056] Step S8, performing coarse-grained semantic stratification on the language interaction fusion data according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information;

[0057] Step S9, after the coarse-grained semantic stratification, fine-grained semantic understanding is performed on the language interaction fusion data according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information.

[0058] Specifically, the method is set in a digital human language interaction terminal, and improves the efficiency of natural language interaction in the digital human interaction process by optimizing the sound source collection during the language interaction process and performing multi-layer semantic understanding and analysis based on accurate and efficient sound source data. The method provides a basic data source for the entire interaction process through the real-time collection of language interaction information in step S1. Through a multi-microphone array device, it can capture sound wave signals from different directions and enhance spatial resolution, so that the digital human can more comprehensively obtain surrounding voice information and avoid missing important voice content, thereby laying the foundation for subsequent accurate understanding of user intentions. The method helps to accurately identify the direction and distance of the sound source in a complex environment through the sound source localization in step S2. The sound source location information can be further refined by combining the environmental model to estimate the physical position, so that the digital human can focus on the target. Sound source, distinguish different speakers in a multi-sound source scenario, and collect and process the target user's voice in a targeted manner. The method can effectively improve the quality of voice signals and reduce the impact of interfering sounds on the target voice by separating and shielding the target sound source signal and background noise in real time, so that the digital human can receive clearer voice commands or information. The sensitivity and directionality of the microphone array are dynamically adjusted so that it can automatically optimize the collection effect according to the sound source position and environmental changes, adapt to the voice interaction needs in different scenarios, and further improve the accuracy and stability of voice collection. The method collects visual perception information in real time and performs parallel multi-user recognition in step S3, enriching the digital human's understanding of user behavior and intentions, making the interaction more natural and intuitive. The method maintains the current collection state in a single-user scenario through step S4 to avoid unnecessary adjustments;In a multi-user scenario, timely adjustment is made to the collection and separation shielding of multiple user sound sources to ensure that each user's voice can be effectively processed, thereby improving the accuracy and adaptability of voice collection in a multi-user interaction scenario and avoiding confusion or omissions in sound collection. The method provides the digital human with contextual information about the interaction environment through step S5. The digital human can judge the type of the current environment by generating an environmental perception feature map through sensor data such as humidity, temperature, and light, and combining it with a user scenario recognition model. The parallel multi-user recognition results are corrected according to the scenario recognition results, thereby improving the accuracy of user recognition and avoiding misjudgment caused by environmental factors. The digital human can more accurately recognize user situations in different scenarios, optimize interaction strategies and The method calculates the correction ratio of parallel multi-user recognition results and compares it with a preset value in step S6 to judge the visual fine-tuning situation. This judgment method based on data statistics is scientific and reasonable, and can accurately evaluate the performance of the current visual model in practical applications. When fine-tuning is required, the visual model is optimized using transfer learning technology, which can make full use of the knowledge and experience of existing models, quickly adapt to the specific needs of digital human language interaction scenarios, and improve the accuracy of the visual model for parallel multi-user recognition, thereby improving the performance and stability of the entire system in complex scenarios. The method integrates language interaction information, visual perception information and environmental perception information into an organic whole through information fusion in step S7, forming a more comprehensive and rich language interaction The mutual fusion data, combined with voice commands and user's behavior in a specific environment, can more accurately understand the user's real needs, provide more powerful data support for subsequent semantic understanding, improve the accuracy and depth of semantic understanding, and thus enhance the digital human's ability to understand the user's intentions and the accuracy of interactive responses. The method performs preliminary processing on the fusion data through coarse-grained semantic stratification in step S8 to obtain intention classification information, so that the digital human can quickly identify the approximate purpose of the user's voice, such as querying information, issuing commands, expressing emotions, etc. The method performs further in-depth analysis on the basis of coarse-grained semantic understanding through step S9 to obtain emotional analysis information, which can enable the digital human to perceive the user's emotional state, such as happiness, sadness, anger, etc. When the user is in a bad mood, the digital human can respond in a gentler and more comforting tone, which helps the digital human understand the tone characteristics of the user's speech, such as statements, questions, imperatives, exclamations, etc., better grasp the user's intentions and attitudes, and make more appropriate responses. Deep context analysis information enables the digital human to deeply understand the contextual meaning behind the voice, accurately understand pronoun references, implicit information, etc., improve the ability to understand complex semantics, and avoid misunderstanding user intentions. Ambiguity resolution information solves possible ambiguous problems in the language, ensuring that the digital human's understanding of the user's voice is accurate, further improving the accuracy and fluency of the interaction, and improving the user experience. ;

[0059] Specifically, in step S1, the language interaction information includes each microphone signal of multiple microphone arrays, the current speech duration of the sound source and each spectral feature of the sound source. Each microphone signal of the multiple microphone arrays refers to the speech signal collected by each microphone in the multiple microphone arrays.

[0060] Specifically, in step S1, a configuration of multiple microphone arrays is used to capture sound wave signals from different directions, enhance spatial resolution, support subsequent sound source localization and noise suppression, and achieve real-time capture of voice signals through fast sampling and data transmission technology. The data is stored in a high-speed buffer to support subsequent fast processing.

[0061] Specifically, in step S2, the microphone signal generalized cross-correlation function Rx1 x2(τ) is calculated according to the first microphone signal x1(t) and the second microphone signal x2(t) in the language interaction information, and the setting Ψ x1x2 (f) is a weighting function, x1(f) is the Fourier transform of x1(t), x2(f) is the Fourier transform of x2(t), x2*(f) is the conjugate complex number of x2(f), e is the base of the natural logarithm, j is the imaginary unit, f is the frequency, which represents the identification of different frequency components in the frequency domain when the signal is converted from the time domain to the frequency domain when analyzing the sound signal received by the microphone, τ is a variable related to the time delay, and the peak value τmax of the generalized cross-correlation function Rx1x2(τ) of the microphone signal is obtained, and the distance difference △r from the sound source to the first microphone and the second microphone is calculated according to the peak value τmax, and △r=τmax×c is set, c is the sound propagation speed, and the angle θ between the sound source and the plane of the microphone array is calculated, and set r is the distance from the sound source to the center of the microphone array, and the distance difference Δr and the angle θ are used as the positioning information of the sound source.

[0062] Specifically, the first microphone signal refers to a microphone signal obtained by a single microphone randomly designated in the multi-microphone array device collecting the sound source, and the second microphone signal refers to a microphone signal obtained by another single microphone randomly designated in the multi-microphone array device collecting the sound source. This embodiment does not limit the random designation method in the multi-microphone array device, and those skilled in the art can freely set it according to actual conditions. For example, each microphone in the multi-microphone array device can be digitally marked, and two microphones corresponding to random numbers are selected according to a random number generator as the first microphone and the second microphone, respectively. The microphone signal collected by the first microphone is used as the first microphone signal, and the microphone signal collected by the second microphone is used as the second microphone signal. The sound source refers to the sound source of the user who performs voice interaction in front of the digital human language interaction terminal. The microphone array plane refers to the plane formed by connecting the microphones in the multi-microphone array device. The microphone array center refers to the center of the microphone array plane. This embodiment does not limit the method for obtaining the distance from the sound source to the center of the microphone array. Those skilled in the art can freely set it according to actual conditions, and only need to meet the collection requirements of the distance from the sound source to the center of the microphone array. For example, the distance from the sound source to the center of the microphone array can be collected by a sonar sensor.

[0063] Specifically, in step S2, after obtaining the localization information of the sound source, the effective coefficient D of the sound source is calculated. 1 ×1 / r+α 2 ×T+α 3 ×△F,α 1 is the first effective coefficient weight parameter, α 2 is the second effective coefficient weight parameter, α 3 is the third effective coefficient weight parameter, T is the duration of the current speech of the sound source, △F refers to the sum of the differences between the sound source and the reference value in each spectral feature, and △F=∑ n k＝1 |Fk-Fk'|, Fk' is the reference value of each spectral feature, the sound source corresponding to the maximum effective coefficient Dmax is taken as the target sound source, and the real-time collection process of the language interaction information of the target sound source is separated and shielded in real time according to the positioning information of the target sound source, and the multi-microphone array device is optimized until the positioning information of the target sound source is that the distance difference is the minimum value and the angle is 90°.

[0064] Specifically, this embodiment does not limit the setting method of the effective coefficient weight parameter. Those skilled in the art can freely set it according to the focus requirements, and only need to meet the recognition requirements of the target sound source. For example, the first effective coefficient weight parameter can be set to 0.5, the second effective coefficient weight parameter can be set to 0.3, and the third effective coefficient weight parameter can be set to 0.2. The current voice duration of the sound source refers to the duration of the continuous voice output of the same sound source in front of the digital human language interaction terminal at the current moment. The reference value of each spectral feature refers to the preset reference value of the energy distribution of the voice signal in different frequency bands. This embodiment does not limit the specific parameters of the reference value of each spectral feature. Those skilled in the art can freely set it according to the actual situation, and only need to meet the reflection requirements of the energy distribution. For example, the reference value of each spectral feature can be preset according to different setting scenarios of the digital human language interaction terminal. The maximum effective coefficient refers to the maximum effective coefficient of the digital human language. The maximum value of the effective coefficients of each sound source performing voice interaction in front of the speech interaction terminal, the positioning information of the target sound source refers to the distance difference △r and the angle θ of the target sound source, the real-time separation and shielding means that the digital human language interaction terminal only accepts the language interaction information from the target sound source, this embodiment does not specifically limit the method of the real-time separation and shielding, and technical personnel in this field can freely set it according to actual conditions, and only need to meet the need for separate reception of the language interaction information of the target sound source, such as real-time separation and shielding through blind source separation technology. This embodiment does not limit the method for optimizing the multi-microphone array device, and technical personnel in this field can freely set it according to actual conditions, and only need to meet the optimization needs, such as controlling each microphone through the built-in controller in the multi-microphone array device, and moving each microphone through the slide rail until the positioning information of the target sound source is that the distance difference is the minimum value and the angle is 90°.

[0065] Specifically, in step S3, the visual perception information is input into a user recognition deep learning model, and a multi-user recognition result output by the user recognition deep learning model is obtained.

[0066] Specifically, the visual perception information refers to the image information in front of the digital human language interaction terminal, including visible light image data, infrared image data and lidar data. The user recognition deep learning model refers to a neural network model that obtains the parallel multi-user recognition result after analyzing the visual perception information. In this embodiment, the basic framework of the user recognition deep learning model is set as a convolutional neural network model, and the historical visual perception information-historical multi-user recognition results are used as the recognition model construction data set, 70% of the recognition model construction data set is used as the recognition model training set, and 30% of the recognition model construction data set is used as the recognition model verification set. The convolutional neural network model is trained according to the recognition model training set to obtain the trained convolutional neural network Model, the trained convolutional neural network model is verified according to the recognition model verification set, and after the accuracy reaches 98%, the trained convolutional neural network model is output as a user recognition deep learning model. This embodiment does not specifically limit the construction parameters of the convolutional neural network model. Those skilled in the art can freely set them according to actual conditions, as long as the multi-user recognition requirements are met. For example, the loss function of the convolutional neural network model can be set to a cross entropy function. The parallel multi-user recognition result refers to the result obtained by the user recognition deep learning model according to the visual perception information, indicating whether the user interacting in front of the digital human language interaction terminal is a single user or multiple users, including a single user and multiple users.

[0067] Specifically, in step S4, the adjustment of the real-time separation and shielding process of the real-time collection process of the language interaction information is judged according to the parallel multi-user recognition result, wherein:

[0068] When the parallel multi-user recognition result is a single user, determining not to adjust the real-time separation and shielding process of the real-time collection process of the language interaction information;

[0069] When the parallel multi-user recognition result is multiple users, it is determined that the real-time separation and shielding process of the real-time collection process of the language interaction information is adjusted, and the adjusted target sound source is the sound source of multiple users. The real-time collection process of the language interaction information of the adjusted target sound source is separated and shielded in real time according to the positioning information of the adjusted target sound source.

[0070] Specifically, in step S5, an environment perception feature map is generated according to the environment perception information, and the environment perception feature map is input into the user scene recognition model, a user scene recognition result output by the user scene recognition model is obtained, and the parallel multi-user recognition result is corrected according to the user scene recognition result, wherein:

[0071] When the user scene recognition result is a multi-person interaction scene, the parallel multi-user recognition result is not corrected;

[0072] When the user scene recognition result is a single-person interaction scene, the parallel multi-user recognition result is corrected to correct the parallel multi-user recognition results of multiple users to a single user.

[0073] Specifically, the environmental perception information refers to the information collected by the Internet of Things sensors such as humidity sensors, temperature sensors and light sensors on the interactive environment in which the digital human language interaction terminal is located. The environmental perception feature map refers to a data feature map related to the interactive environment in which the digital human language interaction terminal is located, which is generated according to the environmental perception information. This embodiment does not limit the method of generating the environmental perception feature map. Those skilled in the art can freely set it according to actual conditions, and only need to meet the needs of converting the graphic features of the environmental perception information. For example, it can be set to generate the environmental perception feature map by direct mapping based on sensor data. When various sensors such as temperature sensors, humidity sensors, light sensors, After collecting environmental perception information, gas sensors, etc., these sensor data can be directly mapped to the pixel values ​​of the image according to certain rules, thereby generating an environmental data feature map. Taking the temperature sensor as an example, it is assumed that multiple temperature sensors are arranged on the digital human language interaction terminal to form a sensor network for monitoring the temperature at different locations. First, determine the size and resolution of the feature map. For example, a 50×50 pixel feature map is generated to represent the indoor temperature distribution. Then, the indoor space is divided according to the division rules, such as uniform grid division, corresponding to the pixels of the feature map. For example, the entire indoor space is divided into 2500 small areas corresponding to 50×50 pixels, and each small area corresponds to a sensor. Monitoring range, normalize the data collected by each temperature sensor so that its value falls within a suitable range, such as 0 to 255, so as to match the image pixel value range. Suppose the temperature value collected by a certain temperature sensor is 20°C, and the value obtained after normalization is 100. Assign the normalized temperature value to the pixel corresponding to the corresponding small area, so as to generate an environmental data feature map representing the indoor temperature distribution. Different pixel grayscale values ​​or color values ​​in the map reflect the temperature conditions at different locations. The user scene recognition model refers to a deep learning model that takes the environmental perception feature map as input and the user scene recognition result as output. This embodiment does not limit the construction method of the user scene recognition model. Technical personnel in the field can freely set it according to actual conditions, and only need to meet the requirements for identifying the interactive environment in which the digital human language interaction terminal is located. For example, the user scenario recognition model can be set as a convolutional neural network model, and the user scenario recognition result refers to the recognition result of the interactive environment in which the digital human language interaction terminal is located obtained according to the environmental perception feature map, including multi-person interaction scenarios and single-person interaction scenarios. The multi-person interaction scenario refers to the situation where the interactive objects in the interactive environment in which the digital human language interaction terminal is located have multi-person interactions, such as schools, hospitals and subway stations. The single interaction scenario refers to the situation where the interactive objects in the interactive environment in which the digital human language interaction terminal is located do not have multi-person interactions, and only single-person interactions are possible, such as private offices.

[0074] Specifically, in step S6, the parallel multi-user recognition result correction ratio A is calculated according to the number of user scene recognition times L0 and the number of parallel multi-user recognition result correction times L1, and A=L1 / L0 is set. The parallel multi-user recognition result correction ratio A is compared with the preset parallel multi-user recognition result correction ratio A0, and the visual fine-tuning situation is judged according to the comparison result, wherein:

[0075] When A≤A0, it is determined that the visual fine-tuning situation does not require fine-tuning;

[0076] When A>A0, it is determined that the visual fine-tuning situation requires fine-tuning, and the model fine-tuning is performed on the parallel multi-user recognition.

[0077] Specifically, the number of user scene recognitions refers to the total number of user scene recognitions performed according to the environmental perception information in step S5 within the fine-tuning monitoring period, the number of parallel multi-user recognition result corrections refers to the total number of parallel multi-user recognition results corrected according to the user scene recognition results in step S5 within the fine-tuning monitoring period, the fine-tuning monitoring period refers to a preset period for monitoring the correction situation, the preset parallel multi-user recognition result correction ratio refers to a parameter reflecting the correction situation, the visual fine-tuning situation refers to the possibility of judging whether to fine-tune the model for parallel multi-user recognition based on the correction situation, including whether fine-tuning is required and whether fine-tuning is required. This embodiment uses transfer learning technology to fine-tune the model for parallel multi-user recognition. The basic idea of ​​transfer learning is to use existing model parameters that have been trained on other related tasks or data sets to initialize the model of the current task, and then further train on the data set of the current task. Assuming that there is already a large-scale communication The convolutional neural network model pre-trained on the image dataset has good feature extraction capabilities, but it is not accurate enough for parallel multi-user recognition in the current digital human language interaction scenario. First, the parameters of some layers of the pre-trained model are kept unchanged as the initialization parameters of the new model, such as some of the previous convolutional layers. These layers have learned common image features, such as edges, textures, etc., and a new training dataset is constructed for the visual perception information collected in the digital human language interaction scenario and its corresponding parallel multi-user recognition result labels, including visible light image data, infrared image data, and lidar data. Then, the following layers of the model are retrained, such as the fully connected layer, and the parameters of the layer are adjusted so that the model can better adapt to the parallel multi-user recognition task in the digital human language interaction scenario. During the training process, according to the characteristics of the new dataset and task requirements, appropriate loss functions, such as cross entropy functions, and optimization algorithms, such as stochastic gradient descent, are selected to update the model parameters.

[0078] Specifically, in step S7, the language interaction information, the visual perception information and the environmental perception information are time synchronized, format unified and normalized to obtain preprocessed language interaction information, preprocessed visual perception information and preprocessed environmental perception information, feature extraction is performed on the preprocessed language interaction information, the preprocessed visual perception information and the preprocessed environmental perception information to obtain language interaction information feature extraction data, visual perception information feature extraction data and environmental perception information feature extraction data, feature layer fusion is performed on the language interaction information feature extraction data, the visual perception information feature extraction data and the environmental perception information feature extraction data according to a multimodal convolutional neural network to obtain feature layer fusion data, which is used as language interaction fusion data.

[0079] Specifically, when the language interaction information, the visual perception information and the environmental perception information are synchronized in time, a timestamp technology is used to mark the time of the language interaction information, the visual perception information and the environmental perception information when they are collected, and then the language interaction information, the visual perception information and the environmental perception information collected within the same time window are associated. For example, when a multi-microphone array device collects language interaction information, the timestamp of the collection time is recorded at the same time; when a camera collects visual perception information, the corresponding timestamp is also marked; when various sensors collect environmental perception information, the timestamp is also recorded. By comparing these timestamps, data within the same or similar time range are regarded as multimodal data at the same time for fusion processing. When the language interaction information, the visual perception information and the environmental perception information are formatted and normalized, the image pixel values ​​in the visual perception information (usually between 0-255) and the temperature values ​​in the environmental perception information (which may be at a larger actual temperature) are combined. range), normalize them to between 0-1 or between -1 and 1 by linear mapping, and when extracting features from the preprocessed language interaction information, the preprocessed visual perception information and the preprocessed environmental perception information, use Fourier transform to obtain the spectrum of the speech in the preprocessed language interaction information, and analyze the frequency components and energy distribution therein to extract spectrum features; calculate the rhythmic features by the duration and pitch changes of the speech signal in the preprocessed language interaction information; use natural language processing tools to perform lexical and syntactic analysis on the text after speech transcription in the preprocessed language interaction information to obtain semantic features; use the face recognition technology in the computer vision algorithm to obtain the coordinates of the facial feature points in the preprocessed visual perception information, determine the human body posture angle by the posture estimation algorithm, analyze the temperature gradient of the infrared image in the preprocessed visual perception information to determine the thermal imaging features, and build a three-dimensional model of the environment according to the point cloud data of the laser radar in the preprocessed visual perception information and extract the geometric features of the object;Calculate the average value and variance of the humidity sensor data collected in the pre-processed environmental perception information within a preset time, analyze the difference between the maximum and minimum values ​​of the temperature sensor data in the pre-processed environmental perception information in different time periods to obtain the temperature fluctuation range, determine the illumination direction according to the measured values ​​of the illumination sensor in different directions in the pre-processed environmental perception information, and when performing feature layer fusion, design each branch of the multimodal convolutional neural network to process the language interaction information feature extraction data, the visual perception information feature extraction data, and the environmental perception information feature extraction data, respectively, and then perform information fusion and interaction in the subsequent layers of the multimodal convolutional neural network. This embodiment does not limit the construction method of the multimodal convolutional neural network, and those skilled in the art can freely set it according to actual conditions, as long as the data fusion requirements are met. For example, when constructing the multimodal convolutional neural network, its input layer determines the number of nodes according to the feature dimension of the fused data, and the hidden layer can be set to multiple layers. The complex relationship in the multimodal data is learned by adjusting the number of neurons and activation functions of each layer. The output layer determines the number of nodes and the output format according to the goal of language interaction fusion data, such as generating intent classification information, context perception information, etc. ;

[0080] Specifically, in step S8, the language interaction fusion data is subjected to coarse-grained semantic stratification according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information. The coarse-grained semantic stratification method includes:

[0081] Step S100, constructing a neural network model including an LSTM layer and a fully connected layer, wherein the LSTM layer is used to model the temporal information in the language interaction fusion data, and the fully connected layer is used to map the output of the LSTM layer to the dimensions of intent classification and context perception;

[0082] Step S200, setting the input of the neural network model including the LSTM layer and the fully connected layer to the language interaction fusion data obtained through the previous information fusion step, setting the output of the neural network model including the LSTM layer and the fully connected layer to two parts, one part is used for intent classification, the number of output categories of the intent classification is determined according to predefined intent categories, including query information, issue instructions, express emotions and other categories, the number of output nodes of the neural network model including the LSTM layer and the fully connected layer can be set to the number of categories, and the probability distribution of each intent classification is obtained through the Softmax activation function, and the other part is used for context perception, outputting feature vectors related to the context; such as vectors representing the current conversation topic, key information of the conversation history, etc., whose dimensions are determined according to specific context modeling requirements;

[0083] Step S300, collecting language interaction fusion data samples, manually annotating the language interaction fusion data samples, obtaining the intent category to which each manually annotated sample belongs, and obtaining manually annotated context information related to each sample; the context information related to each sample includes the conversation topic, the key entities mentioned above, etc.;

[0084] Step S400, dividing the language interaction fusion data samples into a training set, a validation set and a test set; the training set is used for training the neural network model including the LSTM layer and the fully connected layer, accounting for 80% of the language interaction fusion data samples; the validation set is used to evaluate the performance of the neural network model including the LSTM layer and the fully connected layer during the training process, accounting for 10% of the language interaction fusion data samples; the test set is used to finally evaluate the generalization ability of the neural network model including the LSTM layer and the fully connected layer on unseen data, accounting for 10% of the language interaction fusion data samples;

[0085] Step S500, setting the loss function of the neural network model including the LSTM layer and the fully connected layer; for the intent classification part of the neural network model including the LSTM layer and the fully connected layer, the cross entropy loss function is adopted, and the formula is c is the number of intent classification categories, yi is the element (0 or 1) in the one-hot encoding vector corresponding to the category of the sample's true intent classification, yi' is the element in the category probability distribution vector of the intent classification predicted by the model, and for the context-aware part, the mean square error (MSE) loss function can be used, and the formula is n is the dimension of the context-aware vector, bi is the element in the real context-aware vector of the sample, and bi' is the element in the context-aware vector predicted by the neural network model including the LSTM layer and the fully connected layer;

[0086] Step S600, updating the model parameters of the neural network model including the LSTM layer and the fully connected layer; using the stochastic gradient descent (SGD) and its variant (Adam) optimization algorithm to update the model parameters;

[0087] Step S700, outputting the neural network model including the LSTM layer and the fully connected layer; inputting the training set into the neural network model including the LSTM layer and the fully connected layer in batches, calculating the loss function value, and then back-propagating the gradient through the optimization algorithm to update the parameters of the neural network model including the LSTM layer and the fully connected layer; after each training cycle, using the validation set to evaluate the performance of the neural network model including the LSTM layer and the fully connected layer, the performance of the neural network model including the LSTM layer and the fully connected layer includes the accuracy of intent classification, the similarity index between the context-aware vector and the real context information; if the performance does not improve or there is a sign of overfitting, such as the validation set loss starts to rise, stop training in advance, and adjust the model parameters, such as increasing the regularization strength, adjusting the learning rate, etc., continue training until the model achieves good performance on the validation set, and then output the model as the neural network model including the LSTM layer and the fully connected layer;

[0088] Step S800: input the language interaction fusion data into the neural network model including the LSTM layer and the fully connected layer, and obtain the intent classification information and context perception information output by the neural network model including the LSTM layer and the fully connected layer.

[0089] Specifically, in step S9, after coarse-grained semantic stratification, fine-grained semantic understanding is performed on the language interaction fusion data according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information. The fine-grained semantic understanding method includes:

[0090] Step S1000, constructing a sentiment dictionary, and performing fine-grained semantic understanding on the language interaction fusion data according to the sentiment dictionary to obtain sentiment analysis information;

[0091] Step S2000, performing fine-grained semantic understanding on the language interaction fusion data through feature engineering and machine learning models to obtain tone analysis information;

[0092] Step S3000, performing fine-grained semantic understanding on the language interaction fusion data by combining an attention mechanism with a deep learning model to obtain deep context analysis information;

[0093] Step S4000: Perform fine-grained semantic understanding on the language interaction fusion data through a polysemy representation learning model to obtain ambiguous resolution information.

[0094] Specifically, the sentiment dictionary refers to a sentiment dictionary that collects rich sentiment words and their sentiment tendencies such as positive, negative, neutral, and intensity. It builds a custom sentiment dictionary according to specific fields and application scenarios to better adapt to the language characteristics of digital human interaction. When the language interaction fusion data is subjected to fine-grained semantic understanding according to the sentiment dictionary to obtain sentiment analysis information, the text in the language interaction fusion data is matched with the words in the sentiment dictionary. For each matched sentiment word, a corresponding score is assigned according to its sentiment tendency and intensity in the dictionary. For example, "happy" is assigned a positive score of +3, and "sad" is assigned a negative score of -3. Then, by summing the scores of all sentiment words in the text, the sentiment tendency score of the text is obtained. The sentiment category is judged according to the score. For example, a score greater than 0 is a positive emotion, less than 0 is a negative emotion, and close to 0 is a neutral emotion, thereby obtaining sentiment analysis information;When performing fine-grained semantic understanding on the language interaction fusion data through feature engineering and machine learning models, features related to tone are extracted from the language interaction fusion data, such as the use of punctuation marks (the frequency of exclamation marks, question marks, etc.), parts of speech of vocabulary (such as the proportion and type of verbs, adjectives, adverbs, etc.), sentence structure (the proportion of declarative sentences, interrogative sentences, imperative sentences, exclamatory sentences, etc.), intonation change features (if there is voice intonation information, its rising, falling, flat and other features can be analyzed), etc., and the features related to tone are used to train a machine learning model, such as Naive Bayes, to annotate the sentences with different tone types (such as declarative tone, interrogative tone, imperative tone, The language interaction data of the fusion data (such as exclamatory tone, etc.) is used as a training set, and the relationship between the features and the tone type is learned through a machine learning model. The features of the language interaction fusion data are input into the trained machine learning model, and the machine learning model predicts its tone type to obtain tone analysis information. When the language interaction fusion data is subjected to fine-grained semantic understanding by combining the attention mechanism with the deep learning model, the attention mechanism is introduced into the Transformer deep learning model, so that the Transformer deep learning model can pay attention to the important parts of the text related to the context. The attention mechanism assigns different weights according to the relevance of different parts of the input text to the current task, and highlights the key information in the text. Through training on a large amount of language interaction data, the Transformer deep learning model learns how to understand the deep context according to the context and attention weight. For example, in a conversation about tourism, when "that place" is mentioned, the Transformer deep learning model pays attention to the relevant information of the tourist destination mentioned earlier through the attention mechanism, so as to accurately understand the specific content referred to by "that place", and then obtains deep context analysis information, grasps the context of the entire conversation, and then analyzes the language interaction fusion data through the polysemous representation learning model. When combining data for fine-grained semantic understanding, a deep learning model is used as the polysemy representation learning model to learn multiple semantic representations of words and sentences. For example, by training on a large-scale corpus, the polysemy representation learning model can learn different vector representations of "bank" in different contexts. In practical applications, according to the context of language interaction fusion data, the polysemy representation learning model selects the semantic representation that best fits the context. The polysemy representation learning model calculates the correlation or probability between different semantic representations and the context, and selects the semantic representation with the highest probability or the strongest correlation as the final understanding result, thereby achieving ambiguity resolution and obtaining ambiguity resolution information. ;

[0095] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A multi-layer natural language interaction method based on deep learning and pattern recognition, characterized in that: include: Step S1, collecting language interaction information in real time through a multi-microphone array device; Step S2, locating the sound source according to the language interaction information, performing real-time separation and shielding on the real-time collection process of the language interaction information, and optimizing the multi-microphone array device; Step S3, collecting visual perception information in real time, and performing parallel multi-user recognition based on the visual perception information; Step S4, adjusting the real-time separation and shielding process of the real-time collection process of the language interaction information according to the parallel multi-user recognition result; Step S5, collecting environmental perception information in real time, performing user scene recognition based on the environmental perception information, and correcting the parallel multi-user recognition result based on the user scene recognition result; Step S6, judging the visual fine-tuning situation according to the correction ratio of the parallel multi-user recognition result, and fine-tuning the model of the parallel multi-user recognition according to the visual fine-tuning situation; Step S7, fusing the language interaction information, the visual perception information and the environmental perception information to obtain language interaction fusion data; Step S8, performing coarse-grained semantic stratification on the language interaction fusion data according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information; Step S9, after the coarse-grained semantic stratification, fine-grained semantic understanding is performed on the language interaction fusion data according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information.

2. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 1 is characterized in that: In step S2, the microphone signal generalized cross-correlation function Rx1 x2(τ) is calculated according to the first microphone signal x1(t) and the second microphone signal x2(t) in the language interaction information, and the setting Ψ x1x2 (f) is a weighting function, x1(f) is the Fourier transform of x1(t), x2(f) is the Fourier transform of x2(t), x2*(f) is the conjugate complex number of x2(f), e is the base of the natural logarithm, j is the imaginary unit, f is the frequency, which represents the identification of different frequency components in the frequency domain when the signal is converted from the time domain to the frequency domain when analyzing the sound signal received by the microphone, τ is a variable related to the time delay, and the peak value τmax of the generalized cross-correlation function Rx1x2(τ) of the microphone signal is obtained, and the distance difference △r from the sound source to the first microphone and the second microphone is calculated according to the peak value τmax, and △r=τmax×c is set, c is the sound propagation speed, and the angle θ between the sound source and the plane of the microphone array is calculated, and set r is the distance from the sound source to the center of the microphone array, and the distance difference Δr and the angle θ are used as the positioning information of the sound source.

3. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 2 is characterized in that: In step S2, after obtaining the localization information of the sound source, the effective coefficient D of the sound source is calculated as α1×1 / r+α2×T+α3×ΔF, where α1 is a first effective coefficient weight parameter, α2 is a second effective coefficient weight parameter, α3 is a third effective coefficient weight parameter, T is the duration of the current speech of the sound source, ΔF refers to the sum of the differences between the sound source and the reference value in each spectral feature, and setting Fk' is the reference value of each spectral feature, and the sound source corresponding to the maximum effective coefficient Dmax is taken as the target sound source. The real-time collection process of the language interaction information of the target sound source is separated and shielded in real time according to the positioning information of the target sound source, and the multi-microphone array device is optimized until the positioning information of the target sound source is that the distance difference is the minimum value and the angle is 90°.

4. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 3 is characterized in that: In the step S3, the visual perception information is input into a user recognition deep learning model, and a multi-user recognition result output by the user recognition deep learning model is obtained.

5. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 4 is characterized in that: In step S4, the adjustment of the real-time separation and shielding process of the real-time collection process of the language interaction information is judged according to the parallel multi-user recognition result, wherein: When the parallel multi-user recognition result is a single user, determining not to adjust the real-time separation and shielding process of the real-time collection process of the language interaction information; When the parallel multi-user recognition result is multiple users, it is determined that the real-time separation and shielding process of the real-time collection process of the language interaction information is adjusted, and the adjusted target sound source is the sound source of multiple users. The real-time collection process of the language interaction information of the adjusted target sound source is separated and shielded in real time according to the positioning information of the adjusted target sound source.

6. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 5, characterized in that: In step S5, an environment perception feature map is generated according to the environment perception information, and the environment perception feature map is input into the user scene recognition model, a user scene recognition result output by the user scene recognition model is obtained, and the parallel multi-user recognition result is corrected according to the user scene recognition result, wherein: When the user scene recognition result is a multi-person interaction scene, the parallel multi-user recognition result is not corrected; When the user scene recognition result is a single-person interaction scene, the parallel multi-user recognition result is corrected to correct the parallel multi-user recognition results of multiple users to a single user.

7. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 6, characterized in that: In the step S6, the parallel multi-user recognition result correction ratio A is calculated according to the number of user scene recognition times L0 and the number of parallel multi-user recognition result correction times L1, and A=L1 / L0 is set. The parallel multi-user recognition result correction ratio A is compared with the preset parallel multi-user recognition result correction ratio A0, and the visual fine-tuning situation is judged according to the comparison result, wherein: When A≤A0, it is determined that the visual fine-tuning situation does not require fine-tuning; When A>A0, it is determined that the visual fine-tuning situation requires fine-tuning, and the model fine-tuning is performed on the parallel multi-user recognition.

8. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 1, characterized in that: In the step S7, the language interaction information, the visual perception information and the environmental perception information are time synchronized, format unified and normalized to obtain preprocessed language interaction information, preprocessed visual perception information and preprocessed environmental perception information, feature extraction is performed on the preprocessed language interaction information, the preprocessed visual perception information and the preprocessed environmental perception information to obtain language interaction information feature extraction data, visual perception information feature extraction data and environmental perception information feature extraction data, feature layer fusion is performed on the language interaction information feature extraction data, the visual perception information feature extraction data and the environmental perception information feature extraction data according to a multimodal convolutional neural network to obtain feature layer fusion data, which is used as language interaction fusion data.

9. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 8, characterized in that: In the step S8, the language interaction fusion data is subjected to coarse-grained semantic stratification according to a coarse-grained semantic stratification method to obtain intent classification information and context perception information. The coarse-grained semantic stratification method includes: Step S100, constructing a neural network model including an LSTM layer and a fully connected layer; Step S200, setting the input of the neural network model including the LSTM layer and the fully connected layer to the language interaction fusion data obtained through the previous information fusion step, and setting the output of the neural network model including the LSTM layer and the fully connected layer to two parts, one part for intent classification, and the other part for context perception; Step S300, collecting language interaction fusion data samples; Step S400, dividing the language interaction fusion data samples into a training set, a validation set and a test set; Step S500, setting the loss function of the neural network model including the LSTM layer and the fully connected layer; Step S600, updating the model parameters of the neural network model including the LSTM layer and the fully connected layer; Step S700, outputting the neural network model including the LSTM layer and the fully connected layer; Step S800: input the language interaction fusion data into the neural network model including the LSTM layer and the fully connected layer, and obtain the intent classification information and context perception information output by the neural network model including the LSTM layer and the fully connected layer.

10. The multi-layer natural language interaction method based on deep learning and pattern recognition according to claim 9, characterized in that: In the step S9, after the coarse-grained semantic stratification, the language interaction fusion data is subjected to fine-grained semantic understanding according to a fine-grained semantic understanding method to obtain sentiment analysis information, tone analysis information, deep context analysis information and ambiguity resolution information. The fine-grained semantic understanding method includes: Step S1000, constructing a sentiment dictionary, and performing fine-grained semantic understanding on the language interaction fusion data according to the sentiment dictionary to obtain sentiment analysis information; Step S2000, performing fine-grained semantic understanding on the language interaction fusion data through feature engineering and machine learning models to obtain tone analysis information; Step S3000, performing fine-grained semantic understanding on the language interaction fusion data by combining an attention mechanism with a deep learning model to obtain deep context analysis information; Step S4000: Perform fine-grained semantic understanding on the language interaction fusion data through a polysemy representation learning model to obtain ambiguous resolution information.

Images