Human-computer interaction speech recognition system and method based on deep learning

Abstract

The invention discloses a human-computer interaction speech recognition system and method based on deep learning, relates to the technical field of human-computer interaction speech recognition, and remarkably improves the instruction recognition robustness in an industrial multi-person cooperation scene through a multi-modal signal cooperation and dynamic decision mechanism. Equipment exclusive voiceprint mapping is realized with an extremely low sample size by adopting a meta-learning adaptation layer, and the limitation that a traditional voiceprint model depends on large-scale data is broken through; a dynamic threshold value is set, noise spectrum analysis and an equipment state are combined, and misjudgment caused by burst interference is suppressed; by adopting the limited instruction syntax tree and the priority adjustment of the sensor drive, the emergency instruction is executed preferentially, and the security risk is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction voice recognition, and particularly to a human-computer interaction voice recognition system and method based on deep learning. Background Art

[0002] With the popularization of Industry 4.0 and intelligent logistics, voice interaction systems have become key control interfaces in multi-person collaborative operation scenarios.

[0003] Such scenarios require the system to respond to the voice commands of multiple operators in real time and accurately distinguish the source and priority of the commands. Existing solutions mostly adopt multi-microphone array positioning and voiceprint recognition technologies, combined with a semantic understanding module to achieve intention parsing. In the actual application of the current system, when the voices of multiple operators are similar or they speak simultaneously, the existing voiceprint models cannot distinguish due to the too high similarity of the embedded vectors. At the same time, the sound source positioning based on the microphone array has errors in complex reflection environments.

[0004] In recent years, some solutions have tried to aggregate multi-scenario voiceprint data through federated learning to improve the discrimination of the model for similar voices, but a large amount of user data is required and the real-time concurrency problem cannot be solved. There are also some solutions that fuse UWB positioning and the time stamps of voice signals, but they rely on high-precision hardware deployment and the cost is relatively high. Although such solutions partially alleviate the problems, it is difficult to balance real-time performance, cost, and robustness, and there are still significant bottlenecks in industrial scenarios with electromagnetic interference and severe reverberation. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a human-computer interaction voice recognition system and method based on deep learning to solve the problems of voiceprint confusion, positioning error, and instruction conflict faced by existing solutions in multi-person collaborative scenarios, and the improvement direction is limited by the problems of data dependence, hardware cost, and response delay.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a human-computer interaction voice recognition system based on deep learning, which includes, A multi-modal acquisition module for real-time acquisition of voice signals and associated infrared human detection signals; A voiceprint processing module that generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a device-specific adaptation layer that can be dynamically loaded; A dynamic decision-making module that generates an instruction attribution determination result according to the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human detection signal, and a predefined device operation permission list; The execution module responds to the determination result and outputs a control instruction to the target device.

[0008] As a preferred solution of the human-computer interaction voice recognition system based on deep learning according to the present invention, wherein: the voiceprint processing module includes: A pre-trained voiceprint encoder for extracting general voiceprint features from voice signals; A device-specific adaptation layer that generates adaptation parameters through a meta-learning framework and maps the general voiceprint features to the feature space of the current device operator.

[0009] As a preferred solution of the human-computer interaction voice recognition system based on deep learning according to the present invention, wherein: the meta-learning framework adopts the Model-Agnostic Meta-Learning (MAML) algorithm to generate adaptation parameters through the device's historical registered voices.

[0010] As a preferred solution of the human-computer interaction voice recognition system based on deep learning according to the present invention, wherein: in the device-specific adaptation layer, during the registration stage of a new device operator, general voiceprint features are extracted from at most 5 registered voices to form a support set, and the parameters of the meta-learning framework are initialized with this, and subsequently, device-specific adaptation parameters are generated through one gradient update with momentum. The specific process includes: Construct a support set: , wherein, represents the support set, represents the general voiceprint feature vector extracted from the th registered voice, represents the identity label corresponding to the th registered voice, represents the sample index, represents the total number of support set samples; Define the average classification loss of the support set as: , wherein, represents the average classification loss on the support set, represents the initial parameters of the meta-model, represents the classification loss function, represents the voiceprint adaptation mapping function with as the parameter, represents the general voiceprint feature vector of the th voice, represents the identity label corresponding to the th voice; Define the cross-entropy loss as: , wherein, represents the model prediction probability vector, represents the label one-hot encoding vector, represents the class index, represents the total number of classes, represents the dimensional label value, represents the dimensional prediction probability; The first-order gradient update generates the adaptation parameter : , where represents the updated device-specific adaptation layer parameter, represents the learning rate, represents taking the gradient of ; Introduce the momentum coefficient for secondary update, and the update formula is: , where represents the momentum gradient accumulation, represents the momentum coefficient, represents the learning rate.

[0011] As a preferred solution of the human-computer interaction speech recognition system based on deep learning according to the present invention, wherein: the conditions for the dynamic decision module to execute instruction attribution determination include: When multiple voice commands are detected, calculate the matching degree between the voiceprint confidence of each command and the infrared human body space coordinates, and generate a comprehensive confidence score; Determine that the command is valid only when the comprehensive confidence score exceeds the dynamic threshold; The dynamic threshold is automatically adjusted according to the environmental noise level and the device working mode; The adjustment rule of the dynamic threshold is: when the device is in the high-speed movement mode, the threshold is reduced; when the proportion of mechanical impact noise in the environmental noise spectrum exceeds the preset noise value, the threshold is increased.

[0012] As a preferred solution of the human-computer interaction speech recognition system based on deep learning according to the present invention, wherein: the generation method of the comprehensive confidence score is: Assign a first weight to the voiceprint confidence of the voice command, and assign a second weight to the spatial deviation between the infrared human body detection signal and the sound source localization coordinates; When the environmental noise intensity exceeds the preset level, reduce the first weight and increase the second weight.

[0013] As a preferred solution of the human-computer interaction voice recognition system based on deep learning according to the present invention, wherein: in the dynamic decision module, a dynamic threshold is defined , and the automatic adjustment is completed through the linear combination of the environmental noise level and the device working mode and the weighted correction of the mechanical impact noise. The adjustment method is as follows: Based on the environmental noise level , high-speed movement mode indication and the proportion of mechanical impact noise , a threshold calculation formula is constructed: , wherein, represents the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, represents the environmental noise level, represents the reference noise level, represents the mode adjustment coefficient, represents the high-speed movement mode indication, with a value of 1 or 0, represents the impact noise amplification coefficient, represents the proportion of mechanical impact noise, represents the preset mechanical impact noise threshold; The calculation formulas for the environmental noise level and the proportion of impact noise are: , wherein, represents the noise power spectral density, represents the power spectral density of the voice signal, represents the integration frequency range, , wherein, represents the lower and upper limits of the mechanical impact noise frequency band.

[0014] Secondly, the present invention provides a human-computer interaction voice recognition method based on deep learning, including, Step S1, in a multi-person collaborative operation scenario, synchronously collect voice signals and infrared human body distribution data in the target area; Step S2, extract the voiceprint features in the voice signal, and generate an identity confidence vector in combination with the device-specific adaptation layer; Step S3, generate an environmental consistency score according to the spatial deviation between the infrared human body distribution data and the sound source localization result; Step S4, based on the identity confidence vector, the environmental consistency score and the preset permission policy, screen the valid instructions and send them to the execution end.

[0015] As a preferred solution of the human-computer interaction voice recognition method based on deep learning according to the present invention, wherein: the step of screening valid instructions includes: When multiple conflicting instructions are detected, a finite instruction syntax tree is called to perform structured parsing on the instruction content; Give priority to executing emergency instructions that have a logical conflict with the current device state, and the determination condition of the emergency instructions is that the instruction content contains a preset keyword set; The preset keyword set includes safety keywords, device status keywords, and action termination keywords, where the safety keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual.

[0016] As a preferred solution of the human-computer interaction voice recognition method based on deep learning according to the present invention, wherein: the construction method of the finite instruction syntax tree is: Generate a basic instruction set according to the operation manual of the target device, including three types: action instructions, parameter instructions, and safety instructions; Set a dynamic priority weight for each type of instruction, and the weight is updated in real time according to the device running time or external sensor data; The external sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data.

[0017] The beneficial effects of the present invention are as follows: Through the multi-modal signal collaboration and dynamic decision-making mechanism, the present invention significantly improves the robustness of instruction recognition in the industrial multi-person collaboration scenario. The meta-learning adaptation layer is used to achieve device-specific voiceprint mapping with a very low sample size, breaking through the limitation of the traditional voiceprint model relying on large-scale data; setting a dynamic threshold combined with noise spectrum analysis and device status to suppress misjudgments caused by sudden interference; the finite instruction syntax tree and sensor-driven priority adjustment adopted make emergency instructions be executed first, reducing safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0019] Figure 1 It is a framework schematic diagram of a human-computer interaction voice recognition system based on deep learning in Embodiment 1.

[0020] Figure 2 It is a flowchart schematic diagram of a human-computer interaction voice recognition method based on deep learning in Embodiment 1. DETAILED DESCRIPTION OF THE INVENTION

[0021] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the spirit of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0023] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation manner of the present invention. The phrase "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an individual or alternative embodiment that is mutually exclusive with other embodiments.

[0024] Example 1, referring to Figure 1 and Figure 2 , this example provides a human-computer interaction voice recognition system based on deep learning, including: A multi-modal acquisition module for real-time acquisition of voice signals and associated infrared human detection signals; A voiceprint processing module that generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a dynamically loadable device-specific adaptation layer; The voiceprint processing module includes: A pre-trained voiceprint encoder for extracting general voiceprint features from voice signals; A device-specific adaptation layer that generates adaptation parameters through a meta-learning framework and maps the general voiceprint features to the feature space of the current device operator; Among them, the adaptation parameters are dynamically updated according to the operator's registered voice, and the number of registered voices does not exceed 5; The meta-learning framework adopts the model-agnostic meta-learning MAML algorithm to generate adaptation parameters through the device's historical registered voices, enabling the adaptation layer to converge quickly based on gradient updates when receiving new operator voices; In the device-specific adaptation layer, during the registration stage of a new device operator, general voiceprint features are extracted from at most 5 registered voices to form a support set, and the parameters of the meta-learning framework are initialized with this. Subsequently, device-specific adaptation parameters are generated through one gradient update with momentum. The specific process includes: Constructing the support set: , Among them, represents the support set, represents the The general voiceprint feature vector for registered voice extraction, represents the identity label corresponding to the th registered voice, represents the sample index, represents the total number of samples in the support set; Define the average classification loss on the support set as: , where, represents the average classification loss on the support set, represents the initialization parameters of the meta-model, represents the classification loss function, represents the voiceprint adaptation mapping function with as the parameter, represents the general voiceprint feature vector of the th voice, represents the identity label corresponding to the th voice; Define the cross-entropy loss as: , where, represents the model prediction probability vector, represents the label one-hot encoding vector, represents the class index, represents the total number of classes, represents the th dimensional label value, represents the th dimensional prediction probability; The first-order gradient is updated to generate the adaptation parameter : , where, represents the updated device-specific adaptation layer parameter, represents the learning rate, represents taking the gradient of ; Introduce the momentum coefficient for secondary update, and the update formula is: , where, represents the momentum gradient accumulation, represents the momentum coefficient, represents the learning rate; Specifically, the MAML framework is utilized to rapidly generate device-specific adaptation parameters from a small number of registered voice samples: a support set is constructed from at most five registered voices to ensure sample diversity, and then the cross-entropy loss is used to measure the voiceprint adaptation effect. The initial parameters of the general model are mapped to device-specific parameters through first-order gradient updates. Finally, a momentum coefficient is introduced to smooth the gradient updates, accelerating convergence and suppressing noise interference. This solution can complete adaptation with extremely few samples in noisy environments and diverse voice contents, significantly improving the identity recognition accuracy and having the ability of online incremental updates; The dynamic decision-making module generates an instruction attribution determination result based on the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human body detection signal, and a predefined list of device operation permissions; The conditions for the dynamic decision-making module to execute instruction attribution determination include: When multiple voice commands are detected, calculate the matching degree between the voiceprint confidence of each command and the spatial coordinates of the infrared human body to generate a comprehensive confidence score; Determine that the command is valid only when the comprehensive confidence score exceeds the dynamic threshold; The dynamic threshold is automatically adjusted according to the environmental noise level and the device working mode; The adjustment rule of the dynamic threshold is: when the device is in the high-speed movement mode, the threshold is lowered; when the proportion of mechanical impact noise in the environmental noise spectrum exceeds the preset noise value, the threshold is raised; The generation method of the comprehensive confidence score is: Assign a first weight to the voiceprint confidence of the voice command and a second weight to the spatial deviation between the infrared human body detection signal and the sound source localization coordinates; When the environmental noise intensity exceeds the preset level, lower the first weight and raise the second weight; In the dynamic decision-making module, a dynamic threshold is defined and is automatically adjusted through a linear combination of the environmental noise level and the device working mode and weighted correction of mechanical impact noise. The adjustment method is: Based on the environmental noise level , high-speed movement mode indication and the proportion of mechanical impact noise , construct a threshold calculation formula: , where represents the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, represents the environmental noise level, represents the reference noise level, is the mode adjustment coefficient, Indicates the high-speed movement mode indicator, with values of 1 or 0, Indicates the impact noise amplification coefficient, Indicates the proportion of mechanical impact noise, Indicates the preset mechanical impact noise threshold; The calculation formula for the environmental noise level and the proportion of impact noise is: , Among them, Indicates the noise power spectral density, Indicates the power spectral density of the voice signal, Indicates the integration frequency range, , Among them, Indicates the lower and upper limits of the mechanical impact noise frequency band; Specifically, a linear correction of the environmental noise and working mode factors is added to the basic threshold, as well as a non-negative weighted term for the proportion of mechanical impact noise, to achieve adaptive adjustment of the threshold. The noise sensitivity coefficient and mode adjustment coefficient can be obtained through online or offline training according to the actual scenario and device characteristics, ensuring that the risks of misjudgment and missed judgment can be balanced in different environments. The definition of the environmental noise level accurately reflects the overall interference intensity through power spectral integration. The proportion of mechanical impact noise is used to identify sudden impact noise events and avoid misjudging them as instructions. This model has a small computational amount, is suitable for real-time operation of embedded devices, and supports online parameter update, which can effectively improve the robustness and accuracy of instruction recognition in multi-human collaborative scenarios; The execution module responds to the determination result and outputs a control instruction to the target device.

[0025] This embodiment also provides a human-computer interaction voice recognition method based on deep learning, including: Step S1, in a multi-person collaborative operation scenario, synchronously collect the voice signal and infrared human body distribution data in the target area; Step S2, extract the voiceprint features in the voice signal and generate an identity confidence vector in combination with the device-specific adaptation layer; Step S3, generate an environmental consistency score according to the spatial deviation between the infrared human body distribution data and the sound source localization result; Step S4, based on the identity confidence vector, environmental consistency score and preset permission policy, screen out valid instructions and send them to the execution end; The steps of screening out valid instructions include: When multiple conflicting instructions are detected, call the finite instruction syntax tree to perform structured parsing on the instruction content; Give priority to executing the emergency instructions that have a logical conflict with the current device state, and the determination condition for the emergency instructions is that the instruction content contains the preset keyword set; The preset keyword set includes safety keywords, device status keywords, and action termination keywords. Among them, the safety keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual; The construction method of the finite instruction syntax tree is as follows: Generate a basic instruction set according to the operation manual of the target device, including three types: action instructions, parameter instructions, and safety instructions; Set dynamic priority weights for each type of instruction, and the weights are updated in real time according to the device running time or external sensor data; The external sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data. When the acceleration exceeds the set threshold, the weight of the safety instruction class is increased to the highest level.

[0026] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A human-computer interaction speech recognition system based on deep learning, characterized in that, Including, A multi-modal acquisition module for real-time acquisition of voice signals and associated infrared human detection signals; A voiceprint processing module that generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a dynamically loadable device-specific adaptation layer; A dynamic decision-making module that generates an instruction attribution determination result based on the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human detection signal, and a predefined device operation permission list; An execution module that responds to the determination result and outputs a control instruction to a target device.

2. The human-computer interaction voice recognition system based on deep learning according to claim 1, characterized in that, The voiceprint processing module includes: A pre-trained voiceprint encoder for extracting general voiceprint features from voice signals; A device-specific adaptation layer that generates adaptation parameters through a meta-learning framework and maps the general voiceprint features to the current device operator feature space.

3. The human-computer interaction voice recognition system based on deep learning according to claim 2, characterized in that, The meta-learning framework adopts the Model-Agnostic Meta-Learning (MAML) algorithm to generate adaptation parameters through the device's historical registered voices.

4. The human-computer interaction voice recognition system based on deep learning according to claim 3, characterized in that, In the device-specific adaptation layer, during the registration stage of a new device operator, general voiceprint features are extracted from at most 5 registered voices to form a support set, and the parameters of the meta-learning framework are initialized with this support set. Subsequently, device-specific adaptation parameters are generated through one gradient update with momentum. The specific process includes: Constructing the support set: ， in, represents the support set, Indicates The universal voiceprint feature vector extracted from the registered speech, Indicates The identity tag corresponding to the registered voice, represents the sample index, represents the total number of support set samples; Defining the average classification loss of the support set as: ， Among them, represents the average classification loss on the support set, represents the initialization parameters of the meta-model, represents the classification loss function, represents taking as the parameter of the voiceprint adaptation mapping function, represents the general voiceprint feature vector of the th voice, represents the identity label corresponding to the th voice; Defining the cross-entropy loss as: ， Among them, represents the model prediction probability vector, represents the label one-hot encoding vector, represents the class index, represents the total number of classes, represents the -dimensional label value, represents the -dimensional prediction probability; Generate adaptation parameters by first-order gradient update : ， Among them, represents the updated device-specific adaptation layer parameters, represents the learning rate, represents taking the gradient of ; Introducing a momentum coefficient for secondary update, and the update formula is: ， Among them, represents momentum gradient accumulation, represents the momentum coefficient, represents the learning rate.

5. A human-computer interaction voice recognition system based on deep learning according to claim 1, characterized in that, The conditions for the dynamic decision-making module to execute instruction attribution determination include: When multiple voice commands are detected, calculate the matching degree between the voiceprint confidence of each command and the infrared human body spatial coordinates to generate a comprehensive confidence score; Determine that the command is valid only when the comprehensive confidence score exceeds the dynamic threshold; The dynamic threshold is automatically adjusted according to the environmental noise level and the device working mode; The adjustment rule of the dynamic threshold is: when the device is in the high-speed movement mode, the threshold is reduced; when the proportion of mechanical shock noise in the environmental noise spectrum exceeds the preset noise value, the threshold is increased.

6. The human-computer interaction voice recognition system based on deep learning according to claim 5, characterized in that The generation method of the comprehensive confidence score is: Assign a first weight to the voiceprint confidence of the voice command, and assign a second weight to the spatial deviation between the infrared human detection signal and the sound source localization coordinates; When the environmental noise intensity exceeds the preset level, reduce the first weight and increase the second weight.

7. The human-computer interaction voice recognition system based on deep learning according to claim 6, characterized in that, In the dynamic decision-making module, a dynamic threshold is defined , and the automatic adjustment is completed through the linear combination of the environmental noise level and the device working mode and the weighted correction of the mechanical impact noise. The adjustment method is as follows: Based on the environmental noise level , high-speed movement mode indication and the proportion of mechanical shock noise , construct a threshold calculation formula: ， Among them, represents the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, represents the environmental noise level, represents the reference noise level, represents the mode adjustment coefficient, represents the high-speed movement mode indication, with a value of 1 or 0, represents the impact noise amplification coefficient, represents the proportion of mechanical impact noise, represents the preset mechanical impact noise threshold; The calculation formulas for the environmental noise level and the proportion of impact noise are: ， Among them, represents the noise power spectral density, represents the speech signal power spectral density, represents the integration frequency range, ， Among them, represents the lower and upper limits of the mechanical shock noise frequency band.

8. A human-computer interaction voice recognition method based on deep learning, based on the human-computer interaction voice recognition system based on deep learning according to any one of claims 1 to 7, characterized in that, Including: Step S1, in a multi-person collaborative operation scenario, synchronously collect the voice signals and infrared human body distribution data in the target area; Step S2, extract the voiceprint features from the voice signals and generate an identity confidence vector in combination with the device-specific adaptation layer; Step S3, generate an environmental consistency score based on the spatial deviation between the infrared human body distribution data and the sound source localization result; Step S4, based on the identity confidence vector, the environmental consistency score, and a preset permission policy, filter out valid instructions and send them to the execution end.

9. The method for human-computer interaction voice recognition based on deep learning according to claim 8, characterized in that, The steps for filtering out valid instructions include: When multiple conflicting instructions are detected, call a finite instruction syntax tree to perform structured parsing on the instruction content; Give priority to executing emergency instructions that have a logical conflict with the current device state, and the determination condition for the emergency instructions is that the instruction content contains a preset keyword set; The preset keyword set includes safety - related keywords, device - status keywords, and action - termination keywords. Among them, the safety - related keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual.

10. A human-computer interaction voice recognition method based on deep learning according to claim 9, characterized in that, The construction method of the finite instruction syntax tree is as follows: Generate a basic instruction set according to the operation manual of the target device, including three types: action instructions, parameter instructions, and safety instructions; Set dynamic priority weights for each type of instruction, and the weights are updated in real - time according to the device running time or external sensor data; The external sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data.

Images