A human-computer interactive speech recognition system and method based on deep learning

Abstract

The present invention discloses a human-computer interaction speech recognition system and method based on deep learning, which relates to the technical field of human-computer interaction speech recognition. The present invention significantly improves the robustness of command recognition in industrial multi-person collaboration scenarios through multimodal signal collaboration and dynamic decision-making mechanisms, adopts a meta-learning adaptation layer to achieve device-specific voiceprint mapping with extremely low sample volume, breaking through the limitation of traditional voiceprint models relying on large-scale data; sets dynamic thresholds in combination with noise spectrum analysis and device status to suppress misjudgments caused by sudden interference; adopts a limited command syntax tree and sensor-driven priority adjustment to enable emergency commands to be executed first, reducing security risks.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction speech recognition technology, and in particular to a human-computer interaction speech recognition system and method based on deep learning. Background Art

[0002] With the popularization of Industry 4.0 and smart logistics, voice interaction systems have become key control interfaces in multi-person collaborative work scenarios.

[0003] Such scenarios require the system to respond to the voice commands of multiple operators in real time and accurately distinguish the source and priority of the commands. Existing solutions mostly use multi-microphone array positioning and voiceprint recognition technology, combined with a semantic understanding module to implement intent analysis. In actual applications of the current system, when multiple operators have similar voices or speak at the same time, the existing voiceprint model cannot distinguish them due to the high similarity of the embedded vectors. At the same time, the sound source positioning based on the microphone array has errors in complex reflective environments.

[0004] In recent years, some solutions have attempted to aggregate voiceprint data from multiple scenarios through federated learning to improve the model's ability to distinguish similar tones, but this requires a large amount of user data and cannot solve the real-time concurrency problem; some solutions integrate UWB positioning and voice signal timestamps, but rely on high-precision hardware deployment and are relatively costly; although such solutions partially alleviate the problem, it is difficult to balance real-time performance, cost, and robustness, and there are still significant bottlenecks in industrial scenarios with severe electromagnetic interference and reverberation. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a human-computer interactive speech recognition system and method based on deep learning to solve the problems of voiceprint confusion, positioning error and command conflict faced by existing solutions in multi-person collaborative scenarios. The improvement direction is limited by data dependence, hardware cost and response delay.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a human-computer interactive speech recognition system based on deep learning, which includes:

[0009] Multimodal acquisition module, used to acquire voice signals and associated infrared human detection signals in real time;

[0010] The voiceprint processing module generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a dynamically loadable device-specific adaptation layer;

[0011] A dynamic decision module generates a command attribution determination result based on the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human body detection signal, and a predefined device operation permission list;

[0012] The execution module responds to the determination result and outputs a control instruction to the target device.

[0013] As a preferred solution of the deep learning-based human-computer interactive speech recognition system described in the present invention, the voiceprint processing module includes:

[0014] Pre-trained voiceprint encoder for extracting general voiceprint features from speech signals;

[0015] The device-specific adaptation layer generates adaptation parameters through a meta-learning framework and maps the universal voiceprint features to the feature space of the current device operator.

[0016] As a preferred solution of the deep learning-based human-computer interactive speech recognition system described in the present invention, the meta-learning framework adopts the model-independent meta-learning MAML algorithm to generate adaptation parameters through the device's historical registered speech.

[0017] As a preferred solution of the deep learning-based human-computer interactive speech recognition system described in the present invention, in the device-specific adaptation layer, during the new device operator registration phase, universal voiceprint features are extracted from up to five registered voices to form a support set, which is then used to initialize the meta-learning framework parameters. Subsequently, the device-specific adaptation parameters are generated through a gradient update with momentum. The specific process includes:

[0018] Build the support set:

[0019] ,

[0020] in, represents the support set, Indicates the The general voiceprint feature vector extracted from the registered speech, Indicates the The identity tag corresponding to the registered voice, represents the sample index, represents the total number of support set samples;

[0021] The support set average classification loss is defined as:

[0022] ,

[0023] in, represents the average classification loss on the support set, Represents the metamodel initialization parameters, represents the classification loss function, Indicates is the voiceprint adaptation mapping function of the parameters, Indicates the The universal voiceprint feature vector of the speech, Indicates the The identity label corresponding to the voice;

[0024] Define the cross entropy loss as:

[0025] ,

[0026] in, represents the model prediction probability vector, represents the label one-hot encoded vector, Represents the category index, represents the total number of categories, Indicates the Dimension label value, Indicates the dimensional prediction probability;

[0027] First-order gradient update generates adaptation parameters :

[0028] ,

[0029] in, Indicates the updated device-specific adaptation layer parameters. represents the learning rate, Express Find the gradient;

[0030] The momentum coefficient is introduced for secondary update, and the update formula is:

[0031] ,

[0032] in, represents the momentum gradient accumulation, represents the momentum coefficient, Represents the learning rate.

[0033] As a preferred solution of the deep learning-based human-computer interactive speech recognition system described in the present invention, the conditions for the dynamic decision module to perform instruction attribution determination include:

[0034] When multiple voice commands are detected, the matching degree between the voiceprint confidence of each command and the infrared human body spatial coordinates is calculated to generate a comprehensive confidence score;

[0035] The instruction is judged to be valid only when the comprehensive confidence score exceeds the dynamic threshold;

[0036] The dynamic threshold is automatically adjusted according to the ambient noise level and the device operating mode;

[0037] The adjustment rule of the dynamic threshold is: when the device is in high-speed moving mode, the threshold is lowered; when the proportion of mechanical impact noise in the ambient noise spectrum exceeds a preset noise value, the threshold is increased.

[0038] As a preferred solution of the deep learning-based human-computer interactive speech recognition system described in the present invention, the comprehensive confidence score is generated as follows:

[0039] Assign a first weight to the voiceprint confidence of the voice command, and assign a second weight to the spatial deviation between the infrared human body detection signal and the sound source positioning coordinates;

[0040] When the ambient noise intensity exceeds a preset level, the first weight is reduced and the second weight is increased.

[0041] As a preferred solution of the human-computer interactive speech recognition system based on deep learning described in the present invention, wherein: in the dynamic decision module, a dynamic threshold is defined , automatic adjustment is completed through the linear combination of environmental noise level and equipment working mode and weighted correction of mechanical impact noise. The adjustment method is:

[0042] Based on ambient noise level , high-speed moving mode indication Ratio of mechanical impact noise to , construct the threshold calculation formula:

[0043] ,

[0044] in, Indicates the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, Indicates the ambient noise level, represents the reference noise level, represents the mode adjustment coefficient, Indicates high-speed movement mode indication, the value is 1 or 0, represents the impact noise amplification coefficient, Indicates the proportion of mechanical impact noise, Indicates the preset mechanical shock noise threshold;

[0045] The calculation formula for the ratio of ambient noise level to impact noise is:

[0046] ,

[0047] in, represents the noise power spectral density, represents the power spectral density of the speech signal, represents the integral frequency range,

[0048] ,

[0049] in, Indicates the lower and upper limits of the mechanical shock noise frequency band.

[0050] In a second aspect, the present invention provides a human-computer interactive speech recognition method based on deep learning, comprising:

[0051] Step S1, in a multi-person collaborative operation scenario, synchronously collect voice signals and infrared human body distribution data in the target area;

[0052] Step S2: extracting voiceprint features from the speech signal and generating an identity confidence vector based on the device-specific adaptation layer;

[0053] Step S3, generating an environment consistency score based on the spatial deviation between the infrared human body distribution data and the sound source localization result;

[0054] Step S4: Based on the identity confidence vector, environment consistency score and preset permission policy, the valid instructions are screened and sent to the execution end.

[0055] As a preferred solution of the deep learning-based human-computer interactive speech recognition method of the present invention, the step of screening valid instructions includes:

[0056] When multiple conflicting instructions are detected, the limited instruction syntax tree is called to perform structured parsing of the instruction content;

[0057] Prioritize the execution of emergency commands that are in logical conflict with the current device state, and the judgment condition for the emergency command is that the command content contains a preset keyword set;

[0058] The preset keyword set includes security keywords, device status keywords and action termination keywords, among which security keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual.

[0059] As a preferred solution of the human-computer interactive speech recognition method based on deep learning described in the present invention, the finite instruction syntax tree is constructed as follows:

[0060] Generate basic instruction sets based on the target device's operating manual, including action instructions, parameter instructions, and safety instructions;

[0061] Setting dynamic priority weights for each type of instruction, which are updated in real time based on device runtime or external sensor data;

[0062] The external sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data.

[0063] The beneficial effects of the present invention are as follows: the present invention significantly improves the robustness of command recognition in industrial multi-person collaboration scenarios through multimodal signal collaboration and dynamic decision-making mechanisms, adopts a meta-learning adaptation layer to achieve device-specific voiceprint mapping with extremely low sample volume, breaking through the limitation of traditional voiceprint models relying on large-scale data; sets dynamic thresholds in combination with noise spectrum analysis and device status to suppress misjudgments caused by sudden interference; adopts a finite command syntax tree and sensor-driven priority adjustment to enable emergency commands to be executed first, reducing security risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 This is a schematic diagram of the framework of a deep learning-based human-computer interactive speech recognition system in Example 1.

[0066] Figure 2 This is a flow chart of a human-computer interaction speech recognition method based on deep learning in Example 1. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0069] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0070] Example 1, with reference to Figure 1 and Figure 2, this embodiment provides a human-computer interactive speech recognition system based on deep learning, including:

[0071] Multimodal acquisition module, used to acquire voice signals and associated infrared human detection signals in real time;

[0072] The voiceprint processing module generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a dynamically loadable device-specific adaptation layer;

[0073] The voiceprint processing module includes:

[0074] Pre-trained voiceprint encoder for extracting general voiceprint features from speech signals;

[0075] The device-specific adaptation layer generates adaptation parameters through a meta-learning framework and maps universal voiceprint features to the feature space of the current device operator;

[0076] Among them, the adaptation parameters are dynamically updated according to the operator's registered voice, and the number of registered voices does not exceed 5;

[0077] The meta-learning framework uses the model-independent meta-learning (MAML) algorithm to generate adaptation parameters based on the device's historical registered speech. This allows the adaptation layer to converge quickly based on gradient updates when receiving new operator speech.

[0078] In the device-specific adaptation layer, during the registration phase for a new device operator, universal voiceprint features are extracted from up to five registered voices to form a support set, which is then used to initialize the parameters of the meta-learning framework. Subsequently, a gradient update with momentum is performed to generate device-specific adaptation parameters. The specific process includes:

[0079] Build the support set:

[0080] ,

[0081] in, represents the support set, Indicates the The general voiceprint feature vector extracted from the registered speech, Indicates the The identity tag corresponding to the registered voice, represents the sample index, represents the total number of support set samples;

[0082] The support set average classification loss is defined as:

[0083] ,

[0084] in, represents the average classification loss on the support set, Represents the metamodel initialization parameters, represents the classification loss function, Indicates is the voiceprint adaptation mapping function of the parameters, Indicates the The universal voiceprint feature vector of the speech, Indicates the The identity label corresponding to the voice;

[0085] Define the cross entropy loss as:

[0086] ,

[0087] in, represents the model prediction probability vector, represents the label one-hot encoded vector, Represents the category index, represents the total number of categories, Indicates the Dimension label value, Indicates the dimensional prediction probability;

[0088] First-order gradient update generates adaptation parameters :

[0089] ,

[0090] in, Indicates the updated device-specific adaptation layer parameters. represents the learning rate, Express Find the gradient;

[0091] The momentum coefficient is introduced for secondary update, and the update formula is:

[0092] ,

[0093] in, represents the momentum gradient accumulation, represents the momentum coefficient, represents the learning rate;

[0094] Specifically, the MAML framework is used to quickly generate device-specific adaptation parameters based on a small number of registered voice samples: a support set is constructed using up to five registered voices to ensure sample diversity. The voiceprint adaptation effect is then measured using cross-entropy loss, and first-order gradient updates are used to map the general model initialization parameters to device-specific parameters. Finally, a momentum coefficient is introduced to smooth the gradient update, accelerate convergence, and suppress noise interference. This solution can complete adaptation with very few samples in noisy environments and with diverse voice content, significantly improving identity recognition accuracy and providing online incremental update capabilities.

[0095] The dynamic decision module generates a command attribution judgment result based on the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human detection signal, and the predefined device operation permission list;

[0096] The conditions for the dynamic decision module to execute instruction attribution judgment include:

[0097] When multiple voice commands are detected, the matching degree between the voiceprint confidence of each command and the infrared human body spatial coordinates is calculated to generate a comprehensive confidence score;

[0098] The instruction is judged to be valid only when the comprehensive confidence score exceeds the dynamic threshold;

[0099] Dynamic thresholds automatically adjust based on ambient noise levels and device operating modes;

[0100] The dynamic threshold adjustment rules are as follows: when the device is in high-speed mobile mode, the threshold is lowered; when the proportion of mechanical impact noise in the ambient noise spectrum exceeds the preset noise value, the threshold is increased;

[0101] The composite confidence score is generated as follows:

[0102] Assign a first weight to the voiceprint confidence of the voice command, and assign a second weight to the spatial deviation between the infrared human body detection signal and the sound source positioning coordinates;

[0103] When the ambient noise intensity exceeds a preset level, the first weight is reduced and the second weight is increased;

[0104] In the dynamic decision module, define the dynamic threshold , automatic adjustment is completed through the linear combination of environmental noise level and equipment working mode and weighted correction of mechanical impact noise. The adjustment method is:

[0105] Based on ambient noise level , high-speed moving mode indication Ratio of mechanical impact noise to , construct the threshold calculation formula:

[0106] ,

[0107] in, Indicates the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, Indicates the ambient noise level, represents the reference noise level, Mode adjustment coefficient, Indicates high-speed movement mode indication, the value is 1 or 0, represents the impact noise amplification coefficient, Indicates the proportion of mechanical impact noise, Indicates the preset mechanical shock noise threshold;

[0108] The calculation formula for the ratio of ambient noise level to impact noise is:

[0109] ,

[0110] in, represents the noise power spectral density, represents the power spectral density of the speech signal, represents the integral frequency range,

[0111] ,

[0112] in, Indicates the lower and upper limits of the mechanical impact noise frequency band;

[0113] Specifically, a linear correction of environmental noise and working mode factors, as well as a non-negative weighted term for the proportion of mechanical impact noise, is added to the basic threshold to achieve adaptive adjustment of the threshold. The noise sensitivity coefficient and mode adjustment coefficient can be obtained through online or offline training according to the actual scenario and device characteristics, ensuring that the risks of misjudgment and missed judgment can be balanced in different environments. The definition of environmental noise level accurately reflects the overall interference intensity through power spectrum integration. The proportion of mechanical impact noise is used to identify sudden impact noise events to avoid misjudgment as commands. The model has a small amount of computation, is suitable for real-time operation of embedded devices, and supports online parameter updates, which can effectively improve the robustness and accuracy of command recognition in multi-human collaborative scenarios.

[0114] The execution module responds to the determination result and outputs the control instruction to the target device.

[0115] This embodiment also provides a human-computer interactive speech recognition method based on deep learning, including:

[0116] Step S1, in a multi-person collaborative operation scenario, synchronously collect voice signals and infrared human body distribution data in the target area;

[0117] Step S2: extracting voiceprint features from the speech signal and generating an identity confidence vector based on the device-specific adaptation layer;

[0118] Step S3, generating an environment consistency score based on the spatial deviation between the infrared human body distribution data and the sound source localization result;

[0119] Step S4: Based on the identity confidence vector, environment consistency score and preset permission policy, the valid instructions are screened and sent to the execution end;

[0120] The steps to filter valid instructions include:

[0121] When multiple conflicting instructions are detected, the limited instruction syntax tree is called to perform structured parsing of the instruction content;

[0122] Prioritize the execution of emergency commands that are logically conflicting with the current device state, and the judgment condition for emergency commands is that the command content contains a preset keyword set;

[0123] The preset keyword set includes security keywords, device status keywords, and action termination keywords. Security keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual.

[0124] The finite instruction syntax tree is constructed as follows:

[0125] Generate basic instruction sets based on the target device's operating manual, including action instructions, parameter instructions, and safety instructions;

[0126] Set dynamic priority weights for each type of instruction, and the weights are updated in real time with device runtime or external sensor data;

[0127] External sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data. When the acceleration exceeds the set threshold, the weight of the safety instruction category is increased to the highest level.

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A human-computer interactive speech recognition system based on deep learning, characterized in that: include, Multimodal acquisition module, used to acquire voice signals and associated infrared human detection signals in real time; The voiceprint processing module generates a speaker identity vector based on a pre-trained general voiceprint recognition model and a dynamically loadable device-specific adaptation layer; A dynamic decision module generates a command attribution determination result based on the real-time voiceprint confidence of the voice signal, the spatial consistency of the infrared human body detection signal, and a predefined device operation permission list; an execution module, responding to the determination result and outputting a control instruction to a target device; The voiceprint processing module includes: Pre-trained voiceprint encoder for extracting general voiceprint features from speech signals; The device-specific adaptation layer is generated through a meta-learning framework and is used to map universal voiceprint features to the device operator feature space; The spatial consistency of the infrared human body detection signal refers to the deviation matching degree between the sound source positioning coordinates and the infrared human body position coordinates; The meta-learning framework adopts the model-independent meta-learning MAML algorithm to generate adaptation parameters through the device's historical registered speech; In the device-specific adaptation layer, during the registration phase of a new device operator, universal voiceprint features are extracted from up to five registered voices to form a support set, which is used to initialize the meta-learning framework parameters. Subsequently, a gradient update with momentum is performed to generate device-specific adaptation parameters. The specific process includes: Build the support set: ， Among them, S represents the support set, Indicates the The general voiceprint feature vector extracted from the registered speech, Indicates the The identity tag corresponding to the registered voice, represents the sample index, represents the total number of support set samples; The support set average classification loss is defined as: ， in, represents the average classification loss on the support set, Represents the metamodel initialization parameters, represents the classification loss function, Indicates is the voiceprint adaptation mapping function of the parameters, Indicates the The universal voiceprint feature vector of the speech, Indicates the The identity label corresponding to the voice; Define the cross entropy loss as: ， in, represents the model prediction probability vector, represents the label one-hot encoded vector, Represents the category index, represents the total number of categories, Indicates the Dimension label value, Indicates the dimensional prediction probability; First-order gradient update generates adaptation parameters : ， in, Indicates the updated device-specific adaptation layer parameters. represents the learning rate, Express Find the gradient; The momentum coefficient is introduced for secondary update, and the update formula is: ， in, represents the momentum gradient accumulation, represents the momentum coefficient, Represents the learning rate.

2. The human-computer interactive speech recognition system based on deep learning according to claim 1, characterized in that: The conditions for the dynamic decision module to perform instruction attribution determination include: When multiple voice commands are detected, the matching degree between the voiceprint confidence of each command and the infrared human body spatial coordinates is calculated to generate a comprehensive confidence score; The instruction is judged to be valid only when the comprehensive confidence score exceeds the dynamic threshold; The dynamic threshold is automatically adjusted according to the ambient noise level and the device operating mode; The adjustment rule of the dynamic threshold is: when the device is in high-speed moving mode, the threshold is lowered; when the proportion of mechanical impact noise in the ambient noise spectrum exceeds a preset noise value, the threshold is increased.

3. The human-computer interactive speech recognition system based on deep learning according to claim 2, characterized in that: The comprehensive confidence score is generated as follows: Assign a first weight to the voiceprint confidence of the voice command, and assign a second weight to the spatial deviation between the infrared human body detection signal and the sound source positioning coordinates; When the ambient noise intensity exceeds a preset level, the first weight is reduced and the second weight is increased.

4. The human-computer interactive speech recognition system based on deep learning according to claim 3, characterized in that: In the dynamic decision module, a dynamic threshold is defined , automatic adjustment is completed through the linear combination of environmental noise level and equipment working mode and weighted correction of mechanical impact noise. The adjustment method is: Based on ambient noise level , high-speed moving mode indication Ratio to mechanical impact noise , construct the threshold calculation formula: ， in, Indicates the dynamic judgment threshold, represents the basic threshold, represents the noise sensitivity coefficient, Indicates the ambient noise level, represents the reference noise level, represents the mode adjustment coefficient, Indicates high-speed movement mode indication, the value is 1 or 0, represents the impact noise amplification coefficient, Indicates the proportion of mechanical impact noise, Indicates the preset mechanical shock noise threshold; The calculation formula for the ratio of ambient noise level to impact noise is: ， in, represents the noise power spectral density, represents the power spectral density of the speech signal, represents the integral frequency range, ， in, Indicates the lower and upper limits of the mechanical shock noise frequency band.

5. A human-computer interaction speech recognition method based on deep learning, based on a human-computer interaction speech recognition system based on deep learning according to any one of claims 1 to 4, characterized in that: include: Step S1, in a multi-person collaborative operation scenario, synchronously collect voice signals and infrared human body distribution data in the target area; Step S2: extracting voiceprint features from the speech signal and generating an identity confidence vector based on the device-specific adaptation layer; Step S3, generating an environment consistency score based on the spatial deviation between the infrared human body distribution data and the sound source localization result; Step S4: Based on the identity confidence vector, environment consistency score and preset permission policy, the valid instructions are screened and sent to the execution end.

6. The method for human-computer interactive speech recognition based on deep learning according to claim 5, characterized in that: The step of screening valid instructions includes: When multiple conflicting instructions are detected, the limited instruction syntax tree is called to perform structured parsing of the instruction content; Prioritize the execution of emergency commands that are logically conflicting with the current device state, and the judgment condition for the emergency command is that the command content contains a preset keyword set; The preset keyword set includes security keywords, device status keywords and action termination keywords, among which security keywords have the highest priority, and the keyword set is automatically parsed and generated through the device operation manual.

7. The method for human-computer interactive speech recognition based on deep learning according to claim 6, characterized in that: The limited instruction syntax tree is constructed as follows: Generate basic instruction sets based on the target device's operating manual, including action instructions, parameter instructions, and safety instructions; Setting dynamic priority weights for each type of instruction, which are updated in real time based on device runtime or external sensor data; The external sensor data includes device acceleration sensor data, pressure sensor data, and temperature sensor data.

Images