A method and system for recognizing and processing a robot voice interaction signal
By training a basic acoustic model and combining it with fine-tuning on low-resource data and parallel processing, the problems of low speech recognition rate and insufficient hardware resources in low-resource language scenarios are solved, achieving efficient and low-latency speech recognition and processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING QINGFEI TECH CO LTD
- Filing Date
- 2025-06-11
- Publication Date
- 2026-05-01
AI Technical Summary
Existing speech recognition models have low recognition rates in low-resource language scenarios, lack support for contextual relevance and dynamic reasoning of user intent, and deep learning models have high requirements for embedded robot hardware resources, resulting in increased response latency and energy consumption.
The basic acoustic model is trained based on high-resource speech data, the underlying parameters are frozen and the top-level language-specific parameters are fine-tuned using low-resource data, a lightweight end-to-end speech recognition model parallel processing architecture is adopted, and the model is adaptively optimized by combining historical human-computer interaction data.
It achieves accurate recognition of mixed Chinese and English input, dialects, and low-resource languages, reducing latency and improving real-time performance and computational efficiency.
Smart Images
Figure CN120673748B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital processing technology, and in particular to a method and system for recognizing and processing robot voice interaction signals. Background Technology
[0002] With the rapid development of artificial intelligence and robotics, voice interaction has become one of the core methods of human-computer interaction. Compared with traditional touch or command input, voice interaction has advantages such as naturalness, efficiency, and no physical contact required, and is widely used in service robots, smart homes, and industrial automation. However, traditional speech recognition models (such as Hidden Markov Models combined with Gaussian Mixture Models) have limited generalization ability for pronunciation habits, dialects, and accents, especially in low-resource language scenarios where the recognition rate drops significantly. At the same time, most systems rely on preset grammatical templates or semantic parsing frameworks with limited states, lacking support for contextual relevance and dynamic reasoning of user intent, making it difficult to handle complex dialogues or multi-turn interaction scenarios. On the other hand, the existing speech processing workflow suffers from a prominent contradiction between real-time performance and computational efficiency. While deep learning models (such as end-to-end speech recognition networks) improve recognition accuracy, their large number of parameters places high demands on the computing power and memory resources of embedded robot hardware platforms, leading to response latency or increased energy consumption. Therefore, designing a highly robust, low-latency, and resource-efficient speech signal recognition and processing method has become crucial for improving the intelligence level of robot interaction. Summary of the Invention
[0003] This invention provides a method for recognizing and processing robot voice interaction signals, including:
[0004] Step 1: Train a basic acoustic model based on high-resource speech data;
[0005] Step 2: Freeze the underlying parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model.
[0006] Step 3: Accelerate the processing of the multilingual speech signal recognition model;
[0007] Step 4: Decompose the voice signal processing task and parallelize the processing task using the robot's hardware resources.
[0008] Step 5: Adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data.
[0009] The method for recognizing and processing robot voice interaction signals as described above, which trains a basic acoustic model based on high-resource voice data, specifically consists of the following sub-steps:
[0010] The high-resource speech data is converted into numerical sequences as model inputs, and the corresponding phoneme sequences are used as model outputs to construct the first training dataset.
[0011] The basic acoustic model was pre-trained using the first training dataset to enhance its ability to extract common features across languages.
[0012] A joint training strategy is introduced to gradually transition the model to the phoneme classification task.
[0013] The method for recognizing and processing robot voice interaction signals as described above, wherein the underlying parameters of the basic acoustic model are frozen and the top-level language-specific parameters are fine-tuned using low-resource voice data, is specifically divided into the following sub-steps:
[0014] The number of freeze layers is adaptively selected based on the amount of low-resource data.
[0015] The low-resource speech data was processed into a second training dataset;
[0016] Fine-tuning of the unfrozen top-level model parameters was performed using the second training dataset.
[0017] The method for recognizing and processing robot voice interaction signals as described above, wherein the multilingual voice signal recognition model is adaptively optimized based on historical human-computer interaction data, specifically comprises the following sub-steps:
[0018] Incremental training datasets are generated by filtering samples with confidence levels below a threshold from historical human-computer interaction data.
[0019] The multilingual speech signal recognition model was incrementally trained using an incremental training dataset.
[0020] The present invention also provides a system for recognizing and processing robot voice interaction signals, including: an acoustic model preparation module, an acoustic model fine-tuning module, an acoustic model acceleration module, a signal processing task acceleration module, and an acoustic model enhancement module.
[0021] The acoustic model preparation module is used to train a basic acoustic model based on high-resource speech data;
[0022] The acoustic model fine-tuning module is used to freeze the underlying parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model.
[0023] The acoustic model acceleration module is used to accelerate the processing of multilingual speech signal recognition models.
[0024] The signal processing task acceleration module is used to split the speech signal processing task and parallelize the processing task using the robot's hardware resources.
[0025] The acoustic model enhancement module is used to adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data.
[0026] The beneficial effects achieved by this invention are as follows: it supports accurate recognition of mixed Chinese and English input, dialects and low-resource languages, and adopts a lightweight end-to-end speech recognition model and parallel processing architecture to achieve low latency and high real-time performance. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0028] Figure 1 This is a flowchart of a robot voice interaction signal recognition and processing method provided in Embodiment 1 of this application;
[0029] Figure 2 This is a schematic diagram of a robot voice interaction signal recognition and processing system provided in Embodiment 2 of this application. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] Example 1
[0032] like Figure 1 As shown, Embodiment 1 of this application provides a method for recognizing and processing robot voice interaction signals, including:
[0033] Step S10: Train a basic acoustic model based on high-resource speech data;
[0034] Prepare a multilingual speech database, which should include both high-resource speech data such as Chinese and English, and low-resource speech data such as various dialects and Tibetan and Mongolian. The text annotations of the speech data in the database must include phoneme-level alignment (e.g., generated using a forced alignment tool). Train a basic acoustic model based on the high-resource speech data, specifically through the following sub-steps:
[0035] Step S11: Convert the high-resource speech data into a numerical sequence as the model input, and use the corresponding phoneme sequence as the model output to construct the first training dataset;
[0036] The speech frames of the high-resource speech data are sequentially fed into the transformation function. In the process, the transformed numerical sequence A' is used as the model input, and the corresponding phoneme sequence is used as the model output. After processing, the first training dataset is obtained. The transformation function... Represented as: ,in Let i be the i-th frame of speech signal in a certain high-resource speech data A. It is a time-varying kurtosis parameter. It is the phase enhancement coefficient. and These are the i-th frame and the i-th frame in A, respectively. The fundamental frequency of the frame speech signal. It is a floor function, where n is the total number of speech frames in A.
[0037] Step S12: Pre-train the basic acoustic model using the first training dataset to enhance the model's ability to extract cross-linguistic common features;
[0038] The underlying acoustic model employs a Transformer encoder structure, first optimizing the contrastive loss. To enhance the model's ability to extract common features across languages, data samples from the first training dataset are input into the basic acoustic model in batches, and the model is fine-tuned based on the contrastive loss calculation results.
[0039] Comparative loss The calculation formula is: ,in It is the cosine similarity function. Given the phoneme sequence output by the basic acoustic model for the u-th input sample, This represents the actual phoneme sequence corresponding to the u-th input sample. This represents the phoneme sequence actually corresponding to the k-th input sample in the current training batch. This is an adjustable parameter, k takes values from 1 to... N is the number of samples in the current training batch. This represents the number of samples in the current training batch that correspond to the actual phoneme sequence of the u-th input sample.
[0040] Step S13: Introduce a joint training strategy to gradually transition the model to the phoneme classification task;
[0041] When the contrast loss curve no longer shows a significant downward trend, a joint training strategy is introduced to gradually transition the model to the phoneme classification task. At this point, further training of the model is needed based on the total loss calculation results. The total loss at the current stage... The calculation formula is: ,in For adjustable weights, It is a comparison of losses. The input samples are calculated using dynamic programming. To model output The sum of all possible alignment path probabilities, The model outputs the number of phonemes that are replaced by the phoneme sequence relative to the actual phoneme sequence. The model outputs a phoneme sequence representing the number of phonemes removed from the actual phoneme sequence. The model outputs a phoneme sequence with respect to the number of newly inserted phonemes in relation to the actual phonemes. This represents the total number of factors within the actual phoneme sequence.
[0042] Step S20: Freeze the underlying parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model;
[0043] Freeze the low-level parameters of the basic acoustic model and fine-tune only the high-level parameters. This preserves the general acoustic features learned from the low-level model while allowing the high-level model to adapt to the pronunciation characteristics of low-resource languages. This is specifically divided into the following sub-steps:
[0044] Step S21: Adaptively select the number of freeze layers based on the low resource data volume;
[0045] Use the formula: Adaptive selection of the number of layers that the current model should be frozen ,in The total number of layers in the basic acoustic model. For low-resource voice data volume, This refers to the amount of high-resource voice data.
[0046] Step S22: Process the low-resource speech data into a second training dataset;
[0047] Using the aforementioned conversion function The low-resource speech data is converted into a numerical sequence as the model input, and the corresponding phoneme sequence is used as the output. After processing, the second training dataset is obtained.
[0048] Step S23: Fine-tune the parameters of the unfrozen top-level model using the second training dataset;
[0049] Total loss during the top-level fine-tuning phase The calculation formula is expressed as: ,in It is the current model for the input sample The cross-entropy loss of the t-th data point, where t ranges from 1 to T, and T is the input sample value. The total number of data items contained , For adjustable weights, It is the KL divergence calculation function. This indicates that the basic acoustic model is for the input sample The output probability distribution, This indicates that the current model is effective for the input samples. The output probability distribution, , It is the j-th model parameter of the current model. yes The diagonal values of the Fisher information matrix It is the j-th model parameter of the basic acoustic model, where j takes values from 1 to m, and m is the total number of model parameters.
[0050] During the first 1000 steps of fine-tuning training, All values are set to 0, solely based on improving the model's generalization ability to low-resource speech data through cross-entropy loss. Subsequent fine-tuning will be adjusted according to actual conditions during training. Assigning values is used to avoid overfitting and catastrophic forgetting problems in the model. Fine-tuning stops when the total loss value no longer decreases or shows no obvious downward trend, resulting in a multilingual speech signal recognition model.
[0051] Step S30: Accelerate the processing of the multilingual speech signal recognition model;
[0052] To improve the real-time performance of the multilingual speech signal recognition model, it is necessary to accelerate the model, which involves the following sub-steps:
[0053] Step S31: Compress the multilingual speech signal recognition model;
[0054] Using AQW quantization, the model can be compressed several times with almost no impact on performance.
[0055] Step S32: Divide the input data into small segments and input each segment into the model for recognition;
[0056] Inputting segment by segment allows for simultaneous input and processing of voice interaction signals, saving time waiting for complete sentences and improving computational efficiency.
[0057] Step S40: Decompose the speech signal processing task and parallelize the processing task using the robot's hardware resources;
[0058] The accelerated multilingual speech signal recognition model is now compatible with the hardware resources of embedded robots. The next step is to parallelize the complete speech signal processing task to further improve the real-time performance of speech recognition and interaction. First, the speech signal processing task is broken down into multiple sub-tasks. Then, these sub-tasks are deployed to different computing chips for parallel processing. The specific pipeline parallel design is as follows:
[0059] Stage 1 (DSP): Signal acquisition → Framing → Noise suppression;
[0060] Stage 2 (CPU+FPGA): Signal conversion (CPU) + baseband extraction (FPGA);
[0061] Stage 3 (GPU): Context splicing → Phoneme recognition → Streaming decoding.
[0062] Double buffering is used to further optimize parallel speed:
[0063] Each stage maintains two buffers (A / B). When Stage 1 writes A, Stage 2 processes B, and Stage 3 reads the result from the previous cycle.
[0064] Step S50: Adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data;
[0065] Historical human-computer interaction data is crucial for identifying model shortcomings. Integrating this data into the training dataset for incremental model training enables adaptive optimization, allowing the model to grow through use. Specifically:
[0066] Step S51: Filter samples with confidence levels below a threshold from historical human-computer interaction data to generate an incremental training dataset;
[0067] Confidence is the model's prediction probability for a certain interactive speech recognition result. The threshold is a preset value of the system. After the selected samples are labeled with the correct factor sequences, they are organized into a dataset to obtain the incremental training dataset.
[0068] Step S52: Incrementally train the multilingual speech signal recognition model using the incremental training dataset;
[0069] Incremental training follows the same process as fine-tuning the base acoustic model, requiring the bottom layer to be frozen first. The model parameters of the first layer are then fine-tuned using an incremental training dataset to fine-tune the unfrozen parameters of the top layer. The loss function used is also... .
[0070] Example 2
[0071] like Figure 2 As shown, Embodiment 2 of this application provides a robot voice interaction signal recognition and processing system, including: an acoustic model preparation module 21, an acoustic model fine-tuning module 22, an acoustic model acceleration module 23, a signal processing task acceleration module 24, and an acoustic model enhancement module 25.
[0072] The acoustic model preparation module 21 is used to train a basic acoustic model based on high-resource speech data; specifically, it includes: a first training dataset construction submodule and a model pre-training submodule.
[0073] 1. The first training dataset construction submodule is used to convert high-resource speech data into numerical sequences as model inputs and the corresponding phoneme sequences as model outputs to construct the first training dataset.
[0074] The speech frames of the high-resource speech data are sequentially fed into the transformation function. In the process, the transformed numerical sequence A' is used as the model input, and the corresponding phoneme sequence is used as the model output. After processing, the first training dataset is obtained. The transformation function... Represented as: ,in Let i be the i-th frame of speech signal in a certain high-resource speech data A. It is a time-varying kurtosis parameter. It is the phase enhancement coefficient. and These are the i-th frame and the i-th frame in A, respectively. The fundamental frequency of the frame speech signal. It is a floor function, where n is the total number of speech frames in A.
[0075] 2. The model pre-training submodule is used to pre-train the basic acoustic model using the first training dataset, enhance the model's ability to extract cross-language common features, and introduce a joint training strategy during training to gradually transition the model to the phoneme classification task.
[0076] The underlying acoustic model employs a Transformer encoder structure, first optimizing the contrastive loss. To enhance the model's ability to extract common features across languages, data samples from the first training dataset are input into the basic acoustic model in batches, and the model is fine-tuned based on the contrastive loss calculation results.
[0077] Comparative loss The calculation formula is: ,in It is the cosine similarity function. Given the phoneme sequence output by the basic acoustic model for the u-th input sample, This represents the actual phoneme sequence corresponding to the u-th input sample. This represents the phoneme sequence actually corresponding to the k-th input sample in the current training batch. This is an adjustable parameter, k takes values from 1 to... N is the number of samples in the current training batch. This represents the number of samples in the current training batch that correspond to the actual phoneme sequence of the u-th input sample.
[0078] Step S13: Introduce a joint training strategy to gradually transition the model to the phoneme classification task;
[0079] When the contrast loss curve no longer shows a significant downward trend, a joint training strategy is introduced to gradually transition the model to the phoneme classification task. At this point, further training of the model is needed based on the total loss calculation results. The total loss at the current stage... The calculation formula is: ,in For adjustable weights, It is a comparison of losses. The input samples are calculated using dynamic programming. To model output The sum of all possible alignment path probabilities, The model outputs the number of phonemes that are replaced by the phoneme sequence relative to the actual phoneme sequence. The model outputs a phoneme sequence representing the number of phonemes removed from the actual phoneme sequence. The model outputs a phoneme sequence with respect to the number of newly inserted phonemes in relation to the actual phonemes. This represents the total number of factors within the actual phoneme sequence.
[0080] The acoustic model fine-tuning module 22 is used to freeze the low-level parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model; specifically, it includes: a freeze layer selection submodule, a second training dataset construction submodule, and a top-level parameter adjustment submodule.
[0081] 1. Freeze layer selection submodule, used to adaptively select the freeze layer based on low resource data volume;
[0082] Use the formula: Adaptive selection of the number of layers that the current model should be frozen ,in The total number of layers in the basic acoustic model. For low-resource voice data volume, This refers to the amount of high-resource voice data.
[0083] Step S22: Process the low-resource speech data into a second training dataset;
[0084] Using the aforementioned conversion function The low-resource speech data is converted into a numerical sequence as the model input, and the corresponding phoneme sequence is used as the output. After processing, the second training dataset is obtained.
[0085] Step S23: Fine-tune the parameters of the unfrozen top-level model using the second training dataset;
[0086] Total loss during the top-level fine-tuning phase The calculation formula is expressed as: ,in It is the current model for the input sample The cross-entropy loss of the t-th data point, where t ranges from 1 to T, and T is the input sample value. The total number of data items contained , For adjustable weights, It is the KL divergence calculation function. This indicates that the basic acoustic model is for the input sample The output probability distribution, This indicates that the current model is effective for the input samples. The output probability distribution, , It is the j-th model parameter of the current model. yes The diagonal values of the Fisher information matrix It is the j-th model parameter of the basic acoustic model, where j takes values from 1 to m, and m is the total number of model parameters.
[0087] During the first 1000 steps of fine-tuning training, All values are set to 0, solely based on improving the model's generalization ability to low-resource speech data through cross-entropy loss. Subsequent fine-tuning will be adjusted according to actual conditions during training. Assigning values is used to avoid overfitting and catastrophic forgetting problems in the model. Fine-tuning stops when the total loss value no longer decreases or shows no obvious downward trend, resulting in a multilingual speech signal recognition model.
[0088] The acoustic model acceleration module 23 is used to accelerate the processing of multilingual speech signal recognition models; specifically, it includes a model compression submodule and an input data segmentation submodule.
[0089] 1. Model compression submodule, used to compress multilingual speech signal recognition models;
[0090] Using AQW quantization, the model can be compressed several times with almost no impact on performance.
[0091] 2. Input data segmentation submodule, used to segment the input data into small segments, which are then input into the model for recognition segment by segment;
[0092] Inputting segment by segment allows for simultaneous input and processing of voice interaction signals, saving time waiting for complete sentences and improving computational efficiency.
[0093] The signal processing task acceleration module 24 is used to split the speech signal processing task and parallelize the processing task using the robot's hardware resources.
[0094] The acoustic model enhancement module 25 is used to adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data; specifically, it includes: an incremental training dataset generation submodule and an incremental training submodule.
[0095] 1. The incremental training dataset generation submodule is used to filter samples with confidence scores below a threshold from historical human-computer interaction data to generate incremental training datasets.
[0096] Confidence is the model's prediction probability for a certain interactive speech recognition result. The threshold is a preset value of the system. After the selected samples are labeled with the correct factor sequences, they are organized into a dataset to obtain the incremental training dataset.
[0097] 2. Incremental Training Submodule, used to incrementally train the multilingual speech signal recognition model using the incremental training dataset;
[0098] Incremental training follows the same process as fine-tuning the base acoustic model, requiring the bottom layer to be frozen first. The model parameters of the first layer are then fine-tuned using an incremental training dataset to fine-tune the unfrozen parameters of the top layer. The loss function used is also... .
[0099] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor;
[0100] The memory is used to store one or more program instructions;
[0101] A processor is used to run one or more program instructions to execute a method for recognizing and processing robot voice interaction signals.
[0102] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a method for recognizing and processing robot voice interaction signals.
[0103] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer performs the above-described method for recognizing and processing robot voice interaction signals.
[0104] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0105] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.
[0106] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
[0107] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.
[0108] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).
[0109] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.
[0110] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for recognizing and processing robot voice interaction signals, characterized in that, include: Step 1: Train a basic acoustic model based on high-resource speech data, which is divided into the following sub-steps: The high-resource speech data is converted into numerical sequences as model inputs, and the corresponding phoneme sequences are used as model outputs to construct the first training dataset. The basic acoustic model is pre-trained using the first training dataset to enhance its ability to extract cross-linguistic common features. The contrast loss is then optimized to further enhance the model's ability to extract cross-linguistic common features. Data samples from the first training dataset are then input into the basic acoustic model in batches, and the model is fine-tuned based on the contrast loss calculation results. A joint training strategy is introduced to gradually transition the model to the phoneme classification task. When the contrastive loss curve no longer shows a significant downward trend, the joint training strategy is introduced again to gradually transition the model to the phoneme classification task. At this point, further training of the model is needed based on the total loss calculation results. The calculation formula is: ,in For adjustable weights, It is a comparison of losses. The input samples are calculated using dynamic programming. To model output The sum of all possible alignment path probabilities, The model outputs the number of phonemes replaced by the phoneme sequence relative to the actual phoneme sequence. The model outputs a phoneme sequence representing the number of phonemes removed from the actual phoneme sequence. The model outputs a phoneme sequence with respect to the number of newly inserted phonemes in relation to the actual phonemes. This represents the total number of factors within the actual phoneme sequence. Step 2: Freeze the underlying parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model. This is specifically divided into the following sub-steps: The number of freeze layers is adaptively selected based on low resource data volume; using the formula: Adaptive selection of the number of layers that the current model should be frozen ,in The total number of layers in the basic acoustic model. For low-resource voice data volume, For high-resource voice data volume; The low-resource speech data was processed into a second training dataset; Use conversion functions Low-resource speech data is converted into numerical sequences as model input, and the corresponding phoneme sequences are used as output. After processing, a second training dataset is obtained. The conversion function... Represented as: ,in Let i be the i-th frame of speech signal in a certain high-resource speech data A. It is a time-varying kurtosis parameter. It is the phase enhancement coefficient. and These are the i-th frame and the i-th frame in A, respectively. The fundamental frequency of the frame speech signal. It is a floor function, where n is the total number of speech frames in A; Fine-tuning of the unfrozen top-level model parameters was performed using the second training dataset; Total loss during the top-level fine-tuning phase The calculation formula is expressed as: ,in It is the current model for the input sample The cross-entropy loss of the t-th data point, where t ranges from 1 to T, and T is the input sample value. The total number of data items contained For adjustable weights, It is the KL divergence calculation function. This indicates that the basic acoustic model is for the input sample The output probability distribution, This indicates that the current model is effective for the input samples. The output probability distribution, , It is the j-th model parameter of the current model. yes The diagonal values of the Fisher information matrix This is the j-th model parameter of the basic acoustic model, where j ranges from 1 to m, and m is the total number of model parameters; during the first 1000 steps of fine-tuning training, All values are set to 0, solely based on improving the model's generalization ability to low-resource speech data through cross-entropy loss. Subsequent fine-tuning will be adjusted according to actual conditions during training. Assigning values is used to avoid overfitting and catastrophic forgetting problems in the model; Step 3: Accelerate the processing of the multilingual speech signal recognition model; Step 4: Decompose the speech signal processing task and parallelize it using the robot's hardware resources. Specifically, the speech signal processing task is divided into three-stage pipelined parallel subtasks, which are deployed to different computing chips on the robot for parallel processing. Stage 1 is deployed on the DSP chip to perform signal acquisition, framing, and noise suppression tasks; Stage 2 is deployed on the CPU+FPGA to perform signal conversion tasks on the CPU and fundamental frequency extraction tasks on the FPGA; Stage 3 is deployed on the GPU to perform context splicing, phoneme recognition, and streaming decoding tasks. Each stage maintains two buffers, A and B. When Stage 1 writes to buffer A, Stage 2 processes buffer B, and Stage 3 reads the result from the previous cycle. The parallel speed is optimized through the dual buffers. Step 5: Adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data.
2. The method for recognizing and processing robot voice interaction signals according to claim 1, characterized in that, Accelerating the processing of multilingual speech signal recognition models involves the following sub-steps: Compression of multilingual speech signal recognition models; The input data is divided into small segments, which are then fed into the model for recognition one segment at a time.
3. The method for recognizing and processing robot voice interaction signals according to claim 1, characterized in that, The multilingual speech signal recognition model is adaptively optimized based on historical human-computer interaction data, specifically through the following sub-steps: Incremental training datasets are generated by filtering samples with confidence levels below a threshold from historical human-computer interaction data. The multilingual speech signal recognition model was incrementally trained using an incremental training dataset.
4. A system for recognizing and processing robot voice interaction signals, characterized in that, The method for recognizing and processing robot voice interaction signals as described in any one of claims 1-3 includes: an acoustic model preparation module, an acoustic model fine-tuning module, an acoustic model acceleration module, a signal processing task acceleration module, and an acoustic model enhancement module. The acoustic model preparation module is used to train a basic acoustic model based on high-resource speech data; The acoustic model fine-tuning module is used to freeze the underlying parameters of the basic acoustic model and fine-tune the top-level language-specific parameters using low-resource speech data to obtain a multilingual speech signal recognition model. The acoustic model acceleration module is used to accelerate the processing of multilingual speech signal recognition models. The signal processing task acceleration module is used to split the speech signal processing task and parallelize the processing task using the robot's hardware resources. The acoustic model enhancement module is used to adaptively optimize the multilingual speech signal recognition model based on historical human-computer interaction data.
5. The robot voice interaction signal recognition and processing system according to claim 4, characterized in that, The acoustic model preparation module specifically includes: a first training dataset construction submodule and a model pre-training submodule; The first training dataset construction submodule is used to convert high-resource speech data into numerical sequences as model inputs and the corresponding phoneme sequences as model outputs to construct the first training dataset. The model pre-training submodule is used to pre-train the basic acoustic model using the first training dataset, enhance the model's ability to extract cross-linguistic common features, and introduce a joint training strategy during training to enable the model to gradually transition to the phoneme classification task.
6. The robot voice interaction signal recognition and processing system according to claim 4, characterized in that, The acoustic model fine-tuning module specifically includes: a freeze layer selection submodule, a second training dataset construction submodule, and a top-level parameter adjustment submodule; The freeze layer selection submodule is used to adaptively select the freeze layer based on low resource data volume; The second training dataset construction submodule is used to process low-resource speech data into a second training dataset. The top-level parameter tuning submodule is used to fine-tune the parameters of the unfrozen top-level model using the second training dataset.
7. The robot voice interaction signal recognition and processing system according to claim 4, characterized in that, The acoustic model acceleration module specifically includes: a model compression submodule and an input data segmentation submodule; The model compression submodule is used to compress multilingual speech signal recognition models; The input data segmentation submodule is used to segment the input data into small segments, which are then fed into the model for recognition segment by segment.
8. The robot voice interaction signal recognition and processing system according to claim 4, characterized in that, The acoustic model enhancement module specifically includes: an incremental training dataset generation submodule and an incremental training submodule; The incremental training dataset generation submodule is used to filter samples with confidence levels below a threshold from historical human-computer interaction data to generate incremental training datasets. The incremental training submodule is used to incrementally train the multilingual speech signal recognition model using an incremental training dataset.
Citation Information
Patent Citations
Language model training method of native voice mode
CN118471202A
Multi-language speech recognition method and device, equipment and medium
CN119339714A