Speech data processing method and apparatus, and speech control system
By adopting a voice keyword detection model based on the ConvMixer architecture in smart devices, the consumption problem of the voice keyword detection module on power and storage space is solved, low-power and low-latency voice detection is achieved, and the use time of the device is improved and interactive experience.
Patent Information
- Application Number
- PCT/CN2024/088339
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
The consumption of power and storage space by the voice keyword detection module in intelligent devices affects the use time and operation speed of the device.
Using a speech keyword detection model based on the ConvMixer architecture, the speech feature vector is extracted from the programmable logic unit and converted into a multi-channel input feature map, and the in-memory computing chip is used to detect speech keywords to reduce the model Space usage of storage resources and consumption of computing resources.
It realizes low-power and low-latency voice keyword detection, reduces the standby power consumption of smart devices, extends the device usage time, and improves the real-time performance of human-computer interaction.
Smart Images

Figure CN2024088339_05122024_PF_FP_ABST
Abstract
Description
Voice data processing method and device, and voice control system Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for processing voice data, a training method for a voice keyword detection model, a voice control system, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the advancement of artificial intelligence (AI) algorithms and hardware technologies like AI chips, smart devices have become widely used in daily life. Examples include smart home voice control systems, smart speakers, and smart conference systems. Voice interaction is widely used in smart devices and is becoming increasingly mature. Traditionally, voice interaction involved waking up the device by clicking a button, such as the record button, before interacting with it.
[0003] To further enhance the human-computer interaction experience, speech keyword spotting technology has emerged. Speech wakeup or keyword spotting is a subfield of speech recognition. Speech recognition involves extracting useful information from a continuous speech signal and converting it into a computer-recognizable string, typically a binary character stream or text. Speech keyword spotting doesn't require the entire speech signal to be recognized; it simply determines whether the signal contains a specific keyword or keywords.
[0004] Currently, there are three main methods for voice keyword detection: template matching-based wake-up technology, hidden Markov model-based wake-up technology, and deep learning-based wake-up technology. Among them, the most widely used method is the deep learning-based speech recognition wake-up method.
[0005] Summary of the Invention
[0006] The present disclosure provides a speech data processing method and device, a training method for a speech keyword detection model, a speech control system, an electronic device, and a computer-readable storage medium.
[0007] According to a first aspect of the present disclosure, the present disclosure provides a method for processing speech data, the method comprising:
[0008] Obtaining voice data to be processed;
[0009] Inputting the speech data to be processed into a programmable logic unit to perform speech feature extraction on the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed;
[0010] and converting the speech feature vector into a multi-channel input feature map in the programmable logic unit;
[0011] Inputting the input feature graph into an in-memory computing chip, performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtaining a speech keyword detection result;
[0012] The voice keyword detection result is fed back to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
[0013] In some embodiments, extracting speech features from the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed includes:
[0014] A preset speech feature extraction algorithm is used to extract speech features from the speech data to be processed, and a speech feature vector corresponding to the speech data to be processed is obtained.
[0015] In some embodiments, the speech feature vector is a two-dimensional speech feature vector, and converting the speech feature vector into a multi-channel input feature map includes:
[0016] Expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector;
[0017] Copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors;
[0018] The multiple three-dimensional speech feature vectors are spliced according to the channel dimension to obtain the multi-channel input feature map.
[0019] In some embodiments, the speech keyword detection model is a model built based on a convolutional hybrid network ConvMixer architecture, and the speech keyword detection model includes a feature sampling layer, a convolutional hybrid network, and a classification output layer;
[0020] The method of performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip and obtaining a speech keyword detection result includes:
[0021] Downsampling the input feature map using the feature sampling layer to obtain a downsampled feature map;
[0022] Performing convolution mixing processing on the downsampled feature map using the convolution mixing network to obtain a speech recognition feature map;
[0023] The classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.
[0024] In some embodiments, the feature sampling layer includes: a feature block embedding layer, a first batch normalization layer, and a first activation function layer;
[0025] The feature block embedding layer is implemented based on a convolutional layer, and the activation function layer adopts a Swish activation function.
[0026] In some embodiments, the convolutional mixing network includes at least one convolutional mixing network module, and the convolutional mixing network module includes a spatial position convolutional mixing unit and a channel position convolutional mixing unit. The spatial position mixing processing result output by the spatial position convolutional mixing unit and the input of the spatial position convolutional mixing unit are input to the channel position convolutional mixing unit through a residual connection to perform channel position mixing processing;
[0027] The spatial position convolution hybrid unit comprises: a depth-wise separable convolution layer, a second batch normalization layer, and a second activation function layer connected in sequence;
[0028] The channel position convolution mixing unit includes: a point-by-point convolution layer, a third batch normalization layer and a third activation function layer connected in sequence;
[0029] Wherein, the second activation function layer and the third activation function layer both adopt the Swish activation function.
[0030] In some embodiments, the classification output layer includes: an average pooling layer and a fully connected layer;
[0031] Wherein, the fully connected layer is implemented by a convolutional layer.
[0032] According to a second aspect of the present disclosure, the present disclosure provides a method for training a speech keyword detection model, the training method comprising:
[0033] Acquire an initial training data set, wherein the initial training data set includes a plurality of initial training speech data and a keyword label corresponding to each initial training speech data;
[0034] Performing data expansion processing on at least a portion of the initial training voice data using at least one data expansion method to obtain at least one expanded training voice data and a keyword label corresponding to the expanded training voice data;
[0035] The speech keyword detection model is trained according to the initial training speech data and the expanded training speech data to obtain a trained speech keyword detection model.
[0036] In some embodiments, the at least one data expansion method includes: a data mixing method, a data noise addition method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.
[0037] In some embodiments, performing model training on the speech keyword detection model includes:
[0038] The speech keyword detection model is trained based on a quantitative perceptual training method.
[0039] According to a third aspect of the present disclosure, the present disclosure provides a voice data processing device, the voice data processing device comprising:
[0040] an acquiring unit, configured to acquire voice data to be processed;
[0041] a feature extraction unit configured to input the speech data to be processed into the programmable logic unit to perform speech feature extraction on the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed;
[0042] a feature conversion unit, configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit;
[0043] a detection unit configured to input the input feature graph into an in-memory computing chip, perform speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtain a speech keyword detection result;
[0044] The response unit is configured to feed back the voice keyword detection result to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
[0045] According to a fourth aspect of the present disclosure, the present disclosure provides a voice control system, the voice control system comprising:
[0046] A sound collection device is used to collect the voice data to be processed in the environment and send the voice data to be processed to the main control system;
[0047] A main control system, configured to receive the voice data to be processed collected by the sound collection device and send the voice data to be processed to the programmable logic unit;
[0048] A programmable logic unit is used to send the voice data to be processed to the programmable logic unit to perform voice feature extraction on the voice data to be processed to obtain a voice feature vector corresponding to the voice data to be processed, and convert the voice feature vector into a multi-channel input feature map;
[0049] An in-memory computing chip having a preset voice keyword detection model integrated thereon, configured to perform voice keyword detection on the input feature graph using the voice keyword detection model pre-configured on the in-memory computing chip, and obtain a voice keyword detection result;
[0050] The main control system is further configured to receive the voice keyword detection result of the in-memory computing chip and control the terminal device to perform a corresponding response operation based on the voice keyword detection result;
[0051] The terminal device is used to perform a response operation corresponding to the voice keyword detection result.
[0052] According to a fifth aspect of the present disclosure, the present disclosure provides an electronic device, the electronic device comprising:
[0053] at least one processor; and
[0054] a memory communicatively connected to the at least one processor; wherein,
[0055] The memory stores one or more computer programs that can be executed by the at least one processor. The one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned voice data processing method.
[0056] According to a sixth aspect of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned voice data processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:
[0058] FIG1 is a flow chart of a method for processing voice data according to an embodiment of the present disclosure.
[0059] FIG2 is a schematic diagram of a model architecture of a speech keyword detection model provided by an embodiment of the present disclosure.
[0060] FIG3 is a schematic structural diagram of a voice data processing device provided by an embodiment of the present disclosure.
[0061] FIG4 is a schematic structural diagram of a voice control system provided by an embodiment of the present disclosure.
[0062] FIG5 is a schematic diagram of an application scenario of a voice control system provided by an embodiment of the present disclosure.
[0063] FIG6 is a schematic diagram of another application scenario of a voice control system provided by an embodiment of the present disclosure.
[0064] FIG7 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0065] To enable those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0066] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0067] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0068] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.
[0069] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.
[0070] In related technologies, in order to achieve a good interactive experience between people and smart devices, smart devices monitor voices in real time. This creates a serious problem: the voice keyword detection module consumes resources such as power and storage space. Many smart devices are not directly connected to the power supply, but use batteries, such as smart watches and Bluetooth speakers. The power consumption of the voice keyword detection function directly affects the usage time and operating speed of the smart device.
[0071] FIG1 is a flow chart of a method for processing voice data provided by an embodiment of the present disclosure. As shown in FIG1 , the method for processing voice data includes steps S11 to S14 .
[0072] Step S10: Acquire the voice data to be processed.
[0073] Step S11: Input the speech data to be processed into a programmable logic unit to perform speech feature extraction on the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed.
[0074] The voice data to be processed is obtained and input into a programmable logic unit. The voice features of the voice data to be processed are extracted by the programmable logic unit. Specifically, a preset voice feature extraction algorithm is used to extract the voice features of the voice data to be processed, and a voice feature vector corresponding to the voice data to be processed is extracted.
[0075] Exemplarily, speech feature extraction is performed on the speech data to be processed based on Mel Frequency Cepstrum Coefficient (MFCC) to obtain a speech feature vector corresponding to the speech data to be processed, wherein the Mel Frequency Cepstrum Coefficient MFCC is a speech feature extraction algorithm.
[0076] In the embodiment of the present disclosure, the speech feature vector extracted based on the Mel-frequency cepstral coefficient MFCC is an Fbank feature, and the speech feature vector is a two-dimensional speech feature vector.
[0077] In the MFCC algorithm, the extraction of Fbanks features involves several steps, including pre-emphasis processing, framing processing, windowing processing, Fourier transform processing, power spectrum calculation processing, and filtering through Mel filter banks.
[0078] Specifically, pre-emphasis is performed on the voice data. Pre-emphasis is a signal processing method that compensates for the high-frequency components of the input signal at the transmitting end. The function of pre-emphasis is to increase the energy of the high-frequency signal in the voice data and improve the signal-to-noise ratio.
[0079] Furthermore, the voice data after pre-emphasis processing is framed to obtain multiple audio frames. Framing processing refers to dividing the voice data into voice signal segments of fixed size, and each segment of the voice signal is called a frame, and the general frame length range is 20 milliseconds to 40 milliseconds. When performing framing processing, an overlapping segmentation method can be adopted, and the ratio of the frame shift to the frame length ranges from 0 to 1 / 2, wherein the frame shift is the overlapping part of the previous frame and the next frame. By utilizing the short-term stability of the signal, a smooth transition between frames is achieved to maintain its continuity, and at the same time, the problem of information omission caused by the boundary of the time window can be avoided, thereby reducing the impact on the windowing processing step. In the embodiment of the present disclosure, the frame length is 25ms, the frame shift is 10ms, the sampling frequency of the voice data is 16000 / s, and the length of the voice data is 1s.
[0080] Furthermore, windowing is performed on each audio frame. If the beginning and end values of each audio frame in the speech data are the same, it can be assumed that the frame is periodic, and the length of the period is exactly equal to the frame length. The purpose of windowing is to achieve the goal of making the beginning and end values of the frame the same by adding different weights to each sampling point of the speech data, thereby eliminating signal discontinuities that may be caused by the beginning and end of each frame.
[0081] Furthermore, a Fourier transform is performed on each audio frame to obtain spectrum information corresponding to each audio frame. The Fourier transform is used to convert a time-domain signal into a frequency-domain signal. In some embodiments, the voice data is a sampled digital signal, which is a discrete value. Therefore, the Fourier transform can use a discrete Fourier transform (DFT).
[0082] Furthermore, a power spectrum is calculated based on the spectrum information corresponding to the audio frame. The power spectrum is the ratio of the square of the modulus of the spectrum information obtained by Fourier transform to the length.
[0083] Furthermore, the power spectrum of each audio frame is filtered using a Mel filter bank to obtain the spectral features of each audio frame. The Mel filter bank can be a triangular filter bank. The power spectrum is processed through a triangular filter to obtain a one-dimensional vector, which is then processed through a set of triangular filter banks to obtain a two-dimensional matrix. Furthermore, the spectral features of the two-dimensional matrix are logarithmically calculated to obtain Fbank features, where the logarithm calculation is performed using a base-10 logarithmic function.
[0084] For example, the length of the Fbank feature vector is 64, that is, after using MFCC to extract speech features from speech data, a 98×64 two-dimensional Fbank feature vector can be obtained. Expanding the Fbank feature vector into a three-dimensional vector can obtain a tensor with a shape of [1,98,64], which is similar to a feature image with a length of 98, a width of 64, and a channel number of 1.
[0085] Step S12: Convert the speech feature vector into a multi-channel input feature map in the programmable logic unit.
[0086] In the programmable logic unit, the two-dimensional speech feature vector is converted into a multi-channel input feature map by using dimensional expansion, feature replication, feature splicing and other methods.
[0087] In some embodiments, in step S12, converting the speech feature vector into a multi-channel input feature map may further include: expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector; copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors, for example, copying it into three copies, the three three-dimensional speech feature vectors being the first speech feature vector, the second speech feature vector and the third speech feature vector respectively; splicing the multiple three-dimensional speech feature vectors according to the channel dimension to obtain a multi-channel input feature map, where the channel dimension refers to the feature channel of the speech feature vector.
[0088] In some embodiments, the unsequeeze function can be used to expand the feature dimension of the speech feature vector. The unsequeeze function is an operation function used to add a dimension with a dimension value of 1 at a specified position in the data. For example, the speech feature vector is a two-dimensional Fbank feature vector of 98×64. Using the unsequeeze function to expand the speech feature vector into a three-dimensional feature vector, a tensor with a shape of [1,98,64] can be obtained, which is similar to a feature image with a length of 98, a width of 64, and a channel number of 1.
[0089] In some embodiments, a concat function can be used to concatenate multiple three-dimensional speech feature vectors according to the channel dimension to obtain a multi-channel input feature map, where the number of channels of the input feature map is 3. The concat function is an operation function that concatenates two or more tensors in a certain dimension.
[0090] Step S13: input the input feature graph into the in-memory computing chip, and use the pre-configured speech keyword detection model on the in-memory computing chip to perform speech keyword detection on the input feature graph to obtain a speech keyword detection result.
[0091] In the embodiment of the present disclosure, a speech keyword detection model is pre-configured on the in-memory computing chip. In step S13, the multi-channel input feature map is input into the in-memory computing chip, and speech keyword detection processing is performed using the pre-configured speech keyword detection model to obtain a speech keyword detection result.
[0092] Figure 2 is a schematic diagram of the model architecture of a speech keyword detection model provided in an embodiment of the present disclosure. In some embodiments, the speech keyword detection model is a model constructed based on the convolutional hybrid network ConvMixer architecture. As shown in Figure 2, the speech keyword detection model constructed based on the ConvMixer architecture includes an input layer, a feature sampling layer, a convolutional hybrid network ConvMixer and a classification output layer connected in sequence. The input layer is used to input the input feature map into the feature sampling layer.
[0093] Specifically, performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip and obtaining a speech keyword detection result may further include the following steps:
[0094] First, the feature sampling layer is used to downsample the input feature map to obtain a downsampled feature map.
[0095] As shown in Figure 2, the feature sampling layer includes: a feature block embedding layer (Patch Embedding), a first batch normalization layer (BatchNorm, BN) and a first activation function layer; wherein, the feature block embedding layer is implemented based on the convolution layer, and the activation function layer adopts the Swish activation function.
[0096] Among them, the feature block embedding layer (Patch Embedding) is used to downsample the input feature map, the first batch normalization layer, that is, the BN layer, is used to normalize the input features, and the first activation function layer is used to change the linear relationship of the data and perform delinearization processing on the data, that is, nonlinear transformation processing.
[0097] Assuming the feature block patch size is p, the embedding dimension is h, and the number of channels of the input feature map is Cin, the output Z0 of the Patch Embedding layer can be expressed as: z0 = σ(BN(Conv cin→h (X, stride=p, kernel_size=p)))
[0098] For example, in the above formula, p is 64, kernel_size=p is a 2D convolution of 5x5, followed by a Swish activation function σ and a batch normalization layer (BN layer). The output of the Patch Embedding layer, that is, the feature map used next, is a tensor that can be expressed as [Batch size, 64, 25, 13]. That is, the final input to the convolutional hybrid network ConvMixer is a tensor expressed as [Batch size, 64, 25, 13]. The downsampling process of the feature map is completed through the Patch Embedding layer, which reduces the resolution of the feature map, increases the receptive field, makes it easier to find deep spatial information, and reduces the computational complexity and parameter amount of the convolutional hybrid network ConvMixer, thereby reducing the space occupied by the model for storage resources and reducing the model's consumption of computing resources.
[0099] Then, the convolutional mixing network is used to perform convolution mixing processing on the downsampled feature map to obtain the speech recognition feature map.
[0100] As shown in FIG2 , the convolutional mixing network includes at least one convolutional mixing network module (ConvMixer-Block), the number of convolutional mixing network modules is L, and the L convolutional mixing network modules are connected in series in sequence. The value of L can be set according to actual needs. Each convolutional mixing network module includes a spatial position convolutional mixing unit and a channel position convolutional mixing unit. The spatial position mixing processing result output by the spatial position convolutional mixing unit and the input of the spatial position convolutional mixing unit are input to the channel position convolutional mixing unit through a residual connection to perform channel position mixing processing.
[0101] As shown in Figure 2, the spatial position convolution hybrid unit includes: a depth-wise separable convolution (ConvDepthwise) layer, a second batch normalization (BN) layer, and a second activation function layer connected in sequence; as shown in Figure 2, the channel position convolution hybrid unit includes: a point-by-point convolution (ConvPointwise) layer, a third batch normalization layer, and a third activation function layer connected in sequence; wherein the second activation function layer and the third activation function layer both use the Swish activation function.
[0102] In some embodiments, in the convolutional hybrid network module, a depth-separable convolution (ConvDepthwise) layer is selected to mix spatial position features, and a point-by-point convolution (ConvPointwise) layer (with a convolution kernel size of 1x1) is selected to mix channel position features, while effectively ensuring that the model has a small number of network parameters. The number of channels in the convolution layer is consistent with the embedding dimension h of the above-mentioned Patch Embedding layer. The convolutional hybrid network module (ConvMixer-Block) can be expressed by the following formula: z′ l =σ(BN(ConvDepthwise(z l-1 )))+z l-1 z l =σ(BN(ConvPointwise(z′ l )))
[0103] Where BN is the batch normalization layer and σ is the swish activation function.
[0104] Finally, the classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.
[0105] As shown in Figure 2, the classification output layer includes: an average pooling layer and a fully connected layer; the fully connected layer is implemented by a convolutional layer (Conv2d), and the average pooling layer is a 2d average pooling layer. In the fully connected layer, the convolutional layer uses a convolutional layer with a convolution kernel size of 1×1, and the number of output channels is equal to the preset number of keyword categories.
[0106] In the embodiment of the present disclosure, using the convolutional layer as the fully connected layer can significantly reduce the size of the model parameters and reduce the amount of model calculation, thereby helping to reduce the space occupied by the model on storage resources and reducing the model's consumption of computing resources.
[0107] Specifically, the speech recognition features output by the convolutional hybrid network are input into a 2D average pooling layer for pooling processing, and then keyword classification prediction processing is performed through a fully connected layer using a softmax activation function. An N+2 classification output result can be obtained, where N is the total number of preset voice wake-up words and voice command words, and 2 represents the silence category and the unknown category. That is, the categories of voice keywords include voice wake-up words, voice command words, silence, and unknown. Among them, voice wake-up words include but are not limited to strings of at least 4 words such as "Xiao X Xiao X" and "Hello Xiao X", and voice command words include but are not limited to a series of control commands such as "Turn on the air conditioner", "Increase the volume", and "Turn up the brightness". The output result can represent the probability of the voice data belonging to each keyword category. Based on the output result of the fully connected layer, the voice keyword detection result is obtained. For example, the category with the highest probability can be selected as the voice keyword detection result, or the category with a probability greater than a preset threshold can be selected as the voice keyword detection result.
[0108] In some embodiments, as shown in FIG2 , a dimension conversion (View) layer is further provided after the fully connected layer for performing dimension conversion on the output result of the fully connected layer to obtain an output result of a required dimension.
[0109] Step S14: Feedback the voice keyword detection result to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
[0110] For example, when the voice keyword detection result is a certain voice wake-up word, it can be used to control the intelligent voice control device to respond. For example, if the input voice data is "Xiao X Xiao X", the response of the intelligent voice control device can be "Xiao X is here", "Here", etc. When the voice keyword detection result is a certain voice command word, it can be used to control the corresponding smart terminal to execute the corresponding control command. For example, when the voice command word is "turn on the air conditioner", it can be used to control the air conditioner to turn on and execute the preset or last set operation. When the voice command word is "increase the volume", it controls the player currently in the playing state to increase the volume by one level. When the voice command word is "increase the brightness", it controls the lighting device currently in the lighting state to increase the brightness by one level.
[0111] A neural network model is composed of a series of connected network layers. Each layer receives the output of the previous layer and processes the result as the input to the next layer. Each layer of the neural network model has its own parameters. During the training process, changes in the parameters of a layer will cause the output of that layer, that is, the input distribution of the next layer, to change. As the changes accumulate layer by layer, these changes are amplified. With each iteration, as the parameters are updated, the distribution of the input features changes, and the parameters of the neural network model need to readjust to the input distribution. This greatly increases the training complexity of the neural network model. The batch normalization layer (BN) can effectively alleviate this problem. The batch normalization layer performs a similar standardization operation on the input features, which can stabilize the distribution of the input features and accelerate the training process of the neural network model.
[0112] In the embodiment of the present disclosure, in the feature sampling layer and convolutional hybrid network of the speech keyword detection model, since the batch normalization layer is located after the convolution layer and is connected in series with the convolution layer, the batch normalization layer can be fused with the convolution layer, which can help accelerate network calculations and reduce the number of parameters and calculations of the model, thereby reducing the space occupied by the model for storage resources and reducing the model's consumption of computing resources.
[0113] For example, assume that the convolution calculation method with bias is: y = x*w+b
[0114] The calculation formula of the BN layer is:
[0115] Substituting y into z yields:
[0116] Let w n =β·w / y var ,b n =β·(by mean ) / y var +γ, where x represents input, w represents weight, b, γ, and β are constants, mean represents mean, and var represents variance.
[0117] The new convolution parameters after the fusion of the convolution layer and the BN layer can be expressed as: z = x*w n +b n
[0118] In an embodiment of the present disclosure, a voice keyword detection model is integrated into an in-memory computing chip, and the in-memory computing chip is used to perform voice keyword detection.
[0119] In the disclosed embodiments, the in-memory computing chip can effectively reduce computing power consumption compared to traditional von Neumann architecture chips. Specifically, in von Neumann architecture chips, the computing unit is separated from the memory. To complete data calculations, data must be read from the memory and then written back to the memory after the calculation is completed. The data transmission between the computing unit and the memory generates huge power consumption. In contrast, the in-memory computing chip embeds the computing function in the memory, integrating storage and computing. During the operation of the voice keyword detection model, the data transmission between the computing unit and the memory is reduced. In addition, the memory uses analog circuits instead of traditional digital circuits to perform calculations, which reduces computing power consumption.
[0120] In the disclosed embodiment, the speech keyword detection model based on the ConvMixer architecture has good accuracy and maintains a small number of model parameters. It can be easily integrated into a low-power in-memory computing chip to achieve low-power, low-latency speech keyword detection.
[0121] According to the voice data processing method provided by the embodiment of the present disclosure, a lightweight voice keyword detection model is integrated with a low-power in-memory computing chip to perform voice keyword detection and recognition. Voice keyword detection can be achieved with low power consumption and low latency, and wake-up word and command word recognition tasks can be achieved simultaneously without the use of additional voice recognition models. It can be applied to run on edge devices and can also be applied to terminal devices to realize functions such as voice wake-up devices and voice control devices.
[0122] An embodiment of the present disclosure also provides a training method for a speech keyword detection model, which includes: obtaining an initial training data set, the initial training data set including multiple initial training speech data and keyword labels corresponding to each initial training speech data, the keyword labels corresponding to the initial training speech data are pre-labeled keyword categories to which the initial training speech data belongs, such as wake-up words or voice command words; for at least part of the initial training speech data, performing data expansion processing on the initial training speech data through at least one data expansion method to obtain at least one expanded training speech data and keyword labels corresponding to the expanded training speech data; performing model training on the speech keyword detection model based on the initial training speech data and the expanded training speech data to obtain a speech keyword detection model after training.
[0123] In some embodiments, at least one data expansion method includes: a data mixing method, a data noise adding method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.
[0124] Among them, the data mixing method is a data enhancement method based on Mixup. For supervised learning tasks, the data mixing Mixup method is an effective data enhancement strategy. This method can combine the initial training speech data with keyword labels, making the model more robust to adversarial samples and reducing the model's misjudgment of sample categories. The data mixing method is shown in the following formula: X mix =λX i +(1-λ)X j Y mix =λY i +(1-λ)Y j
[0125] Among them, X i represents the i-th initial training speech data, Y i represents the keyword label corresponding to the i-th initial training speech data, X j represents the jth initial training speech data, Y j represents the keyword label corresponding to the jth initial training speech data, and the mixing ratio λ conforms to the Beta distribution, that is, λ~Beta(α,α), where α=10 can be taken, X mix represents the expanded training speech data obtained by data mixing, mix Represents the expanded training speech data X mix The corresponding keyword tag.
[0126] Data noise addition refers to adding a specified type of noise data to the given initial training speech data by specifying the signal-to-noise ratio, such as music, TV drama background, reverberation, etc. Given the initial training speech data S, noise data N, and signal-to-noise ratio SNR, we have: SNR=10lg(Ps / Pn) Ps=avg(s 2 ),Pn=avg(n 2 ) Noise=S+scalar·N
[0127] Where Ps is the average kilometer of the initial training speech data S, Pn is the average power of the noise data N, and Noise represents the expanded training speech data obtained by data expansion processing by adding noise to the data. The keyword label of the expanded training speech data Noise is the same as the keyword label of the initial training speech data S.
[0128] Constructing negative samples: An important metric for voice wakeup / keyword detection is the false wakeup / false recognition rate. For example, if the wakeup word is "Hello, Xiaojing," if the model is overfitting or has poor robustness, the greeting "Hello" during normal conversation may also trigger voice wakeup. To avoid this, construct negative samples based on the initial training dataset.
[0129] The method for constructing negative samples is: based on the speech ASR (Automatic Speech Recognition) model to identify the initial training speech data, and output keyword tags with timestamp information, where ASR can adopt any existing suitable speech recognition model; based on the timestamp information, the original initial training speech data is segmented, for example, the original initial training speech data is "Hello Xiaojing", the original initial training speech data is segmented into two audio clips of "Hello" and "Xiaojing", and the two audio clips can be randomly spliced, so that four negative sample data can be obtained: "Hello", "Xiaojing", "Hello hello", and "Hello Xiaojing".
[0130] In addition, different initial training voice data can be spliced together to form new negative samples. For example, the initial training voice data "turn up the volume" and the initial training voice data "light up the screen" can be segmented into "turn up", "volume", "light up", and "screen". The negative samples obtained by random combination and splicing can be "turn up the screen", "light up the volume", etc.
[0131] Finally, the constructed negative samples are used as augmented training speech data, and their corresponding labels are set to unknown labels.
[0132] In addition, the training data can also be expanded using methods such as velocity perturbation and spectrum enhancement, which will not be described here.
[0133] Through the above data expansion method, the amount of training data can be increased, the diversity of training data can be improved, and the robustness of the model can be increased.
[0134] In the disclosed embodiment, before training the voice keyword detection model based on the initial training voice data and the expanded training voice data, voice feature extraction is performed on the voice data to obtain voice feature vectors corresponding to the voice data. The voice feature vectors are then converted into multi-channel input feature maps. The input feature maps are then input into the model for model training.
[0135] In some embodiments, performing model training on the speech keyword detection model includes: performing model training on the speech keyword detection model based on a Quantification Aware Training (QAT) method.
[0136] In the quantization-aware training method, the entire model can use integer INT8 quantization, which involves pseudo-quantization nodes. Pseudo-quantization nodes are nodes inserted during quantization-aware training to find the model's data distribution and provide feedback on accuracy loss. They have the following two specific functions:
[0137] (1) Find the distribution of network data, that is, find the maximum and minimum values of the parameters to be quantified.
[0138] (2) Simulate the accuracy loss when quantized to low bits, apply this loss to the model, pass it to the loss function, and let the optimizer optimize the loss during training.
[0139] In some embodiments, a speech keyword detection model can be pre-trained using the above-mentioned training method, and then the trained speech keyword detection model can be deployed to an in-memory computing chip for execution.
[0140] In addition, the following effectiveness verification was carried out for the speech data processing method proposed in the embodiment of the present disclosure: the speech data processing method proposed in the embodiment of the present disclosure was experimentally tested using the Google Speech Commands V2 (GSC-V2) dataset. GSC-V2 contains 105,829 command words and 2,618 speakers, including 35 command words such as 'down', 'go', 'left', 'no', 'off', 'on', 'right', 'stop', 'up', and 'yes'. Experiments were conducted using models of different sizes. For example, for the speech keyword detection model of the embodiment of the present disclosure, the model depth is 8, the hidden layer size is 64, which is called KwsConvMixer-S (KCM-S), the model depth is 12, the hidden layer size is 64, which is called KwsConvMixer-M (KCM-M), and the model depth is 12, the hidden layer size is 128, which is called ConvMixer-L (KCM-L). During the experiment, the adamw optimizer was used, the warmup epoch was 10, the number of iterations was 25,000, the learning rate was 0.02, and the batch size was 256. The test results are shown in Table 1.
[0141] Table 1
[0142] It can be seen that the KCM-S model achieves an accuracy of 96.24 with only 96k parameters, while ConvMixer-L achieves an accuracy of 97.9 with 0.26M parameters. This shows that the speech keyword detection model of the embodiment of the present disclosure has better prediction effect than the MLP-based model and the transformer-based model, and the model parameter count is smaller.
[0143] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.
[0144] Figure 3 is a structural diagram of a voice data processing device provided by an embodiment of the present disclosure. As shown in Figure 3, the voice data processing device 300 includes: a feature extraction unit 301, a feature conversion unit 302, a detection unit 303, a response unit 304 and an acquisition unit 305.
[0145] Among them, the acquisition unit 305 is configured to acquire the speech data to be processed; the feature extraction unit 301 is configured to input the speech data to be processed into the programmable logic unit to perform speech feature extraction on the speech data to be processed to obtain the speech feature vector corresponding to the speech data to be processed; the feature conversion unit 302 is configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit; the detection unit 303 is configured to input the input feature map into the in-memory computing chip, use the speech keyword detection model pre-configured on the in-memory computing chip to perform speech keyword detection on the input feature map, and obtain the speech keyword detection result; the response unit 304 is configured to feed back the speech keyword detection result to the main control system, and the main control system is used to execute a response operation corresponding to the speech keyword detection result based on the speech keyword detection result.
[0146] The speech data processing device 300 provided in the embodiment of the present disclosure is used to implement the speech data processing method provided in any of the above embodiments. For specific related descriptions, please refer to the description of the speech data processing method in any of the above embodiments, which will not be repeated here.
[0147] Figure 4 is a structural diagram of a voice control system provided in an embodiment of the present disclosure. As shown in Figure 4, the voice control system 400 includes: a sound collection device 401, a processing in memory chip (PIM chip) 402, a terminal device 403, a main control system (Processing System, PS) 404 and a programmable logic unit (PL) 405.
[0148] Among them, the sound collection device 401 is used to collect the unprocessed voice data in the environment and send the unprocessed voice data to the main control system; the main control system 404 is used to receive the unprocessed voice data collected by the sound collection device 401 and send the unprocessed voice data to the programmable logic unit 405; the programmable logic unit 405 is used to extract voice features from the unprocessed voice data to obtain a voice feature vector corresponding to the unprocessed voice data and convert the voice feature vector into a multi-channel input feature map; the in-memory computing chip 402 is integrated with a preset voice keyword detection model, which is used to use the pre-configured voice keyword detection model on the in-memory computing chip 402 to perform voice keyword detection on the input feature map, obtain voice keyword detection results, and feedback the voice keyword detection results to the main control system 404; the main control system 404 is also used to receive the voice keyword detection results of the in-memory computing chip and control the terminal device 403 to perform corresponding response operations according to the voice keyword detection results; the terminal device 403 is used to perform response operations corresponding to the voice keyword detection results; the main control system 404 and the programmable logic unit 405 can be integrated into an FPGA (Field Programmable Gate Array). The Field Programmable Gate Array (FPGA) device can exchange data with the sound collection device 401, the in-memory computing chip 402, and the terminal device 403.
[0149] In some embodiments, the main control system is a system on chip (SOC), and a Linux operating system can be deployed on the FPGA device as the main control system (PS) to realize the control and scheduling of hardware resources. Based on the operating system, some algorithms with strong versatility and low performance requirements can be implemented.
[0150] Figure 5 is a schematic diagram of an application scenario of a voice control system provided by an embodiment of the present disclosure. As shown in Figures 4 and 5, in some application scenarios, the main control system 404 and the programmable logic unit 405 are deployed in an FPGA device, and the in-memory computing chip 402 is implemented based on the flash memory FLASH architecture. The FPGA device and the in-memory computing chip 402 can interact with each other through a variety of data transmission methods, such as data transmission through interfaces such as UART, SPI, I2C, INT, and I2S. The FPGA device and the terminal device 403 can be connected through an HDMI interface.
[0151] Figure 6 is a schematic diagram of an application scenario of another voice control system provided by an embodiment of the present disclosure. As shown in Figure 6, in some application scenarios, the main control system 404 is deployed on an FPGA device, and a programmable logic unit (PL) 405 is also deployed on the FPGA device. The in-memory computing chip 402 is implemented based on the SRAM memory architecture. The main control system 404 and the in-memory computing chip 402 exchange data through the programmable logic unit 405. The main control system 404 and the terminal device 403 can be connected through an HDMI or DP interface. In addition, the voice control system also includes an external memory DDR406, and the external memory 406 exchanges data with the main control system 404 through a memory interface (Memory interfaces). The main control system 404 can store data from the sound collection device 401 or data from the programmable logic unit 405 to the external memory DDR406. The main control system 404 can send the voice data collected from the sound collection device 401 to the programmable logic unit 405. The programmable logic unit 405 can be used to pre-process the voice data, for example, feature extraction and feature conversion processing, and then send the pre-processed data to the in-memory computing chip 402 to perform voice keyword detection processing using the voice keyword detection model. The in-memory computing chip 402 can send the voice keyword detection result calculated by the voice keyword detection model to the main control system 404 via the programmable logic unit 405. The main control system 403 then sends the voice keyword detection result to the terminal device 403, so that the terminal device 403 performs corresponding response operations based on the voice keyword detection result.
[0152] In the disclosed embodiment, the voice keyword detection model can be deployed on an in-memory computing chip based on the SRAM / Flash architecture for computational acceleration, using an FPGA or application processor as the main controller to deploy a main control system, and perform data interaction with the sound collection device, the in-memory computing chip, and the terminal device, thereby realizing functions such as voice control and voice interaction. In the voice control system, the sound collection device, the main control system, and the in-memory computing chip can be integrated into the terminal device. The in-memory computing chip uses voice keyword detection to control whether to wake up or control the terminal device. The standby power of the device is expected to be at the μW level, which can significantly reduce the standby power consumption of the device. Among them, the application processor can use the RK3399 processor, which is a low-power, high-performance application processor chip.
[0153] In some embodiments, the sound collection device can be integrated into the terminal device. In some embodiments, the main control system can be integrated into the terminal device. In some embodiments, the in-memory computing chip can also be integrated into the terminal device.
[0154] In some embodiments, the terminal device can also be an intelligent voice control device, such as home appliances, wearable devices, smart speakers, smart phones, computers, tablets, televisions and other electronic devices with voice control functions. Among them, smart home appliances with voice control functions are, for example, smart speakers, smart air conditioners, smart refrigerators and other home appliances, and wearable devices are, for example, smart watches, smart glasses, etc.
[0155] In some embodiments, the sound collection device is a device for collecting voice data, such as a microphone array.
[0156] In some application scenarios, the specific process of waking up and controlling the terminal device through the voice control system is as follows:
[0157] 0) The main control system is initialized.
[0158] 1) The main control system receives the voice data sent by the sound collection device;
[0159] 2) The main control system sends a wake-up word detection start instruction to the in-memory computing chip, waits for the response of the in-memory computing chip, and proceeds to the next step after the in-memory computing chip responds and confirms. Otherwise, the main control system sends the instruction again until the in-memory computing chip responds and confirms.
[0160] 3) The main control system receives the voice keyword detection result from the in-memory computing chip and responds with a confirmation; if the voice keyword detection result indicates that the corresponding wake-up word is recognized, it jumps to step 4); otherwise, it jumps to step 3) and continues to wait for the wake-up word detection result;
[0161] 4) The main control system controls the terminal device to provide wake-up feedback, such as playing the voice "I'm here", "I'm here", "How can I help you?", and then waits for further voice keyword detection results from the in-memory computing chip.
[0162] When the terminal device is awakened from the screen-off state for the first time, the screen is also turned on.
[0163] 5) When receiving further voice keyword detection results from the in-memory computing chip, a response confirmation is performed; if the voice keyword detection result indicates that the command word is recognized, jump to step 6), otherwise jump to step 7).
[0164] 6) The main control system controls the terminal device to execute the corresponding instructions, such as "open the video", "turn up the volume, etc.", and jump to step 5).
[0165] 7) Determine whether the command word has not been recognized before the timeout. If so, jump to step 2); otherwise, jump to step 5).
[0166] FIG7 is a block diagram of another electronic device provided by an embodiment of the present disclosure.
[0167] 7 , an embodiment of the present disclosure provides an electronic device 1000, which includes: at least one processor 1001; at least one memory 1002; and one or more I / O interfaces 1003 connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001 so that the at least one processor 1001 can execute the above-mentioned voice data processing method.
[0168] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned voice data processing method. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.
[0169] An embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned voice data processing method.
[0170] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).
[0171] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0172] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0173] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0174] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0175] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0176] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0177] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0178] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0179] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.
Claims
1. A method for processing speech data, wherein: The voice data processing method comprises: Obtaining voice data to be processed; Inputting the speech data to be processed into a programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed; and converting the speech feature vector into a multi-channel input feature map in the programmable logic unit; Inputting the input feature graph into an in-memory computing chip, performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtaining a speech keyword detection result; The voice keyword detection result is fed back to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
2. The method for processing speech data according to claim 1, wherein: The extracting speech features of the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed includes: The speech feature extraction algorithm is used to extract the speech feature of the speech data to be processed, so as to obtain the speech feature vector corresponding to the speech data to be processed.
3. The method for processing speech data according to claim 1, wherein: The speech feature vector is a two-dimensional speech feature vector, and the step of converting the speech feature vector into a multi-channel input feature map includes: Expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector; Copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors; The multiple three-dimensional speech feature vectors are spliced according to the channel dimension to obtain the multi-channel input feature map, where the channel dimension refers to the feature channel of the speech feature vector.
4. The method for processing speech data according to claim 1, wherein: The speech keyword detection model is a model built based on the convolutional hybrid network ConvMixer architecture, and the speech keyword detection model includes a feature sampling layer, a convolutional hybrid network and a classification output layer; The method of performing speech keyword detection on the input feature graph using the speech keyword detection model pre-configured on the in-memory computing chip and obtaining a speech keyword detection result includes: Downsampling the input feature map using the feature sampling layer to obtain a downsampled feature map; Using the convolutional hybrid network to perform convolutional hybrid processing on the downsampled feature map to obtain a speech recognition feature map; The classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.
5. The method for processing speech data according to claim 4, wherein: The feature sampling layer includes: a feature block embedding layer, a first batch normalization layer and a first activation function layer; Among them, the feature block embedding layer is implemented based on the convolution layer, and the activation function layer adopts the Swish activation function.
6. The method for processing speech data according to claim 4, wherein: The convolutional hybrid network includes at least one convolutional hybrid network module, and the convolutional hybrid network module includes a spatial position convolutional hybrid unit and a channel position convolutional hybrid unit. The spatial position mixing processing result output by the spatial position convolutional hybrid unit and the input of the spatial position convolutional hybrid unit are input to the channel position convolutional hybrid unit through a residual connection to perform channel position mixing processing; The spatial position convolutional hybrid unit comprises: a depth-separable convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence; The channel position convolution mixing unit comprises: a point-by-point convolution layer, a third batch normalization layer and a third activation function layer connected in sequence; Wherein, the second activation function layer and the third activation function layer both adopt the Swish activation function.
7. The method for processing speech data according to claim 4, wherein: The classification output layer includes: an average pooling layer and a fully connected layer; Wherein, the fully connected layer is implemented by a convolutional layer.
8. A method for training a speech keyword detection model, wherein: The training method comprises: Acquire an initial training data set, wherein the initial training data set includes a plurality of initial training speech data and a keyword label corresponding to each initial training speech data; For at least part of the initial training voice data, perform data expansion processing on the initial training voice data by at least one data expansion method to obtain at least one expanded training voice data and a keyword label corresponding to the expanded training voice data; The speech keyword detection model is trained according to the initial training speech data and the expanded training speech data to obtain a trained speech keyword detection model.
9. The training method according to claim 8, wherein: The at least one data expansion method includes: a data mixing method, a data noise adding method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.
10. The training method according to claim 8, wherein: The voice keyword detection model is trained, including: The speech keyword detection model is trained based on a quantitative perceptual training method.
11. A speech data processing device, wherein: The speech data processing device comprises: An acquisition unit, configured to acquire voice data to be processed; A feature extraction unit, configured to input the speech data to be processed into the programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed; A feature conversion unit, configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit; A detection unit, configured to input the input feature graph into an in-memory computing chip, perform speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtain a speech keyword detection result; A response unit is configured to feed back the voice keyword detection result to a main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
12. A voice control system, wherein: include: A sound collection device is used to collect the voice data to be processed in the environment and send the voice data to be processed to the main control system; A main control system, used for receiving the voice data to be processed collected by the sound collection device, and sending the voice data to be processed to the programmable logic unit; A programmable logic unit, configured to perform speech feature extraction on the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed, and convert the speech feature vector into a multi-channel input feature map; An in-memory computing chip having a preset voice keyword detection model integrated thereon, for performing voice keyword detection on the input feature graph using the voice keyword detection model pre-configured on the in-memory computing chip, and obtaining a voice keyword detection result; The main control system is further used to receive the voice keyword detection result of the in-memory computing chip, and control the terminal device to perform a corresponding response operation according to the voice keyword detection result; The terminal device is used to execute a response operation corresponding to the voice keyword detection result.
13. An electronic device, wherein: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the speech data processing method as described in any one of claims 1 to 7 or execute the training method of the speech keyword detection model as described in claim 8.
14. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the computer program implements the speech data processing method according to any one of claims 1 to 7 or executes the training method of the speech keyword detection model according to claim 8.
Citation Information
Patent Citations
Voice keyword recognition system and method based on graph convolutional neural network
CN112435652A
Voice wake-up method and device, storage medium and program product
CN112825250A
Data processing method and device, equipment and storage medium
CN113222107A
Voice keyword recognition method and system
CN114937450A
Voice signal processing method and device, equipment and medium
CN115035887A
Cited By
Evaluation data generation method and device, storage medium and program product
CN119443061A
Lumbar lesion recognition method and device, medium and electronic equipment
CN121033542A
