Voice data processing method and device and voice control system

By adopting a voice keyword detection model based on the ConvMixer architecture in smart devices, the high consumption problem of the voice keyword detection module in smart devices for power and storage space is solved, and low power consumption and low latency voice keyword detection is achieved. Improves the service time and operation speed of the equipment.

CN119068873APending Publication Date: 2024-12-03BOE TECHNOLOGY GROUP CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310639508.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The voice keyword detection module consumes a lot of power and storage space in smart devices, which affects the use time and operation speed of the device. Especially in devices that use batteries, such as smart watches and Bluetooth speakers, power consumption problems are serious.

Method used

Using a voice keyword detection model based on the ConvMixer architecture, voice keyword detection is performed through the feature sampling layer, the convolutional hybrid network and the classification output layer, and model training and deployment is performed on the in-memory computing chip to reduce the Consumption of computing and storage resources.

Benefits of technology

It realizes low power consumption and low latency voice keyword detection, reduces the equipment's demand for power and storage, improves the use time and operation speed of smart devices, and maintains a high detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068873A_ABST
    Figure CN119068873A_ABST
Patent Text Reader

Abstract

The invention provides a voice data processing method and device and a voice control system, relates to the technical field of computers, and can be applied to the technical fields of artificial intelligence, voice recognition, deep learning and the like. Sending the to-be-processed voice data to a programmable logic unit to perform voice feature extraction on the to-be-processed voice data, and obtaining a voice feature vector corresponding to the to-be-processed voice data; converting the voice feature vector into a multi-channel input feature map in a programmable logic unit; sending the input feature map to an in-memory computing chip, and performing voice keyword detection on the input feature map by using a voice keyword detection model pre-configured on the in-memory computing chip to obtain a voice keyword detection result; the voice keyword detection result is fed back to the main control system, and the main control system is used for executing response operation corresponding to the voice keyword detection result according to the voice keyword detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a voice data processing method and device, a training method for a voice keyword detection model, a voice control system, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence (AI) algorithms and hardware technologies such as AI chips, smart devices have been widely used in daily life, such as smart home voice control systems, smart speakers, smart conference systems, etc. Voice interaction is widely used in smart devices and is becoming increasingly mature. In traditional voice interaction scenarios, the device was initially woken up by clicking a button, such as pressing the record button, to interact with the device.

[0003] In order to further improve the human-computer interaction experience, speech keyword detection technology came into being. Speech wake-up or keyword detection (Speech Keyword Spotting) is a sub-field of speech recognition (Speech Recognition). Speech recognition refers to extracting useful information from a continuous speech signal and converting it into a string that can be recognized by a computer, usually a binary character stream or text content. Speech keyword detection does not require the entire speech signal to be recognized, but only needs to determine whether the signal contains one or more keywords.

[0004] At present, there are three main methods for voice keyword detection: wake-up technology based on template matching; wake-up technology based on hidden Markov model; wake-up technology based on deep learning. Among them, the most widely used is the voice recognition wake-up method based on deep learning. Summary of the invention

[0005] The present invention provides a speech data processing method and device, a training method for a speech keyword detection model, a speech control system, an electronic device, and a computer-readable storage medium.

[0006] According to a first aspect of the present disclosure, the present disclosure provides a method for processing speech data.

[0007] The law includes:

[0008] Obtaining voice data to be processed;

[0009] Inputting the speech data to be processed into a programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed;

[0010] and converting the speech feature vector into a multi-channel input feature map in the programmable logic unit;

[0011] Inputting the input feature graph into an in-memory computing chip, performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtaining a speech keyword detection result;

[0012] The voice keyword detection result is fed back to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.

[0013] In some embodiments, said performing speech feature extraction on the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed includes:

[0014] The speech feature extraction algorithm is used to extract the speech feature of the speech data to be processed, so as to obtain the speech feature vector corresponding to the speech data to be processed.

[0015] In some embodiments, the speech feature vector is a two-dimensional speech feature vector, and converting the speech feature vector into a multi-channel input feature map includes:

[0016] Expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector;

[0017] Copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors;

[0018] The multiple three-dimensional speech feature vectors are concatenated according to the channel dimension to obtain the multi-channel input feature map.

[0019] In some embodiments, the speech keyword detection model is a model built based on a convolutional hybrid network ConvMixer architecture, and the speech keyword detection model includes a feature sampling layer, a convolutional hybrid network, and a classification output layer;

[0020] The method of performing speech keyword detection on the input feature graph using the speech keyword detection model pre-configured on the in-memory computing chip and obtaining a speech keyword detection result includes:

[0021] Downsampling the input feature map using the feature sampling layer to obtain a downsampled feature map;

[0022] Using the convolutional hybrid network to perform convolutional hybrid processing on the downsampled feature map to obtain a speech recognition feature map;

[0023] The classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.

[0024] In some embodiments, the feature sampling layer includes: a feature block embedding layer, a first batch normalization layer, and a first activation function layer;

[0025] Among them, the feature block embedding layer is implemented based on the convolution layer, and the activation function layer adopts the Swish activation function.

[0026] In some embodiments, the convolutional mixing network includes at least one convolutional mixing network module, and the convolutional mixing network module includes a spatial position convolutional mixing unit and a channel position convolutional mixing unit. The spatial position mixing processing result output by the spatial position convolutional mixing unit and the input of the spatial position convolutional mixing unit are input to the channel position convolutional mixing unit through a residual connection to perform channel position mixing processing;

[0027] The spatial position convolutional hybrid unit comprises: a depth-separable convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence;

[0028] The channel position convolution mixing unit comprises: a point-by-point convolution layer, a third batch normalization layer and a third activation function layer connected in sequence;

[0029] Wherein, the second activation function layer and the third activation function layer both adopt the Swish activation function.

[0030] In some embodiments, the classification output layer includes: an average pooling layer and a fully connected layer;

[0031] Wherein, the fully connected layer is implemented by a convolutional layer.

[0032] According to a second aspect of the present disclosure, the present disclosure provides a method for training a speech keyword detection model, the training method comprising:

[0033] Acquire an initial training data set, wherein the initial training data set includes a plurality of initial training speech data and a keyword label corresponding to each initial training speech data;

[0034] For at least part of the initial training voice data, perform data expansion processing on the initial training voice data by at least one data expansion method to obtain at least one expanded training voice data and a keyword label corresponding to the expanded training voice data;

[0035] The speech keyword detection model is trained according to the initial training speech data and the expanded training speech data to obtain a trained speech keyword detection model.

[0036] In some embodiments, the at least one data expansion method includes: a data mixing method, a data noise adding method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.

[0037] In some embodiments, performing model training on the speech keyword detection model includes:

[0038] The speech keyword detection model is trained based on a quantitative perceptual training method.

[0039] According to a third aspect of the present disclosure, the present disclosure provides a voice data processing device, the voice data processing device comprising:

[0040] An acquisition unit, configured to acquire voice data to be processed;

[0041] A feature extraction unit, configured to input the speech data to be processed into the programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed;

[0042] A feature conversion unit, configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit;

[0043] A detection unit, configured to input the input feature graph into an in-memory computing chip, perform speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtain a speech keyword detection result;

[0044] A response unit is configured to feed back the voice keyword detection result to a main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.

[0045] According to a fourth aspect of the present disclosure, the present disclosure provides a voice control system, the voice control system comprising:

[0046] A sound collection device is used to collect the voice data to be processed in the environment and send the voice data to be processed to the main control system;

[0047] A main control system, used for receiving the voice data to be processed collected by the sound collection device, and sending the voice data to be processed to the programmable logic unit;

[0048] A programmable logic unit, configured to send the voice data to be processed to the programmable logic unit to perform voice feature extraction on the voice data to be processed, so as to obtain a voice feature vector corresponding to the voice data to be processed, and convert the voice feature vector into a multi-channel input feature map;

[0049] An in-memory computing chip having a preset voice keyword detection model integrated thereon, for performing voice keyword detection on the input feature graph using the voice keyword detection model pre-configured on the in-memory computing chip, and obtaining a voice keyword detection result;

[0050] The main control system is further used to receive the voice keyword detection result of the in-memory computing chip, and control the terminal device to perform a corresponding response operation according to the voice keyword detection result;

[0051] The terminal device is used to execute a response operation corresponding to the voice keyword detection result.

[0052] According to a fifth aspect of the present disclosure, the present disclosure provides an electronic device, the electronic device comprising:

[0053] at least one processor; and

[0054] a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned voice data processing method.

[0056] According to a sixth aspect of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned voice data processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0058] Figure 1 A flowchart of a method for processing voice data provided by an embodiment of the present disclosure;

[0059] Figure 2 A schematic diagram of a model architecture of a speech keyword detection model provided in an embodiment of the present disclosure;

[0060] Figure 3 A schematic diagram of the structure of a voice data processing device provided in an embodiment of the present disclosure;

[0061] Figure 4 A schematic diagram of the structure of a voice control system provided by an embodiment of the present disclosure;

[0062] Figure 5 A schematic diagram of an application scenario of a voice control system provided by an embodiment of the present disclosure;

[0063] Figure 6 A schematic diagram of another application scenario of a voice control system provided by an embodiment of the present disclosure;

[0064] Figure 7 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0067] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0068] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0069] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0070] In related technologies, in order to achieve a good interactive experience between people and smart devices, smart devices monitor voice in real time, which creates a serious problem: the consumption of resources such as power and storage space by the voice keyword detection module. Many smart devices are not directly connected to the power supply, but use batteries, such as smart watches and Bluetooth speakers. The power consumption of the voice keyword detection function directly affects the use time and running speed of smart devices.

[0071] Figure 1 A flow chart of a voice data processing method provided by an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the voice data processing method includes: steps S11 to S14.

[0072] Step S10: Acquire voice data to be processed.

[0073] Step S11: Input the speech data to be processed into a programmable logic unit to perform speech feature extraction on the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed.

[0074] The voice data to be processed is obtained, and the voice data to be processed is input into a programmable logic unit. The voice features of the voice data to be processed are extracted by the programmable logic unit. Specifically, a preset voice feature extraction algorithm is used to extract the voice features of the voice data to be processed, and a voice feature vector corresponding to the voice data to be processed is extracted.

[0075] Exemplarily, speech feature extraction is performed on the speech data to be processed based on Mel Frequency Cepstrum Coefficient (MFCC) to obtain a speech feature vector corresponding to the speech data to be processed, wherein the Mel Frequency Cepstrum Coefficient MFCC is a speech feature extraction algorithm.

[0076] In the disclosed embodiment, the speech feature vector extracted based on the Mel-frequency cepstral coefficient MFCC is an Fbank feature, and the speech feature vector is a two-dimensional speech feature vector.

[0077] In the MFCC algorithm, the extraction of Fbanks features involves several steps, including pre-emphasis processing, frame processing, windowing processing, Fourier transform processing, power spectrum calculation processing, and filtering through a Mel filter bank.

[0078] Specifically, the voice data is pre-emphasized. Pre-emphasis is a signal processing method that compensates for the high-frequency components of the input signal at the transmitting end. The function of pre-emphasis is to increase the energy of the high-frequency signal in the voice data and improve the signal-to-noise ratio.

[0079] Furthermore, the voice data after pre-emphasis processing is framed to obtain multiple audio frames. Among them, the frame processing refers to dividing the voice data into voice signal segments of fixed size, and each segment of the voice signal is called a frame, and the general frame length range is 20 milliseconds to 40 milliseconds. When performing the frame processing, the overlapping segmentation method can be adopted, and the ratio of the frame shift to the frame length ranges from 0 to 1 / 2, wherein the frame shift is the overlapping part of the previous frame and the next frame. By utilizing the short-term stability of the signal, a smooth transition between frames is achieved to maintain its continuity, and at the same time, the problem of information omission caused by the boundary of the time window can be avoided, reducing the impact on the windowing processing step. In the embodiment of the present disclosure, the frame length is 25ms, the frame shift is 10ms, the sampling frequency of the voice data is 16000 / s, and the length of the voice data is 1s.

[0080] Furthermore, each audio frame is subjected to windowing. If the frame head and frame tail values ​​of each audio frame in the speech data are the same, it can be assumed that the frame is periodic, and the length of the period is exactly equal to the frame length. The role of windowing is to add different weights to each sampling point of the speech data to achieve the same frame head and frame tail values, so as to eliminate the signal discontinuity that may be caused by the frame head and frame tail of each frame.

[0081] Further, each audio frame is subjected to Fourier transform to obtain spectrum information corresponding to each audio frame. The Fourier transform is used to convert a time domain signal into a frequency domain signal. In some embodiments, the voice data is a sampled digital signal, and the digital signal is a discrete value. Therefore, the Fourier transform can adopt a discrete Fourier transform (DFT for short).

[0082] Furthermore, a power spectrum is calculated according to the frequency spectrum information corresponding to the audio frame, and the power spectrum is a ratio of the square of the modulus of the frequency spectrum information obtained by Fourier transform to the length.

[0083] Furthermore, the power spectrum of each audio frame is filtered using a Mel filter bank to obtain the spectrum characteristics of each audio frame. The Mel filter bank can be a triangular filter bank, and the power spectrum is processed by a triangular filter to obtain a one-dimensional vector, and a two-dimensional matrix is ​​obtained after being processed by a group of triangular filter banks. Further, the spectrum characteristics of the two-dimensional matrix are logarithmically calculated to obtain Fbank characteristics, wherein the logarithmic calculation is calculated using a logarithmic function with a base of 10.

[0084] Exemplarily, the length of the Fbank feature vector is 64, that is, after using MFCC to extract speech features from speech data, a 98×64 two-dimensional Fbank feature vector can be obtained. Expanding the Fbank feature vector into a three-dimensional vector can obtain a tensor with a shape of [1,98,64], which is similar to a feature image with a length of 98, a width of 64, and a channel number of 1.

[0085] Step S12: converting the speech feature vector into a multi-channel input feature map in the programmable logic unit.

[0086] In the programmable logic unit, the two-dimensional speech feature vector is converted into a multi-channel input feature map by using dimensional expansion, feature replication, feature concatenation and other methods.

[0087] In some embodiments, in step S12, converting the speech feature vector into a multi-channel input feature map may further include: expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector; copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors, for example, copying it into three copies, the three three-dimensional speech feature vectors are respectively a first speech feature vector, a second speech feature vector and a third speech feature vector; splicing the multiple three-dimensional speech feature vectors according to the channel dimension to obtain a multi-channel input feature map, where the channel dimension refers to the feature channel of the speech feature vector.

[0088] In some embodiments, the unsequeeze function can be used to expand the feature dimension of the speech feature vector. The unsequeeze function is an operation function used to add a dimension with a dimension value of 1 at a specified position of the data. For example, the speech feature vector is a two-dimensional Fbank feature vector of 98×64. The unsequeeze function is used to expand the speech feature vector into a three-dimensional feature vector, and a tensor with a shape of [1,98,64] can be obtained, which is similar to a feature image with a length of 98, a width of 64, and a channel number of 1.

[0089] In some embodiments, the concat function can be used to concatenate multiple three-dimensional speech feature vectors according to the channel dimension to obtain a multi-channel input feature map, and the number of channels of the input feature map is 3. The concat function is an operation function that concatenates two or more tensors in a certain dimension.

[0090] Step S13: input the input feature graph into the in-memory computing chip, and use the pre-configured speech keyword detection model on the in-memory computing chip to perform speech keyword detection on the input feature graph to obtain a speech keyword detection result.

[0091] In the disclosed embodiment, a speech keyword detection model is pre-configured on the in-memory computing chip. In step S13, the multi-channel input feature map is input into the in-memory computing chip, and speech keyword detection processing is performed using the pre-configured speech keyword detection model to obtain a speech keyword detection result.

[0092] Figure 2 A model architecture diagram of a speech keyword detection model provided in an embodiment of the present disclosure. In some embodiments, the speech keyword detection model is a model constructed based on a convolutional hybrid network ConvMixer architecture, such as Figure 2 As shown in the figure, the speech keyword detection model built based on the ConvMixer architecture includes an input layer, a feature sampling layer, a convolutional hybrid network ConvMixer and a classification output layer connected in sequence, and the input layer is used to input the input feature map into the feature sampling layer.

[0093] Specifically, using a pre-configured voice keyword detection model on an in-memory computing chip to perform voice keyword detection on an input feature graph and obtaining a voice keyword detection result may further include the following steps:

[0094] First, the input feature map is downsampled using the feature sampling layer to obtain a downsampled feature map.

[0095] Among them, Figure 2 As shown, the feature sampling layer includes: a feature block embedding layer (Patch Embedding), a first batch normalization layer (BatchNorm, BN) and a first activation function layer; wherein, the feature block embedding layer is implemented based on the convolution layer, and the activation function layer adopts the Swish activation function.

[0096] Among them, the feature block embedding layer (Patch Embedding) is used to downsample the input feature map, the first batch normalization layer, that is, the BN layer, is used to normalize the input features, and the first activation function layer is used to change the linear relationship of the data and delinearize the data, that is, nonlinear transformation processing.

[0097] Assuming the feature block patch size is p, the embedding dimension is h, and the number of channels of the input feature map is Cin, the output z0 of the Patch Embedding layer can be expressed as:

[0098] z0=σ(BN(Conv cin→h (X, stride=p, kernel_size=p)))

[0099] Exemplarily, in the above formula, p is 64, kernel size = p is a 2D convolution of 5x5, followed by a Swish activation function σ and a batch normalization layer (BN layer), and the output of the Patch Embedding layer, that is, the feature map used next, is a tensor that can be expressed as [Batch size, 64, 25, 13], that is, the final input to the convolutional hybrid network ConvMixer is a tensor expressed as [Batch size, 64, 25, 13]. The downsampling process of the feature map is completed through the Patch Embedding layer, which reduces the resolution of the feature map, increases the receptive field, and makes it easier to find deep spatial information. At the same time, the computational complexity and parameter amount of the convolutional hybrid network ConvMixer are reduced, thereby reducing the space occupancy of the model for storage resources and reducing the model's consumption of computing resources.

[0100] Then, the convolutional mixing network is used to perform convolution mixing processing on the downsampled feature map to obtain the speech recognition feature map.

[0101] Among them, Figure 2 As shown, the convolutional hybrid network includes at least one convolutional hybrid network module (ConvMixer-Block), the number of convolutional hybrid network modules is L, and the L convolutional hybrid network modules are connected in series in sequence. The value of L can be set according to actual needs. Among them, each convolutional hybrid network module includes a spatial position convolutional hybrid unit and a channel position convolutional hybrid unit. The spatial position mixing processing result output by the spatial position convolutional hybrid unit and the input of the spatial position convolutional hybrid unit are input to the channel position convolutional hybrid unit through a residual connection to perform channel position mixing processing.

[0102] Among them, Figure 2 As shown, the spatial position convolution hybrid unit includes: a depth-separable convolution (ConvDepthwise) layer, a second batch normalization (BN) layer, and a second activation function layer connected in sequence; Figure 2 As shown, the channel position convolution hybrid unit includes: a point-by-point convolution (ConvPointwise) layer, a third batch normalization layer and a third activation function layer connected in sequence; wherein the second activation function layer and the third activation function layer both adopt the Swish activation function.

[0103] In some embodiments, in the convolutional hybrid network module, a depth-separable convolution (ConvDepthwise) layer is selected to mix the spatial position features, and a point-wise convolution (ConvPointwise) layer (with a convolution kernel size of 1x1) is selected to mix the channel position features, while effectively ensuring that the model has a small amount of network parameters. The number of channels of the convolution layer is consistent with the embedding dimension h of the above-mentioned PatchEmbedding layer. The convolutional hybrid network module (ConvMixer-Block) can be expressed by the following formula:

[0104] z' l =σ(BN(ConvDepthwise(z l-1 )))+z l-1

[0105] z l =σ(BN(ConvPointwise(z' l )))

[0106] Where BN is the batch normalization layer and σ is the swish activation function.

[0107] Finally, the classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.

[0108] Among them, Figure 2 As shown, the classification output layer includes: an average pooling layer (Average pooling) and a fully connected layer; wherein the fully connected layer is implemented by a convolutional layer (Conv2d), the average pooling layer is a 2d average pooling layer, and in the fully connected layer, the convolutional layer uses a convolutional layer with a convolution kernel size of 1×1, and the number of output channels is equal to the preset number of keyword categories.

[0109] In the disclosed embodiment, the convolutional layer is used as the fully connected layer, which can significantly reduce the size of model parameters and reduce the amount of model calculation, thereby helping to reduce the space occupied by the model for storage resources and reducing the model's consumption of computing resources.

[0110] Specifically, the speech recognition features output by the convolutional hybrid network are input into the 2D average pooling layer for pooling processing, and then the keyword classification prediction processing is performed through the fully connected layer using the softmax activation function, and an N+2 classification output result can be obtained, wherein N is the total number of pre-set voice wake-up words and voice command words, and 2 represents the silence category and the unknown category, that is, the categories of voice keywords include voice wake-up words, voice command words, silence and unknown, wherein the voice wake-up words include but are not limited to "XiaoXiaoX", "Hello XiaoX" and other strings consisting of no less than 4 words, and the voice command words include but are not limited to "turn on the air conditioner", "increase the volume", "turn up the brightness" and other series of control commands, and the output result can represent the probability of each keyword category to which the voice data belongs. According to the output result of the fully connected layer, the voice keyword detection result is obtained. For example, the category with the largest probability can be selected as the voice keyword detection result, or the category with a probability greater than the preset threshold can be selected as the voice keyword detection result.

[0111] In some embodiments, Figure 2 As shown, a dimension conversion (View) layer is provided after the fully connected layer, which is used to perform dimension conversion on the output result of the fully connected layer to obtain an output result of the required dimension.

[0112] Step S14: Feedback the voice keyword detection result to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.

[0113] For example, when the voice keyword detection result is a certain voice wake-up word, it can be used to control the intelligent voice control device to respond. For example, if the input voice data is "Xiao X Xiao X", the response of the intelligent voice control device can be "Xiao X is here", "Here", etc. When the voice keyword detection result is a certain voice command word, it can be used to control the corresponding smart terminal to execute the corresponding control command. For example, when the voice command word is "turn on the air conditioner", it can be used to control the air conditioner to turn on and execute the preset or last set operation. When the voice command word is "increase the volume", it controls the player currently in the playing state to increase the volume by one level. When the voice command word is "increase the brightness", it controls the lighting device currently in the lighting state to increase the brightness by one level.

[0114] The neural network model is composed of a series of network layers. Each layer receives the output of the previous layer and obtains the result as the input of the next layer after processing. Each layer of the neural network model has its own parameters. During the training process, if the parameters of a certain layer change, the output of the layer, that is, the input distribution of the next layer, will change. After continuous accumulation layer by layer, this change will be continuously amplified. With each iteration, as the parameters are updated, the distribution of input features will change, and the parameters of the neural network model need to be re-adapted to the input distribution, which greatly increases the training complexity of the neural network model. The batch normalization layer (Batch Norm, BN) can effectively alleviate this problem. The batch normalization layer performs a similar standardization operation on the input features, which can stabilize the distribution of input features and accelerate the training process of the neural network model.

[0115] In the disclosed embodiments, in the feature sampling layer and the convolutional hybrid network of the speech keyword detection model, since the batch normalization layer is located after the convolutional layer and is connected in series with the convolutional layer, the batch normalization layer can be fused with the convolutional layer, which can help accelerate network calculations and reduce the number of parameters and calculations of the model, thereby reducing the space occupied by the model for storage resources and reducing the model's consumption of computing resources.

[0116] For example, it is assumed that the convolution calculation method with bias is:

[0117] y=x*w+b

[0118] The calculation formula of the BN layer is:

[0119]

[0120] Substituting y into z yields:

[0121]

[0122] Let w n =β·w / y var ,b n =β·(by mean ) / y var +γ, where x represents input, w represents weight, b, γ, β are constants, mean represents mean, and var represents variance.

[0123] The new convolution parameters after the fusion of the convolution layer and the BN layer can be expressed as:

[0124] z=x*w n +b n

[0125] In the disclosed embodiment, the voice keyword detection model is integrated into an in-memory computing chip, and the in-memory computing chip is used to perform voice keyword detection.

[0126] In the disclosed embodiments, the in-memory computing chip can effectively reduce computing power consumption compared to the traditional von Neumann architecture chip. Specifically, in the von Neumann architecture chip, the computing unit is separated from the memory, and data needs to be read from the memory to complete the data calculation, and the data needs to be written back to the memory after the calculation is completed. The data transmission between the computing unit and the memory generates huge power consumption. The in-memory computing chip embeds the computing function in the memory, and the storage and computing are integrated. During the operation of the voice keyword detection model, the data transmission between the computing unit and the memory is reduced, and the memory uses analog circuits instead of traditional digital circuits to realize the calculation, and the computing power consumption is lower.

[0127] In the disclosed embodiment, the speech keyword detection model based on the ConvMixer architecture has good accuracy and maintains a small number of model parameters, and can be easily integrated into a low-power in-memory computing chip to achieve low-power, low-latency speech keyword detection.

[0128] According to the voice data processing method provided by the embodiment of the present disclosure, a lightweight voice keyword detection model is integrated with a low-power in-memory computing chip to perform voice keyword detection and recognition. Voice keyword detection can be achieved with low power consumption and low latency, and wake-up word and command word recognition tasks can be achieved simultaneously without the use of additional voice recognition models. It can be applied to run on edge devices and can also be applied to terminal devices to realize functions such as voice wake-up devices and voice control devices.

[0129] The disclosed embodiments also provide a method for training a speech keyword detection model, the training method comprising: obtaining an initial training data set, the initial training data set comprising a plurality of initial training speech data and a keyword label corresponding to each initial training speech data, the keyword label corresponding to the initial training speech data being a pre-labeled keyword category to which the initial training speech data belongs, such as a wake-up word or a voice command word; for at least a portion of the initial training speech data, performing data expansion processing on the initial training speech data by at least one data expansion method to obtain at least one expanded training speech data and a keyword label corresponding to the expanded training speech data; performing model training on a speech keyword detection model based on the initial training speech data and the expanded training speech data to obtain a speech keyword detection model after training.

[0130] In some embodiments, at least one data expansion method includes: a data mixing method, a data noise adding method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.

[0131] Among them, the data mixing method is a data enhancement method based on Mixup. For supervised learning tasks, the data mixing Mixup method is an effective data enhancement strategy. This method can combine the initial training speech data with keyword labels to make the model more robust to adversarial samples and reduce the model's misjudgment of sample categories. The data mixing method is shown in the following formula:

[0132] X mix =λX i +(1-λ)X j

[0133] Y mix =λY i +(1-λ)Y j

[0134] Among them, X i represents the i-th initial training speech data, Y i represents the keyword label corresponding to the i-th initial training speech data, X j represents the jth initial training speech data, Y j represents the keyword label corresponding to the jth initial training speech data, and the mixing ratio λ conforms to the Beta distribution, that is, λ~Beta(α,α), where α=10 can be taken, X mix represents the expanded training speech data obtained by data expansion processing through data mixing, Y mix Represents the expanded training speech data X mix The corresponding keyword tag.

[0135] Data noise addition refers to adding a specified type of noise data to the given initial training speech data by specifying the signal-to-noise ratio, such as music, TV drama background, reverberation, etc. Given the initial training speech data S, noise data N, and signal-to-noise ratio SNR, we have:

[0136] SNR = 10lg (Ps / Pn)

[0137] Ps=avg(s 2 ),Pn=avg(n 2 )

[0138]

[0139] Noise=S+scalar·N

[0140] Wherein Ps is the average kilometer of the initial training speech data S, Pn is the average power of the noise data N, Noise represents the expanded training speech data obtained by data expansion processing by adding noise to the data, and the keyword label of the expanded training speech data Noise is the same as the keyword label of the initial training speech data S.

[0141] Method for constructing negative samples: There is an important indicator for voice wake-up / keyword detection problems, namely, the false wake-up / false recognition rate. For example, if the wake-up word is "Hello, Xiaojing", if the model is overfitting or has poor robustness, the greeting "Hello" during a normal conversation may also trigger voice wake-up. To avoid such situations, negative samples can be constructed based on the initial training data set.

[0142] The method for constructing negative samples is as follows: based on the speech ASR (Automatic Speech Recognition) model to identify the initial training speech data, and output keyword tags with timestamp information, wherein ASR can adopt any existing suitable speech recognition model; based on the timestamp information, the original initial training speech data is segmented, for example, the original initial training speech data is "Hello Xiaojing", the original initial training speech data is segmented into two audio clips of "Hello" and "Xiaojing", and the two audio clips can be randomly spliced, so that four negative sample data can be obtained: "Hello", "Xiaojing", "Hello hello", "Hello Xiaojing".

[0143] In addition, different initial training voice data can be spliced ​​together to form new negative samples. For example, the initial training voice data "turn up the volume" and the initial training voice data "light up the screen" can be segmented into audio clips of "turn up", "volume", "light up", and "screen". The negative samples obtained by random combination and splicing can be "turn up the screen", "light up the volume", etc.

[0144] Finally, the constructed negative samples are used as augmented training speech data, and their corresponding labels are set to unknown unknow labels.

[0145] In addition, the training data can also be expanded using speed perturbation, spectrum enhancement, etc., which will not be described here.

[0146] Through the above data expansion method, the amount of training data can be increased, the diversity of training data can be improved, and it is beneficial to increase the robustness of the model.

[0147] In the disclosed embodiment, before the voice keyword detection model is trained based on the initial training voice data and the expanded training voice data, voice features are extracted from the voice data to obtain a voice feature vector corresponding to the voice data, and the voice feature vector is converted into a multi-channel input feature map. Then, the input feature map is input into the model for model training.

[0148] In some embodiments, performing model training on the speech keyword detection model includes: performing model training on the speech keyword detection model based on a Quantification Aware Training (QAT) method.

[0149] In the quantization-aware training method, the entire model can be quantized using integer INT8, which involves pseudo-quantization nodes. Pseudo-quantization nodes refer to nodes inserted in quantization-aware training to find the data distribution of the model and feedback the accuracy loss. Its specific functions are as follows:

[0150] (1) Find the distribution of network data, that is, find the maximum and minimum values ​​of the parameters to be quantified.

[0151] (2) Simulate the accuracy loss when quantized to low bits, apply this loss to the model, pass it to the loss function, and let the optimizer optimize the loss during the training process.

[0152] In some embodiments, a speech keyword detection model may be pre-trained using the above-mentioned training method, and then the trained speech keyword detection model may be deployed to an in-memory computing chip for execution.

[0153] In addition, the following effectiveness verification was carried out for the speech data processing method proposed in the embodiment of the present invention: the speech data processing method proposed in the embodiment of the present invention was experimentally tested using the Google Speech Commands V2 (GSC-V2) dataset. GSC-V2 contains 105,829 command words and 2,618 speakers, including 35 command words such as 'down', 'go', 'left', 'no', 'off', 'on', 'right', 'stop', 'up', and 'yes'. Experiments were conducted using models of different sizes. For example, for the speech keyword detection model of the embodiment of the present disclosure, the model depth is 8, the hidden layer size is 64, which is called KwsConvMixer-S (KCM-S), the model depth is 12, the hidden layer size is 64, which is called KwsConvMixer-M (KCM-M), the model depth is 12, the hidden layer size is 128, which is called ConvMixer-L (KCM-L). During the experiment, the adamw optimizer was used, the warmup epoch was 10, the number of iterations was 25000, the learning rate was 0.02, and the batch size was 256. The test results are shown in Table 1.

[0154] Table 1

[0155] Model Parameter quantity Accuracy % DSCNN 0.42M 97.2 MHAtt-RNN 0.784M 97.27 KWT-1 0.607M 96.85 KWT-2 2.394M 97.53 KWT-3 5.361M 97.51 KW-MLP 0.424M 97.56 KCM-S(ours) 0.05M 96.24 KCM-M(ours) 0.08M 96.7 KCM-L(ours) 0.26M 97.9

[0156] It can be seen that the KCM-S model uses only 96k parameters to achieve an accuracy of 96.24, while ConvMixer-L uses 0.26M parameters to achieve an accuracy of 97.9. This shows that the speech keyword detection model of the embodiment of the present disclosure has better prediction effect than the MLP-based model and the tansformer-based model, and the model parameter amount is smaller.

[0157] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0158] Figure 3 A schematic diagram of the structure of a voice data processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the speech data processing device 300 includes: a feature extraction unit 301, a feature conversion unit 302, a detection unit 303, a response unit 304 and an acquisition unit 305.

[0159] Among them, the acquisition unit 305 is configured to acquire the speech data to be processed; the feature extraction unit 301 is configured to input the speech data to be processed into the programmable logic unit to perform speech feature extraction on the speech data to be processed, so as to obtain the speech feature vector corresponding to the speech data to be processed; the feature conversion unit 302 is configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit; the detection unit 303 is configured to input the input feature map into the in-memory computing chip, perform speech keyword detection on the input feature map using the speech keyword detection model pre-configured on the in-memory computing chip, and obtain the speech keyword detection result; the response unit 304 is configured to feed back the speech keyword detection result to the main control system, and the main control system is used to execute a response operation corresponding to the speech keyword detection result according to the speech keyword detection result.

[0160] The speech data processing device 300 provided in the embodiment of the present disclosure is used to implement the speech data processing method provided in any of the above embodiments. For specific related descriptions, please refer to the description of the speech data processing method in any of the above embodiments, which will not be repeated here.

[0161] Figure 4 A schematic diagram of a voice control system provided by an embodiment of the present disclosure is shown in FIG. Figure 4As shown, the voice control system 400 includes: a sound collection device 401, a processing in memory chip (PIMChip) 402, a terminal device 403, a main control system (Processing System, PS) 404 and a programmable logic unit (Programmable Logic, PL) 405.

[0162] The sound collection device 401 is used to collect the voice data to be processed in the environment and send the voice data to be processed to the main control system; the main control system 404 is used to receive the voice data to be processed collected by the sound collection device 401 and send the voice data to be processed to the programmable logic unit 405; the programmable logic unit 405 is used to extract voice features from the voice data to be processed to obtain the voice feature vector corresponding to the voice data to be processed, and convert the voice feature vector into a multi-channel input feature map; the in-memory computing chip 402 is integrated with a preset voice keyword detection model, which is used to perform voice keyword detection on the input feature map using the voice keyword detection model pre-configured on the in-memory computing chip 402, and obtain the voice keyword detection result, and feedback the voice keyword detection result to the main control system 404; the main control system 404 is also used to receive the voice keyword detection result of the in-memory computing chip, and according to the voice keyword detection result, control the terminal device 403 to perform a corresponding response operation; the terminal device 403 is used to perform a response operation corresponding to the voice keyword detection result; the main control system 404 and the programmable logic unit 405 can be integrated into an FPGA (Field Programmable Gate Array). The Field Programmable Gate Array (FPGA) device can exchange data with the sound collection device 401, the in-memory computing chip 402, and the terminal device 403.

[0163] In some embodiments, the main control system is a system on chip (SOC), and a Linux operating system can be deployed on the FPGA device as the main control system (PS) to realize the control and scheduling of hardware resources, and implement some algorithms with strong versatility and low performance requirements based on the operating system.

[0164] Figure 5 A schematic diagram of an application scenario of a voice control system provided by an embodiment of the present disclosure, such as Figure 4 and Figure 5As shown, in some application scenarios, the main control system 404 and the programmable logic unit 405 are deployed in the FPGA device, and the in-memory computing chip 402 is implemented based on the flash memory FLASH architecture. The FPGA device and the in-memory computing chip 402 can interact with each other through a variety of data transmission methods, such as data transmission through UART, SPI, I2C, INT, I2S and other interfaces, and the FPGA device and the terminal device 403 can be connected through an HDMI interface.

[0165] Figure 6 A schematic diagram of another application scenario of a voice control system provided by an embodiment of the present disclosure, such as Figure 6 As shown, in some application scenarios, the main control system 404 is deployed on an FPGA device, and a programmable logic unit (PL) 405 is also deployed on the FPGA device. The in-memory computing chip 402 is implemented based on the SRAM memory architecture. The main control system 404 and the in-memory computing chip 402 exchange data through the programmable logic unit 405. The main control system 404 and the terminal device 403 can be connected through an HDMI or DP interface. In addition, the voice control system also includes an external memory DDR406, and the external memory 406 exchanges data with the main control system 404 through a memory interface (Memory interfaces). The main control system 404 can store data from the sound collection device 401 or data from the programmable logic unit 405 to the external memory DDR406. The main control system 404 can send the voice data collected from the sound collection device 401 to the programmable logic unit 405. The programmable logic unit 405 can be used to pre-process the voice data, for example, feature extraction and feature conversion processing, and then send the pre-processed data to the in-memory computing chip 402 to perform voice keyword detection processing using a voice keyword detection model. The in-memory computing chip 402 can send the voice keyword detection result calculated by the voice keyword detection model to the main control system 404 via the programmable logic unit 405. The main control system 403 then sends the voice keyword detection result to the terminal device 403, so that the terminal device 403 performs corresponding response operations according to the voice keyword detection result.

[0166] In the disclosed embodiment, the voice keyword detection model can be deployed on an in-memory computing chip based on the SRAM / Flash architecture for computing acceleration, using FPGA or application processor as the main controller to deploy the main control system, and perform data interaction with the sound collection device, the in-memory computing chip and the terminal device, so as to realize functions such as voice control and voice interaction. In the voice control system, the sound collection device, the main control system, and the in-memory computing chip can be integrated into the terminal device. The in-memory computing chip uses voice keyword detection to control whether to wake up or control the terminal device. The standby power of the device is expected to be at the μW level, which can greatly reduce the standby power consumption of the device. Among them, the application processor can use the RK3399 processor, which is a low-power, high-performance application processor chip.

[0167] In some embodiments, the sound collection device can be integrated into the terminal device. In some embodiments, the main control system can be integrated into the terminal device. In some embodiments, the in-memory computing chip can also be integrated into the terminal device.

[0168] In some embodiments, the terminal device can also be an intelligent voice-controlled device, such as household appliances, wearable devices, smart speakers, smart phones, computers, tablets, televisions and other electronic devices with voice control functions, among which smart household appliances with voice control functions are, for example, smart speakers, smart air conditioners, smart refrigerators and other household appliances, and wearable devices are, for example, smart watches, smart glasses, etc.

[0169] In some embodiments, the sound collection device is a device for collecting voice data, such as a microphone array.

[0170] In some application scenarios, the specific process of the voice control system waking up and controlling the terminal device is as follows:

[0171] 0) The main control system is initialized.

[0172] 1) The main control system receives the voice data sent by the sound collection device;

[0173] 2) The main control system sends a wake-up word detection start command to the in-memory computing chip, waits for the in-memory computing chip to respond, and proceeds to the next step after the in-memory computing chip responds and confirms. Otherwise, the command is sent again until the in-memory computing chip responds and confirms.

[0174] 3) The main control system receives the voice keyword detection result from the in-memory computing chip and responds to confirm; if the voice keyword detection result indicates that the corresponding wake-up word is recognized, it jumps to step 4), otherwise it jumps to step 3) and continues to wait for the wake-up word detection result;

[0175] 4) The main control system controls the terminal device to provide wake-up feedback, such as playing the voice "I'm here", "Are you there", "How can I help you", and then waits for further voice keyword detection results from the in-memory computing chip.

[0176] When the terminal device is awakened from the screen-off state for the first time, the screen is also turned on.

[0177] 5) When receiving further voice keyword detection results from the in-memory computing chip, respond and confirm; if the voice keyword detection results indicate that the command word has been recognized, jump to step 6), otherwise jump to step 7).

[0178] 6) The main control system controls the terminal device to execute corresponding instructions, such as "open the video", "turn up the volume, etc.", and jumps to step 5).

[0179] 7) Determine whether the command word has not been recognized before the timeout. If so, jump to step 2); otherwise, jump to step 5).

[0180] Figure 7 A block diagram of another electronic device provided in accordance with an embodiment of the present disclosure.

[0181] Reference Figure 7 An embodiment of the present disclosure provides an electronic device 1000, which includes: at least one processor 1001; at least one memory 1002, and one or more I / O interfaces 1003, connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001, so that the at least one processor 1001 can execute the above-mentioned voice data processing method.

[0182] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned voice data processing method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0183] The embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned voice data processing method.

[0184] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0185] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0186] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0187] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0188] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0189] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0190] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0191] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0192] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0193] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for processing speech data, in, The voice data processing method comprises: Obtaining voice data to be processed; Inputting the speech data to be processed into a programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed; and converting the speech feature vector into a multi-channel input feature map in the programmable logic unit; Inputting the input feature graph into an in-memory computing chip, performing speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtaining a speech keyword detection result; The voice keyword detection result is fed back to the main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.

2. The method for processing speech data according to claim 1, in, The extracting speech features of the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed includes: The speech feature extraction algorithm is used to extract the speech feature of the speech data to be processed, so as to obtain the speech feature vector corresponding to the speech data to be processed.

3. The voice data processing method according to claim 1, in, The speech feature vector is a two-dimensional speech feature vector, and the step of converting the speech feature vector into a multi-channel input feature map includes: Expanding the feature dimension of the speech feature vector to obtain a three-dimensional speech feature vector; Copying the three-dimensional speech feature vector into multiple copies to obtain multiple three-dimensional speech feature vectors; The multiple three-dimensional speech feature vectors are spliced ​​according to the channel dimension to obtain the multi-channel input feature map, where the channel dimension refers to the feature channel of the speech feature vector.

4. The voice data processing method according to claim 1, in, The speech keyword detection model is a model built based on the convolutional hybrid network ConvMixer architecture, and the speech keyword detection model includes a feature sampling layer, a convolutional hybrid network and a classification output layer; The method of performing speech keyword detection on the input feature graph using the speech keyword detection model pre-configured on the in-memory computing chip and obtaining a speech keyword detection result includes: Downsampling the input feature map using the feature sampling layer to obtain a downsampled feature map; Using the convolutional hybrid network to perform convolutional hybrid processing on the downsampled feature map to obtain a speech recognition feature map; The classification output layer is used to perform keyword classification prediction processing on the speech recognition feature graph to obtain the speech keyword detection result.

5. The method for processing speech data according to claim 4, in, The feature sampling layer includes: a feature block embedding layer, a first batch normalization layer and a first activation function layer; Among them, the feature block embedding layer is implemented based on the convolution layer, and the activation function layer adopts the Swish activation function.

6. The method for processing speech data according to claim 4, in, The convolutional hybrid network includes at least one convolutional hybrid network module, and the convolutional hybrid network module includes a spatial position convolutional hybrid unit and a channel position convolutional hybrid unit. The spatial position mixing processing result output by the spatial position convolutional hybrid unit and the input of the spatial position convolutional hybrid unit are input to the channel position convolutional hybrid unit through a residual connection to perform channel position mixing processing; The spatial position convolutional hybrid unit comprises: a depth-separable convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence; The channel position convolution mixing unit comprises: a point-by-point convolution layer, a third batch normalization layer and a third activation function layer connected in sequence; Wherein, the second activation function layer and the third activation function layer both adopt the Swish activation function.

7. The method for processing speech data according to claim 4, in, The classification output layer includes: an average pooling layer and a fully connected layer; Wherein, the fully connected layer is implemented by a convolutional layer.

8. A training method for a speech keyword detection model, in, The training method comprises: Acquire an initial training data set, wherein the initial training data set includes a plurality of initial training speech data and a keyword label corresponding to each initial training speech data; For at least part of the initial training voice data, perform data expansion processing on the initial training voice data by at least one data expansion method to obtain at least one expanded training voice data and a keyword label corresponding to the expanded training voice data; The speech keyword detection model is trained according to the initial training speech data and the expanded training speech data to obtain a trained speech keyword detection model.

9. The training method according to claim 8, in, The at least one data expansion method includes: a data mixing method, a data noise adding method, a negative sample construction method, a speed perturbation method, and a spectrum enhancement method.

10. The training method according to claim 8, in, The voice keyword detection model is trained, including: The speech keyword detection model is trained based on a quantitative perceptual training method.

11. A voice data processing device, in, The speech data processing device comprises: An acquisition unit, configured to acquire voice data to be processed; A feature extraction unit, configured to input the speech data to be processed into the programmable logic unit to extract speech features of the speech data to be processed, so as to obtain a speech feature vector corresponding to the speech data to be processed; A feature conversion unit, configured to convert the speech feature vector into a multi-channel input feature map in the programmable logic unit; A detection unit, configured to input the input feature graph into an in-memory computing chip, perform speech keyword detection on the input feature graph using a speech keyword detection model pre-configured on the in-memory computing chip, and obtain a speech keyword detection result; A response unit is configured to feed back the voice keyword detection result to a main control system, and the main control system is used to perform a response operation corresponding to the voice keyword detection result according to the voice keyword detection result.

12. A voice control system, in, include: A sound collection device is used to collect the voice data to be processed in the environment and send the voice data to be processed to the main control system; A main control system, used for receiving the voice data to be processed collected by the sound collection device, and sending the voice data to be processed to the programmable logic unit; A programmable logic unit, configured to perform speech feature extraction on the speech data to be processed to obtain a speech feature vector corresponding to the speech data to be processed, and convert the speech feature vector into a multi-channel input feature map; An in-memory computing chip having a preset voice keyword detection model integrated thereon, for performing voice keyword detection on the input feature graph using the voice keyword detection model pre-configured on the in-memory computing chip, and obtaining a voice keyword detection result; The main control system is further used to receive the voice keyword detection result of the in-memory computing chip, and control the terminal device to perform a corresponding response operation according to the voice keyword detection result; The terminal device is used to execute a response operation corresponding to the voice keyword detection result.

13. An electronic device, in, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the speech data processing method as described in any one of claims 1 to 7 or execute the training method of the speech keyword detection model as described in claim 8.

14. A computer-readable storage medium having a computer program stored thereon, in, When the computer program is executed by a processor, the computer program implements the speech data processing method according to any one of claims 1 to 7 or executes the training method of the speech keyword detection model according to claim 8.

Citation Information

Cited By

  • Voice wake-up method, electronic device, storage medium and computer program product

    CN120673760A

  • Voice wake-up methods, electronic devices, storage media, and computer program products

    CN120673760B