Voice noise reduction recognition method and device in machine room environment

Abstract

The invention discloses a voice noise reduction recognition method and device in a machine room environment, and the method comprises the steps: obtaining noise data in a plurality of types of machine rooms, and carrying out the first preprocessing, and obtaining a noise power spectrum; collecting audio data in the machine room and carrying out second preprocessing to obtain an audio power spectrum; based on the noise power spectrum, performing noise reduction processing on the audio power spectrum to obtain voice data; and based on a pre-constructed speech acceleration recognition model, performing recognition processing on the speech data, and giving text information. According to the invention, the voice command can be quickly and accurately recognized, so that a corresponding response is made.

Description

Technical Field

[0001] The invention belongs to the technical field of computer room speech recognition, and in particular relates to a speech noise reduction recognition method and device in a computer room environment. Background Art

[0002] As the scale of data centers gradually expands, the number of devices in the computer room continues to increase, and the efficiency and accuracy of manual inspections are gradually facing challenges. In order to solve this problem, more and more data centers are beginning to use computer room inspection robots for automatic inspections.

[0003] For example, patent CN112735013A discloses a computer room environment voice comprehensive query alarm system, including an authentication module for identifying the face and / or voiceprint features of people entering the computer room, and optimizing and recording the operations in real time; a voice interaction module for realizing the voice interaction control function of the staff to check and retrieve the monitoring data in the computer room in real time, and to schedule the video monitoring screen, and for voice broadcasting important alarm signals and changes in important data; an intelligent controller for providing Internet connection and Web service capabilities, which has the functions of integrated control, monitoring, data recording, alarm, calendar and network management; and a monitoring management module for managing the power subsystem, environmental subsystem and security subsystem; it can realize voice interaction between people and the computer room, simplify the work process, understand the situation in the computer room at the first time, and ensure the normal operation of various equipment in the computer room.

[0004] For another example, patent CN114043492A discloses a voice recognition room inspection robot, comprising a main body; the main body is provided with a propeller and a detector, and the main body is provided with a voice player and a voice recognition system, and a circle of sealed protective bag is provided on the side wall of the main body; by arranging a circle of protective bag on the side of the robot, when the robot approaches the equipment, the protective bag can collide and fit with the equipment in advance, because the inside of the protective bag is flowing gas, and its side wall is a wave shape that can be freely expanded and contracted, so that the protective bag can maintain sufficient elasticity to cushion the impact of the robot and reduce the risk of the main body crashing. At the same time, one side of the protective bag can rebound after compression, so that the robot can move to the side away from the equipment, thereby effectively keeping it away from obstacles and reducing repeated collisions between the robot and obstacles.

[0005] However, the computer room environment is complex and noisy, which places higher demands on the robot's speech recognition capabilities. As speech recognition technology itself is constantly developing and improving, traditional speech recognition methods may not be able to adapt well to the noise and complexity of the computer room environment.

[0006] Therefore, how to provide a speech recognition algorithm and technology that can improve recognition accuracy and speed is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0007] In view of the defects existing in the above-mentioned prior art, the present invention provides a method and device for speech noise reduction recognition in a computer room environment, which can quickly and accurately recognize speech commands and make corresponding responses.

[0008] In a first aspect, the present invention provides a method for speech noise reduction recognition in a computer room environment, comprising:

[0009] Acquire noise data in multiple types of computer rooms and perform first preprocessing to obtain noise power spectrum;

[0010] Collecting audio data in the computer room and performing a second preprocessing to obtain an audio power spectrum;

[0011] Based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data;

[0012] Based on the pre-built speech acceleration recognition model, the speech data is recognized and processed to give text information.

[0013] Furthermore, noise data in multiple types of computer rooms are obtained and preprocessed to obtain a noise power spectrum, including:

[0014] Acquire noise data in multiple types of computer rooms, and divide the noise data into a number of first short time frames;

[0015] Performing windowing processing on a plurality of first short-time frames to obtain a plurality of frame noise signals;

[0016] All frame noise signals are processed based on short-time Fourier transform to obtain noise complex spectrum values;

[0017] The noise power spectrum is obtained by averaging the complex spectrum values ​​of the noise.

[0018] Furthermore, the complex spectrum value of the noise satisfies the following relationship:

[0019]

[0020] w(n)=α1-β1cos(2πn / (N-1))

[0021] Where X(m,k) is the noise complex spectrum value of the kth noise frequency component of the mth frame, N is the frame length of the mth frame noise signal, x(n+mH) is the sample of the mth frame taken from the noise signal x(n), mH is the starting position of the mth frame, m is the frame index of the noise signal, H is the frame shift of the noise signal, j is the imaginary unit, w(n) is the window function of the noise signal, k is the index of the noise frequency component, n is the window length, and α1 and β1 are both constants;

[0022] The noise power spectrum satisfies the following relationship:

[0023]

[0024] Where P(k) is the noise power spectrum and M is the total number of frames of all frame noise signals.

[0025] Furthermore, the audio data in the computer room is collected and subjected to a second preprocessing to obtain an audio power spectrum, including:

[0026] Collecting audio data in the computer room and dividing it into a number of second short time frames;

[0027] Performing windowing processing on the plurality of second short-time frames to obtain a plurality of frames of audio signals;

[0028] All frame audio signals are processed based on short-time Fourier transform to obtain audio complex spectrum values;

[0029] The audio complex spectrum values ​​are averaged to obtain the audio power spectrum.

[0030] Furthermore, the complex spectrum value of the audio satisfies the following relationship:

[0031]

[0032] w'(n')=α2-β2cos(2πn' / (N'-1))

[0033] Wherein, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame, N' is the frame length of the m'th frame audio signal, s(n'+m'H') is the sample of the m'th frame taken from the audio signal s(n'), m'H' is the starting position of the m'th frame, m' is the frame index of the audio signal, H' is the frame shift of the audio signal, j is the imaginary unit, w'(n') is the window function of the audio signal, k' is the index of the audio frequency component, n' is the window length, and α2 and β2 are both constants;

[0034] The audio power spectrum satisfies the following relationship:

[0035] D(m',k')=|S(m',k')| 2

[0036] Where D(m',k') is the audio power spectrum.

[0037] Furthermore, based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data, including:

[0038] Extract the corresponding real and imaginary parts based on the audio complex spectrum value, and give the audio phase angle;

[0039] Comparing the audio power spectrum with the noise power spectrum, and determining the speech amplitude of the speech data by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum;

[0040] The speech amplitude and the audio phase angle are superimposed and inverse Fourier transformed to obtain speech data.

[0041] Furthermore, the audio phase angle satisfies the following relationship:

[0042]

[0043] In the formula, S angle is the audio phase angle, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame;

[0044] The audio power spectrum is compared with the noise power spectrum, and the voice amplitude of the voice data is determined by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum, which is specifically expressed as:

[0045]

[0046] In the formula, is the speech amplitude of the speech data, a is the over-reduction factor, and b is the gain compensation factor, all of which are constants.

[0047] Furthermore, the pre-built speech acceleration recognition model includes:

[0048] The speech recognition model is trained based on the historical speech data to obtain a trained speech recognition model;

[0049] Quantize the trained speech recognition model to obtain a speech recognition model in ONNX format;

[0050] The speech recognition model in the ONNX format is trimmed to obtain a trimmed speech recognition model in the ONNX format;

[0051] The trimmed ONNX-format speech recognition model is loaded into the ONNX-format inference framework to obtain the final speech acceleration recognition model.

[0052] Furthermore, the trained speech recognition model is quantized to obtain a speech recognition model in ONNX format, including:

[0053] Select the ONNX inference engine that runs the speech recognition model in ONNX format;

[0054] Based on the ONNX inference engine, select the ONNX operator set that matches the speech acceleration recognition model;

[0055] Based on the ONNX operator set, the speech acceleration recognition model is converted into a format to construct an intermediate recognition model in the ONNX format.

[0056] The model quantization module based on the ONNX inference engine performs statistical processing on the weight data of the intermediate state recognition model in ONNX format and gives the minimum and maximum values;

[0057] The minimum and maximum values ​​are processed by asymmetric quantization, and the quantization ratio and critical value are given;

[0058] The weight data is converted by quantization ratio and critical value to obtain weight data converted from floating point numbers to integers;

[0059] The weight data converted into integers is used to replace the weight data of the intermediate state recognition model in ONNX format to obtain a speech recognition model in ONNX format.

[0060] Furthermore, based on the pre-built speech acceleration recognition model, the speech data is recognized and processed to provide text information, including:

[0061] Input the speech data into the pre-built speech acceleration recognition model;

[0062] Based on the speech data, the reasoning framework in ONNX format infers the trimmed speech recognition model in ONNX format and outputs text information in combination with the prior dictionary.

[0063] In a second aspect, the present invention further provides a speech noise reduction recognition device in a computer room environment, using the above-mentioned speech noise reduction recognition method in a computer room environment, the speech noise reduction recognition device comprises:

[0064] A noise processing module, which is used to obtain noise data in multiple types of computer rooms and perform a first preprocessing to obtain a noise power spectrum;

[0065] An audio processing module, which is used to collect audio data in the computer room and perform a second preprocessing to obtain an audio power spectrum;

[0066] A noise reduction processing module, which is used to perform noise reduction processing on the audio power spectrum based on the noise power spectrum to obtain speech data;

[0067] The speech recognition module is used to recognize and process speech data based on a pre-built speech acceleration recognition model and provide text information.

[0068] Furthermore, the noise processing module also includes:

[0069] Acquire noise data in multiple types of computer rooms, and divide the noise data into a number of first short time frames;

[0070] Performing windowing processing on a plurality of first short-time frames to obtain a plurality of frame noise signals;

[0071] All frame noise signals are processed based on short-time Fourier transform to obtain noise complex spectrum values;

[0072] The noise power spectrum is obtained by averaging the complex spectrum values ​​of the noise.

[0073] Furthermore, the audio processing module also includes:

[0074] Collecting audio data in the computer room and dividing it into a number of second short time frames;

[0075] Performing windowing processing on the plurality of second short-time frames to obtain a plurality of frames of audio signals;

[0076] All frame audio signals are processed based on short-time Fourier transform to obtain audio complex spectrum values;

[0077] The audio complex spectrum values ​​are averaged to obtain the audio power spectrum.

[0078] Furthermore, the noise reduction processing module also includes:

[0079] Extract the corresponding real and imaginary parts based on the audio complex spectrum value, and give the audio phase angle;

[0080] Comparing the audio power spectrum with the noise power spectrum, and determining the speech amplitude of the speech data by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum;

[0081] The speech amplitude and the audio phase angle are superimposed and inverse Fourier transformed to obtain speech data.

[0082] Furthermore, the speech noise reduction recognition device also includes a model building module, which includes:

[0083] The speech recognition model is trained based on the historical speech data to obtain a trained speech recognition model;

[0084] Quantize the trained speech recognition model to obtain a speech recognition model in ONNX format;

[0085] The speech recognition model in the ONNX format is trimmed to obtain a trimmed speech recognition model in the ONNX format;

[0086] The trimmed ONNX-format speech recognition model is loaded into the ONNX-format inference framework to obtain the final speech acceleration recognition model.

[0087] Furthermore, the model building module also includes:

[0088] Select the ONNX inference engine that runs the speech recognition model in ONNX format;

[0089] Based on the ONNX inference engine, select the ONNX operator set that matches the speech acceleration recognition model;

[0090] Based on the ONNX operator set, the speech acceleration recognition model is converted into a format to construct an intermediate recognition model in the ONNX format.

[0091] The model quantization module based on the ONNX inference engine performs statistical processing on the weight data of the intermediate state recognition model in ONNX format and gives the minimum and maximum values;

[0092] The minimum and maximum values ​​are processed by asymmetric quantization, and the quantization ratio and critical value are given;

[0093] The weight data is converted by quantization ratio and critical value to obtain weight data converted from floating point numbers to integers;

[0094] The weight data converted into integers is used to replace the weight data of the intermediate state recognition model in ONNX format to obtain a speech recognition model in ONNX format.

[0095] The present invention provides a method and device for speech noise reduction recognition in a computer room environment, which has at least the following beneficial effects:

[0096] (1) The present invention can improve the accuracy of voice command recognition by denoising the audio data in the computer room. In addition, the voice command can be recognized quickly and accurately through the pre-built voice acceleration recognition model.

[0097] (2) By adopting a speech recognition model in the ONNX format and trimming the speech recognition model in the ONNX format, the model size can be minimized while ensuring that the model accuracy is not significantly affected; that is, the present invention enables the speech recognition model to achieve efficient reasoning execution in a cross-platform and cross-framework environment while having good recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1A flow chart of a method for speech noise reduction and recognition in a computer room environment provided by the present invention;

[0099] Figure 2 A flowchart for obtaining a noise power spectrum provided by an embodiment of the present invention;

[0100] Figure 3 A flowchart for obtaining an audio power spectrum provided by an embodiment of the present invention;

[0101] Figure 4 A flowchart of obtaining voice data provided by an embodiment of the present invention;

[0102] Figure 5 A flowchart of constructing a speech acceleration recognition model provided by an embodiment of the present invention;

[0103] Figure 6 A flowchart of obtaining a speech recognition model in ONNX format provided by an embodiment of the present invention;

[0104] Figure 7 A power spectrum density diagram of original audio data with noise provided by an embodiment of the present invention;

[0105] Figure 8 A power spectrum density diagram of the original audio data after noise reduction provided by an embodiment of the present invention;

[0106] Fig. 9 A Mel-frequency cepstrum of original audio data with noise provided by an embodiment of the present invention;

[0107] Fig.10 A Mel-frequency cepstrum of the original audio data after noise reduction provided by an embodiment of the present invention;

[0108] Fig.11 A schematic diagram of a speech noise reduction recognition device in a computer room environment provided by the present invention. DETAILED DESCRIPTION

[0109] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0110] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0111] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0112] The present invention is based on the problem that traditional speech recognition methods cannot adapt well to the noise and complexity of the computer room environment. It adopts a speech recognition model based on deep learning and improves the recognition ability and robustness of the model by training a large amount of data. In addition, the speech recognition of the computer room inspection robot also needs to consider real-time and stability. Since the computer room inspection requires real-time response and accurate judgment, the robot's speech recognition system needs to be able to quickly and accurately recognize voice commands and respond accordingly. At the same time, since the computer room environment may change, such as changes in noise levels, changes in equipment locations, etc., the robot's speech recognition system also needs to have certain adaptability and stability to cope with these changes.

[0113] Based on the above situation, if Figure 1 As shown, the present invention provides a method for speech noise reduction recognition in a computer room environment, which may include:

[0114] Acquire noise data in multiple types of computer rooms and perform first preprocessing to obtain noise power spectrum;

[0115] Collecting audio data in the computer room and performing a second preprocessing to obtain an audio power spectrum;

[0116] Based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data;

[0117] Based on the pre-built speech acceleration recognition model, the speech data is recognized and processed to give text information.

[0118] Among them, Figure 2 As shown, noise data in multiple types of computer rooms are obtained and first preprocessed to obtain noise power spectra, including:

[0119] Acquire noise data in multiple types of computer rooms, and divide the noise data into a number of first short time frames;

[0120] Performing windowing processing on a plurality of first short-time frames to obtain a plurality of frame noise signals;

[0121] All frame noise signals are processed based on short-time Fourier transform to obtain noise complex spectrum values;

[0122] The noise power spectrum is obtained by averaging the complex spectrum values ​​of the noise.

[0123] In actual application scenarios, a six-microphone ring is used to collect noise data in various types of computer rooms for the first preprocessing. The noise data is divided into several short-time frames and windowed to calculate the short-time Fourier transform of the noise and the power spectrum of the noise. The Fourier transform of a discrete noise time series x(i) is the frequency domain signal X(k). Let the time series of the noise data be x(i). After windowing and framing, the mth frame of the speech signal is x(n) = x(0), x(1), ... x(N-1), with a frame length of N. The corresponding complex spectrum value of the noise satisfies the following relationship:

[0124]

[0125] w(n) is a window function of length N, which is used to smoothly intercept the signal frame and satisfies the following relationship:

[0126] w(n)=α1-β1cos(2πn / (N-1))

[0127] Where X(m,k) is the noise complex spectrum value of the kth noise frequency component in the mth frame, N is the frame length of the noise signal in the mth frame, x(n+mH) is the sample of the mth frame taken from the noise signal x(n), mH is the starting position of the mth frame, m is the frame index of the noise signal, H is the frame shift of the noise signal, j is the imaginary unit, w(n) is the window function of the noise signal, and k is the index of the audio frequency component, which is an integer and increases from 0. In Fourier transform, k is used to identify different noise frequency components, n is the window length, also known as the number of points of FFT, which determines the resolution of the spectrum. A larger n will provide higher frequency resolution, but it will also increase the computational complexity. n is usually a power of 2, which can be efficiently implemented using the fast Fourier transform (FFT) algorithm. α1 and β1 are both constants, with values ​​of 0.54 and 0.46 respectively. Among them, the short-time Fourier transform is to localize the signal in time through a sliding window, and then perform FFT on each local signal segment. In addition, the noise frequency component is the representation of the noise signal in the frequency domain, which describes the intensity and phase of the noise signal at each frequency component. Each noise frequency component X(m,k) represents the complex value of the noise signal at a specific time frame m and frequency k. This complex value can be decomposed into amplitude and phase. The amplitude represents the energy or intensity of the noise frequency component; in audio processing, the amplitude spectrum is used to show the relative intensity of different frequency components. The phase represents the time offset of the noise frequency component; phase information is very important for reconstructing the original signal and contains the temporal structure of the signal. The index k of the noise frequency component corresponds to a specific frequency value f k ; The specific frequency value can be obtained through the formula f k =k@f s Calculate, f s is the sampling rate.

[0128] Among them, x(n+mH)w(n) is actually a windowing process performed on the signal, that is, only the signal part within the coverage range of the window function is retained, and the signal part outside the range of the window function is set to zero.

[0129] For a certain frame, the noise power spectrum is represented by the average value of the noise power spectrum, which satisfies the following relationship:

[0130]

[0131] Where P(k) is the noise power spectrum and M is the total number of frames of all frame noise signals.

[0132] After obtaining the noise power spectrum, the noise can be removed during the subsequent processing of the audio data in the computer room. Figure 3 As shown, collecting audio data in the room and performing a second preprocessing to obtain an audio power spectrum may include:

[0133] Collecting audio data in the computer room and dividing it into a number of second short time frames;

[0134] Performing windowing processing on the plurality of second short-time frames to obtain a plurality of frames of audio signals;

[0135] All frame audio signals are processed based on short-time Fourier transform to obtain audio complex spectrum values;

[0136] The audio complex spectrum values ​​are averaged to obtain the audio power spectrum.

[0137] In actual application scenarios, a six-microphone ring is used to collect audio data for the second preprocessing. The audio data is divided into several short-time frames and windowed to calculate the short-time Fourier transform of the audio and the power spectrum of the audio. The calculation method of the audio complex spectrum value is similar to the calculation method of the noise power spectrum. Specifically, the audio complex spectrum value satisfies the following relationship:

[0138]

[0139] w'(n')=α2-β2cos(2πn' / (N'-1))

[0140] Wherein, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame, N' is the frame length of the m'th frame audio signal, s(n'+m'H') is the sample of the m'th frame taken out from the audio signal s(n'), m'H' is the starting position of the m'th frame, m' is the frame index of the audio signal, H' is the frame shift of the audio signal, j is the imaginary unit, w'(n') is the window function of the audio signal, k' is the index of the audio frequency component, n' is the window length, α2 and β2 are constants, and the values ​​are 0.54 and 0.46 respectively; the audio frequency component and the noise frequency component have the same meaning, the only difference is that they are targeted at different objects, and n' can be equal to n;

[0141] The discrete speech signal s(n) is Fourier transformed and recorded as: S(m',k'), then the audio power spectrum satisfies the following relationship:

[0142] D(m',k')=|S(m',k')| 2

[0143] Where D(m',k') is the audio power spectrum.

[0144] like Figure 4 As shown, after obtaining the audio power spectrum, based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data, which may include:

[0145] Extract the corresponding real and imaginary parts based on the audio complex spectrum value, and give the audio phase angle;

[0146] Comparing the audio power spectrum with the noise power spectrum, and determining the speech amplitude of the speech data by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum;

[0147] The speech amplitude and the audio phase angle are superimposed and inverse Fourier transformed to obtain speech data.

[0148] Among them, the audio phase angle satisfies the following relationship:

[0149]

[0150] In the formula, S angle is the audio phase angle, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame;

[0151] After obtaining the audio phase angle, the frequency spectrum subtraction method is used to obtain pure speech data; when the frequency spectrum subtraction method is used, the speech amplitude of the speech data also needs to be obtained, that is, the audio power spectrum is compared with the noise power spectrum, and the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum is combined to determine the speech amplitude of the speech data, which is specifically expressed as:

[0152]

[0153] In the formula, is the speech amplitude of the speech data, a is the over-reduction factor, b is the gain compensation factor, and both a and b are constants.

[0154] When realizing the recognition of voice data, it is necessary to perform recognition processing on it through the pre-built voice acceleration recognition model to obtain the corresponding text information, and finally obtain the corresponding voice command through the text information. Specifically, Figure 5 As shown, the pre-built speech acceleration recognition model may include:

[0155] The speech recognition model is trained based on the historical speech data to obtain a trained speech recognition model;

[0156] Quantize the trained speech recognition model to obtain a speech recognition model in ONNX format;

[0157] The speech recognition model in the ONNX format is trimmed to obtain a trimmed speech recognition model in the ONNX format;

[0158] The trimmed ONNX-format speech recognition model is loaded into the ONNX-format inference framework to obtain the final speech acceleration recognition model.

[0159] The historical speech data is obtained by denoising the historical audio data using the denoising method of the present invention, or it may be pure speech data (i.e., audio data without noise). The speech recognition model when training with the historical speech data is a Paraformer model; at the same time, in order to improve the recognition ability and robustness of the model, the historical speech data for training the speech recognition model may be increased.

[0160] like Figure 6 As shown, after the training of the speech recognition model is completed, the trained speech recognition model can be quantized to obtain a speech recognition model in ONNX format, which may include:

[0161] Select the ONNX inference engine that runs the speech recognition model in ONNX format;

[0162] Based on the ONNX inference engine, select the ONNX operator set that matches the speech acceleration recognition model;

[0163] Based on the ONNX operator set, the speech acceleration recognition model is converted into a format to construct an intermediate recognition model in the ONNX format.

[0164] The model quantization module based on the ONNX inference engine performs statistical processing on the weight data of the intermediate state recognition model in ONNX format and gives the minimum and maximum values;

[0165] The minimum and maximum values ​​are processed by asymmetric quantization, and the quantization ratio and critical value are given;

[0166] The weight data is converted by quantization ratio and critical value to obtain weight data converted from floating point numbers to integers;

[0167] The weight data converted into integers is used to replace the weight data of the intermediate state recognition model in ONNX format to obtain a speech recognition model in ONNX format.

[0168] In actual application scenarios, the trained Paraformer model (speech acceleration recognition model) can be exported to ONNX format using the interface provided by the deep learning framework (PyTorch). This process requires the selection of a suitable ONNX operator set version to ensure that all model operations can be supported by ONNX Runtime. In order to further reduce the model size and computational complexity, the model can be quantized during the export process. Specifically, the quantization tool suitable for the ONNX model (the Quantization Toolkit (model quantization module) provided by ONNX Runtime) is used to perform 8-bit integer quantization on the model channel weights and bias values. The exported speech acceleration recognition model in ONNX format can achieve efficient reasoning execution in a cross-platform and cross-framework environment.

[0169] Among them, the model channel weights and bias values ​​are quantized to 8-bit integers, including:

[0170] Count the min_value and max_value of the input data: First, count the corresponding minimum value (min_value) and maximum value (max_value) in the model weight data;

[0171] Choose the appropriate quantization type: asymmetric quantization (such as uint8), asymmetric quantization uses the actual minimum value as the zero point;

[0172] Calculate quantization parameters: Calculate the quantization weight according to the quantization type, min_value and max_value, which usually includes the critical value and the quantization ratio;

[0173] Among them, the quantized ratio satisfies the following relationship:

[0174] scale=(data_range_max-data_range_min) / (quantization_range_max-quantization_range_min)

[0175] Where scale is the quantization ratio, data_range_max and data_range_min are the maximum and minimum values ​​of the weight data, quantization_range_max and quantization_range_min are the maximum and minimum values ​​of the asymmetric quantization range, respectively;

[0176] The critical value satisfies the following relationship:

[0177] zero_point=quantization_range_min-data_range_min / scale

[0178] In the formula, zero_point is the critical value;

[0179] Perform quantization: Perform quantization on the model based on the weight data, that is, convert it from floating point numbers to integers. In this step, the weights in the model are converted according to the quantization parameters calculated previously, satisfying the following relationship:

[0180] weight_quantized=round(weight_fp32 / scale-zero_point)

[0181] In the formula, weight_quantized is the quantized weight, and weight_fp32 is the 32-bit floating-point representation of the model weight. The model weight can be converted from FP32 (or FP64) to a lower precision representation, such as INT8 (8-bit integer), to reduce the model size and improve the inference speed, especially on resource-constrained devices;

[0182] Verify the performance of the quantized model: After the quantization is completed, the performance of the quantized model needs to be verified. If the performance is poor, then re-perform the quantization operation.

[0183] After the model is quantized, the model can also be pruned to remove unnecessary nodes and weights, minimize the model size, and ensure that the model accuracy is not significantly affected. Specifically, the speech recognition model in ONNX format is pruned to obtain a pruned speech recognition model in ONNX format. The model pruning tool onnx_simplifier can be used to perform unstructured fine pruning on the Paraformer model (speech recognition model in ONNX format) to remove unnecessary nodes and weights, minimize the model size, and ensure that the model accuracy is not significantly affected.

[0184] In actual application scenarios, you can use the pip command to install onnx-simplifier, import the onnx library, and use the onnx.load function to load the speech recognition model in ONNX format to be simplified.

[0185] Next, import the simplify function in the onnxsim library, traverse the nodes in the model graph corresponding to the speech recognition model in ONNX format, check the operation type (op_type) and input and output of each node, determine whether there are obvious unnecessary operations, and perform unstructured fine pruning. Unnecessary operations include useless operations (the result of the operation has never been used, or its output is exactly the same as the input) and repeated operations. Pruning rules: Identically connect useless operations and nodes with zero weight; delete consecutive repeated operations;

[0186] After simplification, validation is required to ensure the correctness of the model. This can be done by comparing the output or performance of the model before and after simplification.

[0187] During the simplification process, the check variable returns a Boolean value, indicating whether the simplified model is successfully verified.

[0188] After simplification, use the onnx.save function of the ONNX library to save the simplified ONNX format speech recognition model to disk and load it into the ONNX format reasoning framework.

[0189] After obtaining the final speech acceleration recognition model, the speech data can be recognized and processed based on the pre-built speech acceleration recognition model to provide text information, including:

[0190] Input the speech data into the pre-built speech acceleration recognition model;

[0191] Based on the speech data, the reasoning framework in ONNX format infers the trimmed speech recognition model in ONNX format and outputs text information in combination with the prior dictionary.

[0192] In actual application scenarios, when voice data is recognized and processed to give text information, ONNX Runtime is used as the inference engine to load and infer Paraformer.onnx. ONNX Runtime can automatically optimize the order of operations and use the underlying hardware acceleration to improve the inference speed. Specifically, first, install ONNX Runtime in the development environment. For the Python environment, use the pip command to install it. Then use the InferenceSession of ONNX Runtime to load the cropped ONNX format speech recognition model and specify the path of the model file. At the same time, according to the hardware configuration, reasonably set the threads in the ONNX Runtime inference process. First, determine the number of available cores and assign them to the environment variable to control the total number of threads used in the overall inference process of the model. Then convert the denoised voice into voice features as input, run the inference by calling the session.run method, process the inference results, retrieve text (text information) according to the prior dictionary and the output of the model, and generate corresponding voice instructions according to the retrieved text. Using ONNX Runtime can automatically optimize the order of operations and use the underlying hardware acceleration to improve the inference speed.

[0193] In actual application scenarios, such as Figure 7 As shown, it represents the power spectrum density diagram of the original audio data with noise, such as Figure 8 As shown, it shows the power spectrum density diagram after the original audio data is denoised by the speech denoising recognition method of the present invention. Figure 7 and Figure 8 It can be seen that the power spectrum density diagram after denoising the original audio data shows the following significant changes:

[0194] 1. Overall reduction: The overall power level of the power spectral density graph after denoising is significantly lower than that of the original audio data. This indicates that the denoising process effectively reduces the noise level.

[0195] 2. Peak-to-valley changes: The peaks in the original audio data become smoother after noise reduction, indicating that some sharp noise components are removed. At the same time, the valley values ​​are also improved, which means that more audio details are retained.

[0196] 3. Noise Residual: The noise is suppressed overall, but some residual noise can still be seen at certain frequency points. This shows that the noise reduction algorithm does not completely eliminate all noise, but balances the relationship between noise suppression and signal fidelity as much as possible.

[0197] 4. Improved signal-to-noise ratio: The signal-to-noise ratio of the denoised audio data is higher than that of the original audio data.

[0198] like Fig. 9 and Fig.10 As shown in the figure, the original audio data and the audio data after noise reduction by the speech noise reduction recognition method of the present invention are represented respectively. Specifically, the original audio data and the audio data after noise reduction are represented by the Mel frequency cepstrum. Fig. 9 and Fig.10 A comparative analysis shows that:

[0199] 1. Noise reduction: The denoised audio data has lower brightness at more time-frequency points, indicating that the noise has been successfully reduced. In particular, in some frequency bands, the originally brighter parts become darker.

[0200] 2. Speech fidelity: Despite the noise reduction process, the main audio features can still be recognized, indicating that the noise reduction algorithm does not overly affect the audio data itself.

[0201] 3. Frequency band selectivity: The noise reduction effect is more obvious in the low-frequency segment.

[0202] 4. Inter-frame consistency: The denoised audio data is more consistent between frames, which may improve the accuracy of speech recognition.

[0203] 5. Dynamic range: The dynamic range of the image after noise reduction is reduced, that is, the difference between the maximum and minimum brightness is smaller, indicating that the noise is compressed to a lower level.

[0204] 6. Improved signal-to-noise ratio: The signal-to-noise ratio of the denoised audio is higher than that of the original audio.

[0205] 7. Spectral structure: The spectral structure of the image after noise reduction is clearer, which is conducive to subsequent speech processing and recognition.

[0206] like Fig.11 As shown, the present invention also provides a speech noise reduction recognition device in a computer room environment, using the above-mentioned speech noise reduction recognition method in a computer room environment, the speech noise reduction recognition device comprises:

[0207] A noise processing module, which is used to obtain noise data in multiple types of computer rooms and perform a first preprocessing to obtain a noise power spectrum;

[0208] An audio processing module, which is used to collect audio data in the computer room and perform a second preprocessing to obtain an audio power spectrum;

[0209] A noise reduction processing module, which is used to perform noise reduction processing on the audio power spectrum based on the noise power spectrum to obtain speech data;

[0210] The speech recognition module is used to recognize and process speech data based on a pre-built speech acceleration recognition model and provide text information.

[0211] Furthermore, the noise processing module also includes:

[0212] Acquire noise data in multiple types of computer rooms, and divide the noise data into a number of first short time frames;

[0213] Performing windowing processing on a plurality of first short-time frames to obtain a plurality of frame noise signals;

[0214] All frame noise signals are processed based on short-time Fourier transform to obtain noise complex spectrum values;

[0215] The noise complex spectrum values ​​are averaged to obtain the noise power spectrum.

[0216] Furthermore, the audio processing module also includes:

[0217] Collecting audio data in the computer room and dividing it into a number of second short time frames;

[0218] Performing windowing processing on the plurality of second short-time frames to obtain a plurality of frames of audio signals;

[0219] All frame audio signals are processed based on short-time Fourier transform to obtain audio complex spectrum values;

[0220] The audio complex spectrum values ​​are averaged to obtain the audio power spectrum.

[0221] Furthermore, the noise reduction processing module also includes:

[0222] Extract the corresponding real and imaginary parts based on the audio complex spectrum value, and give the audio phase angle;

[0223] Comparing the audio power spectrum with the noise power spectrum, and determining the speech amplitude of the speech data by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum;

[0224] The speech amplitude and the audio phase angle are superimposed and inverse Fourier transformed to obtain speech data.

[0225] Furthermore, the speech noise reduction recognition device also includes a model building module, which includes:

[0226] The speech recognition model is trained based on the historical speech data to obtain a trained speech recognition model;

[0227] Quantize the trained speech recognition model to obtain a speech recognition model in ONNX format;

[0228] The speech recognition model in the ONNX format is trimmed to obtain a trimmed speech recognition model in the ONNX format;

[0229] The trimmed ONNX-format speech recognition model is loaded into the ONNX-format inference framework to obtain the final speech acceleration recognition model.

[0230] Furthermore, the model building module also includes:

[0231] Select the ONNX inference engine that runs the speech recognition model in ONNX format;

[0232] Based on the ONNX inference engine, select the ONNX operator set that matches the speech acceleration recognition model;

[0233] Based on the ONNX operator set, the speech acceleration recognition model is converted into a format to construct an intermediate recognition model in the ONNX format.

[0234] The model quantization module based on the ONNX inference engine performs statistical processing on the weight data of the intermediate state recognition model in ONNX format and gives the minimum and maximum values;

[0235] The minimum and maximum values ​​are processed by asymmetric quantization, and the quantization ratio and critical value are given;

[0236] The weight data is converted by quantization ratio and critical value to obtain weight data converted from floating point numbers to integers;

[0237] The weight data converted into integers is used to replace the weight data of the intermediate state recognition model in ONNX format to obtain a speech recognition model in ONNX format.

[0238] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for speech noise reduction recognition in a computer room environment, characterized in that: include: Acquire noise data in multiple types of computer rooms and perform first preprocessing to obtain noise power spectrum; Collecting audio data in the computer room and performing a second preprocessing to obtain an audio power spectrum; Based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data; Based on the pre-built speech acceleration recognition model, the speech data is recognized and processed to give text information.

2. The speech noise reduction recognition method according to claim 1, characterized in that: Acquire noise data in multiple types of computer rooms and perform first preprocessing to obtain noise power spectra, including: Acquire noise data in multiple types of computer rooms, and divide the noise data into a number of first short time frames; Performing windowing processing on a plurality of first short-time frames to obtain a plurality of frame noise signals; All frame noise signals are processed based on short-time Fourier transform to obtain noise complex spectrum values; The noise complex spectrum values ​​are averaged to obtain the noise power spectrum.

3. The speech noise reduction recognition method according to claim 2, characterized in that: The complex spectrum value of the noise satisfies the following relationship: w(n)=α1-β1cos(2πn / (N-1)) Where X(m,k) is the noise complex spectrum value of the kth noise frequency component of the mth frame, N is the frame length of the mth frame noise signal, x(n+mH) is the sample of the mth frame taken from the noise signal x(n), mH is the starting position of the mth frame, m is the frame index of the noise signal, H is the frame shift of the noise signal, j is the imaginary unit, w(n) is the window function of the noise signal, k is the index of the noise frequency component, n is the window length, and α1 and β1 are both constants; The noise power spectrum satisfies the following relationship: Where P(k) is the noise power spectrum and M is the total number of frames of all frame noise signals.

4. The speech noise reduction recognition method according to claim 1, characterized in that: The audio data in the computer room is collected and preprocessed to obtain the audio power spectrum, including: Collecting audio data in the computer room and dividing it into a number of second short time frames; Performing windowing processing on the plurality of second short-time frames to obtain a plurality of frames of audio signals; All frame audio signals are processed based on short-time Fourier transform to obtain audio complex spectrum values; The audio complex spectrum values ​​are averaged to obtain the audio power spectrum.

5. The speech noise reduction recognition method according to claim 4, characterized in that: The audio complex spectrum value satisfies the following relationship: w'(n')=α2-β2cos(2πn' / (N'-1)) Wherein, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame, N' is the frame length of the m'th frame audio signal, s(n'+m'H') is the sample of the m'th frame taken from the audio signal s(n'), m'H' is the starting position of the m'th frame, m' is the frame index of the audio signal, H' is the frame shift of the audio signal, j is the imaginary unit, w'(n') is the window function of the audio signal, k' is the index of the audio frequency component, n' is the window length, and α2 and β2 are both constants; The audio power spectrum satisfies the following relationship: D(m',k')=|S(m',k')| 2 Where D(m',k') is the audio power spectrum.

6. The speech noise reduction recognition method according to claim 4 or 5, characterized in that: Based on the noise power spectrum, the audio power spectrum is subjected to noise reduction processing to obtain speech data, including: Extract the corresponding real and imaginary parts based on the audio complex spectrum value, and give the audio phase angle; Comparing the audio power spectrum with the noise power spectrum, and combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum to determine the speech amplitude of the speech data; The speech amplitude and the audio phase angle are superimposed and inverse Fourier transformed to obtain speech data.

7. The speech noise reduction recognition method according to claim 6, characterized in that: The audio phase angle satisfies the following relationship: In the formula, S angle is the audio phase angle, S(m',k') is the audio complex spectrum value of the k'th audio frequency component of the m'th frame; The audio power spectrum is compared with the noise power spectrum, and the voice amplitude of the voice data is determined by combining the difference between the audio power spectrum and the noise power spectrum or the noise power spectrum, which is specifically expressed as: In the formula, is the speech amplitude of the speech data, a is the over-reduction factor, and b is the gain compensation factor.

8. The speech noise reduction recognition method according to claim 1, characterized in that: Pre-built accelerated speech recognition models, including: The speech recognition model is trained based on the historical speech data to obtain a trained speech recognition model; Quantize the trained speech recognition model to obtain a speech recognition model in ONNX format; The speech recognition model in the ONNX format is trimmed to obtain a trimmed speech recognition model in the ONNX format; The trimmed ONNX-format speech recognition model is loaded into the ONNX-format inference framework to obtain the final speech acceleration recognition model.

9. The speech noise reduction recognition method according to claim 8, characterized in that: The trained speech recognition model is quantized to obtain a speech recognition model in ONNX format, including: Select the ONNX inference engine that runs the speech recognition model in ONNX format; Based on the ONNX inference engine, select the ONNX operator set that matches the speech acceleration recognition model; Based on the ONNX operator set, the speech acceleration recognition model is converted into a format to construct an intermediate recognition model in ONNX format. The model quantization module based on the ONNX inference engine performs statistical processing on the weight data of the intermediate state recognition model in ONNX format and gives the minimum and maximum values; The minimum and maximum values ​​are processed by asymmetric quantization, and the quantization ratio and critical value are given; The weight data is converted by quantization ratio and critical value to obtain weight data converted from floating point numbers to integers; The weight data converted into integers is used to replace the weight data of the intermediate state recognition model in ONNX format to obtain a speech recognition model in ONNX format.

10. A speech noise reduction recognition device in a computer room environment, characterized in that: According to any one of claims 1 to 9, the speech noise reduction recognition method in a computer room environment is adopted, and the speech noise reduction recognition device comprises: A noise processing module, which is used to obtain noise data in multiple types of computer rooms and perform a first preprocessing to obtain a noise power spectrum; An audio processing module, which is used to collect audio data in the computer room and perform a second preprocessing to obtain an audio power spectrum; A noise reduction processing module, which is used to perform noise reduction processing on the audio power spectrum based on the noise power spectrum to obtain speech data; The speech recognition module is used to recognize and process speech data based on a pre-built speech acceleration recognition model and provide text information.

Images