Cooking audio-based menu making method and device, equipment and medium

Abstract

The invention provides a menu making method and device based on cooking audio, equipment and a medium, and relates to the technical field of data processing, and the method comprises the steps: carrying out the adaptive filtering processing of the cooking audio, obtaining a first audio, carrying out the voice enhancement processing of the first audio, carrying out the noise reduction processing of the first audio after the voice enhancement processing, and obtaining a second audio, and voice recognition is performed based on the second audio, and the menu corresponding to the cooking audio is obtained based on the voice recognition result, so that the menu can be reliably made based on the unstructured multimedia audio resources.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a recipe making method, device, equipment and medium based on cooking audio. Background Art

[0002] With the rapid development of Internet technology and multimedia content, cooking videos and pictures have become the main medium for learning and sharing cooking skills. Currently, a large amount of cooking information exists in the form of unstructured multimedia, which contains rich visual and auditory data. However, it is a technical challenge to automatically extract useful cooking information and produce recipes from these unstructured multimedia resources. Summary of the invention

[0003] The purpose of the present invention includes, for example, providing a method, apparatus, device and medium for making a recipe based on cooking audio, so as to at least partially realize the automatic extraction of useful cooking information from unstructured multimedia resources and make it into a recipe.

[0004] The embodiments of the present invention can be implemented as follows:

[0005] In a first aspect, a recipe making method based on cooking audio is provided, comprising:

[0006] Performing adaptive filtering on the cooking audio to obtain a first audio;

[0007] Performing speech enhancement processing on the first audio, and performing noise reduction processing on the first audio after the speech enhancement processing to obtain a second audio;

[0008] Speech recognition is performed based on the second audio, and a recipe corresponding to the cooking audio is obtained based on the speech recognition result.

[0009] In an optional implementation manner, performing adaptive filtering on the cooking audio to obtain the first audio includes:

[0010] Inputting the cooking audio as an input signal into an adaptive filter, wherein a finite impulse response structure is provided in the adaptive filter so that an impulse response of the adaptive filter is equal to a filter coefficient of the adaptive filter;

[0011] A speech signal in the input signal is obtained based on the input signal and the filter coefficient analysis, and the speech signal is used as the first audio.

[0012] In an optional embodiment, the method further comprises:

[0013] Convolving the input signal with the impulse response to obtain an estimated signal;

[0014] Subtracting the speech signal from the estimated signal to obtain an error signal;

[0015] generating a correction factor based on the input signal and the error signal;

[0016] The filter coefficients are updated based on the correction factor.

[0017] In an optional implementation manner, performing noise reduction processing on the first audio after speech enhancement processing to obtain the second audio includes:

[0018] Performing noise detection based on the first audio after speech enhancement processing, and extracting parameters representing noise characteristics;

[0019] Determine a noise type based on the parameter, the noise type including additive noise, multiplicative noise, white noise, and Gaussian white noise;

[0020] When the noise type is additive noise, performing noise reduction processing on the first audio after speech enhancement by using a filter and / or a noise reduction algorithm;

[0021] In a case where the noise type is multiplicative noise, performing noise reduction processing on the first audio after speech enhancement through signal enhancement processing and / or envelope detection;

[0022] When the noise type is white noise, performing noise reduction processing on the first audio after speech enhancement by mean filtering and / or spectrum shaping;

[0023] When the noise type is Gaussian white noise, performing noise reduction processing on the first audio after speech enhancement by Kalman filtering and / or Bayesian filtering;

[0024] Convert the first audio after noise reduction processing into the time-frequency domain, and perform joint noise reduction processing in the time domain and the frequency domain by Fourier transform method to obtain a second audio;

[0025] The window size in the Fourier transform method is adaptively obtained based on the signal change frequency of the first audio, and the overlap ratio in the Fourier transform method is adaptively obtained based on the window size.

[0026] In an optional implementation, the step of performing speech enhancement processing on the first audio includes:

[0027] Based on frequency domain features, separate the speech signal and the noise signal in the first audio, and only retain the frequency spectrum component of the speech signal; or input the first audio into a low-pass filter or a high-pass filter for filtering, and remove high-frequency and low-frequency noise in the first audio;

[0028] The speech signal, or the first audio after high-frequency and low-frequency noises are removed, is subjected to speech signal enhancement processing.

[0029] In an optional implementation, performing speech recognition based on the second audio, and obtaining a recipe corresponding to the cooking audio based on the speech recognition result, includes:

[0030] Normalizing the second audio, performing feature extraction and speech enhancement processing, to obtain a third audio;

[0031] Performing semantic recognition and analysis on the third audio to obtain a corresponding recipe text;

[0032] The recipe text is subjected to word segmentation, entity recognition, grammatical analysis and information extraction to generate a structured recipe.

[0033] In an optional implementation manner, the performing semantic recognition and parsing on the third audio to obtain a corresponding recipe text includes:

[0034] Performing semantic recognition and analysis on the third audio to obtain a speech recognition result;

[0035] Rewriting the speech recognition result according to the corresponding prompt word template to obtain the corresponding recipe prompt word;

[0036] Inputting the recipe prompt words into a corresponding generative large language model to add corresponding semantic expansion content to the recipe text; wherein the semantic expansion content includes: cooking ingredient information, cooking step information, recipe background information, cooking tips information, user comment information, and user rating information;

[0037] Perform integrity check on the semantically expanded recipe text to obtain the final recipe text.

[0038] In a second aspect, the present invention provides a recipe making device based on cooking audio, comprising:

[0039] An information acquisition module is used to perform adaptive filtering processing on the cooking audio to obtain a first audio; perform speech enhancement processing on the first audio, and perform noise reduction processing on the first audio after the speech enhancement processing to obtain a second audio;

[0040] An information processing module is used to perform speech recognition based on the second audio, and obtain a recipe corresponding to the cooking audio based on the speech recognition result.

[0041] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the recipe making method based on cooking audio as described in any one of the aforementioned implementations is implemented.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the recipe preparation method based on cooking audio as described in any one of the aforementioned embodiments.

[0043] The beneficial effects of the embodiments of the present invention include, for example: providing a recipe making method, device, equipment and medium based on cooking audio, by performing adaptive filtering, speech enhancement, noise reduction, speech recognition and other processing on the cooking audio, obtaining a recipe corresponding to the cooking audio based on the speech recognition result, thereby realizing automatic acquisition of cooking information and making recipes based on unstructured multimedia audio resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic flow chart of a recipe making method based on cooking audio provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a sub-step flow chart of S110 provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of an implementation principle of adaptive filtering provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a sub-step flow chart of S130 provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a sub-step flow chart of S132 provided in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the overall process of a recipe making method based on cooking audio provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the architecture of a recipe making device based on cooking audio provided by an embodiment of the present invention;

[0052] Figure 8 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0053] Icons: 11-electronic device; 111-controller; 112-ROM; 113-RAM; 114-bus; 115-I / O interface; 116-input unit; 117-output unit; 118-storage unit; 119-communication unit; 140-recipe making device based on cooking audio; 141-information acquisition module; 142-information processing module. DETAILED DESCRIPTION

[0054] Nowadays, with the rapid development of Internet technology and multimedia content, multimedia resources such as cooking videos and pictures have become the main media for learning and sharing cooking skills. At present, the solution for automatically obtaining recipes mainly processes structured multimedia resources to realize the automatic extraction of recipes.

[0055] For example, static text processing is performed on document data to obtain recipes. Another example is text recognition and processing is performed based on static images to obtain recipes.

[0056] However, for unstructured multimedia resources, such as dynamic cooking videos that contain rich visual and audio information, especially when processing the simultaneous analysis of voice commentary and visual content in cooking videos, the technology that relies solely on text processing such as text feature extraction lacks the ability to process audio content and other visual dynamic information in the video, making it difficult to achieve efficient information integration and accurate recipe generation. In practical applications, this may lead to incomplete information and insufficient precision extracted from complex cooking videos.

[0057] Research has found that the noise problem is particularly prominent in the process of extracting recipes from audio and video. For example, various environmental noises generated during the cooking process, noise from the operation of cooking utensils, and background human voice noise will interfere with the audio signal, thereby affecting the accuracy of speech recognition and the extraction effect of recipe information. Therefore, in the process of extracting recipes from audio and video, how to effectively reduce noise has become an urgent problem to be solved.

[0058] Based on the above research, an embodiment of the present invention provides a recipe preparation method, device, equipment and medium based on cooking audio, which cleverly applies noise reduction processing technology to improve the accuracy of audio recognition in unstructured multimedia resources, thereby ensuring the quality of recipes automatically extracted based on unstructured multimedia audio resources.

[0059] The defects existing in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present invention for the above problems below should all be the contributions made by the inventors in the invention process.

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0063] In the description of the present invention, it should be noted that the terms “first”, “second”, etc., if used, are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0064] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0065] See also Figure 1 , is a flow chart of a recipe making method based on cooking audio provided in an embodiment of the present invention, the method comprising S110, S120 and S130.

[0066] S110, performing adaptive filtering processing on the cooking audio to obtain a first audio.

[0067] S120: Perform speech enhancement processing on the first audio, and perform noise reduction processing on the first audio after the speech enhancement processing to obtain a second audio.

[0068] S130, performing speech recognition based on the second audio, and obtaining a recipe corresponding to the cooking audio based on the speech recognition result.

[0069] The cooking audio may be audio in a cooking scene. By performing adaptive filtering, speech enhancement, noise reduction, speech recognition and other processing on the cooking audio, a recipe corresponding to the cooking audio is obtained based on the speech recognition result, thereby improving the problem of noise interference affecting the accuracy and structure of recipe conversion in the audio-to-recipe process, realizing automatic acquisition of recipes based on unstructured multimedia audio resources, and improving the accuracy of audio recognition and the quality of recipe generation.

[0070] In S110, the cooking audio is adaptively filtered to obtain the first audio, which can be implemented flexibly as long as the audio quality can be improved based on the adaptive filtering process.

[0071] In one implementation, adaptive filtering can be performed based on an adaptive filter. Figure 2 , S110 can be implemented through S111 and S112.

[0072] S111, input the cooking audio as an input signal into an adaptive filter.

[0073] A finite impulse response structure is provided in the adaptive filter so that the impulse response of the adaptive filter is equal to the filter coefficient of the adaptive filter.

[0074] S112: Analyze the input signal and the filter coefficient to obtain a speech signal in the input signal, and use the speech signal as the first audio.

[0075] The adaptive filter can be a digital filter that can automatically adjust the performance according to the input signal to perform digital signal processing. Since the parameters to be operated are not known in advance, such as the characteristics of some noise signals, in this case, the adaptive filter can use feedback to adjust the filter coefficients and frequency response.

[0076] Accordingly, the filter coefficients can be adjusted by feedback in the following manner: convolving the input signal with the impulse response to obtain an estimated signal, subtracting the speech signal from the estimated signal to obtain an error signal, generating a correction factor based on the input signal and the error signal, and updating the filter coefficients based on the correction factor.

[0077] Compared with traditional infinite impulse response (IIR) filters and finite impulse response (FIR) filters, the parameters of the filters are fixed during the processing of input signals. When the environment changes, the filters cannot achieve the originally set goals. Adaptive filters can adjust the weights of the filters according to their own state and environmental changes, thus having better filtering performance.

[0078] IIR filter is a recursive system, that is, the current output sample is a function of the past output and past input samples, and its system impulse h(n) is an infinite length sequence. The phase-frequency characteristic of IIR filter is nonlinear, and stability cannot be guaranteed. The advantage is that the implementation order is low and the amount of calculation is small. FIR filter is a non-recursive system, that is, the current output sample is only a function of the past and present input samples, and its system impulse response h(n) is a finite length sequence. It has a good linear phase, no phase distortion, and good stability.

[0079] Adaptive filters (Variable filter) can be implemented based on IIR or FIR structures. Unlike traditional IIR filters or FIR filters, adaptive filters have the ability to dynamically adjust filter coefficients. Traditional IIR filters and FIR filter parameters are fixed, while adaptive filters can dynamically adjust their coefficients and frequency responses according to input signals and environmental changes, such as implementing dynamic adjustments based on a diffusion update algorithm, thereby ensuring adaptive filtering of the cooking audio, and the first audio obtained has better audio quality improvement.

[0080] For example, please refer to Figure 3 , take cooking audio as input signal x(n), the input signal x(n) is the sum of speech signal d(n) and interference noise v(n), then there is the following expression:

[0081] x(n)=d(n)+v(n)

[0082] Adaptive filters (also called variable filters) can have a finite impulse response structure, where the impulse response of the structure is equal to the filter coefficients. n The definition is as follows:

[0083] w n =[w n (0), w n (i), ..., w n (p)] T

[0084] The error signal (also called cost function) e(n) is the difference between the speech signal d(n) and the estimated signal The difference exists in the following expression:

[0085]

[0086] The adaptive filter estimates the speech signal by convolving the input signal with the impulse response to obtain the estimated signal a(nn), which is represented by a vector:

[0087]

[0088] Among them, the input signal

[0089] The adaptive filter changes the filter coefficient based on the correction factor of the filter coefficient. In order to ensure the timeliness of the filter coefficient adjustment, the filter coefficient can be adjusted immediately by the following formula every time the correction factor is obtained:

[0090] w n+1 =w n +Δw n

[0091] Among them, Δw n is a correction factor of the filter coefficient, which can be generated according to the input signal and the error signal.

[0092] The strategy of generating correction factors according to the input signal and the error signal to update the filter coefficients can be flexibly selected. For example, the filter coefficients can be updated based on the least mean squares algorithm (LMS), the recursive least squares algorithm (RLS), etc., so that the audio quality can be reliably improved after using the adaptive filter.

[0093] Among them, the least mean square algorithm is used to adjust the parameters of the linear filter in real time. The filter coefficient of the filter can be adjusted according to the estimated error to minimize the mean square error between the input signal and the output signal. The recursive least squares algorithm does not need to make assumptions about the statistical characteristics of the input sequence, but minimizes the problem through a purely deterministic method and recursively updates the filter parameters to adapt to new data. Among them, the filter coefficient is one of the core parameters of the filter.

[0094] In the case where the first audio is obtained through adaptive filtering processing based on S110, in S120, voice enhancement processing of the first audio can be flexibly implemented.

[0095] For example, an acoustic noise elimination method may be used to achieve speech enhancement. Based on frequency domain features, the speech signal and interference noise (noise signal) in the first audio are separated, and only the spectral components of the speech signal are retained to achieve noise elimination.

[0096] For another example, a speech enhancement filtering method can be used to achieve speech enhancement, and the first audio is input into a low-pass filter and a high-pass filter for filtering to remove high-frequency and low-frequency noise in the first audio. The speech signal, or the first audio after removing high-frequency and low-frequency noise, is subjected to speech signal enhancement, such as smoothing, enhancement, amplification, etc., so that the enhanced speech signal is clearer.

[0097] Taking the use of MATLAB to perform speech enhancement filtering on the first audio to obtain the second audio as an example, the implementation process can be as follows:

[0098] The cooking audio is adaptively filtered to obtain the first audio, which is used as the original data to be subjected to speech enhancement filtering.

[0099] Use the audioread function in MATLAB to read the speech signal file where the first audio is located, and store the read first audio in the vector shown below:

[0100] [x,fs]=audioread('speech.wav')

[0101] Among them, x is the speech signal vector and fs is the sampling rate.

[0102] The audioread function is used to read audio files in MATLAB. It can read audio files in various formats and return the audio data and sampling rate.

[0103] Design a speech enhancement filter, such as using the designfilt function in MATLAB to design a low-pass filter lpFilt, as follows:

[0104] lpFilt=designfilt('lowpassiir','FilterOrder',8,

[0105] 'PassbandFrequency',1000,'PassbandRipple',0.2,'SampleRate',fs)

[0106] Among them, 'lowpassiir' means designing a low-pass filter with infinite impulse response, 'FilterOrder' means the order of the filter, 'PassbandFrequency' means the passband cutoff frequency, 'PassbandRipple' means the maximum passband ripple, and 'sampleRate' means the sampling rate.

[0107] The designfilt function is a function in MATLAB used to design digital filters. It provides methods for creating various types of digital filters.

[0108] Use the created speech enhancement filter to perform speech enhancement filtering on the speech signal to obtain the first audio y after speech enhancement, as follows:

[0109] y=filter(lpFilt,x)

[0110] In S120, performing noise reduction processing on the first audio after speech enhancement processing to obtain the second audio can be flexibly implemented.

[0111] In one implementation, the method can be implemented as follows: noise detection is performed based on the first audio after speech enhancement processing, and parameters representing noise characteristics are extracted. The noise type is determined based on the parameters, and the noise type includes additive noise, multiplicative noise, white noise, and Gaussian white noise.

[0112] In the case where the noise type is additive noise, noise reduction processing is performed on the first audio after speech enhancement by using a filter and / or a noise reduction algorithm.

[0113] In the case where the noise type is multiplicative noise, noise reduction processing is performed on the first audio after speech enhancement through signal enhancement processing and / or envelope detection.

[0114] When the noise type is white noise, the first audio after speech enhancement is subjected to noise reduction processing by mean filtering and / or spectrum shaping.

[0115] When the noise type is Gaussian white noise, the first audio after speech enhancement is subjected to noise reduction processing by Kalman filtering and / or Bayesian filtering.

[0116] The first audio after noise reduction processing is converted into time-frequency domain, and joint noise reduction processing is performed in time domain and frequency domain by Fourier transform method to obtain second audio.

[0117] The window size in the Fourier transform method is adaptively obtained based on the signal change frequency of the first audio, and the overlap ratio in the Fourier transform method is adaptively obtained based on the window size.

[0118] Noise detection based on the first audio after speech enhancement processing can be flexibly implemented. Given that noise is the sum of various interference signals to which the signal is subjected during transmission, it directly affects the transmission quality of the signal and even destroys the normal signal. In layman's terms, noise is defined as useless signal components in the signal. The noise signal is mixed in the original signal (the desired speech signal), which may cause distortion of the original signal. Therefore, reliable detection and analysis of noise is an important factor in ensuring the reliability of subsequent noise reduction and, in turn, the reliability of recipe extraction.

[0119] Based on different noise characteristics, it is mainly divided into additive noise, multiplicative noise and white noise.

[0120] Additive noise: There is a superposition relationship between noise and the signal transmitted in the channel. Its characteristic is that noise exists independently and has nothing to do with the presence or absence of channel signals. It has a great impact on the quality of signal transmission. In practical applications, additive noise can only be reduced but cannot be completely eliminated.

[0121] Multiplicative noise: There is a multiplicative relationship between noise and the signal transmitted in the channel. The noise exists as long as the signal exists. When the signal disappears, the multiplicative noise will also be annihilated.

[0122] White noise: The power spectral density is constant in the entire frequency domain. Random noise with the same energy density at all frequencies is called white noise. If the probability density function of the noise also follows a Gaussian distribution, it is called Gaussian white noise.

[0123] In the process of noise detection, in order to remove redundant and interfering parts in the signal and enhance the analyzability of the signal, the first audio based on speech enhancement can be preprocessed, and feature extraction can be performed on the preprocessed signal. Parameters that can represent its characteristics are extracted from the signal, and the noise detection result is determined based on the extracted parameters.

[0124] The method of determining the noise detection result based on the parameters can be flexibly selected. For example, the parameters can be input into a trained noise classification model, and noise detection and classification recognition can be performed through the noise classification model. For another example, the parameters can be analyzed based on a preset calculation strategy and function to achieve noise detection and classification recognition.

[0125] Exemplarily, the extracted parameters that can represent the noise characteristics may include time domain features. Accordingly, the parameter mean of the noise signal may be calculated, and the noise type may be determined based on the mean. If the mean is close to zero, it may be white noise or Gaussian white noise. The parameter variance of the noise signal may also be calculated, and the energy of the noise may be reflected based on the variance, and the noise type may be determined based on the energy. The periodic characteristics of the noise may also be determined by the autocorrelation function, and the noise type may be determined based on the periodic characteristics.

[0126] The extracted parameters that can represent the noise characteristics can also include frequency domain features. Accordingly, the noise type can be determined based on the power spectral density (PSD) based on the different performances of different types of noise in the frequency domain. For example, the energy of white noise is evenly distributed in the entire frequency domain. Therefore, if the analysis shows that the energy is evenly distributed in the frequency domain, it can be determined as white noise. The noise type can also be determined based on the different performances of the spectrum peak. For example, some noise types may have obvious peaks at specific frequencies, such as power supply noise has obvious peaks at 50Hz or 60Hz frequencies, so the noise type can be determined based on the spectrum peak situation.

[0127] The extracted parameters that can represent the noise characteristics can also include time-frequency features. Accordingly, the spectrum of the noise signal changing with time can be analyzed according to the short-time Fourier transform (STFT), and then the noise can be classified based on the Mel-Frequency Cepstral Coefficients (MFCC) to obtain the noise type.

[0128] In yet another implementation, a simulated noise signal may be generated based on a program to perform testing and analysis of noise classification in research and experiments.

[0129] For example, noise can be simulated, for example, white noise can be simulated, and the number of sampling points and sampling frequency can be defined, such as the number of sampling points n_sampling = 10000, and the sampling frequency fs = 1000 (Hz), which means that the generated white noise will contain 10000 sample points, and 1000 sample points will be collected per second.

[0130] After obtaining the required sample points, an array is generated based on each sample point. For example, the random.normal function of the numpy library can be used to generate an array that conforms to the normal distribution and represents white noise:

[0131] white_noise=np.random.normal(0,5,n_sampling)

[0132] Among them, white_noise is white noise, parameter 0 is the mean, 5 is the standard deviation, and n_sampling is the number of samples generated.

[0133] Generate a time vector t, indicating the time corresponding to each sample point:

[0134] t=np.linspace(0,n_sampling / fs,n_sampling)

[0135] Here, t starts from 0 and ends at n_sampling / fs (i.e. 10 seconds), containing n_sampling equally spaced points.

[0136] Based on the white noise generated by the above simulation, the power spectrum is calculated as a basis for subsequent classification. The power spectrum can be obtained in the following way:

[0137] Use the fftpack.fft function to calculate the Fast Fourier Transform (FFT) of white noise and get the frequency domain representation p:

[0138] p = fftpack.fft(white_noise)

[0139] Calculate the absolute value of the FFT result p and square it to get the power spectrum power:

[0140] power = np.abs(p)**2

[0141] The power spectrum power represents the energy of each frequency component.

[0142] After obtaining the power spectrum power, the power spectrum power can be processed into a graphic form for easy display and application.

[0143] To convert the power spectrum from a linear scale to a decibel scale:

[0144] power_db = 10*np.log10(power)

[0145] The decibel is a logarithmic unit used to describe the ratio of power or intensity.

[0146] Calculate the corresponding frequency array f of the FFT result p:

[0147] f=np.fft.fftfreq(p.size,1 / fs)

[0148] Among them, the fftfreq function returns an array containing the frequencies of each point in the Fourier transform.

[0149] Create a graphics window and set its size and resolution. For example, if the size is 10x6 inches and the resolution is 100 DPI, the graphics window fig is:

[0150] fig=plt.figure(figsize=(10,6),dpi=100)

[0151] Define the font dictionary font, which is used to set the font, size, and thickness of the text in the chart, as follows:

[0152] font={'family':'Times New Roman','size':14,'weight':'normal'}

[0153] Apply the font settings to the global configuration of matplotlib:

[0154] matplotlib.rc('font',**font)

[0155] Create a grid layout grid to place different charts:

[0156] grid=plt.GridSpec(8,8,hspace=0.1)

[0157] Among them, hspace=0.1 sets the space in the vertical direction.

[0158] Select an area in the grid to place a plot of the time domain signal:

[0159] plt.subplot(grid[0:2,0:8])

[0160] Plot the time domain signal of white noise, using the blue line:

[0161] plt.plot(t,white_noise,'b')

[0162] Set the label of the X-axis to "time(s)" and apply the previously defined font:

[0163] plt.xlabel('time(s)',fontname='Times New Roman',fontsize=14)

[0164] Set the Y-axis label to "Amplitude" and apply the previously defined font:

[0165] plt.ylabel('Amplitude',fontname='Times New Roman',fontsize=14)

[0166] Select another area in the grid to place a plot of the frequency domain signal:

[0167] plt.subplot(grid[3:8,0:8])

[0168] Plot the power spectral density, taking only the first half of the frequency array (due to the symmetry of the FFT result). Use the red line:

[0169] plt.plot(f[0:n_sampling / / 2],power_db[0:n_sampling / / 2],'r')

[0170] Set the label of the X-axis to "Frequency(Hz)" and apply the previously defined font:

[0171] plt.xlabel('Frequency(Hz)',fontname='Times New Roman',fontsize=14)

[0172] Set the Y-axis label to "Magnitude (dB)" and apply the previously defined font:

[0173] plt.ylabel('Magnitude(dB)',fontname='Times New Roman',fontsize=14)

[0174] Show the Graphics window, showing the previously drawn graph:

[0175] plt.show()

[0176] Based on the above process, white noise can be simulated and generated, and the power spectrum, as well as graphical time domain signals, frequency domain signals and power spectrum density can be obtained, thus providing a basis for the classification and identification of white noise.

[0177] The above provides an example of program-based generation of simulated white noise for testing and analyzing white noise classification. It is understandable that the principles of simulation, testing and analysis of other types of noise are similar and will not be described one by one here.

[0178] The above provides some optional implementation schemes for performing noise detection based on the first audio after speech enhancement processing. It can be understood that, based on the actual situation of each scene, one or more of the methods can be flexibly selected for noise detection to obtain parameters representing noise characteristics, thereby determining the noise type based on the parameters.

[0179] After the noise type is obtained, noise reduction strategies are determined based on the noise type, and noise reduction processing can be flexibly performed on the first audio after speech enhancement based on the noise reduction strategy.

[0180] For example, if additive noise is determined based on the noise type, the noise in a specific frequency range can be removed by using filters such as low-pass filters, high-pass filters, and band-pass filters. Adaptive filtering such as LMS adaptive filters can also be used to dynamically adjust filter parameters according to noise characteristics to remove noise. Noise reduction algorithms such as Wiener filtering and wavelet transform can also be used to reduce the impact of noise.

[0181] If the noise type is determined to be multiplicative noise, the strength of the original signal can be increased through signal enhancement technology, thereby reducing the relative impact of the multiplicative noise. The signal envelope can also be extracted through envelope detection methods to remove the noise component.

[0182] If the noise type is determined to be white noise, the noise can be smoothed by calculating the moving average of the signal through mean filtering. The uniform noise component in the spectrum can also be removed through spectrum trimming.

[0183] If the noise type is determined to be Gaussian white noise, the Kalman filter can be used to estimate and track the signal, thereby reducing the impact of Gaussian white noise. The Bayesian filter technology can also be used for signal estimation and noise suppression.

[0184] The above provides an optional implementation scheme for determining a noise reduction strategy based on the noise type and performing noise reduction processing on the first audio after speech enhancement. It can be understood that based on the actual situation of each scenario, one or more of the methods can be flexibly selected to perform noise reduction processing to obtain the first audio after noise reduction processing.

[0185] After obtaining the first audio after noise reduction processing, the first audio after noise reduction processing is converted into time-frequency domain, and joint noise reduction processing is performed in time domain and frequency domain by Fourier transform method to obtain the second audio, so as to further remove residual noise and improve signal quality.

[0186] The window size in the Fourier transform method is adaptively obtained based on the signal change frequency of the first audio, and the overlap ratio in the Fourier transform method is adaptively obtained based on the window size.

[0187] Time-frequency domain joint noise reduction converts the audio signal into the time-frequency domain, such as through short-time Fourier transform (STFT), and performs joint noise reduction on the audio signal in the time domain and frequency domain. Use STFT to convert the time domain signal into the time-frequency domain signal, obtain the spectrum, and set the STFT parameters (such as window size, overlap ratio, etc.) to ensure the effect of the time-frequency domain conversion.

[0188] Through joint denoising in the time and frequency domains, converting the audio signal to the time and frequency domains provides more detailed signal information, which can more accurately estimate the noise characteristics, thereby more effectively removing the noise while retaining the details and structure of the audio signal. Time domain smoothing can reduce artifacts introduced by frequency domain processing and improve the auditory quality of the signal. By adjusting the STFT parameters, the time resolution and frequency resolution can be flexibly balanced to meet different denoising requirements.

[0189] Given that the window size determines the length of each STFT slice, that is, the length of the signal segment analyzed by each Fourier transform, choosing an appropriate window size requires a trade-off between time resolution and frequency resolution:

[0190] Time resolution: A smaller window size provides better time resolution, which can capture the temporal changes of the signal more finely.

[0191] Frequency Resolution: A larger window size provides better frequency resolution, which can capture the frequency components of the signal more finely.

[0192] In order to ensure the adaptability of the window size during the Fourier transform processing, adaptive Fourier transform can be used. Adaptive Fourier transform is an improved Fourier transform technology that can dynamically adjust the window size and overlap ratio according to the characteristics of the input signal to better capture the transient characteristics of the signal.

[0193] Among them, the rules for dynamically adjusting the window size can be set flexibly. For example, the window size can be determined based on the signal characteristics. If the signal changes quickly (high-frequency signal), a smaller window can be used to obtain better time resolution. If the signal changes slowly (low-frequency signal), a larger window can be used to obtain better frequency resolution. For example, the window size can be determined based on experience and big data analysis. For example, a window with a length of 10-20 times the signal sampling rate is usually used. Exemplarily, for a signal with a sampling rate of 1000Hz, a window of 100 to 200 sampling points can be selected.

[0194] Given that the overlap ratio determines the overlap between adjacent windows to avoid information loss and smooth the spectrum graph, a higher overlap ratio can increase the smoothness and continuity of the time-frequency graph, but at the same time it will increase the amount of computation. Therefore, the overlap ratio can be flexibly set according to the actual needs in each scenario. For example, for scenarios that require precise time-frequency analysis, a higher overlap ratio (such as 50% or 75%) can be set to improve the smoothness and continuity of the time-frequency graph. Given that a higher overlap ratio will increase the amount of computation, the overlap ratio can be determined based on device performance, scenario requirements, and a trade-off between accuracy and computation.

[0195] This embodiment uses adaptive Fourier transform (an improved Fourier transform technology) as an example to explain the implementation principle of dynamically adjusting the window size and overlap ratio based on the characteristics of the input signal based on adaptive Fourier transform to better capture the transient characteristics of the signal.

[0196] Among them, the confirmation strategy of the window size may include: for the rapidly changing part of the signal, use a smaller window to improve the time resolution and better capture the transient characteristics, so as to more effectively separate the noise and the useful signal, improve the quality of signal processing, and improve the signal-to-noise ratio; for the slowly changing part of the signal, use a larger window to improve the frequency resolution and better analyze the spectral characteristics of the signal.

[0197] The overlap ratio confirmation strategy may include: adjusting the overlap ratio according to the window size to ensure the continuity of the signal and the smoothness of the spectrum. For example, in the slowly changing (relatively stable) part of the signal, the overlap ratio can be reduced to reduce computational redundancy and improve processing efficiency. At the same time, by dynamically adjusting the window size and overlap ratio, the artifacts introduced by the fixed window and the fixed overlap ratio can be effectively reduced.

[0198] Exemplarily, the adaptive Fourier transform can be implemented based on the function adaptive_fft(signal, fs, window_type='hann'), and the spectrogram can be calculated.

[0199] Where signal is the input time-domain signal; fs is the sampling frequency of the signal. window_type is the type of the window function, and the default value is 'hann'; the spectrogram represents the frequency distribution of the signal at different time points.

[0200] For example, inside the function adaptive_fft, a series of different window sizes window_sizes are defined, such as window_sizes = [128, 256, 512, 1024], which are used to analyze different parts of the signal. A fixed overlap ratio overlap_ratio is set, such as overlap_ratio = 0.5.

[0201] Calculate the step sizes step_sizes corresponding to each window size as follows:

[0202] step_sizes = [int(ws * (1 - overlap_ratio)) for ws in window_sizes].

[0203] The step size determines how many data points to move each time to obtain the next segment of the signal.

[0204] Initialize a list spectrogram, spectrogram = [], which is used to store the calculated spectrogram.

[0205] Perform segmented processing on the signal. Through a while loop, traverse the entire signal. The condition for the loop is that the current index is less than the signal length minus the maximum window size. For example, initialize the index i to 0. In each loop, extract the current segment of the signal, and the length is the maximum window size:

[0206] segment = signal[i:i + max(window_sizes)]

[0207] When the index i is less than the signal length minus the maximum window size:

[0208] while i < len(signal) - max(window_sizes), the loop continues.

[0209] Inside the loop, first extract the current segment of the signal, and select the best window size according to the local signal characteristics:

[0210] local_window_size=choose_optimal_window(segment,window_sizes)

[0211] window=get_window(window_type,local_window_size)

[0212] By calling the choose_optimal_window function, the optimal window size is selected according to the characteristics of the current signal segment, and the corresponding window function is obtained according to the selected window size to intercept the signal segment of appropriate length:

[0213] segment=segment[:local_window_size]

[0214] windowed_segment=segment*window

[0215] Apply the window function to calculate the Fourier transform of the intercepted signal segment, take its first half (corresponding to the positive frequency part), and add the spectrum to the spectrogram list:

[0216] spectrum=np.abs(fft(windowed_segment))[:local_window_size / / 2]

[0217] spectrogram.append(spectrum)

[0218] Update the index according to the chosen window size in order to move to the next signal segment, with a step size depending on the chosen window size:

[0219] step_size=step_sizes[window_sizes.index(local_window_size)]

[0220] i+=step_size

[0221] After the loop is finished, the spectrogram list is converted to an array and transposed for easy plotting of the spectrogram:

[0222] return np.array(spectrogram).T

[0223] Returns the transposed spectrogram array.

[0224] Define the function choose_optimal_window to select the optimal window size based on the characteristics of the signal segment:

[0225] choose_optimal_window(segment,window_sizes)

[0226] Among them, segment is the input signal segment; window_sizes is an optional window size list.

[0227] Specifically, the optimal window size is selected through the following return process:

[0228] Calculate the standard deviation std_dev of the signal segments as a simple criterion for choosing the window size:

[0229] std_dev = np.std(segment)

[0230] Select an appropriate window size from the list based on the size of the standard deviation. For example, the selection strategy can be that if the standard deviation is greater than 1.0, select the first window size in the list; if it is greater than 0.5, select the second, and so on; otherwise select the last one, as follows:

[0231] if std_dev>1.0:return window_sizes[0];

[0232] elif std_dev>0.5:return window_sizes[1]

[0233] elif std_dev>0.2:return window_sizes[2]

[0234] else:return window_sizes[3]

[0235] Returns the selected window size.

[0236] The sample signal and visualization process are as follows:

[0237] Set the sampling frequency (fs) to 1000 Hz, fs=1000.

[0238] Create a time vector (t) from 0 to 1 second, containing fs points, excluding the end point:

[0239] t=np.linspace(0,1,fs,endpoint=False)

[0240] Generate an example signal containing two different frequency components:

[0241] signal=np.sin(2*np.pi*7*t)+np.sin(2*np.pi*13*t)

[0242] Apply the adaptive Fourier transform adaptive_fft function to perform adaptive Fourier transform on the sample signal, and use the imshow function of the matplotlib library to draw the spectrum graph, set the title, axis labels, and color bar of the chart:

[0243] spectrogram=adaptive_fft(signal,fs)

[0244] plt.imshow(spectrogram,aspect='auto',origin='lower',cmap='viridis',exten t=[0,t[-1],0,fs / / 2])

[0245] plt.colorbar(label='Magnitude')

[0246] Before performing speech recognition in S130, a series of speech recognition pre-processing including adaptive filtering, speech enhancement, and noise reduction is performed on the cooking audio using S110 and S120. The resulting second audio is free of noise and includes a speech signal with better clarity, ensuring high-quality input for subsequent speech recognition.

[0247] In S130, voice recognition is performed based on the second audio, and a recipe corresponding to the cooking audio is obtained based on the voice recognition result, which can be flexibly implemented in a variety of ways. Figure 4 In one implementation, S130 may be implemented based on S131, S132, and S133.

[0248] S131, normalize the second audio and perform feature extraction and speech enhancement processing to obtain a third audio.

[0249] S132, performing semantic recognition and analysis on the third audio to obtain a corresponding recipe text.

[0250] S133, performing word segmentation, entity recognition, grammatical analysis and information extraction on the recipe text to generate a structured recipe.

[0251] For example, to ensure the reliability of recipe extraction, semantic supplementation and integrity verification can also be performed. Figure 5 In S132, semantic recognition and analysis are performed on the third audio to obtain a corresponding recipe text, which can be implemented based on S1321, S1322 and S1323.

[0252] S1321, perform semantic recognition and analysis on the third audio to obtain a speech recognition result.

[0253] S1322: rewrite the speech recognition result according to the corresponding prompt word template to obtain the corresponding recipe prompt word.

[0254] S1323, inputting the recipe prompt words into the corresponding generative large language model to add corresponding semantic expansion content to the recipe text.

[0255] The semantic expansion content includes: cooking ingredient information, cooking step information, recipe background information, cooking tips information, user comment information and user rating information.

[0256] S1324, performing integrity check on the recipe text after semantic expansion to obtain the final recipe text.

[0257] In order to more clearly explain the implementation process of performing speech recognition based on the second audio to obtain a recipe text, an implementation principle of obtaining a recipe corresponding to the cooking audio based on the speech recognition result is provided, as shown below.

[0258] Define the path audio_path of the audio file where the second audio is located:

[0259] audio_path='path / to / audio / file.wav'

[0260] Use librosa.load function to read the audio file:

[0261] audio,sr=librosa.load(audio_path,sr=None)

[0262] Among them, audio is the returned audio time series, and sr is the audio sampling rate.

[0263] Define the target sampling rate target_sr. For example, if the target sampling rate is defined as 16000Hz:

[0264] target_sr=16000

[0265] Use the librosa.resample function to convert the audio from the original sampling rate to the target sampling rate:

[0266] audio_resampled=librosa.resample(audio,orig_sr=sr,target_sr=target_sr)

[0267] Normalize the audio signal to the range [-1,1] by dividing by the maximum absolute value of the audio signal:

[0268] audio_normalized=audio_resampled / np.max(np.abs(audio_resampled))

[0269] Use librosa.feature.mfcc function to extract MFCC features of audio signal:

[0270] mfccs=librosa.feature.mfcc(y=audio_normalized,sr=target_sr,n_mfcc=13)

[0271] Among them, n_mfcc=13 specifies the number of MFCC features to be extracted.

[0272] Use the nr.reduce_noise function to perform noise reduction on the audio signal to obtain the third audio enhanced_signal with improved voice quality:

[0273] enhanced_signal=nr.reduce_noise(y=filtered_signal,sr=16000)

[0274] Among them, filtered_signal is the audio signal that needs to be filtered out.

[0275] The third audio enhanced_signal is semantically recognized and parsed by natural language processing (NLP) technology to parse the semantically recognized text into structured data.

[0276] Among them, semantic recognition and parsing can include word segmentation, entity recognition, grammatical analysis and information extraction.

[0277] The speech is converted into discrete words through word segmentation for recognition. You can load the model object nlp to process each text, nlp = spacy.load('en_core_web_sm').

[0278] Among them, spacy.load is a function in the spaCy library that is used to load a pre-trained language model. 'en_core_web_sm' is a small English language model that contains basic vocabulary and grammar knowledge and is suitable for text processing tasks.

[0279] Define the test text. Take the test text as one of the cooking steps, instructing to add 200 grams of flour and stir well:

[0280] text="Add 200grams of flour and mix well."

[0281] Use the loaded model to process text. doc is the processed text object, which contains information such as word segmentation, part-of-speech tagging, and dependency analysis. If there is:

[0282] doc=nlp(text)

[0283] Extract and print the tokens of non-stop words print("Tokens:",tokens), and generate a text list tokens containing all non-stop words:

[0284] tokens=[token.text for token in doc if not token.is_stop]

[0285] Where tokens is a list containing all non-stop-word texts. #token.text is the text representation of the word; #token.is_stop is a Boolean value indicating whether the word is a stop word; # is a marker for extracting non-stop words.

[0286] Named Entity Recognition (NER) identifies named entities in text, such as ingredients, quantities, units, etc. Extract and print named entities in text and their corresponding tags print("Entities:",entities).

[0287] entities=[(ent.text,ent.label_)for ent in doc.ents]

[0288] Among them, entities is a list containing all named entities in the text and their corresponding labels, #ent.text is the text representation of the entity; #ent.label is the label of the entity.

[0289] Then generate and print the structured recipe recipe and output it in json format, print("StructuredRecipe:",json.dumps(recipe,indent=2)).

[0290] The recipe includes a list of ingredients, ingredient quantities, steps, etc. An example of a structured recipe is provided below:

[0291] recipe = {

[0292] "ingredients":ingredients,

[0293] "quantities":quantities,

[0294] "steps": steps#step description list

[0295] }

[0296] Among them, ingredients is the ingredient list; quantities is the corresponding ingredient quantity list, and steps is the step description list.

[0297] In order to further improve the quality and practicality of recipes and provide users with more comprehensive, detailed and valuable cooking guides, after generating structured recipes, the generated recipes can be supplemented, such as through large models. The supplementary content for the recipes may include:

[0298] Ingredient Details: Description of complementary ingredients, substitution options, nutritional information, etc.

[0299] Step Details: Describe each step in detail to provide more precise operation instructions and time suggestions.

[0300] Recipe background: Provide information such as the recipe’s historical background, cultural relevance, and applicable scenarios.

[0301] Tips and Tricks: Add cooking tips, notes, common problems and solutions.

[0302] User Reviews and Ratings: Add reviews and ratings to improve the usefulness of recipes based on user feedback.

[0303] For example, a basic recipe basic_recipe is generated as follows:

[0304] basic_recipe = {

[0305] "ingredients":["flour","sugar","eggs"],

[0306] "quantities":["200grams","100grams","2"],

[0307] "steps":["Add flour and sugar.","Mix well.","Add eggs and stir."]

[0308] }

[0309] An Application Programming Interface (API) request may be constructed to request additional recipe information, for example:

[0310] prompt=(f"Please provide detailed information about the recipe below, including descriptions of each ingredient, substitution options, nutritional information, "f"detailed instructions and time recommendations for each step, as well as cooking tips and precautions:\n\n"f"{json.dumps(basic_recipe,ensure_ascii=False,indent=4)}")

[0311] Use OpenAI's Completion API to send the above constructed request as follows:

[0312] response=openai.Completion.create(engine="davinci-codex", prompt=prompt, max_tokens=300, n=1, stop=None, temperature=0.7)

[0313] Parse the API response supplemented_recipe = response.choices[0].text.strip() to obtain the detailed supplemented recipe information and get the final recipe text.

[0314] Please refer to Figure 6 , provides a schematic diagram of the overall process of a recipe making method based on cooking audio. Based on the recipe extraction scheme in this embodiment, through noise detection and classification, the identification and targeted processing of different noise sources are realized, and adaptive filtering, speech enhancement, noise reduction and other processing are comprehensively applied to adapt to different situations, filter noise in different scenes, reduce errors, minimize audio loss, improve audio quality, and ensure the completeness of recipe extraction.

[0315] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a recipe making device based on cooking audio is given below. Figure 7 , Figure 7A functional module diagram of a recipe making device 140 based on cooking audio provided in an embodiment of the present invention. The recipe making device 140 based on cooking audio can be applied to Figure 8 It should be noted that the basic principle and technical effects of the cooking audio-based recipe making device 140 provided in this embodiment are similar to those of the above-mentioned Figure 1 The method embodiments shown are the same. For the sake of brief description, the parts not mentioned in this embodiment can be referred to the above Figure 1 The cooking audio-based recipe making device 140 comprises an information obtaining module 141 and an information processing module 142 .

[0316] Among them, the information acquisition module 141 is used to perform adaptive filtering processing on the cooking audio to obtain the first audio; perform speech enhancement processing on the first audio, and perform noise reduction processing on the first audio after the speech enhancement processing to obtain the second audio.

[0317] The information processing module 142 is used to perform speech recognition based on the second audio, and obtain a recipe corresponding to the cooking audio based on the speech recognition result.

[0318] The recipe making method, device, equipment and medium based on cooking audio provided by the embodiments of the present invention provide a recipe extraction scheme, which realizes accurate identification and targeted processing of different noise sources through noise detection and classification. Comprehensively apply processing methods such as adaptive filtering, speech enhancement, and noise reduction to adapt to different situations, filter noise in different scenes, reduce errors, minimize audio loss, extract high audio quality, and thus ensure the reliability of making structured recipes based on unstructured multimedia audio resources.

[0319] See also Figure 8 , is a structural block diagram of an electronic device 11 for executing a recipe making method based on cooking audio according to an embodiment of the present invention. The components, their connections and relationships, and their functions shown in this embodiment are merely examples and are not intended to limit the implementation of the embodiments of the present invention described and / or required in this embodiment.

[0320] like Figure 8As shown, the electronic device 11 includes a controller 111 and a memory. There is at least one controller 111, and the memory is such as a read-only memory (ROM112), a random access memory (RAM113), etc. Among them, the memory stores computer executable instructions that can be executed by at least one controller 111, and the controller 111 can perform various appropriate actions and processes according to the computer executable instructions stored in the read-only memory (ROM112) or the computer executable instructions loaded from the storage unit to the random access memory (RAM113). In RAM113, various programs and data required for the operation of the electronic device 11 can also be stored. The controller 111, ROM112 and RAM113 are connected to each other via a bus 114. The input / output interface (I / O interface 115) is also connected to the bus 114.

[0321] Multiple components in the electronic device 11 are connected to the I / O interface 115, including: an input unit 116, such as a keyboard, a mouse, a touch screen, etc.; an output unit 117, such as various types of displays, speaker components, etc.; a storage unit 118, such as a disk, an optical disk, etc.; and a communication unit 119, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 119 allows the electronic device 11 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The controller 111 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the controller 111 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The controller 111 performs the various methods and processes described above, such as a recipe making method based on cooking audio.

[0322] In some embodiments, the recipe making method based on cooking audio can be implemented as computer executable instructions, which are tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer executable instructions can be loaded and / or installed on the electronic device 11 via ROM112 and / or communication unit 119. When the computer executable instructions are loaded into RAM113 and executed by the controller 111, one or more steps of the recipe making method based on cooking audio described above can be performed. Alternatively, in other embodiments, the controller 111 can be configured to execute the recipe making method based on cooking audio in any other appropriate manner (for example, by means of firmware).

[0323] Various implementations of the systems and techniques described above can be realized in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof.

[0324] These various embodiments may include being implemented in one or more computer programs that are executable and / or interpretable on a programmable system that includes at least one programmable processor. The programmable processor may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0325] The computer executable instructions for implementing the method of the embodiment of the present invention can be written in any combination of one or more programming languages. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer executable instructions, when executed by the processor, implement the functions / operations specified in the flow chart and / or block diagram. The computer executable instructions can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0326] In the context of embodiments of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device.

[0327] Computer-readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of computer-readable storage media may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory, and the like. Optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0328] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions of the embodiments of the present invention can be achieved, and this document does not limit this.

[0329] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A recipe making method based on cooking audio, characterized in that: include: Performing adaptive filtering on the cooking audio to obtain a first audio; Performing speech enhancement processing on the first audio, and performing noise reduction processing on the first audio after the speech enhancement processing to obtain a second audio; Speech recognition is performed based on the second audio, and a recipe corresponding to the cooking audio is obtained based on the speech recognition result.

2. The recipe making method based on cooking audio according to claim 1, characterized in that: The step of performing adaptive filtering on the cooking audio to obtain the first audio includes: Inputting the cooking audio as an input signal into an adaptive filter, wherein a finite impulse response structure is provided in the adaptive filter so that an impulse response of the adaptive filter is equal to a filter coefficient of the adaptive filter; A speech signal in the input signal is obtained based on the input signal and the filter coefficient analysis, and the speech signal is used as the first audio.

3. The recipe making method based on cooking audio according to claim 2, characterized in that: The method further comprises: Convolving the input signal with the impulse response to obtain an estimated signal; Subtracting the speech signal from the estimated signal to obtain an error signal; generating a correction factor based on the input signal and the error signal; The filter coefficients are updated based on the correction factor.

4. The recipe making method based on cooking audio according to claim 1, characterized in that: The step of performing noise reduction processing on the first audio after the speech enhancement processing to obtain the second audio includes: Performing noise detection based on the first audio after speech enhancement processing, and extracting parameters representing noise characteristics; Determine a noise type based on the parameter, the noise type including additive noise, multiplicative noise, white noise, and Gaussian white noise; When the noise type is additive noise, performing noise reduction processing on the first audio after speech enhancement by using a filter and / or a noise reduction algorithm; In a case where the noise type is multiplicative noise, performing noise reduction processing on the first audio after speech enhancement through signal enhancement processing and / or envelope detection; When the noise type is white noise, performing noise reduction processing on the first audio after speech enhancement by mean filtering and / or spectrum shaping; When the noise type is Gaussian white noise, performing noise reduction processing on the first audio after speech enhancement by Kalman filtering and / or Bayesian filtering; Convert the first audio after noise reduction processing into the time-frequency domain, and perform joint noise reduction processing in the time domain and the frequency domain by Fourier transform method to obtain a second audio; The window size in the Fourier transform method is adaptively obtained based on the signal change frequency of the first audio, and the overlap ratio in the Fourier transform method is adaptively obtained based on the window size.

5. The recipe making method based on cooking audio according to claim 1, characterized in that: The step of performing speech enhancement processing on the first audio comprises: Based on frequency domain features, separate the speech signal and the noise signal in the first audio, and only retain the frequency spectrum component of the speech signal; or input the first audio into a low-pass filter or a high-pass filter for filtering, and remove high-frequency and low-frequency noise in the first audio; The speech signal, or the first audio after high-frequency and low-frequency noises are removed, is subjected to speech signal enhancement processing.

6. The recipe making method based on cooking audio according to claim 1, characterized in that: The performing speech recognition based on the second audio and obtaining a recipe corresponding to the cooking audio based on the speech recognition result includes: Normalizing the second audio, performing feature extraction and speech enhancement processing, to obtain a third audio; Performing semantic recognition and analysis on the third audio to obtain a corresponding recipe text; The recipe text is subjected to word segmentation, entity recognition, grammatical analysis and information extraction to generate a structured recipe.

7. The recipe making method based on cooking audio according to claim 6, characterized in that: The performing semantic recognition and parsing on the third audio to obtain a corresponding recipe text includes: Performing semantic recognition and analysis on the third audio to obtain a speech recognition result; Rewriting the speech recognition result according to the corresponding prompt word template to obtain the corresponding recipe prompt word; Inputting the recipe prompt words into a corresponding generative large language model to add corresponding semantic expansion content to the recipe text; wherein the semantic expansion content includes: cooking ingredient information, cooking step information, recipe background information, cooking tips information, user comment information, and user rating information; Perform integrity check on the semantically expanded recipe text to obtain the final recipe text.

8. A recipe making device based on cooking audio, characterized in that: include: An information acquisition module, used for performing adaptive filtering on the cooking audio to obtain a first audio; Performing speech enhancement processing on the first audio, and performing noise reduction processing on the first audio after the speech enhancement processing to obtain a second audio; An information processing module is used to perform speech recognition based on the second audio, and obtain a recipe corresponding to the cooking audio based on the speech recognition result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the recipe making method based on cooking audio as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the recipe making method based on cooking audio according to any one of claims 1 to 7.

Images