Video playing method and device based on voice control, electronic equipment and medium

Abstract

Embodiments of the present disclosure disclose a voice control-based video playing method and device, electronic equipment and a medium. A specific implementation of the method comprises: collecting a video playing voice signal in a noisy environment to obtain a video playing audio signal; filtering noise from the video playing audio signal to obtain a filtered audio signal; performing noise separation processing on the filtered audio signal to obtain a target audio signal; performing amplitude normalization processing on the target audio signal to obtain a normalized audio signal; performing text conversion processing on the normalized audio signal to obtain audio text information; generating an audio text vector according to the audio text information; performing video retrieval on a target video corresponding to the audio text vector to obtain a video retrieval result; and generating a video playing control instruction according to the audio text information and the video retrieval result to play the video. The implementation can reduce the time for retrieving and playing the video and the error operation for playing the target video.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a voice-controlled video playback method, apparatus, electronic device, and medium. Background Technology

[0002] With the advancement of technology and the improvement of people's living standards, people's demands for video playback are becoming increasingly diversified. This is especially true for large group activities (such as outdoor movie screenings and large concerts), where video playback has almost become an essential component. Currently, the common methods for playing videos are: using a combination of buttons on a handheld remote control to sequentially search for and play videos, or directly capturing user voice and converting it into video playback commands.

[0003] However, when using the above method to play videos, the following technical problems often occur: On the one hand, searching for videos by clicking buttons can be time-consuming or even fail to retrieve the desired video due to the large number of videos stored in the video database. On the other hand, at large group events, the large number of participants and the surrounding noise (such as passing cars) can create significant noise pollution, leading to lower accuracy in user voice recordings and video playback commands, and consequently, more errors in playing the target video. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure provide a voice-controlled video playback method, apparatus, electronic device, and computer-readable medium to address the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide a voice-controlled video playback method, which includes: acquiring video playback audio signals in a noisy environment using an associated voice signal acquisition device to obtain video playback audio signals; performing noise filtering processing on the video playback audio signals to obtain filtered audio signals; performing noise separation processing on the filtered audio signals to obtain target audio signals; performing amplitude normalization processing on the target audio signals to obtain normalized audio signals; performing text conversion processing on the normalized audio signals to obtain audio text information; generating an audio text vector based on the audio text information; performing video retrieval on a target video corresponding to the audio text vector to obtain video retrieval results; and generating video playback control commands to play the video based on the audio text information and the video retrieval results.

[0007] Secondly, some embodiments of this disclosure provide a voice-controlled video playback device, comprising: a data acquisition unit configured to acquire video playback audio signals in a noisy environment via an associated voice signal acquisition device to obtain video playback audio signals; a noise filtering unit configured to perform noise filtering processing on the video playback audio signals to obtain filtered audio signals; a noise separation unit configured to perform noise separation processing on the filtered audio signals to obtain target audio signals; an amplitude normalization unit configured to perform amplitude normalization processing on the target audio signals to obtain normalized audio signals; a text conversion unit configured to perform text conversion processing on the normalized audio signals to obtain audio text information; a first generation unit configured to generate an audio text vector based on the audio text information; a video retrieval unit configured to perform video retrieval on a target video corresponding to the audio text vector to obtain video retrieval results; and a second generation unit configured to generate video playback control instructions for video playback based on the audio text information and the video retrieval results.

[0008] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0009] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0010] The above-described embodiments of this disclosure have the following beneficial effects: The voice-controlled video playback method of some embodiments of this disclosure can reduce erroneous operations when playing target videos. Specifically, the main reasons for numerous erroneous operations when playing target videos are: firstly, searching for videos by pressing buttons can lead to long search times or even failure to find the desired video due to the large number of videos stored in the video database. Secondly, in large-scale group events, the large number of participants and the surrounding noise (such as passing cars) can easily result in significant noise pollution, leading to lower accuracy of the collected user voice and the generated video playback instructions, thus resulting in more erroneous operations when playing target videos. Therefore, the voice-controlled video playback method of some embodiments of this disclosure first collects video playback voice signals in noisy environments using associated voice signal acquisition devices to obtain video playback audio signals. This provides a video playback audio signal representing the user's video playback needs in noisy environments. Then, the video playback audio signal is subjected to noise filtering to obtain a filtered audio signal. Therefore, a filtered audio signal can be obtained, which can remove broadband noise (such as white noise, ambient noise, and device background noise) from the original audio, improving the signal-to-noise ratio. Next, the filtered audio signal undergoes noise separation processing to obtain the target audio signal. This target audio signal allows for the separation of target speech from non-target noise (such as human voice interference, transient noise, and specific frequency band interference), extracting a clean speech signal. Then, the target audio signal undergoes amplitude normalization processing to obtain a normalized audio signal. This normalized audio signal allows the amplitude of the audio signal to be uniformly adjusted to a preset standard range (e.g., between -1.0 and 1.0), eliminating volume differences caused by different sources, devices, and distances. Then, the normalized audio signal undergoes text conversion processing to obtain audio text information. This audio text information allows the preprocessed speech signal to be converted into readable text, completing the mapping from acoustic signals to semantic symbols. Finally, based on the audio text information, an audio text vector is generated. This yields audio text vectors, which convert discrete text strings into continuous, semantically informative numerical vectors (embedding vectors), facilitating the measurement of semantic similarity between texts. Next, video retrieval is performed on the target videos corresponding to the aforementioned audio text vectors, yielding video retrieval results. Thus, using audio text vectors as query criteria, video retrieval results can be retrieved from the video library that best matches the semantics of the user's needs for the target video.Finally, based on the aforementioned audio text information and video retrieval results, video playback control commands are generated to play the video. Thus, by integrating the user's original command text and the retrieved video information, standardized control commands executable by the device can be generated to drive the player to complete video playback. Furthermore, noise filtering of the video playback audio signal removes broadband noise from the original audio, improving the signal-to-noise ratio. Noise separation processing of the filtered audio signal separates the target speech from non-target noise, extracting a clean speech signal. This improves the accuracy of the collected user speech and the accuracy of the generated video playback commands, thereby reducing errors in playing the target video. Simultaneously, this technical solution replaces video retrieval via button presses, lowering the barrier to entry for video playback and reducing the time spent searching for and playing videos. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of a voice-controlled video playback method according to some embodiments of this disclosure; Figure 2 This is a flowchart of some embodiments of the voice-controlled video playback method according to the present disclosure; Figure 3 This is a schematic diagram of dilated convolution for a voice-controlled video playback method according to this disclosure; Figure 4 This is a schematic diagram of a video retrieval method based on the voice-controlled video playback method disclosed herein; Figure 5 This is a schematic diagram of the structure of some embodiments of a voice-controlled video playback device according to the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a voice-controlled video playback method according to some embodiments of this disclosure.

[0020] exist Figure 1 In the application scenario, firstly, the computing device 101 can acquire data from the video playback audio signal in a noisy environment through the associated voice signal acquisition device 102 to obtain the video playback audio signal 103; then, the computing device 101 can perform noise filtering processing on the video playback audio signal 103 to obtain the filtered audio signal 104; subsequently, the computing device 101 can perform noise separation processing on the filtered audio signal 104 to obtain the target audio signal 105; secondly, the computing device 101 can perform amplitude normalization processing on the target audio signal 105 to obtain the normalized... The computing device 101 performs text conversion processing on the normalized audio signal 106 to obtain audio text information 107. Then, the computing device 101 generates an audio text vector 108 based on the audio text information 107. Next, the computing device 101 performs video retrieval on the target video corresponding to the audio text vector 108 to obtain a video retrieval result 109. Finally, the computing device 101 generates a video playback control instruction 110 to play the video based on the audio text information 107 and the video retrieval result 109.

[0021] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here. It should be understood that... Figure 1 The number of computing devices in the system can be arbitrary, depending on the implementation requirements.

[0022] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of a voice-controlled video playback method according to the present disclosure. This voice-controlled video playback method includes the following steps: Step 201: Collect video playback audio signals from a noisy environment using an associated audio signal acquisition device to obtain video playback audio signals.

[0023] In some embodiments, the executor of the voice-controlled video playback method (e.g.) Figure 1 The computing device 101 shown can acquire video playback audio signals from noisy environments using an associated voice signal acquisition device. The voice signal acquisition device can be a device capable of acquiring voice signals from a user. For example, the voice signal acquisition device can be a directional microphone. The user is not specifically limited here. For example, the user can be any user who needs to acquire video playback voice signals. The video playback audio signal can be an audio signal acquired by the voice signal acquisition device in a noisy environment, corresponding to the user's video playback needs. For example, the video playback audio signal can be an audio signal of "Playing the documentary 'XX'" acquired by a directional microphone in an outdoor movie screening scenario.

[0024] Step 202: Perform noise filtering on the audio signal of the video playback to obtain the filtered audio signal.

[0025] In some embodiments, the aforementioned execution entity may perform noise filtering processing on the aforementioned video playback audio signal to obtain a filtered audio signal.

[0026] As an example, the aforementioned execution entity can use the Wiener filtering algorithm to filter noise from the audio signal played in the video playback, thereby obtaining a filtered audio signal.

[0027] As another example, the aforementioned execution entity can also perform noise filtering on the video playback audio signal using a wavelet threshold denoising algorithm to obtain a filtered audio signal.

[0028] In some optional implementations of certain embodiments, the aforementioned execution entity may perform noise filtering processing on the aforementioned video playback audio signal through the following steps to obtain a filtered audio signal: The first step is to generate an audio amplitude spectrum sequence and an audio phase spectrum sequence based on the audio signal played from the video. The audio amplitude spectrum in the amplitude spectrum sequence and the audio phase spectrum in the phase spectrum sequence correspond one-to-one. In practice, the execution entity can generate the audio amplitude spectrum sequence and the audio phase spectrum sequence through the following steps: The first sub-step involves amplifying the video playback audio signal to obtain an amplified audio signal. In practice, the execution entity can use an operational amplifier or a programmable gain amplifier (PGA) to amplify the video playback audio signal to obtain the amplified audio signal.

[0029] The second sub-step involves performing anti-aliasing filtering on the amplified audio signal to obtain an anti-aliasing filtered signal. In practice, the execution entity can use an active low-pass filter to perform anti-aliasing filtering on the amplified audio signal to obtain the anti-aliasing filtered signal.

[0030] The third sub-step involves performing analog-to-digital conversion (ADC) on the anti-aliasing filtered signal to obtain a digital audio signal. In practice, the execution entity can use a delta-sigma (Δ-Σ) type ADC to perform ADC on the anti-aliasing filtered signal to obtain a digital audio signal.

[0031] The fourth sub-step involves framing the aforementioned digital audio signal to obtain a framed audio signal sequence. In practice, the executing entity can use a preset frame length threshold as the frame length and a preset frame shift threshold as the frame shift to perform framing on the digital audio signal, thereby obtaining a framed audio signal sequence. The preset frame length threshold can be a pre-defined frame length threshold. The preset frame shift threshold can also be a pre-defined frame shift threshold. For example, the preset frame length threshold could be 25ms, and the preset frame shift threshold could be 10ms.

[0032] The fifth sub-step involves windowing the aforementioned framed audio signal sequence to obtain a windowed audio signal sequence. In practice, the executing entity can utilize window functions to window the framed audio signal sequence to obtain the windowed audio signal sequence. These window functions can include, but are not limited to, any of the following: rectangular window, Hamming window, Hanning window, and Blackman window.

[0033] The sixth sub-step involves performing a frequency domain transformation on the windowed audio signal sequence to obtain an audio amplitude spectrum sequence and an audio phase spectrum sequence. The audio amplitude spectrum in the amplitude spectrum sequence and the audio phase spectrum in the phase spectrum sequence correspond one-to-one. In practice, the execution entity can use a Fast Fourier Transform to perform the frequency domain transformation on the windowed audio signal sequence to obtain the audio amplitude spectrum sequence and the audio phase spectrum sequence.

[0034] The second step involves performing the following processing steps for each audio amplitude spectrum in the above audio amplitude spectrum sequence: The first processing step is to determine the audio type of the audio signal corresponding to the aforementioned audio amplitude spectrum. The audio type may include noise and speech. The speech may be a video playback voice signal emitted by the user corresponding to the user's video playback needs. The noise may be a video playback voice signal acquired by the voice signal acquisition device that does not correspond to the user's video playback needs. In practice, the executing entity can determine the audio type of the audio signal corresponding to the aforementioned audio amplitude spectrum through the following steps: The first determining step involves identifying the number of each frequency point included in the aforementioned audio amplitude spectrum as the frequency point quantity information. This frequency point quantity information can be equivalent to the aforementioned preset frame length threshold.

[0035] The second determining step is to determine the total frequency domain energy by summing the squares of the amplitude spectra corresponding to each frequency point included in the above audio amplitude spectrum.

[0036] The third determination step is to determine the ratio of the total energy in the frequency domain to the number of frequency points as the average energy in the frequency domain.

[0037] The fourth determination step involves determining that, in response to the determination that the average energy in the frequency domain is greater than a preset energy threshold, the audio type of the audio signal corresponding to the audio amplitude spectrum is determined to be speech. The preset energy threshold can be a pre-defined energy threshold.

[0038] The fifth determination step involves determining the audio type of the audio signal corresponding to the audio amplitude spectrum as noise in response to determining that the average energy in the frequency domain is less than or equal to the preset energy threshold.

[0039] The second processing step, in response to determining that the audio signal corresponding to the aforementioned audio amplitude spectrum is noise, updates the noise power spectrum corresponding to the aforementioned audio amplitude spectrum sequence using the aforementioned audio amplitude spectrum, obtaining the updated noise power spectrum as the noise power spectrum, and determining the aforementioned audio amplitude spectrum as the target audio amplitude spectrum. The initial noise power spectrum corresponding to the aforementioned audio amplitude spectrum sequence can be obtained by acquiring noise signals before the user sends a video playback voice signal. Specifically, firstly, noise signals within a corresponding preset time period (e.g., 250ms) can be acquired using a directional microphone. Then, the noise signals are subjected to frame segmentation and windowing processing to obtain a windowed noise signal sequence. Next, a Fast Fourier Transform is performed on each windowed noise signal in the windowed noise signal sequence to obtain a noise amplitude spectrum. The noise amplitude spectrum is squared to obtain a noise power spectrum, and the average value of the obtained noise power spectra is determined as the initial noise power spectrum.

[0040] In practice, the aforementioned executing entity can use the aforementioned audio amplitude spectrum to update the noise power spectrum corresponding to the aforementioned audio amplitude spectrum sequence through the following steps, thereby obtaining the updated noise power spectrum as the noise power spectrum: The first update step involves determining the first noise power spectrum by multiplying a preset smoothing factor by the noise power spectrum corresponding to the aforementioned audio amplitude spectrum sequence. The preset smoothing factor can be a pre-defined smoothing factor. This preset smoothing factor controls the update rate of the noise power spectrum. For example, the preset smoothing factor can be 0.95.

[0041] The second update step involves determining the difference between the preset value and the aforementioned preset smoothing factor as the audio amplitude spectrum weight. The preset value can be a pre-defined numerical value. For example, the preset value can be 1.

[0042] The third update step is to determine the second noise power spectrum by multiplying the above-mentioned audio amplitude spectrum weights and the above-mentioned audio amplitude spectrum.

[0043] The fourth update step is to determine the sum of the first noise power spectrum and the second noise power spectrum as the updated noise power spectrum.

[0044] The fifth update step is to determine the above-mentioned updated noise power spectrum as the noise power spectrum in order to update the above-mentioned noise power spectrum.

[0045] The third processing step, in response to determining that the audio signal corresponding to the aforementioned audio amplitude spectrum is speech, involves performing spectral enhancement processing on the aforementioned audio amplitude spectrum using the aforementioned noise power spectrum to obtain an enhanced amplitude spectrum as the target audio amplitude spectrum. Specifically, the frequency point amplitude spectra in the aforementioned audio amplitude spectrum correspond one-to-one with the frequency point noise power spectra in the aforementioned noise power spectrum. In practice, the aforementioned execution entity can perform spectral enhancement processing on the aforementioned audio amplitude spectrum using the aforementioned noise power spectrum to obtain the enhanced amplitude spectrum as the target audio amplitude spectrum through the following steps: The first enhancement step involves performing the following processing steps for the amplitude spectrum at each frequency point in the above audio amplitude spectrum: The first processing step involves determining a first preset enhancement factor as the target enhancement factor in response to the determination that the frequency corresponding to the amplitude spectrum of the aforementioned frequency point satisfies a first frequency condition. The first frequency condition can be a frequency less than a first preset frequency threshold or greater than a second preset frequency threshold. Both the first and second preset frequency thresholds are pre-set frequency thresholds. For example, the first preset frequency threshold can be 1 kHz, and the second preset frequency threshold can be 8 kHz. The first preset enhancement factor can be a pre-set enhancement factor. For example, the first preset enhancement factor can be 1.2. In practice, the frequency corresponding to the amplitude spectrum of the aforementioned frequency point can be obtained using the following formula: .

[0046] Among them, the above This can represent the frequency corresponding to the amplitude spectrum of the above frequency points. The above... This can represent a frequency point index. The N mentioned above can represent a preset frame length threshold. This can represent a preset sampling frequency. For example, the preset sampling frequency mentioned above could be 16000Hz.

[0047] The second processing step involves determining a second preset enhancement factor as the target enhancement factor in response to the determination that the frequency corresponding to the amplitude spectrum of the aforementioned frequency point satisfies a second frequency condition. The second frequency condition can be a frequency greater than or equal to the first preset frequency threshold and less than or equal to the second preset frequency threshold. The second preset enhancement factor can be a pre-set enhancement factor. For example, the second preset enhancement factor can be 0.8.

[0048] The third processing step is to determine the noise power spectrum at the frequency point that corresponds to the frequency point amplitude spectrum in the noise power spectrum as the target frequency point noise power spectrum.

[0049] The fourth processing step is to determine the enhanced frequency noise power spectrum by multiplying the target enhancement factor and the target frequency noise power spectrum.

[0050] The fifth processing step is to determine the difference between the above frequency point amplitude spectrum and the above enhanced frequency point noise power spectrum as the enhanced frequency point amplitude spectrum.

[0051] The second enhancement step involves determining the amplitude spectrum of each enhancement frequency point as the enhancement amplitude spectrum.

[0052] The fourth processing step involves generating a time-domain signal based on the target audio amplitude spectrum and the corresponding audio phase spectrum. In practice, the executing entity can obtain the time-domain signal by performing an inverse Fourier transform on the target audio amplitude spectrum and the corresponding audio phase spectrum.

[0053] The third step involves overlapping and adding the obtained time-domain signals to obtain the overlapped and added time-domain signals, which serve as the filtered audio signal. In practice, the aforementioned execution entity can call a preset overlap and addition function interface to perform overlap and addition on the obtained time-domain signals, resulting in the overlapped and added time-domain signals used as the filtered audio signal. This preset overlap and addition function interface can be a pre-encapsulated function for overlapping and adding the various time-domain signals. Here, overlap and addition can be understood as placing the various time-domain signals according to a set frame shift (preset frame shift threshold), weighting and superimposing them in the overlapping region, and finally normalizing to eliminate the influence of the window function, thereby reconstructing a continuous, smooth, and complete time-domain signal.

[0054] Step 203: Perform noise separation processing on the filtered audio signal to obtain the target audio signal.

[0055] In some embodiments, the aforementioned execution entity may perform noise separation processing on the filtered audio signal to obtain the target audio signal.

[0056] In practice, the aforementioned execution entity can use the Blind Source Separation (BSS) algorithm to perform noise separation processing on the filtered audio signal to obtain the target audio signal. As another example, the aforementioned execution entity can also use the Independent Component Analysis (ICA) algorithm to perform noise separation processing on the filtered audio signal to obtain the target audio signal.

[0057] In some optional implementations of certain embodiments, the aforementioned execution entity may perform noise separation processing on the filtered audio signal through the following steps to obtain the target audio signal: The first step is to extract features from the filtered audio signal to obtain audio features. In practice, the execution entity can use a feature extraction module to extract features from the filtered audio signal to obtain audio features. This feature extraction module may include a one-dimensional convolutional layer, a batch normalization layer, and a LeakyReLU layer. This transforms the time-domain filtered audio signal into a high-dimensional feature space, allowing for the extraction of acoustic features beneficial for subsequent processing.

[0058] The second step involves downsampling the aforementioned audio features to obtain downsampled audio features. This reduces the temporal resolution of the features, expands the receptive field, and allows for the extraction of higher-level, more abstract acoustic features.

[0059] The third step involves performing feature enhancement processing on the downsampled audio features to obtain enhanced audio features, which are then used as the first enhanced audio features. This allows for nonlinear transformation and enhancement of abstract features, highlighting speech-related features while suppressing noise features.

[0060] The fourth step involves generating an enhanced audio feature sequence based on the first enhanced audio feature mentioned above. This enhanced audio feature sequence may include the first enhanced audio feature. In practice, the execution entity generates each enhanced audio feature in the enhanced audio feature sequence in the same way as it generates the first enhanced audio feature (i.e., through feature extraction, downsampling, and feature enhancement via a feature extraction module). Here, each enhanced audio feature can be generated sequentially, layer by layer. The previous enhanced audio feature serves as the input parameter, and the subsequent enhanced audio feature serves as the output of the previous enhanced audio feature. This yields enhanced audio feature sequences representing different downsampling stages, enabling the construction of a multi-scale feature pyramid. Here, shallow features are used to preserve details (such as transients and harmonics), while deep features are used to preserve structure (such as syllables and pauses).

[0061] The fifth step is to determine the enhanced audio feature corresponding to the last position in the above enhanced audio feature sequence as the target enhanced audio feature. This yields the target enhanced audio feature that represents the largest receptive field and the most abstract feature representation.

[0062] Step 6: Perform convolution processing on the aforementioned target enhanced audio features using a first dilation rate, a second dilation rate, and a third dilation rate, respectively, to obtain the first dilated convolution feature, the second dilated convolution feature, and the third dilated convolution feature. The first dilation rate can represent a dilated convolution with 1 dilation point. The second dilation rate can represent a dilated convolution with 2 dilation points. The third dilation rate can represent a dilated convolution with 3 dilation points. A diagram illustrating dilated convolution (dilated convolution) can be found here. Figure 3 , Figure 3In the diagram, the gray area refers to the input image, the orange area refers to the 3x3 convolution kernel, and the green area represents the receptive field. Figure 3 In the diagram, a) represents a normal convolution, also known as a dilated convolution with 1 dilation point; b) represents a dilated convolution with 2 dilation points; and c) represents a dilated convolution with 3 dilation points. The dilation rate can be the spacing between elements in the convolution kernel. When the dilation rate is 1, the dilated convolution is a normal convolution. Dilated convolution controls the spacing between elements in the convolution kernel through the dilation rate, thereby changing the size of the receptive field. The first dilation rate is used to capture local detail features, the second dilation rate is used to capture medium-range context, and the third dilation rate is used to capture long-range dependencies. Therefore, by setting different dilation rates in parallel, the model can simultaneously focus on patterns at short, medium, and long time scales, resulting in stronger noise discrimination.

[0063] Step 7: The first, second, and third dilated convolutional features are concatenated to obtain concatenated dilated convolutional features. Here, the concatenation process can be channel-wise concatenation. This allows for the merging of features from three different receptive fields along the channel dimension, achieving multi-scale information fusion. This enables feature complementarity and enhances feature representation capabilities.

[0064] Step 8: Perform channel compression processing on the above-mentioned concatenated dilated convolution features to obtain channel-compressed convolution features. In practice, the execution entity can perform one-dimensional convolution processing on the above-mentioned concatenated dilated convolution features to obtain one-dimensional concatenated dilated convolution features as channel-compressed convolution features. This reduces the number of channels after concatenation, lowers the computational load, and extracts the essential features after fusion.

[0065] The ninth step involves upsampling the compressed convolutional features from the aforementioned channels to obtain upsampled convolutional features. This upsampling process can be achieved through linear interpolation. This allows the compressed features to be restored to a higher temporal resolution, gradually reconstructing temporal details.

[0066] Step 10: Combine the upsampled convolutional features with the enhanced audio features corresponding to the upsampled convolutional features in the enhanced audio feature sequence to obtain combined audio features. Here, the combination can be achieved through concatenation. The correspondence can be a data dimension correspondence. As an example, the enhanced audio feature corresponding to the upsampled convolutional features in the enhanced audio feature sequence can be the second-to-last enhanced audio feature in the sequence. Thus, the encoder's detailed features (shallow layer) and the decoder's abstract features (deep layer) can be fused.

[0067] The eleventh step involves feature extraction processing of the combined audio features to obtain upsampled audio features. In practice, the execution entity can use the feature extraction module to perform feature extraction processing on the combined audio features to obtain upsampled audio features. This allows for further nonlinear transformation of the fused features, making them more suitable for the final output.

[0068] The twelfth step involves performing feature enhancement processing on the upsampled audio features to obtain upsampled enhanced features. Here, the method by which the execution entity performs feature enhancement processing on the upsampled audio features is the same as the method for performing feature enhancement processing on the downsampled audio features, and will not be repeated here. This allows for the refinement of the decoder's intermediate features, enhancing the quality of feature reconstruction.

[0069] Step 13: Based on the aforementioned upsampling enhancement features and the aforementioned enhanced audio feature sequence, generate the target audio feature. In practice, firstly, the execution entity can upsample the aforementioned upsampling enhancement features to obtain upsampled features. Then, the upsampled features and the enhanced audio features corresponding to the aforementioned upsampled features in the aforementioned enhanced audio feature sequence can be concatenated to obtain concatenated audio features. Next, the aforementioned feature extraction module is used to extract features from the concatenated audio features and enhance the extracted concatenated audio features to obtain enhanced features as upsampled enhancement features. The above steps are then repeated layer by layer, and the result of concatenation, feature extraction, and feature enhancement with the first enhanced audio feature in the aforementioned enhanced audio feature sequence is used as the target audio feature. Thus, by fusing the features of each encoder level with its corresponding decoder features, complete information flow from global to local is achieved, completing full-scale feature fusion.

[0070] Step fourteen involves performing a separation and mapping process on the aforementioned target audio features to obtain the target audio signal. In practice, firstly, the executing entity can perform a one-dimensional convolution on the target audio features to obtain convolutional target audio features. Then, the convolutional target audio features can be input into a third preset activation function to obtain the target audio signal. The third preset activation function can be a pre-defined activation function. Here, the third preset activation function can be a Tanh activation function. The target audio signal can represent the noise-removed video playback speech signal emitted by the user. Thus, the high-dimensional features can be mapped back to a one-dimensional time-domain audio waveform, outputting the final clean speech signal.

[0071] In practice, the processing of data features often employs unified processing (global processing) and single-dimensional (e.g., time-based) approaches, resulting in low accuracy in extracting effective features. To address this technical problem, the following technical solution is proposed.

[0072] In practice, the aforementioned executing entity can perform feature enhancement processing on the downsampled audio features using the following steps to obtain enhanced audio features as the first enhanced audio feature: The first step is to perform convolution processing on the downsampled audio features to obtain a spatial attention map. In practice, the execution entity can use convolutional layers to process the downsampled audio features to obtain the spatial attention map. Here, the size of the convolutional kernel corresponding to the convolutional layer is 1*1.

[0073] The second step involves generating spatial attention features based on the aforementioned spatial attention map, the aforementioned inverted multi-scale aggregated features, and the first preset activation function. In practice, the executing entity can first input the aforementioned spatial attention map into the aforementioned first preset activation function to obtain spatial attention activation features. Then, the Hadamard product (element-wise multiplication along the channel dimension) of the aforementioned spatial attention activation features and the aforementioned inverted multi-scale aggregated features can be used to determine the spatial attention features. The aforementioned first preset activation function can be a pre-defined activation function. Here, the aforementioned first preset activation function can be the Sigmoid activation function.

[0074] The third step involves generating max-pooling and average-pooling feature vectors based on the aforementioned spatial attention features. In practice, firstly, the executing entity can perform global max-pooling on the spatial attention features to obtain the max-pooling feature vector. Then, it can perform global average-pooling on the spatial attention features to obtain the average-pooling feature vector.

[0075] The fourth step involves performing convolution processing on the max-pooling feature vector and the average-pooling feature vector, respectively, to obtain the first max-pooling feature vector and the first average-pooling feature vector. In practice, the execution entity can use convolutional layers to perform convolution processing on the max-pooling feature vector and the average-pooling feature vector, respectively, to obtain the first max-pooling feature vector and the first average-pooling feature vector. Here, the size of the convolutional kernel corresponding to the above convolutional layer can be 1*1. Here, 1*1 convolution processing is mainly used to change the number of feature channels, making the number of channels halved.

[0076] Fifth, based on the first maximum convolution feature vector, the first average convolution feature vector, and the second preset activation function, generate the maximum activation feature vector and the average activation feature vector. In practice, firstly, the execution entity can input the first maximum convolution feature vector into the second preset activation function to obtain the maximum activation feature vector. Then, it can input the first average convolution feature vector into the second preset activation function to obtain the average activation feature vector. The second preset activation function can be a pre-defined activation function. Here, the second preset activation function can be the ReLU activation function.

[0077] Step 6: Perform convolution processing on the aforementioned maximum activation feature vector and the aforementioned average activation feature vector respectively to obtain the second maximum convolution feature vector and the second average convolution feature vector. In practice, the execution entity can use convolutional layers to perform convolution processing on the aforementioned maximum activation feature vector and the aforementioned average activation feature vector respectively to obtain the second maximum convolution feature vector and the second average convolution feature vector. Here, the size of the convolution kernel corresponding to the aforementioned convolutional layer can be 1*1. Here, the aforementioned convolution processing is mainly used to restore the number of feature channels.

[0078] Step 7: Combine the second maximum convolution feature vector and the second average convolution feature vector to obtain a combined convolution feature vector. In practice, the execution entity can perform element-wise addition of the second maximum convolution feature vector and the second average convolution feature vector to obtain the combined convolution feature vector.

[0079] Step 8: Based on the combined convolutional feature vector and the first preset activation function, generate channel attention features. In practice, the execution entity can input the combined convolutional feature vector into the first preset activation function to obtain the channel attention features.

[0080] Step 9: Based on the aforementioned channel attention features and downsampled audio features, generate enhanced audio features as the first enhanced audio feature. In practice, the executing entity can determine the Hadamard product of the aforementioned channel attention features and downsampled audio features as the enhanced audio feature as the first enhanced audio feature.

[0081] To address the aforementioned technical challenges, firstly, by acquiring the spatial attention features of the audio features, the start and end positions of speech on the time axis can be captured, allowing for dynamic focusing of useful frequency band signals. Then, by acquiring the channel attention features of the audio features, it is possible to delve deeper into the features and distinguish which channels are primarily speech and which are primarily noise. Furthermore, adopting a spatial-first, channel-second order better aligns with the physical characteristics of speech signals, thereby improving the accuracy of extracted effective features.

[0082] Step 204: Perform amplitude normalization processing on the target audio signal to obtain a normalized audio signal.

[0083] In some embodiments, the execution entity may perform amplitude normalization processing on the target audio signal to obtain a normalized audio signal. The amplitude normalization processing may include peak value normalization, standard deviation normalization, and root mean square normalization.

[0084] As an example, the aforementioned execution entity can perform peak normalization processing on the aforementioned target audio signal to obtain a normalized audio signal.

[0085] As another example, the aforementioned execution entity can also perform standard deviation normalization on the aforementioned target audio signal to obtain a normalized audio signal.

[0086] Step 205: Perform text conversion processing on the normalized audio signal to obtain audio text information.

[0087] In some embodiments, the aforementioned execution entity may perform text conversion processing on the aforementioned normalized audio signal to obtain audio text information.

[0088] In practice, the aforementioned execution entity can input the normalized audio signal into a pre-trained audio-text information generation model to obtain audio-text information. This audio-text information generation model can include an encoder layer, a prediction network layer, and a joint network layer. The encoder layer can be a network layer that extracts high-dimensional features from the input audio signal. The prediction network layer can be a network layer that predicts the next text based on the already output text information. The joint network layer can be a network layer that fuses the acoustic features output from the encoder layer and the linguistic features output from the prediction network layer to generate audio-text information. For example, the audio-text information generation model can be an RNN-T (Recurrent Neural Network Transducer) network model.

[0089] In practice, due to the presence of many homophones or near-homophones in daily use of Chinese, there are significant deviations in the semantic recognition of speech during speech-to-text conversion (for example, "Gaocheng" is easily converted into "gaocheng" or "gaolou").

[0090] In some optional implementations of certain embodiments, the aforementioned execution entity may perform text conversion processing on the normalized audio signal through the following steps to obtain audio text information: The first step is to encode the normalized audio signal to obtain an acoustic feature vector. In practice, the execution entity can use an encoding layer to encode the normalized audio signal to obtain the acoustic feature vector. This encoding layer can be a network layer capable of deep acoustic feature extraction from the input vector. For example, the encoding layer can be a self-attention layer or a Conformer layer.

[0091] The second step involves performing acoustic prediction processing on the aforementioned acoustic feature vectors to obtain acoustic embedding vectors. In practice, the execution entity can input the acoustic feature vectors into the predictor layer to obtain the acoustic embedding vectors. This predictor layer can be a CIF Predictor-based predictor layer. The predictor layer may include two layers of feedforward neural networks (FFN). This predictor layer is used to predict the number of target characters and extract the corresponding acoustic vectors for the target characters.

[0092] The third step involves decoding the acoustic embedding vector to obtain the first decoded vector. In practice, the execution entity can decode the acoustic embedding vector using a first decoding layer to obtain the first decoded vector. This first decoding layer may include: a SAN-M (Self-Attention with Memory) module, an FFN (Feed-Forward Network) module, and an MHA (Multi-Head Attention) module. The SAN-M module enhances the ability to model long sequences. The FFN module performs nonlinear transformations. The MHA module fuses the acoustic features output by the encoder.

[0093] The fourth step involves embedding the predefined domain word sequence to obtain a context embedding vector. In practice, firstly, the execution entity can perform embedding processing on the predefined domain word sequence to obtain a dense embedding vector. Then, this dense embedding vector can be input into a temporal feature extraction network to obtain a context embedding vector. The predefined domain word sequence can be a pre-defined video domain word sequence. The specific definition of the predefined domain word sequence is not limited here. For example, it could include the names of various videos (e.g., movies). The temporal feature extraction network can be a network capable of extracting temporal features from the input vector. For example, it could be an LSTM (Long Short-Term Memory) network.

[0094] Fifth, based on the aforementioned acoustic feature vector and the first decoding vector, a semantic attention vector is generated. In practice, the executing entity can input the aforementioned acoustic feature vector and the first decoding vector into a multi-head attention network to obtain the semantic attention vector.

[0095] Step 6: Generate a context attention vector based on the aforementioned context embedding vector and the aforementioned first decoding vector. In practice, the executing entity can input the aforementioned context embedding vector and the aforementioned first decoding vector into a multi-head attention network to obtain the context attention vector.

[0096] Step 7: Concatenate the semantic attention vector and the context attention vector to obtain the fused attention vector. In practice, firstly, the executing entity can concatenate the semantic attention vector and the context attention vector through channels to obtain the concatenated attention vector. Then, the executing entity can perform a one-dimensional convolution on the concatenated attention vector to obtain the fused attention vector. Here, the one-dimensional convolution is used to unify the feature dimensions.

[0097] Step 8: Decode the fused attention vector to obtain the decoded vector. In practice, the execution entity can decode the fused attention vector using a second decoding layer to obtain the decoded vector. The network structure of the second decoding layer can be the same as that of the first decoding layer.

[0098] The ninth step is to generate audio text information based on the decoded vectors described above. In practice, firstly, the execution entity can input the decoded vectors into a fully connected layer to obtain a fully connected vector. Then, the fully connected vector can be input into the Softmax function to obtain the audio text information.

[0099] To address the technical challenge of significant semantic discrepancies in speech-to-text conversion, the following steps are taken: First, the normalized audio signal is encoded to obtain an acoustic feature vector. This allows for deep feature extraction of the audio signal, resulting in a more robust acoustic feature vector. Next, acoustic prediction processing is performed on the acoustic feature vector to obtain an acoustic embedding vector. This acoustic embedding vector enables soft alignment in speech prediction and length prediction of the target text. Then, the acoustic embedding vector is decoded to obtain a first decoded vector, representing the initial semantic latent state. Next, a predefined domain word sequence is embedded to obtain a context embedding vector. This context embedding vector allows the conversion of discrete domain words into dynamic contextual representations. Then, a semantic attention vector is generated based on the acoustic feature vector and the first decoded vector. This semantic attention vector allows for the assignment of higher weights to key audio components. Finally, a context attention vector is generated based on the context embedding vector and the first decoded vector. Therefore, a contextual attention vector can be obtained, which can assign higher weights to the target text content in the preset domain word sequence. Next, the semantic attention vector and the contextual attention vector are concatenated to obtain a fused attention vector. This fused attention vector can fuse features representing acoustic information with domain words representing prior features, improving the accuracy of speech-to-semantic conversion. Then, the fused attention vector is decoded to obtain a decoded vector. This decoded vector represents the updated domain information decoding hidden state. Finally, audio text information is generated based on the decoded vector. This yields audio text information biased by domain words. Because introducing a preset domain word sequence representing prior features during speech-to-text conversion improves the accuracy of speech recognition and reduces semantic bias, and because introducing an attention mechanism to assign greater weights to key speech parts and key domain words further improves the accuracy of speech recognition and reduces semantic bias, this process further enhances speech recognition accuracy.

[0100] Step 206: Generate an audio text vector based on the audio text information.

[0101] In some embodiments, the aforementioned execution entity may generate an audio text vector based on the aforementioned audio text information.

[0102] In practice, the aforementioned execution entity can convert audio text information into audio text vectors using a bag-of-words model. As an example, the execution entity can input the audio text information into a pre-trained embedding model to obtain audio text vectors. The pre-trained embedding model can be the all-MiniLM-L6-v2 model. As another example, the execution entity can also input the audio text information into a BERT model to obtain audio text vectors.

[0103] Step 207: Perform video retrieval on the target video corresponding to the audio text vector to obtain the video retrieval results.

[0104] In some embodiments, the execution entity may perform video retrieval on the target video corresponding to the audio text vector to obtain video retrieval results.

[0105] In practice, the aforementioned execution entity can use audio text vectors as query conditions to retrieve target videos from a local video database and obtain video retrieval results. Specifically, firstly, the similarity between the vectors corresponding to each video in the local video database (such as the vector corresponding to the video name or the vector corresponding to the video keywords) and the aforementioned audio text vectors can be compared. Then, videos with a similarity greater than a preset similarity threshold can be identified as video retrieval results. Here, the aforementioned similarity can be cosine similarity. The aforementioned preset similarity threshold can be a pre-defined similarity threshold. For example, the aforementioned preset similarity threshold can be 0.85.

[0106] As another example, the aforementioned executing entity can send the audio text vector to a cloud-based video database to retrieve the target video and obtain the video retrieval results. The cloud-based video database can be a server connected to the executing entity via a network, capable of downloading and playing videos. For a detailed diagram of the video retrieval method, please refer to... Figure 4 .

[0107] Step 208: Generate video playback control instructions based on the audio text information and video retrieval results to play the video.

[0108] In some embodiments, the execution entity may generate video playback control instructions to play the video based on the audio text information and the video retrieval results.

[0109] In practice, firstly, the aforementioned execution entity can input the aforementioned audio text information and video retrieval results into a large language model to obtain the video playback result. The aforementioned semantic large model can be Qwen-1.8B. The aforementioned video playback result can be video information corresponding to the user's playback requirements. Specifically, the output format of the aforementioned video playback result can be structured JSON. For example, the aforementioned video playback result can be {"action":"play", "film_id":"JYL_001", "film_name":"XX", "confidence":0.96, "need_confirm":false}. Then, the video playback result can be parsed into video playback instructions through the instruction parsing module. Finally, the video playback instructions can be input into the video playback module to complete the video playback operation. The aforementioned instruction parsing module can be a module capable of converting the video playback result into standardized instruction codes that the device can execute. As an example, the aforementioned instruction parsing module can convert the video playback result into standardized instruction codes that the device can execute based on the function instruction mapping table stored in the aforementioned local video database. For example, the aforementioned instruction parsing module can convert "action":"play" into CMD_PLAY. "film_id":"JYL_001" can be converted to / films / JYL_001.mp4. The video playback command above can be {player.load(" / films / JYL_001.mp4");player.play()}. The above video playback module is used to call the underlying player component, load the specified movie file, and perform video playback operations.

[0110] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a voice-controlled video playback device, which are similar to... Figure 2 Corresponding to the method embodiments shown, this voice-controlled video playback device can be specifically applied to various electronic devices.

[0111] like Figure 5As shown, some embodiments of the voice-controlled video playback device 500 include: a data acquisition unit 501, a noise filtering unit 502, a noise separation unit 503, an amplitude normalization unit 504, a text conversion unit 505, a first generation unit 506, a video retrieval unit 507, and a second generation unit 508. The data acquisition unit 501 is configured to acquire video playback audio signals in noisy environments through an associated voice signal acquisition device to obtain video playback audio signals; the noise filtering unit 502 is configured to perform noise filtering processing on the video playback audio signals to obtain filtered audio signals; the noise separation unit 503 is configured to perform noise separation processing on the filtered audio signals to obtain target audio signals; the amplitude normalization unit 504 is configured to perform amplitude normalization processing on the target audio signals to obtain normalized audio signals; the text conversion unit 505 is configured to perform text conversion processing on the normalized audio signals to obtain audio text information; the first generation unit 506 is configured to generate audio text vectors based on the audio text information; the video retrieval unit 507 is configured to perform video retrieval on the target video corresponding to the audio text vectors to obtain video retrieval results; and the second generation unit 508 is configured to generate video playback control commands for video playback based on the audio text information and the video retrieval results.

[0112] It is understandable that the units described in the voice-controlled video playback device 500 are similar to those in the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the voice-controlled video playback device 500 and the units contained therein, and will not be repeated here.

[0113] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0114] like Figure 6 As shown, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 into a random access memory 603. The random access memory 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0115] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0116] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0117] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0118] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0119] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire data from a video playback audio signal in a noisy environment using an associated voice signal acquisition device to obtain a video playback audio signal; perform noise filtering processing on the video playback audio signal to obtain a filtered audio signal; perform noise separation processing on the filtered audio signal to obtain a target audio signal; perform amplitude normalization processing on the target audio signal to obtain a normalized audio signal; perform text conversion processing on the normalized audio signal to obtain audio text information; generate an audio text vector based on the audio text information; perform video retrieval on the target video corresponding to the audio text vector to obtain video retrieval results; and generate video playback control instructions to play the video based on the audio text information and the video retrieval results.

[0120] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0123] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A video playback method based on voice control, characterized in that, include: The video playback audio signal is obtained by acquiring the video playback audio signal in a noisy environment using associated voice signal acquisition equipment. The audio signal of the video playback is subjected to noise filtering to obtain a filtered audio signal; The filtered audio signal is subjected to noise separation processing to obtain the target audio signal; The target audio signal is subjected to amplitude normalization processing to obtain a normalized audio signal; The normalized audio signal is subjected to text conversion processing to obtain audio text information; Based on the audio text information, generate an audio text vector; Perform video retrieval on the target video corresponding to the audio text vector to obtain video retrieval results; Based on the audio text information and the video retrieval results, a video playback control command is generated to play the video.

2. The method according to claim 1, characterized in that, The noise filtering process performed on the video playback audio signal to obtain the filtered audio signal includes: Based on the audio signal played from the video, an audio amplitude spectrum sequence and an audio phase spectrum sequence are generated, wherein the audio amplitude spectrum in the audio amplitude spectrum sequence and the audio phase spectrum in the audio phase spectrum sequence correspond one-to-one; For each audio amplitude spectrum in the audio amplitude spectrum sequence, perform the following processing steps: Determine the audio type of the audio signal corresponding to the audio amplitude spectrum, wherein the audio type includes: noise and speech; In response to determining that the audio signal corresponding to the audio amplitude spectrum is noise, the noise power spectrum corresponding to the audio amplitude spectrum sequence is updated using the audio amplitude spectrum to obtain the updated noise power spectrum as the noise power spectrum, and the audio amplitude spectrum is determined as the target audio amplitude spectrum. In response to determining that the audio signal corresponding to the audio amplitude spectrum is speech, the audio amplitude spectrum is subjected to spectral enhancement processing using the noise power spectrum to obtain the enhanced amplitude spectrum as the target audio amplitude spectrum; A time-domain signal is generated based on the target audio amplitude spectrum and the audio phase spectrum corresponding to the target audio amplitude spectrum; The obtained time-domain signals are superimposed and added together to obtain the superimposed and added time-domain signals as the filtered audio signals.

3. The method according to claim 2, characterized in that, Determining the audio type of the audio signal corresponding to the audio amplitude spectrum includes: The number of each frequency point included in the audio amplitude spectrum is determined as the frequency point number information; The sum of the squares of the amplitude spectra corresponding to each frequency point included in the audio amplitude spectrum is determined as the total frequency domain energy. The ratio of the total frequency domain energy to the number of frequency points is determined as the average frequency domain energy. In response to determining that the average energy in the frequency domain is greater than a preset energy threshold, the audio type of the audio signal corresponding to the audio amplitude spectrum is determined to be speech; In response to determining that the average energy in the frequency domain is less than or equal to the preset energy threshold, the audio type of the audio signal corresponding to the audio amplitude spectrum is determined to be noise.

4. The method according to claim 2, characterized in that, The step of updating the noise power spectrum corresponding to the audio amplitude spectrum sequence using the audio amplitude spectrum to obtain the updated noise power spectrum as the noise power spectrum includes: The product of the preset smoothing factor and the noise power spectrum corresponding to the audio amplitude spectrum sequence is determined as the first noise power spectrum. The difference between the preset value and the preset smoothing factor is determined as the audio amplitude spectrum weight; The product of the audio amplitude spectrum weight and the audio amplitude spectrum is determined as the second noise power spectrum; The sum of the first noise power spectrum and the second noise power spectrum is determined as the updated noise power spectrum; The updated noise power spectrum is determined as the noise power spectrum in order to update the noise power spectrum.

5. The method according to claim 3, characterized in that, The step of using the noise power spectrum to perform spectral enhancement processing on the audio amplitude spectrum to obtain the enhanced amplitude spectrum as the target audio amplitude spectrum includes: For each frequency point amplitude spectrum in the audio amplitude spectrum, perform the following processing steps: In response to determining that the frequency corresponding to the amplitude spectrum of the frequency point satisfies the first frequency condition, the first preset enhancement factor is determined as the target enhancement factor; In response to determining that the frequency corresponding to the amplitude spectrum of the frequency point satisfies the second frequency condition, the second preset enhancement factor is determined as the target enhancement factor; The noise power spectrum at the frequency point corresponding to the frequency point amplitude spectrum in the noise power spectrum is determined as the target frequency point noise power spectrum; The product of the target enhancement factor and the target frequency noise power spectrum is determined as the enhanced frequency noise power spectrum; The difference between the frequency point amplitude spectrum and the enhanced frequency point noise power spectrum is determined as the enhanced frequency point amplitude spectrum; The amplitude spectrum of each determined enhancement frequency point is defined as the enhancement amplitude spectrum.

6. The method according to claim 1, characterized in that, The step of performing noise separation processing on the filtered audio signal to obtain the target audio signal includes: Feature extraction is performed on the filtered audio signal to obtain audio features; The audio features are downsampled to obtain downsampled audio features; The downsampled audio features are subjected to feature enhancement processing to obtain enhanced audio features as the first enhanced audio features; An enhanced audio feature sequence is generated based on the first enhanced audio feature, wherein the enhanced audio feature sequence includes the first enhanced audio feature; The enhanced audio feature corresponding to the last position in the enhanced audio feature sequence is determined as the target enhanced audio feature; The target enhanced audio features are subjected to convolution processing with a first dilation rate, a second dilation rate, and a third dilation rate, respectively, to obtain first dilated convolution features, second dilated convolution features, and third dilated convolution features; The first dilated convolution feature, the second dilated convolution feature, and the third dilated convolution feature are concatenated to obtain a concatenated dilated convolution feature. The concatenated dilated convolution features are subjected to channel compression processing to obtain channel compressed convolution features; The channel compressed convolution features are upsampled to obtain upsampled convolution features; The upsampled convolutional features are combined with the enhanced audio features in the enhanced audio feature sequence that correspond to the upsampled convolutional features to obtain combined audio features. The combined audio features are subjected to feature extraction processing to obtain upsampled audio features; The upsampled audio features are subjected to feature enhancement processing to obtain upsampled enhanced features; Based on the upsampled enhanced features and the enhanced audio feature sequence, a target audio feature is generated; The target audio features are separated and mapped to obtain the target audio signal.

7. The method according to claim 6, characterized in that, The step of performing feature enhancement processing on the downsampled audio features to obtain enhanced audio features as the first enhanced audio feature includes: The downsampled audio features are convolved to obtain a spatial attention map; Based on the spatial attention map, the downsampled audio features, and the first preset activation function, spatial attention features are generated; Based on the spatial attention features, generate max pooling feature vectors and average pooling feature vectors; The max pooling feature vector and the average pooling feature vector are convolved respectively to obtain the first max convolution feature vector and the first average convolution feature vector. Based on the first maximum convolution feature vector, the first average convolution feature vector, and the second preset activation function, generate the maximum activation feature vector and the average activation feature vector; The maximum activation feature vector and the average activation feature vector are convolved respectively to obtain the second maximum convolution feature vector and the second average convolution feature vector. The second maximum convolution feature vector and the second average convolution feature vector are combined to obtain a combined convolution feature vector. Based on the combined convolutional feature vector and the first preset activation function, channel attention features are generated; Enhanced audio features are generated based on the channel attention features and the downsampled audio features.

8. A video playback device based on voice control, characterized in that, include: The data acquisition unit is configured to acquire video playback audio signals in noisy environments through associated audio signal acquisition devices to obtain video playback audio signals. A noise filtering unit is configured to perform noise filtering processing on the video playback audio signal to obtain a filtered audio signal. The noise separation unit is configured to perform noise separation processing on the filtered audio signal to obtain the target audio signal; An amplitude normalization unit is configured to perform amplitude normalization processing on the target audio signal to obtain a normalized audio signal; The text conversion unit is configured to perform text conversion processing on the normalized audio signal to obtain audio text information; The first generation unit is configured to generate an audio text vector based on the audio text information; The video retrieval unit is configured to perform video retrieval on the target video corresponding to the audio text vector and obtain video retrieval results. The second generation unit is configured to generate video playback control instructions for video playback based on the audio text information and the video retrieval results.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images