Music source separation method and wearable device

Abstract

The invention discloses a music source separation method and wearable equipment, and relates to the technical field of signal processing, music source separation is performed by adopting a neural network model of a UNet architecture, the neural network model at least comprises an encoder and a decoder, and the method comprises the following steps: obtaining an original signal of a to-be-separated music source, and converting the original signal to a frequency domain to obtain a frequency domain signal; performing feature coding processing on the frequency domain signal through an encoder to obtain coded feature data, the feature coding processing including feature extraction, and the encoder performing feature extraction processing at least by using a TFC-TDF block of causal convolution; and performing feature decoding processing on the coded feature data through a decoder, and outputting to obtain a music source separation result. According to the method and the device, low-delay music source separation is realized in the end side equipment with limited resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a music source separation method and a wearable device. BACKGROUND

[0002] In recent years, with the wide popularity of intelligent terminal devices, end-side music source separation (MSS) technology has gradually become a key entry to improve human-computer interaction experience. Music source separation refers to the technical process of extracting different component audio tracks from a mixed music signal, which has important value in audio enhancement, content creation, interactive entertainment and other applications.

[0003] Music source separation technology has mainly experienced the evolution from traditional methods to deep learning-based technology routes. Early methods are mostly based on traditional machine learning algorithms such as principal component analysis and independent component analysis, which basically assume that different source signals are statistically independent of each other, and try to achieve audio track separation using this property. However, this kind of method has limited separation effect when dealing with real complex music scenes. With the development of deep learning technology, models such as feedforward neural networks, convolutional neural networks and recurrent neural networks are widely used in MSS tasks, and the separation performance is improved through end-to-end learning.

[0004] However, the music source separation method based on deep learning usually has a very high parameter quantity and computational complexity, resulting in a large inference delay, which is difficult to meet the strict requirements of terminal devices for low latency. Therefore, in the resource-limited end-side environment, how to realize low-latency music source separation has become a core bottleneck problem faced by the current technology in the actual landing process. SUMMARY

[0005] The main purpose of the present application is to provide a music source separation method and a wearable device, which aims to solve the technical problem of how to realize low-latency music source separation in a resource-limited situation.

[0006] To achieve the above-mentioned purpose, the present application provides a music source separation method, which uses a neural network model with UNet architecture to perform music source separation. The neural network model at least includes an encoder and a decoder. The music source separation method comprises the following steps: Obtaining an original signal of a music source to be separated, and converting the original signal to a frequency domain to obtain a frequency domain signal; Performing feature encoding processing on the frequency domain signal through the encoder to obtain encoded feature data, wherein the feature encoding processing includes feature extraction, and the encoder at least performs feature extraction processing by using a TFC-TDF block using causal convolution; The encoded feature data is decoded by the decoder to obtain a music source separation result.

[0007] In an embodiment, the encoder comprises a first convolutional layer and a feature encoding network connected in sequence, and the feature extraction of the frequency domain signal is performed by the encoder to obtain encoded feature data. The frequency domain signal is convoluted by the first convolutional layer to obtain low-level feature data. The low-level feature data is encoded by the feature encoding network to obtain encoded feature data.

[0008] In an embodiment, the feature encoding process further comprises downsampling, and the feature encoding network comprises at least one set of a first TFC-TDF block, a first DPRNN block and a downsampling layer connected in sequence, and the feature extraction of the low-level feature data by the feature encoding network to obtain encoded feature data comprises: In each set of the first TFC-TDF block, the first DPRNN block and the downsampling layer connected in sequence, the first input data is extracted by the first TFC-TDF block to obtain first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low-level feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set; The first intermediate feature data is extracted by the DPRNN block to obtain second intermediate feature data. The second intermediate feature data is downsampled by the downsampling layer to obtain downsampled feature data. After the downsampling of the tail layer downsampling layer is completed, the downsampled feature data obtained by the tail layer downsampling layer is determined as the encoded feature data.

[0009] In an embodiment, the feature extraction of the first intermediate feature data by the first DPRNN block to obtain second intermediate feature data comprises: The first intermediate feature data is converted into first tensor data, wherein the feature dimensions of the first tensor data are respectively the channel number multiplied by the batch number dimension, the time frame number dimension and the frequency component number dimension. The first tensor data is time series modeled by a unidirectional gated recurrent unit to obtain time series feature data. convert the time sequence feature data into second tensor data, wherein a feature dimension of the second tensor data is respectively a batch number multiplied by a time frame number dimension, a frequency component number dimension, and a channel number dimension; perform frequency domain structure modeling processing on the time sequence feature data by a bidirectional gate recurrent unit to obtain global feature data; generate second intermediate feature data based on the global feature data, wherein a feature dimension of the second intermediate feature data is consistent with that of the first intermediate feature data.

[0010] In an embodiment, the first TFC-TDF block comprises a first causal convolution two-dimensional layer, a first full connection layer, a second full connection layer, and a second causal convolution two-dimensional layer connected in sequence, and an input residual of the first full connection layer is connected to an output of the second full connection layer, and an input of the first causal convolution two-dimensional layer is connected to an output of the second causal convolution two-dimensional layer through a two-dimensional convolution layer residual.

[0011] In an embodiment, the decoder comprises a feature decoding network and a second convolution layer, and the feature decoding processing of the encoded feature data by the decoder comprises: performing feature decoding processing on the encoded feature data by each group of the feature decoding network in sequence to obtain decoded feature data; performing convolution processing on the decoded feature data by the second convolution layer to output a music source separation result.

[0012] In an embodiment, the feature decoding processing comprises feature extraction and up-sampling, and the feature decoding network comprises at least one group of up-sampling layers, a second TFC-TDF block, and a second DPRNN block connected in sequence, and the feature decoding processing of the encoded feature data by each group of the feature decoding network in sequence to obtain decoded feature data comprises: in each group of the up-sampling layers, the second TFC-TDF block, and the second DPRNN block connected in sequence, performing up-sampling processing on second input data by the up-sampling layer to obtain up-sampled feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is an intermediate layer of the decoder, the second input data is a processing result output by the second DPRNN block of the previous group; performing feature extraction processing on the up-sampled feature data by the second TFC-TDF block to obtain third intermediate feature data; performing feature extraction processing on the third intermediate feature data by the second DPRNN block to obtain fourth intermediate feature data; After the second DPRNN block of the tail layer completes the feature extraction processing, it is determined that the fourth intermediate feature data obtained by the second DPRNN block of the tail layer is the decoding feature data.

[0013] In an embodiment, the neural network model further comprises a DPRNN intermediate layer, and the step of performing feature decoding processing on the encoded feature data by the decoder to output a music source separation result comprises: performing feature extraction processing on the encoded feature data by the DPRNN intermediate layer to obtain high-dimensional feature data; performing feature decoding processing on the high-dimensional feature data by the decoder to output a music source separation result.

[0014] In an embodiment, after the step of performing feature decoding processing on the encoded feature data by the decoder to output a music source separation result, the method further comprises: performing inverse Fourier transform processing on the music source separation result to obtain a separated time-domain audio track signal, wherein the time-domain audio track signal comprises a vocal track signal, a drum track signal, a bass track signal, and other audio track signals.

[0015] In addition, to achieve the above-mentioned purposes, the present application also provides a wearable device, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the music source separation method as described above.

[0016] In addition, to achieve the above-mentioned purposes, the present application also provides a readable storage medium, which is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, wherein the computer program is executed by a processor to implement the steps of the music source separation method as described above.

[0017] The present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the music source separation method as described above.

[0018] The one or more technical solutions proposed in the present application have at least the following technical effects: The application effectively solves the problems of high calculation complexity and long inference delay of a music source separation model based on deep learning by adopting a neural network model based on a UNet architecture and introducing a TFC-TDF (Temporal Frequency Convolution-Time Domain Filtering) block with causal convolution as a feature extraction unit of an encoder. Specifically, after obtaining the original signal of the music source to be separated, the original signal is converted to the frequency domain, and the sparse characteristics and structured features of the audio signal in the frequency domain are fully utilized to reduce the calculation burden for subsequent processing. Then, the encoder uses at least a TFC-TDF block integrating causal convolution to extract features from the frequency domain signal. The application of causal convolution ensures that the model only relies on current and historical information, eliminating the dependence on future frames, so that a streaming processing mode of one frame in and one frame out can be realized, thereby greatly reducing the algorithm delay and meeting the requirements of real-time processing. At the same time, the TFC-TDF block can efficiently model the time-frequency feature relationship in the frequency domain, while maintaining the feature expression ability while compressing the calculation amount. Finally, the decoder reconstructs and recovers the encoded features. The symmetrical encoder-decoder design of the UNet architecture ensures the integrity of information transmission, and the UNet architecture itself has the characteristics of high parameter sharing degree and clear calculation path. The symmetrical encoder-decoder structure of the UNet architecture can efficiently extract and reconstruct multi-level features from the frequency domain signal, significantly reducing the parameter amount and calculation burden of the whole model, and is more conducive to deployment on terminal devices. In this way, low-delay music source separation can be realized under limited resources. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings from these drawings without creative labor.

[0021] Figure 1 Flowchart of the first embodiment of the music source separation method of the present application; Figure 2 Flowchart of the music source separation process involved in an embodiment of the music source separation method of the present application; Figure 3 Structure diagram of the neural network model based on the UNet architecture involved in an embodiment of the music source separation method of the present application; Figure 4A block structure diagram of a TFC-TDF block involved in an embodiment of the music source separation method of the present application; Figure 5 A feature extraction flowchart of a DPRNN block involved in an embodiment of the music source separation method of the present application; Figure 6 A device structure diagram of a hardware running environment involved in the music source separation method of an embodiment of the present application.

[0022] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] In recent years, with the popularity of intelligent terminal devices such as smart phones, smart tablets, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, etc., end-side music source separation technology has become a key entry for human-computer interaction. Music source separation refers to the technology of separating each component track (such as vocals, drums, bass, etc.) from a mixed piece of music. Traditional music source separation systems usually rely on cloud processing, but due to network delay, privacy security and power consumption limitations, end-side music source separation technology has gradually become a research hotspot. Early music source separation technology is based on traditional machine learning algorithms, which assume that different source signals are statistically independent, so principal component analysis or independent component analysis is used to separate different tracks in music. This algorithm does not work well in real music separation tasks. Deep learning-based methods use feedforward neural networks, convolutional neural networks and recurrent neural networks to enhance music source separation performance, achieving excellent performance. However, these models have high parameter quantity and computational complexity, and the algorithm delay is large, making it difficult to run in real time on end-side devices.

[0025] In practical applications, especially on intelligent terminals, the deployment of music source separation systems is usually limited by computing resources and algorithm latency. Although deep learning-based music source separation technology has excellent performance, it is still a direction to explore under the requirements of low latency and low power consumption. Therefore, under the condition of limited resources, how to achieve efficient and real-time music source separation is still a challenge that needs to be broken through.

[0026] Based on this, the main solution of the present application is: a neural network model with UNet architecture is used for music source separation, the neural network model at least includes an encoder and a decoder, an original signal of a music source to be separated is obtained, and the original signal is converted to a frequency domain to obtain a frequency domain signal; the frequency domain signal is processed by the encoder for feature encoding to obtain encoded feature data, wherein the feature encoding processing includes feature extraction, and the encoder at least performs feature extraction processing by using a TFC-TDF block integrating causal convolution; the encoded feature data is processed by the decoder for feature decoding to output a music source separation result.

[0027] After obtaining the original signal of the music source to be separated, the present application converts it to the frequency domain, fully utilizes the sparse characteristics and structured features of the audio signal in the frequency domain, and reduces the computational burden for subsequent processing. Subsequently, the encoder at least uses a TFC-TDF block integrating causal convolution to perform feature extraction on the frequency domain signal. The application of causal convolution ensures that the model only relies on current and historical information, eliminates the dependence on future frames, and enables a streaming processing mode of one frame in and one frame out, thereby greatly reducing the algorithm delay and meeting the requirements of real-time processing. At the same time, the TFC-TDF block can efficiently model the time-frequency feature relationship in the frequency domain, while maintaining the feature expression ability while compressing the computational amount. Finally, the decoder reconstructs and recovers the encoded features. The symmetrical encoder-decoder design of the UNet architecture ensures the integrity of information transmission. Moreover, the UNet architecture itself has the characteristics of high parameter sharing degree and clear calculation path. Its symmetrical encoder-decoder structure can efficiently perform multi-level feature extraction and reconstruction on the frequency domain signal, significantly reducing the parameter amount and computational burden of the whole model, and is more conducive to deployment on terminal devices. In this way, the music source separation with low delay can be realized in the case of limited resources.

[0028] It should be noted that the execution subject of each embodiment of the music source separation method of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a wearable device capable of realizing the above functions, such as AR glasses, VR glasses, earphones, etc. The music source separation method of the present application does not make specific limitations on this.

[0029] Based on this, the present application proposes a music source separation method of a first embodiment, which uses a neural network model with UNet architecture for music source separation. The neural network model at least includes an encoder and a decoder. As shown in Figure 1 The music source separation method includes the following steps S10-S30: Step S10, obtaining an original signal of a music source to be separated, and converting the original signal to a frequency domain to obtain a frequency domain signal; The original signal of the music source to be separated can be acquired in real time by an audio acquisition device or read from a storage medium. This original signal is usually a time-domain waveform signal and can be in single-channel or multi-channel format. For ease of explanation, each embodiment in this application is described using a dual-channel signal as an example.

[0030] After acquiring the original signal, in order to facilitate subsequent neural network processing, the time-domain signal is converted to the frequency-domain representation to obtain the frequency-domain signal.

[0031] Reference Figure 2 As shown, this embodiment preferably employs Short-Time Fourier Transform (STFT) to perform the conversion. STFT uses a fixed-length window function (such as a Hamming window) to slide along the time axis, dividing the continuous time-domain signal into multiple short-overlapping time frames. Then, a Discrete Fourier Transform (DFT) is performed on each frame to obtain the time spectrum, which simultaneously contains time and frequency information. This transformation represents the non-stationary music signal as a time-varying spectral feature, making it more advantageous for separating the structural features of different audio tracks in the frequency domain.

[0032] After STFT transformation, the frequency domain signal can be represented as a tensor with dimensions C×T×F. Here, C represents the number of channels, T represents the number of time frames, and F represents the number of frequency components. For a dual-channel input signal, to fully preserve phase information, this embodiment decomposes the complex spectrum of each channel into two components: a real part and an imaginary part. Therefore, a total of four channels are obtained: the real part of the left channel, the imaginary part of the left channel, the real part of the right channel, and the imaginary part of the right channel.

[0033] Reference Figure 2 As shown, in this embodiment, the obtained 4×T×F-dimensional frequency domain tensor is input into a neural network model based on the UNet architecture for subsequent encoding and decoding processing. By explicitly utilizing both the real and imaginary parts of the spectrum simultaneously, i.e., modeling amplitude and phase at the same time, the time-frequency structure of the audio signal can be preserved more comprehensively, which helps to improve the fidelity and listening quality of the final separated audio tracks.

[0034] Step S20: The frequency domain signal is processed by the encoder to obtain coded feature data. The feature encoding process includes feature extraction. The encoder performs feature extraction at least by using a TFC-TDF block of causal convolution. The encoder includes one or more TFC-TDF blocks employing causal convolution. Each TFC-TDF block comprises a Time-Frequency Convolution (TFC) module and a Time-Frequency-Depth (TDF) module. The TFC module uses causal convolution to perform convolution operations on the frequency domain signal, extracting the signal's time-frequency features. Causal convolution ensures that when processing the signal at the current time frame, it only depends on information from previous time frames, avoiding reliance on information from future frames. The TDF module performs depth processing on the extracted time-frequency features, further extracting more abstract features and enhancing the discriminative power of the features.

[0035] Step S30: The encoded feature data is processed by the decoder to obtain the music source separation result.

[0036] The encoded feature data is processed using a symmetrical decoder structure. The decoder typically contains multiple cascaded feature decoding blocks, each structurally corresponding to an encoded block in the encoder. Each decoding block receives two inputs: one from the upsampled output features of the previous decoding block, and the other from feature outputs at the same level in the encoder path via a skip connection. This skip connection mechanism directly transmits the rich detail captured during the encoding stage to the decoding process, effectively compensating for subtle structures and high-frequency components that may be lost during feature compression, thus significantly improving the quality and clarity of the final reconstructed signal.

[0037] The decoder can upsample using methods such as transposed convolution or interpolation to gradually restore the spatial resolution of the feature maps and reduce the number of channels. Finally, the decoder's output layer (usually a convolutional layer) maps the reconstructed features to the time-frequency domain estimates of each target sound source (e.g., vocals, drums, bass, and other instruments). For each sound source, the output is a 4×T×F dimensional tensor, consistent with the dimension of the complex spectral representation of the input signal, constituting the model's complex spectral estimate of that sound source. The output of this convolutional layer is the music source separation result generated by the neural network model.

[0038] Furthermore, such as Figure 2 As shown, in order to obtain a playable time-domain audio waveform, the frequency domain tensors of each audio track output by the model (i.e., the music source separation results) can be converted back into time-domain signals via inverse short-time fourier transform (ISTFT). Figure 2 This example demonstrates the process of separating the original mixed signal into four independent audio tracks: "Vocals," "Drums," "Bass," and "Others." In practical applications, the type and number of target audio sources can be flexibly set according to specific needs, and this embodiment does not impose any limitations on this.

[0039] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, the encoder comprises a first convolutional layer and a feature encoding network connected in turn, and the feature extraction processing of the frequency domain signal is performed through the encoder to obtain the encoded feature data. Step A10, the frequency domain signal is convolved through the first convolutional layer to obtain low layer feature data. The first convolutional layer refers to the convolutional layer in the encoder, and the first convolutional layer uses a predetermined size of convolution kernel (such as 1*1) to perform convolution operation on the input frequency domain signal. The operation aims to preliminarily extract the local time-frequency features in the input frequency spectrum, such as edge, texture and other basic patterns. By adjusting the number of convolution kernels, the dimension transformation and feature compression of the input channel are realized, and a feature map containing rich low layer semantic information, i.e. the low layer feature data, is formed.

[0040] Step A20, the low layer feature data is processed through the feature encoding network to obtain the encoded feature data.

[0041] The feature encoding network includes at least one or more TFC-TDF blocks. Each TFC-TDF block uses causal convolution (Causal Convolution) for feature extraction to capture the harmonic structure in the frequency domain and the time sequence dependence in the time domain, respectively, while ensuring that the model only depends on the historical and current information, meeting the low delay requirement. The whole encoding process reduces the resolution of the feature map through stepwise downsampling (such as using stride convolution or pooling operation), while increasing the number of channels, and finally outputs the encoded feature data, which provides a strong feature representation for the subsequent decoder reconstruction.

[0042] In one possible implementation, the feature encoding processing further includes downsampling, and the feature encoding network includes at least one set of first TFC-TDF block, first DPRNN block and downsampling layer connected in turn, and the step of performing feature extraction processing on the low layer feature data through the feature encoding network to obtain the encoded feature data comprises: Step B10, in each set of first TFC-TDF block, first DPRNN block and downsampling layer connected in turn, the first TFC-TDF block performs feature extraction processing on the first input data to obtain the first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low layer feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set; The first TFC-TDF block and the first DPRNN block respectively refer to a TFC-TDF block and a DPRNN (Dual-Path Recurrent Neural Network) block in an encoder.

[0043] Specifically, referring to FIG. 2, Figure 4 As shown in FIG. 2, the TFC-TDF block includes a first causal convolution two-dimensional layer (Casual Conv2D), a first fully connected layer (FC), a second fully connected layer, and a second causal convolution two-dimensional layer connected in sequence, and the input residual of the first fully connected layer is connected to the output of the second fully connected layer, and the input of the first causal convolution two-dimensional layer is connected to the output of the second causal convolution two-dimensional layer through a two-dimensional convolution layer (1x1 Conv2D) as shown in FIG. 3, that is, the first input data is first processed through a causal convolution 2D layer, then is transformed through two fully connected layers, and then is processed through a causal convolution 2D layer again. At the same time, the input data is also processed through a 1x1 convolution 2D layer, and finally the outputs of the two paths are combined to form the final output. Figure 4

[0044] The TFC-TDF block uses causal convolution to extract features from the input data to generate first intermediate feature data, which contains more rich time-frequency information.

[0045] In step B20, the DPRNN block is used to extract features from the first intermediate feature data to obtain second intermediate feature data. The DPRNN block is a deep recurrent neural network module, which can further process the first intermediate feature data, capture the time sequence dependency relationship in the feature data, and generate second intermediate feature data, which can better represent the dynamic characteristics of the input signal.

[0046] In step B30, the down-sampling layer is used to down-sample the second intermediate feature data to obtain down-sampled feature data. The down-sampling layer can use pooling or other down-sampling techniques to down-sample the second intermediate feature data, reduce the spatial dimension of the feature data, and retain important feature information to generate down-sampled feature data. The down-sampling process helps to reduce the computational complexity and improve the abstract level of the features.

[0047] In step B40, after the down-sampling layer of the tail layer completes the down-sampling process, the down-sampled feature data obtained by the down-sampling layer of the tail layer is determined as the encoding feature data.

[0048] ​After repeated stacking and processing of multiple groups (first TFC-TDF block + first DPRNN block + down-sampling layer), the feature map is gradually compressed and abstracted. The down-sampled feature data output by the down-sampling layer of the last group (i.e., the tail layer) of the network is the final output of the entire feature encoding network, which is highly abstract and rich in semantic information. The data will be passed to the decoder for subsequent source separation reconstruction.

[0049] In a possible implementation, the step of performing feature extraction processing on the first intermediate feature data by the first DPRNN block to obtain second intermediate feature data comprises: Step C10, converting the first intermediate feature data into first tensor data, wherein the feature dimensions of the first tensor data are respectively the channel number multiplied by the batch number dimension, the time frame number dimension, and the frequency component number dimension; Let the channel number be C, the batch number be B, the time frame number be T, and the frequency component number be F. The first intermediate feature data can be converted into first tensor data with BC*T*F dimensional distribution through reshaping operation.

[0050] Step C20, performing time series modeling processing on the first tensor data by a unidirectional gated recurrent unit to obtain time series feature data; The unidirectional GRU layer is used to perform forward sequence scanning on the first tensor data along the time step dimension. The unidirectional GRU ensures that its output only depends on the current and past input information, strictly maintaining the causality of the model to meet the low delay constraint required by the end-side real-time processing. The GRU layer captures the dynamic changes and dependencies of signals in the time dimension, and outputs feature data containing time series context information.

[0051] Step C30, converting the time series feature data into second tensor data, wherein the feature dimensions of the second tensor data are respectively the batch number multiplied by the time frame number dimension, the frequency component number dimension, and the channel number dimension; After completing the time dimension modeling, the dimensions of the time series feature data are converted and reshaped to organize them into second tensor data with BT*F*C dimensional distribution. This conversion takes the frequency component number dimension as the new sequence dimension, preparing for subsequent recursive modeling in the frequency domain dimension.

[0052] The bidirectional GRU layer is used to perform sequence modeling on the second tensor data along the frequency component number dimension. The bidirectional GRU can simultaneously utilize the global frequency context information before and after each frequency component, thereby efficiently modeling the harmonic structure and global frequency spectrum dependency between different frequency components, and outputting feature data containing global frequency domain context information.

[0053] Step C40, performing frequency domain structure modeling processing on the timing feature data through a bidirectional gate recurrent unit to obtain global feature data; Step C50, generating second intermediate feature data based on the global feature data, wherein the second intermediate feature data is consistent with the feature dimension of the first intermediate feature data.

[0054] Performing final dimension transformation and reshaping operation on the global feature data processed by the bidirectional GRU to restore it to the feature dimension consistent with the input first intermediate feature data, ensuring that the data can seamlessly flow into the subsequent network layer.

[0055] In a possible implementation, the step of generating second intermediate feature data based on the global feature data comprises: Step D10, converting the feature dimension of the global feature data to be consistent with the first intermediate feature data to obtain target data; Step D20, taking the target data as new first intermediate feature data, and returning to execute the step of converting the first intermediate feature data into first tensor data; Step D30, until a preset repetition number threshold is reached, determining the target data obtained after the preset repetition number threshold is reached as second intermediate feature data.

[0056] Let the preset repetition number threshold be L, and the above steps C10 to C50 are encapsulated as a block that can be repeatedly executed. Through multiple repetitions (for example, 4 or 6 times), the model can perform deeper timing and frequency domain joint optimization on the input features, thereby gradually enhancing the feature expression capability. Finally, the target data obtained after L times of cycle processing is determined as the final output of the first DPRNN block, i.e., the second intermediate feature data.

[0057] For example, in a specific implementation, the processing flow of the DPRNN block is as follows Figure 5As shown, first reshape the first intermediate feature data of B*C*T* F dimension distribution into the first tensor data of BC*T*F dimension distribution. In order to ensure that the model does not depend on future frame information, a unidirectional GRU model is used for modeling at this time, followed by a full connection layer (FC) and a layer normalization (Layernorm) process. This module mainly improves the time sequence modeling capability of the music signal. Then, the first tensor data of BC*T*F dimension distribution is subjected to reshape operation, permute operation and reshape operation, so as to obtain the second tensor data of BT*F*C dimension distribution. Since no time sequence information is depended on at this time, a bidirectional GRU model is used for modeling, followed by a full connection layer (FC) and a layer normalization (Layernorm) process. This module mainly improves the local modeling capability of the music signal. Finally, the second tensor data of BT*F*C dimension distribution is subjected to reshape operation and permute operation, so as to restore to the dimension distribution of B*C*T*F. The above process is repeated L times (not shown in the figure), and the target data of B*C*T*F dimension distribution finally obtained is determined as the second intermediate feature data. Figure 5

[0058] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned embodiments one and two can refer to the above introduction, and the subsequent will not be described in detail. On this basis, the decoder includes a feature decoding network and a second convolutional layer, and the step of performing feature decoding processing on the encoded feature data by the decoder to output the music source separation result includes: Step E10, sequentially performing feature decoding processing on the encoded feature data by each group of the feature decoding network to obtain decoded feature data. The feature decoding network is composed of a plurality of cascaded decoding blocks, and each decoding block corresponds to the processing unit of the corresponding level in the encoder in structure. Each decoding block receives two inputs: one from the output feature of the previous decoding layer, and the other from the feature map of the same level in the encoder path through the skip connection. Through the layer-by-layer processing of the multi-level decoding block, the resolution of the feature map is gradually restored, and finally the decoded feature data containing detailed information is output.

[0059] Step E20, performing convolution processing on the decoded feature data by the second convolutional layer to output the music source separation result.

[0060] ​The second convolutional layer refers to a convolutional layer in the decoder, which maps the high-dimensional decoding feature data output by the decoder to the number of target sound sources. The convolutional layer does not change the spatial resolution (i.e., the number of time frames T and the number of frequency components F) of the feature map, but adjusts the number of channels thereof. For each target sound source (such as vocals, drums, bass, etc.), the output is a tensor consistent with the dimension of the original input frequency domain signal (for example, 4 x T x F). The tensor can be directly used as the complex spectrum of the target sound source estimated by the model (i.e., the result of the point multiplication of the mask and the mixed spectrum), and the output of the second convolutional layer is the final music source separation result.

[0061] In a possible implementation, the feature decoding process includes feature extraction and up-sampling, the feature encoding network includes at least one group of sequentially connected up-sampling layers, second TFC-TDF blocks and second DPRNN blocks, and the step of sequentially performing feature decoding on the encoded feature data by each group of the feature decoding network to obtain decoded feature data includes: In step F10, the up-sampling layer in each group of sequentially connected up-sampling layers, second TFC-TDF blocks and second DPRNN blocks performs up-sampling processing on the second input data to obtain up-sampled feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is an intermediate layer of the decoder, the second input data is the processing result output by the second DPRNN block of the previous group. The second TFC-TDF block and the second DPRNN block respectively refer to a TFC-TDF block and a DPRNN block in the decoder.

[0062] The up-sampling layer can perform a spatial resolution enhancement operation on the input feature through transposed convolution or interpolation algorithm to gradually restore the size of the feature map in the time dimension and the frequency dimension. The operation is the inverse process of the down-sampling process of the encoder, and is designed to expand the highly compressed encoded feature map to a higher resolution to lay a foundation for subsequent detail reconstruction.

[0063] In step F20, the second TFC-TDF block performs feature extraction processing on the up-sampled feature data to obtain third intermediate feature data. The structure of the second TFC-TDF block is consistent with that of the first TFC-TDF block, which will not be described here.

[0064] In step F30, the second DPRNN block performs feature extraction processing on the third intermediate feature data to obtain fourth intermediate feature data. Similarly, the feature extraction process of the second DPRNN block is consistent with that of the first DPRNN block. The second DPRNN block also adopts a double-path recurrent neural network structure to sequentially perform sequence modeling on the time dimension and the frequency dimension of the input third intermediate feature data. This module further enhances the discriminability of feature expression, can effectively integrate global context information, and reconstruct the fine structure of the sound source signal, outputting fourth intermediate feature data containing semantic information.

[0065] Step F40, after the second DPRNN block of the tail layer completes the feature extraction process, the fourth intermediate feature data obtained by the second DPRNN block of the tail layer is determined as the decoding feature data.

[0066] After repeated stacking and processing of multiple groups (up-sampling layer + second TFC-TDF block + second DPRNN block), the resolution of the feature map is gradually restored to a size close to that of the original input frequency domain signal. The fourth intermediate feature data output by the second DPRNN block of the last group (i.e., the tail layer) of the network is the decoding feature data finally generated by the entire feature decoding network, which will be passed to the second convolutional layer to generate the final sound source separation result.

[0067] Based on the first, second and / or third embodiments of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned embodiments one, two and three can be referred to the above introduction, and will not be repeated hereinafter. On this basis, the neural network model further includes a DPRNN intermediate layer, and the step of performing feature decoding processing on the encoded feature data by the decoder to output a music source separation result, comprising: Step G10, performing feature extraction processing on the encoded feature data by the DPRNN intermediate layer to obtain high-dimensional feature data; The DPRNN intermediate layer adopts a double-path recurrent neural network structure, receives the encoded feature data output from the encoder, and performs feature extraction processing thereon. The feature extraction process can be consistent with that of the first DPRNN block, which will not be repeated here.

[0068] Step G20, performing feature decoding processing on the high-dimensional feature data by the decoder to output a music source separation result.

[0069] The decoder receives high-dimensional feature data enhanced by the intermediate layers of the DPRNN and performs feature decoding and reconstruction through a series of symmetrical decoding networks (such as an upsampling layer + a second TFC-TDF block + a second DPRNN block). Each decoding network gradually recovers the spatial resolution of the feature map through upsampling operations and uses skip connections to fuse feature details from corresponding layers in the encoder path to accurately reconstruct the time-frequency structure of the audio source signal. Finally, the decoder output is mapped to the time-frequency representation of each target audio source through a convolutional layer, which is the final music source separation result of the model.

[0070] For example, to aid in understanding the neural network model based on the U-NET architecture after combining this embodiment with the first, second, and third embodiments described above, a specific embodiment is now provided. In this specific embodiment, refer to... Figure 3 As shown, the neural network model based on the U-NET architecture consists of three modules: encoder, middle block, and decoder. The skip connections between the encoder and decoder are primarily responsible for passing low-level feature information from the encoder to the decoder to ensure reconstruction quality.

[0071] The encoder initially used 1x1 2D convolutional layers (1*1 Conv2D) to increase the number of channels from 4 to C1. Then, stacked D groups ( Figure 3 The diagram shows three groups of TFC-TDF blocks, DPRNN blocks, and downsampled layers. The number of channels is increased from C1 to C2, and then from C2 to C3, while the number of frequency components is decreased from F to F / 2, and then from F / 2 to F / 4, ultimately resulting in encoded feature data with a B*C3*T*F / 4 dimensional distribution.

[0072] The intermediate layer is implemented through DPRNN blocks. The encoded feature data with a B*C3*T*F / 4 dimension distribution is extracted through the DPRNN blocks of the intermediate layer to obtain high-level feature data, which is then input into the decoder.

[0073] The decoder has a similar structure to the encoder, consisting of stacked D groups ( Figure 3 The diagram shows a three-part structure consisting of an upsample layer, a TFC-TDF block, and a DPRNN block. The upsample block is implemented using a two-dimensional transposed convolution, as shown below. Figure 2 As shown, each upsampling increases the F dimension of the features to twice its original value, 2F. Specifically, for B*C*T*F features, the kernel of the 2D transposed convolution is set to (1, 2), and the stride is set to (1, 2). This ensures that the time dimension remains unaffected, thus preserving the causality of the model. Both the TFC-TDF block and the DPRNN block are consistent with the encoder.

[0074] The TFC-TDF block in the first group of the encoder is connected to the DPRNN block in the third group of the decoder by a skip connection, the TFC-TDF block in the second group of the encoder is connected to the DPRNN block in the second group of the decoder by a skip connection, and the TFC-TDF block in the third group of the encoder is connected to the DPRNN block in the first group of the decoder by a skip connection.

[0075] It should be noted that the above examples are only used to assist in understanding the present embodiment, and do not constitute a limitation on the neural network model based on the U-NET architecture of the present embodiment. Further forms of simple transformation based on this technical concept are within the scope of protection of the present application.

[0076] In addition, the present embodiment further proposes a wearable device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the music source separation method as described above.

[0077] Reference Figure 6 which shows a structural schematic diagram of a wearable device suitable for being used to implement the present embodiment. The wearable device in the present embodiment can also include, but is not limited to, AR glasses, VR glasses, earphones, AR headsets, VR headsets, and the like. Figure 6 The wearable device shown is only an example, and should not bring any limitation to the function and use range of the present embodiment.

[0078] As Figure 6As shown, the wearable device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the wearable device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the wearable device to communicate wirelessly or wired with other devices to exchange data. Although the wearable device with various systems is shown in the figure, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0079] In particular, the processes described above with reference to the flowcharts can be implemented as computer software programs according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0080] The wearable device provided by the embodiments of the present disclosure adopts the music source separation method in the above embodiments, and can solve the technical problem of how to realize low-delay music source separation in a resource-limited case. Compared with the prior art, the wearable device provided by the present disclosure has the same beneficial effects as the music source separation method provided by the above embodiments, and other technical features in the wearable device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0081] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0082] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by those skilled in the art without departing from the spirit of the application are intended to be included in the scope of the application. The scope of the application is defined by the appended claims.

[0083] In addition, to achieve the above object, the embodiment of the application further provides a readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the music source separation method in the above embodiment.

[0084] The computer readable storage medium provided by the embodiment of the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency: radio frequency), etc., or any suitable combination of the above.

[0085] The above computer readable storage medium can be contained in a wearable device; or can exist separately without being assembled into a wearable device.

[0086] The above computer readable storage medium carries one or more programs, which, when executed by the wearable device, cause the wearable device to implement the flow steps of any of the above embodiments.

[0087] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0088] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0089] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0090] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above music source separation method, and can solve the technical problem of how to realize low-delay music source separation in a resource-limited situation. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the music source separation method provided by the above embodiments, which will not be described here.

[0091] In addition, the embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the music source separation method as described above.

[0092] The computer program product embodiment of the present application is basically the same as the above-mentioned music source separation method embodiments, and will not be described here.

[0093] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0094] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by software plus the necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software sensors, which are stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a number of instructions to make a wearable device (which can be a mobile phone, computer, server or network device, etc.) execute the methods described in various embodiments of the present application.

[0096] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of musical source separation, characterized by, The music source separation method adopts a neural network model with a UNet architecture to perform music source separation, and the neural network model at least includes an encoder and a decoder. An original signal of a music source to be separated is obtained, and the original signal is converted into a frequency domain to obtain a frequency domain signal. The frequency domain signal is processed by the encoder for feature encoding to obtain encoded feature data, wherein the feature encoding process includes feature extraction, and the encoder at least performs feature extraction processing by using a TFC-TDF block with causal convolution. The encoded feature data is processed by the decoder for feature decoding to output a music source separation result.

2. The musical source separation method of claim 1, wherein, The encoder includes a first convolution layer and a feature encoding network connected in sequence, and the frequency domain signal is processed by the encoder for feature extraction to obtain encoded feature data. The frequency domain signal is processed by the first convolution layer for convolution to obtain low-level feature data. The low-level feature data is processed by the feature encoding network for feature encoding to obtain encoded feature data.

3. The musical source separation method of claim 2, wherein, The feature encoding process further includes downsampling, and the feature encoding network includes at least one set of a first TFC-TDF block, a first DPRNN block and a downsampling layer connected in sequence, and the step of processing the low-level feature data by the feature encoding network for feature extraction to obtain encoded feature data includes: In each set of the first TFC-TDF block, the first DPRNN block and the downsampling layer connected in sequence, the first TFC-TDF block is used to process first input data for feature extraction to obtain first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low-level feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set; The first intermediate feature data is processed by the DPRNN block for feature extraction to obtain second intermediate feature data; The second intermediate feature data is processed by the downsampling layer for downsampling to obtain downsampled feature data; After the downsampling layer at the tail layer completes the downsampling processing, the downsampled feature data obtained by the downsampling layer at the tail layer is determined as the encoded feature data.

4. The musical source separation method of claim 3, wherein, The step of processing the first intermediate feature data by the first DPRNN block for feature extraction to obtain second intermediate feature data includes: The first intermediate feature data is converted into first tensor data, wherein the feature dimensions of the first tensor data are respectively a channel number multiplied by a batch number dimension, a time frame number dimension and a frequency component number dimension; The first tensor data is processed by a unidirectional gated recurrent unit for time series modeling to obtain time series feature data; The time series feature data is converted into second tensor data, wherein the feature dimensions of the second tensor data are respectively a batch number multiplied by a time frame number dimension, a frequency component number dimension and a channel number dimension; The time sequence feature data is subjected to frequency domain structure modeling processing by a bidirectional gated recurrent unit to obtain global feature data. Second intermediate feature data is generated based on the global feature data, wherein the second intermediate feature data is consistent with the feature dimension of the first intermediate feature data.

5. The musical source separation method of claim 3, wherein, The first TFC-TDF block comprises a first causal convolution two-dimensional layer, a first full connection layer, a second full connection layer and a second causal convolution two-dimensional layer connected in sequence, and the input residual of the first full connection layer is connected to the output of the second full connection layer, and the input of the first causal convolution two-dimensional layer is connected to the output of the second causal convolution two-dimensional layer through a two-dimensional convolution layer residual.

6. The musical source separation method of claim 1, wherein, The decoder comprises a feature decoding network and a second convolution layer, and the feature decoding processing of the encoded feature data by the decoder to output the music source separation result comprises: The encoded feature data is subjected to feature decoding processing by each group of the feature decoding network in sequence to obtain decoded feature data; The decoded feature data is subjected to convolution processing by the second convolution layer to output the music source separation result.

7. The musical source separation method of claim 6, wherein, The feature decoding processing comprises feature extraction and up-sampling, and the feature encoding network comprises at least one group of up-sampling layers, second TFC-TDF blocks and second DPRNN blocks connected in sequence, and the feature decoding processing of the encoded feature data by each group of the feature decoding network in sequence to obtain decoded feature data comprises: In each group of the up-sampling layers, the second TFC-TDF blocks and the second DPRNN blocks connected in sequence, the second input data is subjected to up-sampling processing by the up-sampling layer to obtain up-sampling feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is the middle layer of the decoder, the second input data is the processing result output by the second DPRNN block of the previous group; The up-sampling feature data is subjected to feature extraction processing by the second TFC-TDF block to obtain third intermediate feature data; The third intermediate feature data is subjected to feature extraction processing by the second DPRNN block to obtain fourth intermediate feature data; After the feature extraction processing of the second DPRNN block of the tail layer is completed, the fourth intermediate feature data obtained by the second DPRNN block of the tail layer is determined as the decoded feature data.

8. The musical source separation method according to any one of claims 1 to 7, characterized by, The neural network model further comprises a DPRNN middle layer, and the feature decoding processing of the encoded feature data by the decoder to output the music source separation result comprises: The encoded feature data is subjected to feature extraction processing by the DPRNN middle layer to obtain high-dimensional feature data; The high-dimensional feature data is subjected to feature decoding processing by the decoder to output the music source separation result.

9. The musical source separation method according to any one of claims 1 to 7, wherein, After the feature decoding processing of the encoded feature data by the decoder to output the music source separation result, the method further comprises: The music source separation result is subjected to inverse Fourier transform processing to obtain a separated time-domain audio track signal, wherein the time-domain audio track signal comprises a human voice audio track signal, a drum audio track signal, a bass audio track signal and other audio track signals.

10. A wearable device, comprising: The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage medium storing a computer program. The application also provides a computer readable storage

Images