Audio signal separation method and device, equipment and storage medium

Abstract

The invention discloses an audio signal separation method and device, equipment and a storage medium, and belongs to audio signal processing. The method comprises the following steps: performing Fourier transform on an audio signal to obtain a frequency domain signal; calling a cyclic network to obtain a target masking value corresponding to the target track based on the frequency domain signal; the target masking value is used for indicating a frequency point where the audio signal of the target track is located; the loop network comprises a selection network and a state space model which are connected in sequence, and the selection network is used for indicating focused data in the input data for the state space model; determining a product of the frequency domain signal and the target masking value as a target track frequency domain signal of the target track; and performing inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track. The method can improve the separation efficiency of the audio signals.

Description

Technical Field

[0001] The present application relates to audio signal processing, and in particular to an audio signal separation method, device, equipment and storage medium. Background Art

[0002] Audio signal separation is used to separate an audio signal into multiple audio tracks. For example, the audio signal of a song can be separated into vocal audio and accompaniment audio, or the accompaniment audio can be separated into audio of different instrument tracks such as drum audio, bass audio, piano audio, etc.

[0003] When performing audio signal separation, the related technology uses the Transformer model to perform time domain modeling and frequency domain modeling on the audio signal to extract the correlation of the audio signal in the time domain and frequency domain, and predict the masking value of the target audio track. The track signal of the target audio track is calculated based on the predicted masking value.

[0004] However, when using the Transformer model for inference, the amount of computation of the Transformer model's self-attention mechanism increases quadratically with the increase in context length. When used to calculate long sequence input data such as audio signals, the amount of computation is huge and the efficiency of audio signal separation is low. Summary of the invention

[0005] The present application provides an audio signal separation method, device, equipment and storage medium, which can improve the separation efficiency of audio signals. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for separating an audio signal is provided, the method comprising:

[0007] Performing Fourier transform on the audio signal to obtain a frequency domain signal;

[0008] Calling a recurrent network to obtain a target masking value corresponding to a target track based on the frequency domain signal; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the recurrent network includes a selection network and a state space model connected in sequence, and the selection network is used to indicate key attention data in the input data to the state space model;

[0009] Determine the product of the frequency domain signal and the target masking value as the target track frequency domain signal of the target track;

[0010] Perform inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0011] According to another aspect of the present application, there is provided an audio signal separation device, the device comprising:

[0012] A transform module, used for performing Fourier transform on the audio signal to obtain a frequency domain signal;

[0013] A loop module, used for calling a loop network to obtain a target masking value corresponding to a target track based on the frequency domain signal; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the loop network includes a selection network and a state space model connected in sequence, and the selection network is used to indicate key attention data in the input data to the state space model;

[0014] A separation module, configured to determine a target track frequency domain signal of the target track by multiplying the frequency domain signal by the target masking value;

[0015] The inverse transform module is used to perform inverse Fourier transform on the target track frequency domain signal to obtain the target track audio signal of the target track.

[0016] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the audio signal separation method as described above.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the audio signal separation method described in the above aspect.

[0018] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the audio signal separation method provided in various optional implementations of the above aspects.

[0019] The beneficial effects of the technical solution provided by this application include at least:

[0020] A selection network and a state space model are used to model the frequency domain signal. Since the state space model uses linear operations and predicts the output based on the hidden state stored in the model based on the state transfer equation and the output equation, the amount of computation increases linearly in the scenario of processing long sequence input data such as audio signals or frequency domain signals, which greatly reduces the amount of model computation compared to the quadratic growth of the Transformer model. In addition, a selection network is added in front of the state space model, and the selection network selects key focus data in the input data, filters irrelevant data in the input data, and refines the data content of the input data, thereby further reducing the amount of computation of the state space model and improving the prediction accuracy and computational efficiency of the state space model. The audio signal separation method provided in the embodiment of the present application can accurately and efficiently separate the track audio signal of the target track from the audio signal, thereby facilitating the deployment and application of the audio signal separation function. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of the structure of a computer system provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a schematic diagram of an audio separation model provided by an exemplary embodiment of the present application;

[0025] Figure 4 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a time domain network provided by an exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a frequency domain network provided by an exemplary embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a cyclic network provided by an exemplary embodiment of the present application;

[0029] Figure 8 is a schematic diagram of an audio separation model provided by an exemplary embodiment of the present application;

[0030] Fig. 9 is a schematic diagram of a cyclic network provided by an exemplary embodiment of the present application;

[0031] Fig.10 is a schematic diagram of an audio separation model provided by an exemplary embodiment of the present application;

[0032] Fig.11 is a schematic diagram of a cyclic network provided by an exemplary embodiment of the present application;

[0033] Fig.12 is a structural schematic diagram of an audio signal separation device provided by an exemplary embodiment of the present application;

[0034] Fig.13 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0037] Figure 1 A schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown, and the computer system may include a terminal device 101 and a server 103 .

[0038] Exemplarily, the audio signal separation method shown in the embodiment of the present application can be applied in a terminal device, and the terminal device 101 runs an application 102 with an audio signal separation requirement. The terminal device may include a mobile phone, a tablet computer, a notebook computer, a laptop computer, a desktop computer, an all-in-one computer, an Internet of Things device, an intelligent robot workstation, a television, a set-top box, smart glasses, a smart watch, a digital camera, an MP4 player, an MP5 player, a learning machine, a point-reading machine, an electronic paper book, an electronic dictionary, a vehicle-mounted device, a virtual reality (VR) player or an augmented reality (AR) player, etc.

[0039] Exemplarily, the audio signal separation method provided in the present application can be executed by a client in a terminal device. The client is a client of an application that has an audio signal separation requirement. For example, the application may include at least one of the following: an audio player, an audio application, a video application, a social application, a shopping application, a live broadcast application, an in-car audio playback application, an information application, a browser, a game application, and a recording application.

[0040] For example, an audio player provides a singing function, and the singing function requires the original audio to be separated into vocal audio and accompaniment audio. The audio player can use the audio signal separation method provided in this application to separate the accompaniment audio from the original audio for playback by the singing function.

[0041] The terminal device 101 includes a first memory and a first processor. An audio signal separation program is stored in the first memory; the audio signal separation program is called and executed by the first processor to implement the audio signal separation method provided in the present application. The first memory may include but is not limited to the following: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electric Erasable Programmable Read-Only Memory (EEPROM).

[0042] The first processor may be composed of one or more integrated circuit chips. Optionally, the first processor may be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Optionally, the first processor may implement the audio signal separation method provided in the present application by running a program or code.

[0043] In an optional embodiment, the terminal device 101 and the server 103 may be connected to each other via a wired or wireless network.

[0044] Exemplarily, the audio signal separation method provided in the present application can also be executed by a server. The server 103 is used to provide background services for the target use case system of the terminal device 101. Optionally, the server 103 undertakes the main computing work and the terminal device 101 undertakes the secondary computing work; or, the server 103 undertakes the secondary computing work and the terminal device 101 undertakes the main computing work; or, the server 103 and the terminal device 101 adopt a distributed computing architecture for collaborative computing.

[0045] The server 103 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing service center.

[0046] Optionally, the server 103 includes a second memory and a second processor. The second memory stores an audio signal separation program; the audio signal separation program is called by the second processor to implement the audio signal separation method provided in the present application. Optionally, the second memory may include but is not limited to the following: RAM, ROM, PROM, EPROM, EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0047] Figure 2 FIG. 1 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown in the figure. The method comprises the following steps.

[0048] Step 210: Perform Fourier transform on the audio signal to obtain a frequency domain signal.

[0049] Exemplarily, a short-time Fourier transform (STFT) is performed on the audio signal to obtain a frequency domain signal.

[0050] An audio signal can be separated into track signals of at least one audio track. Audio tracks are different channels of an audio signal, which are used to separate different sound elements from the audio signal, such as musical instruments, human voices, ambient sounds, etc., so as to facilitate editing, mixing and processing of track signals of different tracks. For example, song audio can usually be separated into track signals of a vocal track and an accompaniment track, and the accompaniment audio can be separated into track signals of audio tracks corresponding to multiple instruments. The track signals of at least one audio track separated from the audio signal can be combined and restored to obtain an audio signal.

[0051] Exemplarily, the frequency domain signal includes a frequency domain signal obtained by Fourier transforming at least one frame of the time domain signal of the audio signal. For example, the frequency domain signal can be represented as a three-dimensional signal of T*F*C, where T is time / frame number, F is frequency point, and C is amplitude.

[0052] Step 220: Call the recurrent network to obtain a target masking value corresponding to the target track based on the frequency domain signal; the target masking value is used to indicate the frequency point where the audio signal of the target track is located; the recurrent network includes a selection network and SSM connected in sequence, and the selection network is used to indicate the key focus data in the input data to the SSM.

[0053] The target masking value is used to indicate the frequency point where the track signal of the target track in the frequency domain signal is located, as well as the signal strength at the frequency point. The target masking value has the same dimension as the frequency domain signal, that is, the dimension of the target masking value is also T*F*C, where C represents the masking value corresponding to the T and F positions, and the masking value ranges from 0 to 1. For example, the point (1,1,0) in the target masking value indicates that the masking value of the first frequency point position in the first frame is 0, that is, there is no track signal of the target track at the first frequency point.

[0054] The recurrent network is used to predict the target masking value corresponding to the target track. After obtaining the target masking value, the target masking value is multiplied by the frequency domain signal to obtain the target track frequency domain signal of the target track. Exemplarily, this embodiment takes the target track as an example for explanation. If it is necessary to separate the track signals of multiple tracks from the audio signal, a recurrent network can be trained for each track to predict the masking value corresponding to the track.

[0055] The recurrent network is a neural network model with a recurrent structure. The recurrent network is used to model the input frequency domain signal in at least one dimension in the frequency domain or time domain, and predict the masking value corresponding to the target track based on the characteristic information of the frequency domain signal in the time domain or frequency domain dimension.

[0056] It should be noted that the recurrent network in this embodiment adopts SSM (Structured Space Model). SSM is a model used to describe the state representation of a sequence at each time step and predict its next state based on the input. SSM transforms the input sequence through the hidden state representation and finally generates an output sequence. The core idea is to capture the key information in the sequence through the hidden state, so as to achieve efficient sequence modeling. SSM defines the state transfer equation and the output equation. The state transfer equation describes how the hidden state changes over time, and the output equation describes how to get the output data from the hidden state.

[0057] In the related art, the recurrent network implemented by the Transformer model needs to calculate the attention score between each element and other elements when processing long sequences, which requires a huge amount of calculation. However, the SSM of this embodiment uses linear operations. As the sequence length increases, the amount of calculation of the model increases linearly, which can significantly increase the reasoning speed of the recurrent network.

[0058] The cyclic network in this embodiment adds a selection network before the SSM to filter out irrelevant information in the input data, retain necessary information, and input the selected input data into the SSM, so that the SSM can manage information more effectively and improve the accuracy and operation efficiency of the model.

[0059] Exemplarily, the network structure of the selection network can be arbitrary. For example, the selection network can be implemented using a convolutional network, or the selection network can be implemented using an encoder and a decoder.

[0060] In an optional embodiment, the cyclic network includes a time domain network and a frequency domain network connected sequentially; the time domain network includes a time domain selection network and a time domain SSM, and the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the time domain selection network is used to indicate time domain focus data for the time domain SSM in the time domain dimension; the frequency domain selection network is used to indicate frequency domain focus data for the frequency domain SSM in the frequency domain dimension.

[0061] The time domain network is used to model the frequency domain signal in the time domain dimension. Exemplarily, the frequency domain signal is sequentially input into the time domain network according to the frame sequence in the time domain dimension (T), and the time domain SSM performs state prediction according to the input frame sequence.

[0062] Correspondingly, the frequency domain network is used to model the frequency domain signal in the frequency domain dimension. Exemplarily, in the frequency domain dimension (F), the frequency domain signal is sequentially input into the frequency domain network according to the subband sequence / frequency point sequence, and the frequency domain SSM performs state prediction according to the input subband sequence / frequency point sequence. Among them, at least one frequency point can be divided into a subband according to a fixed frequency domain interval. Exemplarily, the input of the frequency domain network is the output data of the time domain network, and the output data of the time domain network has the same dimension as the frequency domain signal.

[0063] In another optional embodiment, the cyclic network includes a time domain network; the time domain network includes a time domain selection network and a time domain SSM; the time domain selection network is used to indicate time domain focus data for the time domain SSM in the time domain dimension.

[0064] Exemplarily, taking the i-th cycle in the recurrent network as an example, the time domain input data is input into the sequentially connected time domain selection network and the time domain SSM to obtain the time domain output data; the time domain selection network is used to indicate the time domain focus data for the time domain SSM in the time domain dimension; the time domain input data is obtained based on the i-th input data; and the time domain output data is determined as the i-th output data.

[0065] In yet another optional embodiment, the cyclic network includes a frequency domain network; the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the frequency domain selection network is used to indicate frequency domain focus data for the frequency domain SSM in the frequency domain dimension.

[0066] Exemplarily, taking the i-th cycle in the recurrent network as an example, the frequency domain input data is input into the sequentially connected frequency domain selection network and the frequency domain SSM to obtain frequency domain output data; the frequency domain selection network is used to indicate the frequency domain focus data for the frequency domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data; and the frequency domain output data is determined as the i-th output data.

[0067] The following will take the cyclic network including the time domain network and the frequency domain network as an example to explain the network structure of the cyclic network in detail. When the cyclic network only includes the time domain network or the frequency domain network, its network structure can also refer to the time domain network structure or the frequency domain network structure given below.

[0068] Step 230: The product of the frequency domain signal and the target masking value is determined as the target track frequency domain signal of the target track.

[0069] Exemplarily, the target masking value is used to indicate the frequency point where the audio signal of the target track is located, and the signal strength of the target track at the frequency point. The frequency domain signal includes at least one frame of frequency domain signal, and each frame of frequency domain signal includes an amplitude value at at least one frequency point; then the target masking value includes a group of masking values ​​corresponding to at least one frame of frequency domain signal, and a group of masking values ​​includes a masking value (a value from 0 to 1) corresponding to at least one frequency point.

[0070] The amplitude of the frequency domain signal is multiplied by the mask value of the corresponding position in the target mask value (the same frequency point in the same frame) to obtain the amplitude of the target track frequency domain signal at that position. For example, the amplitude of the frequency domain signal at the first frequency point in the first frame is 10, and the mask value of the target mask value at the first frequency point in the first frame is 0.1, then the amplitude of the target track frequency domain signal at the first frequency point in the first frame is 10*0.1=1. In this way, the frequency domain signal is multiplied by the corresponding positions of the target mask value to obtain the target track frequency domain signal of the target track.

[0071] Step 240: Perform inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0072] Exemplarily, a short-time inverse Fourier transform is performed on the target track frequency domain signal to obtain a time domain signal corresponding to the target track: a target track audio signal.

[0073] This embodiment is described by taking the example of separating the track audio signal of the target track from the audio signal. In this way, the track audio signals of other tracks can also be separated from the audio signal.

[0074] In summary, the method provided in this embodiment uses a selection network and a state space model to model the frequency domain signal. Since the state space model uses linear operations, it predicts the output based on the hidden state stored in the model based on the state transfer equation and the output equation. In the scenario of processing long sequence input data such as audio signals or frequency domain signals, its amount of calculation increases linearly, which greatly reduces the amount of model calculation compared to the quadratic growth of the Transformer model. In addition, a selection network is added in front of the state space model, and the key focus data in the input data is selected through the selection network, and irrelevant data in the input data is filtered out, and the data content of the input data is refined, so as to further reduce the amount of calculation of the state space model and improve the prediction accuracy and operation efficiency of the state space model. The audio signal separation method provided in the embodiment of the present application can accurately and efficiently separate the track audio signal of the target track from the audio signal, which is convenient for the deployment and application of the audio signal separation function.

[0075] In an optional embodiment, the recurrent network includes a time domain network and a frequency domain network. Figure 3 As shown, an audio separation model is used to perform the audio signal separation method of the present application. The audio separation model includes a recurrent network, and the audio separation model includes a first multilayer perceptron (MLP) 301, a time domain network 302, a frequency domain network 303, and a second multilayer perceptron 304 connected in sequence.

[0076] Based on this network structure, the audio signal separation method can be briefly summarized as the following process: (1) Convert the audio signal into a frequency domain signal through short-time Fourier transform, divide the frequency domain signal into frames to form a sequence in the time dimension, and divide the frequency domain signal into sub-bands according to frequency points to form a sequence in the frequency domain dimension; (2) Perform sequence modeling on the sequences in the time dimension and frequency domain dimension respectively, and predict the masking value existing in the audio track; (3) Multiply the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and perform short-time Fourier inverse transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0077] Figure 4 FIG. 1 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown. Figure 3 The network structure of the recurrent network shown is based on Figure 2 In the illustrated embodiment, step 220 includes steps 221 to 224 .

[0078] Step 210 : Perform Fourier transform on the audio signal 310 to obtain a frequency domain signal 320 .

[0079] Exemplarily, according to the preset time window length and time window offset, the audio signal is framed to obtain at least one frame of audio signal, the time length of one frame of audio signal is equal to the time window length, and the distance between the starting position of the next frame of time window and the starting position of the current frame of time window is equal to the time window offset. Fourier transform is performed on at least one frame of audio signal to obtain at least one frame of frequency domain signal, and at least one frame of frequency domain signal constitutes frequency domain signal 320. That is, short-time Fourier transform is performed on audio signal 310 to obtain frequency domain signal 320.

[0080] Step 221: Input the frequency domain signal 320 into the first multilayer perceptron 301 to obtain the first input data of the first cycle of the recurrent network.

[0081] Exemplarily, the first multilayer perceptron includes an input layer, at least one hidden layer and an output layer, each layer of the network includes a number of neurons, and the neurons are connected by weights. The first multilayer perceptron is used to perform sub-band segmentation on the frequency domain signal 320, divide at least one frequency point into a sub-band, and obtain the first input data after sub-band segmentation. The dimension of the first input data is T*N*D, where T is time / frame, N is sub-band, and D is amplitude. The first input data is the sub-band segmentation result obtained after sub-band segmentation of the frequency domain signal.

[0082] Exemplarily, the first multilayer perceptron 301 is an optional or replaceable network structure, and when the first multilayer perceptron 301 is deleted, the frequency domain signal 320 can be used as the first input data to perform subsequent steps.

[0083] Step 222: for the i-th cycle in the cyclic network, input the i-th input data into the cyclic network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key focus data in the i-th input data for the SSM.

[0084] That is, the output data of the current loop network is used as the input data for the next loop until the loop ends.

[0085] Exemplarily, the cyclic network includes a time domain network 302 and a frequency domain network 303 connected in sequence; the time domain network 302 includes a time domain selection network and a time domain SSM, and the frequency domain network 303 includes a frequency domain selection network and a frequency domain SSM. Then step 222 may include the following steps:

[0086] Step 1: Input the time domain input data into the time domain selection network and the time domain SSM connected in sequence to obtain the time domain output data; the time domain selection network is used to indicate the time domain focus data to the time domain SSM in the time domain dimension; the time domain input data is obtained based on the i-th input data;

[0087] In an optional embodiment, if Figure 5As shown, the time domain network 302 includes: a first linear network 401 , a second linear network 402 , a third linear network 403 , a first time domain selection network 404 , a second time domain selection network 405 , a first time domain SSM 406 , and a second time domain SSM 407 .

[0088] according to Figure 5 In the network structure of the time domain network 302 shown, step 1 can be specifically implemented as follows:

[0089] 1) Inputting time domain input data into the first linear network 401 to obtain a first linear result. The time domain input data may be the i-th input data; or the time domain input data may be obtained based on the i-th input data.

[0090] Exemplarily, the first linear network 401 is an optional or replaceable network structure. When the first linear network 401 is deleted, the subsequent steps can be directly performed using the time domain input data as the first linear result.

[0091] 2) Input the first linear result into the first time domain selection network 404 to obtain the first time domain selection result. The first time domain selection network may include at least one convolution layer. The first time domain selection network is used to indicate key data in the first linear result from the time domain dimension.

[0092] 3) Calling a first activation function to activate the first time domain selection result to obtain a first activation result.

[0093] Exemplarily, the first activation function is an optional or replaceable network structure. When the first activation function is deleted, the first time domain selection result can be used as the first activation result to perform subsequent steps.

[0094] 4) Input the first activation result into the first time domain SSM 406 to obtain a first prediction result.

[0095] 5) Flip the first linear result in the time domain dimension to obtain a first flip result, so that the time domain network can perform bidirectional modeling in the time domain dimension from both forward and reverse directions, extract the correlation of time domain data from both directions, and improve the prediction accuracy.

[0096] 6) Input the first flip result into the second time domain selection network 405 to obtain a second time domain selection result. The second time domain selection network may include at least one convolution layer. The second time domain selection network is used to indicate key data in the first flip result from the time domain dimension.

[0097] 7) Calling a second activation function to activate the second time domain selection result to obtain a second activation result.

[0098] Exemplarily, the second activation function is an optional or replaceable network structure. When the second activation function is deleted, the second time domain selection result can be used as the second activation result to perform subsequent steps. 8) The second activation result is input into the second time domain SSM 407 to obtain a second prediction result.

[0099] For example, Figure 5 The branch network that flips the input data is an optional or replaceable network structure. When deleting the flipped branch network, the above steps 5)-8) can be deleted, the subsequent steps 12)-14) can be deleted, and the first weighted sum in step 15) can be replaced by the first product.

[0100] 9) Input the time domain input data into the second linear network 402 to obtain a second linear result.

[0101] Exemplarily, the second linear network 402 is an optional or replaceable network structure. When the second linear network 402 is deleted, the time domain input data can be directly used as the second linear result to execute subsequent steps.

[0102] 10) Calling a third activation function to activate the second linear result to obtain a third activation result.

[0103] Exemplarily, the third activation function is an optional or replaceable network structure. When the third activation function is deleted, the second linear result can be used as the third activation result to perform subsequent steps.

[0104] 11) Calculate the first prediction result and multiply the third activation result to obtain a first product.

[0105] 12) Calculate the second prediction result and multiply the third activation result to obtain a second product.

[0106] 13) Flip the second product in the time domain dimension to obtain a second flipped result.

[0107] 14) Calculate a first weighted sum of the first product and the second flip result. Exemplarily, the weight may be a preset value, or may be obtained by training during the model training process. For example, the weights of the first product and the second flip result may both be 0.5.

[0108] 15) Input the first weighted sum into the third linear network 403 to obtain time domain output data. Exemplarily, the time domain network does not change the dimension of the input data, that is, the dimension of the time domain output data is the same as the dimension of the time domain input data, for example, the time domain output data is also data of T*N*D dimension.

[0109] Exemplarily, the third linear network 403 is an optional or replaceable network structure. When the third linear network 403 is deleted, the first weighted sum can be directly used as the time domain output data.

[0110] Step 2: Input the frequency domain input data into the frequency domain selection network and the frequency domain SSM connected in sequence to obtain the frequency domain output data; the frequency domain selection network is used to indicate the time domain focus data to the time domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the time domain output data.

[0111] In an optional embodiment, if Figure 6 As shown, the frequency domain network 303 includes: a fourth linear network 501, a fifth linear network 502, a sixth linear network 503, a first frequency domain selection network 504, a second frequency domain selection network 505, a first frequency domain SSM 506, and a second frequency domain SSM 507. The network structure of the time domain network is the same as that of the frequency domain network, but the network parameters are different, and the time domain network processes the input data according to the sequence of the time domain dimension, while the frequency domain network processes the input data according to the sequence of the frequency domain dimension.

[0112] according to Figure 6 In the network structure of the frequency domain network 303 shown in FIG. 1 , step 2 can be specifically implemented as follows:

[0113] 1) Inputting frequency domain input data into the fourth linear network 501 to obtain a fourth linear result. The frequency domain input data may be time domain output data; or the frequency domain input data may be obtained based on the time domain output data.

[0114] Exemplarily, the fourth linear network 501 is an optional or replaceable network structure. When the fourth linear network 501 is deleted, the frequency domain input data can be directly used as the fourth linear result to perform subsequent steps.

[0115] 2) Input the fourth linear result into the first frequency domain selection network 504 to obtain a first frequency domain selection result. The first frequency domain selection network may include at least one convolution layer. The first frequency domain selection network is used to indicate key data in the fourth linear result from a frequency domain dimension.

[0116] 3) Calling the fourth activation function to activate the first frequency domain selection result to obtain a fourth activation result.

[0117] Exemplarily, the fourth activation function is an optional or replaceable network structure. When the fourth activation function is deleted, the first frequency domain selection result can be used as the fourth activation result to perform subsequent steps.

[0118] 4) Input the fourth activation result into the first frequency domain SSM 506 to obtain a third prediction result.

[0119] 5) Flip the fourth linear result in the frequency domain dimension to obtain the third flip result. This enables the frequency domain network to perform bidirectional modeling in the frequency domain dimension from both the forward and reverse directions, extract the correlation of frequency domain data from both directions, and improve prediction accuracy.

[0120] 6) Input the third flip result into the second frequency domain selection network 505 to obtain a second frequency domain selection result. The second frequency domain selection network may include at least one convolution layer. The second frequency domain selection network is used to indicate key data in the third flip result from the frequency domain dimension.

[0121] 7) Calling the fifth activation function to activate the second frequency domain selection result to obtain a fifth activation result.

[0122] Exemplarily, the fifth activation function is an optional or replaceable network structure. When the fifth activation function is deleted, the second frequency domain selection result can be used as the fifth activation result to perform subsequent steps.

[0123] 8) Input the fifth activation result into the second frequency domain SSM 507 to obtain a fourth prediction result.

[0124] For example, Figure 6 The branch network that flips the input data is an optional or replaceable network structure. When the flipped branch network is deleted, the above steps 5)-8) can be deleted, the subsequent steps 12)-14) can be deleted, and the second weighted sum in step 15) can be replaced by the third product.

[0125] 9) Input the frequency domain input data into the fifth linear network 502 to obtain a fifth linear result.

[0126] Exemplarily, the fifth linear network 502 is an optional or replaceable network structure. When the fifth linear network 502 is deleted, the frequency domain input data can be directly used as the fifth linear result to perform subsequent steps.

[0127] 10) Call the sixth activation function to activate the fifth linear result to obtain a sixth activation result.

[0128] Exemplarily, the sixth activation function is an optional or replaceable network structure. When the sixth activation function is deleted, the fifth linear result can be used as the sixth activation result to perform subsequent steps.

[0129] 11) Calculate the third prediction result and multiply the sixth activation result to obtain a third product.

[0130] 12) Calculate the fourth prediction result and multiply the sixth activation result to obtain a fourth product.

[0131] 13) Flip the fourth product in the frequency domain dimension to obtain a fourth flipped result.

[0132] 14) Calculate a second weighted sum of the third product and the fourth flip result. Exemplarily, the weight may be a preset value, or may be obtained by training during the model training process. For example, the weights of the third product and the fourth flip result may both be 0.5.

[0133] 15) Input the second weighted sum into the sixth linear network 503 to obtain frequency domain output data. Exemplarily, the frequency domain network does not change the dimension of the input data, that is, the dimension of the frequency domain output data is the same as the dimension of the frequency domain input data, for example, the frequency domain output data is also data of T*N*D dimension.

[0134] Exemplarily, the sixth linear network 503 is an optional or replaceable network structure. When the sixth linear network 503 is deleted, the second weighted sum can be directly used as frequency domain output data.

[0135] Step 3: Determine the frequency domain output data as the i-th output data.

[0136] In an optional embodiment, in order to improve model performance and training stability, normalization and skip connection structures are introduced before and after the time domain network 302 and the frequency domain network 303, respectively.

[0137] like Figure 7 As shown, the input data needs to be normalized before entering the time domain network 302; the output data of the time domain network 302 also needs to be convolved by the first convolutional network 408 and spliced ​​with the input data, and the splicing result is used as the input data that needs to be input to the frequency domain network 303. Before entering the frequency domain network 303, the input data also needs to be normalized; the output data of the frequency domain network 303 also needs to be convolved by the second convolutional network 409 and spliced ​​with the input data of the frequency domain network, and the splicing result is used as the final output data. The adjusted method is as follows.

[0138] Before step 1, the method further includes: obtaining time domain input data based on the i-th input data; for example, determining the normalized i-th input data as the time domain input data.

[0139] Step 1: Input the time domain input data into the sequentially connected time domain selection network and the time domain SSM (ie, the time domain network 302) to obtain the time domain output data.

[0140] Before step 2, it also includes: splicing the i-th input data with the time domain output data to obtain frequency domain input data; for example, inputting the time domain output data into the first convolution network 408 to obtain a first convolution result; splicing the first convolution result with the i-th input data to obtain a first splicing result; and determining the normalized first splicing result as the frequency domain input data.

[0141] Step 2: Input the frequency domain input data into the frequency domain selection network and the frequency domain SSM (ie, the frequency domain network 303) connected in sequence to obtain the frequency domain output data.

[0142] Step 3 can be replaced by: splicing the frequency domain output data with the frequency domain input data to obtain the i-th output data. For example, the frequency domain output data is input into the second convolution network 409 to obtain a second convolution result; the second convolution result is spliced ​​with the first splicing result to obtain a second splicing result; and the second splicing result is determined as the i-th output data.

[0143] Step 223: When the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed.

[0144] The loop end condition may be that the number of loops reaches a number threshold, or the loop end condition may be that the i-th output data reaches a data threshold.

[0145] Step 224: When the loop end condition is met, the i-th output data is input into the second multilayer perceptron 304 to obtain the target masking value corresponding to the target track.

[0146] Exemplarily, the second multilayer perceptron includes an input layer, at least one hidden layer and an output layer, each layer of the network includes a plurality of neurons, and the neurons are connected by weights. The second multilayer perceptron is used to predict the target masking value according to the i-th output data.

[0147] Exemplarily, the second multilayer perceptron 304 is an optional or replaceable network structure, and when the second multilayer perceptron 304 is deleted, the i-th output data can be used as the target mask value.

[0148] In an optional embodiment, the time domain network and / or the frequency domain network can also be implemented using the Mamba model. The design of the Mamba model is optimized for currently available high-performance computing resources and uses a hardware-aware algorithm. This design enables Mamba to run more efficiently on computing hardware such as a GPU (Graphic Processing Unit) and improves model performance.

[0149] In summary, the method provided in this embodiment has a slow reasoning speed and consumes a long memory because the amount of calculation of the self-attention mechanism in the Transformer model increases quadratically with the increase of the context length. The cyclic model provided in this application combines the advantages of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), and can selectively focus on or ignore certain parts of the input data, with an effect comparable to that of Transformer. At the same time, the cyclic model uses SSM to model the input data in the time domain and frequency domain. The computational complexity of its modeling is linear with the duration, and the hardware-aware algorithm is used to make it more efficient when running on the GPU.

[0150] In an optional embodiment, the recurrent network includes a time domain network. Figure 8 As shown, an audio separation model is used to perform the audio signal separation method of the present application. The audio separation model includes a recurrent network, and the audio separation model includes a first multilayer perceptron (MLP) 301, a time domain network 302, and a second multilayer perceptron 304 connected in sequence.

[0151] Based on this network structure, the audio signal separation method can be briefly summarized as the following process: (1) Convert the audio signal into a frequency domain signal through short-time Fourier transform, divide the frequency domain signal into frames to form a sequence in the time dimension, and divide the frequency domain signal into sub-bands according to frequency points to form a sequence in the frequency domain dimension; (2) Perform sequence modeling on the sequence in the time dimension and predict the masking value of the audio track; (3) Multiply the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and perform short-time Fourier inverse transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0152] Figure 4 FIG. 1 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown. Figure 8 The network structure of the recurrent network shown is based on Figure 2 In the illustrated embodiment, step 220 includes steps 221 to 224 .

[0153] Step 210 : Perform Fourier transform on the audio signal 310 to obtain a frequency domain signal 320 .

[0154] Step 221: Input the frequency domain signal 320 into the first multilayer perceptron 301 to obtain the first input data of the first cycle of the recurrent network.

[0155] Exemplarily, the first multilayer perceptron 301 is an optional or replaceable network structure, and when the first multilayer perceptron 301 is deleted, the frequency domain signal 320 can be used as the first input data to perform subsequent steps.

[0156] Step 222: for the i-th cycle in the cyclic network, input the i-th input data into the cyclic network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key focus data in the i-th input data for the SSM.

[0157] Exemplarily, the cyclic network includes a time domain network 302; the time domain network 302 includes a time domain selection network and a time domain SSM. Then step 222 may include the following steps:

[0158] Step 1: Input the time domain input data into the time domain selection network and the time domain SSM connected in sequence to obtain the time domain output data; the time domain selection network is used to indicate the time domain focus data to the time domain SSM in the time domain dimension; the time domain input data is obtained based on the i-th input data;

[0159] In an optional embodiment, if Figure 5 As shown, the time domain network 302 includes: a first linear network 401 , a second linear network 402 , a third linear network 403 , a first time domain selection network 404 , a second time domain selection network 405 , a first time domain SSM 406 , and a second time domain SSM 407 .

[0160] according to Figure 5 In the network structure of the time domain network 302 shown, step 1 can be specifically implemented as follows:

[0161] 1) Inputting time domain input data into the first linear network 401 to obtain a first linear result. The time domain input data may be the i-th input data; or the time domain input data may be obtained based on the i-th input data.

[0162] Exemplarily, the first linear network 401 is an optional or replaceable network structure. When the first linear network 401 is deleted, the subsequent steps can be directly performed using the time domain input data as the first linear result.

[0163] 2) Input the first linear result into the first time domain selection network 404 to obtain the first time domain selection result. The first time domain selection network may include at least one convolution layer. The first time domain selection network is used to indicate key data in the first linear result from the time domain dimension.

[0164] 3) Calling a first activation function to activate the first time domain selection result to obtain a first activation result.

[0165] Exemplarily, the first activation function is an optional or replaceable network structure. When the first activation function is deleted, the first time domain selection result can be used as the first activation result to perform subsequent steps.

[0166] 4) Input the first activation result into the first time domain SSM 406 to obtain a first prediction result.

[0167] 5) Flip the first linear result in the time domain dimension to obtain a first flip result, so that the time domain network can perform bidirectional modeling in the time domain dimension from both forward and reverse directions, extract the correlation of time domain data from both directions, and improve the prediction accuracy.

[0168] 6) Input the first flip result into the second time domain selection network 405 to obtain a second time domain selection result. The second time domain selection network may include at least one convolution layer. The second time domain selection network is used to indicate key data in the first flip result from the time domain dimension.

[0169] 7) Calling a second activation function to activate the second time domain selection result to obtain a second activation result.

[0170] Exemplarily, the second activation function is an optional or replaceable network structure. When the second activation function is deleted, the second time domain selection result can be used as the second activation result to perform subsequent steps. 8) The second activation result is input into the second time domain SSM 407 to obtain a second prediction result.

[0171] For example, Figure 5 The branch network that flips the input data is an optional or replaceable network structure. When deleting the flipped branch network, the above steps 5)-8) can be deleted, the subsequent steps 12)-14) can be deleted, and the first weighted sum in step 15) can be replaced by the first product.

[0172] 9) Input the time domain input data into the second linear network 402 to obtain a second linear result.

[0173] Exemplarily, the second linear network 402 is an optional or replaceable network structure. When the second linear network 402 is deleted, the time domain input data can be directly used as the second linear result to execute subsequent steps.

[0174] 10) Calling a third activation function to activate the second linear result to obtain a third activation result.

[0175] Exemplarily, the third activation function is an optional or replaceable network structure. When the third activation function is deleted, the second linear result can be used as the third activation result to perform subsequent steps.

[0176] 11) Calculate the first prediction result and multiply the third activation result to obtain a first product.

[0177] 12) Calculate the second prediction result and multiply the third activation result to obtain a second product.

[0178] 13) Flip the second product in the time domain dimension to obtain a second flipped result.

[0179] 14) Calculate a first weighted sum of the first product and the second flip result. Exemplarily, the weight may be a preset value, or may be obtained by training during the model training process. For example, the weights of the first product and the second flip result may both be 0.5.

[0180] 15) Input the first weighted sum into the third linear network 403 to obtain time domain output data. Exemplarily, the time domain network does not change the dimension of the input data, that is, the dimension of the time domain output data is the same as the dimension of the time domain input data, for example, the time domain output data is also data of T*N*D dimension.

[0181] Exemplarily, the third linear network 403 is an optional or replaceable network structure. When the third linear network 403 is deleted, the first weighted sum can be directly used as the time domain output data.

[0182] Step 2: Determine the time domain output data as the i-th output data.

[0183] In an optional embodiment, in order to improve model performance and training stability, normalization and skip connection structures are introduced before and after the time domain network 302.

[0184] like Fig. 9 As shown, the input data needs to be normalized before being input into the time domain network 302; the output data of the time domain network 302 also needs to be convolved by the first convolutional network 408 and concatenated with the input data, and the concatenated result is used as the final output data. The adjusted method is as follows.

[0185] Before step 1, the method further includes: obtaining time domain input data based on the i-th input data; for example, determining the normalized i-th input data as the time domain input data.

[0186] Step 1: Input the time domain input data into the sequentially connected time domain selection network and the time domain SSM (ie, the time domain network 302) to obtain the time domain output data.

[0187] Step 2 may be replaced by: splicing the time domain output data with the time domain input data to obtain the i-th output data. For example, the time domain output data is input into the first convolutional network 408 to obtain a first convolution result; the first convolution result is spliced ​​with the time domain input data to obtain a first splicing result; and the first splicing result is determined as the i-th output data.

[0188] Step 223: When the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed.

[0189] Step 224: When the loop end condition is met, the i-th output data is input into the second multilayer perceptron to obtain the target masking value corresponding to the target track.

[0190] Exemplarily, the second multilayer perceptron 304 is an optional or replaceable network structure, and when the second multilayer perceptron 304 is deleted, the i-th output data can be used as the target mask value.

[0191] In summary, the method provided in this embodiment has a slow reasoning speed and consumes a long memory because the amount of calculation of the self-attention mechanism in the Transformer model will increase quadratically with the increase of the context length. The cyclic model provided in this application combines the advantages of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), uses SSM to model the input data in the time domain dimension, and can selectively focus on or ignore certain parts of the input data, with an effect comparable to that of Transformer. At the same time, the cyclic model uses SSM to model the input data in the time domain, and the computational complexity of its modeling is linear with the duration, and the hardware-aware algorithm is used to make it more efficient when running on the GPU.

[0192] In an optional embodiment, the recurrent network includes a frequency domain network. Fig.10 As shown, an audio separation model is used to perform the audio signal separation method of the present application. The audio separation model includes a recurrent network, and the audio separation model includes a first multilayer perceptron (MLP) 301, a frequency domain network 303, and a second multilayer perceptron 304 connected in sequence.

[0193] Based on this network structure, the audio signal separation method can be briefly summarized as the following process: (1) Convert the audio signal into a frequency domain signal through short-time Fourier transform, divide the frequency domain signal into frames to form a sequence in the time dimension, and divide the frequency domain signal into sub-bands according to frequency points to form a sequence in the frequency domain dimension; (2) Perform sequence modeling on the sequence in the frequency domain dimension and predict the masking value of the audio track; (3) Multiply the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and perform short-time Fourier inverse transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0194] Figure 4FIG. 1 is a flow chart of an audio signal separation method provided by an exemplary embodiment of the present application. The method can be used for Figure 1 The terminal device or server shown. Fig.10 The network structure of the recurrent network shown is based on Figure 2 In the illustrated embodiment, step 220 includes steps 221 to 224 .

[0195] Step 210 : Perform Fourier transform on the audio signal 310 to obtain a frequency domain signal 320 .

[0196] Step 221: Input the frequency domain signal 320 into the first multilayer perceptron 301 to obtain the first input data of the first cycle of the recurrent network.

[0197] Exemplarily, the first multilayer perceptron 301 is an optional or replaceable network structure, and when the first multilayer perceptron 301 is deleted, the frequency domain signal 320 can be used as the first input data to perform subsequent steps.

[0198] Step 222: for the i-th cycle in the cyclic network, input the i-th input data into the cyclic network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key focus data in the i-th input data for the SSM.

[0199] Exemplarily, the cyclic network includes a frequency domain network 303; the frequency domain network 303 includes a frequency domain selection network and a frequency domain SSM. Then step 222 may include the following steps:

[0200] Step 1: Input the frequency domain input data into the frequency domain selection network and the frequency domain SSM connected in sequence to obtain the frequency domain output data; the frequency domain selection network is used to indicate the time domain focus data to the time domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data.

[0201] In an optional embodiment, if Figure 6 As shown, the frequency domain network 303 includes: a fourth linear network 501, a fifth linear network 502, a sixth linear network 503, a first frequency domain selection network 504, a second frequency domain selection network 505, a first frequency domain SSM 506, and a second frequency domain SSM 507. The network structure of the time domain network is the same as that of the frequency domain network, but the network parameters are different, and the time domain network processes the input data according to the sequence of the time domain dimension, while the frequency domain network processes the input data according to the sequence of the frequency domain dimension.

[0202] according to Figure 6 In the network structure of the frequency domain network 303 shown in FIG. 1 , step 2 can be specifically implemented as follows:

[0203] 1) Inputting frequency domain input data into the fourth linear network 501 to obtain a fourth linear result. The frequency domain input data may be the i-th input data; or the frequency domain input data may be obtained based on the i-th input data.

[0204] Exemplarily, the fourth linear network 501 is an optional or replaceable network structure. When the fourth linear network 501 is deleted, the frequency domain input data can be directly used as the fourth linear result to perform subsequent steps.

[0205] 2) Input the fourth linear result into the first frequency domain selection network 504 to obtain a first frequency domain selection result. The first frequency domain selection network may include at least one convolution layer. The first frequency domain selection network is used to indicate key data in the fourth linear result from a frequency domain dimension.

[0206] 3) Calling the fourth activation function to activate the first frequency domain selection result to obtain a fourth activation result.

[0207] Exemplarily, the fourth activation function is an optional or replaceable network structure. When the fourth activation function is deleted, the first frequency domain selection result can be used as the fourth activation result to perform subsequent steps.

[0208] 4) Input the fourth activation result into the first frequency domain SSM 506 to obtain a third prediction result.

[0209] 5) Flip the fourth linear result in the frequency domain dimension to obtain the third flip result. This enables the frequency domain network to perform bidirectional modeling in the frequency domain dimension from both the forward and reverse directions, extract the correlation of frequency domain data from both directions, and improve prediction accuracy.

[0210] 6) Input the third flip result into the second frequency domain selection network 505 to obtain a second frequency domain selection result. The second frequency domain selection network may include at least one convolution layer. The second frequency domain selection network is used to indicate key data in the third flip result from the frequency domain dimension.

[0211] 7) Calling the fifth activation function to activate the second frequency domain selection result to obtain a fifth activation result.

[0212] Exemplarily, the fifth activation function is an optional or replaceable network structure. When the fifth activation function is deleted, the second frequency domain selection result can be used as the fifth activation result to perform subsequent steps.

[0213] 8) Input the fifth activation result into the second frequency domain SSM 507 to obtain a fourth prediction result.

[0214] For example, Figure 6The branch network that flips the input data is an optional or replaceable network structure. When the flipped branch network is deleted, the above steps 5)-8) can be deleted, the subsequent steps 12)-14) can be deleted, and the second weighted sum in step 15) can be replaced by the third product.

[0215] 9) Input the frequency domain input data into the fifth linear network 502 to obtain a fifth linear result.

[0216] Exemplarily, the fifth linear network 502 is an optional or replaceable network structure. When the fifth linear network 502 is deleted, the frequency domain input data can be directly used as the fifth linear result to perform subsequent steps.

[0217] 10) Call the sixth activation function to activate the fifth linear result to obtain a sixth activation result.

[0218] Exemplarily, the sixth activation function is an optional or replaceable network structure. When the sixth activation function is deleted, the fifth linear result can be used as the sixth activation result to perform subsequent steps.

[0219] 11) Calculate the third prediction result and multiply the sixth activation result to obtain a third product.

[0220] 12) Calculate the fourth prediction result and multiply the sixth activation result to obtain a fourth product.

[0221] 13) Flip the fourth product in the frequency domain dimension to obtain a fourth flipped result.

[0222] 14) Calculate a second weighted sum of the third product and the fourth flip result. Exemplarily, the weight may be a preset value, or may be obtained by training during the model training process. For example, the weights of the third product and the fourth flip result may both be 0.5.

[0223] 15) Input the second weighted sum into the sixth linear network 503 to obtain frequency domain output data. Exemplarily, the frequency domain network does not change the dimension of the input data, that is, the dimension of the frequency domain output data is the same as the dimension of the frequency domain input data, for example, the frequency domain output data is also data of T*N*D dimension.

[0224] Exemplarily, the sixth linear network 503 is an optional or replaceable network structure. When the sixth linear network 503 is deleted, the second weighted sum can be directly used as frequency domain output data.

[0225] Step 2: Determine the frequency domain output data as the i-th output data.

[0226] In an optional embodiment, in order to improve model performance and training stability, normalization and skip connection structures are introduced before and after the frequency domain network 303.

[0227] like Fig.11 As shown, the input data needs to be normalized before being input into the frequency domain network 303; the output data of the frequency domain network 303 needs to be convolved by the second convolutional network 409 and concatenated with the input data of the frequency domain network, and the concatenated result is used as the final output data. The adjusted method is as follows.

[0228] Before step 1, the method further includes: obtaining frequency domain input data based on the i-th input data; for example, determining the normalized i-th input data as the frequency domain input data.

[0229] Step 1: Input frequency domain input data into the frequency domain selection network and the frequency domain SSM (ie, the frequency domain network 303) connected in sequence to obtain frequency domain output data.

[0230] Step 2 can be replaced by: splicing the frequency domain output data with the frequency domain input data to obtain the i-th output data. For example, the frequency domain output data is input into the second convolution network 409 to obtain a second convolution result; the second convolution result is spliced ​​with the frequency domain input data to obtain a second splicing result; and the second splicing result is determined as the i-th output data.

[0231] Step 223: When the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed.

[0232] Step 224: When the loop end condition is met, the i-th output data is input into the second multilayer perceptron 304 to obtain the target masking value corresponding to the target track.

[0233] Exemplarily, the second multilayer perceptron 304 is an optional or replaceable network structure, and when the second multilayer perceptron 304 is deleted, the i-th output data can be used as the target mask value.

[0234] In summary, the method provided in this embodiment has a slow reasoning speed and consumes a long memory because the amount of calculation of the self-attention mechanism in the Transformer model will increase quadratically with the increase of the context length. The cyclic model provided in this application combines the advantages of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), uses SSM to model the input data in the frequency domain dimension, and can selectively focus on or ignore certain parts of the input data, with an effect comparable to that of Transformer. At the same time, the cyclic model uses SSM to model the input data in the frequency domain, and the computational complexity of its modeling is linear with the duration, and the hardware-aware algorithm is used to make it more efficient when running on the GPU.

[0235] It should be noted that the sequence of the method steps provided in the embodiments of the present application can be appropriately adjusted, and the steps can be increased or decreased accordingly according to the circumstances. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated here.

[0236] Fig.12 : is a schematic diagram of the structure of an audio signal separation device provided by an exemplary embodiment of the present application. The device comprises:

[0237] A transform module 601 is used to perform Fourier transform on the audio signal to obtain a frequency domain signal;

[0238] A loop module 602 is used to call a loop network to obtain a target masking value corresponding to a target track based on the frequency domain signal; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the loop network includes a selection network and a state space model SSM connected in sequence, and the selection network is used to indicate key attention data in the input data to the SSM;

[0239] A separation module 603 is used to determine the product of the frequency domain signal and the target masking value as the target track frequency domain signal of the target track;

[0240] The inverse transform module 604 is configured to perform an inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0241] In an optional embodiment, the circulation module 602 is used to:

[0242] Inputting the frequency domain signal into a first multilayer perceptron to obtain first input data of a first cycle of the recurrent network;

[0243] For the i-th cycle in the cyclic network, the i-th input data is input into the cyclic network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key data in the i-th input data to the SSM;

[0244] If the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed;

[0245] When the loop end condition is met, the i-th output data is input into a second multilayer perceptron to obtain the target masking value corresponding to the target track.

[0246] In an optional embodiment, the cyclic network includes a time domain network and a frequency domain network connected in sequence; the time domain network includes a time domain selection network and a time domain SSM, and the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the cyclic module 602 is used to:

[0247] Input the time domain input data into the sequentially connected time domain selection network and the time domain SSM to obtain time domain output data; the time domain selection network is used to indicate the time domain key attention data for the time domain SSM in the time domain dimension; the time domain input data is obtained based on the i-th input data; input the frequency domain input data into the sequentially connected frequency domain selection network and the frequency domain SSM to obtain frequency domain output data; the frequency domain selection network is used to indicate the frequency domain key attention data for the frequency domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the time domain output data;

[0248] The frequency domain output data is determined as the i-th output data.

[0249] In an optional embodiment, the cyclic network includes a time domain network; the time domain network includes a time domain selection network and a time domain SSM; the cyclic module 602 is used to:

[0250] Inputting the time domain input data into the sequentially connected time domain selection network and the time domain SSM to obtain time domain output data; the time domain selection network is used to indicate the time domain key attention data for the time domain SSM in the time domain dimension; the time domain input data is obtained based on the i-th input data;

[0251] The time domain output data is determined as the i-th output data.

[0252] In an optional embodiment, the cyclic network includes a frequency domain network; the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the cyclic module 602 is used to:

[0253] Inputting frequency domain input data into the frequency domain selection network and the frequency domain SSM connected in sequence to obtain frequency domain output data; the frequency domain selection network is used to indicate frequency domain key attention data for the frequency domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data;

[0254] The frequency domain output data is determined as the i-th output data.

[0255] In an optional embodiment, the time domain network includes: a first linear network, a second linear network, a third linear network, a first time domain selection network, a second time domain selection network, a first time domain SSM and a second time domain SSM; the circulation module 602 is used to:

[0256] Inputting the time domain input data into a first linear network to obtain a first linear result;

[0257] Inputting the first linear result into a first time domain selection network to obtain a first time domain selection result;

[0258] Calling a first activation function to activate the first time domain selection result to obtain a first activation result;

[0259] Inputting the first activation result into a first time-domain SSM to obtain a first prediction result;

[0260] Flipping the first linear result in the time domain dimension to obtain a first flipped result;

[0261] Inputting the first flip result into a second time domain selection network to obtain a second time domain selection result;

[0262] Calling a second activation function to activate the second time domain selection result to obtain a second activation result;

[0263] Inputting the second activation result into a second time-domain SSM to obtain a second prediction result;

[0264] Inputting the time domain input data into a second linear network to obtain a second linear result;

[0265] Calling a third activation function to activate the second linear result to obtain a third activation result;

[0266] Calculate the first prediction result and multiply the third activation result to obtain a first product;

[0267] Calculate the second prediction result and multiply the third activation result to obtain a second product;

[0268] Flipping the second product in the time domain dimension to obtain a second flipped result;

[0269] Calculating a first weighted sum of the first product and the second flip result;

[0270] The first weighted sum is input into a third linear network to obtain the time domain output data.

[0271] In an optional embodiment, the frequency domain network includes: a fourth linear network, a fifth linear network, a sixth linear network, a first frequency domain selection network, a second frequency domain selection network, a first frequency domain SSM, and a second frequency domain SSM; the circulation module 602 is used to:

[0272] Inputting the frequency domain input data into a fourth linear network to obtain a fourth linear result;

[0273] Inputting the fourth linear result into the first frequency domain selection network to obtain a first frequency domain selection result;

[0274] Calling a fourth activation function to activate the first frequency domain selection result to obtain a fourth activation result;

[0275] Inputting the fourth activation result into the first frequency domain SSM to obtain a third prediction result;

[0276] Flipping the fourth linear result in the frequency domain dimension to obtain a third flipped result;

[0277] Inputting the third flip result into a second frequency domain selection network to obtain a second frequency domain selection result;

[0278] Calling a fifth activation function to activate the second frequency domain selection result to obtain a fifth activation result;

[0279] Inputting the fifth activation result into the second frequency domain SSM to obtain a fourth prediction result;

[0280] Inputting the frequency domain input data into a fifth linear network to obtain a fifth linear result;

[0281] Calling a sixth activation function to activate the fifth linear result to obtain a sixth activation result;

[0282] Calculate the third prediction result and multiply the sixth activation result to obtain a third product;

[0283] Calculate the fourth prediction result and multiply the sixth activation result to obtain a fourth product;

[0284] Flipping the fourth product in the frequency domain dimension to obtain a fourth flipped result;

[0285] Calculating a second weighted sum of the third product and the fourth flip result;

[0286] The second weighted sum is input into a sixth linear network to obtain the frequency domain output data.

[0287] In an optional embodiment, the circulation module 602 is used to:

[0288] Obtaining the time domain input data based on the i-th input data;

[0289] splicing the i-th input data with the time domain output data to obtain the frequency domain input data;

[0290] The step of determining the frequency domain output data as the i-th output data comprises:

[0291] The frequency domain output data is concatenated with the frequency domain input data to obtain the i-th output data.

[0292] In an optional embodiment, the circulation module 602 is used to:

[0293] Determine the normalized i-th input data as the time domain input data;

[0294] The step of splicing the i-th input data with the time domain output data to obtain the frequency domain input data includes:

[0295] Inputting the time domain output data into a first convolutional network to obtain a first convolution result; splicing the first convolution result with the i-th input data to obtain a first splicing result; and determining the normalized first splicing result as the frequency domain input data;

[0296] The step of splicing the frequency domain output data with the frequency domain input data to obtain the i-th output data includes:

[0297] Input the frequency domain output data into a second convolutional network to obtain a second convolution result; splice the second convolution result with the first splicing result to obtain a second splicing result; and determine the second splicing result as the i-th output data.

[0298] It should be noted that the audio signal separation device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the audio signal separation device provided in the above embodiment and the audio signal separation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0299] The embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the audio signal separation method provided by the above-mentioned method embodiments. The computer device can be implemented as a terminal device.

[0300] For example, Fig.13 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0301] Typically, the computer device 1700 includes a processor 1701 and a memory 1702 .

[0302] The processor 1701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0303] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store at least one instruction, which is used to be executed by the processor 1701 to implement the audio signal separation method provided in the method embodiment of the present application.

[0304] In some embodiments, the computer device 1700 may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, the memory 1702 and the peripheral device interface 1703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707 and a power supply 1708.

[0305] The peripheral device interface 1703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1701 and the memory 1702. In some embodiments, the processor 1701, the memory 1702, and the peripheral device interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1701, the memory 1702, and the peripheral device interface 1703 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0306] The radio frequency circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1704 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1704 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1704 can communicate with other computer devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1704 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0307] The display screen 1705 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1705 is a touch display screen, the display screen 1705 also has the ability to collect touch signals on the surface or above the surface of the display screen 1705. The touch signal can be input to the processor 1701 as a control signal for processing. At this time, the display screen 1705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1705 can be one, and the front panel of the computer device 1700 is set; in other embodiments, the display screen 1705 can be at least two, which are respectively set on different surfaces of the computer device 1700 or are folded; in some other embodiments, the display screen 1705 can be a flexible display screen, which is set on the curved surface or folded surface of the computer device 1700. Even, the display screen 1705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0308] The camera assembly 1706 is used to capture images or videos. Optionally, the camera assembly 1706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the computer device 1700, and the rear camera is arranged on the back of the computer device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0309] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1701 for processing, or input them into the radio frequency circuit 1704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the computer device 1700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0310] The power supply 1708 is used to power various components in the computer device 1700. The power supply 1708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1708 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0311] In some embodiments, the computer device 1700 further includes one or more sensors 1709 , including but not limited to: an acceleration sensor 1710 , a gyroscope sensor 1711 , a pressure sensor 1712 , an optical sensor 1713 , and a proximity sensor 1714 .

[0312] The acceleration sensor 1710 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computer device 1700. For example, the acceleration sensor 1710 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1701 can control the touch display screen 1705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1710. The acceleration sensor 1710 can also be used to collect game or user motion data.

[0313] The gyro sensor 1711 can detect the body direction and rotation angle of the computer device 1700, and the gyro sensor 1711 can cooperate with the acceleration sensor 1710 to collect the user's 3D actions on the computer device 1700. The processor 1701 can implement the following functions based on the data collected by the gyro sensor 1711: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0314] The pressure sensor 1712 can be set on the side frame of the computer device 1700 and / or the lower layer of the touch display screen 1705. When the pressure sensor 1712 is set on the side frame of the computer device 1700, it can detect the user's grip signal of the computer device 1700, and the processor 1701 performs left and right hand recognition or shortcut operations according to the grip signal collected by the pressure sensor 1712. When the pressure sensor 1712 is set on the lower layer of the touch display screen 1705, the processor 1701 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0315] The optical sensor 1713 is used to collect the ambient light intensity. In one embodiment, the processor 1701 can control the display brightness of the touch display screen 1705 according to the ambient light intensity collected by the optical sensor 1713. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1705 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1705 is reduced. In another embodiment, the processor 1701 can also dynamically adjust the shooting parameters of the camera assembly 1706 according to the ambient light intensity collected by the optical sensor 1713.

[0316] The proximity sensor 1714, also called a distance sensor, is usually arranged on the front panel of the computer device 1700. The proximity sensor 1714 is used to collect the distance between the user and the front of the computer device 1700. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually decreasing, the processor 1701 controls the touch display screen 1705 to switch from the screen-on state to the screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually increasing, the processor 1701 controls the touch display screen 1705 to switch from the screen-off state to the screen-on state.

[0317] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the computer device 1700, and the computer device 1700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0318] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor of a computer device, the audio signal separation method provided by the above-mentioned method embodiments is implemented.

[0319] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the audio signal separation method provided by each of the above method embodiments.

[0320] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0321] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for separating an audio signal, characterized in that: The method comprises: Performing Fourier transform on the audio signal to obtain a frequency domain signal; Calling a recurrent network to process the frequency domain signal to obtain a target masking value corresponding to a target track; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the recurrent network includes a selection network and a state space model connected in sequence, and the selection network is used to indicate key attention data in the input data to the state space model; Determine the product of the frequency domain signal and the target masking value as the target track frequency domain signal of the target track; Perform inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

2. The method according to claim 1, characterized in that: The calling recurrent network obtains a target masking value corresponding to the target track based on the frequency domain signal, including: Inputting the frequency domain signal into a first multilayer perceptron to obtain first input data of a first cycle of the recurrent network; For the i-th cycle in the cyclic network, the i-th input data is input into the cyclic network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key data in the i-th input data for the state space model; If the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed; When the loop end condition is met, the i-th output data is input into a second multilayer perceptron to obtain the target masking value corresponding to the target track.

3. The method according to claim 2, characterized in that The cyclic network includes a time domain network and a frequency domain network connected in sequence; the time domain network includes a time domain selection network and a time domain state space model, and the frequency domain network includes a frequency domain selection network and a frequency domain state space model; The step of inputting the i-th input data into the cyclic network to obtain the i-th output data comprises: Inputting the time domain input data into the sequentially connected time domain selection network and the time domain state space model to obtain time domain output data; the time domain selection network is used to indicate the time domain key attention data for the time domain state space model in the time domain dimension; the time domain input data is obtained based on the i-th input data; Inputting frequency domain input data into the frequency domain selection network and the frequency domain state space model connected in sequence to obtain frequency domain output data; the frequency domain selection network is used to indicate frequency domain key attention data for the frequency domain state space model in the frequency domain dimension; the frequency domain input data is obtained based on the time domain output data; The frequency domain output data is determined as the i-th output data.

4. The method according to claim 2, characterized in that: The cyclic network includes a time domain network; the time domain network includes a time domain selection network and a time domain state space model; The step of inputting the i-th input data into the cyclic network to obtain the i-th output data comprises: Inputting the time domain input data into the sequentially connected time domain selection network and the time domain state space model to obtain time domain output data; the time domain selection network is used to indicate the time domain key attention data for the time domain state space model in the time domain dimension; the time domain input data is obtained based on the i-th input data; The time domain output data is determined as the i-th output data.

5. The method according to claim 2, characterized in that: The cyclic network includes a frequency domain network; the frequency domain network includes a frequency domain selection network and a frequency domain state space model; The step of inputting the i-th input data into the cyclic network to obtain the i-th output data comprises: Inputting frequency domain input data into the frequency domain selection network and the frequency domain state space model connected in sequence to obtain frequency domain output data; the frequency domain selection network is used to indicate frequency domain key attention data for the frequency domain state space model in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data; The frequency domain output data is determined as the i-th output data.

6. The method according to claim 3 or 4, characterized in that: The time domain network includes: a first linear network, a second linear network, a third linear network, a first time domain selection network, a second time domain selection network, a first time domain state space model and a second time domain state space model; The step of inputting the time domain input data into the time domain selection network and the time domain state space model connected in sequence to obtain the time domain output data comprises: Inputting the time domain input data into a first linear network to obtain a first linear result; Inputting the first linear result into a first time domain selection network to obtain a first time domain selection result; Calling a first activation function to activate the first time domain selection result to obtain a first activation result; Inputting the first activation result into a first time-domain state-space model to obtain a first prediction result; Flipping the first linear result in the time domain dimension to obtain a first flipped result; Inputting the first flip result into a second time domain selection network to obtain a second time domain selection result; Calling a second activation function to activate the second time domain selection result to obtain a second activation result; Inputting the second activation result into a second time-domain state-space model to obtain a second prediction result; Inputting the time domain input data into a second linear network to obtain a second linear result; Calling a third activation function to activate the second linear result to obtain a third activation result; Calculate the first prediction result and multiply the third activation result to obtain a first product; and calculate the second prediction result and multiply the third activation result to obtain a second product; Flipping the second product in the time domain dimension to obtain a second flipped result; Calculating a first weighted sum of the first product and the second flip result; The first weighted sum is input into a third linear network to obtain the time domain output data.

7. The method according to claim 3 or 5, characterized in that: The frequency domain network includes: a fourth linear network, a fifth linear network, a sixth linear network, a first frequency domain selection network, a second frequency domain selection network, a first frequency domain state space model and a second frequency domain state space model; The step of inputting the frequency domain input data into the frequency domain selection network and the frequency domain state space model connected in sequence to obtain the frequency domain output data comprises: Inputting the frequency domain input data into a fourth linear network to obtain a fourth linear result; Inputting the fourth linear result into the first frequency domain selection network to obtain a first frequency domain selection result; Calling a fourth activation function to activate the first frequency domain selection result to obtain a fourth activation result; Inputting the fourth activation result into the first frequency domain state space model to obtain a third prediction result; Flipping the fourth linear result in the frequency domain dimension to obtain a third flipped result; Inputting the third flip result into a second frequency domain selection network to obtain a second frequency domain selection result; Calling a fifth activation function to activate the second frequency domain selection result to obtain a fifth activation result; Inputting the fifth activation result into the second frequency domain state space model to obtain a fourth prediction result; Inputting the frequency domain input data into a fifth linear network to obtain a fifth linear result; Calling a sixth activation function to activate the fifth linear result to obtain a sixth activation result; Calculating the third prediction result and multiplying the sixth activation result to obtain a third product; and calculating the fourth prediction result and multiplying the sixth activation result to obtain a fourth product; Flipping the fourth product in the frequency domain dimension to obtain a fourth flipped result; Calculating a second weighted sum of the third product and the fourth flip result; The second weighted sum is input into a sixth linear network to obtain the frequency domain output data.

8. The method according to claim 3, characterized in that The method further comprises: Obtaining the time domain input data based on the i-th input data; splicing the i-th input data with the time domain output data to obtain the frequency domain input data; The step of determining the frequency domain output data as the i-th output data comprises: The frequency domain output data is concatenated with the frequency domain input data to obtain the i-th output data.

9. The method according to claim 8, characterized in that The obtaining the time domain input data based on the i-th input data includes: Determine the normalized i-th input data as the time domain input data; The step of splicing the i-th input data with the time domain output data to obtain the frequency domain input data includes: Inputting the time domain output data into a first convolutional network to obtain a first convolution result; splicing the first convolution result with the i-th input data to obtain a first splicing result; and determining the normalized first splicing result as the frequency domain input data; The step of splicing the frequency domain output data with the frequency domain input data to obtain the i-th output data includes: Input the frequency domain output data into a second convolutional network to obtain a second convolution result; splice the second convolution result with the first splicing result to obtain a second splicing result; and determine the second splicing result as the i-th output data.

10. An audio signal separation device, characterized in that: The device comprises: A transform module, used for performing Fourier transform on the audio signal to obtain a frequency domain signal; A loop module, used to call a loop network to obtain a target masking value corresponding to a target track based on the frequency domain signal; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the loop network includes a selection network and a state space model state space model connected in sequence, and the selection network is used to indicate key attention data in the input data to the state space model; A separation module, configured to determine a target track frequency domain signal of the target track by multiplying the frequency domain signal by the target masking value; The inverse transform module is used to perform inverse Fourier transform on the target track frequency domain signal to obtain the target track audio signal of the target track.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the audio signal separation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one program, and the at least one program is loaded and executed by a processor to implement the audio signal separation method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the audio signal separation method according to any one of claims 1 to 9.

Images