Audio signal separation method, apparatus, device, and storage medium

Abstract

The application discloses an audio signal separation method and device, equipment and a storage medium, and belongs to the field of audio signal processing. The method comprises the following steps: performing Fourier transform on an audio signal to obtain a frequency domain signal; calling a recurrent network to obtain a target mask value corresponding to a target track based on the frequency domain signal; the target mask value is used to indicate a frequency point of the audio signal of the target track; the recurrent network comprises a selection network and a state space model connected in sequence, and the selection network is used to indicate key attention data in input data for the state space model; the product of the frequency domain signal and the target mask value is determined as a target track frequency domain signal of the target track; and inverse Fourier transform is performed on the target track frequency domain signal to obtain a target track audio signal of the target track. The method can improve the separation efficiency of the audio signal.

Description

TECHNICAL FIELD

[0001] The present application relates to audio signal processing, in particular to an audio signal separation method, device, equipment and storage medium. BACKGROUND

[0002] Audio signal separation is used to separate an audio signal into multiple audio track signals. For example, separating the audio signal of a song into human voice audio and accompaniment audio, or separating the accompaniment audio into different instrument track audio such as drum audio, bass audio, piano audio, etc.

[0003] In related technologies, when performing audio signal separation, a Transformer model is used to model the time domain and frequency domain of the audio signal to extract the correlation of the audio signal in the time domain and the frequency domain, and to predict the masking value of the target audio track. The track signal of the target audio track is calculated based on the predicted masking value.

[0004] However, when using the Transformer model for inference, the calculation amount of the self-attention mechanism of the Transformer model increases in a square level with the increase of the context length. When used to calculate the input data of the audio signal which is a long sequence, the calculation amount is huge, and the efficiency of audio signal separation is low. SUMMARY

[0005] The present application provides an audio signal separation method, device, equipment and storage medium, which can improve the separation efficiency of the audio signal. The technical solution is as follows:

[0006] According to an aspect of the present application, an audio signal separation method is provided, the method comprising:

[0007] performing Fourier transform on the audio signal to obtain a frequency domain signal;

[0008] calling a recurrent network to obtain a target masking value corresponding to a target track based on the frequency domain signal; the target masking value is used to indicate a frequency point where the audio signal of the target track is located; the recurrent network comprises a selection network and a state space model connected in sequence, and the selection network is used to indicate key attention data in the input data for the state space model;

[0009] determining the product of the frequency domain signal and the target masking value as a target track frequency domain signal of the target track;

[0010] performing inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0011] According to another aspect of the present application, an audio signal separation device is provided, the device comprising:

[0012] a transform module configured to perform Fourier transform on the audio signal to obtain a frequency domain signal;

[0013] a recurrent module configured to call a recurrent network to obtain a target mask value corresponding to a target track based on the frequency domain signal; the target mask value is used to indicate a frequency point where the audio signal of the target track is located; the recurrent network comprises a selection network and a state space model connected in sequence, and the selection network is used to indicate key data in input data for the state space model;

[0014] a separation module configured to determine a target track frequency domain signal of the target track by multiplying the frequency domain signal by the target mask value;

[0015] an inverse transform module configured to perform inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0016] According to another aspect of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the audio signal separation method according to the above aspect.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, the readable storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the audio signal separation method according to the above aspect.

[0018] According to another aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the audio signal separation method provided in various optional implementation manners of the above aspect.

[0019] The technical scheme provided by the present application has at least the following beneficial effects:

[0020] The selection network and the state space model are used to model the frequency domain signal. Since the state space model uses linear operation, the output is predicted based on the hidden state stored in the model according to the state transition equation and the output equation. In the scenario of processing long sequence input data such as audio signals or frequency domain signals, the operation amount increases linearly, which greatly reduces the model operation amount compared with the square growth of the Transformer model. Moreover, the selection network is added in front of the state space model, the key data in the input data is selected through the selection network, the irrelevant data in the input data is filtered, and the data content of the input data is refined, so as to further reduce the operation amount of the state space model and improve the prediction accuracy and operation efficiency of the state space model. By using the audio signal separation method provided in the application, the track audio signal of the target track can be accurately and efficiently separated from the audio signal, and the deployment and application of the audio signal separation function are facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a structural schematic diagram of a computer system provided by an example embodiment of the present application;

[0023] Figure 2 is a flowchart of an audio signal separation method provided by an example embodiment of the present application;

[0024] Figure 3 is a schematic diagram of an audio separation model provided by an example embodiment of the present application;

[0025] Figure 4 is a flowchart of an audio signal separation method provided by an example embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a time domain network provided by an example embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a frequency domain network provided by an example embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a recurrent network provided by an example embodiment of the present application;

[0029] Figure 8 is a schematic diagram of an audio separation model provided by an example embodiment of the present application;

[0030] Figure 9 is a schematic diagram of a recurrent network provided by an example embodiment of the present application;

[0031] Figure 10 is a schematic diagram of an audio separation model provided by an example embodiment of the present application;

[0032] Figure 11 is a schematic diagram of a recurrent network provided by an example embodiment of the present application;

[0033] Figure 12 is a structural schematic diagram of an audio signal separation apparatus provided by an example embodiment of the present application;

[0034] Figure 13 is a structural schematic diagram of a computer device provided by an example embodiment of the present application.

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification. DETAILED DESCRIPTION

[0036] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0037] Figure 1 is a schematic diagram of a computer system provided by an example embodiment of the present application, which can include a terminal device 101 and a server 103.

[0038] For example, the audio signal separation method shown in the embodiments of the present application can be applied in a terminal device, and the terminal device 101 runs an application program 102 with audio signal separation requirements. The terminal device can include a mobile phone, a tablet computer, a notebook computer, a laptop computer, a desktop computer, an all-in-one computer, an Internet of Things device, a smart robot workstation, a television, a set-top box, smart glasses, a smart watch, a digital camera, an MP4 player, an MP5 player, a learning machine, a point-reading machine, an electronic paper, an electronic dictionary, a vehicle-mounted device, a virtual reality (VR) player or an augmented reality (AR) player, etc.

[0039] Exemplarily, the audio signal separation method provided in the present application can be executed by a client in a terminal device. The client is a client of an application program with audio signal separation requirements. For example, the application program can include at least one of the following: an audio player, an audio application program, a video application program, a social application program, a shopping application program, a live broadcast application program, a vehicle-mounted audio playback application program, an information application program, a browser, a game application program, and a recording application program.

[0040] For example, a karaoke function is provided in an audio player, and the karaoke function requires separation of human voice audio and accompaniment audio from original audio. The audio player can use the audio signal separation method provided in the present application to separate the accompaniment audio in the original audio for the karaoke function to play.

[0041] The terminal device 101 includes a first memory and a first processor. The first memory stores an audio signal separation program; the audio signal separation program is called and executed by the first processor to implement the audio signal separation method provided in the present application. The first memory can include, but is not limited to, the following: a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM).

[0042] The first processor can be composed of one or more integrated circuit chips. Alternatively, the first processor can be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Alternatively, the first processor can implement the audio signal separation method provided in the present application by running a program or code.

[0043] In an optional embodiment, the terminal device 101 and the server 103 can be connected to each other through a wired or wireless network.

[0044] Exemplarily, the audio signal separation method provided in the present application can also be executed by a server. The server 103 is configured to provide background service for the target use case system of the terminal device 101. Optionally, the server 103 undertakes the main computing work, and the terminal device 101 undertakes the secondary computing work; or the server 103 undertakes the secondary computing work, and the terminal device 101 undertakes the main computing work; or the server 103 and the terminal device 101 adopt a distributed computing architecture to perform collaborative computing.

[0045] The server 103 can be a single server, a server cluster composed of multiple servers, or a cloud computing service center.

[0046] Optionally, the server 103 comprises a second memory and a second processor. The second memory stores an audio signal separation program; the audio signal separation program is called by the second processor to implement the audio signal separation method provided in the present application. Optionally, the second memory can include but is not limited to the following: RAM, ROM, PROM, EPROM, EEPROM. Optionally, the second processor can be a general-purpose processor, such as CPU or NP.

[0047] Figure 2 FIG. 1 is a flowchart of an audio signal separation method provided in an exemplary embodiment of the present application. The method can be used in a terminal device or a server as shown in FIG. 1. The method comprises the following steps. Figure 1

[0048] Step 210: performing Fourier transform on the audio signal to obtain a frequency domain signal.

[0049] Exemplarily, the short-time Fourier transform (STFT) is performed on the audio signal to obtain the frequency domain signal.

[0050] The audio signal can be separated into at least one track signal of an audio track. The audio track is a different channel of the audio signal, which is used to separate different sound elements, such as musical instruments, vocals, environmental sounds, etc., from the audio signal, so as to facilitate editing, mixing and processing of the track signals of different tracks respectively. For example, the audio of a song can be separated into track signals of a vocal track and an accompaniment track, and the accompaniment audio can be separated into track signals of multiple audio tracks corresponding to different musical instruments respectively. The at least one track signal of the audio track obtained by separating the audio signal can be combined to restore the audio signal.

[0051] Exemplarily, the frequency domain signal comprises a frequency domain signal obtained by performing Fourier transform on at least one frame of time domain signal of the audio signal. For example, the frequency domain signal can be represented as a three-dimensional signal of T*F*C, where T is time / frame number, F is frequency point, and C is amplitude.​

[0052] Step 220: calling a recurrent network to obtain a target mask value corresponding to the target track based on the frequency domain signal; the target mask value is used to indicate a frequency point at which the track signal of the target track is located in the frequency domain signal; the recurrent network comprises a selection network and an SSM connected in sequence, and the selection network is used to indicate key data in the input data for the SSM.

[0053] The target mask value is used to indicate a frequency point at which the track signal of the target track is located in the frequency domain signal, and a signal strength at the frequency point. The target mask value has the same dimension as the frequency domain signal, that is, the dimension of the target mask value is also T*F*C, where C represents a mask value corresponding to the T, F position, and the mask value has a value range of 0 to 1. For example, a point (1, 1, 0) in the target mask value indicates that the mask value of the 1st frequency point position in the 1st frame is 0, that is, the 1st frequency point does not exist the track signal of the target track.

[0054] The recurrent network is used to predict the target mask value corresponding to the target track. After obtaining the target mask value, the target mask value is multiplied by the frequency domain signal, and the target track frequency domain signal of the target track is obtained. For example, the embodiment takes the target track as an example for description. If it is required to separate the track signals of multiple tracks from the audio signal, a recurrent network can be trained for each track to predict the mask value corresponding to the track.

[0055] The recurrent network is a neural network model with a recurrent structure. The recurrent network is used to model at least one dimension of the frequency domain or the time domain of the input frequency domain signal, and predict the mask value corresponding to the target track based on the feature information of the frequency domain signal in the time domain or the frequency domain.

[0056] It should be noted that the recurrent network in the embodiment adopts an SSM (Structured Space Model, state space model). The SSM is a model used to describe the state representation of a sequence at each time step and predict the next state according to the input. The SSM converts the input sequence through the hidden state representation and finally generates the output sequence. The core idea is to capture the key information in the sequence through the hidden state, so as to realize efficient sequence modeling. The SSM defines a state transition equation and an output equation. The state transition equation describes how the hidden state changes over time, and the output equation describes how to obtain the output data from the hidden state.

[0057] The recurrent network implemented by the Transformer model in the related art needs to calculate the attention score between each element and other elements when processing a long sequence, and the calculation amount is huge. The SSM of the embodiment adopts linear operation, and the calculation amount of the model increases linearly with the increase of the sequence length, which can significantly increase the reasoning speed of the recurrent network.

[0058] The recurrent network in the embodiment adds a selection network before the SSM, filters out irrelevant information in the input data, retains necessary information, inputs the selected input data into the SSM, enables the SSM to more effectively manage information, and improves the accuracy and operation efficiency of the model.

[0059] For example, the network structure of the selection network can be arbitrary. For example, the selection network can be implemented by using a convolutional network, or the selection network can be implemented by using an encoder and a decoder.

[0060] In an alternative embodiment, the recurrent network comprises a time domain network and a frequency domain network connected in sequence; the time domain network comprises a time domain selection network and a time domain SSM, and the frequency domain network comprises a frequency domain selection network and a frequency domain SSM; the time domain selection network is configured to indicate time domain focus data to the time domain SSM in a time domain dimension; and the frequency domain selection network is configured to indicate frequency domain focus data to the frequency domain SSM in a frequency domain dimension.

[0061] The time domain network is configured to model the frequency domain signal in the time domain dimension. For example, the frequency domain signal is sequentially input into the time domain network according to a frame sequence in the time domain dimension (T), and the time domain SSM performs state prediction according to the input frame sequence.

[0062] Correspondingly, the frequency domain network is configured to model the frequency domain signal in the frequency domain dimension. For example, the frequency domain signal is sequentially input into the frequency domain network according to a sub-band sequence / frequency point sequence in the frequency domain dimension (F), and the frequency domain SSM performs state prediction according to the input sub-band sequence / frequency point sequence. At least one frequency point can be divided into a sub-band according to a fixed frequency domain interval. For example, the input of the frequency domain network is the output data of the time domain network, and the output data of the time domain network is the same as the dimension of the frequency domain signal.

[0063] In another alternative embodiment, the recurrent network comprises a time domain network; the time domain network comprises a time domain selection network and a time domain SSM; and the time domain selection network is configured to indicate time domain focus data to the time domain SSM in a time domain dimension.

[0064] For example, taking the i-th cycle in the recurrent network as an example, time domain input data is input into the time domain selection network and the time domain SSM connected in sequence to obtain time domain output data; the time domain selection network is configured to indicate time domain focus data to the time domain SSM in a time domain dimension; the time domain input data is obtained based on the i-th input data; and the time domain output data is determined as the i-th output data.

[0065] In another alternative embodiment, the recurrent network comprises a frequency domain network; the frequency domain network comprises a frequency domain selection network and a frequency domain SSM; and the frequency domain selection network is configured to indicate frequency domain focus data to the frequency domain SSM in a frequency domain dimension.

[0066] Exemplarily, taking the i-th cycle in the cycle network as an example, the frequency domain input data is input into the frequency domain selection network and the frequency domain SSM connected in sequence to obtain frequency domain output data; the frequency domain selection network is used to indicate the frequency domain focus data for the frequency domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data; and the frequency domain output data is determined as the i-th output data.

[0067] Hereinafter, the network structure of the cycle network will be described in detail taking the cycle network including the time domain network and the frequency domain network as an example. When the cycle network only includes the time domain network or the frequency domain network, the network structure thereof can also refer to the time domain network structure or the frequency domain network structure given below.

[0068] Step 230: determining the product of the frequency domain signal and the target masking value as the target track frequency domain signal of the target track.

[0069] Exemplarily, the target masking value is used to indicate the frequency point where the audio signal of the target track is located, and the signal strength of the target track at the frequency point. The frequency domain signal includes at least one frame of frequency domain signal, and each frame of frequency domain signal includes an amplitude value at at least one frequency point; and the target masking value includes a set of masking values corresponding to at least one frame of frequency domain signal respectively, and each set of masking values includes a masking value (a value from 0 to 1) corresponding to at least one frequency point.

[0070] The amplitude of the frequency domain signal is multiplied by the masking value at the corresponding position (the same frequency point of the same frame) in the target masking value to obtain the amplitude of the target track frequency domain signal at the position. For example, the amplitude of the frequency domain signal at the first frequency point of the first frame is 10, and the masking value of the target masking value at the first frequency point of the first frame is 0.1, so the amplitude of the target track frequency domain signal at the first frequency point of the first frame is 10*0.1=1. In this way, the corresponding positions of the frequency domain signal and the target masking value are multiplied respectively to obtain the target track frequency domain signal of the target track.

[0071] Step 240: performing inverse Fourier transform on the target track frequency domain signal to obtain the target track audio signal of the target track.

[0072] Exemplarily, performing short-time inverse Fourier transform on the target track frequency domain signal to obtain the time domain signal corresponding to the target track: the target track audio signal.

[0073] This embodiment only takes separating the target track audio signal of the target track from the audio signal as an example for description, and other track audio signals of other tracks can also be separated from the audio signal in this way.

[0074] In summary, the method provided in the embodiment adopts a selection network and a state space model to model the frequency domain signal. Since the state space model adopts linear operation, the output is predicted based on the state transition equation and the output equation according to the hidden state stored in the model. In the scenario of processing long sequence input data such as audio signals or frequency domain signals, the operation amount increases linearly, which greatly reduces the model operation amount compared with the quadratic growth of the Transformer model. Moreover, the selection network is added in front of the state space model, the key data in the input data is selected through the selection network, the irrelevant data in the input data is filtered, the data content of the input data is refined, the operation amount of the state space model is further reduced, and the prediction accuracy and operation efficiency of the state space model are improved. The audio signal separation method provided in the embodiment can accurately and efficiently separate the track audio signal of the target track from the audio signal, facilitating the deployment and application of the audio signal separation function.

[0075] In an alternative embodiment, the recurrent network includes a time domain network and a frequency domain network. For example, as shown in Figure 3 The audio signal separation method of the present application is performed using an audio separation model. The audio separation model includes a recurrent network, and the audio separation model includes a first multilayer perceptron (MLP) 301, a time domain network 302, a frequency domain network 303, and a second multilayer perceptron 304 connected in sequence.

[0076] Based on the network structure, the audio signal separation method can be briefly summarized as follows: (1) converting the audio signal to a frequency domain signal by short-time Fourier transform, dividing the frequency domain signal into a sequence in the time dimension according to frame, and dividing the frequency domain signal into a sequence in the frequency domain dimension according to frequency point; (2) performing sequence modeling on the sequences in the time dimension and the frequency domain dimension respectively to predict the masking value of the audio track; (3) multiplying the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and performing inverse short-time Fourier transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0077] Figure 4 is a flowchart of an audio signal separation method provided by an exemplary embodiment of the present application. The method can be used in a terminal device or a server as shown in Figure 1 According to the network structure of the recurrent network as shown in Figure 3 According to the embodiment as shown in Figure 2 Step 220 includes steps 221 to 224.

[0078] Step 210: performing Fourier transform on the audio signal 310 to obtain the frequency domain signal 320.

[0079] For example, the audio signal is framed according to a preset time window length and a time window offset, to obtain at least one frame of audio signal, the time length of the at least one frame of audio signal is equal to the time window length, and the distance between the starting position of the next frame time window and the starting position of the current frame time window is equal to the time window offset. The at least one frame of audio signal is respectively subjected to Fourier transform to obtain at least one frame of frequency domain signal, and the at least one frame of frequency domain signal constitutes the frequency domain signal 320. That is, the short-time Fourier transform is performed on the audio signal 310 to obtain the frequency domain signal 320.

[0080] Step 221: input the frequency domain signal 320 into the first multi-layer perceptron 301 to obtain first input data of the first cycle of the recurrent network.

[0081] For example, the first multi-layer perceptron includes an input layer, at least one hidden layer and an output layer, each layer of the network includes a plurality of neurons, and the neurons are connected through weights. The first multi-layer perceptron is used for sub-band division of the frequency domain signal 320, at least one frequency point is divided into a sub-band, and first input data after sub-band division is obtained. The dimension of the first input data is T*N*D, T is time / frame, N is a sub-band, and D is amplitude. The first input data is a sub-band division result obtained after the sub-band division of the frequency domain signal.

[0082] For example, the first multi-layer perceptron 301 is an optional or replaceable network structure, and when the first multi-layer perceptron 301 is deleted, the frequency domain signal 320 can be used as the first input data to perform subsequent steps.

[0083] Step 222: for the i-th cycle in the recurrent network, input the i-th input data into the recurrent network to obtain the i-th output data, i is a positive integer; and the network is selected to indicate the key attention data in the i-th input data for the SSM.

[0084] That is, the output data of the current cycle network is used as the input data of the next cycle, until the cycle ends.

[0085] For example, the recurrent network includes a time domain network 302 and a frequency domain network 303 connected in sequence; the time domain network 302 includes a time domain selection network and a time domain SSM, and the frequency domain network 303 includes a frequency domain selection network and a frequency domain SSM. Then, step 222 can include the following steps:

[0086] Step 1: input the time domain input data into the time domain selection network and the time domain SSM connected in sequence to obtain time domain output data; the time domain selection network is used to indicate the time domain key attention data for the time domain SSM in the time domain dimension; and the time domain input data is obtained based on the i-th input data;

[0087] In an optional embodiment, as Figure 5As shown, the time-domain network 302 comprises: a first linear network 401, a second linear network 402, a third linear network 403, a first time-domain selection network 404, a second time-domain selection network 405, a first time-domain SSM 406, and a second time-domain SSM 407.

[0088] According to Figure 5 According to the network structure of the time-domain network 302 as shown, step 1 can be specifically implemented as:

[0089] 1) input the time-domain input data into the first linear network 401 to obtain a first linear result. The time-domain input data can be the i-th input data; or the time-domain input data is obtained based on the i-th input data.

[0090] For example, the first linear network 401 is an optional or replaceable network structure. When the first linear network 401 is deleted, the time-domain input data can be directly used as the first linear result to perform the subsequent steps.

[0091] 2) input the first linear result into the first time-domain selection network 404 to obtain a first time-domain selection result. The first time-domain selection network can comprise at least one convolutional layer. The first time-domain selection network is used to indicate the key data in the first linear result from the time-domain dimension.

[0092] 3) activate the first time-domain selection result by calling a first activation function to obtain a first activation result.

[0093] For example, the first activation function is an optional or replaceable network structure. When the first activation function is deleted, the first time-domain selection result can be directly used as the first activation result to perform the subsequent steps.

[0094] 4) input the first activation result into the first time-domain SSM 406 to obtain a first prediction result.

[0095] 5) flip the first linear result in the time-domain dimension to obtain a first flipped result. This enables the time-domain network to model in both directions in the time-domain dimension, extract the correlation of the time-domain data from two directions, and improve the prediction accuracy.

[0096] 6) input the first flipped result into the second time-domain selection network 405 to obtain a second time-domain selection result. The second time-domain selection network can comprise at least one convolutional layer. The second time-domain selection network is used to indicate the key data in the first flipped result from the time-domain dimension.

[0097] 7) activate the second time-domain selection result by calling a second activation function to obtain a second activation result.

[0098] The second activation function is an optional or alternative network structure. When the second activation function is removed, the second time domain selection result can be taken as the second activation result to perform the subsequent steps.

[0099] The second linear network 402 is an optional or alternative network structure. When the second linear network 402 is removed, the time domain input data can be directly taken as the second linear result to perform the subsequent steps. Figure 5 The branch network for flipping the input data is an optional or alternative network structure. When the flipping branch network is removed, the steps 5) to 8) above can be removed, the subsequent steps 12) to 14) can be removed, and the first weighted sum in step 15) can be replaced by the first product.

[0100] 9) The time domain input data is input into the second linear network 402 to obtain a second linear result.

[0101] The second linear network 402 is an optional or alternative network structure. When the second linear network 402 is removed, the time domain input data can be directly taken as the second linear result to perform the subsequent steps.

[0102] 10) A third activation function is called to activate the second linear result to obtain a third activation result.

[0103] The third activation function is an optional or alternative network structure. When the third activation function is removed, the second linear result can be taken as the third activation result to perform the subsequent steps.

[0104] 11) The first prediction result is multiplied by the third activation result to obtain a first product.

[0105] 12) The second prediction result is multiplied by the third activation result to obtain a second product.

[0106] 13) The second product is flipped in the time domain dimension to obtain a second flipped result.

[0107] 14) A first weighted sum of the first product and the second flipped result is calculated. The weight can be a preset value or can be trained in the model training process. For example, the weight of the first product and the second flipped result can both be 0.5.

[0108] 15) The first weighted sum is input into the third linear network 403 to obtain time domain output data. The time domain network does not change the dimension of the input data, i.e., the dimension of the time domain output data is the same as that of the time domain input data, for example, the time domain output data is also T*N*D dimensional data.

[0109] The third linear network 403 is an optional or alternative network structure. When the third linear network 403 is removed, the first weighted sum can be directly taken as the time domain output data.

[0110] Step 2: input the frequency domain input data into the sequentially connected frequency domain selection network and the frequency domain SSM to obtain frequency domain output data; the frequency domain selection network is used to focus on the time domain data indicated by the time domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the time domain output data.

[0111] In an optional embodiment, as shown in Figure 6 The frequency domain network 303 includes a fourth linear network 501, a fifth linear network 502, a sixth linear network 503, a first frequency domain selection network 504, a second frequency domain selection network 505, a first frequency domain SSM 506, and a second frequency domain SSM 507. The network structure of the time domain network is the same as that of the frequency domain network, but the network parameters are different, and the time domain network processes the input data in the sequence of the time domain dimension, while the frequency domain network processes the input data in the sequence of the frequency domain dimension.

[0112] According to the network structure of the frequency domain network 303 as shown in Figure 6 Step 2 can be specifically implemented as:

[0113] 1) input the frequency domain input data into the fourth linear network 501 to obtain a fourth linear result. The frequency domain input data can be the time domain output data; or the frequency domain input data is obtained based on the time domain output data.

[0114] For example, the fourth linear network 501 is an optional or replaceable network structure. When the fourth linear network 501 is deleted, the frequency domain input data can be directly used as the fourth linear result to perform the subsequent steps.

[0115] 2) input the fourth linear result into the first frequency domain selection network 504 to obtain a first frequency domain selection result. The first frequency domain selection network can include at least one convolutional layer. The first frequency domain selection network is used to indicate the key data in the fourth linear result from the frequency domain dimension.

[0116] 3) activate the first frequency domain selection result by calling a fourth activation function to obtain a fourth activation result.

[0117] For example, the fourth activation function is an optional or replaceable network structure. When the fourth activation function is deleted, the first frequency domain selection result can be directly used as the fourth activation result to perform the subsequent steps.

[0118] 4) input the fourth activation result into the first frequency domain SSM 506 to obtain a third prediction result.

[0119] 5) flip the fourth linear result in the frequency domain dimension to obtain a third flipped result. This enables the frequency domain network to model in both directions in the frequency domain dimension, extract the correlation of the frequency domain data from two directions, and improve the prediction accuracy.

[0120] 6) input the third flipped result into the second frequency domain selection network 505 to obtain a second frequency domain selection result. The second frequency domain selection network can include at least one convolutional layer. The second frequency domain selection network is used to indicate the key data in the third flipped result from the frequency domain dimension.

[0121] 7) activate the second frequency domain selection result by invoking a fifth activation function to obtain a fifth activation result.

[0122] Exemplarily, the fifth activation function is an optional or alternative network structure. When the fifth activation function is deleted, the second frequency domain selection result can be executed as the fifth activation result in the subsequent steps.

[0123] 8) input the fifth activation result into the second frequency domain SSM 507 to obtain a fourth prediction result.

[0124] Exemplarily, Figure 6 The branch network that flips the input data in the above step 5)-8) can be deleted, the subsequent steps 12)-14) can be deleted, and the second weighted sum in step 15) can be replaced by a third product.

[0125] 9) input the frequency domain input data into the fifth linear network 502 to obtain a fifth linear result.

[0126] Exemplarily, the fifth linear network 502 is an optional or alternative network structure. When the fifth linear network 502 is deleted, the frequency domain input data can be directly executed as the fifth linear result in the subsequent steps.

[0127] 10) activate the fifth linear result by invoking a sixth activation function to obtain a sixth activation result.

[0128] Exemplarily, the sixth activation function is an optional or alternative network structure. When the sixth activation function is deleted, the fifth linear result can be executed as the sixth activation result in the subsequent steps.

[0129] 11) calculate a third product by multiplying the third prediction result and the sixth activation result.

[0130] 12) calculate a fourth product by multiplying the fourth prediction result and the sixth activation result.

[0131] 13) flip the fourth product in the frequency domain dimension to obtain a fourth flipped result.

[0132] 14) calculate a second weighted sum of the third product and the fourth flipped result. Exemplarily, the weight can be a preset value or can be trained in the model training process. For example, the weight of the third product and the fourth flipped result can both be 0.5.

[0133] 15) input the second weighted sum into the sixth linear network 503 to obtain the frequency domain output data. Illustratively, the frequency domain network does not change the dimension of the input data, i.e., the frequency domain output data has the same dimension as the frequency domain input data, for example, the frequency domain output data is also T*N*D dimensional data.

[0134] Illustratively, the sixth linear network 503 is an optional or alternative network structure, when the sixth linear network 503 is removed, the second weighted sum can be directly taken as the frequency domain output data.

[0135] Step 3: determine the frequency domain output data as the i-th output data.

[0136] In an optional embodiment, in order to improve the model performance and training stability, normalization and skip connection structures are introduced before and after the time domain network 302 and the frequency domain network 303, respectively.

[0137] As shown in FIG. 4, the input data needs to be normalized before being input into the time domain network 302; the output data of the time domain network 302 also needs to be convoluted by the first convolutional network 408, and is spliced with the input data, and the splicing result is taken as the input data of the frequency domain network 303. Before being input into the frequency domain network 303, the input data also needs to be normalized; the output data of the frequency domain network 303 also needs to be convoluted by the second convolutional network 409, and is spliced with the input data of the frequency domain network, and the splicing result is taken as the final output data. The adjusted method is shown as follows. Figure 7 Step 1 before also includes: obtaining the time domain input data based on the i-th input data; for example, determining the normalized i-th input data as the time domain input data.

[0138] Step 1: input the time domain input data into the sequentially connected time domain selection network and time domain SSM (i.e., the time domain network 302) to obtain the time domain output data.

[0139] Step 2 before also includes: splicing the i-th input data with the time domain output data to obtain the frequency domain input data; for example, inputting the time domain output data into the first convolutional network 408 to obtain the first convolutional result; splicing the first convolutional result with the i-th input data to obtain the first splicing result; determining the normalized first splicing result as the frequency domain input data.

[0140] Step 2: input the frequency domain input data into the sequentially connected frequency domain selection network and frequency domain SSM (i.e., the frequency domain network 303) to obtain the frequency domain output data.

[0141]

[0142] ​Step 3 can be replaced by: splicing the frequency domain output data and the frequency domain input data to obtain the i-th output data. For example, inputting the frequency domain output data into the second convolutional network 409 to obtain a second convolutional result; splicing the second convolutional result and the first splicing result to obtain a second splicing result; and determining the second splicing result as the i-th output data.

[0143] Step 223: In the case where the loop end condition is not met, determining the i-th output data as the i+1-th input data, and performing the i+1-th loop.

[0144] The loop end condition can be that the number of loops reaches a number threshold. Alternatively, the loop end condition can be that the i-th output data reaches a data threshold.

[0145] Step 224: In the case where the loop end condition is met, inputting the i-th output data into the second multi-layer perceptron 304 to obtain a target masking value corresponding to the target track.

[0146] For example, the second multi-layer perceptron includes an input layer, at least one hidden layer, and an output layer, each layer of the network includes a plurality of neurons, and the neurons are connected through weights. The second multi-layer perceptron is used to predict the target masking value according to the i-th output data.

[0147] For example, the second multi-layer perceptron 304 is an optional or replaceable network structure. When the second multi-layer perceptron 304 is deleted, the i-th output data can be used as the target masking value.

[0148] In an alternative embodiment, the time domain network and / or the frequency domain network can also be implemented using a Mamba model. The design of the Mamba model is optimized for currently available high-performance computing resources, and a hardware-aware algorithm is used. This design enables Mamba to run more efficiently on computing hardware such as GPUs (Graphic Processing Units), thereby improving model performance.

[0149] In summary, the method provided by the embodiment has slow reasoning speed and consumes long memory because the calculation amount of the self-attention mechanism in the Transformer model increases in a square level with the increase of the context length. The recurrent model provided by the application combines the advantages of the CNN (Convolutional Neural Network) and the RNN (Recurrent Neural Network), and can selectively focus on or ignore some parts of the input data, and has an effect comparable to the Transformer. Meanwhile, the recurrent model uses the SSM to model the time domain and the frequency domain of the input data, and the calculation complexity of the modeling is linear with the time length, and the hardware-aware algorithm is used to make the recurrent model more efficient when running on the GPU.

[0150] In an alternative embodiment, the recurrent network comprises a time domain network. For example, as shown in Figure 8 The audio signal separation method of the application is performed using an audio separation model. The audio separation model comprises a recurrent network, and the audio separation model comprises a first multilayer perceptron (MLP) 301, a time domain network 302, and a second multilayer perceptron 304 connected in sequence.

[0151] Based on the network structure, the audio signal separation method can be briefly summarized as follows: (1) converting the audio signal to a frequency domain signal by short-time Fourier transform, dividing the frequency domain signal by frame to form a sequence in the time dimension, and dividing the frequency domain signal by frequency point to form a sequence in the frequency domain dimension; (2) performing sequence modeling on the sequence in the time dimension to predict the masking value of the audio track; (3) multiplying the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and performing inverse short-time Fourier transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0152] Figure 4 is a flowchart of an audio signal separation method provided by an exemplary embodiment of the application. The method can be used in a terminal device or a server as shown in Figure 1 According to Figure 8 the network structure of the recurrent network as shown, step 220 comprises steps 221 to 224 based on the embodiment as shown in Figure 2

[0153] Step 210: performing Fourier transform on the audio signal 310 to obtain a frequency domain signal 320.

[0154] Step 221: inputting the frequency domain signal 320 into the first multilayer perceptron 301 to obtain the first input data of the first cycle of the recurrent network.

[0155] ​Exemplarily, the first multi-layer perceptron 301 is an optional or replaceable network structure, and when the first multi-layer perceptron 301 is deleted, the frequency domain signal 320 can be taken as the first input data to perform the subsequent steps.

[0156] Step 222: input the i-th input data into the recurrent network to obtain i-th output data, i is a positive integer; and the network is selected to indicate the key data in the i-th input data for the SSM.

[0157] Exemplarily, the recurrent network includes the time domain network 302; the time domain network 302 includes a time domain selection network and a time domain SSM. Then, step 222 can include the following steps:

[0158] Step 1: input the time domain input data into the sequentially connected time domain selection network and time domain SSM to obtain time domain output data; the time domain selection network is used to indicate the key data in the time domain for the time domain SSM in the time domain dimension; and the time domain input data is obtained based on the i-th input data.

[0159] In an optional embodiment, as shown in Figure 5 the time domain network 302 includes a first linear network 401, a second linear network 402, a third linear network 403, a first time domain selection network 404, a second time domain selection network 405, a first time domain SSM 406, and a second time domain SSM 407.

[0160] According to the network structure of the time domain network 302 as shown in Figure 5 step 1 can be specifically implemented as:

[0161] 1) input the time domain input data into the first linear network 401 to obtain a first linear result. The time domain input data can be the i-th input data; or the time domain input data is obtained based on the i-th input data.

[0162] Exemplarily, the first linear network 401 is an optional or replaceable network structure, and when the first linear network 401 is deleted, the time domain input data can be directly taken as the first linear result to perform the subsequent steps.

[0163] 2) input the first linear result into the first time domain selection network 404 to obtain a first time domain selection result. The first time domain selection network can include at least one convolutional layer. The first time domain selection network is used to indicate the key data in the first linear result in the time domain dimension.

[0164] 3) activate the first time domain selection result by calling a first activation function to obtain a first activation result.

[0165] Exemplarily, the first activation function is an optional or alternative network structure, when the first activation function is removed, the first time domain selection result can be taken as the first activation result to execute the subsequent steps.

[0166] 4) input the first activation result into the first time domain SSM 406 to obtain a first prediction result.

[0167] 5) flip the first linear result in the time domain dimension to obtain a first flipped result. The time domain network can be bidirectionally modeled in the time domain dimension from the forward direction and the reverse direction, and the correlation of the time domain data is extracted from two directions to improve the prediction accuracy.

[0168] 6) input the first flipped result into the second time domain selection network 405 to obtain a second time domain selection result. The second time domain selection network can include at least one convolutional layer. The second time domain selection network is used to indicate the key data in the first flipped result from the time domain dimension.

[0169] 7) call the second activation function to activate the second time domain selection result to obtain a second activation result.

[0170] Exemplarily, the second activation function is an optional or alternative network structure, when the second activation function is removed, the second time domain selection result can be taken as the second activation result to execute the subsequent steps. 8) input the second activation result into the second time domain SSM 407 to obtain a second prediction result.

[0171] Exemplarily, Figure 5 The branch network that flips the input data in the above-mentioned steps 5)-8) can be removed, the subsequent steps 12)-14) can be removed, and the first weighted sum in step 15) can be replaced by the first product.

[0172] 9) input the time domain input data into the second linear network 402 to obtain a second linear result.

[0173] Exemplarily, the second linear network 402 is an optional or alternative network structure, when the second linear network 402 is removed, the time domain input data can be directly taken as the second linear result to execute the subsequent steps.

[0174] 10) call the third activation function to activate the second linear result to obtain a third activation result.

[0175] Exemplarily, the third activation function is an optional or alternative network structure, when the third activation function is removed, the second linear result can be taken as the third activation result to execute the subsequent steps.

[0176] 11) calculate the first prediction result multiplied by the third activation result to obtain a first product.

[0177] 12) Calculate the second prediction result multiplied by the third activation result to obtain the second product.

[0178] 13) Flip the second product in the time domain dimension to obtain the second flipped result.

[0179] 14) Calculate the first weighted sum of the first product and the second flip result. For example, the weights can be preset values ​​or obtained during model training. For instance, the weights of the first product and the second flip result can both be 0.5.

[0180] 15) Input the first weighted sum into the third linear network 403 to obtain the time-domain output data. For example, the time-domain network does not change the dimension of the input data, that is, the time-domain output data has the same dimension as the time-domain input data, for example, the time-domain output data is also data of the T*N*D dimension.

[0181] For example, the third linear network 403 is an optional or replaceable network structure. When the third linear network 403 is deleted, the first weighted sum can be directly used as the time-domain output data.

[0182] Step 2: Determine the time-domain output data as the i-th output data.

[0183] In an alternative embodiment, to improve model performance and training stability, normalization and skip connection structures are introduced before and after the temporal network 302.

[0184] like Figure 9 As shown, the input data needs to be normalized before entering the time-domain network 302; the output data of the time-domain network 302 also needs to be convolved by the first convolutional network 408 and concatenated with the input data. The concatenated result is used as the final output data. The adjusted method is shown below.

[0185] Before step 1, the process also includes: obtaining time-domain input data based on the i-th input data; for example, determining the normalized i-th input data as time-domain input data.

[0186] Step 1: Input the time-domain input data into the sequentially connected time-domain selection network and time-domain SSM (i.e., time-domain network 302) to obtain the time-domain output data.

[0187] Step 2 can be replaced by concatenating the time-domain output data with the time-domain input data to obtain the i-th output data. For example, input the time-domain output data into the first convolutional network 408 to obtain the first convolution result; concatenate the first convolution result with the time-domain input data to obtain the first concatenation result; and determine the first concatenation result as the i-th output data.

[0188] Step 223: In the case where the loop end condition is not met, the i-th output data is determined as the i+1-th input data, and the i+1-th loop is executed.

[0189] Step 224: In the case where the loop end condition is met, the i-th output data is input into the second multilayer perceptron to obtain a target masking value corresponding to the target track.

[0190] For example, the second multilayer perceptron 304 is an optional or alternative network structure, and when the second multilayer perceptron 304 is deleted, the i-th output data can be used as the target masking value.

[0191] In summary, the method provided in the embodiment has slow reasoning speed and consumes long memory because the calculation amount of the self-attention mechanism in the Transformer model increases in a square level with the increase of the context length. The loop model provided in the present application combines the advantages of CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), uses SSM to model the input data in the time domain dimension, can selectively focus on or ignore some parts of the input data, and has a comparable effect to the Transformer. At the same time, the loop model uses SSM to model the input data in the time domain, and the calculation complexity of the modeling is linear with the time length, and the use of hardware perception algorithm makes it more efficient in GPU running.

[0192] In an optional embodiment, the loop network includes a frequency domain network. For example, as shown in Figure 10 The audio signal separation method of the present application is performed using an audio separation model. The audio separation model includes a loop network, and the audio separation model includes a first multilayer perceptron (MLP) 301, a frequency domain network 303, and a second multilayer perceptron 304 connected in sequence.

[0193] Based on the network structure, the audio signal separation method can be briefly summarized as follows: (1) converting the audio signal into a frequency domain signal by short-time Fourier transform, dividing the frequency domain signal into a sequence in the time dimension according to frames, and dividing the frequency domain signal into a sequence in the frequency domain dimension according to frequency points; (2) performing sequence modeling on the sequence in the frequency domain dimension to predict the masking value of the audio track; (3) multiplying the frequency domain signal by the predicted masking value to obtain the track frequency domain signal of the audio track, and performing inverse short-time Fourier transform on the frequency domain signal to obtain the track audio signal of the audio track.

[0194] Figure 4is a flowchart of an audio signal separation method provided by an example embodiment of the present application. The method can be used in a terminal device or a server as shown in Figure 1 According to the network structure of the recurrent network as shown in Figure 10 Based on the network structure of the recurrent network as shown in Figure 2 According to the embodiment as shown in

[0195] Step 210: Perform Fourier transform on the audio signal 310 to obtain a frequency domain signal 320.

[0196] Step 221: Input the frequency domain signal 320 into the first multi-layer perceptron 301 to obtain first input data of the first cycle of the recurrent network.

[0197] For example, the first multi-layer perceptron 301 is an optional or replaceable network structure. When the first multi-layer perceptron 301 is deleted, the frequency domain signal 320 can be used as the first input data to perform the subsequent steps.

[0198] Step 222: For the i-th cycle in the recurrent network, input the i-th input data into the recurrent network to obtain the i-th output data, i is a positive integer; select the network to indicate the key attention data in the i-th input data for the SSM.

[0199] For example, the recurrent network includes a frequency domain network 303; the frequency domain network 303 includes a frequency domain selection network and a frequency domain SSM. Then step 222 can include the following steps:

[0200] Step 1: Input the frequency domain input data into the sequentially connected frequency domain selection network and frequency domain SSM to obtain frequency domain output data; the frequency domain selection network is used to indicate the time domain key attention data for the time domain SSM in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data.

[0201] In an optional embodiment, as shown in Figure 6 The frequency domain network 303 includes a fourth linear network 501, a fifth linear network 502, a sixth linear network 503, a first frequency domain selection network 504, a second frequency domain selection network 505, a first frequency domain SSM 506, and a second frequency domain SSM 507. The network structure of the time domain network is the same as that of the frequency domain network, but the network parameters are different, and the time domain network processes the input data in the sequence of the time domain dimension, while the frequency domain network processes the input data in the sequence of the frequency domain dimension.

[0202] According to the network structure of the frequency domain network 303 as shown in Figure 6 Step 2 can be specifically implemented as:

[0203] 1) input the frequency domain input data into the fourth linear network 501 to obtain a fourth linear result. The frequency domain input data can be the i-th input data; or the frequency domain input data is obtained based on the i-th input data.

[0204] For example, the fourth linear network 501 is an optional or replaceable network structure. When the fourth linear network 501 is deleted, the frequency domain input data can be directly used as the fourth linear result to perform the subsequent steps.

[0205] 2) input the fourth linear result into the first frequency domain selection network 504 to obtain a first frequency domain selection result. The first frequency domain selection network can include at least one convolutional layer. The first frequency domain selection network is used to indicate the key data in the fourth linear result from the frequency domain dimension.

[0206] 3) activate the first frequency domain selection result by calling a fourth activation function to obtain a fourth activation result.

[0207] For example, the fourth activation function is an optional or replaceable network structure. When the fourth activation function is deleted, the first frequency domain selection result can be used as the fourth activation result to perform the subsequent steps.

[0208] 4) input the fourth activation result into the first frequency domain SSM 506 to obtain a third prediction result.

[0209] 5) flip the fourth linear result in the frequency domain dimension to obtain a third flipped result. This enables the frequency domain network to model in both directions in the frequency domain dimension, extract the correlation of the frequency domain data from two directions, and improve the prediction accuracy.

[0210] 6) input the third flipped result into the second frequency domain selection network 505 to obtain a second frequency domain selection result. The second frequency domain selection network can include at least one convolutional layer. The second frequency domain selection network is used to indicate the key data in the third flipped result from the frequency domain dimension.

[0211] 7) activate the second frequency domain selection result by calling a fifth activation function to obtain a fifth activation result.

[0212] For example, the fifth activation function is an optional or replaceable network structure. When the fifth activation function is deleted, the second frequency domain selection result can be used as the fifth activation result to perform the subsequent steps.

[0213] 8) input the fifth activation result into the second frequency domain SSM 507 to obtain a fourth prediction result.

[0214] For example, Figure 6The branch network for flipping the input data is an optional or alternative network structure. When the flipping branch network is removed, steps 5)-8) above can be removed, subsequent steps 12)-14) can be removed, and the second weighted sum in step 15) can be replaced by a third product.

[0215] 9) Input the frequency domain input data into a fifth linear network 502 to obtain a fifth linear result.

[0216] The fifth linear network 502 is an optional or alternative network structure. When the fifth linear network 502 is removed, the frequency domain input data can be directly used as the fifth linear result to perform subsequent steps.

[0217] 10) Call a sixth activation function to activate the fifth linear result to obtain a sixth activation result.

[0218] The sixth activation function is an optional or alternative network structure. When the sixth activation function is removed, the fifth linear result can be used as the sixth activation result to perform subsequent steps.

[0219] 11) Calculate a third product by multiplying the third prediction result and the sixth activation result.

[0220] 12) Calculate a fourth product by multiplying the fourth prediction result and the sixth activation result.

[0221] 13) Flip the fourth product in the frequency domain dimension to obtain a fourth flipped result.

[0222] 14) Calculate a second weighted sum of the third product and the fourth flipped result. The weight can be a preset value or can be trained during model training. For example, the weight of the third product and the fourth flipped result can both be 0.5.

[0223] 15) Input the second weighted sum into a sixth linear network 503 to obtain frequency domain output data. The frequency domain network does not change the dimension of the input data, i.e., the dimension of the frequency domain output data is the same as that of the frequency domain input data, for example, the frequency domain output data is also T*N*D dimensional data.

[0224] The sixth linear network 503 is an optional or alternative network structure. When the sixth linear network 503 is removed, the second weighted sum can be directly used as the frequency domain output data.

[0225] Step 2: Determine the frequency domain output data as the i-th output data.

[0226] In an optional embodiment, to improve model performance and training stability, normalization and skip connection structures are introduced before and after the frequency domain network 303.

[0227] As shown in Figure 11 , the input data needs to be normalized before being input into the input frequency domain network 303; the output data of the frequency domain network 303 needs to be convoluted by the second convolutional network 409 and spliced with the input data of the frequency domain network, and the splicing result is the final output data. The adjusted method is as follows.

[0228] Before step 1, it also includes obtaining the frequency domain input data based on the i-th input data; for example, determining the normalized i-th input data as the frequency domain input data.

[0229] Step 1: input the frequency domain input data into the sequentially connected frequency domain selection network and frequency domain SSM (i.e., the frequency domain network 303) to obtain the frequency domain output data.

[0230] Step 2 can be replaced by: splicing the frequency domain output data with the frequency domain input data to obtain the i-th output data. For example, input the frequency domain output data into the second convolutional network 409 to obtain the second convolutional result; splice the second convolutional result with the frequency domain input data to obtain the second splicing result; and determine the second splicing result as the i-th output data.

[0231] Step 223: in the case of not meeting the loop end condition, determine the i-th output data as the i+1-th input data and execute the i+1-th loop.

[0232] Step 224: in the case of meeting the loop end condition, input the i-th output data into the second multilayer perceptron 304 to obtain the target masking value corresponding to the target track.

[0233] For example, the second multilayer perceptron 304 is an optional or replaceable network structure, and when the second multilayer perceptron 304 is deleted, the i-th output data can be used as the target masking value.

[0234] In summary, the method provided in the embodiment has slow reasoning speed and consumes long memory because the calculation amount of the self-attention mechanism in the Transformer model increases in a square level with the increase of the context length. The recurrent model provided in the application combines the advantages of the CNN (Convolutional Neural Network) and the RNN (Recurrent Neural Network), uses the SSM to model the input data in the frequency domain dimension, can selectively focus on or ignore some parts of the input data, and has an effect comparable to the Transformer. Meanwhile, the recurrent model uses the SSM to model the input data in the frequency domain, and the calculation complexity and length of the modeling are linear, and the hardware-aware algorithm is used to make the recurrent model more efficient when running on the GPU.

[0235] It should be noted that the order of the method steps provided in the embodiments of the application can be appropriately adjusted, and the steps can be increased or decreased as appropriate. Any person skilled in the art can easily think of changes within the technical scope disclosed in the application, which should be covered within the protection scope of the application, and therefore will not be described again.

[0236] Figure 12 FIG. 1 is a structural schematic diagram of an audio signal separation device provided in an example embodiment of the application. The device includes:

[0237] The transform module 601 is configured to perform Fourier transform on the audio signal to obtain a frequency domain signal.

[0238] The recurrent module 602 is configured to call a recurrent network to obtain a target masking value corresponding to a target track based on the frequency domain signal. The target masking value is used to indicate a frequency point of the audio signal of the target track. The recurrent network includes a selection network and a state space model (SSM) connected in sequence. The selection network is used to indicate important data in the input data for the SSM.

[0239] The separation module 603 is configured to determine a product of the frequency domain signal and the target masking value as a target track frequency domain signal of the target track.

[0240] The inverse transform module 604 is configured to perform inverse Fourier transform on the target track frequency domain signal to obtain a target track audio signal of the target track.

[0241] In an optional embodiment, the recurrent module 602 is configured to:

[0242] input the frequency domain signal into a first multi-layer perceptron to obtain first input data of a first cycle of the recurrent network;

[0243] For the i-th loop in the loop network, input the i-th input data into the loop network to obtain i-th output data, i is a positive integer; the selection network is configured to indicate the SSM with focus data in the i-th input data;

[0244] In the case where the loop end condition is not met, determine the i-th output data as i+1-th input data, and perform i+1-th loop;

[0245] In the case where the loop end condition is met, input the i-th output data into a second multi-layer perceptron to obtain the target masking value corresponding to the target trajectory.

[0246] In an optional embodiment, the loop network includes a time domain network and a frequency domain network connected in sequence; the time domain network includes a time domain selection network and a time domain SSM, and the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the loop module 602 is configured to:

[0247] input time domain input data into the time domain selection network and the time domain SSM connected in sequence to obtain time domain output data; the time domain selection network is configured to indicate time domain focus data in the time domain dimension for the time domain SSM; the time domain input data is obtained based on the i-th input data; input frequency domain input data into the frequency domain selection network and the frequency domain SSM connected in sequence to obtain frequency domain output data; the frequency domain selection network is configured to indicate frequency domain focus data in the frequency domain dimension for the frequency domain SSM; the frequency domain input data is obtained based on the time domain output data;

[0248] determine the frequency domain output data as the i-th output data.

[0249] In an optional embodiment, the loop network includes a time domain network; the time domain network includes a time domain selection network and a time domain SSM; the loop module 602 is configured to:

[0250] input time domain input data into the time domain selection network and the time domain SSM connected in sequence to obtain time domain output data; the time domain selection network is configured to indicate time domain focus data in the time domain dimension for the time domain SSM; the time domain input data is obtained based on the i-th input data;

[0251] determine the time domain output data as the i-th output data.

[0252] In an optional embodiment, the loop network includes a frequency domain network; the frequency domain network includes a frequency domain selection network and a frequency domain SSM; the loop module 602 is configured to:

[0253] inputting the frequency domain input data into the sequentially connected frequency domain selection network and the frequency domain SSM to obtain frequency domain output data; the frequency domain selection network is configured to indicate frequency domain focus data for the frequency domain SSM in a frequency domain dimension; the frequency domain input data is obtained based on the ith input data;

[0254] determining the frequency domain output data as the ith output data.

[0255] In an optional embodiment, the time domain network comprises: a first linear network, a second linear network, a third linear network, a first time domain selection network, a second time domain selection network, a first time domain SSM, and a second time domain SSM; and the loop module 602 is configured to:

[0256] inputting the time domain input data into the first linear network to obtain a first linear result;

[0257] inputting the first linear result into the first time domain selection network to obtain a first time domain selection result;

[0258] activating the first time domain selection result by calling a first activation function to obtain a first activation result;

[0259] inputting the first activation result into the first time domain SSM to obtain a first prediction result;

[0260] reversing the first linear result in a time domain dimension to obtain a first reversed result;

[0261] inputting the first reversed result into the second time domain selection network to obtain a second time domain selection result;

[0262] activating the second time domain selection result by calling a second activation function to obtain a second activation result;

[0263] inputting the second activation result into the second time domain SSM to obtain a second prediction result;

[0264] inputting the time domain input data into the second linear network to obtain a second linear result;

[0265] activating the second linear result by calling a third activation function to obtain a third activation result;

[0266] calculating the first prediction result multiplied by the third activation result to obtain a first product;

[0267] calculating the second prediction result multiplied by the third activation result to obtain a second product;

[0268] reversing the second product in the time domain dimension to obtain a second reversed result;

[0269] computing a first weighted sum of the first product and the first flipped result;

[0270] inputting the first weighted sum into a third linear network to obtain the time domain output data.

[0271] In an alternative embodiment, the frequency domain network comprises: a fourth linear network, a fifth linear network, a sixth linear network, a first frequency domain selection network, a second frequency domain selection network, a first frequency domain SSM and a second frequency domain SSM; the loop module 602 is configured to:

[0272] inputting the frequency domain input data into the fourth linear network to obtain a fourth linear result;

[0273] inputting the fourth linear result into the first frequency domain selection network to obtain a first frequency domain selection result;

[0274] activating the first frequency domain selection result by invoking a fourth activation function to obtain a fourth activation result;

[0275] inputting the fourth activation result into the first frequency domain SSM to obtain a third prediction result;

[0276] flipping the fourth linear result in the frequency domain dimension to obtain a third flipped result;

[0277] inputting the third flipped result into the second frequency domain selection network to obtain a second frequency domain selection result;

[0278] activating the second frequency domain selection result by invoking a fifth activation function to obtain a fifth activation result;

[0279] inputting the fifth activation result into the second frequency domain SSM to obtain a fourth prediction result;

[0280] inputting the frequency domain input data into the fifth linear network to obtain a fifth linear result;

[0281] activating the fifth linear result by invoking a sixth activation function to obtain a sixth activation result;

[0282] computing a third product of the third prediction result and the sixth activation result;

[0283] computing a fourth product of the fourth prediction result and the sixth activation result;

[0284] flipping the fourth product in the frequency domain dimension to obtain a fourth flipped result;

[0285] computing a second weighted sum of the third product and the fourth flipped result;

[0286] inputting the second weighted sum into a sixth linear network to obtain the frequency domain output data.

[0287] In an optional embodiment, the loop module 602 is configured to:

[0288] obtain the time domain input data based on the i-th input data;

[0289] splice the i-th input data and the time domain output data to obtain the frequency domain input data;

[0290] The determining the frequency domain output data as the i-th output data comprises:

[0291] splicing the frequency domain output data and the frequency domain input data to obtain the i-th output data.

[0292] In an optional embodiment, the loop module 602 is configured to:

[0293] determine the normalized i-th input data as the time domain input data;

[0294] The splicing the i-th input data and the time domain output data to obtain the frequency domain input data comprises:

[0295] inputting the time domain output data into a first convolutional network to obtain a first convolutional result; splicing the first convolutional result and the i-th input data to obtain a first splicing result; determining the normalized first splicing result as the frequency domain input data;

[0296] The splicing the frequency domain output data and the frequency domain input data to obtain the i-th output data comprises:

[0297] inputting the frequency domain output data into a second convolutional network to obtain a second convolutional result; splicing the second convolutional result and the first splicing result to obtain a second splicing result; determining the second splicing result as the i-th output data.

[0298] It should be noted that the audio signal separation device provided in the above embodiments is only exemplified by the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the audio signal separation device and the audio signal separation method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0299] Embodiments of the present application also provide a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the audio signal separation method provided by the above method embodiments. The computer device can be implemented as a terminal device.

[0300] For example, Figure 13 FIG. 1 is a structural schematic diagram of a computer device provided by an example embodiment of the present application.

[0301] Generally, the computer device 1700 comprises a processor 1701 and a memory 1702.

[0302] The processor 1701 can comprise one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1701 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1701 can also comprise a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1701 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1701 can further comprise an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0303] The memory 1702 can comprise one or more computer-readable storage media, which can be non-transitory. The memory 1702 can also comprise a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store at least one instruction for being executed by the processor 1701 to implement the audio signal separation method provided by the method embodiments.

[0304] In some embodiments, the computer device 1700 can also optionally include a peripheral interface 1703 and at least one peripheral device. The processor 1701, the memory 1702 and the peripheral interface 1703 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral interface 1703 through a bus, a signal line or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 1704, a display screen 1705, a camera component 1706, an audio circuit 1707 and a power supply 1708.

[0305] The peripheral interface 1703 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1701 and the memory 1702. In some embodiments, the processor 1701, the memory 1702 and the peripheral interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1701, the memory 1702 and the peripheral interface 1703 can be implemented on a separate chip or circuit board, and the embodiments of the present application do not limit this.

[0306] The radio frequency circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1704 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1704 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1704 can communicate with other computer devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1704 can also include NFC (Near Field Communication) related circuitry, and the present application does not limit this.

[0307] The display screen 1705 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1705 is a touch display screen, the display screen 1705 is further configured to capture touch signals on or above the surface of the display screen 1705. The touch signals can be input to the processor 1701 as control signals for processing. In this case, the display screen 1705 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1705 can be one, configured on the front panel of the computer device 1700; in other embodiments, the display screen 1705 can be at least two, respectively configured on different surfaces of the computer device 1700 or in a folding design; in still other embodiments, the display screen 1705 can be a flexible display screen, configured on a curved surface or a folding surface of the computer device 1700. Even, the display screen 1705 can also be configured in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 1705 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0308] The camera assembly 1706 is configured to capture images or videos. Optionally, the camera assembly 1706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is configured on the front panel of the computer device 1700, and the rear-facing camera is configured on the back of the computer device. In some embodiments, the rear-facing camera is at least two, respectively any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1706 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0309] The audio circuit 1707 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1701 for processing, or input to the radio frequency circuit 1704 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the computer device 1700. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 1707 can also include a headphone jack.

[0310] The power supply 1708 is used to supply power to various components in the computer device 1700. The power supply 1708 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1708 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery that is charged through a wired line, and the wireless charging battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0311] In some embodiments, the computer device 1700 further includes one or more sensors 1709. The one or more sensors 1709 include, but are not limited to, an acceleration sensor 1710, a gyroscope sensor 1711, a pressure sensor 1712, an optical sensor 1713, and a proximity sensor 1714.

[0312] The acceleration sensor 1710 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the computer device 1700. For example, the acceleration sensor 1710 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1701 can control the touch display 1705 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1710. The acceleration sensor 1710 can also be used for game or user motion data collection.

[0313] The gyroscope sensor 1711 can detect the body orientation and rotation angle of the computer device 1700, and the gyroscope sensor 1711 can collect 3D actions of the user on the computer device 1700 in cooperation with the acceleration sensor 1710. The processor 1701 can realize the following functions according to the data collected by the gyroscope sensor 1711: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0314] Pressure sensor 1712 can be disposed on the side bezel of computer device 1700 and / or on the lower layer of touch display screen 1705. When pressure sensor 1712 is disposed on the side bezel of computer device 1700, it can detect the user's grip signal on computer device 1700, and processor 1701 can perform left / right hand recognition or quick operation based on the grip signal collected by pressure sensor 1712. When pressure sensor 1712 is disposed on the lower layer of touch display screen 1705, processor 1701 can control operable controls on the UI interface based on the user's pressure operation on touch display screen 1705. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0315] Optical sensor 1713 is used to collect ambient light intensity. In one embodiment, processor 1701 can control the display brightness of touch display screen 1705 based on the ambient light intensity collected by optical sensor 1713. Specifically, when the ambient light intensity is high, the display brightness of touch display screen 1705 is increased; when the ambient light intensity is low, the display brightness of touch display screen 1705 is decreased. In another embodiment, processor 1701 can also dynamically adjust the shooting parameters of camera assembly 1706 based on the ambient light intensity collected by optical sensor 1713.

[0316] The proximity sensor 1714, also known as a distance sensor, is typically located on the front panel of the computer device 1700. The proximity sensor 1714 is used to detect the distance between the user and the front of the computer device 1700. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually decreasing, the processor 1701 controls the touch display screen 1705 to switch from a screen-on state to a screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually increasing, the processor 1701 controls the touch display screen 1705 to switch from a screen-off state to a screen-on state.

[0317] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on the computer device 1700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0318] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the audio signal separation method provided in the above-described method embodiments is implemented.

[0319] The application further provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the audio signal separation method provided by each method embodiment.

[0320] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The readable storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0321] The above only describes optional embodiments of the application and is not used to limit the application. Any modification, equivalent switching, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. An audio signal separation method, characterized in that, The method includes: Perform a Fourier transform on the audio signal to obtain the frequency domain signal; The frequency domain signal is processed by a recurrent network to obtain the target masking value corresponding to the target track; the target masking value is used to indicate the frequency point where the audio signal of the target track is located; the recurrent network includes a sequentially connected selection network and a state space model, and the selection network is used to indicate the key data of interest in the input data to the state space model; The product of the frequency domain signal and the target masking value is determined as the target orbit frequency domain signal of the target orbit; Perform an inverse Fourier transform on the target orbit frequency domain signal to obtain the target orbit audio signal; The method of calling the loop network to obtain the target masking value corresponding to the target orbit based on the frequency domain signal includes: The frequency domain signal is input into the first multilayer perceptron to obtain the first input data of the first cycle of the recurrent network; For the i-th iteration in the recurrent network, the i-th input data is input into the recurrent network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key data in the i-th input data for the state space model; If the loop termination condition is not met, the i-th output data is determined as the (i+1)-th input data, and the (i+1)-th loop is executed. If the loop termination condition is met, the i-th output data is input into the second multilayer sensor to obtain the target masking value corresponding to the target orbit.

2. The method according to claim 1, characterized in that, The cyclic network includes a time-domain network and a frequency-domain network connected in sequence; the time-domain network includes a time-domain selection network and a time-domain state-space model, and the frequency-domain network includes a frequency-domain selection network and a frequency-domain state-space model. The step of inputting the i-th input data into the recurrent network to obtain the i-th output data includes: The time-domain input data is sequentially connected to the time-domain selection network and the time-domain state-space model to obtain time-domain output data; the time-domain selection network is used to indicate the time-domain focus data for the time-domain state-space model in the time-domain dimension; the time-domain input data is obtained based on the i-th input data. The frequency domain input data is sequentially connected to the frequency domain selection network and the frequency domain state space model to obtain frequency domain output data; the frequency domain selection network is used to indicate the frequency domain focus data for the frequency domain state space model in the frequency domain dimension; the frequency domain input data is obtained based on the time domain output data. The frequency domain output data is determined as the i-th output data.

3. The method according to claim 1, characterized in that, The recurrent network includes a time-domain network; the time-domain network includes a time-domain selection network and a time-domain state-space model; The step of inputting the i-th input data into the recurrent network to obtain the i-th output data includes: The time-domain input data is sequentially connected to the time-domain selection network and the time-domain state-space model to obtain time-domain output data; the time-domain selection network is used to indicate the time-domain focus data for the time-domain state-space model in the time-domain dimension; the time-domain input data is obtained based on the i-th input data. The time-domain output data is determined as the i-th output data.

4. The method according to claim 1, characterized in that, The cyclic network includes a frequency domain network; the frequency domain network includes a frequency domain selection network and a frequency domain state space model; The step of inputting the i-th input data into the recurrent network to obtain the i-th output data includes: The frequency domain input data is sequentially connected to the frequency domain selection network and the frequency domain state space model to obtain frequency domain output data; the frequency domain selection network is used to indicate the frequency domain focus data for the frequency domain state space model in the frequency domain dimension; the frequency domain input data is obtained based on the i-th input data; The frequency domain output data is determined as the i-th output data.

5. The method according to claim 2 or 3, characterized in that, The time-domain network includes: a first linear network, a second linear network, a third linear network, a first time-domain selection network, a second time-domain selection network, a first time-domain state-space model, and a second time-domain state-space model; The process of sequentially connecting the time-domain input data into the time-domain selection network and the time-domain state-space model to obtain time-domain output data includes: The time-domain input data is input into the first linear network to obtain the first linear result; The first linear result is input into the first time-domain selection network to obtain the first time-domain selection result; The first activation function is called to activate the first time-domain selection result, resulting in the first activation result. The first activation result is input into the first time-domain state-space model to obtain the first prediction result; The first linear result is flipped in the time domain to obtain the first flipped result; The first flipping result is input into the second time-domain selection network to obtain the second time-domain selection result; The second activation function is called to activate the second time-domain selection result, resulting in the second activation result. The second activation result is input into the second time-domain state-space model to obtain the second prediction result; The time-domain input data is input into the second linear network to obtain the second linear result; The third activation function is called to activate the second linear result, resulting in the third activation result; Calculate the first prediction result multiplied by the third activation result to obtain a first product; and calculate the second prediction result multiplied by the third activation result to obtain a second product; The second product is flipped in the time domain to obtain the second flipped result; Calculate the first weighted sum of the first product and the second flipped result; The first weighted sum is input into the third linear network to obtain the time-domain output data.

6. The method according to claim 2 or 4, characterized in that, The frequency domain network includes: a fourth linear network, a fifth linear network, a sixth linear network, a first frequency domain selection network, a second frequency domain selection network, a first frequency domain state space model, and a second frequency domain state space model; The process of sequentially connecting the frequency domain input data into the frequency domain selection network and the frequency domain state space model to obtain frequency domain output data includes: The frequency domain input data is input into the fourth linear network to obtain the fourth linear result; The fourth linear result is input into the first frequency domain selection network to obtain the first frequency domain selection result; The fourth activation function is called to activate the first frequency domain selection result, resulting in the fourth activation result. The fourth activation result is input into the first frequency domain state space model to obtain the third prediction result; The fourth linear result is flipped in the frequency domain to obtain the third flipped result; The third flip result is input into the second frequency domain selection network to obtain the second frequency domain selection result; The fifth activation function is called to activate the second frequency domain selection result, resulting in the fifth activation result. The fifth activation result is input into the second frequency domain state space model to obtain the fourth prediction result; The frequency domain input data is input into the fifth linear network to obtain the fifth linear result; The sixth activation function is called to activate the fifth linear result, resulting in the sixth activation result. The third prediction result is multiplied by the sixth activation result to obtain the third product; and the fourth prediction result is multiplied by the sixth activation result to obtain the fourth product. The fourth product is flipped in the frequency domain dimension to obtain the fourth flipped result; Calculate the second weighted sum of the third product and the fourth flip result; The second weighted sum is input into the sixth linear network to obtain the frequency domain output data.

7. The method according to claim 2, characterized in that, The method further includes: The time-domain input data is obtained based on the i-th input data; The i-th input data is concatenated with the time-domain output data to obtain the frequency-domain input data; Determining the frequency domain output data as the i-th output data includes: The frequency domain output data is concatenated with the frequency domain input data to obtain the i-th output data.

8. The method according to claim 7, characterized in that, The process of obtaining the time-domain input data based on the i-th input data includes: The normalized i-th input data is determined as the time-domain input data; The step of concatenating the i-th input data with the time-domain output data to obtain the frequency-domain input data includes: The time-domain output data is input into a first convolutional network to obtain a first convolution result; the first convolution result is concatenated with the i-th input data to obtain a first concatenation result; the normalized first concatenation result is determined as the frequency-domain input data. The step of concatenating the frequency domain output data with the frequency domain input data to obtain the i-th output data includes: The frequency domain output data is input into a second convolutional network to obtain a second convolution result; the second convolution result is concatenated with the first concatenation result to obtain a second concatenation result; the second concatenation result is determined as the i-th output data.

9. An audio signal separation device, characterized in that, The device includes: The transformation module is used to perform Fourier transform on the audio signal to obtain the frequency domain signal; The loop module is used to call the loop network to obtain the target masking value corresponding to the target track based on the frequency domain signal; the target masking value is used to indicate the frequency point where the audio signal of the target track is located; the loop network includes a sequentially connected selection network and a state space model, the selection network is used to indicate the key data of interest in the input data to the state space model; A separation module is used to determine the target orbit frequency domain signal by multiplying the frequency domain signal with the target masking value; The inverse transform module is used to perform an inverse Fourier transform on the target track frequency domain signal to obtain the target track audio signal; The method of calling the loop network to obtain the target masking value corresponding to the target orbit based on the frequency domain signal includes: The frequency domain signal is input into the first multilayer perceptron to obtain the first input data of the first cycle of the recurrent network; For the i-th iteration in the recurrent network, the i-th input data is input into the recurrent network to obtain the i-th output data, where i is a positive integer; the selection network is used to indicate the key data in the i-th input data for the state space model; If the loop termination condition is not met, the i-th output data is determined as the (i+1)-th input data, and the (i+1)-th loop is executed. If the loop termination condition is met, the i-th output data is input into the second multilayer sensor to obtain the target masking value corresponding to the target orbit.

10. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the audio signal separation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one program, which is loaded and executed by a processor to implement the audio signal separation method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the audio signal separation method as described in any one of claims 1 to 8.

Images