Accompaniment extraction method and system based on modal decomposition

Abstract

The invention discloses an accompaniment extraction method and system based on modal decomposition. The method comprises the steps that S1, a microphone array is adopted to collect multi-source sound signals; s2, constructing a multi-source sound signal array; performing modal decomposition on the multi-source sound signal array by adopting a principal component analysis method to obtain a characteristic value; s3, solving a principal component signal corresponding to the characteristic value, and identifying the principal component signal to obtain a principal component signal marked as an accompaniment component; s4, converting the principal component signal marked as the accompaniment component into a frequency domain signal, and adding an error correction signal to obtain an updated frequency domain signal; s5, performing inverse Fourier transform on the updated frequency domain signal to obtain an updated principal component signal; and combining to obtain an accompaniment signal. A modal decomposition method is adopted, principal component signals of the multi-source sound signal array are accurately separated, and accompaniment components can be recognized; and adding an error correction signal to the principal component signal marked as the accompaniment component to realize accurate transmission of the accompaniment signal.

Description

Technical Field

[0001] The present invention relates to the technical field of accompaniment extraction, and in particular to an accompaniment extraction method and system based on modal decomposition. Background Art

[0002] Accompaniment extraction refers to removing the human voice from the sound signal, separating the accompaniment part, and obtaining a sound signal with only the music background. Accompaniment extraction has a wide range of applications in many fields, such as: in the process of music production, extracting the accompaniment signal based on the recorded sound signal in order to remix and create a new version of music; extracting the accompaniment signal for the user's accompaniment; repairing the sound signal based on the extracted accompaniment signal, removing noise, and improving the quality of the accompaniment signal. Accompaniment extraction has brought convenience to many fields such as music creation, education, and entertainment.

[0003] Accompaniment extraction methods are mainly divided into methods based on spectrum analysis and methods based on sound signal processing. Among them, the method based on spectrum analysis distinguishes between human voice and accompaniment by observing the spectrum graph; the method based on audio signal processing separates human voice and accompaniment through filtering and noise reduction. These two methods will introduce sound quality loss in the process of extracting accompaniment, affecting the musical effect of the final accompaniment signal; for complex sound signals, due to the overlap of spectral features of different instruments and sounds, it is difficult to extract accompaniment signals through simple spectrum analysis or filtering methods; at the same time, accompaniment extraction involves copyright issues of sound signals, and it is necessary to ensure the authenticity and integrity of the accompaniment signal. Summary of the invention

[0004] In view of this, the present invention first performs signal processing on the collected multi-source sound signals, and adopts the modal decomposition method to accurately separate the main component signals of the multi-source sound signal array; then, based on the frequency spectrum of the accompaniment and the human voice, the accompaniment component in the main component signal is identified; finally, an error correction signal is added to the main component signal marked as the accompaniment component to achieve accurate and complete transmission of the accompaniment signal.

[0005] The technical solution proposed by the present invention is an accompaniment extraction method based on modal decomposition, comprising the following steps: S1: Use microphone array to collect multi-source sound signals; S2: Based on the multi-source acoustic signals, a multi-source acoustic signal array is constructed; the multi-source acoustic signal array is modally decomposed using the principal component analysis method to obtain eigenvalues; S3: solving the principal component signal corresponding to the eigenvalue, and identifying the principal component signal to obtain the principal component signal marked as the accompaniment component; S4: converting the main component signal marked as the accompaniment component into a frequency domain signal, and adding an error correction signal to obtain an updated frequency domain signal; S5: performing inverse Fourier transform on the updated frequency domain signal to obtain an updated principal component signal; combining the updated principal component signals to obtain an accompaniment signal.

[0006] Optionally, in step S1, using a microphone array to collect multi-source sound signals includes: No. The acoustic signal collected by the microphone is: ; in, represents the time variable, Indicates The time-varying sound signal collected by the microphones Indicates The microphones collect data with time delay The sound source signal, Indicates The noise signal collected by the microphone varies with time. Indicates The time delay of the microphone collecting the sound signal; ; in, Indicates the location of the sound source, Indicates The location of the microphones, Indicates the propagation speed of sound signal; The multi-source sound signal collected by the microphone array is expressed as: ; in, Represents multi-source sound signals, Indicates the number of microphones in the microphone array.

[0007] Optionally, in step S2, based on the multi-source acoustic signals, a multi-source acoustic signal array is constructed; and a principal component analysis method is used to perform modal decomposition on the multi-source acoustic signal array to obtain eigenvalues, including: S21: Construct a multi-source sound signal array based on the multi-source sound signals collected by the microphone array: ; in, represents a multi-source acoustic signal array, represents the microphone array acquisition time, Indicates the maximum value of the time series collected by the microphone array; Multi-source acoustic signal array Perform a de-averaging operation to obtain an updated multi-source sound signal array : ; in, Represents a multi-source acoustic signal array The index is Elements of Represents a multi-source acoustic signal array The mean of ; Represents the updated multi-source acoustic signal array The index is Elements of S22: Updated multi-source acoustic signal array , calculate the covariance matrix: ; in, represents the covariance matrix, express The transposed matrix of S23: Pair covariance matrix Perform eigendecomposition to obtain eigenvalues ​​and eigenvectors: ; in, represents the eigenvalue, Represents eigenvalue The corresponding feature vector.

[0008] Optionally, in step S3, solving the principal component signal corresponding to the eigenvalue and identifying the principal component signal to obtain the principal component signal marked as the accompaniment component includes: S31: Sort the eigenvalues ​​obtained in step S23 in ascending order to obtain an eigenvalue sequence: ; in, represents the eigenvalue sequence, represents the total number of eigenvalues ​​obtained in step S23; Define the intermediate variables of the eigenvalue ,make ; S32: Calculate the intermediate variables of eigenvalues The corresponding principal component signal: ; in, Indicates the intermediate variable of the eigenvalue The corresponding eigenvector, Indicates the intermediate variable of the eigenvalue The corresponding principal component signal; S33: Judgment Whether it is an accompaniment component; like Belongs to the accompaniment component, the main component signal Marked as accompaniment component; like If it does not belong to the accompaniment component, no operation will be performed; S34: Judgment Is it 2? if Not 2, update ,make , jump to step S32; if If it is 2, it will not be executed anymore; Get all the principal component signals marked as accompaniment components.

[0009] Optionally, in step S4, the main component signal marked as the accompaniment component is converted into a frequency domain signal, and an error correction signal is added to obtain an updated frequency domain signal, including: S41: Perform discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal: ; in, Represents the main component signal marked as the accompaniment component The discrete values, is the main component signal marked as the accompaniment component The total number of sampling points, is the main component signal marked as the accompaniment component The index value sampled in the time domain, is the index value of the frequency domain signal, Represents the main component signal marked as the accompaniment component The frequency domain signal, represents exponential notation, Represents the imaginary number symbol, Represents the symbol for pi; S42: Add the error correction signal to obtain an updated frequency domain signal: ; in, represents the updated frequency domain signal, Represents the error correction signal strength scaling factor, which is used to control the impact of the error correction signal on the frequency domain signal. Represents binary error correction signal information, Represents the error correction signal added to the frequency domain signal; At the same time, the error correction signal Satisfy the constraints: ; That is, the error correction signals added to all frequency domain signals cancel each other out; Optionally, in step S5, the updated frequency domain signal is subjected to inverse Fourier transform to obtain an updated principal component signal; and the updated principal component signal is combined to obtain an accompaniment signal, including: Perform an inverse Fourier transform on the updated frequency domain signal to obtain the updated principal component signal: ; in, represents the updated principal component signal obtained after inverse Fourier transform, Express Perform an inverse Fourier transform; All updated principal component signals are combined to obtain an accompaniment signal containing an error correction signal.

[0010] The present invention also provides an accompaniment extraction system based on modal decomposition, comprising: Multi-source sound signal acquisition module: uses a microphone to collect sound signals, and uses a microphone array to collect multi-source sound signals; Modal decomposition module: construct a multi-source acoustic signal array, perform a de-meaning operation on the multi-source acoustic signal array to obtain an updated multi-source acoustic signal array; calculate the covariance matrix of the updated multi-source acoustic signal array; perform eigendecomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; The main component signal recognition module: sorts the eigenvalues ​​to obtain the eigenvalue sequence; defines the eigenvalue intermediate variables, calculates the main component signals corresponding to the eigenvalue intermediate variables, determines whether the main component signals belong to the accompaniment components, marks the main component signals as the accompaniment components, and obtains all the main component signals marked as the accompaniment components; Error correction signal module: performing discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal; adding the error correction signal to obtain an updated frequency domain signal; Accompaniment signal extraction module: performs inverse Fourier transform on the updated frequency domain signal to obtain an updated main component signal; combines all updated main component signals to obtain an accompaniment signal containing an error correction signal.

[0011] Beneficial effects: The present invention constructs a multi-source sound signal array based on multi-source sound signals collected by a microphone array, and further adopts a modal decomposition method to obtain eigenvalues ​​and eigenvectors related to the multi-source sound signals; based on the characteristics of the accompaniment signal, a suitable eigenvalue is selected from the eigenvalues ​​related to the multi-source sound signal to accurately separate the main component signal of the multi-source sound signal array; then, based on the frequency spectrum of the accompaniment and the human voice, the accompaniment component in the main component signal is accurately identified; after converting the main component signal marked as the accompaniment component into a frequency domain signal, an error correction signal is added to ensure that the error correction signal does not interfere with the auditory perception of the human being; at the same time, the error correction signal satisfies the constraint condition to ensure that all the main component signals marked as the accompaniment component can be accurately and completely transmitted, and the obtained accompaniment signal is complete and error-free. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic flow chart of an accompaniment extraction method based on modal decomposition provided by an embodiment of the present invention.

[0013] Figure 2 This is a multi-source sound signal graph obtained in step S1 according to an embodiment of the present invention.

[0014] Figure 3 This is an accompaniment signal diagram obtained in step S5 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0016] Embodiment 1: An accompaniment extraction method based on modal decomposition, such as Figure 1 As shown, the following steps are included: S1: Using microphone array to collect multi-source sound signals: No. The acoustic signal collected by the microphone is: ; in, represents the time variable, Indicates The time-varying sound signal collected by the microphones Indicates The microphones collect data with time delay The sound source signal, Indicates The noise signal collected by the microphone varies with time. Indicates The time delay of the microphone collecting the sound signal; ; in, Indicates the location of the sound source, Indicates The location of the microphones, Indicates the propagation speed of sound signal; The multi-source sound signal collected by the microphone array is expressed as: ; in, Represents multi-source sound signals, Indicates the number of microphones in the microphone array.

[0017] S2: Based on the multi-source acoustic signals, a multi-source acoustic signal array is constructed; the principal component analysis method is used to perform modal decomposition on the multi-source acoustic signal array to obtain the eigenvalues: S21: Construct a multi-source sound signal array based on the multi-source sound signals collected by the microphone array: ; in, represents a multi-source acoustic signal array, represents the microphone array acquisition time, Indicates the maximum value of the time series collected by the microphone array; Multi-source acoustic signal array Perform a de-averaging operation to obtain an updated multi-source sound signal array : ; in, Represents a multi-source acoustic signal array The index is Elements of Represents a multi-source acoustic signal array The mean of ; Represents the updated multi-source acoustic signal array The index is Elements of S22: Updated multi-source acoustic signal array , calculate the covariance matrix: ; in, represents the covariance matrix, express The transposed matrix of S23: Pair covariance matrix Perform eigendecomposition to obtain eigenvalues ​​and eigenvectors: ; in, represents the eigenvalue, Represents eigenvalue The corresponding feature vector.

[0018] S3: Solve the principal component signal corresponding to the eigenvalue, identify the principal component signal, and obtain the principal component signal marked as the accompaniment component: S31: Sort the eigenvalues ​​obtained in step S23 in ascending order to obtain an eigenvalue sequence: ; in, represents the eigenvalue sequence, represents the total number of eigenvalues ​​obtained in step S23; Define the intermediate variables of the eigenvalue ,make ; S32: Calculate the intermediate variables of eigenvalues The corresponding principal component signal: ; in, Indicates the intermediate variable of the eigenvalue The corresponding eigenvector, Indicates the intermediate variable of the eigenvalue The corresponding principal component signal; S33: Judgment Whether it is an accompaniment component; like Belongs to the accompaniment component, the main component signal Marked as accompaniment component; like If it does not belong to the accompaniment component, no operation will be performed; S34: Judgment Is it 2? if Not 2, update ,make , jump to step S32; if If it is 2, it will not be executed anymore; Get all the principal component signals marked as accompaniment components.

[0019] In the embodiment of the present invention, in step S31, the characteristic value intermediate variable is defined , and initialize the eigenvalue intermediate variables , that is, the eigenvalue is selected from the second largest eigenvalue First, the main signal of human voice singing in the multi-source sound signal array is removed; the eigenvalues ​​are traversed in descending order, and the principal component signals corresponding to the eigenvalues ​​are solved in turn, and the accompaniment components exist in these principal component signals; In the embodiment of the present invention, step S33 first determines Whether it belongs to the accompaniment component, whether the main component signal belongs to the accompaniment component can be judged based on the frequency interval of the main component signal: if the frequency interval of the main component signal contains a part that cannot be covered by the frequency interval of the human voice, then the main component signal is considered to belong to the accompaniment component; it can be based on the characteristics of the main component signal: use the trained machine learning model to identify the main component signal to determine whether it belongs to the accompaniment component; it can be based on the significant characteristics of the main component signal: such as amplitude and phase changes.

[0020] S4: Convert the main component signal marked as the accompaniment component into a frequency domain signal, and add the error correction signal to obtain an updated frequency domain signal: S41: Perform discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal: ; in, Represents the main component signal marked as the accompaniment component The discrete values, is the main component signal marked as the accompaniment component The total number of sampling points, is the main component signal marked as the accompaniment component The index value sampled in the time domain, is the index value of the frequency domain signal, Represents the main component signal marked as the accompaniment component The frequency domain signal, represents the exponential symbol, Represents the imaginary number symbol, Represents the symbol for pi; S42: Add the error correction signal to obtain an updated frequency domain signal: ; in, represents the updated frequency domain signal, Represents the error correction signal strength scaling factor, which is used to control the impact of the error correction signal on the frequency domain signal. Represents binary error correction signal information, Represents the error correction signal added to the frequency domain signal; At the same time, the error correction signal Satisfy the constraints: ; That is, the error correction signals added to all frequency domain signals cancel each other out.

[0021] In the embodiment of the present invention, in step S42, a frequency domain signal having a high frequency component is selected. , modify the frequency domain signal The human ear is not sensitive to high-frequency components and is sensitive to frequency domain signals with high-frequency components. Modification is performed to reduce the sensitivity of the human ear to signal modification.

[0022] S5: Perform inverse Fourier transform on the updated frequency domain signal to obtain an updated main component signal; merge the updated main component signals to obtain an accompaniment signal: Perform an inverse Fourier transform on the updated frequency domain signal to obtain the updated principal component signal: ; in, represents the updated principal component signal obtained after inverse Fourier transform, Express Perform an inverse Fourier transform; All updated principal component signals are combined to obtain an accompaniment signal containing an error correction signal.

[0023] In the embodiment of the present invention, if an error occurs in the updated main component signal during the transmission and merging process, after all the main component signals are merged, the frequency domain error correction code part will not be 0, and the accompaniment signal obtained in step S5 will suffer from sound quality loss, and the human ear can feel the change of the accompaniment signal.

[0024] In the embodiment of the present invention, taking the song "Two Tigers" as an example, Figure 2 is the multi-source sound signal obtained in step S1, Figure 3 is the accompaniment signal diagram obtained in step S5; Figure 2 , Figure 3 The largest main component signal, the human voice, was removed. Figure 3 The signal amplitude is reduced; at the same time, Figure 3 After removing some noise, Figure 2 , Figure 3 The signal amplitude is more concentrated.

[0025] Embodiment 2: The present invention also provides an accompaniment extraction system based on modal decomposition, comprising the following five modules: Multi-source sound signal acquisition module: uses a microphone to collect sound signals, and uses a microphone array to collect multi-source sound signals; Modal decomposition module: construct a multi-source acoustic signal array, perform a de-meaning operation on the multi-source acoustic signal array to obtain an updated multi-source acoustic signal array; calculate the covariance matrix of the updated multi-source acoustic signal array; perform eigendecomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; The main component signal recognition module: sorts the eigenvalues ​​to obtain the eigenvalue sequence; defines the eigenvalue intermediate variables, calculates the main component signals corresponding to the eigenvalue intermediate variables, determines whether the main component signals belong to the accompaniment components, marks the main component signals as the accompaniment components, and obtains all the main component signals marked as the accompaniment components; Error correction signal module: performing discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal; adding the error correction signal to obtain an updated frequency domain signal; Accompaniment signal extraction module: performs inverse Fourier transform on the updated frequency domain signal to obtain an updated main component signal; combines all updated main component signals to obtain an accompaniment signal containing an error correction signal.

[0026] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0027] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0028] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An accompaniment extraction method based on modal decomposition, characterized in that: The method comprises: S1: Use microphone array to collect multi-source sound signals; S2: Based on the multi-source acoustic signals, a multi-source acoustic signal array is constructed; the multi-source acoustic signal array is modally decomposed using the principal component analysis method to obtain eigenvalues; S3: solving the principal component signal corresponding to the eigenvalue, and identifying the principal component signal to obtain the principal component signal marked as the accompaniment component; S4: converting the main component signal marked as the accompaniment component into a frequency domain signal, and adding an error correction signal to obtain an updated frequency domain signal; S5: performing inverse Fourier transform on the updated frequency domain signal to obtain an updated principal component signal; combining the updated principal component signals to obtain an accompaniment signal.

2. The accompaniment extraction method based on modal decomposition according to claim 1, characterized in that: The step S1 comprises: No. The acoustic signal collected by the microphone is: ； in, represents the time variable, Indicates The time-varying sound signal collected by the microphones Indicates The microphones collect data with time delay The sound source signal, Indicates The noise signal collected by the microphone varies with time. Indicates The time delay of the microphone collecting the sound signal; ； in, Indicates the location of the sound source, Indicates The location of the microphones, Indicates the propagation speed of sound signal; The multi-source sound signal collected by the microphone array is expressed as: ； in, Represents multi-source sound signals, Indicates the number of microphones in the microphone array.

3. The accompaniment extraction method based on modal decomposition according to claim 2, characterized in that: The step S2 comprises: S21: Construct a multi-source sound signal array based on the multi-source sound signals collected by the microphone array: ； in, represents a multi-source acoustic signal array, represents the microphone array acquisition time, Indicates the maximum value of the time series collected by the microphone array; Multi-source acoustic signal array Perform a de-averaging operation to obtain an updated multi-source sound signal array : ； in, Represents a multi-source acoustic signal array The index is Elements of Represents a multi-source acoustic signal array The mean of ; Represents the updated multi-source acoustic signal array The index is Elements of S22: Updated multi-source acoustic signal array , calculate the covariance matrix: ； in, represents the covariance matrix, express The transposed matrix of S23: Pair covariance matrix Perform eigendecomposition to obtain eigenvalues ​​and eigenvectors: ； in, represents the eigenvalue, Represents eigenvalue The corresponding feature vector.

4. The accompaniment extraction method based on modal decomposition according to claim 3 is characterized in that: The step S3 comprises: S31: Sort the eigenvalues ​​obtained in step S23 in ascending order to obtain an eigenvalue sequence: ； in, represents the eigenvalue sequence, represents the total number of eigenvalues ​​obtained in step S23; Define the intermediate variables of the eigenvalue ,make ; S32: Calculate the intermediate variables of eigenvalues The corresponding principal component signal: ； in, Indicates the intermediate variable of the eigenvalue The corresponding eigenvector, Indicates the intermediate variable of the eigenvalue The corresponding principal component signal; S33: Judgment Whether it is an accompaniment component; like Belongs to the accompaniment component, the main component signal Marked as accompaniment component; like If it does not belong to the accompaniment component, no operation will be performed; S34: Judgment Is it 2? if Not 2, update ,make , jump to step S32; if If it is 2, it will not be executed anymore; Get all the principal component signals marked as accompaniment components.

5. The accompaniment extraction method based on modal decomposition according to claim 4, characterized in that: The step S4 comprises: S41: Perform discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal: ； in, Represents the main component signal marked as the accompaniment component The discrete values, is the main component signal marked as the accompaniment component The total number of sampling points, is the principal component signal marked as the accompaniment component The index value sampled in the time domain, is the index value of the frequency domain signal, Represents the main component signal marked as the accompaniment component The frequency domain signal, represents exponential notation, Represents the imaginary number symbol, Represents the symbol for pi; S42: Add the error correction signal to obtain an updated frequency domain signal: ； in, represents the updated frequency domain signal, Represents the error correction signal strength scaling factor, which is used to control the impact of the error correction signal on the frequency domain signal. Represents binary error correction signal information, Represents the error correction signal added to the frequency domain signal; At the same time, the error correction signal Satisfy the constraints: ； That is, the error correction signals added to all frequency domain signals cancel each other out.

6. The accompaniment extraction method based on modal decomposition according to claim 5, characterized in that: The step S5 comprises: Perform an inverse Fourier transform on the updated frequency domain signal to obtain the updated principal component signal: ； in, represents the updated principal component signal obtained after inverse Fourier transform, Express Perform an inverse Fourier transform; All updated principal component signals are combined to obtain an accompaniment signal containing an error correction signal.

7. An accompaniment extraction system based on modal decomposition, characterized in that: include: Multi-source sound signal acquisition module: uses a microphone to collect sound signals, and uses a microphone array to collect multi-source sound signals; Mode decomposition module: construct a multi-source sound signal array, perform a de-meaning operation on the multi-source sound signal array to obtain an updated multi-source sound signal array; calculate the covariance matrix of the updated multi-source sound signal array; Perform eigendecomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; Principal component signal recognition module: sort the eigenvalues ​​to obtain the eigenvalue sequence; definition Eigenvalue intermediate variable, calculating the principal component signal corresponding to the eigenvalue intermediate variable, judging whether the principal component signal belongs to the accompaniment component, marking the principal component signal as the accompaniment component, and obtaining all principal component signals marked as the accompaniment component; Error correction signal module: performing discrete Fourier transform on the main component signal marked as the accompaniment component to obtain a frequency domain signal; adding the error correction signal to obtain an updated frequency domain signal; Accompaniment signal extraction module: performing inverse Fourier transform on the updated frequency domain signal to obtain an updated main component signal; merging all updated main component signals to obtain an accompaniment signal containing an error correction signal; To realize an accompaniment extraction method based on modal decomposition as described in any one of claims 1-6.

Images