A live reading noise processing method, system, device and medium

Abstract

The application relates to a live reading noise processing method and system. The method is first divided into two routes. The first route is to extract an audio graph of an audio segment, and then convert the audio graph into a graph structure. The second route is to extract a two-dimensional time sequence graph of the audio segment, and then convert the two-dimensional time sequence graph into a graph structure. Then, based on a graph attention mechanism, the audio graph structure and the time sequence graph structure in different domains are fused to sufficiently extract information distribution in the two domains. Then, the time sequence graph structure and the first graph structure are fused to obtain a second graph structure in which the time sequence graph structure is enhanced based on the first graph structure. The audio graph structure and the first graph structure are fused to obtain a third graph structure in which the audio graph structure is enhanced based on the first graph structure. The ability to extract more representative graph features from the second graph structure and the third graph structure is improved. Finally, a live reading audio segment after noise removal is generated based on the second graph structure and the third graph structure.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of audio denoising, and in particular to a live book reading noise processing method, system, device and medium. BACKGROUND

[0002] The main purpose of live audio denoising is to improve the clarity and audibility of the audio. Background noise will interfere with the clarity of the audio and affect the understanding and enjoyment of the audience. By removing these noises, the audio can be made more pure and the information conveyed can be more clear.

[0003] Traditional live audio denoising uses PCA (Principal Component Analysis) / ICA (Independent Component Analysis) based noise separation, but this scheme is computationally complex and relies on the signal sparsity assumption. With the development of neural network technology, traditional methods are gradually replaced by convolutional neural network based audio denoising technology schemes, but due to the changing noise characteristics of live broadcasts, the loss of topological information caused by the local convolution window cutting of convolutional neural networks, and the poor performance of convolutional neural networks in capturing the spatial and temporal dependencies of noise in complex systems, the noise filtering effect in dynamic environments is easily reduced. SUMMARY

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The main purpose of the embodiments of the present disclosure is to propose a live book reading noise processing method and system, which can improve the clarity and intelligibility of audio signals under complex environmental noise interference.

[0006] The first aspect of the embodiments of the present application proposes a live book reading noise processing method, which comprises:

[0007] extracting a live book reading audio segment;

[0008] extracting an audio graph from the live book reading audio segment, and generating an audio graph structure according to the audio graph;

[0009] extracting a two-dimensional time sequence graph from the live book reading audio segment, and generating a time sequence graph structure according to the two-dimensional time sequence graph;

[0010] input the audio graph structure and the time sequence graph structure into a preset graph convolution network, to obtain a first graph structure by fusing the audio graph structure and the time sequence graph structure of different domains based on a graph attention mechanism of the graph convolution network, fuse the time sequence graph structure and the first graph structure based on the graph attention mechanism to obtain a second graph structure, fuse the audio graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure, and generate the live reading audio segment after noise reduction based on the second graph structure and the third graph structure.

[0011] In some embodiments, the audio graph is extracted from the live reading audio segment, and an audio graph structure is generated based on the audio graph, including:

[0012] performing short-time Fourier transform on the live reading audio segment to obtain the audio graph;

[0013] detecting time-frequency points in the audio graph, taking the time-frequency points as nodes of the audio graph structure, and taking the amplitude values of the time-frequency points as node features;

[0014] building edges between the time-frequency points based on time-frequency neighborhoods.

[0015] In some embodiments, a two-dimensional time sequence graph is extracted from the live reading audio segment, and a time sequence graph structure is generated based on the two-dimensional time sequence graph, including:

[0016] normalizing and scaling the live reading audio segment;

[0017] mapping the normalized and scaled live reading audio segment to polar coordinates;

[0018] converting the live reading audio segment mapped to polar coordinates into a two-dimensional time sequence graph based on Gram angle and field transformation;

[0019] detecting pixel points in the two-dimensional time sequence graph as nodes of the time sequence graph structure;

[0020] building edges between the pixel points according to the pixel spatial relationship in the two-dimensional time sequence graph.

[0021] In some embodiments, the audio graph structure and the time sequence graph structure of different domains are fused based on a graph attention mechanism to obtain a first graph structure, including:

[0022] fusing the audio graph structure and the time sequence graph structure of different domains based on a graph attention network to obtain a first graph structure.

[0023] In some embodiments, the time sequence graph structure and the first graph structure are fused based on a graph attention mechanism to obtain a second graph structure, including:

[0024] calculating a first mapping graph structure of the timing graph structure;

[0025] normalizing the first mapping graph structure and the first graph structure after element-wise multiplication, to obtain a first weight;

[0026] multiplying the first weight and the first graph structure, to obtain a first intermediate graph structure;

[0027] fusing the first intermediate graph structure and the first graph structure, to obtain the second graph structure.

[0028] In some embodiments, the fusing the audio-frequency graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure comprises:

[0029] calculating a second mapping graph structure of the audio-frequency graph structure;

[0030] normalizing the second mapping graph structure and the first graph structure after element-wise multiplication, to obtain a second weight;

[0031] multiplying the second weight and the first graph structure, to obtain a second intermediate graph structure;

[0032] fusing the second intermediate graph structure and the first graph structure, to obtain the third graph structure.

[0033] In some embodiments, the generating the denoised live reading audio segment based on the second graph structure and the third graph structure comprises:

[0034] extracting a first graph feature in the second graph structure based on a first branch of a graph convolution network;

[0035] extracting a second graph feature in the third graph structure based on a second branch of the graph convolution network;

[0036] concatenating the first graph feature and the second graph feature, to obtain a third graph feature;

[0037] inputting the third graph feature into a multi-layer perception machine, to obtain an output feature;

[0038] reconstructing the output feature into the denoised live reading audio segment.

[0039] A second aspect of the embodiments of the present application proposes a live reading noise processing system, the system comprising:

[0040] an audio acquisition module, configured to extract a live reading audio segment;

[0041] An audio graph structure generation module is configured to extract an audio graph from the live reading audio segment, and generate an audio graph structure according to the audio graph;

[0042] A time sequence graph structure generation module is configured to extract a two-dimensional time sequence graph from the live reading audio segment, and generate a time sequence graph structure according to the two-dimensional time sequence graph;

[0043] An audio denoising module is configured to input the audio graph structure and the time sequence graph structure into a preset graph convolutional network, to obtain a first graph structure by fusing the audio graph structure and the time sequence graph structure in different domains based on a graph attention mechanism of the graph convolutional network, fuse the time sequence graph structure and the first graph structure based on the graph attention mechanism to obtain a second graph structure, fuse the audio graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure, and generate the live reading audio segment after denoising based on the second graph structure and the third graph structure.

[0044] A third aspect of the embodiments of the present application provides an electronic device, including at least one controller and a memory connected to the controller for communication; the memory stores instructions executable by the at least one controller, and the instructions are executed by the at least one controller to make the at least one controller execute the live reading noise processing method.

[0045] A fourth aspect of the embodiments of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed to implement the live reading noise processing method.

[0046] The live reading noise processing method provided by the embodiments has the following advantages:

[0047] The method first divides into two routes, wherein the first route is to extract an audio spectrogram of the audio segment, the audio spectrogram shows the energy distribution of the audio as a time sequence signal at different times and frequencies, and in order to use graph neural processing, the audio spectrogram is further converted into a graph structure; the second route is to extract a two-dimensional time sequence graph of the audio segment, the two-dimensional time sequence graph can retain the global dependency relationship of the time sequence signal, and in order to use graph neural processing, the two-dimensional time sequence graph is further converted into a graph structure; then a graph convolutional network is used to fuse the audio spectrogram structure and the time sequence graph structure in different domains based on a graph attention mechanism, so that the information distribution in the two domains can be fully extracted, then the time sequence graph structure and the first graph structure are fused based on the graph attention mechanism, to obtain a second graph structure in which the time sequence graph structure is enhanced based on the first graph structure, the audio spectrogram structure and the first graph structure are fused based on the graph attention mechanism, to obtain a third graph structure in which the audio spectrogram structure is enhanced based on the first graph structure, and the ability to extract more representative graph features from the second graph structure and the third graph structure is improved; finally, a live reading audio segment after noise reduction is generated based on the second graph structure and the third graph structure. The joint use of the audio spectrogram and the two-dimensional time sequence graph fully excavates the frequency domain characteristics and time characteristics of the audio signal, the double-branch structure of the graph convolutional network enhances the comprehensiveness of feature extraction, and the feature fusion strategy of the attention mechanism further optimizes the noise suppression effect. Therefore, the intelligibility and clarity of the audio signal are significantly improved under complex environmental noise interference.

[0048] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or related technical description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a flow diagram of a live reading noise processing method provided by an embodiment of the present application;

[0051] Fig. 2(a) is a schematic diagram of a live reading audio sequence provided by an embodiment of the present application;

[0052] Fig. 2(b) is a schematic diagram of a two-dimensional time sequence graph corresponding to Fig. 2(a);

[0053] Figure 3 is a model block diagram of a live reading noise processing method provided by an embodiment of the present application;

[0054] Figure 4 is a structural schematic diagram of a live reading noise processing system provided by an embodiment of the present application.

[0055] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0057] In the description of the present application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0058] In the description of the present application, it should be understood that the position description, such as the position or location relationship indicated by up, down, etc., is based on the position or location relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present application.

[0059] The main purpose of live audio denoising is to improve the clarity and audibility of the audio. Background noise will interfere with the clarity of the audio and affect the understanding and enjoyment of the audience. By removing these noises, the audio can be more pure and the information conveyed can be more clear.

[0060] Traditional live audio denoising uses PCA (Principal Component Analysis) / ICA (Independent Component Analysis) based noise separation, but this scheme is computationally complex and depends on the sparsity assumption of the signal. With the development of neural network technology, traditional methods are gradually replaced by convolutional neural network audio denoising technology schemes, but due to the constantly changing noise characteristics of live broadcasts, and the loss of topological information caused by the local convolution window cutting of convolutional neural networks, in addition, the convolutional neural network has poor performance in capturing the spatio-temporal dependence of noise in complex systems, which can easily reduce the noise filtering effect in dynamic environments.

[0061] As Figures 1 to 3 , one embodiment of the present application provides a live reading noise processing method, the method comprising the following steps S100 to S400:

[0062] Step S100, extracting a live reading audio segment.

[0063] At the present stage, for any live broadcast software, during live broadcast, a delay and a segment are usually set, that is, there is a delay of several seconds to several minutes between the time when the host speaks and the time when the audience hears the audio, and the live broadcast audio segment can be collected within the delay time.

[0064] Step S200 extracts a sound frequency graph from the live broadcast audio segment, and generates a sound frequency graph structure according to the sound frequency graph.

[0065] In the embodiment, the sound frequency graph refers to a frequency spectrum graph generated by short-time Fourier transform, which can be implemented by frame division and windowing, frequency domain transformation extraction, and is used to represent the frequency domain distribution characteristics of the audio signal and can reveal more physical properties in the live broadcast audio segment.

[0066] In order to enable the graph convolutional neural network to extract features from the sound frequency graph, the sound frequency graph needs to be generated into a graph structure.

[0067] Step S300 extracts a two-dimensional time sequence graph from the live broadcast audio segment, and generates a time sequence graph structure according to the two-dimensional time sequence graph.

[0068] The two-dimensional time sequence graph refers to a graph generated by Gram angle and field transformation after the audio signal is mapped to a polar coordinate system, for example, by using a normalized scaling and polar coordinate mapping method, which is used to retain the time sequence dynamic information of the audio signal and can retain the global dependency relationship of the time sequence.

[0069] In order to enable the graph convolutional neural network to extract features from the two-dimensional time sequence graph, the two-dimensional time sequence graph needs to be converted into a graph structure.

[0070] Step S400 inputs the sound frequency graph structure and the time sequence graph structure into a preset graph convolutional network, so as to obtain a first graph structure by fusing the sound frequency graph structure and the time sequence graph structure in different domains based on a graph attention mechanism of the graph convolutional network, fuse the time sequence graph structure and the first graph structure based on the graph attention mechanism, obtain a second graph structure, fuse the sound frequency graph structure and the first graph structure based on the graph attention mechanism, obtain a third graph structure, and generate a live broadcast audio segment after noise reduction based on the second graph structure and the third graph structure.

[0071] The method first divides into two routes, wherein the first route is to extract an audio spectrogram of the audio segment, the audio spectrogram shows the energy distribution of the audio as a time sequence signal at different times and frequencies, and in order to use graph neural processing, the audio spectrogram is further converted into a graph structure; the second route is to extract a two-dimensional time sequence graph of the audio segment, the two-dimensional time sequence graph can retain the global dependency relationship of the time sequence signal, and in order to use graph neural processing, the two-dimensional time sequence graph is further converted into a graph structure; then a graph convolutional network is used to fuse the audio spectrogram structure and the time sequence graph structure in different domains based on a graph attention mechanism, so as to sufficiently extract the information distribution in the two domains, then the time sequence graph structure and the first graph structure are fused based on the graph attention mechanism, to obtain a second graph structure in which the time sequence graph structure is enhanced based on the first graph structure, the audio spectrogram structure and the first graph structure are fused based on the graph attention mechanism, to obtain a third graph structure in which the audio spectrogram structure is enhanced based on the first graph structure, and the ability to extract more representative graph features from the second graph structure and the third graph structure is improved; finally, a live reading audio segment after noise reduction is generated based on the second graph structure and the third graph structure. The joint use of the audio spectrogram and the two-dimensional time sequence graph sufficiently excavates the frequency domain characteristics and time characteristics of the audio signal, the double-branch structure of the graph convolutional network enhances the comprehensiveness of feature extraction, and the feature fusion strategy of the attention mechanism further optimizes the noise suppression effect. Thus, the intelligibility and clarity of the audio signal are significantly improved under complex environmental noise interference.

[0072] Further, the step S200 of extracting an audio spectrogram from the live reading audio segment, and generating an audio spectrogram structure according to the audio spectrogram, includes steps S210 to S230:

[0073] Step S210, performing short-time Fourier transform on the live reading audio segment to obtain an audio spectrogram.

[0074] The short-time Fourier transform refers to dividing the live reading audio segment into short-time windows for Fourier transform, which can be specifically implemented by using a Hamming window function, by dividing the audio signal into an overlapping frame sequence, calculating the spectrum information of each frame to obtain time-frequency domain features.

[0075] Step S220, detecting time-frequency points in the audio spectrogram, taking the time-frequency points as nodes of the audio spectrogram structure, and taking the amplitude values of the time-frequency points as node features.

[0076] Step S230, building edges between the time-frequency points based on time-frequency neighborhoods.

[0077] The feature of each node is composed of the amplitude value of the corresponding key point. By calculating the correlation degree between the node features, for example, using cosine similarity to measure the similarity of two node features, if the similarity exceeds the preset threshold, an edge connection is established. The finally formed graph structure can retain the local features and global correlation of the audio graph, and provide input data containing spatial and temporal dependence for the subsequent graph convolution network.

[0078] Further, the two-dimensional time sequence graph is extracted from the live reading audio segment in step S300, and the time sequence graph structure is generated according to the two-dimensional time sequence graph, including steps S310 to S330:

[0079] In step S310, the live reading audio segment is normalized and scaled.

[0080] The normalization and scaling refers to adjusting the amplitude of the audio signal to a unified range, which can be implemented by linear scaling or quantile normalization method, for eliminating the amplitude difference between different audio inputs and ensuring the stability of subsequent processing.

[0081] In step S320, the normalized and scaled live reading audio segment is mapped to polar coordinates.

[0082] The polar coordinate mapping refers to converting the time sequence data into angle and radius parameters in the polar coordinate system, which can be implemented by mapping the time step to the angle and the signal value to the radius, for converting the one-dimensional time sequence data into geometric distribution in two-dimensional space, and facilitating the capture of the correlation in the time dimension.

[0083] In step S330, the live reading audio segment mapped to the polar coordinates is converted into a two-dimensional time sequence graph based on the Gram angle and field transformation.

[0084] The Gram angle and field transformation refers to generating a two-dimensional image through a Gram matrix, which can be implemented by sum Gram angle field or difference Gram angle field method, for converting the time sequence data in polar coordinates into a two-dimensional image with spatial structure, and retaining the dynamic change characteristics in the time sequence.

[0085] Specifically, the live reading audio segment is scaled by normalization, and its amplitude range is limited in a fixed interval, such as [-1, 1] or [0, 1], so as to avoid deviation caused by amplitude difference in subsequent processing. Subsequently, the normalized audio signal is mapped into a polar coordinate system, for example, the signal value x(t) corresponding to the time step t is converted into an angle θ(t) = t·(2π / N) and a radius r(t) = x(t) + C, where N is the total number of time steps, and C is a constant offset. Through Gram angle and field transformation, the time series data in the polar coordinate system is converted into a Gram angle field image in the form of a two-dimensional matrix, where the matrix elements represent the angle difference or cosine similarity between different time points, thereby encoding the time dependence relationship into spatial structure information.

[0086] The method converts the time series data into a geometric structure in a two-dimensional space through polar coordinate mapping and Gram angle field transformation, so that the subsequent graph convolution network can directly model the time dependence by utilizing the spatial topological relationship, thereby enhancing the recognition ability of noise patterns, and further improving the modeling accuracy of complex noise features in a dynamic environment and the denoising effect.

[0087] Step S340, detecting a pixel point in the two-dimensional time series graph as a node of the time series graph structure;

[0088] Specifically, after the two-dimensional time series graph is generated through Gram angle and field transformation, it is converted into a matrix form (composed of pixel points). Each row or column in the matrix is extracted as a node of the graph structure, and the features of the nodes are obtained by analyzing the statistical characteristics or embedding representation of the elements in the row or column. The construction of edges is based on the position of the non-zero elements in the matrix, for example, if there are non-zero cross columns corresponding to the rows of two row nodes in the matrix, an edge is established to connect the two nodes. The finally generated time series graph structure can capture the global dependence relationship in the time series data through the graph convolution network.

[0089] The method can preserve the global topological information by converting the two-dimensional time series graph into a graph structure and constructing edges based on non-zero values. In addition, the node features are extracted based on the characteristics of the row or column elements, which avoids the feature loss problem caused by the signal sparsity assumption in traditional methods, thereby more effectively capturing the noise patterns in a complex environment.

[0090] Step S350, constructing edges between pixel points according to the spatial relationship of pixels in the two-dimensional time series graph.

[0091] Further, the step S400 of fusing the audio graph structure and the time series graph structure in different domains based on the graph attention mechanism to obtain the first graph structure includes:

[0092] Step S410, fusing the audio graph structure and the time series graph structure in different domains based on the graph attention network to obtain the first graph structure.

[0093] In some embodiments, the graph attention network (GAT) can fuse the graph structures of two different domains in the following ways:

[0094] 1) Cross-graph attention mechanism; realized by establishing attention connection between two graphs.

[0095] 2) Hierarchical fusion; process the two graphs respectively to obtain node representations using GAT; perform fusion operation in the middle layer or the last layer:

[0096] Further, the step S400 of fusing the time series graph structure and the first graph structure based on the graph attention mechanism to obtain a second graph structure includes the following steps S510 to S540:

[0097] Step S510, calculate the first mapping graph structure of the time series graph structure. The following are mathematical calculations using the graph structure matrix form.

[0098] In this step, because the time series graph structure and the first graph structure are in different domains, the time series graph structure can be mapped to the same domain space as the first graph structure (mapping can be based on the common matrix mapping method in the art, which can be based on the training and improvement of the model), and then the time series graph structure is enhanced based on the first graph structure.

[0099] Step S520, multiply the first mapping graph structure and the first graph structure element by element and then normalize to obtain the first weight.

[0100] In this step, the first mapping graph structure and the first graph structure are multiplied element by element, the multiplication result is activated, then a learnable matrix is multiplied, and finally normalized to obtain the first weight.

[0101] Step S530, multiply the first weight and the first graph structure to obtain the first intermediate graph structure.

[0102] In this embodiment, by multiplying the first weight and the first graph structure, the first graph structure is enhanced using the first weight to obtain the first intermediate graph structure.

[0103] Step S540, fuse the first intermediate graph structure and the first graph structure to obtain the second graph structure.

[0104] Finally, the first intermediate graph structure and the first graph structure are fused, and the fusion method can be: control the first intermediate graph structure and the first graph structure to keep the same dimension, then splice, and finally normalize to obtain the second graph structure.

[0105] This method fuses the audio graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure that enhances the audio graph structure features based on the first graph structure. This makes the third graph structure rich in representative noise features and can significantly improve the recognition ability of noise features.

[0106] Furthermore, in step S400, the audio graph structure and the first graph structure are fused based on the graph attention mechanism to obtain a third graph structure, which includes the following steps S610 to S640:

[0107] Step S610, calculating a second mapping structure of the audio graph structure;

[0108] Step S620: multiply the second mapping structure and the first mapping structure element by element and then normalize them to obtain a second weight;

[0109] Step S630: multiply the second weight and the first graph structure to obtain a second intermediate graph structure;

[0110] Step S640: Fusing the second intermediate graph structure and the first graph structure to obtain a third graph structure.

[0111] Similar to step S510 to step S540, detailed description is omitted here.

[0112] Furthermore, generating a denoised live book-reading audio segment based on the second graph structure and the third graph structure in step S400 includes the following steps S710 to S750:

[0113] Step S710: extracting first graph features from the second graph structure based on the first branch of the graph convolutional network.

[0114] Step S720: extract the second graph features in the third graph structure based on the second branch of the graph convolutional network.

[0115] Step S730: Concatenate the first image feature and the second image feature to obtain a third image feature.

[0116] Step S740: Input the third graph feature into a multi-layer perceptron to obtain an output feature.

[0117] Step S750: reconstruct the output features into a denoised live book-reading audio clip.

[0118] Among them, the multi-layer perceptron refers to a feedforward neural network composed of fully connected layers, which can be specifically implemented using a network structure containing multiple hidden layers. It is used to perform nonlinear transformation and dimensionality reduction on the fused third feature, thereby extracting high-level abstract features to generate a denoised graph structure.

[0119] The method cooperates the graph structure with the multi-layer perception machine to remove the background noise while completely retaining the fundamental frequency feature information of the voice, thereby improving the clarity and naturalness of real-time audio processing in a live scene.

[0120] The graph amplitude value of the audio graph output by the graph convolution network can be clean audio. Then the time-domain signal is synthesized based on an iterative reconstruction algorithm (such as Griffin-Lim).

[0121] Part of the training parameters:

[0122] The loss function can be: mean square error or multi-scale STFT loss.

[0123] Dataset: Use DNS Challenge or VoiceBank+DEMAND, etc. noisy / clean paired audio dataset.

[0124] As Figure 4 , one embodiment of the present application provides a live reading noise processing system, the system comprises:

[0125] The audio acquisition module 1100 is used to extract a live reading audio segment;

[0126] The audio graph structure generation module 1200 is used to extract an audio graph from the live reading audio segment, and generate an audio graph structure according to the audio graph;

[0127] The timing graph structure generation module 1300 is used to extract a two-dimensional timing graph from the live reading audio segment, and generate a timing graph structure according to the two-dimensional timing graph;

[0128] The audio denoising module 1400 is used to input the audio graph structure and the timing graph structure into a preset graph convolution network, to fuse the audio graph structure and the timing graph structure in different domains based on a graph attention mechanism of the graph convolution network, to obtain a first graph structure, to fuse the timing graph structure and the first graph structure based on the graph attention mechanism, to obtain a second graph structure, to fuse the audio graph structure and the first graph structure based on the graph attention mechanism, to obtain a third graph structure, and to generate a live reading audio segment after denoising based on the second graph structure and the third graph structure.

[0129] It should be noted that the live reading noise processing system embodiment and the live reading noise processing method embodiment described above are based on the same inventive concept, so the related content of the live reading noise processing method embodiment described above is also applicable to the live reading noise processing system embodiment, which will not be repeated here.

[0130] Firstly, two routes are divided, wherein the first route is to extract an audio spectrogram of the audio segment, the audio spectrogram shows the energy distribution of the audio as a time sequence signal at different times and frequencies, and the audio spectrogram is further converted into a graph structure for using graph neural processing; the second route is to extract a two-dimensional time sequence graph of the audio segment, the two-dimensional time sequence graph can retain the global dependency relationship of the time sequence signal, and the two-dimensional time sequence graph is further converted into a graph structure for using graph neural processing; then, a graph convolution network is used to fuse the audio spectrogram structure and the time sequence graph structure in different domains based on a graph attention mechanism, so that the information distribution in the two domains can be fully extracted, then the time sequence graph structure and the first graph structure are fused based on the graph attention mechanism, to obtain a second graph structure in which the time sequence graph structure is enhanced based on the first graph structure, the audio spectrogram structure and the first graph structure are fused based on the graph attention mechanism, to obtain a third graph structure in which the audio spectrogram structure is enhanced based on the first graph structure, and the ability to extract more representative graph features from the second graph structure and the third graph structure is improved; finally, the live reading audio segment after noise reduction is generated based on the second graph structure and the third graph structure. The joint use of the audio spectrogram and the two-dimensional time sequence graph fully excavates the frequency domain characteristics and time characteristics of the audio signal, the double-branch structure of the graph convolution network enhances the comprehensiveness of feature extraction, and the feature fusion strategy of the attention mechanism further optimizes the noise suppression effect. Therefore, under the interference of complex environmental noise, the intelligibility and clarity of the audio signal are significantly improved.

[0131] Reference Figure 5 The embodiments of the present application also provide an electronic device, and the electronic device comprises:

[0132] at least one memory;

[0133] at least one processor;

[0134] at least one program;

[0135] The program is stored in the memory, and the processor executes the at least one program to implement the live reading noise processing method described above.

[0136] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0137] The electronic device of the embodiments of the present application will be described in detail below.

[0138] The processor 1600 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0139] The memory 1700 can be implemented by a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the program codes are stored in the memory 1700 and are called and executed by the processor 1600.

[0140] The input / output interface 1800 is configured to realize information input and output.

[0141] The communication interface 1900 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0142] The bus 2000 is configured to transmit information between various components (for example, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device.

[0143] The processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are connected to each other by the bus 2000 to realize the communication connection between them in the device.

[0144] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and stores computer executable instructions. The computer executable instructions are configured to make a computer execute the live reading noise processing method.

[0145] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0148] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0149] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0150] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0152] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0154] In addition, the functional units in each embodiment of the application can be integrated into a processing unit, or each unit can be physically present, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0155] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0156] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above implementation. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for processing noise during live book lectures, characterized in that: The method comprises: Extract audio clips of live book lectures; Extracting an audiogram from the live book-reading audio segment, and generating an audiogram structure based on the audiogram; Extracting a two-dimensional timing diagram from the live audio segment of the book lecture, and generating a timing diagram structure according to the two-dimensional timing diagram; The audio graph structure and the timing graph structure are input into a preset graph convolutional network to fuse the audio graph structure and the timing graph structure of different domains based on the graph attention mechanism according to the graph convolutional network to obtain a first graph structure, and the timing graph structure and the first graph structure are fused based on the graph attention mechanism to obtain a second graph structure, and the audio graph structure and the first graph structure are fused based on the graph attention mechanism to obtain a third graph structure, and the denoised live book reading audio clip is generated based on the second graph structure and the third graph structure.

2. The method for processing noise during live lectures according to claim 1, wherein: The step of extracting an audio graph from the live audio segment of the book lecture and generating an audio graph structure according to the audio graph comprises: Performing short-time Fourier transform on the live book-lecture audio clip to obtain the audio frequency graph; Detecting time-frequency points in the audio graph, using the time-frequency points as nodes of the audio graph structure, and using amplitude values ​​of the time-frequency points as node features; The edges between the time-frequency points are constructed based on the time-frequency neighborhood.

3. The method for processing noise during live lectures according to claim 1, wherein: The step of extracting a two-dimensional timing diagram from the live audio segment of the book lecture and generating a timing diagram structure according to the two-dimensional timing diagram includes: Normalizing and scaling the live book-lecture audio clip; Mapping the normalized and scaled live book-reading audio clip to polar coordinates; Based on Gram angle and field transformation, the live book-reading audio clip mapped to polar coordinates is converted into a two-dimensional time sequence diagram; Detecting pixel points in the two-dimensional timing graph as nodes of the timing graph structure; The edges between the pixels are constructed according to the spatial relationship of the pixels in the two-dimensional time series graph.

4. The method for processing noise during live lectures according to claim 1, wherein: The graph-attention mechanism is used to fuse the audio graph structure and the time sequence graph structure of different domains to obtain a first graph structure, including: The audio graph structure and the time sequence graph structure of different domains are fused based on a graph attention network to obtain a first graph structure.

5. The method for processing noise during live lectures according to claim 4, characterized in that: The method of fusing the time series graph structure and the first graph structure based on the graph attention mechanism to obtain a second graph structure includes: Calculating a first mapping structure of the timing diagram structure; Normalizing the first mapping structure and the first graph structure after element-by-element multiplication to obtain a first weight; multiplying the first weight and the first graph structure to obtain a first intermediate graph structure; The first intermediate graph structure and the first graph structure are fused to obtain the second graph structure.

6. The method for processing noise during live lectures according to claim 4, characterized in that: The method of fusing the audio graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure includes: calculating a second mapping structure of the audio graph structure; Multiplying the second mapping structure and the first mapping structure element by element and then normalizing them to obtain a second weight; multiplying the second weight and the first graph structure to obtain a second intermediate graph structure; The second intermediate graph structure and the first graph structure are fused to obtain the third graph structure.

7. The method for processing noise during live lectures according to claim 1, wherein: The generating of the denoised live book-reading audio segment based on the second graph structure and the third graph structure includes: Extracting first graph features in the second graph structure based on the first branch of the graph convolutional network; Extracting second graph features in the third graph structure based on the second branch of the graph convolutional network; Concatenate the first image feature and the second image feature to obtain a third image feature; Inputting the third graph feature into a multi-layer perceptron to obtain an output feature; The output features are reconstructed into the denoised audio segment of the live book lecture.

8. A noise processing system for live book lectures, characterized in that: The system comprises: Audio acquisition module, used to extract audio clips of live book lectures; An audio graph structure generation module is used to extract an audio graph from the live book-reading audio segment and generate an audio graph structure based on the audio graph; A timing diagram structure generating module is used to extract a two-dimensional timing diagram from the live audio clip of the book lecture and generate a timing diagram structure according to the two-dimensional timing diagram; An audio denoising module is used to input the audio graph structure and the timing graph structure into a preset graph convolutional network, so as to fuse the audio graph structure and the timing graph structure of different domains based on the graph attention mechanism according to the graph convolutional network to obtain a first graph structure, and to fuse the timing graph structure and the first graph structure based on the graph attention mechanism to obtain a second graph structure, and to fuse the audio graph structure and the first graph structure based on the graph attention mechanism to obtain a third graph structure, and to generate the denoised live book-reading audio clip based on the second graph structure and the third graph structure.

9. An electronic device, characterized in that: It includes at least one controller and a memory for communicating with the controller; the memory stores instructions that can be executed by the at least one controller, and the instructions are executed by at least one of the controllers to enable at least one of the controllers to execute the live lecture noise processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the live lecture noise processing method according to any one of claims 1 to 7.

Images