Audio content repairing method, apparatus, device, and storage medium

Abstract

This application provides an audio content restoration method, apparatus, device, and storage medium. The method includes: determining an input feature matrix based on a first character, a second character, a third character, the original damaged audio, a content description, and the region to be restored; determining a first target audio point based on a fusion feature matrix corresponding to the input feature matrix; determining a first preset replacement position in the original damaged audio based on the features of the region to be restored; replacing the original damaged audio point located at the first preset replacement position with the first target audio point to obtain a first new audio; and continuously replacing the original damaged audio point by sampling point according to the above steps of determining the first target audio point and the first new audio, until all modifications to the original damaged audio are completed. This application achieves seamless and smooth insertion of audio content through an audio content restoration method based on sampling point + autoregressive iteration, achieving the technical effect of restoring audio to its original state.

Description

Technical Field

[0001] This application belongs to the field of audio processing technology and relates to an audio content repair method, apparatus, device, and storage medium. Background Technology

[0002] When audio media suffers from prolonged signal loss or severe popping due to scratches, breakage, mold, or adhesion, traditional restoration methods typically employ waveform interpolation, spectral restoration, or signal statistical model-based filling techniques. However, these methods can only mathematically complete the signal based on the numerical characteristics of the waveforms before and after the missing section, essentially restoring the "physical waveform" but failing to reconstruct the lost "historical semantic content." For example, when a piece of opera lyrics or a specific historical ambient sound is completely missing, existing algorithms cannot understand the original sound category, rhythm, timbre, and semantic information that should exist in that section, resulting in restoration results lacking historical authenticity and content coherence.

[0003] Even when using manual sound splicing or deep learning-based audio generation methods for overlay restoration, the unique acoustic fingerprints of recordings from different historical periods—including specific signal-to-noise ratios, mechanical resonant frequencies, media noise floor, and reverberation characteristics—mean that directly introducing externally generated audio clips will disrupt the acoustic consistency of the original recording. While this restoration method fills in the auditory gaps, it destroys the historical authenticity and immersive experience of the audio, failing to achieve the universally accepted principle of "restoring the old as it was" in the field of cultural heritage restoration.

[0004] Therefore, how to provide an audio restoration method that can simultaneously understand the semantic content of historical audio and adaptively simulate its original acoustic environment, in order to overcome the dual defects of existing technologies that cannot recover semantic information and destroy the original acoustic consistency, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an audio content restoration method, apparatus, device, and storage medium, which achieves the technical effect of "restoring the old as before" by understanding the semantic content of historical audio and adaptively simulating its original acoustic environment.

[0006] In a first aspect, this application provides an audio content repair method, the method comprising: receiving original damaged audio; extracting a content description and a region to be repaired corresponding to the original damaged audio; setting a first character at the start position of the original damaged audio, setting a second character at the start position of the content description, and setting a third character at the start position of the region to be repaired; generating an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the region to be repaired; determining a fusion feature matrix corresponding to the input feature matrix; determining a first target audio point based on the fusion feature matrix; and determining a first target audio point in the original damaged audio based on the target character. The repair interval is located at a first preset replacement position; the original damaged audio point located at the first preset replacement position is replaced by the first target audio point to obtain a first new audio; based on the step of determining the first target audio point, the first new audio containing the first target audio point is used as the update input, and combined with the first character, the second character, the third character, the content description and the interval to be repaired, a second target audio point is determined and a second new audio is generated in a sample-by-sample autoregressive iteration manner; this process is repeated, and the replacement is carried out sample-by-sample until the original damaged audio point at the last position in the interval to be repaired is replaced with the target sampling point, thus completing the repair of the original damaged audio.

[0007] Compared to traditional overlay splicing, since the noise floor, reverberation and phase of the generated audio and the original audio are independent, a spectral abrupt change (acoustic tomography) will inevitably occur at the junction. The core innovation of this application lies in "sampling point by sampling point + autoregressive iteration": only one audio point is generated and replaced each time, and the audio containing the new audio point is forced to be used as the context for the next generation. This breaks the limitation of traditional restoration methods that can only make blind guesses based on the previous and next waveforms, so that the restoration process has clear semantic guidance. Whether it is missing opera lyrics or specific environmental sounds, they can be consciously generated according to the input content description, realizing the leap from "physical waveform completion" to "semantic content reconstruction", and completely solving the semantic gap problem caused by physical degradation. This mechanism forces that when generating the next sampling point, the mechanical noise, material reverberation and phase state of the original audio at the previous moment must be strictly inherited. At the micro-physical level, the generated waveform and the historical waveform are perfectly matched, completely eliminating the abruptness at the seam. It breaks through the pain point of auditory discontinuity caused by the "hard splicing" of traditional audio editing, realizes the seamless and smooth insertion of audio content, and achieves the ultimate immersive feeling of "restoring the old as before" in audio restoration.

[0008] In one implementation of the first aspect, the original damaged audio includes damaged audio with physical medium degradation characteristics and damaged audio containing specific semantic content; wherein, the damaged audio with physical medium degradation characteristics includes recordings with mechanical noise or physical breakage of the carrier, and the damaged audio containing specific semantic content includes recordings of specific dialects or traditional vocal music.

[0009] In one implementation of the first aspect, determining the first target audio point based on the fused feature matrix includes:

[0010] The fused feature matrix is ​​subjected to global average pooling to obtain a fused feature vector; the first audio element value corresponding to the fused feature vector is output based on the multilayer perceptron; the first audio element value is mapped to a first target audio point through a mapping function.

[0011] In one implementation of the first aspect, the expression corresponding to the first audio element value of the fused feature vector output based on the multilayer perceptron is:

[0012]

[0013] Where X represents the fused feature vector. and These are the weight matrix and bias vector of the hidden layer of the multilayer perceptron, respectively. Represents a non-linear activation function. and These are the weight vector and bias scalar of the output layer, respectively. This represents the audio element value.

[0014] In one implementation of the first aspect, generating an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the interval to be repaired includes: extracting the first character feature corresponding to the first character, the second character feature corresponding to the second character, and the third character feature corresponding to the third character; extracting the original damaged audio feature corresponding to the original damaged audio; extracting the content description feature corresponding to the content description; mapping the interval to be repaired to a high-dimensional space to obtain the interval features to be repaired; and merging the first character feature, the second character feature, the third character feature, the original damaged audio feature, the content description feature, and the interval features to be repaired into an input feature matrix.

[0015] In one implementation of the first aspect, determining the fusion feature matrix corresponding to the input feature matrix includes:

[0016] The input feature matrix is ​​input into a preset language model, and the preset language model outputs the initial semantic features corresponding to the input feature matrix; the initial semantic features are input into an encoder network to perform global relation modeling, and the fused feature matrix corresponding to the initial semantic features is obtained.

[0017] In one implementation of the first aspect, locating a first preset replacement position in the original damaged audio based on the interval to be repaired includes: obtaining the sampling rate of the original damaged audio; determining a replacement time point based on the characteristics of the interval to be repaired; and determining the first preset replacement position based on the sampling rate and the replacement time point.

[0018] Secondly, this application provides an audio content repair device, the device comprising: an audio information acquisition module for acquiring a first character, a second character, a third character, an original damaged audio, a content description, and a region to be repaired; an input feature matrix determination module for generating an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the region to be repaired; a fusion feature matrix determination module for determining a fusion feature matrix corresponding to the input feature matrix; a first target audio point determination module for determining a first target audio point based on the fusion feature matrix; and a first preset replacement position determination module for locating a first preset position in the original damaged audio based on the region to be repaired. The system includes a replacement position module and a first new audio determination module, which replaces the original damaged audio point located at the first preset replacement position with the first target audio point to obtain a first new audio. The original damaged audio complete repair module is used to, based on the step of determining the first target audio point, take the first new audio containing the first target audio point as update input, combine the first character, the second character, the third character, the content description, and the interval to be repaired, and determine the second target audio point and generate the second new audio in a sample-by-sample autoregressive iterative manner. This process is repeated, continuously replacing sample-by-sample until the original damaged audio point at the last position in the interval to be repaired is replaced with the target sample point, thus completing the repair of the original damaged audio.

[0019] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the audio content repair method described in any one of the first aspects of embodiments of this application.

[0020] Fourthly, embodiments of this application provide an electronic device, the electronic device comprising: a memory storing a computer program; and a processor communicatively connected to the memory, which executes the audio content repair method described in any one of the first aspects of embodiments of this application when the computer program is invoked.

[0021] As described above, the audio content restoration method, apparatus, device, and storage medium of this application have the following beneficial effects:

[0022] 1) Compared with traditional overlay splicing, since the noise floor, reverberation and phase of the generated audio and the original audio are independent, a spectral abrupt change (acoustic tomography) will inevitably occur at the junction. The core innovation of this application lies in "sampling point by sampling point + autoregressive iteration": only one audio point is generated and replaced each time, and the audio containing the new audio point is forced to be used as the context for the next generation. This breaks the limitation of traditional restoration methods that can only make blind guesses based on the waveforms before and after, so that the restoration process has clear semantic guidance.

[0023] 2) Whether it is missing opera lyrics or specific environmental sounds, they can be consciously generated based on the input content description, realizing the leap from "physical waveform completion" to "semantic content reconstruction". This completely solves the problem of semantic discontinuity caused by physical degradation. This mechanism forces that when generating the next sampling point, the mechanical noise, material reverberation and phase state of the original audio at the previous moment must be strictly inherited. At the micro-physical level, the generated waveform and the historical waveform are perfectly matched, completely eliminating the abruptness at the seam. It breaks through the auditory discontinuity pain point caused by the "hard splicing" of traditional audio editing, realizes the seamless and smooth insertion of audio content, and achieves the ultimate immersive feeling of "restoring the old as before" in audio restoration.

[0024] 3) In this application, an input feature matrix is ​​obtained based on the first character feature, the second character feature, the third character feature, the original damaged audio feature, the content description feature, and the feature of the interval to be repaired. Each feature in the input feature matrix can interact with all other features for calculation. This means that the "character to be modified" will refer to the "tail note of the original damaged audio", and the "interval to be repaired" will guide the "duration of the character", providing an accurate input feature matrix to ensure the continuity of the modified audio in the entire long audio segment.

[0025] 4) In the process of determining the fusion feature matrix, this application presupposes that the language model and the encoder network are independent of each other to achieve task decoupling: the language model focuses on cross-modal semantic understanding to maintain generalization, while the encoder network focuses on global structural analysis to ensure the purity of feature extraction; the independent architecture enables the encoder to get rid of the limitation of the causal mask of the language model and realize true bidirectional self-attention computation. This independent design has plug-and-play modularity, which greatly improves the iterative efficiency of the system and the feasibility of engineering implementation. Attached Figure Description

[0026] Figure 1 The flowchart shown is a method for repairing audio content provided in an embodiment of this application.

[0027] Figure 2The flowchart shown is a process for determining a first target audio point provided in an embodiment of this application.

[0028] Figure 3 The flowchart shown is a process for determining the input feature matrix provided in an embodiment of this application.

[0029] Figure 4 The flowchart shown is a process for determining the fusion feature matrix provided in an embodiment of this application.

[0030] Figure 5 The flowchart shown is a process for determining a first preset replacement position provided in an embodiment of this application.

[0031] Figure 6 The diagram shown corresponds to the audio content repair method provided in the embodiments of this application.

[0032] Figure 7 The diagram shown is a structural diagram of the audio content restoration device provided in an embodiment of this application.

[0033] Figure 8 The diagram shown is a structural diagram of an electronic device provided in an embodiment of this application.

[0034] Component designation explanation

[0035] S11~S19 step 75 First preset replacement position determination module S21~S23 step 76 First new audio determination module S31~S35 step 77 Original damaged audio repair module S41~S42 step 80 electronic devices S51~S53 step 81 processor 70 Audio content restoration device 82 Non-volatile storage media 71 Audio information acquisition module 83 System bus 72 Input feature matrix determination module 84 Internal memory 73 Fusion Feature Matrix Determination Module 85 Network interface 74 First target audio point determination module Detailed Implementation

[0036] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0037] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0038] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown in the flowchart, this application embodiment provides an audio content repair method. Figure 1As shown, the audio content repair method provided in this application embodiment includes the following steps S11 to S19.

[0040] S11 receives the original corrupted audio.

[0041] In some embodiments, the original damaged audio includes damaged audio with physical media degradation characteristics and damaged audio containing specific semantic content; wherein, the damaged audio with physical media degradation characteristics includes recordings with mechanical noise or physical breakage of the carrier, and the damaged audio containing specific semantic content includes recordings of specific dialects or traditional vocal music.

[0042] For example, the damaged audio with physical medium degradation characteristics includes recordings with mechanical noise or physical breakage of the carrier, specifically including audio recorded on physical media such as early phonograph records and magnetic tapes.

[0043] For example, damaged audio containing specific semantic content includes recordings of specific dialects or traditional vocal music, specifically including: endangered dialects, traditional operas, folk oral traditions and other living heritage recordings.

[0044] S12, extract the content description and the area to be repaired corresponding to the original damaged audio.

[0045] For example, methods for extracting content descriptions corresponding to the original damaged audio include:

[0046] 1) Automatic inference and extraction based on contextual semantics

[0047] Extraction method: Since the audio in the area to be repaired is damaged and its content cannot be directly obtained, the system extracts the edge audio of a preset length (e.g., 2 seconds before and after) before and after the area to be repaired; performs speech recognition (ASR) on the edge audio to obtain the text context, or performs audio event classification to identify ambient sound; inputs the recognition results into the Large Language Model (LLM) or rule reasoning engine, and automatically infers and generates the missing content description in the area to be repaired based on contextual coherence.

[0048] 2) Retrieval and extraction based on historical archive data

[0049] Obtain metadata of the original damaged audio (e.g., track name, recording year, singer information); input the metadata into a pre-built historical audio archive for matching and retrieval, and extract the historical text records (e.g., the lyrics of the track and performance records from that year) corresponding to the metadata as content description.

[0050] For example, the region to be repaired (e.g., [23, 56]) represents the start and end points in the original damaged audio where a physical break, severe degradation, or replacement is required. Its extraction methods include:

[0051] 1) Interactive annotation extraction

[0052] In the audio waveform visualization interface (such as audio editing software) corresponding to the original damaged audio, the repair personnel can manually select or input the start and end time points of the damaged area by listening to and observing the waveform / spectrum. The system directly receives this interactive instruction as the area to be repaired.

[0053] 2) Adaptive detection and extraction based on acoustic features

[0054] The original damaged audio is processed by frame segmentation, and the acoustic features of each frame (such as short-time energy, zero-crossing rate, spectral centroid, signal-to-noise ratio, etc.) are extracted. The feature values ​​corresponding to the acoustic features are compared with the preset damage thresholds (for example, short-time energy below the silence threshold is judged as band breakage, and spectral abrupt change is judged as popping sound). The intervals corresponding to the time frames that continuously meet the damage feature conditions are judged as the intervals to be repaired.

[0055] S13, set a first character at the beginning position of the original damaged audio, set a second character at the beginning position of the content description, and set a third character at the beginning position of the section to be repaired.

[0056] For example, the first character can be used <audio>This indicates that the second character is available. <content>This indicates that the third character is available. <timestep>express.

[0057] It should be noted that the character representation methods of the first, second, and third characters listed in the above examples are merely illustrative. In actual applications, other suitable character representations can be selected based on specific application requirements, and this application does not impose any restrictions on this.

[0058] S14, generate an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the interval to be repaired.

[0059] S15, determine the fusion feature matrix corresponding to the input feature matrix.

[0060] S16, determine the first target audio point based on the fusion feature matrix.

[0061] S17, locate a first preset replacement position in the original damaged audio based on the interval to be repaired.

[0062] For example, if the sampling rate is 16K and the specific representation of the interval to be repaired is [23, 56], then the first preset replacement position can be 23 x 16K.

[0063] S18, replace the original damaged audio point located at the first preset replacement position with the first target audio point to obtain the first new audio.

[0064] For example, if the first target audio point is represented by 'a', and the first preset replacement position is 23 x 16K, the original damaged audio point located at the first preset replacement position is replaced by 'a', while the audio points at other positions in the original damaged audio remain unchanged, thus obtaining the first new audio.

[0065] S19, based on the step of determining the first target audio point, the first new audio containing the first target audio point is used as the update input. Combining the first character, the second character, the third character, the content description, and the interval to be repaired, the second target audio point is determined and the second new audio is generated in a sample-by-sample autoregressive iteration manner. This process is repeated, and the sample-by-sample point is continuously replaced until the original damaged audio point at the last position in the interval to be repaired is replaced with the target sample point, thus completing the repair of the original damaged audio.

[0066] For example, if the first new audio is B, then "B," <audio> 、 <content>"Content Description" <timestep>Based on S12 to S16 above, the second target audio point b is obtained for the "range to be repaired". b replaces the original damaged audio point at the second first preset replacement position (e.g., position 23 x 16K + 1) in B, resulting in C. Similarly, the above steps are repeated, replacing each sample point until the original damaged audio point at the last position in the range to be repaired is replaced with the target sample point, thus completing the repair of the original damaged audio.

[0067] It should be noted that the first preset replacement position 23 x 16K corresponding to the first target audio point and the first preset replacement position 23 x 16K+1 corresponding to the second target audio point listed in the above example, that is, replacing the original damaged audio sequentially from the start time point, are only used as examples. In actual applications, the original damaged audio can be replaced from any time point and any suitable first preset replacement position can be determined based on specific application requirements. This application does not impose any restrictions on this.

[0068] This application provides an audio content restoration method. Compared with traditional overlay splicing, since the noise floor, reverberation and phase of the generated audio and the original audio are independent, a spectral abrupt change (acoustic tomography) will inevitably occur at the junction. The core innovation of this application lies in "sampling point by sampling point + autoregressive iteration": only one audio point is generated and replaced each time, and the audio containing the new audio point is forced to be used as the context for the next generation. This breaks the limitation of traditional restoration methods that can only make blind guesses based on the previous and next waveforms, so that the restoration process has clear semantic guidance. Whether it is missing opera lyrics or specific environmental sounds, they can be consciously generated according to the input content description, realizing the leap from "physical waveform completion" to "semantic content reconstruction", and completely solving the semantic gap problem caused by physical degradation. This mechanism forces that when generating the next sampling point, the mechanical noise, material reverberation and phase state of the original audio at the previous moment must be strictly inherited. At the micro-physical level, the generated waveform and the historical waveform are perfectly matched, completely eliminating the abruptness at the seam. It breaks through the pain point of auditory discontinuity caused by the "hard splicing" of traditional audio editing, realizes the seamless and smooth insertion of audio content, and achieves the ultimate immersive feeling of "restoring the old as before" in audio restoration.

[0069] like Figure 2 As shown in the figure, this application embodiment provides a flowchart for determining a first target audio point, as follows: Figure 2 As shown, the method for determining the first target audio point provided in this application embodiment includes the following steps S21 to S23.

[0070] S21, perform global average pooling on the fused feature matrix to obtain the fused feature vector.

[0071] For example, the size of the fused feature matrix is ​​L×D, where L is the sequence length and D is the feature dimension;

[0072] On the dimension corresponding to the sequence length L, a parameterless arithmetic mean operation is performed on the fusion feature matrix, that is, for each feature dimension d, the mean of L feature segments under that dimension is calculated; through the above mean calculation, the influence of dynamic changes in sequence length is eliminated, and the two-dimensional fusion feature matrix is ​​reduced and collapsed into a one-dimensional fusion feature vector with a fixed dimension D, so as to serve as the standardized input for subsequent single-point numerical mapping of the multilayer perceptron.

[0073] S22, based on the multilayer perceptron, output the first audio element value corresponding to the fused feature vector.

[0074] In some embodiments, the expression corresponding to the first audio element value of the fused feature vector output based on the multilayer perceptron is:

[0075]

[0076] Where X represents the fused feature vector. and These are the weight matrix and bias vector of the hidden layer of the multilayer perceptron, respectively. Represents a non-linear activation function. and These are the weight vector and bias scalar of the output layer, respectively. This represents the value of the first audio element.

[0077] In this configuration, the output layer of the multilayer perceptron is set as a purely linear mapping layer without an activation function, such that the value of the first audio element is... Maintaining an unbounded, continuous state of real numbers allows for adaptation to the physical constraints imposed on the input range by the subsequent hyperbolic tangent function.

[0078] S23, map the first audio element value to the first target audio point through a mapping function.

[0079] For example, the expression that maps the first audio element value to the first target audio point using a mapping function is as follows:

[0080]

[0081] in, Indicates the first target audio point. This represents the audio element value.

[0082] This application provides a method for determining a first target audio point. In this method, a hierarchical design of "dimensionality reduction aggregation → nonlinear mapping → legal constraints" is used to achieve a balance between feature utilization efficiency, task adaptability and output physical legality. This is a key technical support for "from features to audio" in audio generation tasks.

[0083] like Figure 3 As shown in the figure, this application provides a flowchart for determining the input feature matrix, as follows: Figure 3 As shown, the method for determining the input feature matrix provided in this application embodiment includes the following steps S31 to S35.

[0084] S31, extract the first character feature corresponding to the first character, the second character feature corresponding to the second character, and the third character feature corresponding to the third character, respectively.

[0085] For example, the first character feature corresponding to the first character, the second character feature corresponding to the second character, and the third character feature corresponding to the third character can be extracted based on the bidirectional encoder representations from the transformers (BERT).

[0086] S32, Extract the original damaged audio features corresponding to the original damaged audio.

[0087] For example, the original damaged audio features corresponding to the original damaged audio can be extracted based on waveform-to-vector (wav2vec).

[0088] S33, Extract the content description features corresponding to the content description.

[0089] For example, the content description features corresponding to the content description can be extracted based on the CLIP text encoder.

[0090] S34, map the interval to be repaired to a high-dimensional space to obtain the features of the interval to be repaired.

[0091] For example, the region to be repaired can be mapped to a high-dimensional space based on a multi-layer perceptron (MLP) to obtain the features of the region to be repaired.

[0092] S35, the first character feature, the second character feature, the third character feature, the original damaged audio feature, the content description feature, and the feature of the area to be repaired are merged into an input feature matrix.

[0093] For example, the first character feature, the second character feature, and the third character feature can be mapped to a fixed-dimensional identifier vector through a special token embedding layer; using an audio coding network, a text segmentation embedding network, and a time-value projection network, the original damaged audio features, content description features, and features to be repaired interval features are respectively transformed into continuous feature sequences with aligned dimensions; according to a preset temporal logic, the identifier vector corresponding to the first character feature is used as a prefix and concatenated with the original damaged audio feature at the beginning and end of the sequence dimension, and so on, to complete the concatenation of the second character feature with the content description feature, and the third character feature with the features to be repaired interval, forming an original concatenation matrix of length L and dimension D for each feature; introducing positional encoding, a positional matrix of dimension L×D is added to the original concatenation matrix to obtain the input feature matrix.

[0094] This application provides a method for determining an input feature matrix. In this method, the first character feature corresponding to the first character, the second character feature corresponding to the second character, the third character feature corresponding to the third character, the original damaged audio feature corresponding to the original damaged audio, the content description feature corresponding to the content description, and the feature of the interval to be repaired corresponding to the interval to be repaired are extracted respectively. Based on these six features, an input feature matrix is ​​obtained. This six types of features are combined into a single "input feature matrix" and fed into the network. Under the self-attention mechanism, each feature in the input feature matrix can interact and be calculated with all other features. This means that the "character to be modified" will reference the "tail note of the original damaged audio," and the "interval to be repaired" will guide the "duration of the character," providing an accurate input feature matrix to ensure the continuity of the modified audio within the entire long audio segment.

[0095] like Figure 4 As shown in the figure, this application embodiment provides a flowchart for determining the fusion feature matrix, as follows: Figure 4 As shown, the determination of the fusion feature matrix provided in this application embodiment includes the following steps S41 to S42.

[0096] S41, input the input feature matrix into a preset language model, and output the initial semantic features corresponding to the input feature matrix through the preset language model.

[0097] For example, the default language model is the Large Language Model (LLM).

[0098] For example, through a multi-head self-attention layer in a preset language model, cross-attention weights are calculated for the first character feature, the original damaged audio feature, the second character feature, the content description feature, the third character feature, and the feature of the interval to be repaired in the input feature matrix. This establishes a cross-modal association mapping between text features and audio features in the feature space, resulting in an attention feature matrix. From the attention feature matrix output by the last hidden layer of the preset language model, the hidden state vector corresponding to the position of the first character feature is extracted. The hidden state vector is used as the initial semantic feature that integrates the contextual information of the original damaged audio feature, the content description feature, and the feature of the interval to be repaired.

[0099] S42, the initial semantic features are input into the encoder network to perform global relation modeling, and the fusion feature matrix corresponding to the initial semantic features is obtained.

[0100] The encoder network and the pre-defined language model are independent of each other.

[0101] For example, the steps of inputting the initial semantic features into the encoder network for global relation modeling to obtain the fused feature matrix corresponding to the initial semantic features are as follows:

[0102] 1) Feature interface alignment: The initial semantic features extracted by the pre-trained language model are first transformed in dimension through a linear mapping layer to match the input dimension requirements of the independent encoder (Transformer Encoder) network, and the transformed features are serialized and unfolded into a two-dimensional matrix.

[0103] 2) Structural position injection: Add a pre-defined structured position code to the two-dimensional matrix to give each feature segment in the matrix absolute prior position information;

[0104] 3) Maskless bidirectional global interaction: The matrix carrying the prior position information is input into the self-attention layer of the independent TransformerEncoder network. Under the condition of removing the causal mask, the bidirectional attention weight between any two feature segments in the sequence is calculated, so that each feature segment can aggregate the global context information of the whole sequence.

[0105] 4) Nonlinear fusion and iteration: The features that have been globally aggregated are sequentially fused through a feedforward neural network, residual connections and layer normalization operations, and after being processed by stacking multiple encoder layers, a fused feature matrix with deep correlation of internal features is output.

[0106] This application provides a method for determining a fusion feature matrix. In this method, the preset language model and encoder network are independent of each other to achieve task decoupling: the language model focuses on cross-modal semantic understanding to maintain generalization, while the encoder network focuses on global structure analysis to ensure the purity of feature extraction. The independent architecture enables the encoder to break free from the limitation of the language model's causal mask and achieve true bidirectional self-attention computation. This independent design has plug-and-play modularity, which greatly improves the system's iteration efficiency and engineering feasibility.

[0107] like Figure 5 As shown, this application embodiment provides a flowchart for determining a first preset replacement position, as follows: Figure 5 As shown, the method for determining the first preset replacement position provided in this application embodiment includes the following steps S51 to S53.

[0108] S51, obtain the sampling rate of the original damaged audio.

[0109] For example, the sampling rate of the original damaged audio could be 16K, 20K, etc.

[0110] For example, this can be achieved by calling the application programming interface (API) of the audio processing library.

[0111] S52, determine the replacement time point based on the characteristics of the interval to be repaired.

[0112] For example, the replacement time point can be any suitable time within the interval to be repaired.

[0113] For example, if the interval to be repaired is [23, 56], the replacement time point can be the start time point 23X, the end time point 56X, or any suitable time point within the interval to be repaired [23, 56].

[0114] S53, determine the first preset replacement position based on the sampling rate and the replacement time point.

[0115] For example, if the original damaged audio has a sampling rate of 16K, the replacement time point can be the start time point 23X, and the first preset replacement position is the sampling audio point at the position 23X 16K.

[0116] This application provides a method for determining a first preset replacement position. In this method, the replacement time point can be arbitrarily selected within the interval to be repaired, supporting multiple replacement strategies such as start time point, end time point, and intermediate time point to adapt to different editing needs. Through the logic of sampling rate → replacement time point → sampling position, it achieves precise positioning of a specific time period in the original damaged audio, ensuring the accuracy and reliability of the replacement operation. It is the core technical support for the "positioning-replacement" link in audio editing or processing, and is especially suitable for scenarios that require high-precision processing.

[0117] Please see Figure 6 , Figure 6 The diagram shown is a structural diagram corresponding to an audio content restoration method provided in an embodiment of this application. Figure 6 The steps in the above Figures 1 to 5 The above has already been explained in detail, and this application will not repeat it here.

[0118] This application provides an embodiment of an application scenario for a method to repair audio content corresponding to damaged audio with physical media degradation characteristics. The specific repair steps are as follows:

[0119] 1. Application Scenario: An archive contains an early 1940s phonograph record (audio recorded on a physical medium). Due to its age and improper preservation, the record has a deep scratch between the 23rd and 25th seconds (physical breakage of the medium), resulting in a sharp popping sound and tape breakage for 2 seconds during playback. Furthermore, the original Peking Opera lyrics recorded in that section are completely lost. Additionally, the entire audio file is accompanied by the characteristic hissing sound of a phonograph (mechanical noise).

[0120] The system receives the original damaged audio that exhibits physical media degradation characteristics.

[0121] 2. Extract content description and areas to be repaired.

[0122] Extraction of the region to be repaired: By detecting acoustic features (or manually annotating), the physical location of the popping sound and the broken band is identified, and the region to be repaired is extracted as [23, 25] (s).

[0123] Content description extraction: Based on historical archives such as opera scripts from that year, the restoration team verified that the missing 2s content was "a Peking Opera Qingyi singing segment with the background noise of an old-fashioned needle", and extracted it as the content description.

[0124] 3. Special character settings and input feature matrix generation

[0125] Set the first character at the beginning of the original damaged audio. <audio>Set the second character at the beginning of the content description. <content>Set the third character at the beginning of the interval [23, 25] to be repaired. <timestep>.

[0126] The damaged audio features with hissing noise are extracted by wav2vec, the composite semantic features of "a Peking Opera Qingyi singing segment with hissing noise from an old-fashioned needle" are extracted by CLIP, and three special character features are extracted by BERT. [23, 25] is mapped into a high-dimensional time boundary vector by MLP, and the above 6 features are merged into the input feature matrix.

[0127] 4. Obtaining the fused feature matrix using an independent two-stage network.

[0128] Cross-modal semantic understanding: The input feature matrix first enters the preset language model (LLM). With its powerful generalization ability, the LLM understands the semantic relationship between the content "Peking Opera Qingyi singing segment" and the damaged audio context, and outputs the initial semantic features.

[0129] Global structural relationship refinement: Due to the specific mechanical resonance and continuous background noise texture of phonograph audio, the initial semantic features are then fed into an independent encoder network. This network abandons causal masking and performs true bidirectional self-attention computation, capturing the long-distance dependencies of the audio before and after the scratch (especially the continuous evolution of the mechanical noise background) without any blind spots, and outputting a fusion feature matrix that accurately characterizes the global acoustic structure.

[0130] 5. Autoregressive iterative replacement for each sampling point

[0131] Locating the first replacement point: Based on the interval to be repaired [23, 25] and a sampling rate of 16K, the first preset replacement position is located at the 23×16000 = 368,000th sampling point.

[0132] The first step involves generation and replacement: global average pooling and MLP mapping are performed on the fused feature matrix, and the first target audio point 'a' is output via tanh. At this point, the value of 'a' not only conforms to the semantics of "Qingyi singing segment" but is also constrained by the "hissing background noise" in the first 367,999 real sampling points. The audio corruption point at position 368,000 is replaced with 'a' to obtain the first new audio B.

[0133] Autoregressive rolling stitch: The first new audio B containing the new point a is used as the update input. Combining the aforementioned character and interval features, the second target audio point b is determined again through the model. The position 368,001 is replaced to obtain the second new audio C.

[0134] Continuous iteration: This process is repeated, with continuous replacement using a self-regressive iteration method for each sample point. This microscopic point-by-point stitching ensures that the generation of each new waveform forcibly inherits the mechanical noise floor amplitude and phase characteristics of the previous instant, completely avoiding the acoustic gap between the "clean generated sound and the noisy original sound" caused by traditional overlay methods.

[0135] 6. Repair complete

[0136] The repair is completed when the 25 × 16000 = 400,000th sampling point (the last position in the interval to be repaired) is replaced with the target sampling point.

[0137] The restored audio not only filled in the missing semantics of the Peking Opera lyrics, but the newly generated 2s audio also perfectly matched the original recordings in terms of background noise texture, stylus resonance and phase. It sounded as if the scratches on the record had never existed, achieving "restoring the old to its original state" for audio with physical media degradation.

[0138] This application provides an embodiment of an audio content repair method based on damaged audio containing specific semantic content. The specific repair steps are as follows:

[0139] 1. Scenario Setting: A non-material cultural heritage protection organization rescued a recording of a centenarian speaking an extremely endangered dialect (such as a Tujia language understood by only a few people). Due to the harsh outdoor recording conditions, the 15th to 18th seconds were completely obscured by a sudden burst of strong wind noise, resulting in the complete loss of this highly valuable dialect passage, creating a "semantic gap." This dialect lacks sufficient general corpus to train traditional speech synthesis models, and traditional waveform interpolation is completely unable to guess what the elderly person said.

[0140] The system receives the original, corrupted audio file containing missing semantic content.

[0141] 2. Extract content description and areas to be repaired.

[0142] Extraction of the region to be repaired: The area covered by strong wind noise was located by acoustic feature detection, and the region to be repaired was extracted as [15, 18] (s).

[0143] Content description extraction: Based on the oral context and the grammatical habits of the dialect, linguists inferred that the missing content was "a sacrificial prayer uttered in this endangered dialect," and extracted it as the content description. This step solves the key prior knowledge input for "semantic dating."

[0144] 3. Special character settings and input feature matrix generation

[0145] Set the first character at the beginning of the original damaged audio. <audio>Set the second character at the beginning of the content description. <content>Set the third character at the beginning of the interval [15, 18] to be repaired. <timestep>.

[0146] The acoustic features of the elderly man’s speech before and after were extracted using wav2vec (including his unique voice characteristics and dialect timbre). The deep semantic features of “sacrificial prayers in endangered dialect narratives” were extracted using CLIP text encoder. Special character features were extracted using BERT. [15, 18] were then mapped into a high-dimensional temporal boundary vector using MLP.

[0147] The above features are combined into an input feature matrix. At this point, both the "target semantics" and the "speaker acoustic context" are available.

[0148] 4. Obtaining the fused feature matrix using an independent two-stage network.

[0149] The input feature matrix is ​​fed into a pre-defined language model (LLM). Faced with this rare dialect, the LLM, with its powerful cross-modal generalization ability, aligns the text semantics with the preceding and following audio context, understands "the need to express the meaning of blessing with this specific timbre and rhythm", and outputs the initial semantic features.

[0150] Global structural relationship refinement: Initial semantic features are fed into an independent encoder network. Due to the unique intonation and long-distance prosodic dependence of dialect speech, the independent encoder uses maskless bidirectional self-attention to globally capture the intonation fluctuations and dialect tone sandhi features of the elderly before and after the interval [15, 18], and outputs a fusion feature matrix that accurately describes the prosodic structure of the dialect.

[0151] 5. Autoregressive iterative replacement for each sampling point

[0152] Locating the first replacement point: Based on the interval to be repaired [15, 18] and a sampling rate of 16K, the first preset replacement position is located at 15 × 16000 = 240,000 sampling points.

[0153] The first step involves generation and replacement: pooling and mapping the fused feature matrix to output the first target audio point 'a'. Here, 'a' not only contains the semantic meaning of blessing but is also constrained by the elderly person's unique voice and speech flow state from the previous 239,999 sampling points. The damaged point at position 240,000 is replaced with 'a' to obtain the first new audio point 'B'.

[0154] Autoregressive rolling stitch: The first new audio B containing the new point a is used as the update input. Combining semantics and interval features, the second target audio point b is determined again and the position 240,001 is replaced.

[0155] Continuous iteration: This process is repeated, continuously replacing sounds using an autoregressive method based on each sampling point. This mechanism ensures that the generated dialect pronunciations are perfectly consistent with the context in terms of phoneme transitions and tone sandhi. It avoids the "dialect accent abrupt changes" or "machine accent" caused by traditional splicing methods.

[0156] 6. Repair complete

[0157] The repair is completed when the 18th × 16000 = 288,000th sampling point is replaced with the target sampling point.

[0158] The restored audio not only accurately reconstructed the sacrificial prayers that were obscured by wind noise (semantic revival), but also generated a dialect that was indistinguishable from the old man's original pronunciation in terms of timbre, intonation, and pronunciation habits, thus achieving a living inheritance and restoration of endangered intangible cultural heritage materials that was both "form and spirit" intact.

[0159] The scope of protection of the audio content restoration method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0160] This application also provides an audio content restoration device, which can implement the audio content restoration method described in this application. However, the implementation device of the audio content restoration method described in this application includes, but is not limited to, the structure of the audio content restoration device listed in this embodiment. Any structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0161] like Figure 7 As shown, in one embodiment, the audio content restoration device 70 of this application includes an audio information acquisition module 71.

[0162] Input feature matrix determination module 72, fusion feature matrix determination module 73, first target audio point determination module 74, first preset replacement position determination module 75, first new audio determination module 76, original damaged audio full repair module 77.

[0163] The audio information acquisition module 71 is used to acquire the first character, the second character, the third character, the original damaged audio, the content description, and the area to be repaired;

[0164] The input feature matrix determination module 72 is used to generate an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the interval to be repaired;

[0165] The fusion feature matrix determination module 73 is used to determine the fusion feature matrix corresponding to the input feature matrix;

[0166] The first target audio point determination module 74 is used to determine the first target audio point based on the fusion feature matrix;

[0167] The first preset replacement position determination module 75 is used to locate the first preset replacement position in the original damaged audio based on the interval to be repaired.

[0168] The first new audio determination module 76 is used to replace the original damaged audio point located at the first preset replacement position with the first target audio point to obtain the first new audio;

[0169] The original damaged audio repair module 77 is used to, based on the step of determining the first target audio point, take the first new audio containing the first target audio point as the update input, combine the first character, the second character, the third character, the content description and the interval to be repaired, and determine the second target audio point and generate the second new audio in a sample-by-sample autoregressive iterative manner; and so on, continuously replacing sample points until the original damaged audio point at the last position in the interval to be repaired is replaced with the target sample point, thus completing the repair of the original damaged audio.

[0170] The structure and principle of the audio information acquisition module 71, the input feature matrix determination module 72, the fusion feature matrix determination module 73, the first target audio point determination module 74, the first preset replacement position determination module 75, the first new audio determination module 76, and the original damaged audio complete repair module 77 correspond one-to-one with the steps in the above audio content repair method, so they will not be described in detail here.

[0171] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0172] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0173] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] This application also provides an electronic device. Figure 8 The diagram shown is a structural schematic of an electronic device 80 in one embodiment of this application. The audio content restoration method provided in this embodiment can be applied to... Figure 8 The electronic devices shown are 80, but not limited to these. For example... Figure 8 As shown, the electronic device 80 includes a processor 81, a memory, a system bus 83, and a network interface 85. The memory may include a non-volatile storage medium 82 and internal memory 84.

[0175] The non-volatile storage medium 82 can store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor to perform any of the audio content restoration methods provided in the embodiments of this application.

[0176] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0177] The internal memory 84 provides an environment for the execution of a computer program in a non-volatile storage medium. When the computer program is executed by the processor, it enables the processor to perform any of the audio content restoration methods provided in the embodiments of this application.

[0178] This network interface 85 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] It should be understood that processor 81 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0180] The electronic device 80 in this application embodiment may include terminal devices such as tablet computers, laptop computers, mobile phones, supercomputers, and smart wearable devices. It can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0181] For example, electronic devices can be stations (STAION, ST) in WLANs, cellular phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, computers, laptops, handheld communication devices, handheld computing devices, and / or other devices for communicating over wireless systems, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0182] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0183] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0184] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0185] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0186] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.< / timestep> < / content> < / audio> < / timestep> < / content> < / audio> < / timestep> < / content> < / audio> < / timestep> < / content> < / audio>

Claims

1. An audio content restoration method, characterized in that, The method includes: Receive the original damaged audio; Extract the content description and the area to be repaired corresponding to the original damaged audio; A first character is set at the beginning position of the original damaged audio, a second character is set at the beginning position of the content description, and a third character is set at the beginning position of the section to be repaired. An input feature matrix is ​​generated based on the first character, the second character, the third character, the original damaged audio, the content description, and the area to be repaired; Determine the fusion feature matrix corresponding to the input feature matrix; The first target audio point is determined based on the fusion feature matrix; Locate a first preset replacement position in the original damaged audio based on the interval to be repaired; The original damaged audio point located at the first preset replacement position is replaced by the first target audio point to obtain the first new audio; Based on the step of determining the first target audio point, the first new audio containing the first target audio point is used as the update input. Combining the first character, the second character, the third character, the content description, and the interval to be repaired, the second target audio point is determined and the second new audio is generated in a sample-by-sample autoregressive iterative manner. This process is repeated, continuously replacing the original damaged audio point at the last position in the interval to be repaired with the target sampling point, thus completing the repair of the original damaged audio.

2. The method according to claim 1, characterized in that, The original damaged audio includes damaged audio with physical media degradation characteristics and damaged audio containing specific semantic content; wherein, the damaged audio with physical media degradation characteristics includes recordings with mechanical noise or physical breakage of the carrier, and the damaged audio containing specific semantic content includes recordings of specific language dialects or traditional vocal music.

3. The method according to claim 1, characterized in that, The step of determining the first target audio point based on the fused feature matrix includes: The fused feature matrix is ​​subjected to global average pooling to obtain the fused feature vector; The first audio element value corresponding to the fused feature vector is output based on the multilayer perceptron; The first audio element value is mapped to the first target audio point using a mapping function.

4. The method according to claim 3, characterized in that, The expression corresponding to the first audio element value of the fused feature vector output by the multilayer perceptron is: Where X represents the fused feature vector. and These are the weight matrix and bias vector of the hidden layer of the multilayer perceptron, respectively. Represents a non-linear activation function. and These are the weight vector and bias scalar of the output layer, respectively. This represents the audio element value.

5. The method according to claim 1, characterized in that, The process of generating an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the region to be repaired includes: Extract the first character feature corresponding to the first character, the second character feature corresponding to the second character, and the third character feature corresponding to the third character, respectively. Extract the original damaged audio features corresponding to the original damaged audio; Extract the content description features corresponding to the content description; The region to be repaired is mapped to a high-dimensional space to obtain the features of the region to be repaired; The first character feature, the second character feature, the third character feature, the original damaged audio feature, the content description feature, and the feature of the region to be repaired are merged into an input feature matrix.

6. The method according to claim 1, characterized in that, Determining the fused feature matrix corresponding to the input feature matrix includes: The input feature matrix is ​​input into a preset language model, and the preset language model outputs the initial semantic features corresponding to the input feature matrix. The initial semantic features are input into the encoder network for global relation modeling to obtain the fused feature matrix corresponding to the initial semantic features.

7. The method according to claim 1, characterized in that, Locating a first preset replacement position in the original damaged audio based on the area to be repaired includes: Obtain the sampling rate of the original damaged audio; The replacement time point is determined based on the characteristics of the section to be repaired; The first preset replacement position is determined based on the sampling rate and the replacement time point.

8. An audio content restoration device, characterized in that, The device includes: An audio information acquisition module is used to receive original damaged audio; extract the content description and the area to be repaired corresponding to the original damaged audio; set a first character at the beginning position of the original damaged audio, a second character at the beginning position of the content description, and a third character at the beginning position of the area to be repaired; An input feature matrix determination module is used to generate an input feature matrix based on the first character, the second character, the third character, the original damaged audio, the content description, and the interval to be repaired; The fusion feature matrix determination module is used to determine the fusion feature matrix corresponding to the input feature matrix; The first target audio point determination module is used to determine the first target audio point based on the fused feature matrix; The first preset replacement position determination module is used to locate the first preset replacement position in the original damaged audio based on the interval to be repaired. The first new audio determination module is used to replace the original damaged audio point located at the first preset replacement position with the first target audio point to obtain the first new audio; The original damaged audio repair module is used to determine the second target audio point and generate the second new audio point by taking the first new audio containing the first target audio point as the update input, and combining the first character, the second character, the third character, the content description and the interval to be repaired in a sample-by-sample autoregressive iterative manner, based on the step of determining the first target audio point. This process is repeated, continuously replacing the original damaged audio point at the last position in the interval to be repaired with the target sampling point, thus completing the repair of the original damaged audio.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: A memory that stores a computer program; The processor, which is communicatively connected to the memory, executes the method of any one of claims 1 to 7 when the computer program is invoked.

Images