Multi-parameter driven stylized sound synthesis method and device, electronic equipment and medium

Abstract

The invention discloses a multi-parameter driven stylized sound synthesis method and device, electronic equipment and a medium, and the method comprises the steps: obtaining an original audio signal, and carrying out the offline particle library construction processing of the original audio signal, the off-line particle library construction processing comprises main period estimation, phase alignment cutting based on a prediction and rollback mechanism, feature extraction and redundancy control based on clustering and similarity, and an off-line particle library is obtained; acquiring an external control parameter in a real-time interaction process, and mapping the external control parameter into a synthetic driving parameter; and dynamically scheduling the target particles from the offline particle library based on the synthetic driving parameters, performing segmented reading and smooth splicing processing on the scheduled target particles, and executing cross fading at a particle switching boundary to output an audio signal. Splicing artifacts are eliminated through phase alignment, storage occupation is remarkably reduced through redundancy control, and real-time sound synthesis which is low in time delay, high in sound quality and rich in stylized control capacity can be achieved on a low-computing-power platform.

Description

Technical Field

[0001] This invention belongs to the field of digital audio signal processing and computer music technology, specifically relating to a multi-parameter driven stylized sound synthesis method, apparatus, electronic device, and medium. Background Technology

[0002] With the development of digital entertainment, extended reality (XR), and smart cockpit technologies, the demand for sound synthesis with a technological, futuristic, and electronic feel is growing. Granular synthesis, as an important sound design technique, is widely used in texture generation and sound effect design.

[0003] However, existing particle synthesis and related technologies still have significant technical shortcomings in practical applications, especially on resource-constrained platforms (such as mobile devices and automotive chips): Traditional particle synthesis typically employs equidistant cutting or energy-threshold-based cutting. This approach often neglects waveform phase information, leading to phase mismatch at particle joints and producing noticeable "pop / click" sounds or high-frequency glitches. While existing semi-automatic cutting tools incorporate zero-crossing detection, they lack in-window alignment and jump constraints, making them prone to mismatch when encountering noise or missing zero-crossing points.

[0004] To ensure richness of timbre, traditional samplers often need to load a large library of audio materials. Due to the lack of systematic feature analysis and redundancy control, the library contains a large number of similar particles, resulting in high memory consumption and slow loading speed, making it difficult to meet the lightweight requirements of IoT or mobile applications.

[0005] While frequency domain vocoders or neural network-based synthesis schemes (such as DDSP and diffusion models) can improve sound quality, they involve complex FFT transformations or deep learning inference, which consume a lot of computing power and have high latency, making it difficult to achieve low-latency real-time interaction.

[0006] Existing lightweight solutions often lack a unified parameter mapping mechanism, making it difficult to perform fine stylistic modulation of sound pitch, texture density, and dynamic envelope based on real-time changing physical parameters (such as vehicle speed and game status). Summary of the Invention

[0007] The purpose of this invention is to provide a multi-parameter driven stylized sound synthesis method, apparatus, electronic device, and storage medium, which can solve the technical problems of obvious phase stitching artifacts, high resource consumption, and poor real-time performance of low computing power platforms in the prior art.

[0008] To achieve the above objectives, a specific embodiment of the present invention provides a multi-parameter driven stylized sound synthesis method, the method comprising: acquiring an original audio signal, performing offline particle library construction processing on the original audio signal, the offline particle library construction processing comprising main period estimation, phase alignment cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity, to obtain an offline particle library containing deredundant particles and metadata; Obtain external control parameters during real-time interaction, and map the external control parameters to synthesis driving parameters, wherein the synthesis driving parameters include at least the playback step size and the gain envelope; Based on the synthesis driving parameters, target particles are dynamically scheduled from the offline particle library. The scheduled target particles are read in segments and smoothly spliced. Crossfading is performed at the particle switching boundary, and the synthesized stylized audio signal is output.

[0009] In one or more embodiments of the present invention, the specific steps of phase alignment cutting in the offline particle library construction process include: The main period of the input signal is obtained by using the autocorrelation function or the fundamental frequency estimation algorithm, and the set of positive zero-crossing points is extracted. Using the first positive zero-crossing point as the initial boundary, the approximate position of the next boundary is predicted based on the principal period; Find the nearest actual positive zero-crossing point within a preset search window near the approximate position as the alignment boundary; If there is no zero-crossing point that meets the conditions within the preset search window, a backoff strategy is executed to determine the alignment boundary.

[0010] In one or more embodiments of the present invention, the rollback strategy specifically includes a two-layer rollback mechanism: Set the first layer of backoff, and search for the first zero-crossing point that satisfies the maximum jump constraint within the preset interval as the alignment boundary; A second backtracking layer is set up. If the first backtracking search fails, an equidistant cut is forced, and the predicted approximate position is directly used as the next boundary.

[0011] In one or more embodiments of the present invention, the specific steps of the redundancy control based on clustering and similarity include: Determine the length and start and end points of the particle, wherein the end of the particle includes a sampling point after the last positive zero crossing; Statistical features of each particle are extracted, including spectral centroid, spectral bandwidth, zero-crossing rate, RMS energy, and Mel frequency cepstral coefficients (MFCC). Based on the statistical characteristics, a clustering algorithm is used to divide the particles into several clusters, and the intra-cluster variance of each cluster is calculated. If the variance within a cluster is lower than a preset variance threshold, the M particles closest to the cluster center are selected as representatives; if the variance within a cluster is not lower than a preset variance threshold or the number of particle clusters is less than a preset value, a combination of sampling from the center point and the farthest point is used to select representative particles. The similarity between particles is calculated using cosine similarity. Particles with similarity higher than the redundancy threshold are removed, and the remaining particles and their metadata are packaged and stored.

[0012] In one or more embodiments of the present invention, the redundancy control further includes: Calculate the cosine similarity between particles after representative particle selection; If the cosine similarity between two particles is greater than the preset similarity threshold, they are identified as redundant particles and a removal operation is performed. The remaining particles and their metadata are then packaged and stored. The metadata includes particle offset length, particle length, crossfade length, and sampling rate.

[0013] In one or more embodiments of the present invention, mapping the external control parameters to synthetic driving parameters includes: The playback step of the particle is calculated using a step mapping function, which includes a pitch ratio mapping based on a semitone offset or a velocity envelope mapping based on a low-frequency oscillator. A gain envelope is generated using a gain mapping function, which includes ADSR envelope generation or equal-loudness correction based on the target loudness.

[0014] In one or more embodiments of the present invention, the segmented reading and smooth splicing process of the scheduled target particles includes: Maintain a playback state machine, which includes the current particle segment and the crossfade segment; In the non-cross-fading segment, particle samples are read by linear interpolation based on the current read pointer position; During the crossfade phase, the scheduling of the next particle is triggered, and the tail of the current particle and the head of the next particle are weighted and superimposed using an equal power window function. At any sampling point in the crossfade phase, the sum of the squares of the crossfade-in weighting coefficient and the crossfade-out weighting coefficient is 1.

[0015] In another aspect of the invention, a multi-parameter driven stylized sound synthesis apparatus is provided, the multi-parameter driven stylized sound synthesis apparatus being used in the aforementioned multi-parameter driven stylized sound synthesis method, characterized in that the multi-parameter driven stylized sound synthesis apparatus comprises: an offline construction unit, an online mapping unit, and a real-time synthesis unit, wherein... The offline construction unit is used to acquire the original audio signal and perform offline particle library construction processing on the original audio signal. The offline particle library construction processing includes main period estimation, phase alignment and cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity to obtain an offline particle library containing deredundant particles and metadata. The online mapping unit is used to acquire external control parameters during real-time interaction and map the external control parameters into synthetic driving parameters. The synthetic driving parameters include at least the playback step size and the gain envelope. The real-time synthesis unit is used to dynamically schedule target particles from the offline particle library based on the synthesis driving parameters, perform segmented reading and smooth splicing of the scheduled target particles, perform crossfading at particle switching boundaries, and output the synthesized stylized audio signal.

[0016] In another aspect of the invention, an electronic device is provided, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for implementing multi-parameter driven stylized sound synthesis.

[0017] In another aspect of the invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of a multi-parameter driven stylized sound synthesis method.

[0018] Compared with existing technologies, the multi-parameter driven stylized sound synthesis method of this invention has the following significant advancements: This invention solves the phase abruptness problem at the particle splicing point from the source by using a phase alignment strategy of periodic prediction and in-window search. With the help of a double-layer backoff mechanism, it can ensure the smoothness of the boundary even when processing complex signals containing DC bias or noise, and completely eliminate the "click" sound and the feeling of burrs.

[0019] This invention uses a closed-loop redundancy control system of "feature clustering + variance adaptive selection + similarity deduplication" to compress the particle library volume to a very small proportion of the original material, significantly reducing memory and storage overhead and enabling it to run smoothly on mobile, embedded and IoT devices.

[0020] This invention eliminates complex frequency domain transformations or neural network inferences in the online synthesis stage, and only uses linear interpolation and constant-level cross-fading operations, resulting in extremely low computational complexity and enabling ultra-low latency response at the millisecond level.

[0021] This invention establishes a unified parameter mapping framework that supports real-time mapping of physical quantities such as beat, rotation speed, and interactive events into pitch, texture speed, and dynamic envelope, enabling the generation of dynamic sound effects with a sense of technology and futurism.

[0022] This invention reduces the cost of cross-platform integration and middleware development based on standardized data packaging formats and robust construction processes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a multi-parameter driven stylized sound synthesis method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a multi-parameter driven stylized sound synthesis processing flow according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the limiting cut curve in one embodiment of the present invention; Figure 4 This is a schematic diagram of the curves for zero-crossing backoff and maximum jump in one embodiment of the present invention; Figure 5 This is a schematic diagram (a) of particle clustering results in one embodiment of the present invention; Figure 6 This is a schematic diagram (II) of particle clustering results in one embodiment of the present invention; Figure 7 This is a schematic diagram of a multi-parameter driven stylized sound synthesis device according to an embodiment of the present invention; Figure 8 This is a hardware structure diagram of a computing device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0026] As described in the background section, existing sound synthesis technologies generally suffer from artifact problems caused by phase discontinuities, as well as the problem that high-quality synthesis requires excessive computing power and storage, making it difficult to deploy efficiently on mobile or in-vehicle devices.

[0027] To address the aforementioned technical issues, such as Figure 1 As shown, this invention introduces a multi-parameter driven stylized sound synthesis method, including the following steps: Step 1: Obtain the original audio signal and perform offline particle library construction processing on the original audio signal. The offline particle library construction processing includes main period estimation, phase alignment and cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity to obtain an offline particle library containing deredundant particles and metadata. Step 2: Obtain the external control parameters during the real-time interaction process, and map the external control parameters to the synthesis driving parameters. The synthesis driving parameters include at least the playback step size and the gain envelope. Step 3: Dynamically schedule target particles from the offline particle library based on the synthesis driving parameters, perform segmented reading and smooth splicing of the scheduled target particles, perform crossfading at particle switching boundaries, and output the synthesized stylized audio signal.

[0028] In a further embodiment, the period and anchor point estimation includes: for the input signal x [ n This method first estimates the principal period. T 0 and f 0 (satisfied) T 0= 1 / f 0). Fundamental frequency estimation can be performed using the pyin / YIN algorithm, or based on the autocorrelation function. ,in: ; In the formula: The lag period of the time series is used; when the autocorrelation confidence is low, the main peaks are identified, and a regression to the spectral peak detection method is adopted to improve robustness in noisy or weakly periodic scenarios. Then, positive zero-crossings are used as phase anchors, and their set... Defined as The above anchor points Particle cutting for subsequent periodic alignment.

[0029] For scenarios with insignificant fundamental frequency: If the fundamental frequency... If unreliable, autocorrelation peaks or spectral flux can be used: ; In the formula: This represents the amount of change in the spectrum from one frame to the next. The minimum point is used as the anchor point; in extreme cases, it reverts to an equidistant cut.

[0030] In a further embodiment, the phase alignment cutting and backtracking mechanism specifically includes: to ensure the phase continuity of the particles at the beginning and end, the present invention uses the first positive zero crossing as the initial boundary. b 0, and according to the sampling period Where: T0 represents the main period of the input signal; This indicates the sampling rate.

[0031] Predict the approximate location of the next boundary: At the prediction point Nearby search window middle ， Find the nearest actual positive zero crossing point, where the window width is... .

[0032] If no suitable candidate is found in the window, a backoff strategy is adopted: within the interval Take the first one that satisfies the jump constraint The point crossing zero, If no point meets the conditions, the process reverts to equidistant cuts to maintain the progress and completeness of library construction. This two-layer revert significantly enhances robustness in unsteady, low-energy regions.

[0033] like Figure 3 As shown, this demonstrates how the system finds the nearest positive zero-crossing point within the search window to achieve phase alignment and cutting when "period prediction is correct and local waveform is normal". Figure 4 As shown, when a DC bias is artificially added, a certain segment of the waveform is raised overall. The system cannot find a positive zero-crossing point at the approximate location of this boundary. At this point, the system begins to execute a backoff strategy and applies the maximum jump constraint. γ·T0 This avoids boundary overflow or misalignment. It can be seen that the above segmentation method maintains the stability of particle boundaries in both steady-state and abnormal situations.

[0034] In a further embodiment, the particle shaping and desalination process includes: the particle length can be determined by the number of periods. Or fixed duration It is decided that the start and end points of the particles are respectively set as , To maintain phase closure, it is recommended that the particle tip include a sampling point after the last positive zero crossing. Particle edges can be faded using a half-cosine window, an equal-power window, or a Tukey window. Taking an equal-power half-window as an example (window length...) ): This method can effectively eliminate energy jumps at particle joints.

[0035] In a further embodiment, the particle feature extraction includes: to support clustering and redundancy screening, this method extracts the following statistical features from the particles: Spectral centroid: ; Spectral bandwidth: ; Zero crossing rate and RMS energy: Statistical by frame or whole particle. MFCC features: DCT is performed on the logarithmic Mel energy, and the mean and variance are used to describe the overall spectral morphology of the particles; in this embodiment, these features are used to describe attributes such as timbre, brightness, and noise components. Steady-state particle screening can eliminate particles with sudden changes based on spectral flux or energy variance, ensuring the consistency and splicability of particles in the library.

[0036] In a further embodiment, the clustering and representative particle selection include: this invention supports algorithms such as K-Means, K-Medoids, and DBSCAN, and can use metrics such as Mahalanobis distance to improve clustering robustness. One implementation of this invention is to use K-Means clustering to minimize the within-cluster squared error:

[0037] In the formula: Indicates the first i The eigenvectors of each particle; Indicates the first i Cluster number of each particle; Indicates the first i The center of mass of each particle; K This indicates the number of clusters. Within each cluster, the clusters closest to the cluster center can be selected. Particles Alternatively, a combination of "center and farthest point" can be used to balance representativeness and diversity.

[0038] like Figure 5 As shown, when the variance within a cluster is low, and when it is necessary to improve the stability of the sound and achieve smooth splicing, the cluster with the smallest distance from the cluster center can be selected. One particle.

[0039] like Figure 6 As shown, when the number of final clusters is less than the preset threshold K_min, the system switches to a global particle selection strategy. The final particle set is obtained through a combination of "Farthest-Point Sampling", filtering based on steady-state indices, and cross-cluster weighted selection to ensure the coverage of the acoustic space and the diversity of the particle library.

[0040] In a further embodiment, the criteria for selecting the nearest M points and combining the center with the farthest point can be found in Table 1.

[0041] Table 1. Reference Table for Particle Selection Strategies

[0042] In a further embodiment, the redundancy control and capacity constraint specifically include using cosine similarity. Controlling particle redundancy:

[0043] In the formula: They represent the first i The and the first j The eigenvectors of each particle; This represents the similarity threshold, used to control the strictness of selecting particle diversity. < This indicates that two particles are significantly different and should be retained; conversely, it indicates that two particles are almost identical and one of them should be removed. (This is within the particle library capacity limit.) Under the constraint of (typical values ​​32–256), a greedy "farthest point sampling" strategy is adopted, prioritizing the retention of particles with a wider coverage area and eliminating near-duplicates to ensure a small library size, fast loading, and stable performance. In a further embodiment, particles and their metadata (particle offset, particle length, particle crossfade length, sampling rate sr, etc.) are uniformly packaged. The sample pool can use a 16-bit fixed-point format to reduce space usage; uniform resampling to 24kHz is recommended to balance quality and size.

[0044] In a further embodiment, based on an offline particle library, a real-time engine is responsible for dynamically selecting particles from the offline particle library and performing smooth splicing under the control parameters, thereby achieving low-latency audio synthesis.

[0045] Specifically, this includes: adding control vectors The control vector This can include beat / BPM, pitch, envelope, LFO, interaction parameters, etc. The system uses a mapping function. Calculate step size (such as RPM ratio, pitch ratio), and through Generate gain envelope Among them, the mapping function The particle's playback speed / step Δ(t) determines the pitch, rhythm, and duration changes. Mapping function The implementation methods are shown in Table 2 below: Table 2. Mapping Functions Reference Table

[0046] In a further embodiment, Used to output the volume envelope A(t), controlling the loudness, dynamics, and intensity of particles. Mapping function The implementation methods are shown in Table 3 below: Table 3. Mapping Functions Reference Table

[0047] In this process, a particle triggering decision is performed once every hop (a sampling point, approximately 10–30 ms): if the current particle is not in the crossfading interval, the next particle is triggered. Particle selection can employ a random strategy, combined with a "recent deduplication window" to avoid selecting the same particle repeatedly within a short period of time.

[0048] In a further embodiment, segmented playback and pointer interpolation, including real-time playback, are managed using a state machine: current particle segment → crossfade segment (length) → Mapping function ( → Switch to the next particle and build a new intersection segment → Loop playback. The read pointer pos inside the particle is a real number; linear interpolation is used.

[0049] In the formula: ] represents the value of the i-th point of the current particle. These are represented as interpolation coefficients calculated based on the read pointer. They are used to generate continuous samples, wrapping around to the starting point with the same phase when crossing boundaries.

[0050] In a further embodiment, equal-power crossfading involves using an equal-power window to achieve a smooth splicing of the two particle signals without abrupt energy changes:

[0051] In the formula: These represent the values ​​synthesized from the current particle and the next particle, respectively. These represent the window functions for crossfade-in and fade-out, respectively. This indicates the length of the cross fade-in / fade-out transition. In this embodiment, the cross window length is... It should not be larger than the size of a hop. Avoid excessive overlap; , The invention can adaptively adjust according to the fundamental frequency confidence level, and can effectively eliminate step artifacts in traditional particle splicing; another implementation is "pre-sum buffering": the current tail is added to the head of the next particle and pre-stored, which is more suitable for platforms with limited computing power.

[0052] In a further embodiment, the final synthesized audio NMR output is: In the formula: Indicates passage Generate gain envelope , This represents the final synthesized audio output. Stylized rendering modules (filtering, modulation, spatialization, etc.) can be overlaid on top of this. The complexity of real-time synthesis is [value missing] per sample. The system performs pointer updates and constant-time interpolation and addition operations; it requires only a particle pool, metadata, and a small fade buffer to operate, making it suitable for low-power and low-latency applications. In this embodiment, the quality monitoring metrics for the final output audio may include: click / glitch rate: statistically determined by high-frequency energy impact events at the boundary; usage: number of bytes in the sample pool + index table size; real-time performance: CPU utilization and peak latency (should not exceed hop time).

[0053] This invention significantly reduces particle library size while ensuring sound quality and diversity through a dual-core strategy combining phase alignment and redundant control. It also achieves a real-time stylized audio synthesis system with low computing power, low latency, and multi-parameter controllability by combining equal-power crossover and segmented playback methods.

[0054] like Figure 7 As shown, an apparatus for implementing a multi-parameter driven stylized sound synthesis method according to a specific embodiment of the present invention is described. The apparatus includes: an offline construction unit 201, an online mapping unit 202, and a real-time synthesis unit 203, wherein... The offline construction unit 201 is used to acquire the original audio signal and perform offline particle library construction processing on the original audio signal. The offline particle library construction processing includes main period estimation, phase alignment cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity to obtain an offline particle library containing deredundant particles and metadata. The online mapping unit 202 is used to acquire external control parameters during real-time interaction and map the external control parameters into synthesis driving parameters. The synthesis driving parameters include at least the playback step size and the gain envelope. The real-time synthesis unit 203 is used to dynamically schedule target particles from the offline particle library based on the synthesis driving parameters, perform segmented reading and smooth splicing of the scheduled target particles, perform crossfading at particle switching boundaries, and output the synthesized stylized audio signal.

[0055] The multi-parameter driven stylized voice synthesis method in this invention acquires reference and target signals from a voice call, processes them using a conventional echo cancellation algorithm to obtain a first residual signal, processes the first residual signal using a power comparison algorithm to obtain a voice signal with further suppressed residual echoes, and processes the resulting voice signal using a smoothing algorithm as the final output signal. This invention achieves dual suppression based on echo cancellation and power comparison algorithms, further enhancing the system's echo suppression capability and improving call quality without significantly increasing the system's computational burden.

[0056] Figure 8 A hardware structure diagram of a computing device 30 for a multi-parameter driven stylized sound synthesis method is shown according to an embodiment of this specification. Figure 8 As shown, the computing device 30 may include at least one processor 301, a memory 302 (e.g., non-volatile memory), a main memory 303, and a communication interface 304, and the at least one processor 301, memory 302, main memory 303, and communication interface 304 are connected together via a bus 305. At least one processor 301 executes at least one computer-readable instruction stored or encoded in the memory 302.

[0057] It should be understood that the computer-executable instructions stored in memory 302, when executed, cause at least one processor 301 to perform the above-described combinations in the various embodiments of this specification. Figure 1-6 The description includes various operations and functions.

[0058] In the embodiments of this specification, the computing device 30 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile computing device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld system, messaging device, wearable computing device, consumer electronic device, etc.

[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0064] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-parameter driven stylized sound synthesis method, characterized in that, The method includes: The original audio signal is acquired, and an offline particle library construction process is performed on the original audio signal. The offline particle library construction process includes main period estimation, phase alignment and cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity to obtain an offline particle library containing deredundant particles and metadata. Obtain external control parameters during real-time interaction, and map the external control parameters to synthesis driving parameters, wherein the synthesis driving parameters include at least the playback step size and the gain envelope; Based on the synthesis driving parameters, target particles are dynamically scheduled from the offline particle library. The scheduled target particles are read in segments and smoothly spliced. Crossfading is performed at the particle switching boundary, and the synthesized stylized audio signal is output.

2. The multi-parameter driven stylized sound synthesis method according to claim 1, characterized in that, The specific steps of phase alignment and cutting in the offline particle library construction process include: The main period of the input signal is obtained by using the autocorrelation function or the fundamental frequency estimation algorithm, and the set of positive zero-crossing points is extracted. Using the first positive zero-crossing point as the initial boundary, the approximate position of the next boundary is predicted based on the principal period; Find the nearest actual positive zero-crossing point within a preset search window near the approximate position as the alignment boundary; If there is no zero-crossing point that meets the conditions within the preset search window, a backoff strategy is executed to determine the alignment boundary.

3. The multi-parameter driven stylized sound synthesis method according to claim 2, characterized in that, The rollback strategy specifically includes a two-layer rollback mechanism: Set the first layer of backoff, and search for the first zero-crossing point that satisfies the maximum jump constraint within the preset interval as the alignment boundary; A second backtracking layer is set up. If the first backtracking search fails, an equidistant cut is forced, and the predicted approximate position is directly used as the next boundary.

4. The multi-parameter driven stylized sound synthesis method according to claim 1, characterized in that, The specific steps of the redundancy control based on clustering and similarity include: Determine the length and start and end points of the particle, wherein the end of the particle includes a sampling point after the last positive zero crossing; Statistical features of each particle are extracted, including spectral centroid, spectral bandwidth, zero-crossing rate, RMS energy, and Mel frequency cepstral coefficients (MFCC). Based on the statistical characteristics, a clustering algorithm is used to divide the particles into several clusters, and the intra-cluster variance of each cluster is calculated. If the variance within a cluster is lower than a preset variance threshold, the M particles closest to the cluster center are selected as representatives; if the variance within a cluster is not lower than a preset variance threshold or the number of particle clusters is less than a preset value, a combination of sampling from the center point and the farthest point is used to select representative particles. The similarity between particles is calculated using cosine similarity. Particles with similarity higher than the redundancy threshold are removed, and the remaining particles and their metadata are packaged and stored.

5. The multi-parameter driven stylized sound synthesis method according to claim 4, characterized in that, The redundancy control also includes: Calculate the cosine similarity between particles after representative particle selection; If the cosine similarity between two particles is greater than the preset similarity threshold, they are identified as redundant particles and a removal operation is performed. The remaining particles and their metadata are then packaged and stored. The metadata includes particle offset length, particle length, crossfade length, and sampling rate.

6. The multi-parameter driven stylized sound synthesis method according to claim 1, characterized in that, Mapping the external control parameters to synthetic drive parameters includes: The playback step of the particle is calculated using a step mapping function, which includes a pitch ratio mapping based on a semitone offset or a velocity envelope mapping based on a low-frequency oscillator. A gain envelope is generated using a gain mapping function, which includes ADSR envelope generation or equal-loudness correction based on the target loudness.

7. The multi-parameter driven stylized sound synthesis method according to claim 1, characterized in that, The segmented reading and smooth splicing process of the scheduled target particles includes: Maintain a playback state machine, which includes the current particle segment and the crossfade segment; In the non-cross-fading segment, particle samples are read by linear interpolation based on the current read pointer position; During the crossfade phase, the scheduling of the next particle is triggered, and the tail of the current particle and the head of the next particle are weighted and superimposed using an equal power window function. At any sampling point in the crossfade phase, the sum of the squares of the crossfade-in weighting coefficient and the crossfade-out weighting coefficient is 1.

8. A multi-parameter driven stylized sound synthesis apparatus, wherein the multi-parameter driven stylized sound synthesis apparatus is used in the multi-parameter driven stylized sound synthesis method according to any one of claims 1 to 7, characterized in that, The multi-parameter driven stylized sound synthesis device includes: an offline construction unit, an online mapping unit, and a real-time synthesis unit, wherein... The offline construction unit is used to acquire the original audio signal and perform offline particle library construction processing on the original audio signal. The offline particle library construction processing includes main period estimation, phase alignment and cutting based on prediction and backoff mechanism, feature extraction, and redundancy control based on clustering and similarity to obtain an offline particle library containing deredundant particles and metadata. The online mapping unit is used to acquire external control parameters during real-time interaction and map the external control parameters into synthetic driving parameters. The synthetic driving parameters include at least the playback step size and the gain envelope. The real-time synthesis unit is used to dynamically schedule target particles from the offline particle library based on the synthesis driving parameters, perform segmented reading and smooth splicing of the scheduled target particles, perform crossfading at particle switching boundaries, and output the synthesized stylized audio signal.

9. An electronic device, characterized in that, include: At least one processor; And a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the multi-parameter driven stylized sound synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-parameter driven stylized sound synthesis method as described in any one of claims 1 to 7.

Images