A low-latency voice communication method based on an end-to-end large model

Abstract

The application relates to the technical field of voice communication, and discloses a low-latency voice communication method based on an end-to-end large model, which comprises the following steps: a sending end collects voice waveforms, constructs current input features, inputs a streaming voice communication model, obtains temporary tokens and stability results, combines historical reasoning records to determine a current confirmation anchor point, generates correction information when the confirmation anchor point is moved, and sends a fast packet and a confirmation packet. An audio segment to be played is generated according to the fast packet, and a mapping relationship is recorded; the temporary tokens are confirmed, replaced and affected segment reconstruction is performed according to the confirmation packet; and the audio is output at a planned playing time. The method takes into account the real-time performance and output stability of voice communication.

Description

Technical Field

[0001] This application relates to the field of voice communication technology, specifically a low-latency voice communication method based on an end-to-end large model. Background Technology

[0002] With the development of applications such as two-way voice calls, multi-person conferencing, real-time voice translation, and voice assistant call links, voice communication systems not only need to ensure the continuity and intelligibility of voice transmission, but also need to minimize end-to-end latency to meet real-time interaction requirements. Among existing voice communication technologies, one type of solution focuses on traditional voice encoding, transmission, and playback processes. While the implementation is relatively mature, it often struggles to balance low-latency output and result stability when combined with large-scale end-to-end models for streaming voice processing.

[0003] Another type of solution can perform streaming inference on the input speech and generate intermediate results earlier. However, as the model is continuously supplemented with subsequent context, the inference results of the preceding positions may change. If the receiver directly uses the intermediate results for speech synthesis and playback, it is easy to encounter problems such as inconsistencies in the broadcast content, local speech distortion, semantic jumps, or excessively large ranges of repeated corrections.

[0004] Conversely, if the receiving end waits for more context or even the entire audio segment to stabilize before performing unified synthesis and playback, it will significantly increase the playback waiting time and affect the real-time performance of voice communication. In actual network transmission, there are also issues such as out-of-order packets, network jitter, and delayed arrival of acknowledgment information, which further increase the difficulty of implementing buffer management, segment replacement, and playback control at the receiving end.

[0005] Therefore, how to effectively manage the unstable inference results during streaming voice communication, accurately correct the affected voice segments while ensuring low communication latency, and reduce unnecessary overall reconstruction and playback interruptions has become an urgent technical problem to be solved in the existing technology. Summary of the Invention

[0006] The purpose of this application is to provide a low-latency voice communication method and system based on an end-to-end large model to solve the problems mentioned in the background art.

[0007] According to one aspect of this application, a low-latency voice communication method based on an end-to-end large model is provided, comprising the following steps:

[0008] Collect the speech waveform from the transmitting end, construct the current input features and input them into the streaming speech communication model to obtain temporary tokens and stability results;

[0009] Based on the multiple reasoning results in the historical reasoning record and the stability result, the current confirmation anchor point is determined, and when the current confirmation anchor point is moved forward, correction information is generated based on the difference between the temporary token obtained in this reasoning and the sent temporary token in the unconfirmed interval.

[0010] Send fast packets and send acknowledgments when the current acknowledgment anchor point moves forward. Fast packets carry the newly added temporary token, and acknowledgments carry the current acknowledgment anchor point and correction information.

[0011] After receiving the fast packet, the receiving end writes the new temporary token into the temporary token buffer, generates the audio segment to be played based on the new temporary token that has not yet been synthesized, writes the audio segment to be played into the audio queue to be played, and records the mapping relationship between the token range and the audio range corresponding to the new temporary token.

[0012] After receiving the confirmation packet, the receiving end writes the temporary token up to the current confirmation anchor point into the confirmation token buffer, replaces the corresponding temporary token in the temporary token buffer according to the correction information, and reconstructs the affected audio segment to be played according to the mapping relationship; when the audio segment to be played in the audio queue reaches the planned playback time, the receiving end outputs the corresponding audio segment to be played.

[0013] Preferably, constructing the current input features includes: performing frame segmentation and feature extraction on the speech waveform to obtain feature frames arranged in chronological order; extracting the feature frame corresponding to the current processing block from the feature buffer and concatenating it with the feature frame corresponding to the right context to obtain the current input features.

[0014] Preferably, the streaming voice communication model sequentially includes a temporal downsampling module, a streaming coding module, a token projection module, and a stability estimation module; the temporal downsampling module receives the current input features and outputs a compressed feature sequence, the streaming coding module receives the compressed feature sequence and outputs a high-level coded representation, the token projection module outputs a temporary token based on the high-level coded representation, and the stability estimation module outputs the stability result based on the high-level coded representation and the temporary token.

[0015] Preferably, determining the current confirmation anchor point includes: extracting multiple inference results corresponding to each temporary token from the historical inference record to obtain the consistency result of each temporary token; obtaining the comprehensive stability result of each temporary token based on the consistency result and the corresponding stability result of each temporary token; starting from the temporary token after the last confirmation anchor point, determining the prefix interval that continuously satisfies the confirmation condition according to the token order, and determining the end position of the prefix interval as the current confirmation anchor point.

[0016] Preferably, the consistency result is the proportion of the number of inferences that are the same as the current temporary token in the most recent inference results to the total number of most recent inferences, the comprehensive stability result is the result obtained by combining the consistency result and the stability result according to a preset weight, and the confirmation condition includes that the comprehensive stability result is not lower than the confirmation threshold.

[0017] Preferably, the fast group includes a session identifier, group type, group sequence number, timestamp, token start index, token end index, and the newly added temporary token; the confirmation group includes a session identifier, group type, group sequence number, timestamp, the current confirmation anchor point, and the correction information; the correction information is obtained by comparing the differences between the temporary token obtained in this inference and the sent temporary token at each token position in the unconfirmed interval, and the difference positions and the corresponding replacement tokens are recorded.

[0018] Preferably, when writing the audio segment to be played into the audio queue, a planned playback time is set for the audio segment to be played. The planned playback time is determined by the enqueue time of the audio segment to be played and the playback hold duration. The playback hold duration is determined based on the upper bound of network jitter, the maximum arrival delay of acknowledgment packets, and the reserved time for rebuilding the audio segment to be played.

[0019] Preferably, the mapping relationship includes a token range, an audio range, and a status identifier; after the receiving end replaces the corresponding temporary token in the temporary token buffer according to the correction information, it finds the audio segment to be played that intersects with the replaced temporary token and whose planned playback time has not yet arrived according to the mapping relationship, deletes the audio segment to be played, and generates a replaced audio segment to be played based on the replaced temporary token and writes it back to the audio queue to be played.

[0020] Preferably, the step of generating the audio segment to be played based on the newly added temporary token that has not yet been synthesized includes: inputting the newly added temporary token into a token embedding layer to obtain a continuous vector sequence; inputting the continuous vector sequence into a causal spectrum decoder to obtain a spectrum frame sequence; and inputting the spectrum frame sequence into a vocoder to obtain the audio segment to be played.

[0021] In another aspect, this application also provides a low-latency voice communication system based on an end-to-end large model, comprising:

[0022] Includes the sending end and the receiving end;

[0023] The transmitting end is used to collect the voice waveform of the transmitting end, construct the current input features and input them into the streaming voice communication model to obtain temporary tokens and stability results;

[0024] The sending end is also used to determine the current confirmation anchor point based on the multiple inference results in the historical inference record and the stability result, and when the current confirmation anchor point moves forward, generate correction information based on the difference between the temporary token obtained in this inference and the sent temporary token in the unconfirmed interval;

[0025] The sending end is also used to send fast packets and send confirmation packets when the current confirmation anchor point moves forward. The fast packets carry a newly added temporary token, and the confirmation packets carry the current confirmation anchor point and correction information.

[0026] The receiving end is used to write the newly added temporary token into the temporary token buffer after receiving the fast packet, generate the audio segment to be played based on the newly added temporary token that has not yet been synthesized, write the audio segment to be played into the audio queue to be played, and record the mapping relationship between the token range and the audio range corresponding to the newly added temporary token.

[0027] The receiving end is also configured to, after receiving the confirmation packet, write the temporary token up to the current confirmation anchor point into the confirmation token buffer, replace the corresponding temporary token in the temporary token buffer according to the correction information, and reconstruct the affected audio segment to be played according to the mapping relationship;

[0028] The receiving end is also used to output the corresponding audio segment to be played when the audio segment to be played in the audio queue reaches the planned playback time.

[0029] Compared with existing technologies, this invention determines the confirmation anchor point at the sending end based on historical inference records and stability results, and uses fast grouping and confirmation grouping in combination to ensure that new temporary tokens are sent in a timely manner. At the same time, it makes targeted corrections to the content that has changed in the unconfirmed interval, thus balancing the real-time performance and result stability of voice communication. Furthermore, at the receiving end, by establishing a mapping relationship between token intervals and audio intervals, and combining the audio queue to be played, the planned playback time, and the affected segment reconstruction mechanism, it only replaces and reconstructs the audio segments to be played that have not yet been played and are indeed affected. This avoids the overall rollback or large-scale repeated synthesis of the generated audio, which helps to reduce end-to-end playback latency, reduce the impact of network jitter and confirmation information delay on playback continuity, and improve the consistency, smoothness, and system implementation efficiency of voice output. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0032] Figure 1 This is a schematic diagram of a low-latency voice communication method based on an end-to-end large model, provided in an embodiment of this application.

[0033] Figure 2 This is a schematic diagram illustrating the stability results and temporary token acquisition process provided in the embodiments of this application.

[0034] Figure 3 This is a schematic diagram of the correction information flow provided in the embodiments of this disclosure.

[0035] Figure 4 This is a schematic diagram of the process for generating an audio segment to be played, provided in an embodiment of this disclosure.

[0036] Figure 5 This is a schematic diagram of a low-latency voice communication system based on an end-to-end large model, provided as an embodiment of this application.

[0037] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this application are information and data authorized by the user or fully authorized by all parties.

[0040] This implementation is applicable to scenarios such as two-way voice calls, multi-person conference voice links, real-time voice translation links, and voice assistant call links with dialogue interaction. The execution entities include at least a sender and a receiver, which can communicate via network forwarding nodes. The forwarding nodes are only responsible for packet forwarding and do not participate in token reasoning, confirmation anchor determination, or voice reconstruction. The sender has voice acquisition, streaming reasoning, and packet encapsulation capabilities, while the receiver has packet rearrangement, token caching, voice reconstruction, and delayed playback capabilities. During the session establishment phase, the sender and receiver pre-exchange processing block length, right-side context length, confirmation decision window length, and pending playback configurations. After completing the above negotiation, they enter a continuous voice communication state.

[0041] The implementation process of the low-latency voice communication method based on an end-to-end large model described in this application will be explained in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the protection scope of this application.

[0042] like Figure 1 As shown in the figure, this application discloses a schematic diagram of a low-latency voice communication method based on an end-to-end large model, which includes the following steps:

[0043] S1: Acquire the speech waveform from the transmitting end, construct the current input features and input them into the streaming speech communication model to obtain the temporary token and stability results;

[0044] S2, based on the multiple reasoning results in the historical reasoning record and the stability result, determine the current confirmation anchor point, and when the current confirmation anchor point moves forward, generate correction information based on the difference between the temporary token obtained in this reasoning and the sent temporary token in the unconfirmed interval;

[0045] S3, send a fast packet and send an acknowledgment packet when the current acknowledgment anchor point moves forward. The fast packet carries the newly added temporary token, and the acknowledgment packet carries the current acknowledgment anchor point and correction information.

[0046] S4. After receiving the fast packet, the receiving end writes the new temporary token into the temporary token buffer, generates the audio segment to be played based on the new temporary token that has not yet been synthesized, writes the audio segment to be played into the audio queue to be played, and records the mapping relationship between the token range and the audio range corresponding to the new temporary token.

[0047] S5, after receiving the confirmation packet, the receiving end writes the temporary token up to the current confirmation anchor point into the confirmation token buffer, replaces the corresponding temporary token in the temporary token buffer according to the correction information, and reconstructs the affected audio segment to be played according to the mapping relationship; when the audio segment to be played in the audio queue reaches the planned playback time, the receiving end outputs the corresponding audio segment to be played.

[0048] According to embodiments of this disclosure, the transmitting end pre-establishes a feature buffer, a historical encoding state buffer, a historical inference record, a temporary token sequence, and a sent temporary token sequence; the receiving end correspondingly establishes a temporary token buffer, an acknowledgment token buffer, a queue of audio to be played, and a mapping table between token intervals and audio intervals. The temporary token sequence stores discrete speech representations that the transmitting end has inferred but have not yet fully acknowledged, and the sent temporary token sequence stores discrete speech representations that have been sent with fast packets. The temporary token buffer stores discrete speech representations received by the receiving end but not yet fully acknowledged, and the acknowledgment token buffer stores discrete speech representations that have been defined by acknowledgment anchors and will not be replaced subsequently.

[0049] The transmitting end deploys a streaming speech communication model after the input feature processing link, which serves as a preprocessing stage independent of the model itself. The streaming speech communication model, arranged according to its connection relationships, includes a temporal downsampling module, a streaming coding module, a token projection module, and a stability estimation module. The speech reconstruction model is deployed at the receiving end and includes a token embedding layer, a causal spectrum decoder, and a vocoder. The transmitting and receiving models can be trained separately within the same training framework or obtained independently through different training processes, as long as the temporary tokens output by the transmitting end can be consistently interpreted by the receiving end's speech reconstruction model.

[0050] In some embodiments, the training of the transmitting model can be completed offline. Specifically, a token projection backbone is first trained with a large number of speech samples, enabling the model to learn the mapping relationship from speech features to discrete speech representations. The target tokens in the training samples can be extracted by a pre-trained neural codec. Subsequently, the same speech sample is replayed using a streaming block approach, repeatedly feeding the same speech segment into the model under different right-hand context conditions, and recording whether the tokens at the same logical position change, thereby constructing the supervision signal required by the stability estimation module.

[0051] The stability estimation module receives high-level encoded representations, the current token posterior distribution, and consistency features of neighboring block overlap positions during training, outputting the probability that the logical position remains unchanged in subsequent inference. The receiver speech reconstruction model can be trained independently, with the target token used by the transmitter model as its training input and the corresponding spectrum frame and speech waveform as its training output. The token embedding layer maps discrete indices to continuous vectors, the causal spectrum decoder outputs spectrum frames in temporal order, and the vocoder restores the spectrum frames to a time-domain waveform. The transmitter and receiver maintain alignment in the discrete token space, allowing the receiver to generate the corresponding local audio based on local tokens without waiting for the end of the entire sentence.

[0052] In some embodiments, for step S1, such as Figure 2 As shown, Figure 2 This diagram illustrates the stability results and temporary token acquisition process provided in this application embodiment. In step S201, the sending end continuously samples the speech waveform and constructs the current input features based on the device's sampling capability.

[0053] After the session begins, the transmitting end continuously samples the speech waveform according to the device's sampling capabilities; for example, mono sampling can be used, with the sampling rate configured within the range commonly used in voice communication scenarios. The acquired speech waveform enters the input feature processing link, which first performs framing and windowing processing on the waveform, then calculates the spectral features to obtain feature frames arranged in chronological order. Optionally, a log-Mel spectrum can be used as the feature frame, and the frequency band dimension of each frame can be set according to the model size. The feature frames output by the input feature processing link are first written to the feature buffer for subsequent block-by-block reading.

[0054] The sending end extracts the feature frame corresponding to the current processing block from the feature buffer, and appends the feature frame corresponding to the right context to the end of the buffer to form the current input features. The length of the current processing block is denoted as... The length of the right-hand context is denoted as Both are measured in time frames, which are calibrated offline by the sender before the session is established, based on device computing power, network latency budget, and model accuracy constraints. These values ​​remain unchanged after the session begins. If the device is in a long session scenario, the sender can also retransmit for new session rounds without interrupting the current session. and .

[0055] In step S202, the current input features are then fed into the streaming speech communication model for temporal downsampling and streaming coding.

[0056] The temporal downsampling module first performs temporal compression on the current input features. In one example, this module consists of two concatenated one-dimensional convolutional layers, each followed by a normalization layer and a non-linear activation layer. The convolutional stride is set to a preset value to reduce the temporal resolution, thus compressing the input feature frame sequence into a shorter feature sequence in the temporal dimension. The input to the temporal downsampling module is the current input features, and the output is the compressed feature sequence. This sequence has fewer time steps than the input feature frames but retains sufficient semantic information for subsequent encoding.

[0057] The compressed feature sequence is then fed into the streaming encoding module. In one implementation, the streaming encoding module employs an enhanced memory converter encoder architecture. This encoder contains multiple encoding units, each consisting of a local self-attention sublayer, a history memory retrieval sublayer, and a feedforward network sublayer, arranged sequentially from front to back. The local self-attention sublayer calculates attention weights within the current block, fusing features from different locations within the block; the history memory retrieval sublayer reads contextual information from the historical encoding state retained from the previous block, incorporating historical semantics into the representation of the current block; the feedforward network sublayer, composed of two fully connected layers and an activation function, performs a nonlinear transformation on the fused features.

[0058] The compressed feature sequence of the current block and the historical encoding state retained from the previous block are fed into this module. The module outputs the high-level encoding representation of the current block and updates the historical encoding state cache. The historical encoding state cache only saves state vectors that will still be used in subsequent inference. Old states that exceed the set retention length are overwritten by the sender and will not be included in subsequent calculations.

[0059] In step S203, the high-level encoded representation is fed into the token projection module to obtain a temporary token. This module first projects the high-level encoded representation onto the codebook space through a linear mapping layer, and then quantizes the continuous vector into a discrete index through a residual vector quantizer, outputting the temporary token.

[0060] The residual vector quantizer works as follows: The input vector is sequentially processed through multiple levels of codebooks, selecting nearest-neighbor codewords. Each level further quantizes the residual from the previous level, and the codeword indices at each level are combined to form the final discrete token. Each temporary token corresponds to a discrete codebook entry. The codebook is fixed during training and not updated during inference. The token projection module also retains the posterior distribution of the current temporary token. This posterior distribution is directly used as one of the inputs to the stability estimation module; it is neither written into the data packets nor sent to the receiver.

[0061] In step S204, the high-level encoded representation of the current block, the temporary token of the current block and its posterior distribution, and the statistical features related to the overlapping position of the current block in the historical inference records are input into the stability evaluation module to obtain the stability result.

[0062] In some embodiments, the stability assessment module consists of two layers of Multilayer Perceptrons (MLPs) and one layer of Gated Recurrent Units (GRUs). The two MLPs receive the encoded representation and posterior distribution of the current position and extract local discriminative features. The first layer concatenates the high-level encoded representation and posterior distribution and projects them onto the hidden space. The second layer performs further nonlinear transformations within the hidden space. The GRU processes the output of the MLPs along the logical position direction, aggregates the changing trends of adjacent tokens, and finally outputs the stability result token by token. The stability result can be represented as the stability probability at each logical position, ranging from 0 to 1. A higher value indicates a greater probability that the token at that position will remain unchanged in subsequent inference.

[0063] At the end of the current processing cycle, the sender simultaneously obtains a temporary token and a stability result. It writes the temporary token into the temporary token sequence and the stability result, along with the current inference timestamp and the start and end positions of the processing block, into the historical inference record. In a session-end scenario, the final confirmation anchor point can be determined based on the end position of the remaining unconfirmed area.

[0064] Optionally, to avoid duplicate counting of overlapping areas from different processing blocks in the temporary token sequence, the sender performs a token alignment operation before writing the temporary token sequence. Based on the time overlap between the current processing block and the previous processing block, the sender compares the temporary token obtained in the current inference with the temporary token at the corresponding position in the previous inference within the overlapping area. If the logical positions are the same and the token sequence lengths are aligned, only the additional length after the overlapping tail of the subsequent inference is retained; the overlapping portion is only used to update historical inference records and is not appended to the temporary token sequence. After the above processing, the temporary token sequence always represents the temporary result after deduplication according to logical position from the start of the session to the current time. Subsequent fast grouping uses the newly added tail in this sequence as the encapsulation object.

[0065] In some embodiments, for step S2, after obtaining the temporary token and stability result, the sender first calculates the current confirmation anchor point based on the historical inference record and stability result, instead of immediately treating all temporary tokens as the final commit content. The current confirmation anchor point indicates that the temporary tokens up to this logical position have met the confirmation conditions. Once the confirmation packet arrives at the receiver, the tokens before this position are transferred to the confirmation token buffer and will not participate in the replacement thereafter. The position of the confirmation anchor point moves monotonically forward during the session, and the sender will not backtrack the confirmed position to the unconfirmed area, thereby ensuring that the segments to be played that have been confirmed by the receiver are not modified repeatedly.

[0066] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the correction information flow provided in an embodiment of this disclosure. In step S301, the sending end extracts the most recent inference results corresponding to each current temporary token from the historical inference record.

[0067] The number of most recent inferences is denoted as That is, the length of the confirmation and judgment window, which is a positive integer. The values ​​are determined before model deployment based on the balance between false confirmation rate and confirmation latency, and are written into the session configuration after deployment, remaining unchanged during the session. If the historical records for a certain logical location are insufficient... If the result is the same, then all the existing reasoning results at that position will be used in the calculation, and the denominator will be adjusted accordingly.

[0068] The sender counts the number of times the current temporary token has remained consistent in recent inferences, and obtains the consistency result accordingly. Specifically, if the... The number of times each logical location is consistent across available records is: The total number of available records is Then the consistency result at that position Determine as follows:

[0069]

[0070] in For consistent results, For the same number of inferences as the current temporary token, This represents the number of inferences involved in the statistics. This calculation is only performed within this step; after the calculation is complete, and It will no longer proceed to subsequent stages separately; the sending end will only retain [the necessary information]. To participate in the calculation of the overall stability results.

[0071] In step S302, the sending end then combines the consistency result and the stability result into a comprehensive stable result.

[0072] To achieve a proper balance between consistency across inferences and the model's instantaneous stability output, weighting coefficients are introduced. The value range is 0-1. The values ​​are obtained from offline validation sets. During calibration, the development team calculates the false confirmation rate and average confirmation latency under multiple candidate values, and selects the value with the smaller average confirmation latency while satisfying the false confirmation rate constraint. This value remains fixed after the session starts and does not change with individual processing blocks. Overall stability results are obtained. From the consistency results Stability results and weighting coefficients Jointly determined:

[0073]

[0074] in Indicates the first The overall stable result of each logical position, This is the stability result output by the stability estimation module. This comprehensive stability result is directly used in subsequent confirmation condition determinations in this step and is not sent to the receiving end as a separate field.

[0075] In step S303, the continuous prefix confirmation and anchor point determination are determined by the combined constraints of the comprehensive stability results and the positional order.

[0076] The sending end maintains the position of the last acknowledgment anchor point. At the start of the current processing cycle, all temporary tokens after the last acknowledgment anchor point are in the unacknowledgment zone. The sending end starts from the first temporary token after the last acknowledgment anchor point and checks them one by one in logical order: if the overall stability result at a certain position is not lower than the acknowledgment threshold... If so, then the position meets the numerical judgment condition. Confirmation threshold. Similarly, the values ​​are obtained during the offline calibration phase, ranging from 0 to 1. The calibration objective focuses on controlling the false confirmation probability. The specific calibration method is similar to that of the weighting coefficient, and will not be elaborated further.

[0077] The transmitter identifies the longest consecutive prefix interval that meets the numerical criteria starting from the previous confirmation anchor point as the new confirmation interval, and the end of this prefix interval becomes the current confirmation anchor point. If any position fails to meet the numerical criteria, the transmitter immediately stops moving forward, and this position and its subsequent positions, even if they have a high overall stability result in this inference, are temporarily retained in the unconfirmed region. This sequential constraint ensures that the confirmation interval remains continuous, preventing the receiver from experiencing a break in the process where some parts are unconfirmed while others are confirmed later.

[0078] In step S304, when the current confirmed anchor point position is greater than the previous confirmed anchor point, the sending end generates correction information.

[0079] The comparison range for correction information is limited to the unconfirmed interval. The temporary token obtained in this inference is compared position-by-position with the previously sent temporary tokens. The previously sent temporary tokens come from a sequence of sent temporary tokens stored at the sender, which is updated synchronously after the fast packet is sent. If the current temporary token at a certain logical position is inconsistent with the previously sent temporary token, the sender records that logical position and the replaced temporary token, forming a correction information entry. All difference entries are arranged in ascending order of logical position, which constitutes the correction information for the current processing cycle.

[0080] No entries are generated for locations where no differences have occurred; for portions that have already entered the new confirmation zone, the sender uses the current confirmation anchor point to uniformly express their status of entering the confirmation zone. Therefore, correction information only covers locations still in the unconfirmed zone where the content has indeed changed, neither repeating descriptions of confirmed locations nor limiting the receiver's subsequent reconstruction to the affected segments to be played.

[0081] In some embodiments, for step S3, after the sending end completes the determination of the acknowledgment anchor point and the generation of correction information, it constructs fast packets and acknowledgment packets according to the packetization rules. Fast packets are responsible for the immediate transmission of newly added temporary tokens, while acknowledgment packets are responsible for transmitting acknowledgment boundaries and partial replacement information. The two types of packets have different contents, but are both based on the same session identifier and the same logical index system. The receiving end processes them separately within the same buffer system accordingly.

[0082] Fast packet encapsulation begins with the addition of a new ephemeral token. The new ephemeral token refers to the tail portion of the ephemeral token sequence that has not yet been sent via fast packet. The sender determines the starting point based on the end position of the already sent ephemeral token sequence and extracts the corresponding new tail portion from the ephemeral token sequence as the fast packet payload.

[0083] Each fast packet must contain at least a session identifier, packet type, packet sequence number, timestamp, token start index, token end index, and a newly added temporary token. The session identifier corresponds to the current voice session, the packet type indicates that the packet belongs to the fast packet category, the packet sequence number monotonically increases in the order of transmission, the timestamp corresponds to the inference time of the first position of the newly added temporary token, and the token start index and token end index define the position range of the payload within the logical token sequence. Fast packets are transmitted immediately after formation. After transmission, the newly added temporary token is synchronously appended to the sequence of transmitted temporary tokens, serving as the transmitting-side reference for subsequent correction information comparison.

[0084] If the current acknowledgment anchor point has moved forward compared to the previous acknowledgment anchor point, the sender constructs an acknowledgment packet. The acknowledgment packet must contain at least the session identifier, packet type, packet sequence number, timestamp, current acknowledgment anchor point, and correction information. The correction information has already been obtained in the previous step and will not be re-compared here. If there are no discrepancies in the current processing cycle, the correction information is empty, and the acknowledgment packet is still sent to notify the receiver that the acknowledgment boundary has moved forward; if the correction information is not empty, the acknowledgment packet carries both the acknowledgment boundary and the replacement position of the unacknowledgment area.

[0085] The sender can send fast packets and acknowledgment packets corresponding to the same round of inference at close intervals, but logically they are not interchangeable. The receiver must process them separately according to their respective packet types. There is no single limitation on the packet transmission protocol; a reliability-enhanced real-time transmission protocol can be used, or a custom header format can be carried on top of an existing communication protocol, as long as the receiver can parse the fields in the packet.

[0086] In some implementations, the sender can include the start and end indices of the affected unacknowledged segments in the acknowledgment packet, allowing the receiver to quickly locate the affected segments. This additional field does not affect the main process; if omitted, the receiver can still locate the affected segments one by one based on the correction information.

[0087] In some embodiments, for step S4, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the audio segment generation process provided in an embodiment of this disclosure. In step S401, after receiving the fast packet, the receiving end first completes the packet parsing and reordering, and then writes the newly added temporary token into the temporary token buffer.

[0088] The receiving end determines the write position of the newly added temporary tokens in the temporary token buffer based on the packet sequence number, token start index, and token end index in the fast packets. If the fast packets arrive in the order they were sent, the receiving end directly writes the new temporary tokens to the corresponding positions; if the fast packets arrive out of order, the receiving end first checks for missing intervals based on the packet sequence number, rearranges them provided there are no logical conflicts, and then triggers subsequent voice reconstruction. The rearrangement process itself does not change the token content, only adjusts the writing order at the receiving end.

[0089] In step S402, after the writing is completed, the receiving end generates the audio segment to be played based on the newly added temporary tokens that have not yet been synthesized. The speech reconstruction model processes the newly added temporary tokens sequentially at the receiving end.

[0090] In this process, the token embedding layer first queries the embedding table, mapping each discrete token to a continuous vector of fixed dimension. The size of the embedding table is equal to the codebook vocabulary multiplied by the embedding dimension. The resulting continuous vector sequence then enters the causal spectrum decoder. In one implementation, the causal spectrum decoder consists of alternating stacks of multiple causal self-attention layers and pointwise feedforward layers. Each causal self-attention layer only accesses the embedding features of the current and previous positions, without using information from future positions, thus ensuring that the decoding process can be executed progressively in a streaming scenario.

[0091] The attention mechanism within the causal self-attention layer is consistent with standard multi-head self-attention, the difference being that the attention mask restricts each position to only focusing on itself and the positions preceding it. The pointwise feedforward layer consists of two fully connected layers and an activation function, performing further nonlinear transformations on the attention output. The output of the causal spectral decoder is a time-aligned sequence of spectral frames.

[0092] The spectral frame sequence is further fed into a vocoder. The vocoder can employ a causal version of a generative adversarial network vocoder, such as a streaming variant of a high-fidelity generative adversarial network vocoder. Its generator consists of multi-scale deconvolutional upsampling layers and residual convolutional blocks. The multi-scale deconvolutional upsampling layers progressively increase the temporal resolution of the spectral frames to the waveform sampling rate, and the residual convolutional blocks refine the features after each upsampling.

[0093] The discriminator only participates in adversarial training during the training phase and is not used during inference. During the training phase, the receiver model uses the real waveform corresponding to the token as a supervision signal to jointly optimize the spectrum reconstruction loss and waveform perception loss. After deployment, the receiver only retains the token embedding layer, the causal spectrum decoder, and the generator network, thereby converting newly added temporary tokens into audio segments to be played.

[0094] In step S403, the receiving end does not play the audio segment to be played immediately after it is generated, but writes it into the audio queue to be played and sets a scheduled playback time for each audio segment to be played.

[0095] The planned playback time equals the enqueue time of the segment plus the waiting time. The waiting time is denoted as... This corresponds to the buffer time that the receiving end retains before the actual output, the purpose of which is to allow the confirmation group and the local reconstruction triggered by it to be completed before the actual playback action occurs.

[0096] The upper bound of network jitter is determined based on the upper bound of network jitter, the maximum arrival delay of acknowledgment packets, and the reconstruction allowance for the audio segment to be played. The upper bound of network jitter is denoted as... The maximum arrival delay of the confirmed packet is The reconstruction time is reserved for ,but:

[0097]

[0098] in The jitter upper bound is estimated by the receiver based on the statistics of packet arrival intervals over a recent period. Optionally, a sliding window percentile estimation method can be used, which sorts the packet arrival intervals within a sliding time window and takes a set percentile value as the estimate. It is provided by the sending end according to the confirmation anchor point formation strategy when the session is established, and stored by the receiving end after receiving it; The time factor is determined by the receiver's average reconstruction time on the current device based on the local speech reconstruction model, plus a safety margin. The receiver can update this value periodically. This will then update the new induction footage. However, segments that have already been queued will continue to play at their original scheduled times to avoid disrupting the order within the queue.

[0099] In step S404, while writing to the audio queue to be played, the receiving end records the mapping relationship between the token range and the audio range corresponding to the newly added temporary token. The mapping table includes at least the token start index, token end index, audio sample start offset, audio sample end offset, and status identifier.

[0100] In this implementation, the status indicators are divided into three types: pending confirmation, confirmed and ready for playback, and invalidated. Pending confirmation indicates that the segment is still after the confirmation anchor point and may be affected by correction information later. Confirmed and ready for playback indicates that the token interval corresponding to the segment has entered before the confirmation anchor point and will not be replaced later. Invalidated indicates that the segment has been replaced by a new reconstructed segment. The mapping table is responsible for segment location after subsequent confirmation groups arrive; its write time must be earlier than the confirmation group processing time to maintain consistency with the processing order. The mapping table can also determine the planned playback time inherited by the replaced audio segment during segment merging and reconstruction.

[0101] In some embodiments, for step S5, after receiving the acknowledgment packet, the receiving end first reads the current acknowledgment anchor point and correction information from it, and then updates the acknowledgment token buffer and the audio queue to be played in two stages.

[0102] The first phase handles the confirmation boundary. The receiving end writes the temporary tokens from the temporary token buffer up to the current confirmation anchor point into the confirmation token buffer. The corresponding positions can retain mirrored content in the temporary token buffer, but the status is no longer considered pending confirmation. If the audio segments to be played corresponding to these token intervals have not yet been actually played, the receiving end updates their status to confirmed and pending playback in the mapping table. This phase does not replace the content of confirmed and pending playback segments; it only changes their status as whether they can be reconstructed later.

[0103] The second stage processes the correction information. If the correction information is not empty, the receiving end updates the temporary token buffer line by line according to the difference positions and replacement tokens in the correction information. The updated temporary tokens remain in the temporary token buffer as the current valid value of the unacknowledged area. The receiving end then uses the mapping table to find the audio segment to be played that simultaneously meets the following two conditions: first, the token interval of the segment covers at least one replaced logical position (i.e., there is an intersection between the token interval and the set of replaced positions); second, the scheduled playback time of the segment has not yet arrived. The former condition ensures that the reconstructed object is related to the correction information, and the latter condition ensures that the reconstruction occurs before actual playback, and will not cause timing conflicts between the output audio and the control operation.

[0104] The receiving end marks audio segments that meet the above conditions as invalid and removes them from the audio queue. Then, using the replaced temporary token as input, it re-invokes the speech reconstruction model to generate the replaced audio segment, writes it back to the queue position corresponding to its original logical location, and updates the audio interval and status flag in the mapping table. If the replaced audio segment covers multiple deleted segments, its planned playback time inherits the earliest planned playback time among the covered segments, and it is written back to the audio queue in the order corresponding to that time.

[0105] When the correction information only covers a small part of the unconfirmed area, the receiver does not need to reconstruct the entire unconfirmed area; it can select only the smallest continuous segment that intersects with the difference location to perform reconstruction.

[0106] Specifically, the receiving end first merges adjacent and affected unacknowledged segments in the mapping table to obtain the smallest continuous audio interval to be reconstructed. Then, it calls the speech reconstruction model based on the smallest continuous token interval corresponding to this audio interval. The planned playback time of the replaced audio segment is inherited from the earliest planned playback time of the covered segments, thus keeping the reconstruction burden within the affected range. After the receiving end completes the partial replacement, the original unacknowledged segment becomes invalid, and the new segment replaces its position in the audio queue to be played. Subsequent playback threads only read the content of the new segment.

[0107] In some embodiments, the receiving end sets up a separate playback thread to continuously check the head segment of the audio queue to be played. When the current time reaches the scheduled playback time of the segment and the segment's status is not invalid, the playback thread sends the segment to the speaker output. If the segment has previously undergone partial replacement, the playback thread reads the replaced audio segment to be played; if the segment has been marked as confirmed for playback before the arrival of the acknowledgment packet, it is played directly according to the original plan. Since the playback hold time already covers the maximum arrival delay of the acknowledgment packet and the local reconstruction reservation time, under normal communication conditions, acknowledgment packet processing and partial replacement are completed before the scheduled playback time, and the actual output does not need to be rolled back after playback occurs.

[0108] The sending end continuously repeats the process of constructing current input features, generating temporary tokens, determining confirmation anchors, and sending packets; the receiving end continuously repeats the process of writing fast packets, generating audio segments to be played, replacing confirmation packets, and scheduling playback. At the end of the session, after the last round of inference, the sending end performs confirmation anchor determination again on the remaining unconfirmed area, and uses the end of the session as the final confirmation condition, determining the end position of the remaining unconfirmed area as the final confirmation anchor, and sending the final confirmation packet.

[0109] After receiving the final acknowledgment packet, the receiving end updates the remaining unacknowledged areas up to the final acknowledgment anchor point to the acknowledgment token buffer, completing the output of the remaining audio segments to be played in the queue. If the session has stopped acquiring data before the final acknowledgment packet arrives, the receiving end will still consume the remaining undiscarded audio segments in the queue according to the original scheduled playback time order until the audio queue is empty.

[0110] In some embodiments, if network jitter increases significantly, the receiver can increase the hold-up time for newly enqueued segments to provide a more ample arrival window for acknowledgment packets; if the network returns to stability, the hold-up time for subsequent segments can be reduced to decrease end-to-end call latency. This adjustment only applies to segments enqueued afterward; segments with currently scheduled playback times remain on their original schedules.

[0111] On the sending end, if no new current confirmation anchor point is generated in multiple consecutive rounds of inference, the sending end continues to send fast packets, while confirmation packets are only sent when the current confirmation anchor point moves forward. If the correction information corresponding to this confirmation packet is empty, the confirmation packet is only used to notify that the confirmation boundary has moved forward. The transmission of temporary tokens and the transmission of confirmation boundaries are always distinguished, so that the receiving end can complete the pre-play construction and partial replacement operations according to the two types of packets.

[0112] In some embodiments, the sender further compresses and encodes the difference entries between the temporary token and the sent temporary token before writing them into the acknowledgment packet. The compression method can be adjacent position merging or run-length encoding, but the logical position expressed by the correction information and the replacement token remain unchanged before and after compression. After parsing the acknowledgment packet, the receiver first decompresses it and then updates the temporary token buffer according to the aforementioned process.

[0113] In some embodiments, at the model deployment level, the streaming voice communication model at the sending end and the voice reconstruction model at the receiving end can be deployed and executed on different hardware platforms. The sending end mainly undertakes streaming inference and anchor point determination, and is suitable for running on a mobile neural network inference engine; the receiving end mainly undertakes local reconstruction from token to audio, and is suitable for execution on the receiving terminal or near-end edge nodes. If the receiving end voice reconstruction model is deployed on an edge node in some scenarios, the edge node is still logically part of the receiving end processing chain, and its output audio segment to be played is then transmitted to the final speaker device, with the playback duration and confirmation packet replacement order remaining unchanged.

[0114] Through the above implementation process, the sending end breaks down each round-robin inference into two consecutive actions: temporary token generation and confirmation anchor point determination. The two actions are then mapped to fast packets and confirmation packets, respectively. The receiving end first generates and buffers the audio segments to be played based on the fast packets. Then, before the scheduled playback time arrives, it replaces the affected audio segments to be played based on the confirmation packets. The tokens before the confirmation anchor point enter the confirmation token buffer.

[0115] In low-latency voice communication links, the temporary tokens generated in real time are processed separately from the subsequent confirmation boundaries. The receiving end maintains two states: the area to be confirmed and the area already confirmed. By mapping the token interval to the audio interval, only the affected audio segments to be played are reconstructed. This reduces the voice distortion caused by inference instability while reducing the end-to-end call latency from the control end. It realizes a low-latency voice communication process of first generating the audio to be played and then performing local replacement according to the confirmation result.

[0116] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0117] Please see Figure 5 , Figure 5 This application provides a low-latency voice communication system based on an end-to-end large model. The system embodiments are similar to... Figure 1 Corresponding to the illustrated method embodiments, this system can be specifically applied to various electronic devices. The system specifically includes:

[0118] Includes the sending end and the receiving end;

[0119] The transmitting end is used to collect the voice waveform of the transmitting end, construct the current input features and input them into the streaming voice communication model to obtain temporary tokens and stability results;

[0120] The sending end is also used to determine the current confirmation anchor point based on the multiple inference results in the historical inference record and the stability result, and when the current confirmation anchor point moves forward, generate correction information based on the difference between the temporary token obtained in this inference and the sent temporary token in the unconfirmed interval;

[0121] The sending end is also used to send fast packets and send confirmation packets when the current confirmation anchor point moves forward. The fast packets carry a newly added temporary token, and the confirmation packets carry the current confirmation anchor point and correction information.

[0122] The receiving end is used to write the newly added temporary token into the temporary token buffer after receiving the fast packet, generate the audio segment to be played based on the newly added temporary token that has not yet been synthesized, write the audio segment to be played into the audio queue to be played, and record the mapping relationship between the token range and the audio range corresponding to the newly added temporary token.

[0123] The receiving end is also configured to, after receiving the confirmation packet, write the temporary token up to the current confirmation anchor point into the confirmation token buffer, replace the corresponding temporary token in the temporary token buffer according to the correction information, and reconstruct the affected audio segment to be played according to the mapping relationship;

[0124] The receiving end is also used to output the corresponding audio segment to be played when the audio segment to be played in the audio queue reaches the planned playback time.

[0125] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0126] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0127] Figure 6 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input section 606, an output section 607, a communication section 609, and an input / output (I / O) interface 605 are also connected to the bus 604.

[0128] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 601, it performs the functions defined in the methods of this application.

[0129] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0130] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A low-latency voice communication method based on an end-to-end large model, characterized in that, include: Collect the speech waveform from the transmitting end, construct the current input features and input them into the streaming speech communication model to obtain temporary tokens and stability results; The streaming speech communication model includes, in sequence, a temporal downsampling module, a streaming coding module, a token projection module, and a stability estimation module. The temporal downsampling module receives the current input features and outputs a compressed feature sequence. The streaming coding module uses an enhanced memory converter encoder architecture to receive the compressed feature sequence and output a high-level coded representation. The token projection module uses a residual vector quantizer to output a temporary token based on the high-level coded representation. The stability estimation module is composed of two layers of multilayer perceptron (MLP) and one layer of gated recurrent unit (GRU), and outputs the stability result based on the high-level coded representation and the temporary token. Based on multiple inference results in the historical inference record and the stability result, the current confirmation anchor point is determined. When the current confirmation anchor point is moved forward, correction information is generated based on the difference between the temporary token obtained in this inference and the previously sent temporary tokens within the unconfirmed interval. Determining the current confirmation anchor point includes: extracting multiple inference results corresponding to each temporary token from the historical inference record to obtain the consistency result of each temporary token; obtaining the comprehensive stability result of each temporary token based on the consistency result and the corresponding stability result; starting from the temporary tokens after the last confirmation anchor point, determining a prefix interval that continuously satisfies the confirmation condition according to the token order, and determining the end position of the prefix interval as the current confirmation anchor point; wherein the current confirmation anchor point indicates that the temporary tokens up to the end position have satisfied the confirmation condition; moving the current confirmation anchor point forward means determining the end position of the prefix interval that continuously satisfies the confirmation condition backward according to the token order, starting from the temporary tokens after the last confirmation anchor point. Send fast packets and send acknowledgments when the current acknowledgment anchor point moves forward. Fast packets carry the newly added temporary token, and acknowledgments carry the current acknowledgment anchor point and correction information. After receiving the fast packet, the receiving end writes the new temporary token into the temporary token buffer, generates the audio segment to be played based on the new temporary token that has not yet been synthesized, writes the audio segment to be played into the audio queue to be played, and records the mapping relationship between the token range and the audio range corresponding to the new temporary token. After receiving the confirmation packet, the receiving end writes the temporary token up to the current confirmation anchor point into the confirmation token buffer, replaces the corresponding temporary token in the temporary token buffer according to the correction information, and reconstructs the affected audio segment to be played according to the mapping relationship; when the audio segment to be played in the audio queue reaches the planned playback time, the receiving end outputs the corresponding audio segment to be played.

2. The low-latency voice communication method based on an end-to-end large model according to claim 1, characterized in that, The construction of the current input features includes: performing frame segmentation and feature extraction on the speech waveform to obtain feature frames arranged in chronological order; extracting the feature frame corresponding to the current processing block from the feature buffer and concatenating it with the feature frame corresponding to the right context to obtain the current input features.

3. The low-latency voice communication method based on an end-to-end large model according to claim 2, characterized in that, The consistency result is the proportion of the number of times the current temporary token is the same as the current temporary token in the most recent inference results. The overall stability result is the result obtained by combining the consistency result and the stability result according to a preset weight. The confirmation condition includes that the overall stability result is not lower than the confirmation threshold.

4. The low-latency voice communication method based on an end-to-end large model according to claim 3, characterized in that, The fast group includes a session identifier, group type, group sequence number, timestamp, token start index, token end index, and the newly added temporary token; the confirmation group includes a session identifier, group type, group sequence number, timestamp, the current confirmation anchor point, and the correction information; The correction information is obtained by comparing the differences between the temporary token obtained in this inference and the sent temporary token at each token position in the unconfirmed interval, and the difference position and the corresponding replacement token are recorded.

5. The low-latency voice communication method based on an end-to-end large model according to claim 1, characterized in that, When the audio segment to be played is written into the audio queue to be played, a planned playback time is set for the audio segment to be played. The planned playback time is determined by the enqueue time of the audio segment to be played and the holding time for playing. The holding time for playing is determined based on the upper bound of network jitter, the maximum arrival delay of acknowledgment packets, and the reserved time for rebuilding the audio segment to be played.

6. The low-latency voice communication method based on an end-to-end large model according to claim 5, characterized in that, The mapping relationship includes a token range, an audio range, and a status identifier. After the receiving end replaces the corresponding temporary token in the temporary token buffer according to the correction information, it searches for an audio segment to be played that simultaneously meets the following two conditions according to the mapping relationship: first, the token range of the segment covers at least one replaced logical position, that is, there is an intersection between the token range and the set of replaced positions; second, the planned playback time of the segment has not yet arrived. The audio segment to be played is then deleted, and a replaced audio segment to be played is generated according to the replaced temporary token and written back to the audio queue to be played.

7. The method according to claim 1, characterized in that, The step of generating the audio segment to be played based on the newly added temporary token that has not yet been synthesized includes: inputting the newly added temporary token into a token embedding layer to obtain a continuous vector sequence; inputting the continuous vector sequence into a causal spectrum decoder to obtain a spectrum frame sequence; and inputting the spectrum frame sequence into a vocoder to obtain the audio segment to be played.

8. A low-latency voice communication system based on an end-to-end large model, characterized in that, include: Sender and receiver; The transmitting end is used to collect the voice waveform of the transmitting end, construct the current input features and input them into the streaming voice communication model to obtain temporary tokens and stability results; The streaming speech communication model includes, in sequence, a temporal downsampling module, a streaming coding module, a token projection module, and a stability estimation module. The temporal downsampling module receives the current input features and outputs a compressed feature sequence. The streaming coding module uses an enhanced memory converter encoder architecture to receive the compressed feature sequence and output a high-level coded representation. The token projection module uses a residual vector quantizer to output a temporary token based on the high-level coded representation. The stability estimation module is composed of two layers of multilayer perceptron (MLP) and one layer of gated recurrent unit (GRU), and outputs the stability result based on the high-level coded representation and the temporary token. The sending end is further configured to determine the current confirmation anchor point based on multiple inference results in the historical inference record and the stability result, and to generate correction information based on the difference between the temporary token obtained in this inference and the sent temporary token in the unconfirmed interval when the current confirmation anchor point is moved forward; wherein, determining the current confirmation anchor point includes: extracting multiple inference results corresponding to each temporary token from the historical inference record to obtain the consistency result of each temporary token; obtaining the comprehensive stability result of each temporary token based on the consistency result of each temporary token and the corresponding stability result; starting from the temporary tokens after the last confirmation anchor point, determining the prefix interval that continuously satisfies the confirmation condition according to the token order, and determining the end position of the prefix interval as the current confirmation anchor point; wherein the current confirmation anchor point indicates that the temporary tokens up to the end position have satisfied the confirmation condition; the moving forward of the current confirmation anchor point means determining the end position of the prefix interval that continuously satisfies the confirmation condition backward according to the token order starting from the temporary tokens after the last confirmation anchor point; The sending end is also used to send fast packets and send confirmation packets when the current confirmation anchor point moves forward. The fast packets carry a newly added temporary token, and the confirmation packets carry the current confirmation anchor point and correction information. The receiving end is used to write the newly added temporary token into the temporary token buffer after receiving the fast packet, generate the audio segment to be played based on the newly added temporary token that has not yet been synthesized, write the audio segment to be played into the audio queue to be played, and record the mapping relationship between the token range and the audio range corresponding to the newly added temporary token. The receiving end is also configured to, after receiving the confirmation packet, write the temporary token up to the current confirmation anchor point into the confirmation token buffer, replace the corresponding temporary token in the temporary token buffer according to the correction information, and reconstruct the affected audio segment to be played according to the mapping relationship; The receiving end is also used to output the corresponding audio segment to be played when the audio segment to be played in the audio queue reaches the planned playback time.

Images