Text segmentation voice streaming processing method, system and device, medium and program product

Abstract

The invention discloses a text segmentation voice streaming processing method, system and device, a medium and a program product, and the method comprises the steps: firstly, carrying out segmentation processing according to a semantic structure of a long text, and obtaining ordered text segments with complete semantics; speech synthesis is carried out on the first text segment to obtain a first audio segment, and SSE connection with the client is established while the first audio segment is synthesized; after the synthesis of the first audio clip is completed, the first audio clip is immediately transmitted to a segmentation buffer queue of the client through SSE connection, and meanwhile, speech synthesis is sequentially performed on subsequent text clips; and after the segmented buffering queue receives the first audio clip, the first audio clip is played immediately, and subsequent audio clips are continuously received and buffered while the first audio clip is played. According to the invention, the coordinated control of the AI speech synthesis progress and the streaming transmission progress is realized, the response speed of AI speech synthesis is obviously improved, and the switching fluency of segmented audios is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of large-scale model technology, and in particular to a text segmentation and speech streaming processing method, system, device, medium, and program product. Background Technology

[0002] Existing AI speech synthesis technology faces key technical bottlenecks when processing long texts:

[0003] First, the existing AI speech synthesis technology requires the AI ​​model to complete the synthesis of the entire text before playback can begin, resulting in long waiting times for users.

[0004] While existing technologies employ a simple segmented composition model that can process data in segments, they still suffer from the following core problems:

[0005] 1. Difficulty in coordinating AI segmented synthesis and streaming delivery: AI model segmented synthesis has uneven time characteristics (simple segments are fast, complex segments are slow). Therefore, existing technologies cannot effectively coordinate the AI ​​synthesis progress with the streaming delivery progress, resulting in either waiting for all segments to be completed (losing the advantages of streaming) or transmitting in advance, affecting the synthesis quality.

[0006] 2. Issues with seamless playback of segmented audio: There are differences in audio parameters between the audio segments synthesized by AI, and existing technologies are prone to pauses, pops, and audio quality jumps when switching between segments.

[0007] 3. Inaccurate judgment of streaming completion status: Existing technologies lack a completion status detection mechanism for the characteristics of AI segmented synthesis, and cannot accurately judge the "AI synthesis completed" status and the "playback completed" status.

[0008] Therefore, there is an urgent need to invent a new AI speech processing method to solve the problems of inconsistency between the speech segmentation and synthesis progress and the streaming progress in existing technologies, unsmooth audio segment playback, and inability to accurately detect the completion status of streaming. Summary of the Invention

[0009] In view of this, embodiments of the present invention provide a text segmentation and speech streaming processing method, system, device, medium, and program product, which at least partially solves the problems existing in the prior art.

[0010] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0011] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0012] According to a first aspect of the present invention, a text segmentation and speech streaming processing method is provided, the method comprising:

[0013] The long text is segmented according to its semantic structure to obtain semantically complete and ordered text fragments;

[0014] The first text segment is processed into speech to obtain the first audio segment, and an SSE connection with the client is established simultaneously with the synthesis of the first audio segment;

[0015] After the first audio segment is synthesized, it is immediately transmitted to the segmented buffer queue of the client via the SSE connection, and the subsequent text segments are synthesized into speech in sequence.

[0016] Upon receiving the first audio segment, the segmented buffer queue immediately plays the first audio segment and continuously receives and buffers subsequent audio segments while playing the first audio segment.

[0017] Furthermore, based on the semantic structure of the long text, it is segmented to obtain semantically complete ordered text fragments, including:

[0018] The segmentation points of the long text are dynamically determined based on its semantic structure.

[0019] Based on the segmentation points, the long text is sequentially segmented into semantically complete ordered text fragments.

[0020] Furthermore, the audio segment includes audio metadata;

[0021] The audio metadata includes a segment identifier, a final segment flag, and a content type tag. The final segment flag is used to record whether the audio segment is the last audio segment.

[0022] Furthermore, the method also includes:

[0023] The audio segment is subjected to standardization processing, which includes volume normalization, sampling rate unification, and audio fade-in / fade-out processing.

[0024] Furthermore, the method also includes:

[0025] Based on the metadata of the audio segment, determine whether the current audio segment is the last audio segment;

[0026] If the current audio segment is the last audio segment and all audio segments have finished playing, a playback completion notification will be sent.

[0027] Furthermore, while playing the first audio segment, subsequent audio segments are continuously received and buffered, including:

[0028] While playing the first audio segment, subsequent audio segments are continuously received via the SSE connection and stored in the segmented buffer queue;

[0029] While playing the current audio segment, the next audio segment in the segmented buffer queue is preloaded.

[0030] According to a second aspect of the present invention, a text segmentation and speech streaming processing system is provided, the system comprising:

[0031] A semantic segmenter is used to segment long texts according to their semantic structure, resulting in semantically complete and ordered text fragments.

[0032] The speech synthesis module is used to synthesize speech on the first text segment to obtain the first audio segment, and while the first audio segment is transmitted to the segmented buffer queue of the client, it sequentially synthesizes speech on subsequent text segments;

[0033] The SSE transmission module is used to establish an SSE connection with the client while synthesizing the first audio segment, and immediately transmit the first audio segment to the client's segment buffer queue through the SSE connection after the first audio segment is synthesized.

[0034] A segmented audio streaming receiver is used to receive audio segments transmitted via the SSE connection and place the audio segments into a segmented buffer queue.

[0035] A playback controller is configured to immediately play the first audio segment after the segmented buffer queue receives the first audio segment, and continuously receive and buffer subsequent audio segments while playing the first audio segment.

[0036] According to a third aspect of the present invention, a text segmentation speech streaming device is provided, the device comprising: a processor and a memory;

[0037] The memory is used to store one or more program instructions;

[0038] The processor is configured to run one or more program instructions to perform the steps of a text segmentation speech streaming method as described in any of the preceding claims.

[0039] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of a text segmentation speech streaming process as described in any of the preceding claims.

[0040] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product comprising a computing program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to implement the steps of a text segmentation speech streaming processing method as described in any of the preceding claims.

[0041] This invention discloses a method, system, device, medium, and program product for text segmentation and speech streaming. The method includes: first, segmenting a long text according to its semantic structure to obtain semantically complete ordered text segments; then, synthesizing speech on the first text segment to obtain the first audio segment, and establishing an SSE connection with the client while synthesizing the first audio segment; immediately transmitting the first audio segment to the segmented buffer queue on the client via the SSE connection after the first audio segment is synthesized, and simultaneously synthesizing speech on subsequent text segments sequentially; after receiving the first audio segment, the segmented buffer queue immediately plays the first audio segment, and while playing the first audio segment, continuously receives and buffers subsequent audio segments. This invention achieves coordinated control of the AI ​​speech synthesis progress and the streaming transmission progress, significantly improving the response speed of AI speech synthesis and effectively enhancing the smoothness of segmented audio switching. Attached Figure Description

[0042] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a text segmentation and speech streaming processing method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of a text segmentation and speech streaming system provided in an embodiment of the present invention;

[0045] Figure 3 This is an interactive timing diagram of a text segmentation and speech streaming processing system provided in an embodiment of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] Figure 1 A flowchart of a text segmentation speech streaming method according to an embodiment of the present invention is shown.

[0049] like Figure 1 As shown, the text segmentation and speech streaming processing method according to an embodiment of the present invention may include steps S100, S200, S300 and S400.

[0050] To better illustrate the practical application effects of the technical solution of this invention, the following application scenario, in which a psychological expert AI assistant helps users solve psychological problems, demonstrates the application methods and technical characteristics of this invention in the field of mental health services.

[0051] A user sent a request to a mental health expert AI assistant: "I've been under a lot of work pressure lately, I often suffer from insomnia, and my relationship with my family has become strained. I feel like I'm about to break down and I don't know what to do. I hope you can help me analyze the reasons and give me some advice."

[0052] In step S100, the long text is segmented according to its semantic structure to obtain semantically complete ordered text segments.

[0053] Specifically, the above steps include:

[0054] After receiving a user's request for psychological counseling, the AI ​​psychological expert model generates a detailed response text, which includes problem analysis (approximately 150 words), explanation of psychological mechanisms (approximately 200 words), specific suggestions (approximately 180 words), practical methods (approximately 160 words), and encouragement and support (approximately 80 words).

[0055] Using an intelligent semantic segmenter, based on the semantic structure of the above-mentioned response text and combined with the professional characteristics of psychological counseling, the segmentation points of the response text are dynamically determined, dividing the response text into 5 semantically complete text segments. This ensures that each text segment is an independent and coherent psychological suggestion unit, while also taking into account the optimal efficiency of the AI ​​model.

[0056] Next, in step S200, the first text segment is processed by speech synthesis to obtain the first audio segment, and an SSE connection with the client is established at the same time as the first audio segment is synthesized.

[0057] Specifically, the above steps include:

[0058] Immediately begin speech synthesis on the first text segment (problem analysis) to obtain the first audio segment, and establish an SSE connection with the client while synthesizing the first audio segment to monitor the progress of AI speech synthesis.

[0059] The above SSE connection stands for Server-Sent Events, which is a protocol for sending events via the server.

[0060] The aforementioned audio segment includes audio metadata, which includes segment identifier (segment_id), final segment identifier (is_final), and content type label (content_type).

[0061] For example, the segment ID of the first audio segment (problem analysis) is 1, the final segment ID is false, and the content type label is analysis; the segment ID of the last audio segment (encouragement and support) is 5, the final segment ID is true, and the content type label is encouragement.

[0062] In step S300, after the first audio segment is synthesized, the first audio segment is immediately transmitted to the segmented buffer queue of the client via the SSE connection, and the subsequent text segments are synthesized into speech in sequence.

[0063] Specifically, the above steps include:

[0064] After the first audio segment is synthesized (approximately 2-3 seconds), the first audio segment is immediately transmitted to the segmented buffer queue on the client via the SSE connection, and speech synthesis of the second text segment (psychological explanation) begins simultaneously.

[0065] Finally, in step S400, after the segmented buffer queue receives the first audio segment, it immediately plays the first audio segment, and while playing the first audio segment, it continuously receives and buffers subsequent audio segments.

[0066] Specifically, the above steps include:

[0067] Once the first audio segment in the segmented buffer queue is detected as ready, playback begins immediately. Simultaneously, the second audio segment (psychological interpretation) is received via the SSE connection and stored in the segmented buffer queue. When the first audio segment has played to 80%, the synthesis and reception of the second audio segment (psychological interpretation) are completed.

[0068] Preferably, the second audio segment is normalized, which includes volume normalization (to ensure that the volume is consistent with the first audio segment), sampling rate unification (48kHz), and 0.1-second fade-in (to eliminate audio quality jumps).

[0069] While playing the current audio segment, the next audio segment in the segment buffer queue is preloaded, so that seamless switching can be achieved after the current audio segment has finished playing.

[0070] Similarly, while playing the second audio segment, the third audio segment can be synthesized in parallel (specific suggestions).

[0071] When the first audio segment is played, the status of each segment is as follows: segment 1 (playing) → segment 2 (buffered) → segment 3 (compositing) → segment 4 (to be composited) → segment 5 (to be composited).

[0072] The embodiments of the present invention achieve efficient pipelined audio synthesis and playback by means of the above-mentioned pipelined parallel processing mechanism, which involves "playing a segment, buffering a segment, and synthesizing a segment".

[0073] In addition, embodiments of the present invention also provide a precise detection mechanism for the synthesis and playback states, specifically including:

[0074] By listening to the metadata of audio segments, it can be determined whether the current audio segment is the last audio segment. If the current audio segment is the last audio segment (encouragement and support, segment_id is 5, is_final is true, and content_type is encouragement), and all audio segments have finished playing, a playback completion notification is sent, and the user can then engage in further interaction.

[0075] In addition, corresponding to the above-mentioned text segmentation and speech streaming processing method, this embodiment of the invention also provides a text segmentation and speech streaming processing system, which includes:

[0076] A semantic segmenter is used to segment long texts according to their semantic structure, resulting in semantically complete and ordered text fragments.

[0077] The speech synthesis module is used to synthesize speech from the first text segment to obtain the first audio segment, and while the first audio segment is transmitted to the segmented buffer queue of the client, it sequentially synthesizes speech from subsequent text segments;

[0078] The SSE transmission module is used to establish an SSE connection with the client while synthesizing the first audio segment, and immediately transmit the first audio segment to the client's segment buffer queue via the SSE connection after the first audio segment is synthesized.

[0079] A segmented audio streaming receiver is used to receive audio segments transmitted via an SSE connection and place the audio segments into a segmented buffer queue.

[0080] The playback controller is used to immediately play the first audio segment after the segmented buffer queue receives the first audio segment, and to continuously receive and buffer subsequent audio segments while playing the first audio segment.

[0081] Figure 2 A schematic diagram of the system structure of a text segmentation and speech streaming processing system provided by an embodiment of the present invention is shown.

[0082] Figure 3 The diagram shows the interactive timing of a text segmentation and speech streaming system provided by an embodiment of the present invention.

[0083] In addition, embodiments of the present invention also provide a text segmentation speech streaming processing device, the device comprising: a processor and a memory; the memory for storing one or more program instructions; the processor for running one or more program instructions to perform the steps of a text segmentation speech streaming processing method as described above.

[0084] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a text segmentation and speech streaming process as described above.

[0085] In addition, embodiments of the present invention also provide a computer program product, which includes computer program instructions that, when executed by a processor, implement the steps of a text segmentation and speech streaming processing method as described above.

[0086] The text segmentation and speech streaming processing method, system, device, medium, and program product provided in this invention have the following advantages:

[0087] This invention, through a real-time coordination mechanism between AI segmented synthesis and streaming transmission, solves for the first time the problem of precise coordination between the progress of AI model segmented audio synthesis and network streaming transmission, significantly improving the response speed of AI speech synthesis. Users can hear the sound quickly after AI starts synthesis without having to wait for the complete synthesis, which is especially suitable for long text AI speech synthesis scenarios.

[0088] This invention eliminates pauses and audio quality jumps between AI segmented synthesized audio through seamless connection technology, achieving high-quality continuous playback of AI segmented synthesized audio so that users cannot perceive the segment switching process.

[0089] The embodiments of the present invention ensure the accuracy of playback status judgment through a dual completion detection mechanism, avoiding the problem of prematurely ending playback before AI synthesis is completed.

[0090] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods. The storage medium can be memory, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory. Those skilled in the art will recognize that the functions described in the above examples can be implemented using a combination of hardware and software. When applied software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers. Although the invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be apparent to those skilled in the art. Therefore, such modifications or improvements made without departing from the spirit of the invention are all within the scope of protection claimed by the invention.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.

Claims

1. A method for text segmentation and speech streaming processing, characterized in that, The method includes: The long text is segmented according to its semantic structure to obtain semantically complete and ordered text fragments. The first text segment is processed into speech to obtain the first audio segment, and an SSE connection with the client is established simultaneously with the synthesis of the first audio segment; After the first audio segment is synthesized, it is immediately transmitted to the segmented buffer queue of the client via the SSE connection, and the subsequent text segments are synthesized into speech in sequence. Upon receiving the first audio segment, the segmented buffer queue immediately plays the first audio segment and continuously receives and buffers subsequent audio segments while playing the first audio segment.

2. The text segmentation and speech streaming processing method according to claim 1, characterized in that, Based on the semantic structure of the long text, segmentation is performed to obtain semantically complete ordered text fragments, including: The segmentation points of the long text are dynamically determined based on its semantic structure. Based on the segmentation points, the long text is sequentially segmented into semantically complete ordered text fragments.

3. The text segmentation and speech streaming processing method according to claim 1, characterized in that, The audio segment includes audio metadata; The audio metadata includes a segment identifier, a final segment flag, and a content type tag. The final segment flag is used to record whether the audio segment is the last audio segment.

4. The text segmentation and speech streaming processing method according to claim 1, characterized in that, The method further includes: The audio segment is subjected to standardization processing, which includes volume normalization, sampling rate unification, and audio fade-in / fade-out processing.

5. The text segmentation and speech streaming processing method according to claim 3, characterized in that, The method further includes: Based on the metadata of the audio segment, determine whether the current audio segment is the last audio segment; If the current audio segment is the last audio segment and all audio segments have finished playing, a playback completion notification will be sent.

6. The text segmentation and speech streaming processing method according to claim 1, characterized in that, While playing the first audio segment, subsequent audio segments are continuously received and buffered, including: While playing the first audio segment, subsequent audio segments are continuously received via the SSE connection and stored in the segmented buffer queue; While playing the current audio segment, the next audio segment in the segmented buffer queue is preloaded.

7. A text segmentation and speech streaming processing system, characterized in that, The system includes: A semantic segmenter is used to segment long texts according to their semantic structure, resulting in semantically complete and ordered text fragments. The speech synthesis module is used to synthesize speech on the first text segment to obtain the first audio segment, and while the first audio segment is transmitted to the segmented buffer queue of the client, it sequentially synthesizes speech on subsequent text segments; The SSE transmission module is used to establish an SSE connection with the client while synthesizing the first audio segment, and immediately transmit the first audio segment to the client's segment buffer queue through the SSE connection after the first audio segment is synthesized. A segmented audio streaming receiver is used to receive audio segments transmitted via the SSE connection and place the audio segments into a segmented buffer queue. A playback controller is configured to immediately play the first audio segment after the segmented buffer queue receives the first audio segment, and continuously receive and buffer subsequent audio segments while playing the first audio segment.

8. A text segmentation and speech streaming processing device, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of a text segmentation speech streaming method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a text segmentation speech streaming process as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed by a processor, implement the steps of a text segmentation speech streaming method as described in any one of claims 1 to 6.

Images