Vehicle-mounted voice interaction method, electronic device, storage medium and program product

Abstract

The application provides a vehicle-mounted voice interaction method, an electronic device, a storage medium and a program product, relates to the technical field of vehicle voice synthesis, and obtains streaming output text of a large language model; performs semantic boundary perception processing on the streaming output text to obtain a first complete text segment in the streaming output text; performs voice synthesis processing on the first complete text segment based on a preset first-character special acceleration channel to obtain audio data of the first complete text segment; and performs vehicle-mounted voice interaction based on the audio data. The application can solve the problem of excessively long link delay in a traditional vehicle-mounted voice interaction scheme, reduce first-character response delay to meet the use requirement of low delay and instant feedback of vehicle-mounted intelligent cabin voice interaction, and improve user experience.

Description

Technical Field

[0001] This application relates to the field of vehicle speech synthesis (Text-to-Speech, TTS) technology, and in particular to an in-vehicle voice interaction method, electronic device, storage medium, and program product. Background Technology

[0002] In related technologies, the voice interaction solution combining a Large Language Model (LLM) with cloud-based TTS in an in-vehicle intelligent cockpit has an inherent defect of excessive link latency. Its complete interaction link consists of multiple latency stages superimposed: the generation of the first token of the LLM (the first valid text segment that meets the semantic integrity requirements output by the LLM in the current session) takes 200-500ms, the cloud-based TTS synthesis startup takes 2-3 seconds, the network transmission latency from the cloud to the vehicle is 100-300ms, and the latency of audio decoding and playback preparation at the vehicle end is 100-200ms. After the latency of each stage is added up, the overall first-word response latency usually reaches 3-5 seconds, which far exceeds the user's acceptable first-word response latency threshold (such as 500ms-1s). This will cause the user's voice interaction feedback to be lagging and the interaction to be fragmented, which will seriously affect the real-time performance and naturalness of the voice interaction. Summary of the Invention

[0003] The main objective of this application is to propose an in-vehicle voice interaction method, electronic device, storage medium, and program product, which aims to solve the problem of excessive link latency in traditional in-vehicle voice interaction solutions, reduce the first-word response latency to meet the low latency and real-time feedback requirements of in-vehicle smart cockpit voice interaction, and improve the user experience.

[0004] To achieve the above objectives, a first aspect of this application proposes an in-vehicle voice interaction method, the method comprising: Obtain the streaming output text of a large language model; Semantic boundary awareness processing is performed on the streaming output text to obtain the first complete text segment in the streaming output text; The first complete text segment is processed by speech synthesis based on a preset first-character-dedicated acceleration channel to obtain the audio data of the first complete text segment; In-vehicle voice interaction is performed based on the audio data.

[0005] In some embodiments, the semantic boundary-aware processing of the streaming output text includes: The streaming output text is processed by word boundary recognition to obtain the complete words in the streaming output text; The semantic verification process is performed on the complete word based on the preset minimum semantic unit rule to obtain the semantic integrity verification result of the complete word; If the semantic integrity verification result indicates that the semantic integrity of the complete word conforms to the minimum semantic unit rule, the complete word is subjected to punctuation filtering to obtain the cleaned text after removing punctuation; the first complete text segment includes the cleaned text.

[0006] In some embodiments, the word boundary recognition processing of the streaming output text includes: The semantic boundary detection strategy is determined based on the language type of the streaming output text. The semantic boundary detection strategy is dynamically adjusted based on the text features of the streaming output text to obtain the adjusted semantic boundary detection strategy. The adjusted semantic boundary detection strategy is used to perform word boundary recognition processing on the streaming output text.

[0007] In some embodiments, the text features include complexity, and the dynamic adjustment of the semantic boundary detection strategy based on the text features of the streaming output text includes: If the streaming output text is determined to be complex text based on the complexity, the character count threshold in the semantic boundary detection strategy is increased. If the streaming output text is determined to be simple text based on the complexity, the character count threshold in the semantic boundary detection strategy is reduced.

[0008] In some embodiments, the speech synthesis processing of the first complete text segment based on a preset first-character-dedicated acceleration channel includes: Add the first complete text fragment to the high-priority queue in the first-character-dedicated acceleration channel; The dedicated inference board resources of the first character-specific acceleration channel are invoked to perform speech synthesis processing on the first complete text segment in the high-priority queue.

[0009] In some embodiments, the in-vehicle voice interaction based on the audio data includes: The audio data is pushed and played in real time based on the dedicated acceleration channel for the first character. After playing the audio data, subsequent audio data continues to be played; the subsequent audio data is obtained by speech synthesis processing of the subsequent text of the first complete text segment based on the conventional processing channel, and the speech synthesis task is performed in parallel with the first character dedicated acceleration channel.

[0010] In some embodiments, the method further includes: Delay monitoring is performed on multiple preset key nodes to obtain the delay information of each key node; the multiple key nodes include a first step of acquiring the streaming output text, a second step of semantic boundary awareness processing of the streaming output text, a third step of speech synthesis processing of the first complete text segment, and a fourth step of in-vehicle voice interaction based on the audio data. If the target stage times out based on the delay information, a preset response degradation strategy is triggered to directly play audio data for in-vehicle voice interaction; the target stage is any one of the first stage, the second stage, the third stage, and the fourth stage.

[0011] To achieve the above objectives, a second aspect of this application provides an in-vehicle voice interaction system, the system comprising: The acquisition module is used to acquire the streaming output text of a large language model; The semantic boundary awareness module is used to perform semantic boundary awareness processing on the streaming output text to obtain the first complete text segment in the streaming output text; The speech acceleration synthesis module is used to perform speech synthesis processing on the first complete text segment based on a preset first-character dedicated acceleration channel to obtain the audio data of the first complete text segment; An audio processing module is used for in-vehicle voice interaction based on the audio data.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the in-vehicle voice interaction method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the in-vehicle voice interaction method described in the first aspect.

[0014] To achieve the above objectives, a fifth aspect of the present application provides a computer program product, which includes a computer program that, when executed by a processor, implements the in-vehicle voice interaction method provided in the first aspect above.

[0015] The in-vehicle voice interaction method, system, electronic device, computer-readable storage medium, and computer program product proposed in this application acquire the streaming output text of a large language model; perform semantic boundary awareness processing on the streaming output text to obtain the first complete text segment in the streaming output text; perform speech synthesis processing on the first complete text segment based on a preset first-character dedicated acceleration channel to obtain the audio data of the first complete text segment; and perform in-vehicle voice interaction based on the audio data.

[0016] In this embodiment of the application, after obtaining the streaming output text of the large language model, the streaming output text is first subjected to semantic boundary awareness processing to obtain the first complete text segment in the streaming output text. Then, based on the preset first character dedicated acceleration channel, the audio data of the first complete text segment is processed by speech synthesis. Finally, in-vehicle voice interaction is performed based on the audio data.

[0017] Thus, compared to the traditional LLM+TTS voice interaction solution, the embodiments of this application, by performing semantic awareness of the first token recognition of the streaming output text and performing speech synthesis based on the dedicated acceleration channel for the first character, can reduce the first character response latency in in-vehicle voice interaction from 3-5 seconds to 300-500 milliseconds. That is, it solves the problem of excessive link latency in traditional in-vehicle voice interaction solutions, reduces the first character response latency, thereby meeting the low latency and instant feedback requirements of in-vehicle intelligent cockpit voice interaction and improving the user experience.

[0018] Furthermore, this application embodiment obtains the first complete text segment by performing semantic boundary awareness on the streaming output text to intelligently identify the first token. This ensures that the first complete text segment is segmented at the boundary of complete words, thereby guaranteeing the naturalness of pronunciation when using audio data based on the first complete text segment for in-vehicle voice interaction. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the steps of the in-vehicle voice interaction method provided in some embodiments of this application; Figure 2 for Figure 1 A detailed flowchart of step S102; Figure 3 A schematic diagram of the first token recognition and acceleration process for semantic boundary awareness involved in some embodiments of the in-vehicle voice interaction method provided in this application; Figure 4 for Figure 2 A detailed flowchart of step S201; Figure 5A schematic diagram of the semantic boundary detection algorithm involved in some embodiments of the in-vehicle voice interaction method provided in this application; Figure 6 for Figure 1 A detailed flowchart of step S103; Figure 7 for Figure 1 A detailed flowchart of step S104; Figure 8 A schematic diagram of the parallel processing flow of the dedicated acceleration channel for the first character and the conventional processing channel involved in some embodiments of the in-vehicle voice interaction method provided in this application; Figure 9 A flowchart illustrating the steps of the in-vehicle voice interaction method provided in this application in other embodiments; Figure 10 The system architecture diagram of the in-vehicle voice interaction method provided in the embodiments of this application is shown in some embodiments; Figure 11 This is a schematic diagram of the structure of the in-vehicle voice interaction system provided in the embodiments of this application; Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] First, a brief explanation of the overall concept of the in-vehicle voice interaction method provided in the embodiments of this application will be given.

[0024] Currently, in-vehicle intelligent cockpits have entered the stage of deep empowerment by large language models (LLM). Large language models output text using a streaming text output method, and then combine this with cloud-based TTS speech synthesis services to convert the text into speech in real time for broadcasting. This has become the mainstream implementation form of in-vehicle voice interaction. In in-vehicle voice interaction scenarios, rapid voice feedback to user-initiated voice interaction requests is the core key to ensuring natural and smooth human-vehicle interaction. The latency of the first character response directly determines the quality of the user interaction experience. Therefore, the industry's requirements for low latency and real-time performance in in-vehicle voice interaction are becoming increasingly stringent.

[0025] According to user experience research, the threshold distribution of the first-word response delay in voice interaction can be as follows: when the first-word response delay is <500ms, the user perceives it as smooth and the experience is excellent; when the first-word response delay is between 500ms and 1s, the user finds it acceptable and the experience is good; when the first-word response delay is between 1s and 2s, the user perceives a significant delay and the experience is average; when the first-word response delay is >2s, the user perceives it as choppy and the experience is poor.

[0026] In addition, in-vehicle voice interaction has certain unique characteristics compared to other voice interaction scenarios. For example, in order to ensure driving safety, navigation prompts and vehicle control feedback often need to respond quickly, and response delays may affect the vehicle's driving decisions. Furthermore, in order to ensure the naturalness of in-vehicle voice interaction, a response delay of more than 1 second in human-computer dialogue will significantly reduce the smoothness of the interaction. Moreover, in terms of semantic integrity, in-vehicle voice broadcasts are usually required to ensure natural pronunciation and avoid pronunciation abnormalities caused by word segmentation.

[0027] In related technologies, the voice interaction solution combining a large language model (LLM) with cloud-based text-to-speech (TTS) in an in-vehicle intelligent cockpit has an inherent defect of excessive link latency. Its complete interaction link consists of multiple latency stages superimposed: the generation of the first token of the large language model (the first valid text segment output by the LLM in the current session that meets the semantic integrity requirements) takes 200-500ms, the cloud-based TTS synthesis startup takes 2-3 seconds, the network transmission latency from the cloud to the vehicle is 100-300ms, and the latency of audio decoding and playback preparation at the vehicle end is 100-200ms. After the latency of each stage is added up, the overall first-word response latency usually reaches 3-5 seconds, which far exceeds the user's acceptable first-word response latency threshold (such as 500ms-1s mentioned above). This will cause the user's voice interaction feedback to be lagging and the interaction to be fragmented, which will seriously affect the real-time performance and naturalness of the voice interaction.

[0028] To address the aforementioned issues, this application proposes an in-vehicle voice interaction method, system, electronic device, computer-readable storage medium, and computer program product. The aim is to solve the problem of excessive latency in traditional in-vehicle voice interaction solutions, reduce first-word response latency to meet the low-latency, real-time feedback requirements of in-vehicle smart cockpit voice interaction, and improve the user experience.

[0029] In this embodiment, the following steps are taken: First, the streaming output text of a large language model is acquired; second, semantic boundary awareness processing is performed on the streaming output text to obtain the first complete text segment; third, speech synthesis processing is performed on the first complete text segment based on a preset first-character dedicated acceleration channel to obtain the audio data of the first complete text segment; and fourth, in-vehicle voice interaction is performed based on the audio data.

[0030] In this embodiment of the application, after obtaining the streaming output text of the large language model, the streaming output text is first subjected to semantic boundary awareness processing to obtain the first complete text segment in the streaming output text. Then, based on the preset first character dedicated acceleration channel, the audio data of the first complete text segment is processed by speech synthesis. Finally, in-vehicle voice interaction is performed based on the audio data.

[0031] Thus, compared to the traditional LLM+TTS voice interaction solution, the embodiments of this application, by performing semantic awareness of the first token recognition of the streaming output text and performing speech synthesis based on the dedicated acceleration channel for the first character, can reduce the first character response latency in in-vehicle voice interaction from 3-5 seconds to 300-500 milliseconds. That is, it solves the problem of excessive link latency in traditional in-vehicle voice interaction solutions, reduces the first character response latency, thereby meeting the low latency and instant feedback requirements of in-vehicle intelligent cockpit voice interaction and improving the user experience.

[0032] Furthermore, this application embodiment obtains the first complete text segment by performing semantic boundary awareness on the streaming output text to intelligently identify the first token. This ensures that the first complete text segment is segmented at the boundary of complete words, thereby guaranteeing the naturalness of pronunciation when using audio data based on the first complete text segment for in-vehicle voice interaction.

[0033] In some embodiments, when performing semantic boundary awareness processing on streaming output text, an adaptive semantic boundary detection strategy can also be adopted. The semantic boundary detection strategy is adaptively adjusted according to different languages ​​and text types, thereby improving the accuracy of semantic boundary recognition for the streaming output text. It can also be adapted to various text types for semantic boundary awareness processing and can effectively improve the success rate of anomaly handling in the process of semantic boundary awareness of streaming output text.

[0034] In some embodiments, an end-to-end optimized link can be established for in-vehicle voice interaction from LLM output to first character playback, minimizing the latency of each link and achieving a stable end-to-end first character response latency of 300-500ms, which greatly reduces the timeout rate of in-vehicle voice interaction.

[0035] Next, the in-vehicle voice interaction method, system, electronic device, computer-readable storage medium, and computer program product provided in this application will be specifically described through the following embodiments, and firstly, the various detailed embodiments of the in-vehicle voice interaction method provided in this application will be described in detail.

[0036] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0037] It should be noted that the in-vehicle voice interaction method provided in this application relates to the field of vehicle voice synthesis technology. The in-vehicle voice interaction method provided in this application can be applied to a terminal, a server, or software running on either the terminal or the server. In some embodiments, the terminal can be a vehicle, an in-vehicle terminal, or an electronic device such as a smartphone, tablet, laptop, or desktop computer associated with the vehicle and capable of communicating and interacting with it via a network. The server can be the backend server terminal device of the terminal, which can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The software can be an application implementing the in-vehicle voice interaction method, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminal, server, and software of the in-vehicle voice interaction method provided in this application may also be other forms not listed here, and the in-vehicle voice interaction method provided in this application does not specifically limit these.

[0038] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: vehicles, in-vehicle terminals, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0039] For ease of understanding and explanation, the following text will use the in-vehicle voice interaction method provided in the embodiments of this application as an example to describe the various specific embodiments of this application in detail. In some descriptions, the in-vehicle terminal may be simply referred to as an intelligent agent. The implementation of the in-vehicle voice interaction method provided in the embodiments of this application for any of the above-described subjects can refer to the implementation process of the in-vehicle voice interaction method described below.

[0040] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the in-vehicle voice interaction method provided in some embodiments of this application. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the in-vehicle voice interaction method provided in this application embodiment can, of course, adopt an execution order of method steps different from that shown in the figures. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the in-vehicle voice interaction method provided in the embodiments of this application. Any other order based on... Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the in-vehicle voice interaction method provided in the embodiments of this application.

[0041] like Figure 1 As shown, in some embodiments, the in-vehicle voice interaction method provided in this application may include steps S101 to S103 as shown below.

[0042] Step S101: Obtain the streaming output text of the large language model.

[0043] It should be noted that streaming output text refers to the text output by a large language model (LLM) in a streaming manner. The large language model (LLM) can be an artificial intelligence model that can generate natural language text, such as a generative pre-trained transformer (GPT).

[0044] When the in-vehicle terminal responds to user-initiated voice interaction requests in the in-vehicle smart cockpit, in order to quickly respond to user voice interaction requests based on voice feedback, it abandons the strategy of treating the first token and subsequent tokens the same as in traditional streaming TTS solutions. Instead, it adopts a solution specifically optimized for the first character response. Through semantic boundary-aware intelligent recognition of the first token and a dedicated acceleration channel, it achieves rapid first character response in in-vehicle voice interaction, thereby reducing the first character response latency (e.g., from 3-5 seconds to 300-500 milliseconds). Based on this, when the in-vehicle terminal responds to user voice interaction in the in-vehicle smart cockpit, it first obtains the streaming output text of the Large Language Model (LLM).

[0045] In some embodiments, the in-vehicle terminal can output text in a streaming, token-by-token manner using a Large Language Model (LLM), sending each token immediately after its generation. In this way, in in-vehicle voice interaction scenarios, the in-vehicle terminal can continuously acquire the streaming text output by the LLM in a token-by-token manner.

[0046] Step S102: Perform semantic boundary awareness processing on the streaming output text to obtain the first complete text segment in the streaming output text.

[0047] It should be noted that the first complete text fragment can be the first token mentioned above, that is, the first valid text fragment in the streaming output text. For example, the first complete text fragment can be the first valid text fragment output by LLM in the current session between the in-vehicle intelligent cockpit and the user, and this text fragment must meet the semantic integrity requirements (such as containing at least one complete word).

[0048] After acquiring the streaming output text, the vehicle terminal further performs semantic boundary awareness processing on the streaming output text to obtain the first complete text segment in the streaming output text.

[0049] In some embodiments, the in-vehicle terminal can identify semantic boundaries in the streaming output text to segment the first complete text fragment based on these semantic boundaries. Here, semantic boundaries refer to the dividing positions between words in the streaming output text. The in-vehicle terminal can identify these boundaries using a tokenizer, thereby ensuring that the first token is not segmented from the middle of a word. A tokenizer is a tool that segments continuous text into word units, such as jieba (a Chinese tokenizer) and NLTK (an English tokenizer).

[0050] In some embodiments, the in-vehicle terminal can maintain a text stream buffer to provide a data foundation for first-token recognition. That is, in an in-vehicle voice interaction scenario, the in-vehicle terminal can temporarily store the streaming output text, which is output token-by-token by the Large Language Model (LLM), in a pre-maintained text stream buffer. Then, the in-vehicle terminal can extract the streaming output text from this buffer and perform semantically aware first-token recognition processing based on a word segmenter, thereby obtaining the first complete text segment in the streaming output text. The buffer can be a core component of TTS streaming playback, serving as a memory area for temporarily storing received but not yet played audio data.

[0051] Step S103: Perform speech synthesis processing on the first complete text segment based on the preset first-character dedicated acceleration channel to obtain the audio data of the first complete text segment.

[0052] It should be noted that the first-character dedicated acceleration channel is a channel pre-designed by the vehicle terminal to provide high-priority processing for the first token. The first token can skip the regular queuing process in this channel and perform rapid voice synthesis and playback push.

[0053] After receiving the first complete text segment from the streaming output text, the vehicle terminal further performs speech synthesis processing on the first complete text segment to generate its audio data. At this point, instead of queuing the first token with other text segments for TTS synthesis, the vehicle terminal uses a pre-defined dedicated acceleration channel for the first character to add the first complete text segment to a high-priority queue. This allows it to skip the conventional queuing process and quickly perform speech synthesis processing on the first complete text segment, obtaining its audio data.

[0054] In some embodiments, the vehicle terminal can provide dedicated synthesis resources for the first token in the first character dedicated acceleration channel, thereby ensuring that the first complete text segment can be immediately processed for speech synthesis based on the dedicated synthesis resources after entering the first character dedicated acceleration channel.

[0055] In some embodiments, the vehicle terminal can perform speech synthesis processing on the first complete text segment based on the first character-dedicated acceleration channel, and simultaneously perform speech synthesis processing on subsequent tokens after the first complete text segment based on the conventional processing channel parallel to the first character-dedicated acceleration channel.

[0056] Step S104: Perform in-vehicle voice interaction based on the audio data.

[0057] After receiving the audio data of the first complete text segment, the in-vehicle terminal immediately controls the in-vehicle smart cockpit to play the audio data, thereby providing a low-latency, natural, and seamless first-word response to the user's voice interaction request.

[0058] In this embodiment, when the vehicle terminal responds to a user's voice interaction in the in-vehicle smart cockpit, it first acquires the streaming output text of the Large Language Model (LLM). Then, it performs semantic boundary awareness processing on the streaming output text to obtain the first complete text segment. Next, based on a preset first-character dedicated acceleration channel, the first complete text segment is added to a high-priority queue, thereby skipping the regular queuing and quickly performing speech synthesis processing on the first complete text segment to obtain the audio data of the first complete text segment. Finally, the in-vehicle smart cockpit is controlled to play the audio data, thereby providing a low-latency, natural, and uninterrupted first-character response to the user's voice interaction request.

[0059] Thus, compared to the traditional LLM+TTS voice interaction solution, the embodiments of this application, by performing semantic awareness of the first token recognition of the streaming output text and performing speech synthesis based on the dedicated acceleration channel for the first character, can reduce the first character response latency in in-vehicle voice interaction from 3-5 seconds to 300-500 milliseconds. That is, it solves the problem of excessive link latency in traditional in-vehicle voice interaction solutions, reduces the first character response latency, thereby meeting the low latency and instant feedback requirements of in-vehicle intelligent cockpit voice interaction and improving the user experience.

[0060] Furthermore, this application embodiment obtains the first complete text segment by performing semantic boundary awareness on the streaming output text to intelligently identify the first token. This ensures that the first complete text segment is segmented at the boundary of complete words, thereby guaranteeing the naturalness of pronunciation when using audio data based on the first complete text segment for in-vehicle voice interaction.

[0061] Please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S102.

[0062] like Figure 2As shown, in some embodiments, the step of "performing semantic boundary awareness processing on the streaming output text" in the above step S102 may include the following steps S201 to S203.

[0063] Step S201: Perform word boundary recognition processing on the streaming output text to obtain complete words in the streaming output text.

[0064] It should be noted that the complete word is the first text segment with complete semantics in the streaming output text. For example, when the streaming output text is the navigation prompt "turn right 500 meters ahead", the complete word may be "ahead"; when the streaming output text is the multilingual mixed text "Hello, welcome to use", the complete word may be "Hello". The first complete text segment in the streaming output text must contain at least one complete word.

[0065] When the in-vehicle terminal performs semantic boundary awareness processing on the streaming output text, it may first perform word boundary recognition processing on the streaming output text, so as to obtain the complete words in the streaming output text.

[0066] In some embodiments, the in-vehicle terminal may adopt a preset semantic boundary detector to identify word boundaries in the streaming output text based on the above-mentioned word segmenter (such as jieba or HanLP, etc.), and then segment out complete words with complete semantics from the word boundaries. For example, when the streaming output text is the text stream "Hello, I am an AI assistant, it is a pleasure to serve you", the relevant details of the in-vehicle terminal performing semantic boundary detection on the text stream based on the semantic boundary detector may be as shown in the following table.

[0067]

[0068] Step S202: Perform semantic verification processing on the complete word based on a preset minimum semantic unit rule to obtain a semantic integrity verification result of the complete word.

[0069] It should be noted that the minimum semantic unit rule may be that Chinese has at least 2 characters (such as "Hello"), English has at least one complete word, and numbers need to be ensured to be complete (such as "2024" is not split). For example, the minimum semantic unit rule may specifically be as shown in the following table:

[0070] After the in-vehicle terminal obtains the complete words in the streaming output text, in order to further ensure semantic integrity, it may also perform semantic verification processing on the complete words based on the minimum semantic unit rule to verify the semantic integrity of the complete words, so as to obtain the semantic integrity verification result of the complete words.

[0071] In some embodiments, the vehicle terminal can use a preset first-token recognizer to apply the minimum semantic unit rule to perform semantic verification processing on the complete word to verify the semantic integrity of the complete word, thereby intelligently identifying the first valid text segment in the streaming output text. For example, the vehicle terminal can apply the minimum semantic unit rule based on the first-token recognizer to check whether the complete word includes the minimum semantic unit. That is, when the complete word is Chinese, it checks whether the complete word is greater than or equal to 2 characters; when the complete word is English, it checks whether the complete word is a complete word; and when the complete word is a number, it checks whether the complete word is complete and has not been incorrectly split.

[0072] Step S203: If the semantic integrity verification result indicates that the semantic integrity of the complete word conforms to the minimum semantic unit rule, perform punctuation filtering on the complete word to obtain the cleaned text after removing punctuation; the first complete text segment includes the cleaned text.

[0073] It should be noted that the semantic integrity verification result indicates that the semantic integrity of the complete word conforms to the minimum semantic unit rule, which can be: when the complete word is Chinese, the complete word is greater than or equal to 2 characters; when the complete word is English, the complete word is a complete word; and when the complete word is a number, the complete word has not been incorrectly split.

[0074] After receiving the semantic integrity verification result of the complete word, if the result indicates that the semantic integrity of the complete word conforms to the minimum semantic unit rule, then the vehicle terminal further performs punctuation filtering on the complete word according to a preset punctuation filtering mechanism (the first token does not start with a punctuation mark, avoiding meaningless first audio, such as avoiding voice interaction starting with commas, periods, etc.), resulting in purified text after removing punctuation. In this way, the vehicle terminal can identify this purified text as the first complete text segment in the streaming output text.

[0075] In this embodiment, when performing semantic boundary awareness processing on the streaming output text by the vehicle terminal, word boundary recognition processing is first performed on the streaming output text to obtain the complete words in the streaming output text. Then, based on the minimum semantic unit rule, semantic verification processing is performed on the complete words to verify the semantic integrity of the complete words. Finally, if the semantic integrity verification result indicates that the semantic integrity of the complete words conforms to the minimum semantic unit rule, punctuation filtering processing is performed on the complete words, thereby determining the purified text after removing punctuation as the first complete text segment.

[0076] Thus, addressing the issue that traditional solutions may contain incomplete words in the first token when synthesizing speech from streaming text to achieve the first-word response, thereby affecting pronunciation quality and user experience, this embodiment uses an intelligent first-token recognition mechanism to determine the first complete text segment in the streaming output text for the first-word response. This can effectively improve the semantic integrity rate of the first token, ensuring that the pronunciation of the first word response is natural and without any breaks, thereby significantly improving the user's subjective evaluation of in-vehicle voice interaction.

[0077] In some embodiments, the in-vehicle terminal can also design an intelligent buffer window to obtain the minimum semantic unit when acquiring streaming output text for semantic boundary awareness processing. The intelligent buffer window can be a 50ms buffer window, designed based on the following: LLM output speed is approximately 20-50 tokens / second (e.g., GPT-4 averages 35 tokens / second, Claude-3 averages 28 tokens / second, and Wenxin Yiyan averages 32 tokens / second), thus 50ms can receive 1-2.5 tokens, sufficient to obtain the minimum semantic unit. Furthermore, the 50ms delay is imperceptible to the user (the threshold for human auditory perception of delay is approximately 100ms: in a test of 100 users, the perceived delay rate was <5% for delays below 50ms and >80% for delays above 100ms).

[0078] In some embodiments, the vehicle terminal can also ensure low-latency response of the first character by designing a fallback mechanism with timeout protection.

[0079] In some embodiments, the vehicle terminal may use the semantic boundary detection algorithm shown in the table below to perform semantic boundary awareness processing on the output text.

[0080]

[0081] For example, the vehicle-mounted terminal can adopt such as Figure 3The semantic boundary-aware first-token recognition and acceleration process shown can be implemented in an in-vehicle voice interaction scenario. When performing semantic boundary-aware first-token recognition and acceleration on streaming output text, after the LLM starts streaming output, text buffering can be performed. At this time, the in-vehicle terminal accumulates the streaming output text into the buffer and starts a 50ms buffering window. Then, the in-vehicle terminal performs word segmentation detection on the streaming output text, that is, it uses a word segmenter to identify word boundaries to check whether the streaming output text contains complete words. If a complete word is detected, the in-vehicle terminal further performs semantic verification on the complete word, that is, it checks whether the complete word contains the smallest semantic unit, obtaining the semantic integrity verification result of the complete word. Subsequently, if the semantic integrity verification result indicates that the semantic integrity of the complete word conforms to the smallest semantic unit rule, the in-vehicle terminal performs punctuation filtering on the complete word, removing the punctuation marks at the beginning of the complete word to avoid meaningless first audio and ensure the validity of the first character. Then, the in-vehicle terminal performs boundary verification on the cleaned text after removing punctuation to generate a FirstTokenResult, thus identifying the cleaned text as the first complete text segment. Finally, the in-vehicle terminal prioritizes the speech synthesis and push of the first complete text segment based on the dedicated acceleration channel for the first character. This allows the first complete text segment to skip the regular queuing and use dedicated TTS resources for speech synthesis. The synthesized audio data is then immediately pushed to the player to start playing the first character.

[0082] In addition, when the vehicle terminal performs word segmentation detection on the streaming output text, if it finds that the streaming output text does not contain a complete word, the vehicle terminal continues to receive tokens and checks whether the current time exceeds the 50ms buffer window. If it does not exceed 50ms, it continues to execute text buffering, word segmentation detection and subsequent processes. If it exceeds 50ms, it triggers timeout protection, forces the output of the current content to generate FirstTokenResult, and uses the current content as the first complete text segment to start priority speech synthesis and push for low-latency response of the first character.

[0083] In some embodiments, in simple broadcast scenarios where semantic integrity requirements are not high, the vehicle terminal may also adopt a fixed-length segmentation scheme to segment the first token of the streaming output text by a fixed number of characters (such as 2 characters) to obtain the first complete text segment in the streaming output text for subsequent priority speech synthesis and push.

[0084] In some embodiments, when the latency requirements are not strict, the vehicle terminal can also use a fixed time window scheme, taking the accumulated text of the LLM streaming output after waiting for a fixed time (such as 100ms) as the first token, and then using the first token for subsequent priority speech synthesis and push.

[0085] In some embodiments, in the context of structured text broadcasting, the vehicle terminal can also segment the first token based on the first punctuation mark as the boundary, and then perform subsequent priority speech synthesis and push on the first token.

[0086] Please refer to Figure 4 , Figure 4 for Figure 2 A detailed flowchart of step S201.

[0087] like Figure 4 As shown, in some embodiments, the step of "performing word boundary recognition processing on the streaming output text" in step S201 above may include steps S401 to S403 as shown below.

[0088] Step S401: Determine the semantic boundary detection strategy based on the language type of the streaming output text.

[0089] It should be noted that the semantic boundary detection strategy identifies the smallest semantic units in the streaming text in real time during the token-by-token output of the Large Language Model (LLM). It supports the identification of the smallest semantic units in Chinese, English, numbers, and various mixed texts. The semantic boundary detection strategy can adopt a three-layer detection architecture: the basic rule layer quickly fills in the gaps using rules for punctuation, numbers, and English words; the semantic understanding layer uses a lightweight model to verify semantic integrity and contextual dependencies; and the mixed text layer is specifically optimized for scenarios such as combinations of Chinese and English numbers and number quantifiers.

[0090] When performing word boundary recognition on streaming output text, the in-vehicle terminal can first determine the semantic boundary detection strategy based on the language type of the streaming output text. For example, when the language type of the streaming output text is Chinese, the in-vehicle terminal can choose a semantic boundary detection strategy that supports the recognition of the smallest semantic unit of Chinese text; when the language type of the streaming output text is English, the in-vehicle terminal can choose a semantic boundary detection strategy that supports the recognition of the smallest semantic unit of English text; when the language type of the streaming output text is numbers, the in-vehicle terminal can choose a semantic boundary detection strategy that supports the recognition of the smallest semantic unit of numbers; and when the language type of the streaming output text is mixed text, the in-vehicle terminal can choose a semantic boundary detection strategy that supports the recognition of the smallest semantic unit of mixed text.

[0091] Step S402: Dynamically adjust the semantic boundary detection strategy based on the text features of the streaming output text to obtain the adjusted semantic boundary detection strategy.

[0092] After determining the semantic boundary detection strategy based on the language type of the streaming output text, the vehicle terminal further dynamically adjusts the semantic boundary detection strategy based on the text features of the streaming output text to obtain the adjusted semantic boundary detection strategy.

[0093] In some embodiments, the vehicle terminal can adjust the length of the minimum semantic unit of the semantic boundary detection strategy to obtain an adjusted semantic boundary detection strategy.

[0094] In some embodiments, the text feature includes complexity.

[0095] It should be noted that complexity is a quantitative indicator reflecting the degree of nesting in the structure of the streamed output text, the degree of discreteness of the content, and the degree of cross-language / cross-format mixing. The higher the complexity, the more difficult it is to accurately identify semantic boundaries, and the more likely semantic truncation and synthesis stuttering issues will occur, directly affecting the balance between initial sound delay and playback smoothness.

[0096] In some embodiments, the vehicle terminal can determine the complexity of the streaming output text by statistically analyzing parameters such as the number of syntactic nesting levels, the number of related words, and the length of consecutive text without punctuation.

[0097] Based on this, the step S402 above, "dynamically adjusting the semantic boundary detection strategy based on the text features of the streaming output text," can include the following steps: If the streaming output text is determined to be complex text based on the complexity, the character count threshold in the semantic boundary detection strategy is increased. If the streaming output text is determined to be simple text based on the complexity, the character count threshold in the semantic boundary detection strategy is reduced.

[0098] When the vehicle terminal dynamically adjusts the semantic boundary detection strategy based on the text features of the streaming output text, it can dynamically adjust the length of the minimum semantic unit of the semantic boundary detection strategy based on the complexity of the streaming output text. That is, the vehicle terminal first determines whether the streaming output text is complex or simple based on the complexity. Then, if the streaming output text is determined to be complex based on the complexity, the basic threshold for the number of characters in the semantic boundary detection strategy is increased. If the streaming output text is determined to be simple based on the complexity, the basic threshold for the number of characters in the semantic boundary detection strategy is decreased.

[0099] In some embodiments, the vehicle terminal may pre-design a complexity threshold to determine whether the streaming output text is complex text or simple text. For example, if the complexity of the streaming output text is greater than the complexity threshold, the streaming output text is determined to be complex text; if the complexity is less than the complexity threshold, the streaming output text is determined to be simple text; and if the complexity is equal to the complexity threshold, the streaming output text is determined to be ordinary text.

[0100] In some embodiments, when the streaming output text is plain text, the vehicle terminal does not adjust the basic threshold for the number of characters in the semantic boundary detection strategy.

[0101] Step S403: Perform word boundary recognition processing on the streaming output text based on the adjusted semantic boundary detection strategy.

[0102] The vehicle terminal performs word boundary recognition on the streaming output text according to the adjusted semantic boundary detection strategy. Then, it performs semantic integrity verification on the complete words in the recognized streaming output text and filters the complete words for punctuation marks, thus obtaining the purified text after removing punctuation marks as the first complete text segment.

[0103] In some embodiments, when the vehicle terminal performs word boundary recognition on the streaming output text, after determining the semantic boundary detection strategy based on the language type of the streaming output text, it may also directly perform word boundary recognition processing on the streaming output text without adjusting the semantic boundary detection strategy.

[0104] In some embodiments, when the vehicle terminal performs word boundary recognition on the streaming output text according to the semantic boundary detection strategy / adjusted semantic boundary detection strategy, it can also determine the word boundaries of the streaming output text based on context awareness and the preceding context of the voice interaction between the vehicle voice interaction system and the user, thereby further improving the accuracy of word segmentation.

[0105] In some embodiments, the vehicle terminal may also be pre-designed with an exception handling mechanism. When performing word boundary recognition on the streaming output text according to the semantic boundary detection strategy / adjusted semantic boundary detection strategy, a timeout protection mechanism is adopted for special cases in which semantic boundaries cannot be recognized.

[0106] In some embodiments, the exception handling mechanism may be as shown in the table below.

[0107]

[0108] In some embodiments, when the vehicle terminal performs word boundary recognition on the streaming output text according to the semantic boundary detection strategy / adjusted semantic boundary detection strategy, it can also perform quality assessment on the semantic integrity of the first complete text segment in real time, thereby ensuring the pronunciation quality when the audio data of the first complete text segment is played to respond to the user's first word.

[0109] For example, the vehicle-mounted terminal can adopt such as Figure 5 The semantic boundary detection algorithm shown takes the streaming text output by LLM token as input. First, it performs language type identification to distinguish whether the streaming text is Chinese, English, numbers, or mixed text. Then, it adaptively matches the corresponding semantic boundary detection strategy: Chinese uses jieba segmentation with a minimum semantic unit of ≥2 characters; English uses NLTK segmentation with a minimum semantic unit of a complete word; numbers use regular expression matching with a minimum unit of a complete numerical string; and mixed text uses a hybrid strategy, segmenting according to the dominant language rules. After segmentation, the vehicle terminal performs semantic integrity verification on the complete words in the streaming text and performs punctuation filtering, finally outputting the first complete text fragment, FirstTokenResult.

[0110] In some embodiments, when determining the semantic boundary detection strategy based on the language type of the streaming output text, the vehicle terminal may adopt the multilingual adaptation strategy shown in the table below:

[0111] In some embodiments, when the vehicle terminal dynamically adjusts the semantic boundary detection strategy based on the text features of the output text, it can use the following dynamic threshold adjustment function: calculate_adaptive_threshold(text_complexity): base_threshold = 2 / / Base threshold (number of characters) if text_complexity == HIGH: return base_threshold + 1 / / Increase the threshold for complex text elif text_complexity == LOW: return max(base_threshold - 1, 1) / / Lower the threshold for simple text Else: return base_threshold / / Use the base threshold for plain text.

[0112] In this embodiment, when performing word boundary recognition on streaming output text via the vehicle-mounted terminal, a semantic boundary detection strategy is first determined based on the language type of the streaming output text. Then, based on the text features of the streaming output text, the semantic boundary detection strategy is dynamically adjusted to obtain an adjusted semantic boundary detection strategy. Finally, word boundary recognition is performed on the streaming output text according to the adjusted semantic boundary detection strategy. Afterwards, semantic integrity verification is performed on the identified complete words in the streaming output text, and punctuation is filtered out from these complete words, resulting in a purified text with punctuation removed as the first complete text segment. This effectively improves the accuracy of semantic boundary recognition on streaming output text, can adapt to various text types for semantic boundary recognition on streaming output text, and also improves the success rate of anomaly handling in semantic boundary recognition on streaming output text.

[0113] Please refer to Figure 6 , Figure 6 for Figure 1 A detailed flowchart of step S103.

[0114] like Figure 6 As shown, in some embodiments, the step of “performing speech synthesis processing on the first complete text segment based on a preset first-character dedicated acceleration channel” in step S103 above may include steps S601 and S602 as shown below.

[0115] Step S601: Add the first complete text fragment to the high-priority queue in the first character dedicated acceleration channel.

[0116] It should be noted that the high-priority queue is a key component in the first-character-dedicated acceleration channel. The processing latency of the high-priority queue can be less than 5ms, meaning that the first complete text segment can begin speech synthesis processing within 5ms after entering the high-priority queue. The first-character-dedicated acceleration channel, based on the high-priority queue and other components, provides a priority processing path for the first complete text segment, ensuring that it is quickly synthesized into speech and played back. For example, the design of the first-character-dedicated acceleration channel can be shown in the table below:

[0117] When the vehicle terminal performs speech synthesis processing on the first complete text segment based on the first character-dedicated acceleration channel, it first adds the first complete text segment to the high-priority queue in the first character-dedicated acceleration channel, thereby skipping the regular queuing.

[0118] Step S602: Call the dedicated inference board resources of the first character dedicated acceleration channel to perform speech synthesis processing on the first complete text segment in the high priority queue.

[0119] It should be noted that dedicated inference board resources refer to dedicated hardware acceleration card resources used for artificial intelligence (AI) inference computing. In the context of vehicle systems, dedicated inference board resources often use NPUs or dedicated AI chips for TTS model inference.

[0120] After adding the first complete text segment to the high-priority queue, the vehicle terminal skips the regular queuing process and directly calls the dedicated inference board resources to perform speech synthesis processing on the first complete text segment in the high-priority queue.

[0121] In some embodiments, the vehicle terminal can call upon the dedicated inference board resources from the resource reservation pool of the first character dedicated acceleration channel to perform speech synthesis processing on the first complete text segment.

[0122] For example, when the vehicle terminal performs speech synthesis processing on the first complete text segment based on the first character dedicated acceleration channel, after obtaining the first complete text segment by recognizing the first token, it immediately adds the segment to the high-priority queue of the first character dedicated acceleration channel, and allocates dedicated inference board resources from the resource reservation pool of the first character dedicated acceleration channel. Then, it skips the regular queuing for the first complete text segment and immediately starts the speech synthesis process to use the dedicated inference board resources to perform speech synthesis processing on the first complete text segment.

[0123] Please refer to Figure 7 , Figure 7 for Figure 1 A detailed flowchart of step S104.

[0124] like Figure 7 As shown, in some embodiments, step S104 above: performing in-vehicle voice interaction based on the audio data may include steps S701 and S702 as shown below.

[0125] Step S701: Perform real-time push playback processing on the audio data based on the dedicated acceleration channel for the first character.

[0126] After the in-vehicle terminal performs speech synthesis processing on the first complete text segment in the high-priority queue to generate the corresponding audio data, it immediately pushes the audio data to the vehicle's client. Upon receiving the audio data, the client can immediately start playing the audio data, thus completing a rapid first-word response.

[0127] In some embodiments, the in-vehicle terminal can push the audio data of the first complete text segment to the in-vehicle client for playback via a high-speed in-vehicle bus.

[0128] Step S702: After playing the audio data, continue playing subsequent audio data; the subsequent audio data is obtained by speech synthesis processing of the subsequent text of the first complete text segment based on the conventional processing channel, and the conventional processing channel and the first character dedicated acceleration channel perform the speech synthesis task in parallel.

[0129] During the speech synthesis of the first complete text segment using the dedicated acceleration channel for the first character, if the in-vehicle terminal continuously receives subsequent text (subsequent tokens) output by the large language model LL, these subsequent texts can be automatically queued in the regular processing channel. The speech synthesis task for the first complete text segment using the dedicated acceleration channel for the first character is executed in parallel with the speech synthesis task for subsequent texts using the regular processing channel, without blocking each other. After the subsequent texts have been synthesized, the in-vehicle terminal also pushes the corresponding subsequent audio data to the vehicle's infotainment client. The vehicle's infotainment client can then seamlessly connect the preceding and following audio segments through its local audio buffer. After playing the audio data of the first complete text segment, it can continue playing the subsequent audio data, thus forming a continuous and smooth speech playback.

[0130] For example, the vehicle-mounted terminal can adopt such as Figure 8 The dedicated acceleration channel for the first character and the regular processing channel operate in parallel. After the first complete text segment (first token) is recognized, the complete text segment is added to the high-priority queue of the dedicated acceleration channel for the first character (processing latency <5ms, skipping the regular queue). Then, dedicated resource allocation is immediately performed. 20% of the reserved dedicated inference board resources are allocated from the resource reservation pool of the dedicated acceleration channel for the first character to start the priority synthesis of the first complete text segment. After the audio data of the first complete text segment is generated, it is immediately pushed to the vehicle client without buffering, achieving a total latency of 300-500ms for the first character playback. After that, the vehicle client continues to play subsequent audio data. In this process, the vehicle terminal executes a regular queue for subsequent text (subsequent tokens) through a regular processing channel. That is, the subsequent text enters the regular queue in sequence, waits for resource allocation, and is then synthesized in batches (multiple tokens are synthesized into audio segments together). After the subsequent audio data corresponding to the subsequent text is synthesized, it is buffered and pushed. That is, the subsequent audio data is accumulated to a certain amount before it is pushed. In this way, the continuous audio stream can arrive at the vehicle client one after another. The vehicle client can then achieve seamless connection between the preceding and following audio segments through the local audio buffer. After playing the audio data of the first complete text segment, it continues to play the subsequent audio data, forming a continuous and smooth merged voice playback.

[0131] In this embodiment, when the in-vehicle terminal performs speech synthesis processing on the first complete text segment based on the dedicated acceleration channel for the first character, the first complete text segment is first added to the high-priority queue in the dedicated acceleration channel. Then, for the first complete text segment in the high-priority queue, the regular queuing is skipped, and the dedicated inference board resources are directly invoked to perform speech synthesis processing on the first complete text segment. In this way, by designing a dedicated acceleration channel with dedicated resource reservation and high-priority queue scheduling, the response latency of the first character in the vehicle's voice interaction can be effectively reduced, while ensuring the continuous playback of subsequent voice messages, thereby significantly improving the naturalness and fluency of human-computer interaction.

[0132] Please refer to Figure 9 , Figure 9 The following are schematic flowcharts illustrating the steps of the in-vehicle voice interaction method provided in this application embodiment in other embodiments.

[0133] like Figure 9 As shown, in some embodiments, the in-vehicle voice interaction method provided in this application may further include steps S901 and S902 as shown below.

[0134] Step S901: Perform delay monitoring on multiple preset key nodes to obtain delay information for each of the multiple key nodes; the multiple key nodes include a first step of acquiring the streaming output text, a second step of semantic boundary awareness processing on the streaming output text, a third step of speech synthesis processing on the first complete text segment, and a fourth step of in-vehicle voice interaction based on the audio data.

[0135] The in-vehicle terminal can also employ an end-to-end first-word response optimization strategy. By establishing an end-to-end optimized link from the output of the Large Language Model (LLM) to the playback of the first word, the latency of each stage can be minimized. Based on this, the in-vehicle terminal can perform latency monitoring on multiple preset key nodes (the first stage of acquiring the streaming output text, the second stage of semantic boundary awareness processing of the streaming output text, the third stage of speech synthesis processing of the first complete text segment, and the fourth stage of in-vehicle voice interaction based on audio data) to obtain the latency information of each of these key nodes.

[0136] In some embodiments, the first stage can be the arrival of the first LLM token, the second stage can be the completion of semantic recognition, the third stage can be the start of TTS synthesis, and the fourth stage can be the push of the first audio. The vehicle terminal can set timestamps at these key nodes so as to monitor the delay information of each stage in real time.

[0137] Step S902: If the target stage times out based on the delay information, a preset response degradation strategy is triggered to directly play audio data for in-vehicle voice interaction; the target stage is any one of the first stage, the second stage, the third stage, and the fourth stage.

[0138] It should be noted that the response degradation strategy can be to output directly without waiting for semantic completeness. That is, the text fragments in the received streaming output text are immediately processed for speech synthesis and the synthesized audio data is pushed for playback, regardless of whether the semantics of the text fragments are complete.

[0139] After obtaining the delay information of each of the multiple key nodes, the vehicle terminal first determines whether any of the target stages in the first, second, third, and fourth stages has timed out based on the delay information. Then, if one or more target stages have timed out, a preset response degradation strategy is triggered to directly play audio data for in-vehicle voice interaction.

[0140] In some embodiments, if the vehicle terminal determines whether any of the target stages in the first, second, third, and fourth stages has timed out based on the delay information, it can determine one or more stages with a delay exceeding 100ms as the timed-out target stages if the timeout threshold for a certain stage is 100ms and the total timeout is 500ms.

[0141] In some embodiments, when minimizing latency in each stage by adopting an end-to-end first-word response optimization strategy, the vehicle terminal can also employ a pipelined parallel mechanism to enable LLM generation, first-token recognition, TTS synthesis, and audio transmission to be executed in parallel, with each stage decoupled through an asynchronous message queue.

[0142] In some embodiments, when the vehicle terminal adopts an end-to-end first-word response optimization strategy to minimize the latency of each link, it can also adopt a cache preheating mechanism. By preloading audio segments of high-frequency words (such as "okay", "currently", "for you" and other Top 100 commonly used first tokens), when the first complete text segment in the streaming output text hits the high-frequency word, the corresponding audio segment is played directly, thereby reducing the latency of first-word speech synthesis to 0.

[0143] In some embodiments, when minimizing latency at each stage by employing an end-to-end first-word response optimization strategy, the vehicle terminal may also employ network optimization mechanisms, such as WebSocket long connection reuse and Opus audio compression (16kbps bitrate), to reduce transmission latency (e.g., reducing transmission latency by approximately 30%).

[0144] In this embodiment, an end-to-end optimized link is established through the terminal device from the output of the Large Language Model (LLM) to the playback of the first character, minimizing the latency of each link. This can achieve a stable end-to-end first character response latency of 300-500ms, which greatly reduces the timeout rate of in-vehicle voice interaction.

[0145] Next, a complete embodiment of the in-vehicle voice interaction method provided in this application will be presented.

[0146] Please refer to Figure 10 , Figure 10 The system architecture diagram of the in-vehicle voice interaction method provided in the embodiments of this application is shown in some embodiments.

[0147] like Figure 10 As shown, in some embodiments, the in-vehicle voice interaction system may include three core layers: an LLM service layer, a first token processing layer, and a TTS synthesis layer. The LLM service layer includes streaming text generation (the large language model outputs text token by token in a streaming manner, and each token is sent immediately after generation) and a text stream buffer (maintaining a text stream buffer to provide a data foundation for first token recognition). The first token processing layer includes a semantic boundary detector (identifying word boundaries based on the jieba / HanLP word segmenter to ensure the semantic integrity of the first token), a first token recognizer (applying the minimum semantic unit rule to intelligently identify the first valid text segment), a dedicated acceleration queue (providing a high-priority processing channel for the first token, skipping the regular queuing), and buffer window management (a 50ms intelligent buffer window to balance latency and semantic integrity). The TTS synthesis layer includes a priority synthesizer (providing dedicated synthesis resources for the first token to ensure immediate processing), a parallel processor (synthesizing the first token and subsequent tokens in parallel to improve overall efficiency), a real-time pusher (pushing the first audio segment immediately after generation, without waiting for subsequent buffers), and audio stream transmission (using WebSocket to transmit audio data in real time).

[0148] The in-vehicle voice interaction system achieves intelligent recognition and accelerated processing of the first token when responding to user-initiated voice interactions through the collaborative work of the LLM service layer, the first token processing layer, and the TTS synthesis layer.

[0149] In some embodiments, when the in-vehicle voice interaction system performs intelligent first token recognition and accelerated processing of user voice interaction needs using the above-mentioned levels, the relevant data flow may include: LLM streaming output → text stream buffer → semantic boundary detection, first token recognition → dedicated acceleration queue → priority synthesis, first audio generation → real-time push → client playback, subsequent tokens → parallel processing → continuous audio stream.

[0150] In some embodiments, when the in-vehicle voice interaction system uses the above-mentioned levels to perform first-token intelligent recognition and accelerated processing of user voice interaction needs, the data structure of the relevant core data can be as follows: / / Structure of the first token recognition result: struct FirstTokenResult { string text; / / First token text content bool is_semantic_complete; / / Whether the semantics are complete int word_count; / / Number of complete words included long detect_time; / / Time taken for detection (milliseconds) SemanticBoundary boundary; / / Semantic boundary information float confidence_score; / / Recognition confidence score [0,1] }; / / Semantic boundary information structure: struct SemanticBoundary { int start_pos; / / Starting character position int end_pos; / / End character position string language_type; / / Language type: CN / EN / NUM / MIXED bool has_punctuation; / / Whether to include punctuation marks string[] word_list; / / List of words contained in the string. }; / / Speed-up queue element structure: struct AcceleratedQueueItem { string token_id; / / Unique identifier for the token string text_content; / / Text content Priority priority_level; / / Priority: FIRST_TOKEN / NORMAL long enqueue_time; / / Enqueue timestamp long processing_deadline; / / Processing deadline }; / / Buffer window state structure: struct BufferWindowState { string accumulated_text; / / Accumulated text content long window_start_time; / / Window start time bool semantic_complete; / / Whether semantic completeness has been achieved int received_token_count; / / Number of tokens received bool timeout_triggered; / / Whether to trigger a timeout }

[0151] In this embodiment, the in-vehicle voice interaction system employs the aforementioned layers to perform intelligent first-token recognition and accelerated processing of user voice interaction needs. Compared to traditional in-vehicle voice interaction solutions, it achieves comprehensive performance and experience upgrades. In terms of response speed, through intelligent semantic boundary recognition and a dedicated acceleration channel, the first token skips the conventional queuing and performs immediate speech synthesis processing, significantly reducing the first-word response latency from 3-5 seconds to 300-500ms, effectively lowering the overall latency. Regarding semantic quality, it abandons the simple length-based first-token recognition method and adopts semantic boundary awareness technology combined with the minimum semantic unit rule for first-token recognition, effectively improving semantic integrity and ensuring natural pronunciation and uninterrupted semantics. In terms of compatibility, it fully supports adaptive processing of mixed Chinese, English, and numerical text, solving the problem of limited multilingual support in traditional solutions. In terms of reliability, the newly added 50ms timeout protection and degradation mechanism fills the gap in traditional solutions' lack of anomaly handling, ensuring stable execution of the first-word response in in-vehicle voice interaction even under extreme conditions.

[0152] Please refer to Figure 11 This application also provides an in-vehicle voice interaction system, which can implement the above-mentioned in-vehicle voice interaction method.

[0153] like Figure 11 As shown in the embodiments of this application, the in-vehicle voice interaction system may include: The acquisition module is used to acquire the streaming output text of a large language model; The semantic boundary awareness module is used to perform semantic boundary awareness processing on the streaming output text to obtain the first complete text segment in the streaming output text; The speech acceleration synthesis module is used to perform speech synthesis processing on the first complete text segment based on a preset first-character dedicated acceleration channel to obtain the audio data of the first complete text segment; An audio processing module is used for in-vehicle voice interaction based on the audio data.

[0154] In some embodiments, the semantic boundary awareness module is further configured to perform word boundary recognition processing on the streaming output text to obtain complete words in the streaming output text; perform semantic verification processing on the complete words based on a preset minimum semantic unit rule to obtain a semantic integrity verification result of the complete words; when the semantic integrity verification result indicates that the semantic integrity of the complete words conforms to the minimum semantic unit rule, perform punctuation filtering processing on the complete words to obtain cleaned text after removing punctuation; the first complete text segment includes the cleaned text.

[0155] In some embodiments, the semantic boundary awareness module is further configured to determine a semantic boundary detection strategy based on the language type of the streaming output text; dynamically adjust the semantic boundary detection strategy based on the text features of the streaming output text to obtain an adjusted semantic boundary detection strategy; and perform word boundary recognition processing on the streaming output text based on the adjusted semantic boundary detection strategy.

[0156] In some embodiments, the text features include complexity, and the semantic boundary awareness module is further configured to increase the basic threshold of the number of characters in the semantic boundary detection strategy when the streaming output text is determined to be complex text based on the complexity; and to decrease the basic threshold of the number of characters in the semantic boundary detection strategy when the streaming output text is determined to be simple text based on the complexity.

[0157] In some embodiments, the speech acceleration synthesis module is further configured to add the first complete text segment to a high-priority queue in the first-character dedicated acceleration channel; and invoke the dedicated inference board resources of the first-character dedicated acceleration channel to perform speech synthesis processing on the first complete text segment in the high-priority queue.

[0158] In some embodiments, the audio processing module is further configured to perform real-time push playback processing on the audio data based on the first-character dedicated acceleration channel; continue playing subsequent audio data after playing the audio data; the subsequent audio data is obtained by speech synthesis processing of the subsequent text of the first complete text segment based on the conventional processing channel, and the conventional processing channel and the first-character dedicated acceleration channel perform the speech synthesis task in parallel.

[0159] In some embodiments, the in-vehicle voice interaction system provided in this application may further include: An end-to-end response optimization module is used to perform latency monitoring on multiple preset key nodes to obtain latency information for each of the multiple key nodes. The multiple key nodes include a first stage of acquiring the streaming output text, a second stage of semantic boundary awareness processing on the streaming output text, a third stage of speech synthesis processing on the first complete text segment, and a fourth stage of in-vehicle voice interaction based on the audio data. If a target stage times out based on the latency information, a preset response degradation strategy is triggered to directly play the audio data for in-vehicle voice interaction. The target stage is any one of the first stage, the second stage, the third stage, and the fourth stage.

[0160] It should be noted that the specific implementation of the in-vehicle voice interaction system provided in this application is basically the same as the specific implementation of the in-vehicle voice interaction method described above, and will not be repeated here.

[0161] Please see Figure 12 This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned vehicle voice interaction method.

[0162] In some embodiments, the electronic device may be any smart terminal such as a vehicle, an in-vehicle terminal, an in-vehicle hardware platform (e.g., an in-vehicle computer), a tablet computer, a smartphone, or a wearable device; or, the electronic device may be a vehicle including a memory and a processor.

[0163] like Figure 12 As shown, the electronic device provided in this application embodiment may include: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 using the in-vehicle voice interaction method of the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0164] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described in-vehicle voice interaction method.

[0165] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] This application also provides a computer program product, including a computer program. The steps implemented by the computer program when executed by a processor are basically the same as those in the specific embodiments of the above-described in-vehicle voice interaction method, and will not be repeated here.

[0167] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0168] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0169] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0171] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0172] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding factor is divided by the following factor, or that the related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0173] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0174] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A vehicle-mounted voice interaction method, characterized in that, The method includes: Obtain the streaming output text of a large language model; Semantic boundary awareness processing is performed on the streaming output text to obtain the first complete text segment in the streaming output text; The first complete text segment is processed by speech synthesis based on a preset first-character-dedicated acceleration channel to obtain the audio data of the first complete text segment; In-vehicle voice interaction is performed based on the audio data.

2. The method according to claim 1, characterized in that, The semantic boundary awareness processing of the streaming output text includes: The streaming output text is processed by word boundary recognition to obtain the complete words in the streaming output text; The semantic verification process is performed on the complete word based on the preset minimum semantic unit rule to obtain the semantic integrity verification result of the complete word; If the semantic integrity verification result indicates that the semantic integrity of the complete word conforms to the minimum semantic unit rule, the complete word is subjected to punctuation filtering to obtain the cleaned text after removing punctuation; the first complete text segment includes the cleaned text.

3. The method according to claim 2, characterized in that, The word boundary recognition processing of the streaming output text includes: The semantic boundary detection strategy is determined based on the language type of the streaming output text. The semantic boundary detection strategy is dynamically adjusted based on the text features of the streaming output text to obtain the adjusted semantic boundary detection strategy. The adjusted semantic boundary detection strategy is used to perform word boundary recognition processing on the streaming output text.

4. The method according to claim 3, characterized in that, The text features include complexity, and the dynamic adjustment of the semantic boundary detection strategy based on the text features of the streaming output text includes: If the streaming output text is determined to be complex text based on the complexity, the character count threshold in the semantic boundary detection strategy is increased. If the streaming output text is determined to be simple text based on the complexity, the character count threshold in the semantic boundary detection strategy is reduced.

5. The method according to claim 1, characterized in that, The process of performing speech synthesis on the first complete text segment based on a preset first-character-dedicated acceleration channel includes: Add the first complete text fragment to the high-priority queue in the first-character-dedicated acceleration channel; The dedicated inference board resources of the first character-specific acceleration channel are invoked to perform speech synthesis processing on the first complete text segment in the high-priority queue.

6. The method according to claim 1, characterized in that, The in-vehicle voice interaction based on the audio data includes: The audio data is pushed and played in real time based on the dedicated acceleration channel for the first character. After playing the audio data, subsequent audio data continues to be played; the subsequent audio data is obtained by speech synthesis processing of the subsequent text of the first complete text segment based on the conventional processing channel, and the speech synthesis task is performed in parallel with the first character dedicated acceleration channel.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Delay monitoring is performed on multiple preset key nodes to obtain the delay information of each key node; the multiple key nodes include a first step of acquiring the streaming output text, a second step of semantic boundary awareness processing of the streaming output text, a third step of speech synthesis processing of the first complete text segment, and a fourth step of in-vehicle voice interaction based on the audio data. If the target stage times out based on the delay information, a preset response degradation strategy is triggered to directly play audio data for in-vehicle voice interaction; the target stage is any one of the first stage, the second stage, the third stage, and the fourth stage.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the in-vehicle voice interaction method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the in-vehicle voice interaction method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the in-vehicle voice interaction method as described in any one of claims 1 to 7.

Images