Semantic response method and device, vehicle and storage medium

Abstract

The invention relates to the technical field of voice processing, and discloses a semantic response method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining acoustic features and semantic features of a streaming target audio segment of a user; obtaining fusion features according to the acoustic features and the semantic features; determining a semantic state tag corresponding to the streaming target audio segment according to the fusion feature; the semantic state tag comprises one of a semantic incomplete tag, a semantic complete tag or a semantic broken tag; and determining a corresponding response control strategy according to the semantic state tag corresponding to the streaming target audio segment, wherein different semantic state tags correspond to different response control strategies. According to the method, the semantic state can be accurately judged, the response strategy is dynamically determined, and the semantic response better fits the real intention of the user.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, specifically to semantic response methods, devices, vehicles, and storage media. Background Technology

[0002] With the development of intelligent customer service and in-vehicle voice systems, streaming dialogue systems need to balance real-time interaction with semantic accuracy. However, traditional segmented processing and static pause techniques have significant drawbacks. They rely on silence time thresholds (such as a preset 1-second pause to determine semantic termination) or text / audio length thresholds to trigger responses. These techniques cannot distinguish between pauses in user speech and natural semantic termination, nor can they capture the user's real-time interruption intent. For example, when a user says, "Please introduce the background of a song title... (pause 1 second)," the system may mistakenly interpret this incomplete phrase as a complete semantic segment, triggering a response prematurely. Furthermore, if the user intends to interrupt the introduction of the song's background (e.g., by suddenly making a new request), the system will continue outputting the original content before the silence time threshold is reached, failing to immediately terminate the invalid response, resulting in a significant decrease in interaction efficiency and user experience.

[0003] Therefore, how to accurately determine semantic state and dynamically determine response strategies has become a technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a semantic response method, apparatus, vehicle, and storage medium to solve the problems of inaccurate, erroneous, or untimely semantic state judgment.

[0005] In a first aspect, embodiments of this application provide a semantic response method, the method comprising: Acquire the acoustic and semantic features of the user's streaming target audio segment; The fused features are obtained based on the acoustic features and the semantic features; The semantic state label corresponding to the streaming target audio segment is determined based on the fusion features; the semantic state label includes one of the following: semantic incomplete label, semantic complete label, or semantic interruption label; The corresponding response control strategy is determined based on the semantic state label corresponding to the streaming target audio segment. Different semantic state labels correspond to different response control strategies.

[0006] The method provided in this application embodiment judges semantic state by fusing acoustic and semantic features of streaming target audio segments, and dynamically adjusts the response strategy based on the judgment result. This breaks the limitations of traditional static stop rules and achieves synergistic optimization of semantic understanding accuracy and interactive real-time performance.

[0007] The method provided in this application can solve the problem of being unable to distinguish between a pause in thought and the termination of semantics. Traditional techniques rely on a fixed 1-second silence time threshold or length threshold, which easily misjudges pauses in user statements as the end of semantics (such as the scenario of "Please introduce the name of a song... (pause 1 second) and its creative background"). The method provided in this application, however, simultaneously acquires the acoustic features (such as the fluctuation of speech rate and intonation continuity of pauses) and semantic features (such as the semantic correlation between "the name of a song" and "creative background") of the streaming target audio segment. By fusing the two, it can accurately identify that the pause is a user's thought process to complete the semantics, rather than a natural termination of semantics. This avoids judging incomplete segments such as "Please introduce the name of a song" as complete semantics, solves the problem of prematurely triggering invalid responses, and ensures the integrity of semantic understanding.

[0008] The method provided in this application can capture user interruption intentions in real time. When a user interrupts the system response midway (such as suddenly raising a new request during the introduction of the background of song creation), traditional solutions need to wait until a silence time threshold is reached before recognizing the interruption intention, and continue to output the original content during the period before the silence time threshold is reached. The method provided in this application is based on the characteristics of streaming processing, which can update the acoustic and semantic features of the audio segment in real time. When the user interrupts, the acoustic features will show changes such as sudden changes in tone and increased speech rate, and the semantic features will show new request keywords (such as "stop the introduction, help me find lyrics"). After the two are combined, a semantic interruption tag can be quickly generated, thereby timely terminating the original response and switching to the control strategy of handling the new request, completely solving the problem of continuous invalid response. Compared with the traditional solution that needs to wait until the silence time threshold is reached to recognize the interruption intention, this application can timely recognize the semantic interruption tag based on the streaming target audio segment, and can timely stop the original response, improving the interaction efficiency.

[0009] The method provided in this application can significantly improve interaction efficiency and user experience. Traditional static judgment logic lacks adaptability and is prone to frequent misjudgments when faced with different users' expression habits (such as varying speech speed and pause duration). In contrast, the method provided in this application dynamically determines semantic state by fusing features and then matches corresponding response control strategies. This adapts to diverse user interaction scenarios, eliminating the need for users to consciously adjust their expression rhythm to match the system's silent time threshold, and also eliminating the need for users to wait for the system to complete an invalid response before repeating their request. This significantly reduces interaction costs and makes semantic responses more aligned with the user's true intentions.

[0010] In one optional implementation, determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment includes: When the semantic state label is the semantically incomplete label, if the next audio segment is obtained within the first preset time period, the next audio segment is used as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly judged; or, if the next audio segment is not obtained within the first preset time period, the corresponding rhetorical question content is generated according to the semantic features; the rhetorical question content is used to guide the user to complete the semantic information.

[0011] The method provided in this application effectively solves the problem of misjudging user pauses due to static interval thresholds in traditional technologies by setting targeted processing logic for semantically incomplete fragments. When semantic incompleteness is detected, the method does not respond immediately, but waits for the user to supplement the content (as in the next audio segment) for a first preset time. This avoids misjudging incomplete fragments such as "Please introduce the name of a song... (pause 1s)" as complete semantics and triggering an invalid response. If the user does not continue to supplement the content, a guiding question is generated (such as "What information do you want to know about this song?"). This maintains the continuity of interaction and actively guides the user to complete the semantics, reducing interaction interruptions caused by semantic fragments and improving the fluency of the dialogue.

[0012] In an optional implementation, determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment further includes: When the semantic state label is the semantic complete label, if the next audio segment is not obtained within the second preset time period, the corresponding response content is generated according to the semantic features; or, if the next audio segment is obtained within the second preset time period, the next audio segment is used as a new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly judged; the response content is the generated content that responds to the user's needs, and the first preset time period is longer than the second preset time period.

[0013] The method provided in this application achieves fine-grained control over response timing by differentiating the waiting time between semantically complete and incomplete scenarios (a first preset duration is longer than a second preset duration). For semantically complete segments, a shorter second preset duration is used to ensure the system generates a response quickly (e.g., when the user explicitly says "introduce the background of a song's creation," a timely response can be provided), ensuring real-time interaction. Simultaneously, the ability to detect the next audio segment is retained. If the user adds content within a short time (e.g., continuing with "there's a release date"), the new content is incorporated into semantic understanding, avoiding the splitting of progressive semantics into independent segments and solving the problem of contextual fragmentation in traditional technologies. The setting of the duration difference balances the needs of "rapid response when semantically complete" and "allowing users to add progressive content," achieving a balance between efficiency and accuracy.

[0014] In an optional implementation, determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment further includes: When the semantic state label is the semantic interruption label, the generation of the rhetorical question content or the response content is stopped.

[0015] The method provided in this application effectively solves the problem of delayed interruption response in traditional technologies by recognizing semantic interruption tags and immediately terminating the current response generation. When a user expresses an intention to interrupt (such as "Wait a minute, I want to ask something else"), the system can quickly mark a semantic interruption tag by fusing acoustic features (such as tone changes and increased speech rate) and semantic features (such as keywords like "wait a minute" and "interrupt"), and immediately stop generating rhetorical questions or responses, avoiding invalid output that occupies interaction resources. This real-time interruption mechanism significantly improves the system's response speed to the user's immediate intentions, making the interaction more in line with human conversation habits and reducing the user's waiting experience.

[0016] In one optional implementation, acquiring the acoustic and semantic features of the user's streaming target audio segment includes: The target audio segment is input into the streaming audio encoder, and the acoustic features of the target audio segment are extracted by the streaming audio encoder. The acoustic features of the streaming target audio segment are input into the adapter layer, and the semantic features corresponding to the acoustic features are determined by the adapter layer.

[0017] The method provided in this application achieves high efficiency and relevance in feature extraction through the collaborative design of a streaming audio encoder and an adapter layer. The streaming audio encoder can process continuously input audio segments in real time, avoiding delays caused by waiting for complete audio and ensuring real-time interaction. The adapter layer establishes a mapping bridge from acoustic features to semantic features (such as converting acoustic signals like pauses and intonation into semantic tendencies like "thinking" or "affirming"), so that semantic features not only rely on text conversion but also integrate key information such as prosody and rhythm in speech, improving the comprehensiveness of semantic understanding. For example, when a user says "Please introduce… (hesitating and pausing)," the pause duration and intonation changes in the acoustic features can help the adapter layer more accurately judge "semantic incompleteness," reducing misjudgments based solely on text features.

[0018] In one optional implementation, obtaining the fused features based on the acoustic features and the semantic features includes: The acoustic features and semantic features are dimensionally aligned to obtain acoustic feature vectors and semantic feature vectors; The acoustic feature vector and the semantic feature vector are normalized respectively to obtain the normalized acoustic feature vector and the normalized semantic feature vector; The fused feature is obtained by calculating the sum of the elements of the same dimension in the normalized acoustic feature vector and the normalized semantic feature vector.

[0019] The method provided in this application, through a fusion strategy of dimension alignment, normalization, and element summation, ensures the effective combination of acoustic and semantic features. Dimension alignment resolves the fusion barrier caused by the different representation dimensions of the two types of features, laying the foundation for cross-modal information fusion. Normalization eliminates differences in feature magnitude (such as the amplitude range of acoustic features differing from the probability value range of semantic features), preventing one type of feature from dominating the fusion process and allowing acoustic features (such as pause duration) and semantic features (such as keyword completeness) to participate equally in semantic state judgment. The element summation fusion method strengthens the synergistic effect of the two types of features on the same dimension (such as "pause acoustic features" and "semantic incompleteness tendency" superimposed on the same dimension), improving the ability of the fused features to represent semantic states and making subsequent label judgment more accurate.

[0020] In one optional implementation, determining the semantic state label corresponding to the streaming target audio segment based on the fusion features includes: The fused features are input into the encoder of the recurrent neural network transformer to obtain the temporal coding features corresponding to the fused features; Obtain the contextual features generated by the prediction network of the recurrent neural network transformer based on the semantic state labels of the historical output; By fusing the temporal coding features and the contextual association features through the joint network of the recurrent neural network transformer, the probability distribution of each semantic state label corresponding to the streaming target audio segment is obtained; Based on the probability distribution, the semantic state label with the highest probability is selected as the semantic state label corresponding to the streaming target audio segment.

[0021] The method provided in this application significantly improves the accuracy of semantic state label judgment through the temporal modeling and context association mechanism of the recurrent neural network transformer. The temporal coding features generated by the encoder can capture the temporal dependencies of streaming audio segments (such as the semantic connection between the previous segment and the current segment), solving the semantic fragmentation problem caused by independent judgment of single segments in traditional technologies; the context association features generated by the prediction network incorporate historical labels (such as the previous segment being "semantically incomplete") into the current judgment, enabling the system to dynamically adjust label prediction based on the dialogue history (such as two consecutive "semantically incomplete" segments can strengthen the judgment of "the user is supplementing content"). The fusion of the two types of features by the joint network results in a probability distribution that more comprehensively reflects the possibility of the current semantic state, making label selection (such as "semantically complete" or "interrupted") more in line with the actual dialogue scenario, especially in multi-turn progressive dialogues, significantly reducing the mislabeling rate caused by isolated judgments.

[0022] Secondly, embodiments of this application provide a semantic response device, the device comprising: The first processing module is used to acquire the acoustic and semantic features of the user's streaming target audio segment; The second processing module is used to obtain fused features based on the acoustic features and the semantic features; The third processing module is used to determine the semantic state label corresponding to the streaming target audio segment based on the fusion features; the semantic state label includes one of a semantically incomplete label, a semantically complete label, or a semantically interrupted label. The fourth processing module is used to determine the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment. Different semantic state labels correspond to different response control strategies.

[0023] Thirdly, embodiments of this application provide a vehicle, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the semantic response method of the first aspect or any of its corresponding optional embodiments.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform the semantic response method of the first aspect or any of its corresponding optional embodiments.

[0025] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the semantic response method of the first aspect or any of its corresponding optional embodiments. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the semantic response method according to an embodiment of this application; Figure 2 This is a schematic diagram of the decoding process of the RNN-T structure according to an embodiment of this application; Figure 3 This is a schematic diagram of the semantic stop determination system for a streaming voice dialogue large model based on the RNN-T structure, according to an embodiment of this application. Figure 4 This is a structural block diagram of the semantic response device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the vehicle according to an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Traditional segmented processing and static pause judgment techniques have significant drawbacks. Their core reliance on fixed-length speech segments or preset silence time thresholds as the basis for judging semantic completeness makes them ill-suited to the fragmented, hierarchical, and dynamic nature of natural dialogue. This leads to semantic fragmentation misjudgment. They cannot effectively distinguish between incomplete semantic segments and sub-parts of higher-level semantics. Relying solely on preset rules (such as silence time thresholds) to determine semantic completeness easily misjudges incomplete segments during user pauses as complete semantics. For example, with a silence time threshold set to 0.8 seconds, when a user states "Please introduce the song title… (pause 1 second) and its background," the system might, upon receiving the incomplete segment "Please introduce the song title," determine that the silence time threshold has been reached, the semantics have ended, and trigger a response based on "Please introduce the song title," generating a reply that deviates from the user's actual need ("Please introduce the song title and its background").

[0030] There is a lag in the response to interruptions. Faced with real-time interruptions from users, the technology lacks the ability to immediately perceive and process them. Even if the user has clearly expressed their intention to interrupt (e.g., by making a new request midway), the model will continue to complete the current response according to the preset process before the silence time threshold is reached, failing to terminate invalid output in a timely manner. For example, while the system is outputting a response on a certain topic (e.g., the background of a song's creation), if the user suddenly requests an interruption such as "Wait a minute, I want to learn about another song," the existing technology will continue to output the original topic content before the silence time threshold is reached, significantly reducing the real-time nature of the interaction and prolonging the user's waiting time.

[0031] There is a problem of contextual fragmentation. The static pausing strategy does not consider the nesting and progression of semantics in multi-turn dialogues. It pauses each segment of streaming audio independently, failing to establish semantic connections across segments. This results in the complete semantics supplemented by the user in stages being broken down into multiple isolated semantics. For example, if a user inputs streaming audio segments twice, the first segment being "Please introduce the name of a certain building," and the second segment being "a certain pavilion inside," the complete semantics should be "Please introduce a certain pavilion inside a certain building." However, traditional technology would determine the semantic completeness and trigger a response upon receiving the first segment, outputting an introduction to the Forbidden City as a whole. The subsequent second segment would then be treated as a new, independent request, causing fragmentation and bias in semantic understanding.

[0032] These shortcomings lead to reduced interaction efficiency and impaired user experience in streaming dialogue systems, making it difficult to meet the requirements for accuracy and real-time interaction in natural dialogue scenarios.

[0033] Therefore, this application provides a semantic response method embodiment to solve the above-mentioned problems. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides a semantic response method. Figure 1 This is a flowchart illustrating the semantic response method according to an embodiment of this application, as shown below. Figure 1 As shown, the process includes the following steps: S101, Obtain the acoustic and semantic features of the user's streaming target audio segment.

[0035] In this embodiment, it is necessary to acquire the user's streaming audio. An audio segment of a preset duration can be used as the target streaming audio segment; for example, the preset duration could be 320ms. In vehicle application scenarios, the driver's / passenger's voice can be captured using an in-vehicle microphone.

[0036] Acoustic features include, but are not limited to: prosodic features of audio (such as pause duration, intonation variation trend, speech rate fluctuation), spectral features (such as Mel frequency cepstral coefficient, audio frequency distribution characteristics), etc. Acoustic features reflect the acoustic properties of speech and can reflect non-verbal information such as rhythm and tone when a user speaks.

[0037] Semantic features include, but are not limited to: the text content of the streaming target audio segment, the completeness of keywords, the grammatical structure of sentences, and the logical connection tendency of semantic segments. Semantic features are used to reflect the semantic information of speech content, such as whether the segment carries sufficient semantic information and whether there is logical incompleteness.

[0038] S102, fused features are obtained based on acoustic features and semantic features.

[0039] In this embodiment, feature fusion can be achieved through steps of dimension alignment, normalization, and vector addition to obtain fused features.

[0040] S103, determine the semantic state label corresponding to the streaming target audio segment based on the fusion features.

[0041] In this embodiment, different semantic state tags correspond to different response control strategies. Semantic state tags include one of the following: semantically incomplete tag, semantically complete tag, or semantically interrupted tag. A semantically incomplete tag indicates that the target audio segment of the streaming audio is semantically incomplete and requires waiting for subsequent audio segments to determine whether the current target audio segment and the next audio segment constitute complete semantics. A semantically complete tag indicates that the target audio segment of the streaming audio is semantically complete. A semantically interrupted tag indicates that the user wants to interrupt the ongoing response. In this embodiment, the target audio segment of the streaming audio includes at least one audio frame, and the semantic state tag corresponding to each audio frame in the target audio segment can be determined. In this embodiment, the next audio segment is the audio segment that is temporally adjacent to the current target audio segment and has the same preset time length (e.g., 320ms) as the current segment. For example, if the current target audio segment corresponds to a time interval of (t, t+320]ms, then the next audio segment corresponds to an audio segment with a time interval of (t+320, t+640]ms.

[0042] S104, determine the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment.

[0043] In this embodiment, different semantic state labels correspond to different response control strategies; one semantic state label corresponds to one response control strategy. After determining the corresponding response control strategy based on the semantic state label of the streaming target audio segment, response control is performed according to the determined response control strategy. In this embodiment, when the semantic state label corresponding to each audio frame in the streaming target audio segment is determined, the corresponding response control strategy can be determined based on the semantic state label of each audio frame in the streaming target audio segment.

[0044] As an example, the specific implementation of determining the corresponding response control strategy based on the semantic state label of each audio frame in the streaming target audio segment is as follows: The dominant semantic state label is determined based on the semantic state label corresponding to each audio frame in the streaming target audio segment, and the corresponding response control strategy is determined based on the dominant semantic state label. The dominant semantic state label belongs to one of the following: semantically incomplete label, semantically complete label, or semantically interrupted label.

[0045] For example, if a semantic interruption label exists in the semantic state tags corresponding to each audio frame in the streaming target audio segment, the semantic interruption label is used as the dominant semantic state label. If no semantic interruption label exists, the proportion of semantically incomplete labels and semantically complete labels in the semantic state tags corresponding to each audio frame in the streaming target audio segment is determined, and the label with the larger proportion is determined as the dominant semantic state label. Alternatively, the proportion of semantically incomplete labels and semantically complete labels in a predetermined number of subsequent audio frames is calculated, and the label with the larger proportion is determined as the dominant semantic state label. Subsequent audio frames, being closer to the end of the current speech, have more timely semantic judgments.

[0046] As an example, a 320ms streaming target audio segment contains 10 consecutive audio frames (referred to as frames 1 to 10 in chronological order), with each audio frame corresponding to a semantic state label.

[0047] Prioritize detecting the presence of semantic interruption tags: If any frame out of 10 frames is a semantic interruption tag (e.g., frame 5 is a semantic interruption tag), then directly determine the dominant semantic state tag of the streaming target audio segment as a semantic interruption tag.

[0048] If there is no semantic interruption label, then the subsequent frames are judged comprehensively based on the principle of higher weight. The label distribution of frames 6 to 10 is statistically analyzed. If the proportion of semantically complete labels is ≥60% (e.g., 3 frames from frames 6 to 10 have semantically complete labels), then the dominant semantic state label is determined to be a semantically complete label.

[0049] If the percentage of semantically complete labels in subsequent frames is less than 60%, and most of them are semantically incomplete labels (e.g., 4 frames in frames 6-10 have semantically incomplete labels), then the dominant semantic state label is determined to be a semantically incomplete label.

[0050] This application embodiment judges semantic state by fusing acoustic and semantic features of streaming target audio segments, and dynamically adjusts the response strategy based on the judgment result, breaking the limitations of traditional static stop rules and achieving synergistic optimization of semantic understanding accuracy and interactive real-time performance.

[0051] The method provided in this application can solve the problem of being unable to distinguish between a pause in thought and the termination of semantics. Traditional techniques rely on a fixed 1-second silence time threshold or length threshold, which easily misjudges pauses in user statements as the end of semantics (such as the scenario of "Please introduce the name of a song... (pause 1 second) and its creative background"). The method provided in this application, however, simultaneously acquires the acoustic features (such as the fluctuation of speech rate and intonation continuity of pauses) and semantic features (such as the semantic correlation between "the name of a song" and "creative background") of the streaming target audio segment. By fusing the two, it can accurately identify that the pause is a user's thought process to complete the semantics, rather than a natural termination of semantics. This avoids judging incomplete segments such as "Please introduce the name of a song" as complete semantics, solves the problem of prematurely triggering invalid responses, and ensures the integrity of semantic understanding.

[0052] The method provided in this application can capture user interruption intentions in real time. When a user interrupts the system response midway (such as suddenly raising a new request during the introduction of the background of song creation), traditional solutions need to wait until a silence time threshold is reached before recognizing the interruption intention, and continue to output the original content during the period before the silence time threshold is reached. The method provided in this application is based on the characteristics of streaming processing, which can update the acoustic and semantic features of the audio segment in real time. When the user interrupts, the acoustic features will show changes such as sudden changes in tone and increased speech rate, and the semantic features will show new request keywords (such as "stop the introduction, help me find lyrics"). After the two are combined, a semantic interruption tag can be quickly generated, thereby timely terminating the original response and switching to the control strategy of handling the new request, completely solving the problem of continuous invalid response. Compared with the traditional solution that needs to wait until the silence time threshold is reached to recognize the interruption intention, this application can timely recognize the semantic interruption tag based on the streaming target audio segment, and can timely stop the original response, improving the interaction efficiency.

[0053] The method provided in this application can significantly improve interaction efficiency and user experience. Traditional static judgment logic lacks adaptability and is prone to frequent misjudgments when faced with different users' expression habits (such as varying speech speed and pause duration). In contrast, the method provided in this application dynamically determines semantic state by fusing features and then matches corresponding response control strategies. This adapts to diverse user interaction scenarios, eliminating the need for users to consciously adjust their expression rhythm to match the system's silent time threshold, and also eliminating the need for users to wait for the system to complete an invalid response before repeating their request. This significantly reduces interaction costs and makes semantic responses more aligned with the user's true intentions.

[0054] This application takes semantic state labels as examples of semantic incomplete label, semantic complete label, and semantic interruption label, and provides a detailed explanation of the response control strategies corresponding to the semantic state labels.

[0055] In an optional implementation, the semantic state label is a semantically incomplete label. S104 determines the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment, including: When the semantic state label is a semantically incomplete label, if the next audio segment is obtained within the first preset time period, the next audio segment is taken as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly judged. Alternatively, if the next audio segment is not obtained within the first preset time period, the corresponding rhetorical question is generated based on the semantic features.

[0056] In this embodiment of the application, the rhetorical question is used to guide the user to complete the semantic content.

[0057] In this embodiment of the application, as an example, the first preset duration can be 1.5 seconds. In this embodiment of the application, the specific output and presentation format of the rhetorical question and the response content in the subsequent content are not limited. As an example, the rhetorical question and the response content in the subsequent content can be output and presented in the form of text or voice.

[0058] If the next audio segment is not obtained within the first preset time period, the semantic features (incomplete semantics) are input into the Large Language Model (LLM) to obtain the rhetorical question. For example, if the current request is "I want to know...", the LLM might generate the rhetorical question: "What do you want to know?" In this application, the specific type of LLM is not limited; the LLM can be a commonly used LLM designed for dialogue scenarios. Based on the input semantic features, it can generate natural language content that conforms to dialogue logic, guides semantic completion, or meets complete requirements.

[0059] As an example, the next audio segment is taken as the new streaming target audio segment. The semantic state label corresponding to the new streaming target audio segment is repeatedly judged. When the semantic state label corresponding to the new streaming target audio segment is a semantically complete label, the semantic features corresponding to the original streaming target audio segment and the semantic features corresponding to the new streaming target audio segment can be input into the large language model to obtain the response content.

[0060] As an example, the next audio segment is taken as the new streaming target audio segment. The semantic state label corresponding to the new streaming target audio segment is repeatedly judged. When the semantic state label corresponding to the new streaming target audio segment is a semantically incomplete label, it can be re-judged whether the next audio segment has been obtained within the first preset duration.

[0061] As an example, the next audio segment is taken as the new streaming target audio segment. The semantic state label corresponding to the new streaming target audio segment is repeatedly judged. When the semantic state label corresponding to the new streaming target audio segment is a semantic interruption label, the semantic interruption label can be sent to the large language model, and the large language model stops generating rhetorical questions or response content.

[0062] This application's embodiments effectively solve the problem of misjudging user pauses due to static interval thresholds in traditional technologies by setting targeted processing logic for semantically incomplete fragments. When semantic incompleteness is detected, the system does not respond immediately but waits for the user to supplement the content (as in the next audio segment) for a first preset time. This avoids misjudging incomplete fragments such as "Please introduce the name of a song... (pause 1s)" as complete semantics and triggering an invalid response. If the user does not continue to supplement, guiding questions are generated (such as "What information would you like to know about this song?"), maintaining interactive continuity while proactively guiding the user to complete the semantics, reducing interaction interruptions caused by semantic fragments, and improving dialogue fluency.

[0063] In an optional implementation, the semantic state label is a semantically complete label. S104 determines the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment, and further includes: When the semantic state label is a semantically complete label, if the next audio segment is not obtained within the second preset time period, the corresponding response content is generated based on the semantic features. Alternatively, if the next audio segment is obtained within the second preset time period, the next audio segment is used as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly determined.

[0064] In this embodiment of the application, the response content is the generated content responding to the user's request, and the first preset duration is longer than the second preset duration.

[0065] In this embodiment of the application, as an example, the second preset duration can be 0.8s.

[0066] In this embodiment of the application, if the next audio segment is not obtained within the second preset time period, the semantic features (complete semantics) are input into the large language model to obtain the response content.

[0067] In this embodiment, to achieve rapid response, when the semantic state label is a semantically complete label, the semantic features (complete semantics) can be input into the large language model to obtain the response content, but the response content is not output or presented. The response content is only output if the next audio segment is not obtained within a second preset time period. The semantic features (complete semantics) can be input into the large language model in advance when the semantically complete label is obtained to obtain the response content.

[0068] If the next audio segment is obtained within the second preset time period, the next audio segment will be used as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment will be judged repeatedly.

[0069] As an example, when the semantic state label corresponding to the new streaming target audio segment is a semantically complete label, the semantic features corresponding to the original streaming target audio segment and the semantic features corresponding to the new streaming target audio segment can be input into the large language model to obtain the response content.

[0070] As an example, when the semantic state label corresponding to the new streaming target audio segment is a semantically incomplete label, the semantic features corresponding to the original streaming target audio segment can be input into the large language model to obtain the response content.

[0071] As an example, when the semantic state label corresponding to a new streaming target audio segment is a semantic break label, the semantic break label can be sent to the large language model, and the large language model will stop generating rhetorical questions or response content.

[0072] This application's embodiments achieve fine-grained control over response timing by differentiating the waiting time between semantically complete and incomplete scenarios (the first preset duration is longer than the second preset duration). For semantically complete segments, a shorter second preset duration is used to ensure the system generates a rapid response (e.g., when the user explicitly says "introduce the background of a song's creation," a timely response can be provided), ensuring real-time interaction. Simultaneously, the ability to detect the next audio segment is retained. If the user adds content within a short time (e.g., continuing with "there's a release date"), the new content is incorporated into semantic understanding, avoiding the splitting of progressive semantics into independent segments and solving the problem of contextual fragmentation in traditional technologies. The duration difference setting balances the needs of "rapid response when semantically complete" and "allowing users to add progressive content," achieving a balance between efficiency and accuracy.

[0073] In an optional implementation, the semantic state label is a semantic interruption label. S104 determines the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment, and further includes: When the semantic state label is a semantic interruption label, stop generating rhetorical questions or response content.

[0074] In this embodiment of the application, when the semantic state label is a semantic interruption label, the semantic interruption label can be sent to the large language model. When the large language model receives the semantic interruption label, it stops generating rhetorical questions or response content.

[0075] This application's embodiments effectively solve the problem of delayed interruption responses in traditional technologies by recognizing semantic interruption tags and immediately terminating the current response generation. When a user expresses an intention to interrupt (such as "Wait a minute, I want to ask something else"), the system can quickly mark the semantic interruption tag by fusing acoustic features (such as tone changes and increased speech rate) and semantic features (such as keywords like "wait a minute" and "interrupt"), and immediately stop generating the rhetorical question or response content, avoiding invalid output that occupies interaction resources. This real-time interruption mechanism significantly improves the system's response speed to the user's immediate intentions, making the interaction more in line with human conversation habits and reducing the user's waiting experience.

[0076] In one optional implementation, S101 acquires the acoustic and semantic features of the user's streaming target audio segment, specifically including steps Sa1 to Sa2.

[0077] Sa1 inputs the streaming target audio segment into the streaming audio encoder, which extracts the acoustic features of the streaming target audio segment.

[0078] In this embodiment of the application, the streaming audio encoder extracts features of the streaming target audio segment layer by layer through its internal neural network (such as convolutional layer, recurrent layer, etc.), gradually abstracting from the basic acoustic information at the bottom layer (such as waveform, frequency, etc.) upwards, and finally taking the features output from the top layer of the neural network as the acoustic features (i.e., high-level acoustic features) of the streaming target audio segment. These high-level acoustic features can reflect the key acoustic properties of the audio.

[0079] Sa2 inputs the acoustic features of the target audio segment of the streaming into the adapter layer, and determines the semantic features corresponding to the acoustic features through the adapter layer.

[0080] In this embodiment, the acoustic features extracted from Sa1 are input into an adapter layer, which consists of two fully connected layers. The first fully connected layer performs linear transformations and nonlinear activations (such as ReLU activation) on the acoustic features, initially converting them from the acoustic feature space to an intermediate feature space. The second fully connected layer then further processes these features, completing the mapping from the intermediate feature space to the semantic space of the large language model. Finally, the hidden layer representation output by the adapter layer (two fully connected layers) is taken as the semantic feature of the streaming target audio segment. This semantic feature reflects the semantic information of the audio content.

[0081] In this embodiment, the acoustic features of the streaming audio encoder and the semantic features of the adapter layer can be directly reused, saving computing resources and improving response speed.

[0082] This application's embodiments achieve high efficiency and relevance in feature extraction through the collaborative design of a streaming audio encoder and an adapter layer. The streaming audio encoder can process continuously input audio segments in real time, avoiding delays caused by waiting for complete audio and ensuring real-time interaction. The adapter layer establishes a mapping bridge from acoustic features to semantic features (such as converting acoustic signals like pauses and intonation into semantic tendencies like "thinking" or "affirmative"), enabling semantic features to not only rely on text conversion but also integrate key information such as prosody and rhythm in speech, thus improving the comprehensiveness of semantic understanding. For example, when a user says "Please introduce… (hesitating and pausing)," the pause duration and intonation changes in the acoustic features can help the adapter layer more accurately determine "semantic incompleteness," reducing misjudgments based solely on text features.

[0083] In an optional implementation, S102 obtains fused features based on acoustic features and semantic features, specifically including steps Sb1 to Sb3.

[0084] Sb1 aligns the acoustic and semantic features in terms of dimensions to obtain acoustic feature vectors and semantic feature vectors.

[0085] In this embodiment, since the acoustic features output by the streaming audio encoder and the semantic features output by the adapter layer (two fully connected layers) may have different initial dimensions, the acoustic features are converted into acoustic feature vectors and the semantic features output by the adapter layer are converted into semantic feature vectors by setting the same output dimension as the acoustic features in the second fully connected layer of the adapter layer, or by adding a lightweight fully connected layer to adjust the dimension after the acoustic features are extracted. This ensures that the number of dimensions of the two are completely consistent (e.g., both are adjusted to 512 dimensions), and that subsequent element-level operations can be performed normally.

[0086] Sb2 normalizes the acoustic feature vector and the semantic feature vector respectively, resulting in normalized acoustic feature vector and normalized semantic feature vector.

[0087] In this embodiment, the dimension-aligned acoustic feature vector and semantic feature vector are normalized. For example, Z-score normalization is used, which calculates the mean and standard deviation of each feature vector to convert the element values ​​of each dimension into values ​​conforming to a standard normal distribution (mean of 0 and standard deviation of 1). Alternatively, min-max normalization is used to map the element values ​​to the interval [0, 1]. Normalization balances the numerical distribution range of the two types of features, preventing a certain type of feature from being ignored during fusion due to excessively large differences in the numerical magnitudes of acoustic and semantic features.

[0088] Sb3 calculates the sum of elements of the same dimension in the normalized acoustic feature vector and the normalized semantic feature vector to obtain the fused feature.

[0089] In this embodiment of the application, the normalized acoustic feature vector (denoted as [A1, A2, ..., A...) is... n [[S1, S2, ..., Sn], where n is the number of dimensions] and semantic feature vectors (denoted as [S1, S2, ..., Sn]). n The vector addition operation is used to sum the elements element by element along the same dimension. That is, the sum of the elements in the first dimension is A1 + S1, the sum of the elements in the second dimension is A2 + S2, ..., the sum of the elements in the nth dimension is A... n +S n Finally, a fusion feature vector [F1, F2, ..., Fn] of dimension n is obtained. n (where F) i =A i +S i This fusion feature retains both the prosody and spectrum information of acoustic features and the semantic meaning and logical connections of semantic features, providing a comprehensive feature basis for subsequent semantic state label judgment.

[0090] This application's embodiments employ a fusion strategy involving dimension alignment, normalization, and element summation to ensure the effective combination of acoustic and semantic features. Dimension alignment resolves fusion barriers caused by the different representation dimensions of the two types of features, laying the foundation for cross-modal information fusion. Normalization eliminates differences in feature magnitude (e.g., the amplitude range of acoustic features differs from the probability value range of semantic features), preventing one type of feature from dominating the fusion process and ensuring that acoustic features (e.g., pause duration) and semantic features (e.g., keyword completeness) participate equally in semantic state judgment. The element summation fusion method strengthens the synergistic effect of the two types of features on the same dimension (e.g., "pause acoustic features" and "semantic incompleteness tendency" are superimposed on the same dimension), improving the fused features' ability to represent semantic states and making subsequent label judgments more accurate.

[0091] In an optional implementation, S103 determines the semantic state label corresponding to the streaming target audio segment based on the fusion features, specifically including steps Sc1 to Sc4.

[0092] Sc1 inputs the fused features into the encoder of the Recurrent Neural Network Transducer (RNN-T) to obtain the temporal encoded features corresponding to the fused features.

[0093] In this embodiment, the fusion features (feature vectors of dimension n) obtained from Sb3 are input into the encoder part of the recurrent neural network transformer in the temporal order of the streaming target audio segments. The encoder performs temporal modeling on the fusion features through an internal recurrent neural network (such as LSTM) or Transformer structure, capturing the dependencies between fusion features at different time steps (such as the semantic association between the current streaming target audio segment and the previous audio segment), and finally outputs temporal encoded features containing temporal context information. These temporal encoded features not only retain the acoustic and semantic integration information in the fusion features, but also enhance the representation of the temporal continuity of the speech stream.

[0094] Sc2 obtains the contextual features generated by the predictive network of the recurrent neural network transformer based on the semantic state labels of the historical outputs.

[0095] In this embodiment, the prediction network of the recurrent neural network transformer takes the semantic state labels of historical outputs (i.e., the semantic state labels corresponding to the preceding streaming target audio segments, including semantically incomplete labels, semantically complete labels, and semantically interrupted labels) as input. It first maps each historical output semantic state label to a preset embedding vector (the three types of labels correspond to three independent embedding representations). Then, it uses a recurrent structure (such as a unidirectional LSTM) within the prediction network to temporally encode these embedding vectors, generating contextual features that reflect the dependencies between historical labels. For example, if the preceding labels are mostly semantically incomplete labels, this contextual feature will carry a tendency to suggest that "the current semantics may still be incomplete."

[0096] Sc3 uses a joint network of recurrent neural network transformers to fuse temporal coding features and contextual features to obtain the probability distribution of each semantic state label corresponding to the streaming target audio segment.

[0097] In this embodiment, the joint network of the recurrent neural network transformer (typically composed of fully connected layers and activation functions) receives the temporal coding features output by Sc1 and the contextual features output by Sc2. It then deeply fuses these two features through vector concatenation or element-level interactions (such as addition and multiplication) to obtain fused data. The fused data is mapped by a fully connected layer to a dimension matching the number of semantic state labels (3-dimensional in this embodiment, corresponding to three types of labels). It is then converted by a softmax function into a probability distribution of the three semantic state labels (incomplete semantic label, complete semantic label, and interrupted semantic label) corresponding to the streaming target audio segment. The value of each dimension represents the probability that the current streaming target audio segment belongs to the corresponding label.

[0098] Sc4 selects the semantic state label with the highest probability as the semantic state label corresponding to the streaming target audio segment based on the probability distribution.

[0099] In this embodiment, the probability distribution output by Sc3 is filtered by maximum value, and the semantic state label corresponding to the dimension with the highest probability value is selected as the final semantic state label of the current streaming target audio segment. For example, if the probability of the semantically incomplete label is 0.1, the probability of the semantically complete label is 0.7, and the probability of the semantically interrupted label is 0.2, then the semantically complete label with the highest probability is selected as the semantic state label corresponding to the streaming target audio segment.

[0100] This application significantly improves the accuracy of semantic state labeling through temporal modeling and contextual association mechanisms using recurrent neural network transformers. The temporal coding features generated by the encoder can capture the temporal dependencies of streaming audio segments (such as the semantic connection between previous and current segments), solving the semantic fragmentation problem caused by independent judgment of single segments in traditional technologies. The contextual association features generated by the prediction network incorporate historical labels (such as "semantically incomplete" in the previous segment) into the current judgment, enabling the system to dynamically adjust label predictions based on dialogue history (such as two consecutive "semantically incomplete" segments can strengthen the judgment of "user is supplementing content"). The fusion of the two types of features by the joint network results in a probability distribution that more comprehensively reflects the probability of the current semantic state, making label selection (such as "semantically complete" or "interrupted") more consistent with actual dialogue scenarios, especially in multi-turn progressive dialogues, significantly reducing the mislabeling rate caused by isolated judgments.

[0101] In this application, the decoding process of the RNN-T structure is as follows: Figure 2As shown, the fused features (such as T1, T2, and T3, obtained by fusing acoustic and semantic features) are input into the fully connected layer of the recurrent neural network transformer joint network. The fully connected layer processes the fused features (e.g., dimensionality mapping, nonlinear transformations, etc.) and outputs corresponding semantic state labels. Different labels correspond to different interaction logics. If the output label indicates incomplete semantics, it determines whether the next audio segment has been acquired within a first preset time period (used to complete the semantics; at this time, the large language model does not generate a complete response and is in a waiting state). If the output label indicates complete semantics, it determines whether the next audio segment has been acquired within a second preset time period (if the semantics are sufficiently complete, the large language model will generate a response based on this semantics). If the output label indicates semantic interruption, the ongoing response of the large language model is directly interrupted, for example, if the user actively interrupts the current dialogue flow.

[0102] In this application, a semantic stop determination system for a large-scale streaming speech dialogue model with an RNN-T structure can be constructed based on the semantic response method of this application. The overall structure of the system is as follows: Figure 3 As shown, streaming audio (i.e., dialogue requests initiated by the user via voice, carrying the user's dialogue needs or instructions that the large language model needs to process) is first input into the streaming audio encoder, which extracts acoustic features. Subsequently, the adapter layer receives these acoustic features and maps them to semantic features. These semantic features are then passed to the large language model, providing a semantic basis for the large language model to generate response content. Simultaneously, the acoustic and semantic features are also input into a recurrent neural network transformer, which analyzes and determines the semantic state label corresponding to the target audio segment of the streaming audio, thereby precisely controlling the timing and manner of the large language model's semantic response. The semantic state label includes three types: semantic integrity label, semantic incomplete label, and semantic interruption label, corresponding to different logics of the large language model: "generating a response," "waiting to continue acquiring speech to complete the semantics," and "pausing the current request and restarting the response after semantic integrity." This application applies the RNN-T structure to the semantic pause decision of the streaming voice dialogue system, achieving end-to-end mapping between the speech feature stream and the semantic integrity label through joint modeling of the streaming audio encoder and the adapter layer. This architecture breaks through the traditional approach of separating the semantic stop detection module in large-scale voice dialogue models, significantly improving the real-time performance and accuracy of semantic boundary judgment.

[0103] This embodiment also provides a semantic response device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] Figure 4This is a structural block diagram of the semantic response device according to an embodiment of this application.

[0105] This embodiment provides a semantic response device, such as Figure 4 As shown, the semantic response device includes: The first processing module 11 is used to acquire the acoustic and semantic features of the user's streaming target audio segment.

[0106] The second processing module 12 is used to obtain fused features based on acoustic features and semantic features.

[0107] The third processing module 13 is used to determine the semantic state label corresponding to the streaming target audio segment based on the fusion features. The semantic state label includes one of the following: semantic incomplete label, semantic complete label, or semantic interruption label.

[0108] The fourth processing module 14 is used to determine the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment. Different semantic state labels correspond to different response control strategies.

[0109] In one alternative implementation, the fourth processing module 14 includes a first processing unit.

[0110] The first processing unit is used to, when the semantic state label is a semantically incomplete label, if the next audio segment is obtained within a first preset time period, then the next audio segment is used as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly determined. Alternatively, if the next audio segment is not obtained within the first preset time period, corresponding rhetorical questions are generated based on semantic features. The rhetorical questions are used to guide the user to complete the semantic information.

[0111] In one alternative implementation, the fourth processing module 14 includes a second processing unit.

[0112] The second processing unit is used to generate corresponding response content based on semantic features if the next audio segment is not obtained within a second preset time period when the semantic state label is a semantically complete label. Alternatively, if the next audio segment is obtained within the second preset time period, the next audio segment is used as a new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly determined. The response content is the generated content responding to the user's request, and the first preset time period is longer than the second preset time period.

[0113] In one alternative implementation, the fourth processing module 14 includes a third processing unit.

[0114] The third processing unit is used to stop generating rhetorical questions or responses when the semantic state label is a semantic interruption label.

[0115] In one optional implementation, the first processing module 11 is specifically used to input the streaming target audio segment into the streaming audio encoder and extract the acoustic features of the streaming target audio segment through the streaming audio encoder.

[0116] The acoustic features of the target audio segment in the streaming are input into the adapter layer, and the semantic features corresponding to the acoustic features are determined by the adapter layer.

[0117] In one optional implementation, the second processing module 12 is specifically used to perform dimensional alignment on the acoustic features and semantic features to obtain acoustic feature vectors and semantic feature vectors.

[0118] The acoustic feature vector and the semantic feature vector are normalized respectively to obtain the normalized acoustic feature vector and the normalized semantic feature vector.

[0119] The fused features are obtained by summing the elements of the same dimension in the normalized acoustic feature vector and the normalized semantic feature vector.

[0120] In one optional implementation, the third processing module 13 is specifically used to input the fused features into the encoder of the recurrent neural network transformer to obtain the temporal coding features corresponding to the fused features.

[0121] Obtain context-related features generated by the predictive network of the recurrent neural network transformer based on the semantic state labels of historical outputs.

[0122] By fusing temporal coding features and contextual features through a joint network of recurrent neural network transformers, the probability distribution of each semantic state label corresponding to the streaming target audio segment is obtained.

[0123] Based on the probability distribution, the semantic state label with the highest probability is selected as the semantic state label corresponding to the streaming target audio segment.

[0124] In this embodiment, the semantic response device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0125] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0126] This application also provides a vehicle having the above-described features. Figure 4 The semantic response device shown.

[0127] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a vehicle provided in an optional embodiment of this application, such as... Figure 5As shown, the vehicle includes one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the vehicle, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple vehicles can be connected, with each device providing some of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0128] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0129] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0130] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on vehicle usage as displayed on a mini-program landing page. Furthermore, the memory 20 may include high-speed random access memory and non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] Memory 20 may include volatile memory, such as random access memory. Memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive. Memory 20 may also include combinations of the above types of memory.

[0132] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the vehicle, such as a touchscreen, keypad, mouse, trackpad, touchpad, indicator, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0133] The vehicle also includes a communication interface 30 for communicating with other devices or communication networks.

[0134] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc. Further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0135] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0136] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A semantic response method, characterized in that, The method includes: Acquire the acoustic and semantic features of the user's streaming target audio segment; The fused features are obtained based on the acoustic features and the semantic features; The semantic state label corresponding to the streaming target audio segment is determined based on the fusion features; the semantic state label includes one of the following: semantic incomplete label, semantic complete label, or semantic interruption label; The corresponding response control strategy is determined based on the semantic state label corresponding to the streaming target audio segment. Different semantic state labels correspond to different response control strategies.

2. The method according to claim 1, characterized in that, The step of determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment includes: When the semantic state label is the semantically incomplete label, if the next audio segment is obtained within the first preset time period, the next audio segment is used as the new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly judged; or, if the next audio segment is not obtained within the first preset time period, the corresponding rhetorical question content is generated according to the semantic features; the rhetorical question content is used to guide the user to complete the semantic information.

3. The method according to claim 2, characterized in that, The step of determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment further includes: When the semantic state label is the semantic complete label, if the next audio segment is not obtained within the second preset time period, the corresponding response content is generated according to the semantic features; or, if the next audio segment is obtained within the second preset time period, the next audio segment is used as a new streaming target audio segment, and the semantic state label corresponding to the new streaming target audio segment is repeatedly judged; the response content is the generated content that responds to the user's needs, and the first preset time period is longer than the second preset time period.

4. The method according to claim 3, characterized in that, The step of determining the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment further includes: When the semantic state label is the semantic interruption label, the generation of the rhetorical question content or the response content is stopped.

5. The method according to claim 1, characterized in that, The acquisition of acoustic and semantic features of the user's streaming target audio segment includes: The target audio segment is input into the streaming audio encoder, and the acoustic features of the target audio segment are extracted by the streaming audio encoder. The acoustic features of the streaming target audio segment are input into the adapter layer, and the semantic features corresponding to the acoustic features are determined by the adapter layer.

6. The method according to claim 1, characterized in that, The process of obtaining fused features based on the acoustic features and the semantic features includes: The acoustic features and semantic features are dimensionally aligned to obtain acoustic feature vectors and semantic feature vectors; The acoustic feature vector and the semantic feature vector are normalized respectively to obtain the normalized acoustic feature vector and the normalized semantic feature vector; The fused feature is obtained by calculating the sum of the elements of the same dimension in the normalized acoustic feature vector and the normalized semantic feature vector.

7. The method according to claim 1, characterized in that, The step of determining the semantic state label corresponding to the streaming target audio segment based on the fusion features includes: The fused features are input into the encoder of the recurrent neural network transformer to obtain the temporal coding features corresponding to the fused features; Obtain the contextual features generated by the prediction network of the recurrent neural network transformer based on the semantic state labels of the historical output; By fusing the temporal coding features and the contextual association features through the joint network of the recurrent neural network transformer, the probability distribution of each semantic state label corresponding to the streaming target audio segment is obtained; Based on the probability distribution, the semantic state label with the highest probability is selected as the semantic state label corresponding to the streaming target audio segment.

8. A semantic response device, characterized in that, The device includes: The first processing module is used to acquire the acoustic and semantic features of the user's streaming target audio segment; The second processing module is used to obtain fused features based on the acoustic features and the semantic features; The third processing module is used to determine the semantic state label corresponding to the streaming target audio segment based on the fusion features; the semantic state label includes one of a semantically incomplete label, a semantically complete label, or a semantically interrupted label. The fourth processing module is used to determine the corresponding response control strategy based on the semantic state label corresponding to the streaming target audio segment. Different semantic state labels correspond to different response control strategies.

9. A vehicle, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images