Human-computer interaction method, device, equipment, storage medium and program product

Abstract

The application discloses a human-computer interaction method and device, equipment, a storage medium and a program product, and relates to the technical field of intelligent interaction. When a human-computer interaction instruction is received, initial feedback content for responding to the human-computer interaction instruction is generated, the initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content; voice broadcast is performed according to the summary feedback content, and the summary feedback content and / or the detailed feedback content are synchronously displayed. The voice broadcast is performed only according to the summary feedback content, so that a user can quickly obtain core information; the user can see specific broadcast content through the displayed summary feedback content, enhance understanding of the voice broadcast content, and / or understand detailed feedback content corresponding to the summary feedback content through the displayed detailed feedback content; and therefore, separation of voice broadcast and display is realized, information integrity is ensured, and the interactive requirement of the user for efficiently obtaining information in a driving scenario is met.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to a human-computer interaction method, device, equipment, storage medium, and program product. Background Technology

[0002] Currently, in-vehicle human-machine interaction systems generally use a combination of voice broadcasting and on-screen text display to provide information to users. For example, when a user inquires about vehicle energy consumption, navigation routes, or troubleshooting, the vehicle's infotainment system will read the entire response aloud, sentence by sentence, while simultaneously displaying the same text or simple icons on the screen.

[0003] However, this feedback method is difficult to meet the interactive needs of users in driving scenarios to obtain information efficiently. Summary of the Invention

[0004] The main objective of this application is to provide a human-computer interaction method, device, equipment, storage medium, and program product, which aims to solve the technical problem that current information feedback methods are insufficient to meet the user's need for efficient information acquisition in driving scenarios.

[0005] To achieve the above objectives, this application proposes a human-computer interaction method, which includes: When a human-computer interaction command is received, initial feedback content is generated to respond to the human-computer interaction command. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The summary feedback is read aloud, and the summary feedback and / or the detailed feedback are displayed simultaneously.

[0006] In one embodiment, the initial feedback content includes multiple feedback segments, and the step of generating the initial feedback content for responding to the human-computer interaction command includes: Based on the human-computer interaction instructions, a first interaction requirement is determined through a multimodal large model. The first interaction requirement is then broken down into multiple sub-requirements according to a preset logical hierarchy. Multiple feedback segments corresponding to the multiple sub-requirements are generated sequentially according to the preset logical hierarchy.

[0007] In one embodiment, the summary feedback content includes summary content corresponding to each of the feedback segments, and the detailed feedback content includes detailed content corresponding to each of the feedback segments. The step of performing voice broadcast based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content includes: The summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content and / or detailed content of each segment are displayed in sequence according to the preset logical hierarchy.

[0008] In one embodiment, the preset logical hierarchy includes at least one hierarchy, and the step of performing voice broadcast based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content further includes: The system broadcasts a summary of the target segment corresponding to the target logical level, and simultaneously displays the summary of the target segment and / or the detailed content of the target segment corresponding to the target logical level, wherein the target logical level is any level among the preset logical levels. After the steps of simultaneously displaying the summary content of the target segment corresponding to the target logical level in the voice broadcast, and simultaneously displaying the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level, the method further includes: If a follow-up question instruction is detected for the voice broadcast content, or a further operation is detected for the displayed content, a second interaction requirement is determined. Based on the second interaction requirement, the content to be fed back is determined, and the voice broadcast and display of the content to be fed back continue. The follow-up question instruction includes at least one of a repeat playback instruction and a further question, and the further operation includes at least one of clicking the replay button and clicking the displayed detailed feedback content. If no follow-up question is detected regarding the voice broadcast content, or no further operation is detected regarding the displayed content, then the remaining feedback segment will no longer be broadcast or displayed via voice.

[0009] In one embodiment, the step of determining the content to be fed back based on the second interaction request includes: When a target feedback segment that matches the second interaction requirement exists, the content to be fed back is determined to be the content corresponding to the target feedback segment; When all feedback fragments do not match the second interaction requirement, follow-up feedback content is generated based on the second interaction requirement and the corresponding fragment summary content, and the follow-up feedback content is used as the feedback content to be provided.

[0010] In one embodiment, the step of determining the second interaction requirement includes at least one of the following: Based on the keywords in the follow-up question, determine the second interaction requirement; Obtain the user's historical interaction preference data, and determine the second interaction requirement based on the historical interaction preference data; Determine the complexity of the content involved in the follow-up question and / or the further operation, and determine the second interaction requirement based on the complexity. The step of determining the content to be fed back based on the second interaction request further includes: Based on the type of the second interaction request, determine the content to be fed back. The types of the second interaction request include summary follow-up questions and detailed follow-up questions.

[0011] In one embodiment, both the initial feedback content and the follow-up feedback content include a voice priority tag and a rendering priority tag. The value of the voice priority tag is used to characterize the voice playback priority of the corresponding content, and the value of the rendering priority tag is used to characterize the display priority of the corresponding content. Before the step of performing voice playback based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content, at least one of the following is further included: Based on the value of the voice priority label, determine the content that needs to be broadcast by voice; The content to be displayed is determined based on the value of the rendering priority tag.

[0012] In one embodiment, each feedback segment corresponds to a timestamp, the follow-up feedback content corresponds to a sub-timestamp, and the step of sequentially broadcasting the summary content of each segment according to the preset logical hierarchy, and simultaneously displaying the summary content and / or detailed content of each segment according to the preset logical hierarchy, includes: Based on each timestamp, the summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content of each segment and / or the detailed content of each segment is displayed in sequence according to the preset logical hierarchy. The step of continuing to broadcast and display the content to be fed back via voice includes: Based on the sub-timestamp, the follow-up question feedback content will continue to be read aloud and displayed via voice.

[0013] In one embodiment, the step of simultaneously displaying the summary feedback content and / or the detailed feedback content includes: The summary content and / or detailed content of each feedback segment are displayed simultaneously in different functional areas of the in-vehicle screen. The functional areas include a core display area, a thumbnail preview area, and a voice-synchronized prompt area. The detailed content of the segments includes charts, code blocks, tables, and thumbnails. The step of simultaneously displaying the summary content and / or detailed content of each feedback segment in different functional areas of the in-vehicle screen includes: The core display area displays charts, code blocks, and / or tables of the current feedback segment; the thumbnail preview area displays a thumbnail of the next feedback segment; and the voice synchronization prompt area displays the key content of the current feedback segment. The current feedback segment is the feedback segment corresponding to the summary content of the current voice broadcast segment, and the next feedback segment is the first feedback segment arranged according to the preset logical hierarchy after the current feedback segment.

[0014] In one embodiment, the step of broadcasting the summary feedback content via voice and simultaneously displaying the summary feedback content and / or the detailed feedback content includes at least one of the following: Based on the summary feedback content, determine the content to be broadcast, and determine the content to be displayed in the summary feedback content and / or the detailed feedback content other than the content corresponding to the content to be broadcast. Then, broadcast the content to be broadcast by voice and display the content to be displayed simultaneously. The system reads the content to be read aloud and simultaneously highlights related content within the content to be displayed.

[0015] In one embodiment, the summary feedback content is in the form of a written expression, and the step of broadcasting the summary feedback content via voice includes at least one of the following: The written summary feedback content is transformed using a preset language processing model to generate a spoken expression text, which is then read aloud. The complexity and nature of the summary feedback content are determined. Based on the complexity and nature of the content, the speech playback speed and tone are determined. The summary feedback content is then played at the specified speech playback speed and tone. The nature of the content includes suggestive content and explanatory content.

[0016] In one embodiment, the step of generating initial feedback content for responding to the human-computer interaction command includes: Generate structured marked text, which is used to describe the initial feedback content; The structured marked text includes structured visual content and structured text content; The structured visualization content is displayed in at least one of the following formats: code blocks, charts, and tables, wherein different types of code fields in the code blocks have different visual distinguishing identifiers; The structured text content includes hierarchical headings and annotations to help explain the meaning of charts or codes. The hierarchical headings are used to distinguish different logical levels.

[0017] In one embodiment, the step of generating initial feedback content for responding to the human-computer interaction command includes: Obtain the display preference data of the user corresponding to the human-computer interaction command, determine the target display format based on the display preference data, and generate visual content to respond to the human-computer interaction command based on the target display format.

[0018] In one embodiment, the detailed feedback content includes visual content, and the step of displaying the visual content includes at least one of the following: The visualized content is displayed according to the characteristics of the in-vehicle screen; Highlight the visualized content; The visualization content includes at least one of code blocks, charts, and tables; the charts include at least one of line charts, flowcharts, and thumbnails; and the step of highlighting the visualization content includes: Adjust the row height or column width of the table according to the driving scenario, and then display it.

[0019] Furthermore, to achieve the above objectives, this application also proposes a human-computer interaction device, which includes: The content generation module is used to generate initial feedback content for responding to the human-computer interaction instruction when a human-computer interaction instruction is received. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The interactive feedback module is used to broadcast the summary feedback content via voice and simultaneously display the summary feedback content and / or the detailed feedback content.

[0020] In addition, to achieve the above objectives, this application also proposes a human-computer interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the human-computer interaction method as described above.

[0021] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the human-computer interaction method described above.

[0022] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the human-computer interaction method described above.

[0023] One or more technical solutions proposed in this application have at least the following technical effects: When this application receives a human-computer interaction command, it generates initial feedback content to respond to the command. This initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The application then performs voice broadcast based on the summary feedback content, simultaneously displaying the summary feedback content and / or the detailed feedback content. It is understood that the summary feedback content removes redundant details, retaining only the most essential information. The voice broadcast is based solely on this summary feedback content, enabling users to quickly obtain core information. Simultaneously, users can see the specific broadcast content through the displayed summary feedback content, thereby enhancing their understanding of the voice broadcast content, and / or learn about the detailed feedback content corresponding to the summary feedback content through the displayed detailed feedback content. This achieves separation of voice broadcast and display, ensuring information integrity and meeting the user's need for efficient information acquisition in driving scenarios. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the first embodiment of the applicant's human-computer interaction method; Figure 2 A schematic diagram of the first scenario provided for the applicant's human-computer interaction method; Figure 3 A second scenario diagram provided for the applicant's human-computer interaction method; Figure 4 A schematic diagram of the third scenario provided for the applicant's human-computer interaction method; Figure 5 This is a schematic diagram of the module structure of the human-computer interaction device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the human-computer interaction method in the embodiments of this application.

[0027] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0029] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0030] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or human-computer interaction device capable of performing the above functions. The following description uses a human-computer interaction device as an example to illustrate this embodiment and the subsequent embodiments.

[0031] Based on this, the embodiments of this application provide a human-computer interaction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the applicant's human-computer interaction method.

[0032] In this embodiment, the human-computer interaction method includes steps S10 to S20: Step S10: When a human-computer interaction instruction is received, an initial feedback content for responding to the human-computer interaction instruction is generated. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. Step S20: Based on the summary feedback content, a voice broadcast is performed, simultaneously displaying the summary feedback content and / or the detailed feedback content.

[0033] It should be noted that the human-computer interaction method in this embodiment is applied to a human-computer interaction device, which may belong to a vehicle system or a smart home system, etc. This embodiment takes the example of the human-computer device belonging to a vehicle system for specific explanation.

[0034] Users can trigger human-computer interaction commands through voice input, touchscreen operation, physical button pressing, or gesture recognition. These commands can be a question, a query, or a control command, such as, "Analyze this week's vehicle energy consumption."

[0035] Upon receiving a human-computer interaction command, the vehicle-side multimodal large model can generate initial feedback content to respond to the command. This initial feedback content includes: detailed feedback content containing all original details, and summary feedback content automatically extracted from the detailed feedback content, retaining only the core conclusions.

[0036] Understandably, detailed feedback contains all the details and is typically information-rich and varied in format. In contrast, summary feedback extracts, summarizes, and simplifies information from the detailed feedback, providing core conclusions or key summaries; summary feedback is less information-rich and concisely expressed.

[0037] For example, upon receiving the human-machine interaction command "Check if my vehicle has any recent malfunctions," the vehicle-side big data model can first call the vehicle's self-diagnostic system interface to obtain fault code data. Then, instead of directly outputting all fault codes and raw data, the vehicle-side multimodal big data model generates detailed feedback content containing all fault code numbers, occurrence times, associated systems, detailed parameters, and repair suggestions. Finally, it performs an automatic summarization algorithm on this detailed feedback content to extract the fault severity level, total number of faults, and a summary feedback content describing the most urgent fault. For example, the detailed feedback content might include details such as five fault codes and their causes, while the summary feedback content might be "Three general faults found; braking system inspection recommended."

[0038] Furthermore, the summary feedback can be read aloud via the in-vehicle speaker system. Simultaneously, the summary feedback and detailed feedback can be displayed on the in-vehicle screen, or only the summary feedback or only the detailed feedback can be displayed. Whether it is necessary to display both summary feedback and detailed feedback can be determined based on the vehicle's driving scenario.

[0039] For example, when the vehicle is in motion and the speed is higher than 30 kilometers per hour, in order to minimize the driver's visual distraction, only the summary feedback content can be displayed while the summary content is read out by voice; when the vehicle is stationary (such as waiting at a red light or in a parked state), the summary feedback content and detailed feedback content can be displayed simultaneously while the summary content is read out by voice; thus achieving an adaptive balance between driving safety and the richness of information access.

[0040] In this embodiment, the summary feedback is preferably read aloud via a vehicle speaker system, while detailed feedback is simultaneously displayed on the vehicle screen. This allows users to quickly obtain the core conclusions through the auditory channel and independently view detailed information through the visual channel. The two processes do not interfere with each other and complement each other, improving the efficiency of information acquisition in driving scenarios. At the same time, users do not need to look at the screen for a long time to obtain the core conclusions, reducing the visual attention required.

[0041] Specifically, the method for broadcasting the summary feedback content via voice can be to directly broadcast all the text of the summary feedback content via voice, or to further refine the summary feedback content before broadcasting it via voice.

[0042] In one feasible implementation, the method for generating initial feedback content in response to the human-computer interaction command may be: generating structured marked text, which is used to describe the initial feedback content.

[0043] It should be noted that structured markup text can be in Markdown or HTML format; Markdown text refers to plain text in a lightweight markup language format, which uses specific markup symbols to annotate the semantic structure and presentation style of the content.

[0044] Optionally, when generating initial feedback content to respond to human-computer interaction commands, the vehicle-side multimodal large model can directly output a piece of Markdown text. This Markdown text can use specific markers to clearly indicate which parts are ordinary text descriptions, which parts need to be rendered as tables, which parts need to be presented as code blocks, and which parts need to be generated as charts, etc.

[0045] The Markdown format supports flexible expansion of new elements and has strong scalability. By describing the initial feedback content as Markdown text, the richness of the content display can be improved.

[0046] Specifically, the vehicle infotainment system includes an in-vehicle human-machine interface, which can parse markdown text through a multi-element rendering module, identify various markup syntaxes in the markdown text, and convert the structured content in the markdown text into corresponding visual elements and display them on the screen according to preset rendering rules.

[0047] In this embodiment, the voice only broadcasts the core conclusions extracted from the detailed feedback content, avoiding lengthy expressions that verbatim read out table data or code parameters. This allows the driver to quickly obtain key information within 30 seconds, reducing auditory burden and attention distraction. The screen simultaneously displays the complete detailed feedback content, which the user can review independently. This ensures both the integrity and depth of the information, while also meeting the interactive needs for efficient information acquisition in driving scenarios, thus improving information transmission efficiency and driving safety. Furthermore, by describing the initial feedback content as structured marked text, the vehicle system can use a unified text format to carry multimodal content, eliminating the need to maintain different data channels for different types of content.

[0048] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, the initial feedback content includes multiple feedback segments, and the specific implementation of generating the initial feedback content for responding to the human-computer interaction command can be: Based on the human-computer interaction instructions, a first interaction requirement is determined through a multimodal large model. The first interaction requirement is then broken down into multiple sub-requirements according to a preset logical hierarchy. Multiple feedback segments corresponding to the multiple sub-requirements are generated sequentially according to the preset logical hierarchy.

[0049] After receiving a human-machine interaction command, the human-machine interaction device can input the command into a multimodal big data model. The multimodal big data model can determine the user's first interaction requirement by analyzing the semantic information of the human-machine interaction command. Alternatively, the multimodal big data model can also comprehensively understand the semantic information of the human-machine interaction command and the vehicle status data (such as driving data, fault information, location information, etc.) obtained by the human-machine interaction device to determine the user's first interaction requirement.

[0050] Optionally, the multimodal large model may be a multimodal large language model based on the Transformer architecture, or a multimodal model including a text recognition module, a visual feature extraction module and / or a language recognition model.

[0051] Optionally, the multimodal large model can be deployed on the local processor of the vehicle's infotainment system or in the cloud. The vehicle's infotainment system sends human-machine interaction commands to the cloud, and the cloud returns the generated initial feedback content to the vehicle's infotainment system after completing inference through the multimodal large model.

[0052] Furthermore, the first interaction requirement can be broken down into multiple sub-requirements using a multimodal large model according to a preset logical hierarchy, with each sub-requirement corresponding to an independent logical hierarchy unit. Finally, a corresponding feedback segment is generated for each sub-requirement in sequence according to the order of the preset logical hierarchy.

[0053] Among them, the preset logical hierarchy refers to a predefined logical structure for organizing information, used to divide complex requirements into multiple levels according to cognitive order or importance. The preset logical hierarchy can be a logical hierarchy such as overview layer-detail layer-suggestion layer, conclusion layer-data layer-explanation layer, or problem layer-analysis layer-solution layer.

[0054] Optionally, prompt words and output examples can be input into the multimodal large model. The multimodal large model can decompose the first interaction requirement into multiple sub-requirements according to the prompt words and output examples and according to the preset logical hierarchy, and generate multiple feedback fragments corresponding to the multiple sub-requirements in sequence according to the preset logical hierarchy.

[0055] Optionally, a mind chain prompting method can be used to guide the multimodal large model through step-by-step reasoning and requirement decomposition; for example, the prompt words are: "

Task

[0056] [User Requirements]: Human-computer interaction commands.

[0057] [Logical hierarchy]: Overview - Details - Recommendations.

[0058] [Thinking Steps]: Step 1: Understand the human-computer interaction instructions and determine the user's overall goals and core needs.

[0059] Step 2: Analyze which aspects of information or operations are involved in user needs.

[0060] Step 3: Break down user needs into several sub-needs according to logical hierarchy.

[0061] Step 4: Determine the corresponding logical hierarchy label for each sub-requirement. Example output: Detailed feedback: [{"level": "Overview": "An overview of vehicle energy consumption this week"}, {"level": "Details": "Daily energy consumption data, energy consumption comparison under different road conditions"} {"level": "recommendation": "Energy-saving driving recommendations based on energy consumption data"}; Summary feedback content: {"Summary": "Total energy consumption this week was 15.2 kWh / 100km, up 5% from last week, and total mileage was 320km. It is recommended to reduce sudden acceleration and deceleration and check tire pressure."}

[0062] Alternatively, after the multimodal large model generates detailed feedback content, you can input summary prompts into the multimodal large model: generate a concise summary description for the detailed feedback content. The summary should include at least the following: the word count should be within a certain range (e.g., 30 words), cover the core conclusions and key data, the language should be concise and clear, suitable for voice broadcast, and retain the key numbers and entity names in the detailed feedback content.

[0063] Alternatively, post-processing algorithms can be used to automatically extract summary feedback from the detailed feedback content. For example, after generating detailed feedback content from a multimodal large model, key entities (such as numbers, dates, locations, object names, etc.) can be extracted from the detailed feedback content through named entity recognition; the subject-verb-object structure in the detailed feedback content can be identified through dependency parsing to extract the core declarative sentence; and the extracted key entities and the core declarative sentence can be combined to generate concise summary feedback content.

[0064] By breaking down user human-computer interaction commands into multiple sub-requirements according to a preset logical hierarchy, and responding to each one according to the preset logical hierarchy, subsequent voice broadcasts and screen displays can proceed segment by segment in a logical order, avoiding the cognitive overload caused by all information being presented at once.

[0065] Specifically, multiple sub-markdown segments can be generated in a streaming manner through a multimodal large model according to a preset logical hierarchy, with each sub-markdown segment corresponding to a feedback segment of a logical hierarchy.

[0066] For example, when a user triggers the human-computer interaction command "Help me check my car's energy consumption this week", the multimodal big model first determines the first interaction requirement as "analyze this week's energy consumption", and breaks down the first interaction requirement into three sub-requirements according to the logical hierarchy of "overview-details-suggestions"; and then generates three sub-segments in a streaming manner for the three sub-requirements.

[0067] It should be noted that the summary feedback content includes the summary content corresponding to each of the feedback segments, and the detailed feedback content includes the detailed content corresponding to each of the feedback segments. That is, each feedback segment has a corresponding summary content, and the summary content corresponding to all feedback segments together constitutes the complete summary feedback content. Each feedback segment also has a corresponding detailed content, and the detailed content corresponding to all feedback segments together constitutes the complete detailed feedback content.

[0068] For example, in an energy consumption analysis scenario, the summary content of the first feedback segment, ordered according to a preset logical hierarchy, is "The average energy consumption this week has decreased by eight percent," and the summary content of the second feedback segment is "Energy consumption was higher on Monday and Friday due to air conditioning." The corresponding detailed content of the first feedback segment is an energy consumption trend line chart containing specific values, and the detailed content of the second feedback segment is a daily energy consumption table.

[0069] In one feasible implementation, the method of simultaneously displaying the summary feedback content and / or the detailed feedback content through voice broadcast based on the summary feedback content can be: The summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content and / or detailed content of each segment are displayed in sequence according to the preset logical hierarchy.

[0070] After obtaining initial feedback content consisting of multiple feedback segments, each segment can be processed sequentially according to the order specified by the preset logical hierarchy. For the currently being processed feedback segment, the summary content is first extracted and broadcast directly through the voice channel; simultaneously, the corresponding detailed content is displayed on the screen. After processing the current feedback segment, the next logical hierarchy's feedback segments are automatically broadcast and displayed via voice.

[0071] This allows users to receive information step-by-step according to a pre-defined logical hierarchy (e.g., from general to specific, from simple to complex), and at each stage, they can simultaneously access the core and detailed content through both auditory and visual channels. This significantly lowers the barrier to understanding for users and avoids the confusion caused by a sudden influx of information.

[0072] In one feasible implementation, the method of simultaneously displaying the summary feedback content and / or the detailed feedback content may be: The in-vehicle screen simultaneously displays summary content and / or detailed content of each feedback segment in different functional areas; the functional areas include a core display area, a thumbnail preview area, and a voice-synchronized prompt area.

[0073] It should be noted that the core display area refers to the main display area occupying the largest area of ​​the in-vehicle human-machine interaction screen, which is the main focus of the user's gaze; the core display area can occupy 70% of the total screen area. The thumbnail preview area refers to the auxiliary display area on the in-vehicle screen. This thumbnail preview area can be located on the side of the screen (e.g., the right side or bottom), and can occupy 20% of the total screen area. The voice-synchronized prompt area refers to the text prompt area on the in-vehicle screen used to dynamically display the current voice broadcast content. This voice-synchronized prompt area can occupy 10% of the total screen area and can be presented in the form of a floating card, bottom bar, or top bar.

[0074] This embodiment can simultaneously display summary content and / or detailed content of each feedback segment in different functional areas of the in-vehicle screen; thereby reducing the user's eye movement and screen scanning frequency during voice broadcasting. Users do not need to frequently touch or switch views, thus increasing the proportion of time their visual attention is focused on driving conditions.

[0075] In one feasible implementation, the detailed content of the segment includes at least one of charts, code blocks, tables, and thumbnails. The implementation of simultaneously displaying the summary content and / or detailed content of each feedback segment in different functional areas of the in-vehicle screen can be as follows: The core display area shows the charts, code blocks, and / or tables of the current feedback segment, the thumbnail preview area shows the thumbnail of the next feedback segment, and the voice-synchronized prompt area shows the key content of the current feedback segment.

[0076] It should be noted that the current feedback segment is the feedback segment corresponding to the summary content of the current voice broadcast segment, and the next feedback segment is the first feedback segment arranged according to the preset logical hierarchy after the current feedback segment.

[0077] This embodiment displays detailed content such as charts, code blocks, or tables from the current feedback segment in the core display area, requiring careful interpretation by the user. This content has high information density and rich visual detail, necessitating a significant portion of the display area. A thumbnail preview area displays the next feedback segment, serving as a visual preview of the core information of the upcoming segment. For example, if the next feedback segment is navigation route details, a route overview map can be displayed in the thumbnail preview area; if the next feedback segment is an energy consumption detail table, a compressed preview of the table can be displayed. Key content (e.g., keywords or core phrases) from the current feedback segment can be displayed in text form in the voice synchronization prompt area, helping users quickly identify the core points conveyed by the voice in noisy environments. Users can acquire three layers of information in a short time through habitual eye scanning: details through the core display area, understanding the follow-up through the thumbnail preview area, and confirmation of keywords through the voice synchronization prompt area; eliminating the need to search or switch views on the screen, thus significantly reducing visual cognitive load and the time spent looking away from the road.

[0078] In one feasible implementation, each feedback segment corresponds to a timestamp, and the method of sequentially broadcasting the summary content of each segment according to the preset logical hierarchy, and simultaneously displaying the summary content and / or detailed content of each segment according to the preset logical hierarchy, can be as follows: Based on the timestamps, the summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content and / or detailed content of each segment are displayed in sequence according to the preset logical hierarchy.

[0079] In this embodiment, the human-computer interaction device can set a timestamp for each feedback segment and arrange the feedback segments in ascending order of timestamps. When the system clock or timer reaches the time specified by the timestamp of a certain feedback segment, the voice broadcast action corresponding to that feedback segment is triggered, and the screen display action is triggered at the same time.

[0080] This ensures that the broadcasting and display of all feedback segments strictly follow the timeline sequence preset by the timestamps, executing each segment sequentially until all feedback segments have been processed. This guarantees that the summary content and detailed content of each logical level's segments remain time-aligned.

[0081] In this embodiment, the user's first interaction requirement is broken down into multiple sub-requirements according to a preset logical hierarchy using a multimodal large model. Feedback segments corresponding to each sub-requirement are generated sequentially, and each feedback segment is broadcast and displayed by voice according to the preset logical hierarchy. This significantly reduces the user's cognitive load. Furthermore, by adopting a streaming generation method, voice broadcasting and screen display can begin immediately after the first feedback segment is generated, without waiting for all content to be generated, thus reducing the user's anxiety about waiting and making complex information interaction more efficient.

[0082] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, the preset logical hierarchy includes at least one level, and the step of performing voice broadcast based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content further includes: The system broadcasts a summary of the target segment corresponding to the target logical level, and simultaneously displays the summary of the target segment and / or the detailed content of the target segment corresponding to the target logical level, wherein the target logical level is any level among the preset logical levels.

[0083] The system can determine whether the user's emotions are stable or urgent based on the human-computer interaction commands. Based on the user's emotions, it can selectively broadcast the summary content of the target segment corresponding to the target logical level, and simultaneously selectively display the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level based on the user's emotions.

[0084] For example, when a user is in a hurry and the preset logical hierarchy is problem layer-analysis layer-solution layer, the summary content of the corresponding segment of the solution layer can be broadcast directly, and the summary content of the segment of the solution layer and / or the detailed content of the segment of the solution layer can be displayed simultaneously.

[0085] Alternatively, the current time period can be determined based on human-computer interaction instructions. Based on the current time period, the summary content of the target segment corresponding to the target logical level can be selectively broadcast. Simultaneously, based on the user's emotions, the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level can be selectively displayed.

[0086] For example, when the current time period is outside of working hours, and the preset logical hierarchy is problem layer-analysis layer-solution layer, the summary content of the analysis layer can be broadcast first, and the summary content of the analysis layer and / or the detailed content of the solution layer can be displayed simultaneously.

[0087] In one feasible implementation, after the steps of simultaneously displaying the summary content of the target segment corresponding to the target logical level and / or the detailed content of the target segment corresponding to the target logical level, the method further includes: If a follow-up question or further action is detected regarding the voice-over content, a second interaction requirement is determined. Based on the second interaction requirement, feedback content is determined, and the voice-over and display of the feedback content continue. The follow-up question includes at least one of a repeat playback instruction and a further question, and the further action includes at least one of clicking the replay button and clicking the displayed detailed feedback content. If no follow-up question or further action is detected regarding the voice-over content, the voice-over and display of the remaining feedback segments cease.

[0088] It should be noted that follow-up questions refer to requests made by users after listening to the current voice broadcast, through voice or text input, to supplement, repeat, or explain the broadcast content in more detail. For example, follow-up questions may include repeat playback instructions to request the replay of the summary content (e.g., say it again), and further questions to request more detailed explanations of a specific detail (e.g., why was energy consumption so high on Monday).

[0089] Further actions refer to user interactions with the currently displayed content through non-voice methods (such as touching the screen, pressing physical buttons, etc.). Further actions may include clicking a preset replay button on the screen to trigger repeated playback, or directly clicking an element in the detailed feedback content displayed on the screen (such as clicking a row in a table, clicking a data point in a chart, clicking a thumbnail, etc.) to request the system to provide voice interpretation of that element.

[0090] After the summary content of the target segment corresponding to the target logical level is broadcast in voice, and the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level are displayed simultaneously, it is detected whether there is a follow-up question instruction for the content of the voice broadcast.

[0091] That is, during the process of sequentially broadcasting the summary content of each segment according to the preset logical hierarchy, and simultaneously displaying the summary content of each segment and / or the detailed content of each segment according to the preset logical hierarchy, after broadcasting the summary content of any segment and simultaneously displaying the summary content of any segment and / or the detailed content of any segment, it is detected whether the user has initiated a follow-up question or further operation on the broadcast or displayed content.

[0092] If a follow-up question or further action is detected regarding the voice-over content, a second interaction request can be parsed from the request or action. Based on this second interaction request, the required feedback content can be determined; alternatively, the required feedback content can be determined based on the second interaction request and the interaction context. The required feedback content is then read aloud and displayed on the screen, thus responding to the user's further requests.

[0093] If no follow-up instructions or further operations are detected within a preset time window (e.g., five or ten seconds after the broadcast ends), it is determined that the user is satisfied with the currently broadcast or displayed content or has no intention of learning more. In this case, any broadcast of the summary content of the remaining feedback segments that have not yet been broadcast will be stopped, and the content corresponding to the remaining feedback segments that have not yet been broadcast will no longer be actively displayed. The screen can remain on the current content or restore the default interface.

[0094] The above approach avoids auditory disturbance and driving distraction caused by the system forcibly broadcasting the remaining content after the user has obtained the required information, while providing the user with a flexible channel for in-depth understanding.

[0095] In one feasible implementation, the method for determining the content to be fed back based on the second interaction requirement may be: When a target feedback segment that matches the second interaction requirement exists, the content to be fed back is determined to be the content corresponding to the target feedback segment; when all feedback segments do not match the second interaction requirement, follow-up feedback content is generated based on the second interaction requirement and the corresponding segment summary content, and the follow-up feedback content is used as the content to be fed back.

[0096] After identifying the user's second interaction request, a matching search can be performed among multiple feedback fragments contained in the generated initial feedback content. Specifically, semantic similarity calculation or keyword matching can be used to determine whether there exists a target feedback fragment that can directly respond to the current follow-up question; for example, if the user asks "details on Monday's data," and there already exists a feedback fragment containing detailed data for Monday, then that feedback fragment is the matching target feedback fragment, and the response content to be provided is the corresponding content of that target feedback fragment.

[0097] If a user's follow-up question exceeds the coverage of all existing feedback snippets—for example, if a user asks, "If I switch to the economy mode, how much will energy consumption decrease?"—and the initial feedback does not include this analysis, then it is determined that no matching feedback snippet exists. In this case, the second interaction requirement and the corresponding summary content of the follow-up question can be input into the multimodal large model to generate follow-up question feedback content in real time, and this follow-up question feedback content will be used as the content to be fed back. This maximizes the reuse efficiency of existing generated content while ensuring good responsiveness to follow-up questions that exceed the preset range.

[0098] In one feasible implementation, the method for determining the second interaction requirement may be at least one of the following: Based on the keywords in the follow-up question, determine the second interaction requirement; Obtain the user's historical interaction preference data, and determine the second interaction requirement based on the historical interaction preference data; Determine the complexity of the content involved in the follow-up question and / or the further operation, and determine the second interaction requirement based on the complexity.

[0099] Specifically, after receiving a follow-up question from the user or detecting further action, the second interaction requirement can be determined through three optional judgment dimensions.

[0100] The first dimension can be to analyze the keywords in the follow-up question using natural language processing technology. If the follow-up question contains keywords with summarizing semantics such as "explain again", "summarize", or "what is the core", then the second interaction requirement is to summarize the current voice broadcast content. If the follow-up question contains detailed words such as specific data values, element names, or scene parameters, then the second interaction requirement is to provide detailed content of the current voice broadcast content.

[0101] The second dimension could be accessing the user's historical interaction preference data. Based on this data, a second interaction request can be determined. Historical interaction preference data refers to personalized behavioral data collected and stored by the system for the current user (or user account) during historical interactions. For example, if the system records show that the user requested detailed explanations in four out of the past five follow-up questions, then when the user issues a vague instruction (e.g., "Tell me more about this"), it is assumed that the user wants to know more details. Conversely, if the user frequently interrupts detailed explanations or never requests details, it is assumed that the user only wants to understand the summary content first.

[0102] The third dimension can be analyzing the complexity of the content involved in the user's follow-up questions. The complexity of the content can be the information density or interpretation difficulty of the target content that the user's follow-up questions or further actions are pointing to. High-complexity content (such as multiple lines of code in a code block, complex line graphs with multiple sequences, and flowcharts with a large number of branches) and low-complexity content (such as single-line text, simple tables, and summarized key data) can be distinguished based on the detailed broadcast adaptation threshold.

[0103] For example, for highly complex content such as code blocks and complex flowcharts, it can be determined that the user wants to know more details only when the specific element identifier is explicitly mentioned in the follow-up question (such as "the second line of code" or "the peak point"). Otherwise, it can be determined that the user wants to know the summary content first, so as to avoid lengthy output. For low-complexity content such as tables and simple text, it can be determined that the user wants to know more details when the user makes a vague request for details (such as "specific data").

[0104] The method for determining the content to be fed back based on the second interaction requirement can also be: Based on the type of the second interaction request, determine the content to be fed back. The types of the second interaction request include summary follow-up questions and detailed follow-up questions.

[0105] Specifically, after determining the type of the second interaction request, the content to be fed back can be determined accordingly. When the second interaction request is a summary follow-up question, the content to be fed back can be determined to be a summary of the segment the user is referring to in the follow-up question, simply repeating or displaying the core conclusions that have already been extracted, without involving any detailed data such as tables, code blocks, or charts. When the second interaction request is a detailed follow-up question, the content to be fed back can be determined to be a local detail related to the follow-up question within the detailed content of the segment the user is referring to in the follow-up question, such as the explanation of a row in a table, the explanation of a data point in a chart, or the meaning of a parameter in a line of code.

[0106] For example, in a scenario involving detailed follow-up questions, after listening to the summary report of the second feedback segment of the energy consumption analysis (the daily energy consumption details table), a user might ask via voice, "What exactly caused the 15 kWh consumption on Monday?" The second interaction request can be identified as a detailed follow-up question based on the keywords "Monday" and the specific value "15 kWh." According to this type of second interaction request, the feedback content can be determined to be a specific detail related to "Monday" within the detailed content of the second feedback segment. Simultaneously, a detailed explanation text can be generated by combining the vehicle's stored daily driving data (average speed, air conditioning set temperature, outdoor temperature, etc.). That is, the feedback content would include: "Monday's energy consumption of 15 kWh was the highest this week. There are two main reasons: first, the morning rush hour traffic was congested, with an average speed of only 20 km / h, leading to increased energy consumption at low speeds; second, the outdoor temperature was 35 degrees Celsius, and the air conditioning was set to 22 degrees Celsius, increasing air conditioning power consumption by approximately two kWh."

[0107] In a scenario involving a summary follow-up question, after listening to the summary report of the first feedback segment (energy consumption overview and trend chart), the user asks a voice follow-up question, "Reiterate this week's energy consumption conclusions." Based on the keywords "reiterate" and "conclusions," the type of the second interaction request can be determined as a summary follow-up question. According to this type of second interaction request, the feedback content to be provided is determined to be the summary content already stored in the first feedback segment, namely, "This week's average energy consumption was 12 kWh, a decrease of 8% compared to last week."

[0108] Using the above method can avoid inefficient interaction caused by content type mismatch, that is, avoid outputting a large amount of detail when users only ask for a repetitive summary, or only repetitive summary when users ask for details.

[0109] It should be noted that both the initial feedback content and the follow-up feedback content include a voice priority tag and a rendering priority tag. The value of the voice priority tag is used to represent the voice playback priority of the corresponding content, and the value of the rendering priority tag is used to represent the display priority of the corresponding content. The priority tag can be a metadata tag attached to each content unit in the initial feedback content or the follow-up feedback content.

[0110] In one feasible implementation, before the step of broadcasting the summary feedback content via voice and simultaneously displaying the summary feedback content and / or the detailed feedback content, the method further includes at least one of the following: Based on the value of the voice priority label, determine the content that needs to be broadcast by voice; The content to be displayed is determined based on the value of the rendering priority tag.

[0111] Specifically, before broadcasting and simultaneously displaying the summary feedback, the individual content units within the initial feedback (or follow-up feedback) can be pre-filtered. Scan the entire feedback content and check the voice priority tag on each content unit. Content units with a voice priority tag of "core summary" can be marked as content to be broadcast; content units with a tag of "supplementary explanation" can be excluded from the broadcast content.

[0112] At the same time, the rendering priority label can be checked. For content units with a value of "must be displayed", it can be determined whether to display them on the screen and how to display them, based on the real-time screen layout, whether the user clicks on the relevant area, or the vehicle's driving status (such as reducing the display of optional content when the vehicle speed is higher than the threshold).

[0113] Through the aforementioned priority-based filtering mechanism, the content allocation and trimming of the voice and visual channels are completed before the broadcast and display actions occur, thus ensuring the separation of interaction from the source: "voice only broadcasts the core summary, and the screen displays details as needed."

[0114] Specifically, the multimodal large model can output a custom comment line after generating each content unit. For example, when generating a summary statement containing "This week's average energy consumption was 12 kWh, a decrease of 8% compared to last week," the model automatically inserts a comment at that summary statement, such as "[Voice tag: Core summary], [Rendering tag: Required]." When generating daily data lines (such as "Tuesday, 13, Daily commute"), the multimodal large model inserts a comment below that data line: "[Voice tag: Supplementary explanation], [Rendering tag: Required]."

[0115] In one feasible implementation, the follow-up feedback content corresponds to a sub-timestamp, and the method for continuing to broadcast and display the feedback content via voice can be: Based on the sub-timestamp, the follow-up question feedback content will continue to be read aloud and displayed via voice.

[0116] After generating follow-up feedback content, a sub-timestamp can be assigned to the follow-up feedback content; this sub-timestamp is set to be executed immediately in terms of timing, and its priority is higher than the timestamp of the original feedback fragment that has not yet been processed.

[0117] Based on the triggering time of this sub-timestamp, the voice broadcast of the follow-up feedback content can be initiated, and the corresponding display can be shown on the screen simultaneously. Specifically, in addition to presenting the visual information of the follow-up feedback content itself, a highlighting linkage mechanism on the screen can also be triggered; for example, if the follow-up question is about the details of a certain row of data in a table, the screen can highlight the background color of that row of data while broadcasting the details, or flash the corresponding data point in a chart.

[0118] This enables instant insertion of follow-up questions and precise voice-visual linkage, allowing users to hear detailed explanations while their gaze is automatically guided to the corresponding position on the screen, thus quickly establishing a correspondence between auditory and visual information and reducing the cost of comprehension.

[0119] In this embodiment, by actively listening for user follow-up questions or further operations after the broadcast display, and differentiating the content to be fed back based on the type of the second interaction request, summary follow-up questions only repeat the summary content of the segment, while detailed follow-up questions extract local details from the detailed content of the segment and simultaneously trigger screen highlighting. At the same time, the subsequent segment broadcast is automatically stopped when no follow-up action is detected. This reduces information overload and auditory interference in driving scenarios, improves the accuracy and efficiency of follow-up response, enhances the fluency of information understanding, and improves the naturalness of interaction and user satisfaction.

[0120] Based on the first, second, and third embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, the step of performing voice broadcast based on the summary feedback content, and simultaneously displaying the summary feedback content and / or the detailed feedback content, includes at least one of the following: Based on the summary feedback content, determine the content to be broadcast, and determine the content to be displayed in the summary feedback content and / or the detailed feedback content other than the content corresponding to the content to be broadcast. Then, broadcast the content to be broadcast by voice and display the content to be displayed simultaneously. The system reads the content to be read aloud and simultaneously highlights related content within the content to be displayed.

[0121] Specifically, when preparing voice broadcasts and screen displays, you can first determine the content to be broadcast from the summary feedback. Combine the summary feedback and detailed feedback, remove the parts that correspond to the content to be broadcast, and use all the remaining content as the content to be displayed.

[0122] For example, if the summary feedback includes "average energy consumption this week is 12 kWh", while the detailed feedback includes a daily energy consumption table, then "average energy consumption this week is 12 kWh" can be used as the content to be broadcast, and the content to be displayed on the screen should no longer include the table data corresponding to "average energy consumption this week is 12 kWh", but only the specific daily energy consumption table should be displayed.

[0123] By using the above methods, the content of the voice broadcast can be avoided from being repeated with the content displayed on the screen, thus reducing redundant information.

[0124] In addition, while the content to be broadcast is being read aloud, the system can not only display the filtered content on the screen, but also identify semantically related elements from the content to be displayed and perform highlighting operations (such as highlighting, flashing, and adding boxes) on these related elements.

[0125] Taking a detailed broadcast scenario as an example, when the voice broadcasts "Monday's energy consumption is 15 kWh due to the air conditioner being turned on", the row containing "Monday" and the column containing "air conditioner being turned on" can be found in the table to be displayed on the screen, and the intersecting cells (or the entire row and column) of these two dimensions can be highlighted with a background color; or, if there is an energy consumption trend chart on the screen, the data points corresponding to Monday can be highlighted with flashing red dots or enlarged circles.

[0126] Alternatively, after setting "average energy consumption this week is 12 kWh" as the content to be broadcast, and ensuring that the content to be displayed on the screen no longer includes the table data corresponding to "average energy consumption this week is 12 kWh", but only displays the specific daily energy consumption table, when the voice broadcasts "average energy consumption this week is 12 kWh", the associated "daily energy consumption table" can be highlighted.

[0127] Using the above method, it is possible to achieve precise alignment and mutual enhancement of auditory and visual information. Users do not need to search for the object pointed to by the voice on the screen; their eyes will be automatically guided to the corresponding position, thereby reducing comprehension time and cognitive load.

[0128] In one feasible implementation, the summarizing feedback content is in written form, and the method of broadcasting the summarizing feedback content via voice can be at least one of the following: The written summary feedback content is transformed using a preset language processing model to generate a spoken expression text, which is then read aloud. The complexity and nature of the summary feedback content are determined. Based on the complexity and nature of the content, the speech playback speed and tone are determined. The summary feedback content is then played at the specified speech playback speed and tone. The nature of the content includes suggestive content and explanatory content.

[0129] It should be noted that written expression refers to an expression that conforms to written grammar, uses formal vocabulary, and has a complete structure. For example, "The average energy consumption of vehicles this week was 12 kilowatt-hours per 100 kilometers, a decrease of 8% compared to last week." Colloquial expression refers to text that has been converted to conform to everyday oral communication habits. The conversion from written to colloquial expression can be achieved by adding everyday vocabulary (such as "we," "we saved a little," etc.), using short sentences and simple structures, adding modal particles (such as "ne," "la," etc.), appropriately using colloquial conjunctions (such as "that," "look," etc.), and / or replacing technical terms with colloquial explanations.

[0130] After receiving the written summary feedback, you can first input it into a pre-defined language processing model. This model can perform format transformation operations on the written text, such as replacing formal vocabulary with common colloquial words (e.g., changing "decreased compared to last week" to "spent less than last week"), changing passive voice to active voice (e.g., changing "energy consumption was recorded as" to "energy consumption is"), breaking long sentences into short sentences, adding modal particles and colloquial opening phrases (e.g., "Let's take a look first" or "Then next"), ultimately generating a colloquial expression.

[0131] By adopting the above approach, the understanding threshold for users can be lowered, especially for the elderly, novice drivers, or users who are not familiar with professional terminology. The colloquial expression is more in line with daily listening habits, and the core content can be understood without having to work hard to analyze it, thereby reducing distraction while driving.

[0132] In addition, after receiving summary feedback, the complexity and nature of the content can be assessed first. For content with low complexity and explanatory nature (such as the simple core conclusion "energy consumption reduced by 8%), it can be broadcast at a normal speaking speed (e.g., 220 words per minute) and in a steady tone to ensure efficient information delivery.

[0133] For content of medium to high complexity that is informational in nature (e.g., "Energy consumption is higher on Mondays because the air conditioning is on"), a slightly slower speaking speed (e.g., 180 words per minute) can be used, and key words (e.g., "higher" and "air conditioning") can be emphasized, or short pauses can be added before and after words to ensure that users are less likely to miss this information while driving.

[0134] This embodiment adaptively adjusts the playback speed and tone by the content, making the voice output more in line with the human auditory cognition pattern. Important or noteworthy information is slowed down and emphasized, while routine information is delivered at a normal speed.

[0135] In one feasible implementation, the structured marked text includes structured visual content and structured text content; the structured text content is used to carry the textual descriptive information in the initial feedback content, and the structured visual content is used to carry the information in the initial feedback content that needs to be presented in a graphical or tabular manner.

[0136] Optionally, the structured visualization content may be presented in at least one of the following formats: code blocks, charts, and tables. A code block is a code segment defined by specific markers. This code block is used to display program code, configuration parameters, instruction sequences, or examples of structured data formats involved in the initial feedback content.

[0137] For example, in Markdown syntax, a code snippet is enclosed in three consecutive backticks (the symbols that are usually located to the left of the number 1 on the keyboard) to identify the content as a code block, so that the rendering engine can recognize it as a code region that requires special formatting.

[0138] Optionally, different types of code fields in the code block have different visual distinguishing identifiers; the visual distinguishing identifiers can be colors. For example, after the structured marked text of the vehicle system is rendered, different types of code fields such as keywords, strings, comments, and variable names in the code block are highlighted with different colors to help users quickly distinguish different syntax elements in the code.

[0139] Alternatively, visual distinguishing markers can also include font weight, underlines, etc.

[0140] Optionally, a chart refers to a graphical representation of content generated from a chart description statement. Chart description statements can be written using Mermaid syntax, and the charts generated based on these description statements include at least one of the following: flowcharts, sequence diagrams, line charts, and pie charts.

[0141] Alternatively, charts can also be generated by embedding image links and combining them with thumbnail identifiers. That is, the structured markup text contains the image's URL (Uniform Resource Locator) link and thumbnail size identifier. When rendering, the vehicle system loads the image data based on the link and scales it according to the size parameters of the thumbnail identifier.

[0142] For example, in scenarios that need to display data trends or logical processes, chart blocks defined by Mermaid syntax can be embedded in Markdown text. For instance, a line chart can be generated using the `lineChart` keyword, and a troubleshooting flowchart can be generated using the `graph` keyword. During rendering, the corresponding graphs can be automatically drawn and adapted to the screen size.

[0143] Optionally, the table can use table tags (such as the vertical bar "|" and hyphen "-" in Markdown syntax) to display structured data. The table uses vertical bars to separate columns, arranging the structured data from the initial feedback content in a row-column matrix for easy comparison and review by the user.

[0144] Optionally, the table header is visually highlighted when displayed. For example, the header row is rendered with a different background color, bold font, or highlighted border than the data rows to help users quickly locate the meaning of the fields in the table.

[0145] For example, when displaying code blocks, different colors, font styles, or background colors can be assigned to different types of fields in the code (such as keywords, variable names, strings, comments, etc.) based on the code language type (such as Python, Java, Shell, etc.) to improve the readability and semantic distinguishability of the code.

[0146] For scenarios requiring the display of road condition thumbnails or component diagrams, standard image link syntax can be embedded within Markdown text, followed by a custom thumbnail identifier. During rendering, the image can be displayed at a reduced scale (e.g., a quarter of the screen width), and clickable zoom is supported. For scenarios requiring the display of structured data, vertically separated table syntax is used in the text, with short horizontal lines separating the table header rows. During rendering, the system displays the table header rows in bold with a highlighted background color.

[0147] Optionally, the structured text content includes hierarchical headings and annotations to help explain the meaning of charts or codes, wherein the hierarchical headings are used to distinguish different logical levels.

[0148] Optionally, the hierarchical headings are identified using hierarchical markers (e.g., the hash symbol "#" in Markdown syntax, where one "#" corresponds to a first-level heading, two "##" correspond to a second-level heading, and so on).

[0149] For example, you can use hash symbols to mark hierarchical headings (such as "# Weekly Energy Consumption Overview" or "## Daily Energy Consumption Details") to distinguish information blocks at different logical levels, and add annotations written in plain text near charts or code blocks to help users understand the meaning of charts or the function of code.

[0150] Optionally, hierarchical headings are used to distinguish different logical levels within the initial feedback content. For example, the initial feedback content can be divided into multiple feedback segments according to the logical hierarchy of "general overview - specific points - detailed information," with each feedback segment corresponding to a content area defined by a hierarchical heading. Through hierarchical headings, the vehicle system can accurately identify the logical position of the current feedback content during voice broadcasting and screen display.

[0151] Optionally, the annotations are used to help explain the meaning of the charts or code. For example, a text description can be added before or after a code block, or before or after a chart, to explain to the user the purpose of the code block, the data source or interpretation method reflected in the chart, etc. The annotations are associated with the display position of the chart / code block (e.g., the annotation is above or below the chart), thereby helping users to understand the meaning of the visualization more efficiently.

[0152] For example, in scenarios where software code or diagnostic instructions need to be displayed, code blocks enclosed in three backticks can be embedded in Markdown text, with the language type (such as Python, JSON, shell, etc.) indicated at the beginning of the code block. During rendering, the code can be highlighted according to the language type, making keywords (such as if, def) appear in blue, comments in gray, and strings in green, greatly improving code reading efficiency.

[0153] Through the diverse Markdown elements mentioned above, a single text can carry various content formats, from simple text and structured tables to syntax-highlighted code, as well as visual charts and thumbnails, covering a variety of in-vehicle scenarios such as energy consumption analysis, vehicle diagnostics, and navigation guidance. Furthermore, due to the openness of Markdown syntax, if a 3D part model needs to be displayed in the future, a new extended tag (such as "3d_model") can be defined for seamless integration.

[0154] In one feasible implementation, the step of generating initial feedback content for responding to the human-computer interaction command includes: Obtain the display preference data of the user corresponding to the human-computer interaction command, determine the target display format based on the display preference data, and generate visual content to respond to the human-computer interaction command based on the target display format.

[0155] Before generating visual content to respond to user interaction commands, the current user's display preference data can be retrieved from the storage system. Display preference data can include the display methods preferred by the user for the same type of information (such as energy consumption data, diagnostic reports, and navigation routes) in historical interactions.

[0156] For example, if a user is found to have actively clicked the "Switch to Bar Chart" button twice out of their last three energy consumption analysis requests, or manually expanded the detailed daily data table three times, then based on this preference data, the target display format can be determined as follows: energy consumption trends will default to bar charts instead of line charts, and the full daily detailed table will be expanded by default instead of just displaying a summary. Furthermore, visualization content can be constructed and generated according to this target display format.

[0157] The above methods allow users to proactively adapt to their personalized habits without having to manually adjust the display mode every time they interact.

[0158] In one feasible implementation, the detailed feedback content includes visual content, and the step of displaying the visual content includes at least one of the following: The visualized content is displayed according to the characteristics of the in-vehicle screen; Highlight the visualization content.

[0159] It should be noted that in-vehicle screen characteristics refer to the set of physical and display parameters of the screen used for information display on the vehicle's infotainment system, including but not limited to screen size, screen resolution, screen aspect ratio, screen type, ambient light sensing capability, and anti-reflective coating characteristics. Visualized content refers to non-plain text information within detailed feedback that needs to be presented to the user visually.

[0160] When displaying visual elements within detailed feedback content, the parameters of the current in-vehicle screen's characteristics can be read first. The visual content can then be adaptively adjusted based on these parameters before rendering. This ensures that the visual content achieves a clear, readable, and user-friendly presentation on central control screens of different sizes, resolutions, and types.

[0161] During the presentation of visual content, specific elements within the visual content can be highlighted based on the current interaction context (e.g., summary content being broadcast, detailed areas pointed to by user follow-up questions, or important information points identified by the system). This allows users to quickly scan and locate areas on the screen relevant to the current auditory information, significantly reducing eye search time, thereby lowering the risk of driver distraction and improving information comprehension efficiency.

[0162] In one feasible implementation, the visualization content includes at least one of code blocks, charts, and tables, and the charts include at least one of line charts, flowcharts, and thumbnails. The step of displaying the visualization content according to the characteristics of the in-vehicle screen includes at least one of the following: The code block is displayed in the main area of ​​the vehicle's screen; Adjust the font size of the code block according to the screen size of the vehicle screen; Adjust the syntax highlight contrast of the code fields according to the resolution of the in-vehicle screen; Adjust the display ratio of the line graph, flowchart, or table according to the screen ratio of the in-vehicle screen.

[0163] Specifically, when a code block needs to be displayed, it can be rendered in the main area of ​​the in-vehicle screen. The main area can occupy 60% to 80% of the total screen area, providing ample horizontal and vertical space for the code block, allowing long lines of code to be displayed completely or wrapped appropriately, and multiple lines of code to be displayed at once, reducing the frequency of user scrolling.

[0164] When rendering code blocks, the screen size of the in-vehicle screen can be dynamically read, and the font size of the text within the code block can be automatically adjusted accordingly. For in-vehicle infotainment systems with smaller screens (such as 10.25 inches), the code font can be set to a larger value (such as 18 pixels) to ensure clear readability and reduce the visual burden and safety hazards caused by drivers having to lean close to the screen due to small font sizes. For in-vehicle infotainment systems with larger screens (such as 15.6 inches), the font can be set to a standard value (such as 16 pixels) to increase the number of lines that can be displayed on a single screen while maintaining readability. Through this screen size-adaptive font adjustment, optimal readability of code blocks can be ensured on in-vehicle infotainment systems of different models and screen sizes.

[0165] When rendering code blocks and highlighting syntax, the color contrast between different code fields (such as keywords, variables, strings, comments, etc.) can be dynamically adjusted according to the resolution of the in-vehicle screen. On high-resolution screens (such as 1920×720), due to the high pixel density and smooth color transitions, a higher contrast (such as dark blue keywords against a light gray background) can be used to enhance differentiation. On low-resolution screens (such as 1280×720), to avoid glare or color bleeding caused by high-contrast colors due to pixelation, the contrast can be appropriately reduced (for example, darkening the blue of keywords by one level and brightening the gray of comments by one level) to make the overall visual experience softer and more comfortable. By adjusting the contrast, it can be ensured that the syntax highlighting function can perform its semantic differentiation function under different display resolutions without causing visual fatigue.

[0166] When rendering line charts, flowcharts, or tables, the display ratio can be adaptively adjusted according to the aspect ratio of the in-vehicle screen. For a standard landscape screen with a 16:9 aspect ratio, line charts and flowcharts can be set to a horizontally elongated, vertically moderate ratio, allowing time series or process nodes to expand fully horizontally and avoiding node crowding. For ultrawide screens with an 8:3 aspect ratio (common in continuous screens), the horizontal ratio can be further extended, or longer process branches or more table columns can be displayed in a single row, making full use of the space on both sides of the screen to place legends and labels. For screens with a 4:3 aspect ratio, the horizontal ratio can be appropriately compressed, and the vertical display height can be increased, making the peaks and troughs of line charts more prominent in the vertical direction, and allowing tables to display more rows at once. By adaptively adjusting the screen ratio, it can be ensured that the visualized content is not cropped, distorted, or excessively white spaced under any screen shape, and is always presented at the ratio most suitable for that screen shape.

[0167] In one feasible implementation, the step of highlighting the visualized content includes: Adjust the row height or column width of the table according to the driving scenario, and then display it.

[0168] When displaying the table, vehicle operation data (including vehicle speed, driving mode, etc.) can be dynamically acquired. Based on the vehicle operation data, the current driving scenario can be determined, and the visual focus requirements and screen size of the in-vehicle screen under the current driving scenario can be further determined.

[0169] Optionally, the current driving scenario can be high-speed driving (e.g., vehicle speed above 30 kilometers per hour) or low-speed driving (e.g., vehicle speed below 30 kilometers per hour). The row height or column width of the table can be adjusted and displayed according to the visual focus requirements of the current driving scenario.

[0170] When it is determined that the user is driving at high speed and needs to quickly scan the table, the table's row height (e.g., from the default 40 pixels to 56 pixels) and / or column width (e.g., from 100 pixels to 130 pixels) can be automatically increased, making the text in each cell larger and the spacing more spacious, reducing the risk of misreading.

[0171] The table's row height or column width can be adjusted and displayed according to the visual focus requirements of the current driving scenario and the screen size of the in-vehicle screen.

[0172] When the screen size is small (e.g., 10.25 inches) and the vehicle is traveling at high speed, the row height increases more significantly. When the vehicle is stationary or traveling at low speed, the default row height and column width can be restored to display more data rows on a single screen and reduce the need for swiping.

[0173] In one feasible implementation, the step of highlighting the visualized content further includes at least one of the following: In response to a double-click zoom operation on a code block, the code block is zoomed in and displayed. Highlight and annotate the key data points in the charts; The thumbnail is displayed at a first display ratio. In response to a click operation on the thumbnail, the content of the thumbnail is displayed in full screen, and a text description of the thumbnail is displayed. Display a portion of the line chart, flowchart, or table at a second display scale, and in response to a sliding operation on the portion of the content, display the undisplayed content in the sliding direction corresponding to the sliding operation; Highlight the key cells in the table.

[0174] Specifically, after the code block is displayed on the screen, user touch events on the code block area can be monitored in real time. When a double-tap to zoom in is detected within the code block area, a zoom response is immediately triggered, enlarging the current code block's display area on the screen. The zoomed-in code block is typically presented as a floating layer or a full-screen modal view, with larger font sizes, increased line spacing, and possibly a horizontal scrollbar to accommodate long lines of code. In the zoomed-in state, the syntax highlight color of the code block can remain unchanged or have its contrast further optimized. When the user double-tap the zoomed-in code block area again, it can be restored to its original size.

[0175] When displaying charts (such as energy consumption trend line charts, troubleshooting flowcharts, etc.), one or more key data points can be identified and highlighted. Highlighting methods include changing the color of the data point's marker (such as a dot) to a highly saturated red, enlarging its size to 1.5 times that of a normal dot, adding a pulsating halo or flashing effect around it, or displaying its numerical label floating above the point. For flowcharts, key nodes can be highlighted using bold borders, changing fill colors, or adding star icons. This type of highlighting allows users to immediately grasp the most important information when scanning the chart.

[0176] When displaying thumbnails, they can be initially shown at a smaller initial display scale in a designated area of ​​the screen (such as the thumbnail preview area or a corner of the core display area). If a user is interested in the content of the thumbnail and wants to view details, they can click on the thumbnail to trigger a full-screen display. In response to the click, the content of the thumbnail can be enlarged to a full-screen or near-full-screen modal view, while a pre-prepared text description is overlaid on this view, such as explaining the meaning of the various parts of the image, key parameter values, or operation suggestions. The user can exit full-screen mode by clicking the full-screen image again or by clicking the close button, and the screen returns to its original layout.

[0177] When the content size of a line chart, flowchart, or table exceeds the display capacity of the core display area of ​​the screen, the complete content can be rendered at a second display ratio (usually the original 1:1 ratio), but only the content within the current viewport is displayed. For example, an energy consumption detail table with fifteen columns can only display the first five columns on the screen at a time. By supporting user swipe gestures (left / right or up / down) to change the viewport position, when the user swipes their finger to the left in the table area, the viewport moves to the right, the previously hidden sixth column gradually comes into view, and the first column on the left gradually moves out, thus achieving dynamic loading and display of undisplayed content. For excessively long flowcharts, up / down swiping is supported; for excessively long time series line charts, left / right swiping is supported.

[0178] When displaying tables, one or more key cells can be identified based on preset display rules (such as extreme values, user follow-up questions, and system-predicted importance) and highlighted. Highlighting methods can include changing the cell's background color to a highlight color (such as yellow or light red), bolding and enlarging the text within the cell, adding a border or shadow around the cell, or adding a small icon inside the cell. When a user points to specific data via voice (such as "Monday's energy consumption"), the corresponding cell can be dynamically highlighted, and the highlight can flash once while the data is read aloud. By highlighting key cells, users can quickly locate the specific data cell they need to focus on, without having to scan the entire table one by one.

[0179] In this embodiment, the separation strategy of "voice only reads the core information, and the screen displays details without repetition" eliminates information redundancy in traditional interactions; conversational expression and intelligent rhythm control enable users to easily capture key points even in distracted driving scenarios; adaptive rendering based on screen characteristics and driving scenarios ensures clear readability and ease of operation for various visual content under different vehicle models, speeds, and ambient light conditions; and rich highlighting and gesture interactions (zoom in, swipe, click to fill the screen) guide the user's gaze to the location of key information, significantly shortening visual search time.

[0180] For example, to help understand the implementation flow of the human-computer interaction method obtained by combining this embodiment with the above embodiments, please refer to... Figure 2-4 : like Figure 2 As shown, when a user issues the voice command "Analyze this week's energy consumption", the multimodal big model first generates the first sub-markdown segment (logic segment 1) for the first sub-requirement (energy consumption overview). This segment contains a first-level heading "This week's energy consumption overview", the core conclusion presented as a short-lined list "This week's average energy consumption is 12 kWh per 100 kilometers, a decrease of 8% compared to last week", and an energy consumption trend line chart defined using Mermaid syntax (the horizontal axis is Monday to Sunday, the vertical axis is the energy consumption value, and the data sequence is 15, 13, 12, 11, 14, 10, 9). The segment also adds labels at the end indicating "voice priority equals core summary" and "rendering priority equals mandatory display".

[0181] like Figure 3 As shown, the large model stream outputs the second markdown segment (logic segment 2), corresponding to the sub-requirement "Daily Energy Consumption Details". This segment contains a second-level heading, a table marker consisting of vertical and horizontal lines (the table columns include date, energy consumption, and main usage scenarios, and the row data are the specific values ​​from Monday to Sunday), and additional label information, indicating that the voice part only extracts the general conclusion that "energy consumption is higher on Monday and Friday because the air conditioning is on, and the lowest on weekends because of driving in the suburbs", instead of reading out all the data in the table line by line.

[0182] like Figure 4 As shown, the large model streams the third sub-markdown segment (logic segment 3), corresponding to the "energy consumption optimization suggestions" sub-requirement. This segment contains text suggestions in the form of an ordered list (avoid idling during morning rush hour, set the air conditioner to 24 degrees Celsius, maintain a speed of 60 to 80 kilometers per hour in suburban areas) and an image marker pointing to a thumbnail, with an additional label indicating that the voice only extracts the core content of the three suggestions. Through the above streaming, segmented, and labeled generation method, the system can start voice playback and screen rendering immediately after the first segment is generated without waiting for all segments to be generated, significantly reducing the initial response latency. At the same time, the label information in each segment provides a precise basis for voice filtering and rendering control for subsequent modules, thereby realizing a streaming interactive experience of "generating, playing, and displaying simultaneously," and ensuring that the voice only plays the core summary content specified by the label, while the screen fully displays all details, including charts, tables, and thumbnails, achieving precise separation of voice and display.

[0183] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the applicant's human-computer interaction method. Any simple modifications based on this technical concept are within the protection scope of this application.

[0184] This application also provides a human-computer interaction device, please refer to... Figure 5 The human-computer interaction device includes: The content generation module 10 is used to generate initial feedback content for responding to the human-computer interaction instruction when a human-computer interaction instruction is received. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The interactive feedback module 20 is used to broadcast the summary feedback content via voice and simultaneously display the summary feedback content and / or the detailed feedback content.

[0185] In one embodiment, the initial feedback content includes multiple feedback segments, and the content generation module 10 includes: The content generation submodule is used to determine the first interaction requirement based on the human-computer interaction instruction through a multimodal large model, decompose the first interaction requirement into multiple sub-requirements according to a preset logical hierarchy, and generate multiple feedback segments corresponding to the multiple sub-requirements in sequence according to the preset logical hierarchy.

[0186] In one embodiment, the summary feedback content includes summary content corresponding to each of the feedback segments, the detailed feedback content includes detailed content corresponding to each of the feedback segments, and the interactive feedback module 20 includes: The first interactive feedback submodule is used to sequentially broadcast the summary content of each segment in the preset logical hierarchy, and simultaneously display the summary content and / or detailed content of each segment in the preset logical hierarchy.

[0187] In one embodiment, the preset logic hierarchy includes at least one hierarchy, and the interactive feedback module 20 further includes: The second interactive feedback submodule is used to broadcast the summary content of the target segment corresponding to the target logical level via voice, and simultaneously display the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level, wherein the target logical level is any level in the preset logical levels; After the steps of simultaneously displaying the summary content of the target segment corresponding to the target logical level via voice broadcast, and / or the detailed content of the target segment corresponding to the target logical level, the device further includes: The follow-up feedback submodule is used to determine a second interaction requirement if a follow-up question instruction is detected for the voice broadcast content or a further operation is detected for the displayed content. Based on the second interaction requirement, the content to be fed back is determined, and the voice broadcast and display of the content to be fed back are continued. The follow-up question instruction includes at least one of a repeat playback instruction and a further question, and the further operation includes at least one of clicking the replay button and clicking the displayed detailed feedback content. The follow-up question detection submodule is used to stop the voice broadcast and display of the remaining feedback segment if no follow-up question instruction or further operation on the displayed content is detected.

[0188] In one embodiment, the follow-up feedback submodule includes: The content determination submodule is used to determine the content to be fed back as the content corresponding to the target feedback segment when there is a target feedback segment that matches the second interaction requirement; The follow-up feedback submodule is used to generate follow-up feedback content based on the second interaction requirement and the corresponding summary content of the fragments when all feedback fragments do not match the second interaction requirement, and to use the follow-up feedback content as the feedback content to be fed back.

[0189] In one embodiment, the follow-up feedback submodule further includes at least one of the following: The first requirement determination submodule is used to determine the second interaction requirement based on the keywords in the follow-up question instruction; The second requirement determination submodule is used to obtain the user's historical interaction preference data and determine the second interaction requirement based on the historical interaction preference data. The third requirement determination submodule is used to determine the complexity of the content involved in the follow-up question and / or the further operation, and to determine the second interaction requirement based on the complexity. The follow-up feedback submodule also includes: The feedback content determination unit is used to determine the feedback content based on the type of the second interaction request, which includes summary follow-up questions and detailed follow-up questions.

[0190] In one embodiment, both the initial feedback content and the follow-up feedback content include a voice priority tag and a rendering priority tag. The value of the voice priority tag is used to characterize the voice playback priority of the corresponding content, and the value of the rendering priority tag is used to characterize the display priority of the corresponding content. Before the step of performing voice playback based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content, the device further includes at least one of the following: The first content determination module is used to determine the content to be broadcast by voice based on the value of the voice priority tag; The second content determination module is used to determine the content to be displayed based on the value of the rendering priority tag.

[0191] In one embodiment, each feedback segment corresponds to a timestamp, the follow-up question content corresponds to a sub-timestamp, and the interactive feedback submodule includes: An interactive feedback unit is used to broadcast the summary content of each segment in sequence according to the preset logical hierarchy based on each timestamp, and simultaneously display the summary content of each segment and / or the detailed content of each segment in sequence according to the preset logical hierarchy. The follow-up questioning and feedback module also includes: The first broadcast and display submodule is used to continue broadcasting and displaying the follow-up feedback content by voice according to the sub-timestamp.

[0192] In one embodiment, the interactive feedback module 20 further includes: The display submodule is used to simultaneously display the summary content and / or detailed content of each feedback segment in different functional areas of the in-vehicle screen. The functional areas include a core display area, a thumbnail preview area, and a voice-synchronized prompt area. The detailed content of the segments includes charts, code blocks, tables, and thumbnails. The display sub-modules include: The display unit is used to display the charts, code blocks, and / or tables of the current feedback segment in the core display area, display the thumbnail of the next feedback segment in the thumbnail preview area, and display the key content of the current feedback segment in the voice synchronization prompt area; wherein, the current feedback segment is the feedback segment corresponding to the summary content of the current voice broadcast segment, and the next feedback segment is the first feedback segment arranged after the current feedback segment according to the preset logical hierarchy.

[0193] In one embodiment, the interactive feedback module 20 further includes at least one of the following: The content to be broadcast and displayed submodule is used to determine the content to be broadcast based on the summary feedback content, determine the content to be displayed in the summary feedback content and / or the detailed feedback content other than the content corresponding to the content to be broadcast, broadcast the content to be broadcast by voice, and display the content to be displayed synchronously. The second broadcasting and display submodule is used to broadcast the content to be broadcast via voice and simultaneously highlight related content in the content to be displayed that is related to the content to be broadcast.

[0194] In one embodiment, the summary feedback content is in the form of a written expression, and the interactive feedback module 20 further includes at least one of the following: The form conversion submodule is used to perform form conversion processing on the written summary feedback content through a preset language processing model to generate colloquial expression text, and then broadcast the colloquial expression text by voice. The broadcast speed and tone determination submodule is used to determine the content complexity and content nature of the summary feedback content, and to determine the voice broadcast speed and voice broadcast tone based on the content complexity and the content nature, and to broadcast the summary feedback content at the voice broadcast speed and the voice broadcast tone, wherein the content nature includes prompting content and explanatory content.

[0195] In one embodiment, the interactive feedback module 20 further includes: Generate structured marked text, which is used to describe the initial feedback content; The structured marked text includes structured visual content and structured text content; The structured visualization content is displayed in at least one of the following formats: code blocks, charts, and tables, wherein different types of code fields in the code blocks have different visual distinguishing identifiers; The structured text content includes hierarchical headings and annotations to help explain the meaning of charts or codes. The hierarchical headings are used to distinguish different logical levels.

[0196] In one embodiment, the interactive feedback module 20 further includes: The display format determination submodule is used to obtain the display preference data of the user corresponding to the human-computer interaction command, determine the target display format based on the display preference data, and generate visual content for responding to the human-computer interaction command based on the target display format.

[0197] In one embodiment, the detailed feedback content includes visual content, and the interactive feedback module 20 further includes at least one of the following: The display submodule is used to display the visual content according to the characteristics of the vehicle screen; The highlighting submodule is used to highlight the visualized content; The visualization content includes at least one of code blocks, charts, and tables; the charts include at least one of line charts, flowcharts, and thumbnails; and the highlighting submodule includes: The table display unit is used to adjust the row height or column width of the table according to the driving scenario and then display it.

[0198] The human-computer interaction device provided in this application, employing the human-computer interaction method described in the above embodiments, can solve the technical problem that current information feedback methods are insufficient to meet the user's need for efficient information acquisition in driving scenarios. Compared with the prior art, the beneficial effects of the human-computer interaction device provided in this application are the same as those of the human-computer interaction method described in the above embodiments, and other technical features in the human-computer interaction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0199] This application provides a human-computer interaction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the human-computer interaction method in the first embodiment described above.

[0200] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a human-computer interaction device suitable for implementing the embodiments of this application. The human-computer interaction device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 6 The human-computer interaction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0201] like Figure 6As shown, the human-computer interaction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the human-computer interaction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the human-machine interface device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show human-machine interface devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0202] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0203] The human-computer interaction device provided in this application, employing the human-computer interaction method described in the above embodiments, can solve the technical problem that current information feedback methods are insufficient to meet the user's need for efficient information acquisition in driving scenarios. Compared with the prior art, the beneficial effects of the human-computer interaction device provided in this application are the same as those of the human-computer interaction method described in the above embodiments, and other technical features of this human-computer interaction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0204] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0205] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0206] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the human-computer interaction method in the above embodiments.

[0207] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0208] The aforementioned computer-readable storage medium may be included in a human-computer interaction device; or it may exist independently and not be assembled into a human-computer interaction device.

[0209] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a human-computer interaction device, cause the human-computer interaction device to perform the aforementioned human-computer interaction method.

[0210] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0212] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0213] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described human-computer interaction method. This addresses the technical problem that current information feedback methods are insufficient to meet the user's need for efficient information acquisition in driving scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the human-computer interaction method provided in the above embodiments, and will not be elaborated upon here.

[0214] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the human-computer interaction method described above.

[0215] The computer program product provided in this application can solve the technical problem that current information feedback methods are insufficient to meet the interactive needs of users in driving scenarios for efficient information acquisition. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the human-computer interaction method provided in the above embodiments, and will not be repeated here.

[0216] The above descriptions are merely some embodiments of this application and do not limit the scope of protection of this application. Any equivalent structural transformations made based on the technical concept of this application and the content of this specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the corresponding device.

Claims

1. A human-computer interaction method, characterized in that, The human-computer interaction method includes: When a human-computer interaction command is received, initial feedback content is generated to respond to the human-computer interaction command. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The summary feedback is read aloud, and the summary feedback and / or the detailed feedback are displayed simultaneously.

2. The human-computer interaction method as described in claim 1, characterized in that, The initial feedback content includes multiple feedback segments, and the step of generating the initial feedback content for responding to the human-computer interaction command includes: Based on the human-computer interaction instructions, a first interaction requirement is determined through a multimodal large model. The first interaction requirement is then broken down into multiple sub-requirements according to a preset logical hierarchy. Multiple feedback segments corresponding to the multiple sub-requirements are generated sequentially according to the preset logical hierarchy.

3. The human-computer interaction method as described in claim 2, characterized in that, The summary feedback content includes summary content corresponding to each of the feedback segments, and the detailed feedback content includes detailed content corresponding to each of the feedback segments. The step of broadcasting the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content includes: The summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content and / or detailed content of each segment are displayed in sequence according to the preset logical hierarchy.

4. The human-computer interaction method as described in claim 3, characterized in that, The preset logical hierarchy includes at least one hierarchy, and the step of broadcasting the summary feedback content via voice and simultaneously displaying the summary feedback content and / or the detailed feedback content further includes: The system broadcasts a summary of the target segment corresponding to the target logical level, and simultaneously displays the summary of the target segment and / or the detailed content of the target segment corresponding to the target logical level, wherein the target logical level is any level among the preset logical levels. After the steps of simultaneously displaying the summary content of the target segment corresponding to the target logical level in the voice broadcast, and simultaneously displaying the summary content of the target segment and / or the detailed content of the target segment corresponding to the target logical level, the method further includes: If a follow-up question instruction is detected for the voice broadcast content, or a further operation is detected for the displayed content, a second interaction requirement is determined. Based on the second interaction requirement, the content to be fed back is determined, and the voice broadcast and display of the content to be fed back continue. The follow-up question instruction includes at least one of a repeat playback instruction and a further question, and the further operation includes at least one of clicking the replay button and clicking the displayed detailed feedback content. If no follow-up question is detected regarding the voice broadcast content, or no further operation is detected regarding the displayed content, then the remaining feedback segment will no longer be broadcast or displayed via voice.

5. The human-computer interaction method as described in claim 4, characterized in that, The step of determining the content to be fed back based on the second interaction request includes: When a target feedback segment that matches the second interaction requirement exists, the content to be fed back is determined to be the content corresponding to the target feedback segment; When all feedback fragments do not match the second interaction requirement, follow-up feedback content is generated based on the second interaction requirement and the corresponding fragment summary content, and the follow-up feedback content is used as the feedback content to be provided.

6. The human-computer interaction method as described in claim 4, characterized in that, The step of determining the second interaction requirement includes at least one of the following: Based on the keywords in the follow-up question, determine the second interaction requirement; Obtain the user's historical interaction preference data, and determine the second interaction requirement based on the historical interaction preference data; Determine the complexity of the content involved in the follow-up question and / or the further operation, and determine the second interaction requirement based on the complexity. The step of determining the content to be fed back based on the second interaction request further includes: Based on the type of the second interaction request, determine the content to be fed back. The types of the second interaction request include summary follow-up questions and detailed follow-up questions.

7. The human-computer interaction method as described in claim 5, characterized in that, Both the initial feedback content and the follow-up feedback content include a voice priority tag and a rendering priority tag. The value of the voice priority tag is used to represent the voice playback priority of the corresponding content, and the value of the rendering priority tag is used to represent the display priority of the corresponding content. Before the step of performing voice playback based on the summary feedback content and simultaneously displaying the summary feedback content and / or the detailed feedback content, at least one of the following is also included: Based on the value of the voice priority label, determine the content that needs to be broadcast by voice; The content to be displayed is determined based on the value of the rendering priority tag.

8. The human-computer interaction method as described in claim 5, characterized in that, Each feedback segment corresponds to a timestamp, and the follow-up feedback content corresponds to a sub-timestamp. The step of sequentially broadcasting the summary content of each segment according to the preset logical hierarchy, and simultaneously displaying the summary content and / or detailed content of each segment according to the preset logical hierarchy, includes: Based on each timestamp, the summary content of each segment is broadcast in sequence according to the preset logical hierarchy, and the summary content of each segment and / or the detailed content of each segment is displayed in sequence according to the preset logical hierarchy. The step of continuing to broadcast and display the content to be fed back via voice includes: Based on the sub-timestamp, the follow-up question feedback content will continue to be read aloud and displayed via voice.

9. The human-computer interaction method as described in claim 3, characterized in that, The step of simultaneously displaying the summary feedback content and / or the detailed feedback content includes: The summary content and / or detailed content of each feedback segment are displayed simultaneously in different functional areas of the in-vehicle screen. The functional areas include a core display area, a thumbnail preview area, and a voice-synchronized prompt area. The detailed content of the segments includes charts, code blocks, tables, and thumbnails. The step of simultaneously displaying the summary content and / or detailed content of each feedback segment in different functional areas of the in-vehicle screen includes: The core display area displays charts, code blocks, and / or tables of the current feedback segment; the thumbnail preview area displays a thumbnail of the next feedback segment; and the voice synchronization prompt area displays the key content of the current feedback segment. The current feedback segment is the feedback segment corresponding to the summary content of the current voice broadcast segment, and the next feedback segment is the first feedback segment arranged according to the preset logical hierarchy after the current feedback segment.

10. The human-computer interaction method according to any one of claims 1 to 9, characterized in that, The step of broadcasting the summary feedback content via voice and simultaneously displaying the summary feedback content and / or the detailed feedback content includes at least one of the following: Based on the summary feedback content, determine the content to be broadcast, and determine the content to be displayed in the summary feedback content and / or the detailed feedback content other than the content corresponding to the content to be broadcast. Then, broadcast the content to be broadcast by voice and display the content to be displayed simultaneously. The system reads the content to be read aloud and simultaneously highlights related content within the content to be displayed.

11. The human-computer interaction method according to any one of claims 1 to 9, characterized in that, The summary feedback is in written form, and the step of broadcasting the summary feedback by voice includes at least one of the following: The written summary feedback content is transformed using a preset language processing model to generate a spoken expression text, which is then read aloud. The complexity and nature of the summary feedback content are determined. Based on the complexity and nature of the content, the speech playback speed and tone are determined. The summary feedback content is then played at the specified speech playback speed and tone. The nature of the content includes suggestive content and explanatory content.

12. The human-computer interaction method according to any one of claims 1 to 9, characterized in that, The step of generating initial feedback content for responding to the human-computer interaction command includes: Generate structured marked text, which is used to describe the initial feedback content; The structured marked text includes structured visual content and structured text content; The structured visualization content is displayed in at least one of the following formats: code blocks, charts, and tables, wherein different types of code fields in the code blocks have different visual distinguishing identifiers; The structured text content includes hierarchical headings and annotations to help explain the meaning of charts or codes. The hierarchical headings are used to distinguish different logical levels.

13. The human-computer interaction method as described in claim 12, characterized in that, The step of generating initial feedback content for responding to the human-computer interaction command includes: Obtain the display preference data of the user corresponding to the human-computer interaction command, determine the target display format based on the display preference data, and generate visual content to respond to the human-computer interaction command based on the target display format.

14. The human-computer interaction method according to any one of claims 1 to 9, characterized in that, The detailed feedback content includes visual content, and the steps for displaying the visual content include at least one of the following: The visualized content is displayed according to the characteristics of the in-vehicle screen; Highlight the visualized content; The visualization content includes at least one of code blocks, charts, and tables; the charts include at least one of line charts, flowcharts, and thumbnails; and the step of highlighting the visualization content includes: Adjust the row height or column width of the table according to the driving scenario, and then display it.

15. A human-computer interaction device, characterized in that, The human-computer interaction device includes: The content generation module is used to generate initial feedback content for responding to the human-computer interaction instruction when a human-computer interaction instruction is received. The initial feedback content includes detailed feedback content and summary feedback content extracted from the detailed feedback content. The interactive feedback module is used to broadcast the summary feedback content via voice and simultaneously display the summary feedback content and / or the detailed feedback content.

16. A human-computer interaction device, characterized in that, The human-computer interaction device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the human-computer interaction method as described in any one of claims 1 to 14.

17. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the human-computer interaction method as described in any one of claims 1 to 14.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the human-computer interaction method as described in any one of claims 1 to 14.

Images