A method and system for driving state rendering and human-machine interaction

Abstract

This invention relates to a method for rendering driving states and human-computer interaction, the method comprising: Step 1, a data source module generating real-time driving environment and vehicle state data; Step 2, a service-oriented communication proxy module receiving the data generated by the data source module, parsing it, and providing a parsed list of environmental objects, a planned trajectory, and vehicle signals, providing raw or pre-processed data; Step 3, an interaction and configuration management module receiving user input from a touchscreen, physical buttons, and voice commands; Step 4, a state management core module maintaining the high-level state of the system in real time, generating standardized event codes, and broadcasting the current state to a multi-layer rendering engine module; Step 5, a multi-layer rendering engine module generating a graphical interface that the driver can directly perceive; and Step 6, a multi-modal interaction execution module receiving event codes from the state management core module and converting state change events into multi-channel feedback that the driver can perceive.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method for rendering driving status and human-computer interaction. Background Technology

[0002] Currently, Level 2 and above autonomous driving systems are becoming increasingly common. However, how to efficiently, clearly, and reliably convey the system's real-time status, environmental perception results, and driving intentions to the driver has become a key bottleneck affecting the safety and user experience of human-machine co-driving. Existing technical solutions mainly suffer from the following shortcomings: Most in-vehicle HMIs simply overlay elements such as the vehicle icon, lane lines, traffic participants (SR), and navigation routes onto the same visual plane. In complex urban scenarios, a large amount of information emerges simultaneously, overwhelming key decision-making intentions (such as lane change targets and risk warnings), resulting in a heavy cognitive load and potential safety hazards. Existing rendering content lacks deep integration and dynamic linkage with the internal state of the autonomous driving system. For example, when the system exits from an active state or degrades, often only one icon changes, while visual elements on the screen, such as the planned route and tracking target, do not provide synchronous status prompts (such as graying out, flashing, or changing styles), causing a delay in the driver's perception of system status changes and potentially missing the optimal time for takeover. Fixed information feedback channels, such as using the same intensity of voice alarms in all states, can easily lead to "alarm fatigue" and limit the driver's understanding and supervision of the system's capabilities. Therefore, there is an urgent need for a solution that can deeply integrate the internal state of the intelligent driving system, realize multi-level intelligent information association and collaborative presentation, and provide flexible, natural, context-aware multi-channel interaction. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method and system for rendering driving status and human-computer interaction, so as to solve the technical problems of existing technologies such as mixed information presentation, high cognitive load, disconnect between display logic and system status, single and rigid interaction mode, and lack of contextual intelligence.

[0004] This invention provides a system for driving state rendering and human-computer interaction, applied to intelligent driving vehicles. The system includes: a data source module, located within the intelligent driving domain controller, for generating real-time driving environment and vehicle state data; a service-oriented communication proxy module, located in the application layer of the intelligent cockpit domain controller and connected to the data source module, for receiving and parsing the data generated by the data source module, providing a parsed list of environmental objects, planned trajectories, and vehicle signals to the multi-layer rendering engine module, and providing raw or pre-processed data to the state management core module for state machine updates; and a state management core module, connected to the service-oriented communication proxy module, for maintaining the system's higher-order states in real time based on data from the service-oriented communication proxy module and user input, and responding to changes in state. Standardized event codes are generated and sent to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. The multi-layer rendering engine module, connected to the state management core module and the service communication proxy module, generates a graphical image that the driver can directly perceive based on data from the service communication proxy module and the state management core module, through the vehicle state rendering layer, environment perception rendering layer, decision planning rendering layer, and image synthesizer. The multimodal interaction execution module, connected to the state management core module, receives event codes from the state management core and converts state change events into multi-channel feedback that the driver can perceive based on the event codes. The interaction and configuration management module, connected to the state management core module, receives user input from the touch screen, physical buttons, and voice commands.

[0005] Furthermore, the data source module includes: a perception submodule, used to fuse sensor data from cameras, millimeter-wave radar, and lidar to identify and track traffic participants and road structures, and generate a list of environmental objects; a positioning submodule, used to fuse GPS, IMU, and high-precision map matching to output the vehicle's precise location, attitude, and lane information; a planning submodule, used to generate local trajectories and driving intentions based on the global path, real-time environment, and decision-making algorithms, and plan path point sequences and target information; and a vehicle bus, used to acquire vehicle physical state signals, including vehicle speed, gear position, door status, and headlight status.

[0006] This invention also includes a method for rendering driving states and human-computer interaction, the method comprising: Step 1, a data source module generating real-time driving environment and vehicle state data based on sensor data, positioning data, route planning data, and vehicle physical state signals, and sending it to a service-oriented communication proxy module; Step 2, the service-oriented communication proxy module receiving the data generated by the data source module, parsing it, providing the multi-layer rendering engine module with a parsed list of environmental objects, planned trajectory, and vehicle signals, and providing the state management core module with raw or pre-processed data for state machine updates; Step 3, an interaction and configuration management module receiving user input from a touchscreen, physical buttons, and voice commands, and sending it to the state management core module. Step 4: The state management core module maintains the system's high-level state in real time based on data from the service communication proxy module and user input. When the state changes, it generates standardized event codes and sends them to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. Step 5: The multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, environment perception rendering layer, decision planning rendering layer, and image synthesizer, based on data from the service communication proxy module and the state management core module. Step 6: The multimodal interaction execution module receives event codes from the state management core module and transforms state change events into multi-channel feedback that the driver can perceive based on the event codes.

[0007] Furthermore, step 1 includes: Step 11, the perception submodule fuses sensor data from cameras, millimeter-wave radar, and lidar to identify and track traffic participants and road structures, generating a list of environmental objects; Step 12, the positioning submodule fuses GPS, IMU, and high-precision map matching to output the vehicle's precise location, attitude, and lane information; Step 13, the planning submodule generates local trajectories and driving intentions based on the global path, real-time environment, and decision algorithm, planning a sequence of path points and target information; Step 14, the vehicle bus acquires vehicle physical status signals, including vehicle speed, gear position, door status, and headlight status; Step 15, the obtained data is encapsulated into an independent rendering data service via the SOME / IP protocol and sent to the service-oriented communication proxy module based on a ProtoBuf encoded data interface.

[0008] Furthermore, step 2 includes: step 21, automatically discovering and subscribing to various rendering data services provided by the data source module based on the SOME / IP Service Discovery mechanism; step 22, after receiving the SOME / IP protocol packet containing the rendering data service, extracting the ProtoBuf-encoded payload data and deserializing it into a structured object for use by the upper-layer rendering module; step 23, monitoring the link status through the SOME / IP communication mechanism, handling abnormal situations, and ensuring the real-time performance and reliability of the rendering data; step 24, providing the multi-layer rendering engine module with a parsed list of environmental objects, planned trajectories, and vehicle signals, and providing the state management core module with raw or pre-processed data for state updates.

[0009] Furthermore, step 4 includes: Step 41, defining system states, including: defining OFF as intelligent driving function off; defining STANDBY as intelligent driving function in standby mode, waiting to be activated; defining ACTIVE_NORMAL as intelligent driving function normally activated; defining ACTIVE_DEGRADED as some functions are limited due to sensor performance degradation; defining TAKEOVER_REQUEST as the system requests the driver to take over immediately; defining FAILURE as a system fault; Step 42, using data from the service communication proxy module and user input, performing system state transitions when state transition triggers occur; Step 43, continuously updating the system state to ensure that the state reflects the real system situation. When the state changes, a standardized event code is generated according to the definition and sent to the multimodal interaction execution module; Step 44, broadcasting the current state to the multi-layer rendering engine module so that each rendering layer can synchronously adjust the visual style.

[0010] Furthermore, the triggering factors include: the data quality of the perception, positioning, and planning modules, the decisions output by the planning module, user operations, and vehicle signals.

[0011] Furthermore, step 5 includes: Step 51, the vehicle state rendering layer renders the vehicle 3D / 2D model, door opening / closing status, headlight effects, instrument information, and coupling status based on the vehicle's physical state signals; Step 52, the environment perception rendering layer renders static road elements, dynamic traffic participants, and state coupling based on sensor data and positioning data; Step 53, the decision planning rendering layer renders the planned path line, following / lane-changing target highlighting, virtual area prompts, motion effects, and semantic associations based on route planning data and system status; Step 54, the image compositer superimposes and synthesizes the outputs of the three rendering layers in a preset order, and adjusts the overall tone, brightness, and contrast according to the current theme, ultimately generating a graphic image that the driver can directly perceive, and outputting it to the display screen.

[0012] Furthermore, step 6 includes: step 61, receiving event codes from the state management core, and performing multi-dimensional dynamic decision output based on the event codes; step 62, sending a high-priority visual update instruction to the multi-layer rendering engine module, playing the corresponding TTS voice or pre-recorded audio clip, and displaying modal or non-modal text prompt boxes in a specified area of ​​the screen to explain the state or failure reason in detail.

[0013] Furthermore, the multi-dimensional dynamic decision output includes: step 611, outputting the priority level of the event as urgent / important / normal / alert according to the event corresponding to the event code; step 612, displaying the current driving mode as manual driving mode / autonomous driving mode; step 613, issuing visual and voice commands according to user preferences and context.

[0014] This invention provides a method and system for driving state rendering and human-computer interaction, mainly to solve the problems existing in the prior art, such as the difficulty for drivers to quickly understand what the system is "doing" and "will do," the inability to provide a global or detailed perspective that matches autonomous driving decisions, and the inability to adaptively adjust according to different driving modes (manual / automatic), user preferences, or specific driving scenarios (such as nighttime highways or congested intersections). Attached Figure Description

[0015] Figure 1 A flowchart of a method for rendering driving status and human-computer interaction provided by the present invention; Figure 2 This invention provides a flowchart of a method for generating real-time driving environment and vehicle status data; Figure 3 This is a flowchart of the method for receiving and parsing data provided by the present invention; Figure 4 This is a flowchart of the method for generating standardized event codes for the high-order state of the maintenance system provided by the present invention; Figure 5 This is a flowchart of the method for generating a graphical image that can be directly perceived by the driver, provided by the present invention. Figure 6 This is a flowchart of a method provided by the present invention for converting state change events into multi-channel feedback that can be perceived by the driver. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Example 1: This invention provides a system and method for rendering driving states and human-computer interaction. The system includes a data source module, a service-oriented communication proxy module, a state management core module, a multi-layer rendering engine module, a multi-modal interaction execution module, and an interaction and configuration management module. As follows... Figure 1 As shown, the method includes: Step 1: The data source module generates real-time driving environment and vehicle status data based on sensor data, positioning data, route planning data, and vehicle physical status signals, and sends it to the service-oriented communication proxy module. The data source module, located within the intelligent driving domain controller, is used to generate real-time driving environment and vehicle status data. This data source module, also known as the data source layer, is a service provider layer. Situated within the intelligent driving domain controller, it comprises sub-modules such as perception, localization, planning, and vehicle bus, and is responsible for generating real-time driving environment and vehicle status data. Each module acts as a service provider, encapsulating data into independent rendering data services via the SOME / IP protocol.

[0018] Step 2: The service-oriented communication proxy module receives the data generated by the data source module, parses it, provides the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provides the raw or pre-processed data to the state management core module for state machine updates. The service-oriented communication proxy module is located in the application layer of the intelligent cockpit domain controller. It serves as a bridge between the front-end rendering and the back-end service. It is connected to the data source module to receive the data generated by the data source module, parse it, provide the multi-layer rendering engine module with the parsed list of environmental objects, planned trajectory and vehicle signals, and provide the state management core module with raw or pre-processed data for state machine updates. Step 3: The interaction and configuration management module receives user input from the touch screen, physical buttons, and voice commands, and sends it to the status management core module. The interaction and configuration management module, connected to the state management core module, is used to receive user input from the touch screen, physical buttons, and voice commands.

[0019] Step 4: The state management core module maintains the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates a standardized event code and sends it to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. The state management core module is connected to the service communication proxy module. It is used to maintain the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates standardized event codes and sends them to the multimodal interaction execution module, and broadcasts the current state to the multi-layer rendering engine module. Step 5: Based on the data from the service communication proxy module and the state management core module, the multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer, and the image synthesizer. The multi-layer rendering engine module is connected to the state management core module and the service communication proxy module respectively. It is used to generate a graphical image that the driver can directly perceive based on the data from the service communication proxy module and the state management core module, through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer and the image synthesizer. Step 6: The multimodal interaction execution module receives event codes from the state management core module and transforms the state change events into multi-channel feedback that the driver can perceive based on the event codes.

[0020] The multimodal interaction execution module, connected to the state management core module, is used to receive event codes from the state management core and transform state change events into multi-channel feedback that the driver can perceive based on the event codes.

[0021] This invention provides a method and system for rendering driving states and human-computer interaction. This technical solution uses the core state of the intelligent driving system as a unified driving source, performing hierarchical, semantically related, and dynamically responsive visual rendering of the vehicle model, environmental scenes, and decision-making planning, ensuring clear information presentation structure, highlighting key points, and real-time accuracy. The solution supports multi-channel intelligent collaboration and adaptive distribution, including visual, auditory (voice), and text (pop-up windows), automatically optimizing information transmission methods based on event urgency and driving context to reduce interference and enhance prompt effectiveness. The solution also provides an efficient, scalable, and service-oriented data communication and processing framework to ensure low-latency, high-reliability transmission of massive amounts of perception and planning data within the vehicle network, supporting complex real-time rendering at the front end.

[0022] Example 2: This invention provides a method for rendering driving status and human-computer interaction, such as... Figure 1 As shown, the method includes: Step 1: The data source module generates real-time driving environment and vehicle status data based on sensor data, positioning data, route planning data, and vehicle physical status signals, and sends it to the service-oriented communication proxy module. As mentioned above, the data source module consists of sub-modules such as perception, localization, planning, and vehicle bus, providing data services. All services use the SOME / IP protocol for registration and announcement, and provide a data interface based on ProtoBuf encoding. Figure 2 As shown, step 1 includes: Step 11: The perception submodule fuses sensor data from cameras, millimeter-wave radar, and lidar to identify and track traffic participants and road structures, and generates a list of environmental objects. Step 12: The positioning submodule integrates GPS, IMU, and high-precision map matching to output the vehicle's precise location, attitude, and lane information; Step 13: The planning submodule generates local trajectories and driving intentions, planning path point sequences and target information based on the global path, real-time environment and decision algorithm; Step 14: The vehicle bus acquires vehicle physical status signals, including vehicle speed, gear position, door status, and headlight status. Step 15: Encapsulate the obtained data into an independent rendering data service using the SOME / IP protocol, and send it to the service-oriented communication proxy module based on the ProtoBuf encoded data interface.

[0023] Step 2: The service-oriented communication proxy module receives the data generated by the data source module, parses it, provides the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provides the raw or pre-processed data to the state management core module for state machine updates. To support the front-end rendering engine's requirements for real-time performance, accuracy, and structured data, this invention encapsulates the core data provided by the back-end into an independent SOME / IP service and uses Protocol Buffers (ProtoBuf) as the data serialization protocol. Detailed descriptions of these data are as follows: Environment Object List Service (SR_Objects_Service): Service ID: 0x1001; Release Frequency: 10Hz ~ 20Hz (dynamically adjusted according to the scene; high frequency for urban scenes, appropriately lower for high-speed scenes); Purpose: Provides real-time environmental perception results, including a list of all dynamic traffic participants and static obstacles; Timing Constraints: Each frame of data must contain a timestamp, which the rendering engine uses to determine data timeliness; data not updated for more than 100ms should be marked as stale; the sequence number (seq) should monotonically increase; if a jump (greater than 1) is found, it indicates frame loss, and the rendering engine can perform interpolation or discarding. Planning Trajectory Service: Service ID: 0x1002; Release frequency: 10Hz ~ 50Hz (consistent with the planning module cycle, usually 10Hz); Purpose: Provides the expected driving trajectory over a future period, as well as the current driving intent (such as following target, lane change status); Timing constraints: The relative_time of trajectory points should start from 0 and increment, and the time of the last point should not exceed 10 seconds (can be dynamically adjusted according to vehicle speed); The rendering engine needs to calculate the real-time position of the trajectory based on the current timestamp and relative_time to draw the future path. Vehicle Signal Service: Service ID: 0x1003 Publication frequency: Up to 50Hz (matching the CAN signal update frequency); Purpose: Provides the vehicle's own physical status signals for L1 layer rendering; Timing constraints: All signals must include a timestamp so the rendering engine can determine data freshness; For high-frequency changing signals (such as vehicle speed, turn signals), differential encoding or sending only when changing is recommended to reduce bandwidth; however, ProtoBuf itself does not support sending a flag when changing, which can be implemented at the SOME / IP level through service design (using events instead of field updates). This service uses event-based publishing, i.e., updates are only sent when the signal changes, but the timestamp and seq must be incremented. Serialization and communication protocol description: All the above messages are defined using ProtoBuf 3 (proto3) syntax and follow these conventions: Field numbers (= 1, = 2, etc.) cannot be changed once assigned to maintain forward / backward compatibility. Numeric types use float or double to represent physical quantities, and integers are used for enumeration and bitmasks. Enumerated values ​​are recommended to be explicitly defined (e.g., using `enum`), but for simplicity, integers are used in the above definition; in actual code, they should be defined as enumeration types. Transmission process: The service provider (perception, planning, vehicle bus modules) fills the data into the corresponding ProtoBuf message structure. The `SerializeToString()` or `SerializeToArray()` method of the ProtoBuf library is called to serialize the message into a binary byte stream. The binary payload is encapsulated in the payload field of the SOME / IP protocol and sent to the subscriber via the SOME / IP Notification event. Upon receiving the SOME / IP message, the subscriber (service communication proxy module) extracts the payload, calls the `ParseFromString()` method of ProtoBuf to deserialize it, restoring it to a structured object for use by the upper layer. Bandwidth optimization: For high-frequency data (such as a list of trajectory points), the `packed` option (for repetitive numeric types) can be considered for further compression, but ProtoBuf is already quite efficient by default and usually requires no additional configuration. Figure 3 As shown, step 2 includes: Step 21: Based on the SOME / IP Service Discovery mechanism, automatically discover and subscribe to various rendering data services provided by the data source module; This step utilizes SOME / IP's Service Discovery mechanism to automatically discover and subscribe to various rendering data services provided by the backend. Subscription rules can be preset by the system or dynamically adjusted based on the current driving mode (for example, the planned trajectory service can be omitted when navigation is not enabled).

[0024] Step 22: After receiving the SOME / IP protocol packet containing the rendering data service, extract the payload data encoded by ProtoBuf and deserialize it into a structured object for use by the upper-layer rendering module. The structured objects are C++, Java objects, etc.

[0025] Step 23: Monitor the link status through the SOME / IP communication mechanism, handle abnormal situations, and ensure the real-time performance and reliability of the rendering data; The abnormal situations mentioned include data processing timeouts, reconnection, and other anomalies.

[0026] Step 24: Provide the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provide the raw or pre-processed data to the state management core module for state updates.

[0027] The state update refers to changing the system state; for example, a low perception confidence level may trigger a state degradation.

[0028] Step 3: The interaction and configuration management module receives user input from the touch screen, physical buttons, and voice commands, and sends it to the status management core module. The interaction and configuration management module receives input from the touchscreen, physical buttons, and voice commands, converting user intentions into system instructions such as "activate intelligent driving," "switch theme," and "rotate view." This module can also switch themes, supporting preset themes such as day, night, and morning, and allows users to customize themes, with theme parameters (such as brightness, saturation, and element visibility) saved as configuration files. This module is also used for view control, supporting manual rotation, zoom, and panning of the observation view, and allowing users to save specific view and theme combinations as "scenes" (such as "intersection overhead view + night mode") for one-click recall. Furthermore, user operations may affect the system state; for example, pressing the activation button sends a "request activation" signal to the state management core, where the state machine, combined with other conditions, ultimately decides whether to switch states. This will be detailed later.

[0029] Step 4: The state management core module maintains the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates a standardized event code and sends it to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. Step 5: Based on the data from the service communication proxy module and the state management core module, the multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer, and the image synthesizer. Step 6: The multimodal interaction execution module receives the event code from the state management core and transforms the state change event into multi-channel feedback that the driver can perceive based on the event code.

[0030] This invention provides a method and system for rendering driving states and human-computer interaction. This technical solution uses the core state of the intelligent driving system as a unified driving source, performing hierarchical, semantically related, and dynamically responsive visual rendering of the vehicle model, environmental scenes, and decision-making planning, ensuring clear information presentation structure, highlighting key points, and real-time accuracy. The solution supports multi-channel intelligent collaboration and adaptive distribution, including visual, auditory (voice), and text (pop-up windows), automatically optimizing information transmission methods based on event urgency and driving context to reduce interference and enhance prompt effectiveness. The solution also provides an efficient, scalable, and service-oriented data communication and processing framework to ensure low-latency, high-reliability transmission of massive amounts of perception and planning data within the vehicle network, supporting complex real-time rendering at the front end.

[0031] Example 3: This invention provides a method for rendering driving status and human-computer interaction, such as... Figure 1 As shown, the method includes: Step 1: The data source module generates real-time driving environment and vehicle status data based on sensor data, positioning data, route planning data, and vehicle physical status signals, and sends it to the service-oriented communication proxy module. Step 2: Receive the data generated by the data source module, parse it, provide the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provide the raw or pre-processed data to the state management core module for state machine updates; Step 3: The interaction and configuration management module receives user input from the touch screen, physical buttons, and voice commands, and sends it to the status management core module. Step 4: The state management core module maintains the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates a standardized event code and sends it to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. The state management core module is the control center of the entire system. Its core is a centralized driving state machine. This state machine defines a set of enumerated system states. Based on data from the service communication agent (service data from the perception module, planning module, positioning module, and vehicle bus) and user input (data from the interaction and configuration management module), it maintains the system's high-level state in real time and drives all downstream modules to work collaboratively. It outputs the current system state and standardized event codes to directly drive style changes in L1, L2, and L3 layers and trigger multi-channel outputs (visual / voice / pop-up windows). For example... Figure 4As shown, step 4 includes: Step 41, define the system status, including: define OFF as the intelligent driving function is off; define STANDBY as the intelligent driving function is in standby mode, waiting to be activated; define ACTIVE_NORMAL as the intelligent driving function is normally activated; define ACTIVE_DEGRADED as some functions are limited due to sensor performance degradation, etc.; define TAKEOVER_REQUEST as the system requests the driver to take over immediately; define FAILURE as a system malfunction. The state machine described above is defined by a finite set of states, input events, and transition rules to ensure the determinism of system behavior, as shown in Tables 1, 2, and 3. Event codes are mapped one-to-one with state changes, and some events can also be directly triggered by specific input events (e.g., DRIVER_OVERRIDE can generate EVENT_DRIVER_TAKEOVER independently of state changes).

[0032]

[0033] Table 1 State Definition Table

[0034] Table 2 Input Event Definitions

[0035] Table 3 State Transition Rules Step 42: Using data from the service communication agent module and user input, perform a system state transition when a state transition trigger occurs; The triggering factors include: data quality of the perception, positioning, and planning modules (such as object tracking loss, positioning deviation exceeding limits), decisions output by the planning module (such as inability to generate a safe trajectory), user operations (such as pressing the activation / cancel button), and vehicle signals (such as driver intervention in braking).

[0036] Step 43: Continuously update the system status to ensure that the status reflects the real system situation. When the status changes, generate standardized event codes according to the definition and send them to the multimodal interaction execution module. When the state changes, a standardized event code (such as EVENT_ACTIVATION_SUCCESS, EVENT_FCW_URGENT) is generated and sent to the multimodal interaction execution module. Furthermore, the core state management module clearly defines its responsibilities, handling only high-order state logic. This avoids over-coupling of the underlying algorithms, ensures the accuracy of the system state and the timeliness of the response, and improves the state machine transition logic. In other words, this module can maintain the system's high-order states in real time.

[0037] Step 44: Broadcast the current status to the multi-layer rendering engine module so that each rendering layer can adjust its visual style synchronously.

[0038] Step 5: Based on the data from the service communication proxy module and the state management core module, the multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer, and the image synthesizer. Step 6: The multimodal interaction execution module receives the event code from the state management core and transforms the state change event into multi-channel feedback that the driver can perceive based on the event code.

[0039] This invention provides a method and system for rendering driving states and human-computer interaction. This technical solution uses the core state of the intelligent driving system as a unified driving source, performing hierarchical, semantically related, and dynamically responsive visual rendering of the vehicle model, environmental scenes, and decision-making planning, ensuring clear information presentation structure, highlighting key points, and real-time accuracy. The solution supports multi-channel intelligent collaboration and adaptive distribution, including visual, auditory (voice), and text (pop-up windows), automatically optimizing information transmission methods based on event urgency and driving context to reduce interference and enhance prompt effectiveness. The solution also provides an efficient, scalable, and service-oriented data communication and processing framework to ensure low-latency, high-reliability transmission of massive amounts of perception and planning data within the vehicle network, supporting complex real-time rendering at the front end.

[0040] Example 4: This invention provides a method for rendering driving status and human-computer interaction, such as... Figure 1 As shown, the method includes: Step 1: The data source module generates real-time driving environment and vehicle status data based on sensor data, positioning data, route planning data, and vehicle physical status signals, and sends it to the service-oriented communication proxy module. Step 2: Receive the data generated by the data source module, parse it, provide the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provide the raw or pre-processed data to the state management core module for state machine updates; Step 3: The interaction and configuration management module receives user input from the touch screen, physical buttons, and voice commands, and sends it to the status management core module. Step 4: The state management core module maintains the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates a standardized event code and sends it to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. Step 5: Based on the data from the service communication proxy module and the state management core module, the multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer, and the image synthesizer. The multi-layer rendering engine module consists of three logically independent rendering layers and a compositer. Each layer receives current state instructions from the state management core in real time and adjusts the rendering content and style accordingly. Figure 5 As shown, step 5 includes: Step 51: The vehicle state rendering layer renders the vehicle 3D / 2D model, door opening and closing status, headlight effects, instrument information and coupling status based on the vehicle body physical state signals. The data source for the vehicle rendering layer (L1) is the vehicle's physical state signals obtained from the vehicle bus, such as vehicle speed, gear position, and headlights. The rendered content includes: a detailed 3D / 2D vehicle model that displays door open / closed status; headlight effects: turn signals (left / right), brake lights, hazard lights, and high / low beam headlights; instrument panel information: vehicle speedometer and gear position indicator; and state coupling: changing the outline color of the vehicle model (e.g., blue when active, flashing red when a takeover request is made) and the color and flashing mode of the intelligent driving icon based on the state machine state.

[0041] Step 52: The environmental perception rendering layer renders static road elements, dynamic traffic participants, and state couplings based on sensor data and positioning data. The data sources for the environmental perception layer (L2) are data provided by the perception module and the localization module, such as traffic participants, vehicle location, and lightweight map data. The rendered content includes: static road elements: lane lines, stop lines, pedestrian crossings, road arrows, and road boundaries; dynamic traffic participants: light trucks, heavy trucks, adults, two-wheeled electric vehicles (with passengers), police cars, repair vehicles, warning triangles, etc.; state coupling: in the TAKEOVER_REQUEST state, red warning boxes are automatically added to obstacles that may pose a risk (such as vehicles braking suddenly in front).

[0042] Step 53: The decision planning rendering layer renders the planned route line, the target highlighting for following / lane changing, the virtual area prompts, the animation effects and semantic associations based on the route planning data and the system status. The data source for the Decision Planning Rendering Layer (L3) is data provided by the planning module, such as the planned trajectory, target information, and system status. Rendering content includes: planned path lines: dynamic lines representing the vehicle trajectory for the next few seconds; target highlighting for following / lane changing: a semi-transparent blue block (ACTIVE_NORMAL) or a flashing red block (TAKEOVER_REQUEST) is overlaid on the current target vehicle; virtual area prompts: such as "invisible walls" for parking at intersections, and road background rendering (indicating drivable / non-drivable areas); animation effects: L3 startup effects, path generation flow effects, and lane deceleration flow effects; semantic association rendering: the virtual areas of the L3 layer must be associated with semantic elements of the L2 layer. For example, the rendering of the "invisible wall for parking at intersections" must simultaneously meet three conditions: the L2 layer recognizes the stop line, the location information confirms the vehicle is in the intersection area, and the planning outputs a parking decision. All animation effects in the L3 layer (startup effects, path generation flow, lane deceleration flow, etc.) adopt a refresh strategy bound to the display device's vertical synchronization (VSync) to ensure smooth animation without tearing or stuttering. Implementation: The rendering engine drives animation updates by listening to the VSync signal of the display device. The animation timer calculates inter-frame interpolation based on a precise system clock (microsecond level) to ensure time consistency. Path generation flow refers to the animation where the path line spreads forward from the vehicle's position in the form of flowing light spots or ripples when the planned trajectory is updated, visually indicating that the system is "planning" or "the path has been updated." Flow rate coefficient (α): Definition: The flow rate coefficient α controls the speed attenuation factor of the flowing light spots along the path, and its value is between 0.8 and 0.95. Calculation formula: v_flow = v_vehicle × α, where v_vehicle is the vehicle's current speed (m / s). Typical value: α = 0.85, making the flow speed slightly lower than the vehicle speed, creating the visual effect of "the path being chased by the vehicle"; at low speeds (<5m / s), α automatically increases to 0.95 to avoid animation stagnation. Dynamic adjustment: When the system is in the ACTIVE_DEGRADED state, α decreases to 0.5 to indicate the degraded state with a smoother flow. Anti-flicker filter: To avoid path line flickering caused by trajectory point jitter, a first-order low-pass filter is applied to the position input of the path points. Cutoff frequency: fc = 12Hz (typical value, range 10-15Hz), filtering out high-frequency jitter while retaining trajectory updates below 10Hz (the planning module usually releases at 10Hz). Filter transfer function: y[n] = k × x[n] + (1-k) × y[n-1], where k = 2π × fc × Ts, and Ts is the sampling period (consistent with the planning update period). This filter effectively smooths out small jitters at the endpoints of the path line, ensuring stable and smooth flow animation. The deceleration flow effect refers to the visual effect of the lane lines on both sides flowing rapidly backward when the vehicle decelerates or there is a risk ahead, simulating a "sense of speed" to warn the driver.Flow speed mapping: The flow speed v_flow_lines is related to the vehicle deceleration a_decel and risk level: v_flow_lines = v_base + β × |a_decel|, where v_base is the base flow speed, taken as 5 m / s (the minimum speed for clear visual perception), and β is the deceleration gain coefficient, typically β = 2.0 (i.e., for every 1 m / s² increase in deceleration, the flow speed increases by 2 m / s). When a_decel > 2 m / s² or TAKEOVER_REQUEST is triggered, the flow speed is directly set to the maximum value of 30 m / s, and the flickering effect is enabled. Flow speed coefficient: Here, the flow speed coefficient is used to control the repetition period of the flow texture. Set λ = 0.9, it represents the ratio of texture repetition length to vehicle speed, ensuring visual continuity of the flow. Anti-flicker filter: Shares the same low-pass filter (fc = 12 Hz) as the path generation flow to smooth lane line jitter. In addition, a first-order hysteresis filter is applied to the flow velocity itself to avoid abrupt changes in flow velocity caused by sudden deceleration. The cutoff frequency is fc_speed = 5Hz, making the flow velocity changes more consistent with the inertia of human visual perception. When the system switches from STANDBY to ACTIVE_NORMAL, the "L3 startup effect" is triggered, which is usually manifested as a blue halo appearing around the vehicle, spreading outwards and disappearing, while the path line extends from the vehicle to the distance in a "growth" animation. Key parameters: Diffusion speed: The halo diffusion speed is constant at 20m / s, fading out after 0.5 seconds. Path growth speed: The path line starts from the front of the vehicle and is drawn forward at a speed of v_grow = 30m / s. The total growth time depends on the path length (usually not exceeding 0.5 seconds). Transparency curve: An exponential decay of f(t) = exp(-5t) is used to ensure that the startup effect does not interfere with subsequent normal display. Refresh rate: Forced to 60FPS to ensure smooth, jagged edges in the animation. Anti-flicker: During the activation of the animation, low-pass filtering at all path points is paused (to prevent the growth process from being smoothed), but zero-orderhold is enabled to ensure smooth updates per frame. GPU resource reservation: The system reserves at least 30% of the GPU rendering capacity for L3 layer animations to ensure that the animations can still maintain a minimum of 30 FPS under high load. Dynamic degradation strategy: When a rendering latency exceeding 16ms (a frame period of 60 FPS) is detected, the complexity of some non-critical animations (such as deceleration flows on both sides) is automatically reduced, prioritizing the real-time performance of path lines and target highlighting. Anti-flicker verification: Trajectory data is collected from real vehicles to calibrate the filter parameters, ensuring that the path line jitter is less than 0.1 meters (visually imperceptible) under bumpy road conditions or sensor noise.

[0043] Step 54: The image compositer overlays and combines the outputs of the three rendering layers in a preset order, and adjusts the overall tone, brightness, and contrast according to the current theme to finally generate a graphic image that the driver can directly perceive and output it to the display screen.

[0044] The system receives rendering outputs from layers L1, L2, and L3, and overlays and composites them according to a preset Z-order (vehicle layer at the top, decision layer next, environment layer at the bottom). It also adjusts the overall tone, brightness, and contrast based on the current theme (day / night) before finally outputting to the display screen. The basic Z-order principles are: Environment at the bottom: Layer L2, as the foundation of the scene, is placed at the bottom. Intent centered: Layer L3, representing system decisions, is placed above L2 and below L1, ensuring that paths and targets are not obscured by the vehicle and are clearly visible. Vehicle at the front: Layer L1, representing the vehicle's own state, is placed at the top (except for emergency coverage), conforming to the "first-person" visual habit. The dynamic Z-order principle dynamically adjusts priority: Emergency events > Regular events; Dynamic elements > Static elements (when of equal importance); Near objects > Distant objects (within Layer L2); User-focused targets (e.g., following targets) > Ordinary targets; Information above the model: Instrument panel information, pop-ups, and other text-based elements are placed above the vehicle model to ensure readability. To ensure a natural, edge-free image after overlay, the image compositor employs the following techniques for transparency blending and anti-aliasing: Alpha Blending: All semi-transparent elements (such as invisible walls, target overlay boxes, and motion particles) use premultiplied alpha blending to ensure smooth edges without dark borders. Depth Testing: Depth testing is enabled, but Z-order priority is higher than depth value to ensure overlay is based on logical hierarchy rather than actual distance. Anti-aliasing: 4x multisampling anti-aliasing (MSAA) is enabled for line elements (lane lines, path lines) in L2 and L3 layers to avoid stepped jagged edges. Transparency Ordering: For multiple semi-transparent elements within the same Z-order range, they are rendered in backward order to ensure correct blending (from far to near). Emergency Overlay Top Layer: Full-screen effects of emergency events are placed at the top layer to ensure the driver perceives them immediately. Through the above Z-order rules, the image compositor ensures that: in any scene, critical information (such as takeover request warnings) is always at the top visual layer. The vehicle model is not obscured by environmental elements or path lines, maintaining a "first-person" realism. The path lines are always clearly visible and will not be disturbed by the vehicle's dynamic effects (such as flashing turn signals). The pop-up window and instrument panel information do not overlap, and the pop-up window will not obscure important driving information for an extended period.

[0045] Step 6: The multimodal interaction execution module receives the event code from the state management core and transforms the state change event into multi-channel feedback that the driver can perceive based on the event code.

[0046] The multimodal interaction execution module is responsible for converting state change events into multi-channel feedback (visual, auditory, and text) that is perceptible to the driver, and intelligently deciding the combination of feedback based on the context. To achieve efficient transmission, parsing, and priority decision-making for events, this invention defines a standardized 32-bit event encoding format. This encoding is embedded in the state change event, generated by the state management core module, and passed to the multimodal interaction execution module to drive multi-channel output such as visual, voice, and pop-up windows, as shown in Table 4. Status Code (8 bits): The status code is used for quick filtering or classification of events, usually corresponding to the state of the driving state machine, and can also be extended to identify the event source or type. Event ID (16 bits): The event ID uniquely identifies a specific interaction event, ranging from 0x0000 to 0xFFFF. The high byte can be further grouped for easier management. Priority (8 bits): A higher priority value indicates greater urgency; the multimodal interaction execution module determines the output channel combination based on the priority.

[0047]

[0048] Table 4 Event Coding Structure Definition Table like Figure 6 As shown, step 6 includes: Step 61: Receive event codes from the state management core and make multi-dimensional dynamic decision outputs based on the event codes; Before outputting multi-dimensional dynamic decisions, the system needs to predefine a set of standardized event codes covering all interaction intentions, such as: EVENT_ACTIVATION_SUCCESS (intelligent driving activation successful), EVENT_ACTIVATION_FAIL_SENSOR_BLOCKED (activation failure: sensor occlusion), EVENT_FCW_URGENT (forward collision warning - emergency), and EVENT_LANE_CHANGE_PLANNED (planned lane change). After receiving event codes from the state management core, the system dynamically decides the output method based on the following dimensions: event priority, driving mode, user preference, and context. The specific processing steps are as follows: Extract priority: directly take the lower 8 bits to determine whether the event should immediately trigger the high-priority channel; Match strategy: look up the predefined event handling strategy table based on the event ID (or combined with the status code) to obtain the default output channel combination, voice / text ID, pop-up template ID, etc.; Dynamic adjustment: if the priority is extremely high (e.g., ≥240), the strategy can be temporarily overridden to force full-channel output.

[0049] Recording and Statistics: Event IDs can be used for logging and user behavior analysis. The multi-dimensional dynamic decision outputs include: Step 611: Based on the event code, output the priority level of the event as urgent / important / normal / notice; Step 612: Displays the current driving mode as manual driving mode / autonomous driving mode; Step 613: Issue visual and voice commands based on user preferences and context.

[0050] As can be seen from the above, the event priority is divided into emergency, important, routine, and alert levels; driving mode: manual driving mode vs. autonomous driving mode (in autonomous driving mode, low-priority voice prompts are suppressed and L3 visual expression is enhanced); user preferences and context: such as automatically reducing interface brightness in the night theme and tilting some information to the voice channel; user mute settings.

[0051] Step 62: Send a high-priority visual update command to the multi-layer rendering engine module, play the corresponding TTS voice or pre-recorded audio clip, and display a modal or non-modal text prompt box in a specified area of ​​the screen to explain the status or failure reason in detail.

[0052] The visual update instructions include: requiring the L3 layer path to blink, popping up a full-screen pop-up window, etc.

[0053] The following example, using a typical scenario, illustrates the overall workflow.

[0054] Scenario A: The user activates intelligent driving and follows the vehicle normally. User Input: The user issues the "Activate Intelligent Driving" command via touchscreen or steering wheel buttons. Interaction and Configuration Management Module: Receives the command and forwards it to the State Management Core Module. State Management Core Module: Based on the current vehicle status (e.g., speed), perception data (whether there is a target ahead), etc., decides to switch the state from STANDBY to ACTIVE_NORMAL; generates the event code EVENT_ACTIVATION_SUCCESS and sends it to the multimodal interaction execution module; broadcasts the new state to the multi-layer rendering engine module. Multimodal Interaction Execution Module: Query Strategy: The event priority is "Important," and the current mode is autonomous driving, so a combination of "significant visual update + brief voice confirmation" is used; a visual update command (requiring the L3 layer to display the planned path and highlight the target) and a voice command (playing "Intelligent driving activated") are sent. Multi-layer rendering engine module: L1 layer: The vehicle model is highlighted with a blue active outline, and the intelligent driving icon turns blue; L2 layer: Continuously renders lane lines, stop lines, pedestrian crossings, road boundaries, and surrounding traffic participants (such as other vehicles and pedestrians), providing a complete environmental background for the driving scene; L3 layer: Based on the data from Planning_Trajectory_Service, it draws the planned path and overlays a blue semi-transparent frame on the target vehicle ahead; the image compositor combines the three layers of images and outputs them to the display screen. Service communication proxy module: Continuously receives data from SR_Objects_Service and Planning_Trajectory_Service at a frequency of 10Hz and supplies the rendering engine with real-time updates to the image (the target frame moves smoothly as the vehicle in front moves).

[0055] Scenario B: A sudden stop by the vehicle ahead triggers a system takeover request. Service Communication Agent Module: Receives the SR_Objects_Service from the Perception Module, indicating a sudden drop in the vehicle's speed. Simultaneously, the Planning_Trajectory_Service output by the Planning Module shows a high risk for the current trajectory. State Management Core Module: Based on data, determines that driver intervention is required and switches the state from ACTIVE_NORMAL to TAKEOVER_REQUEST; generates an event code EVENT_FCW_URGENT and sends it to the Multimodal Interaction Execution Module, broadcasting the new state to the Multi-Layer Rendering Engine Module. Multimodal Interaction Execution Module: Query strategy: The event priority is "urgent," triggering a full-channel response; sends a "full-screen red flash, forced pop-up" command to the vision actuator; sends an "alarm tone + 'Please take over immediately' TTS" command to the voice broadcaster; sends a "display takeover pop-up" command to the pop-up manager. Multi-layer rendering engine module: L1 layer: Intelligent driving icon flashes red brightly; L2 layer: Adds red flashing warning box to the vehicle in front (key obstacle); L3 layer: The planned path becomes a red flashing line, and the target overlay box becomes red; Image compositor: Overlays a semi-transparent red mask and synthesizes a large pop-up window; Output: The display screen presents a strong visual warning, and the speaker plays an alarm sound and voice, which together remind the driver to take over immediately.

[0056] This invention provides a method and system for rendering driving states and human-computer interaction. This technical solution uses the core state of the intelligent driving system as a unified driving source, performing hierarchical, semantically related, and dynamically responsive visual rendering of the vehicle model, environmental scenes, and decision-making planning, ensuring clear information presentation structure, highlighting key points, and real-time accuracy. The solution supports multi-channel intelligent collaboration and adaptive distribution, including visual, auditory (voice), and text (pop-up windows), automatically optimizing information transmission methods based on event urgency and driving context to reduce interference and enhance prompt effectiveness. The solution also provides an efficient, scalable, and service-oriented data communication and processing framework to ensure low-latency, high-reliability transmission of massive amounts of perception and planning data within the vehicle network, supporting complex real-time rendering at the front end.

[0057] In summary, this invention provides a method and system for driving state rendering and human-computer interaction. This technical solution, through state-driven three-layer dynamic coupling rendering, transforms raw data into a hierarchical and intuitive "driving context map," significantly reducing the driver's cognitive load and enhancing their understanding and trust in the system's behavior. It also enables context-based intelligent multi-channel distribution, ensuring the accuracy, timeliness, and comfort of information transmission, making human-computer communication as natural and smooth as a "co-pilot instructor," significantly improving the user experience. Simultaneously, service-oriented communication and efficient serialization protocols ensure the system's real-time performance and stability under large data volumes. The modular design facilitates functional expansion and integration, adapting to the rapid iteration of intelligent driving technology. Furthermore, clear visualization and intelligent interaction enable drivers to understand system states and risks earlier and more accurately, promoting timely and effective supervision and takeover, fundamentally improving active safety levels.

[0058] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A system for rendering driving state and human-machine interaction, applied to a smart driving vehicle, characterized in that, The system includes: The data source module, located within the intelligent driving domain controller, is used to generate real-time driving environment and vehicle status data. The service-oriented communication proxy module is located in the application layer of the intelligent cockpit domain controller and is connected to the data source module. It is used to receive data generated by the data source module, parse it, provide the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provide raw or pre-processed data to the state management core module for state machine updates. The state management core module is connected to the service communication proxy module. It is used to maintain the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates standardized event codes and sends them to the multimodal interaction execution module, and broadcasts the current state to the multi-layer rendering engine module. The multi-layer rendering engine module is connected to the state management core module and the service communication proxy module respectively. It is used to generate a graphical image that the driver can directly perceive based on the data from the service communication proxy module and the state management core module, through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer and the image synthesizer. The multimodal interaction execution module, connected to the state management core module, is used to receive event codes from the state management core and convert state change events into multi-channel feedback that the driver can perceive based on the event codes. The interaction and configuration management module, connected to the state management core module, is used to receive user input from the touch screen, physical buttons, and voice commands.

2. The system for driving state rendering and human-machine interaction according to claim 1, characterized in that, The data source module includes: The perception submodule is used to fuse sensor data from cameras, millimeter-wave radar, and lidar to identify and track traffic participants and road structures, and generate a list of environmental objects. The positioning submodule is used to integrate GPS, IMU, and high-precision map matching to output the vehicle's precise location, attitude, and lane information; The planning submodule is used to generate local trajectories and driving intentions, plan path point sequences and target information based on the global path, real-time environment and decision-making algorithms; The vehicle bus is used to acquire vehicle physical status signals, including vehicle speed, gear position, door status, and headlight status.

3. A method of employing the system of any of claims 1-2 for rendering a driving state with human-machine interaction, characterized in that, The method includes: Step 1: The data source module generates real-time driving environment and vehicle status data based on sensor data, positioning data, route planning data, and vehicle physical status signals, and sends it to the service-oriented communication proxy module. Step 2: The service-oriented communication proxy module receives the data generated by the data source module, parses it, provides the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provides the raw or pre-processed data to the state management core module for state machine updates. Step 3: The interaction and configuration management module receives user input from the touch screen, physical buttons, and voice commands, and sends it to the status management core module. Step 4: The state management core module maintains the high-level state of the system in real time based on the data from the service communication proxy module and user input. When the state changes, it generates a standardized event code and sends it to the multimodal interaction execution module, broadcasting the current state to the multi-layer rendering engine module. Step 5: Based on the data from the service communication proxy module and the state management core module, the multi-layer rendering engine module generates a graphical image that the driver can directly perceive through the vehicle state rendering layer, the environment perception rendering layer, the decision planning rendering layer, and the image synthesizer. Step 6: The multimodal interaction execution module receives event codes from the state management core module and transforms the state change events into multi-channel feedback that the driver can perceive based on the event codes.

4. The method of claim 3, wherein, Step 1 includes: Step 11: The perception submodule fuses sensor data from cameras, millimeter-wave radar, and lidar to identify and track traffic participants and road structures, and generates a list of environmental objects. Step 12: The positioning submodule integrates GPS, IMU, and high-precision map matching to output the vehicle's precise location, attitude, and lane information; Step 13: The planning submodule generates local trajectories and driving intentions, planning path point sequences and target information based on the global path, real-time environment and decision algorithm; Step 14: The vehicle bus acquires vehicle physical status signals, including vehicle speed, gear position, door status, and headlight status. Step 15: Encapsulate the obtained data into an independent rendering data service using the SOME / IP protocol, and send it to the service-oriented communication proxy module based on the ProtoBuf encoded data interface.

5. The method of claim 4, wherein, Step 2 includes: Step 21: Based on the SOME / IP Service Discovery mechanism, automatically discover and subscribe to various rendering data services provided by the data source module; Step 22: After receiving the SOME / IP protocol packet containing the rendering data service, extract the payload data encoded by ProtoBuf and deserialize it into a structured object for use by the upper-layer rendering module. Step 23: Monitor the link status through the SOME / IP communication mechanism, handle abnormal situations, and ensure the real-time performance and reliability of the rendering data; Step 24: Provide the parsed list of environmental objects, planned trajectory and vehicle signals to the multi-layer rendering engine module, and provide the raw or pre-processed data to the state management core module for state updates.

6. The method of claim 3, wherein, Step 4 includes: Step 41, define the system status, including: define OFF as the intelligent driving function is off; define STANDBY as the intelligent driving function is in standby mode, waiting to be activated; define ACTIVE_NORMAL as the intelligent driving function is normally activated; define ACTIVE_DEGRADED as some functions are limited due to sensor performance degradation, etc.; define TAKEOVER_REQUEST as the system requests the driver to take over immediately; define FAILURE as a system malfunction. Step 42: Using data from the service communication agent module and user input, perform a system state transition when a state transition trigger occurs; Step 43: Continuously update the system status to ensure that the status reflects the real system situation. When the status changes, generate standardized event codes according to the definition and send them to the multimodal interaction execution module. Step 44: Broadcast the current status to the multi-layer rendering engine module so that each rendering layer can adjust its visual style synchronously.

7. The method of claim 6, wherein, The triggering factors include: the data quality of the perception, positioning, and planning modules, the decisions output by the planning module, user operations, and vehicle signals.

8. The method of claim 3, wherein, Step 5 includes: Step 51: The vehicle state rendering layer renders the vehicle 3D / 2D model, door opening and closing status, headlight effects, instrument information and coupling status based on the vehicle body physical state signals. Step 52: The environmental perception rendering layer renders static road elements, dynamic traffic participants, and state couplings based on sensor data and positioning data. Step 53: The decision planning rendering layer renders the planned route line, the target highlighting for following / lane changing, the virtual area prompts, the animation effects and semantic associations based on the route planning data and the system status. Step 54: The image compositer overlays and combines the outputs of the three rendering layers in a preset order, and adjusts the overall tone, brightness, and contrast according to the current theme to finally generate a graphic image that the driver can directly perceive and output it to the display screen.

9. The method of claim 3, wherein, Step 6 includes: Step 61: Receive event codes from the state management core and make multi-dimensional dynamic decision outputs based on the event codes; Step 62: Send a high-priority visual update command to the multi-layer rendering engine module, play the corresponding TTS voice or pre-recorded audio clip, and display a modal or non-modal text prompt box in a specified area of ​​the screen to explain the status or failure reason in detail.

10. The method of claim 9, wherein, The multi-dimensional dynamic decision output includes: Step 611: Based on the event code, output the priority level of the event as urgent / important / normal / notice; Step 612: Displays the current driving mode as manual driving mode / autonomous driving mode; Step 613: Issue visual and voice commands based on user preferences and context.

Images