Vehicle-mounted graphical interface intelligent agent system based on multi-modal large model and voice interaction

Abstract

The application relates to the field of intelligent cockpit human-computer interaction, and discloses a vehicle-mounted graphical interface intelligent agent system based on a multimodal large model and voice interaction, a vehicle-mounted end subsystem and a cloud end intelligent agent subsystem. The vehicle-mounted end subsystem collects voice streams and screen images, and sends the voice streams and the screen images to the cloud end through a full-duplex long connection channel; the cloud end intelligent agent subsystem jointly reasons voice intentions and interface visual elements by using a multimodal large model, generates an operation action sequence, and returns the operation action sequence as a control instruction stream; the vehicle-mounted end analyzes the instruction and converts the instruction into a bottom layer input event to inject the graphical interface. The system adapts to different physical resolutions by using a virtual normalized coordinate system, guarantees the real-time performance of voice by using a transmission queue priority strategy, and realizes the monitoring and self-recovery of operation abnormalities by introducing a visual feedback closed loop mechanism. The application realizes the automatic control of a general graphical interface based on natural language, and improves the intelligent level and generalization ability of vehicle-mounted interaction.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit human-computer interaction, specifically to an in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction. Background Technology

[0002] With the evolution of intelligent vehicle electronic architecture and cockpit interaction technology, the interaction mode of in-vehicle infotainment systems has gradually shifted from traditional physical button control to a multimodal interaction mode that combines touch, voice, and visual recognition. Currently, most mainstream in-vehicle human-machine interaction systems integrate voice assistant functions, allowing users to complete basic operations such as navigation settings, multimedia playback, and phone calls via voice commands. These voice assistants are mostly based on predefined intent slots and application programming interface (API) calls, relying heavily on deep adaptation by specific application vendors, making it difficult to cover the increasingly complex third-party application ecosystem and non-standardized long-tail task requirements within the in-vehicle system.

[0003] Although some existing research attempts to introduce multimodal models to automate the operation of graphical user interfaces, it still faces many technical bottlenecks when applied to the in-vehicle environment.

[0004] First, the hardware specifications of automotive display components are highly fragmented, with significant differences in screen resolution, aspect ratio, and pixel density across different vehicle models. Existing visual language models or automated control solutions are typically trained or inferred directly based on absolute pixel coordinates, resulting in strong coupling between the model and the display hardware. Without targeted training or fine-tuning, existing solutions struggle to directly adapt to heterogeneous automotive screen hardware, failing to guarantee the accuracy of interface element positioning and operation across different device scenarios.

[0005] Secondly, in-vehicle terminals typically operate in mobile network environments, where network bandwidth and connection stability fluctuate. Achieving multimodal interaction requires the simultaneous transmission of high-frequency voice data streams and high-resolution screen image data streams. Existing interaction architectures mostly employ traditional request-response communication models, lacking a fine-grained management mechanism for prioritizing multimodal data stream transmission. When network bandwidth is limited, image data transmission often blocks the voice command channel, leading to increased command response latency and failing to meet the stringent requirements of real-time operation and seamless interaction in in-vehicle scenarios.

[0006] Furthermore, most existing in-vehicle voice control systems employ open-loop control logic. The system is only responsible for issuing commands and lacks a visual perception and feedback verification mechanism for the execution results. When unexpected dynamic changes occur in the in-vehicle graphical interface, such as advertising pop-ups obscuring the view, page loading delays, or clicks not responding due to system lag, the system cannot perceive the deviation between the current interface state and the expected goal. It lacks the ability to automatically correct errors or dynamically adjust the operation path, resulting in a low success rate for complex task chains and affecting the interactive experience and safety during driving. Summary of the Invention

[0007] This invention provides an intelligent agent system for in-vehicle graphical interfaces based on a multimodal large model and voice interaction, aiming to solve the technical problems of existing in-vehicle voice assistants, such as their inability to understand complex graphical interface contexts, their inability to perform continuous operations, and their weak cross-application generalization capabilities. This invention utilizes a distributed architecture with edge-cloud collaboration, leveraging a multimodal large model to perform joint semantic analysis of voice commands and dynamic screen images in the in-vehicle environment, thereby achieving intelligent perception and automated control of the in-vehicle graphical user interface.

[0008] This invention provides a system architecture that runs in a distributed computing environment. The architecture includes an in-vehicle terminal subsystem deployed on an in-vehicle terminal device and a cloud-based intelligent agent subsystem deployed on a cloud server.

[0009] The in-vehicle terminal subsystem is responsible for real-time data acquisition and low-level execution, while the cloud-based intelligent agent subsystem is responsible for high-performance inference and decision-making. Specifically, the in-vehicle terminal subsystem integrates a voice stream acquisition unit and a visual stream perception unit, capable of simultaneously capturing the digital audio stream within the vehicle and the graphical interface image data currently displayed by the in-vehicle display components. The communication transmission unit encapsulates the aforementioned audio and image data into a multimodal input stream and sends it to the cloud via a network connection channel. Upon receiving the input stream, the cloud-based intelligent agent subsystem performs deep inference using a pre-trained multimodal large-scale model, identifies user intent, parses control elements in the interface, generates a control command stream containing specific operation sequences, and returns it to the vehicle. The operation execution simulation unit parses this command stream, converts it into input events at the operating system's underlying level, and injects it into the specified coordinates of the graphical user interface, thereby completing interactive operations such as clicking, swiping, or text input.

[0010] To address the mismatch between the output of the multimodal large model and the hardware specifications of different in-vehicle screens, this invention proposes a coordinate mapping and normalization mechanism. The cloud-based intelligent agent subsystem constructs a virtual normalized coordinate system, uniformly representing the control positions or operation target points identified by the multimodal large model as standardized floating-point relative coordinates. This relative coordinate system eliminates differences in physical resolution. The coordinate mapping subunit of the in-vehicle terminal subsystem reads the horizontal and vertical physical pixel counts of the in-vehicle display component. By multiplying the standardized floating-point relative coordinates by the horizontal and vertical physical pixel counts respectively, it calculates the target physical horizontal and vertical coordinates adapted to the current screen, ensuring the accurate execution of operation commands on screens of different vehicle models.

[0011] To ensure real-time voice interaction and adapt to fluctuations in the in-vehicle network environment, this invention employs a full-duplex connection channel based on the transmission control protocol, supporting asynchronous parallel data transmission and reception. The system introduces a transmission queue management mechanism, setting the highest transmission priority for voice data packets to ensure the continuity of voice recognition. Simultaneously, for large screenshot data packets, the system adopts a frame dropping strategy, proactively discarding old timestamp image data accumulated in the transmission queue when network bandwidth is limited, retaining only the most recently generated image data, thereby reducing the overall end-to-end latency of the system.

[0012] Furthermore, to improve the system's robustness and response speed, this embodiment of the invention adopts an edge-cloud hybrid computing architecture. The system integrates a task complexity assessment unit and a lightweight edge inference unit. A classifier determines the complexity of user voice commands. For low-complexity commands such as volume adjustment and song switching, the system directly uses a lightweight model on the local processor to generate control signals and call the system interface, without going through the cloud, thus achieving zero-latency response and basic functional availability even in offline environments. For high-complexity commands involving complex semantic understanding or cross-application navigation, the commands are routed to a large cloud model for processing.

[0013] This invention also introduces a visual feedback closed-loop control mechanism. An anomaly monitoring unit is added to the vehicle-mounted subsystem, which automatically triggers the acquisition of verification image data containing the execution result after an operation is performed and feeds it back to the cloud. The cloud-based intelligent agent subsystem compares the verification image with the expected state characteristics. If it finds that the interface has not changed as expected or that an interfering pop-up appears, it automatically generates error correction or retry instructions, thus realizing the self-healing capability of the intelligent agent.

[0014] In processing multimodal data, the system achieves information fusion through a multimodal data alignment unit. The visual encoder segments the screenshot into image patches and maps them to visual embedding vectors. The modal alignment projection layer maps these visual embedding vectors to the same dimensional space as the text embedding vectors. Subsequently, the system concatenates the mapped visual embedding vector sequence with the text embedding vector sequence obtained from speech conversion to form a multimodal mixed input sequence, enabling large models to understand both visual and linguistic information simultaneously.

[0015] Furthermore, this invention supports multi-zone and multi-screen collaborative control in smart cockpit scenarios. The voice stream acquisition unit uses multi-zone sound source localization technology to calculate the location identifier of the command issuer. The cloud model parses the user's referential intent (such as turning off my screen) based on this identifier, determines the target display component, and assigns the corresponding component's position and operation. The operation execution simulation unit then injects the input event into the corresponding driver, passenger, or rear-seat display component drive pipeline, realizing intelligent management of the entire vehicle's interaction.

[0016] In summary, this invention constructs a complete closed loop encompassing perception, decision-making, execution, and feedback, and utilizes technologies such as coordinate normalization, full-duplex streaming transmission, and end-to-cloud collaboration to achieve automated control of a universal graphical interface based on natural language in the vehicle environment, thereby improving the interaction efficiency and user experience of the in-vehicle infotainment system.

[0017] This invention provides an in-vehicle graphical interface intelligent agent system based on a multimodal large model and voice interaction. It has the following beneficial effects:

[0018] 1. This invention utilizes coordinate mapping and normalization mechanisms to construct a virtual normalized coordinate system in the cloud to output relative coordinates, and then restores the target's physical coordinates on the in-vehicle device by combining the physical resolution. This design decouples the model inference space from the physical display space, enabling multimodal large models to directly adapt to in-vehicle display components with different resolutions and aspect ratios without requiring model retraining or fine-tuning for specific hardware specifications, thus improving the system's generalization ability and compatibility in heterogeneous in-vehicle hardware environments.

[0019] 2. This invention employs a full-duplex long-distance connection channel and transmission queue management strategy based on the transmission control protocol, supporting asynchronous parallel transmission of multimodal data and assigning high priority to audio data. Simultaneously, it executes logic to discard outdated image frames when bandwidth is limited. This streaming interaction and congestion control mechanism effectively reduces end-to-end communication latency, ensuring the continuity of voice interaction while guaranteeing the real-time acquisition of visual information by the cloud model. It solves the problems of delayed command response and data backlog in fluctuating environments of vehicular mobile networks.

[0020] 3. This invention establishes a closed-loop control and anomaly self-healing system based on visual feedback. By acquiring verification images after an action is executed and comparing them with expected state characteristics, it can dynamically detect abnormal situations such as interference pop-ups, page loading timeouts, or unresponsive clicks. The cloud-based intelligent agent then generates error correction or retry instructions in real time, correcting the high task execution interruption rate problem in the traditional voice assistant open-loop control mode and ensuring the integrity and stability of automated task flow execution in complex dynamic graphical interfaces. Attached Figure Description

[0021] Figure 1 This is a block diagram of the logical architecture of the edge-cloud collaborative system of the present invention; Figure 2 This is a schematic diagram illustrating the coordinate mapping and normalization processing principle of the present invention; Figure 3 This is a flowchart of the multimodal data parallel transmission and queue management based on full-duplex channels of the present invention; Figure 4 This is a flowchart of the anomaly monitoring and closed-loop self-healing process of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see the appendix Figure 1 This invention provides an in-vehicle graphical interface intelligent agent system based on a multimodal large model and voice interaction. The system operates in a distributed computing environment including an in-vehicle terminal device and a cloud server. The in-vehicle terminal device establishes a data connection with the cloud server via a wireless communication network.

[0024] The vehicle-mounted terminal equipment includes a central processing unit (CPU), a memory, a display component, an audio acquisition component, an audio playback component, and a vehicle communication component. The CPU establishes electrical or bus connections with the memory, display component, audio acquisition component, audio playback component, and vehicle communication component, respectively.

[0025] The memory is configured to store computer program instructions and the operating data of the vehicle operating system. The central processing unit is configured to execute the computer program instructions stored in the memory to realize the data acquisition, data transmission, instruction reception, and instruction execution functions of the vehicle terminal device.

[0026] The display component is configured to present the graphical user interface of the in-vehicle infotainment system and provide a touch input interface. The screen resolution of the display component is defined as the width value. and height value The data is measured in pixels. The display component is configured to support image data reading operations from the frame buffer to obtain the screen image data at the current moment.

[0027] The audio acquisition component is configured to receive sound wave signals from the external environment and convert them into digital audio signals. The audio acquisition component includes a microphone array located inside the passenger compartment, configured to capture voice commands issued by the driver or passengers.

[0028] The vehicle communication component is configured to establish a communication link between the vehicle terminal equipment and an external network. The vehicle communication component supports fourth-generation (4G), fifth-generation (5G), or wireless local area network (WLAN) communication protocols.

[0029] The cloud server includes a multimodal large model inference processor, server storage, and network communication interfaces. The multimodal large model inference processor, comprising a cluster of graphics processing units (GPUs) or tensor processing units (TMUs), is configured to load and run the multimodal large model. The server storage is configured to store the network parameters, weight files, and contextual history data of the multimodal large model.

[0030] Logically, the system includes an in-vehicle terminal subsystem and a cloud-based intelligent agent subsystem.

[0031] The vehicle-mounted terminal subsystem includes a voice stream acquisition unit, a visual stream perception unit, a communication transmission unit, and an operation execution simulation unit.

[0032] The voice stream acquisition unit is configured to capture the digital audio stream output by the audio acquisition component in real time. The voice stream acquisition unit performs sampling rate conversion and encoding compression on the raw digital audio stream to generate voice data packets that conform to the transmission protocol.

[0033] The visual flow perception unit is configured to acquire image data of the current graphical user interface from the underlying video memory of the display component or the operating system framework layer at preset time intervals or triggered events. The visual flow perception unit is configured to compress and encode the acquired image data to generate a screenshot data packet.

[0034] The communication transmission unit is configured to establish a full-duplex network connection between the vehicle-mounted terminal device and the cloud server. The communication transmission unit encapsulates voice data packets and screenshot data packets into a multimodal input stream and sends it to the cloud server through the full-duplex network connection. Simultaneously, the communication transmission unit is configured to receive control command streams from the cloud server.

[0035] The operation execution simulation unit is configured to receive control commands from a cloud server. It is also configured to parse these commands and convert them into low-level input events that the vehicle operating system can recognize. For example, the operation execution simulation unit can inject click, swipe, or text input events at specified coordinates in the graphical user interface by calling the vehicle operating system's accessibility service interface or debug bridge interface.

[0036] The cloud-based intelligent agent subsystem includes a multimodal data alignment unit, a multimodal large model inference unit, and an instruction serialization generation unit.

[0037] The multimodal data alignment unit is configured to receive multimodal input streams and perform timing synchronization and format preprocessing on speech and image data. The multimodal data alignment unit converts speech data into text sequences and image data into visual feature vectors.

[0038] The multimodal large model inference unit is configured to receive text sequences and visual feature vectors as input. The multimodal large model inference unit utilizes a pre-trained deep neural network model to perform semantic analysis on the user intent expressed by the input text sequence, and combines this with the visual feature vectors to identify and locate control elements in the graphical user interface.

[0039] The input-output relationship of a multimodal large model inference unit is defined as a mapping function. Set the input multimodal context set as follows: ,but .in, This represents the image feature matrix of the graphical user interface at the current moment. This represents the text embedding vector after the user's voice command has been converted. The state vector represents the historical interaction context. The output of the multimodal large model inference unit is a sequence of action actions. Operation sequence It is obtained by calculation using the following formula: ; in, It contains one or more ordered atomic operation instructions. Each atomic operation instruction is defined as a tuple. ,in, Indicates the operation type, with possible values ​​including click, long press, swipe, text input, etc. Indicates the screen coordinates of the location where the operation occurred. ; This indicates, for example, the input text content or the sliding distance parameter.

[0040] The instruction serialization generation unit is configured to generate operation action sequences The data is converted into a structured data format that conforms to the communication protocol. The instruction serialization generation unit then sends the structured data back to the vehicle terminal device via the network communication interface.

[0041] The vehicle-mounted subsystem also includes an anomaly monitoring unit. This unit is configured to monitor the execution results of the operation execution simulation unit. When the operation execution simulation unit completes an atomic operation command, the anomaly monitoring unit triggers the visual flow perception unit to collect new screen image data and feeds this data back to the cloud-based intelligent agent subsystem, forming a closed-loop control circuit.

[0042] The vehicle-mounted end-side subsystem includes a voice stream acquisition unit and a visual stream perception unit, which work in parallel to acquire multimodal input data.

[0043] The voice stream acquisition unit establishes a communication connection with the vehicle audio hardware driver layer. The voice stream acquisition unit is configured to monitor the pulse-code modulation data stream from the microphone array. An integrated voice activity detection subunit is included within the voice stream acquisition unit. This subunit determines whether there is valid human voice input at the current moment by calculating the short-time energy value and zero-crossing rate of the current audio frame.

[0044] The voice activity detection subunit is set with an energy threshold. When continuous detection The short-time energy value of the frame audio data is greater than the energy threshold. When the voice activity detection subunit determines that the current state is the start of voice activity, it triggers the voice stream acquisition unit to start recording audio data. When continuous voice activity is detected... The short-time energy value of the frame audio data is less than the energy threshold. When the voice activity detection subunit determines that the voice has ended, it triggers the voice stream acquisition unit to stop recording.

[0045] The voice stream acquisition unit may also include an echo cancellation and noise reduction processing subunit. This subunit receives the original pulse-code modulation data stream and a reference audio stream. The reference audio stream is the audio signal currently being played by the vehicle's loudspeakers. The echo cancellation and noise reduction processing subunit uses an adaptive filtering algorithm to subtract the signal component corresponding to the reference audio stream from the original pulse-code modulation data stream, thereby eliminating interference from the content played by the vehicle's loudspeakers on voice commands. Simultaneously, the echo cancellation and noise reduction processing subunit uses spectral subtraction to filter out stable background noise, outputting clean voice audio data.

[0046] The speech stream acquisition unit inputs the processed speech audio data into the audio encoder. The audio encoder compresses and encodes the speech audio data using, for example, an advanced audio coding format or an Opus audio coding format, generating speech data packets. Each speech data packet contains timestamp information and a compressed binary audio stream.

[0047] The visual flow perception unit is configured to capture image information from the vehicle's central control screen. For example, the visual flow perception unit can establish a connection with the window manager service or display composition service of the vehicle's operating system. The visual flow perception unit can obtain the current screen display content, for example, by calling the operating system's screen projection interface or directly reading the frame buffer memory area.

[0048] The visual flow perception unit includes a screenshot triggering logic subunit. The screenshot triggering logic subunit is configured to operate based on a status signal from the voice activity detection subunit. According to one embodiment of the invention, the screenshot triggering logic subunit performs screen capture in response to the following two triggering events: (1) When the voice activity detection subunit determines that the current state is the end of voice (i.e. the user's voice command is detected to have ended), the screenshot triggering logic subunit immediately triggers the screen capture operation to obtain the graphical user interface at the current moment as the visual context for understanding the user's intent. (2) After the operation execution simulation unit completes the execution of the previous round of atomic operation instructions, the screenshot triggering logic subunit triggers the screen capture operation again to obtain the interface feedback after the operation is executed. The new screen image data is fed back to the cloud intelligent agent subsystem, which is used by the multimodal large model inference unit to judge the current task status and perform the inference and generation of the next round of actions, thereby forming a closed-loop control loop between the end and the cloud.

[0049] The raw screen images acquired by the visual flow perception unit typically have high resolution and bit depth. The visual flow perception unit includes an image preprocessing subunit. The image preprocessing subunit is configured to perform resizing and format conversion on the raw screen images.

[0050] The image preprocessing subunit reads the width of the original screen image. and height The image preprocessing subunit sets the target width. and target height The image preprocessing subunit employs, for example, bilinear interpolation or bicubic interpolation to scale the original screen image to a target size. The target size is set based on the input resolution limitations of the multimodal large model to reduce bandwidth consumption and improve cloud inference speed. According to one embodiment of the invention, the image preprocessing subunit also converts the scaled image data from the original RGB alpha channel format to an RGB three-channel format, removing the alpha channel data.

[0051] The image preprocessing subunit further includes an image compression encoder. The image compression encoder performs lossy compression on the image data using, for example, the Joint Group of Image Experts (JEP) JPEG format or the Web Image BP (WEBP) format. According to one embodiment of the invention, the image compression encoder is configured with a compression quality factor. The visual flow sensing unit is configured to dynamically adjust the compression quality factor based on the current network bandwidth status. When network bandwidth is below a preset threshold, the compression quality factor is reduced. To reduce data packet size; and to increase the compression quality factor when network bandwidth exceeds a preset threshold. To retain more interface details.

[0052] The visual stream perception unit ultimately outputs a screenshot data packet containing timestamps, image format metadata, and binary image data. The speech data packet output by the speech stream acquisition unit and the screenshot data packet output by the visual stream perception unit are synchronized on the timeline, together forming the basis of the multimodal input data.

[0053] The vehicle-mounted terminal subsystem includes an execution control module, which further includes an instruction parsing unit and a low-level simulation unit.

[0054] The instruction parsing unit is configured to establish a data connection with the communication transmission unit of the vehicle-mounted terminal device. The instruction parsing unit receives structured control instruction data packets from a cloud server. These structured control instruction data packets are encoded using formats such as JavaScript Object Notation (JSON) or Extensible Markup Language (XML). The structured control instruction data packets include operation type fields, location coordinate fields, parameter value fields, and execution timestamp fields.

[0055] The instruction parsing unit deserializes the structured control instruction data packet to extract the operation type, location coordinates, and parameter values. Operation types include, for example, click operations, long press operations, swipe operations, text input operations, and physical button simulation operations.

[0056] The instruction parsing unit includes a coordinate mapping subunit. This subunit is configured to resolve the inconsistency between the normalized coordinates output by the multimodal large model and the actual physical resolution of the in-vehicle central control screen. The position coordinate field output by the multimodal large model is typically a normalized floating-point value relative to the image size or a relative integer value within a preset range.

[0057] The coordinate mapping subunit reads the horizontal physical pixel count of the display component. and vertical physical pixel count Assume the location coordinate field contains normalized x-coordinates. and normalized ordinate ,in and The value range is from 0 to 1. The coordinate mapping sub-unit calculates the target's physical abscissa through multiplication. and target physical coordinates The calculation process is as follows: ; ; The underlying simulation unit is configured to synthesize and distribute human-machine interaction events at the vehicle operating system level based on the parsed operation parameters. The underlying simulation unit can, for example, implement event injection based on the vehicle operating system's Accessibility Service framework or the Android Debug Bridge interface.

[0058] When the operation type is a click operation, the underlying simulation unit constructs a click gesture description object. The click gesture description object defines the touch point's position on the target's physical x-coordinate. and target physical coordinates The press action, after a preset short pause, triggers a release action. The underlying simulation unit calls an event dispatch interface (e.g., an accessibility service dispatch Gesture) to send the click gesture description object to the system event queue.

[0059] When the operation type is a swipe operation, the position coordinate field includes the starting coordinates. and termination coordinates The underlying simulation unit constructs a path object. The path object defines the trajectory of the touch point moving from the starting coordinates to the ending coordinates. The underlying simulation unit sets the swipe duration parameter. Based on the path object and the swipe duration parameter, the underlying simulation unit generates a swipe gesture description object and injects it into the system through the event dispatch interface.

[0060] When the operation type is text input, the instruction parsing unit extracts the text content to be input from the structured control instruction data packet. The underlying simulation unit first performs a click operation to activate the focus of the input box. Subsequently, the underlying simulation unit can, for example, directly fill the text content to be input into the currently focused user interface element through the ACTION-SET-TEXT interface of the accessibility service node. Alternatively, the underlying simulation unit can simulate keyboard key events by sending the key codes corresponding to the character codes one by one to simulate the manual typing process.

[0061] The underlying simulation unit may also include an execution status feedback subunit. This subunit is configured to listen for the return results of the event injection interface. When the event distribution interface returns a success status code, the execution status feedback subunit generates an execution success signal; when the event distribution interface returns a failure status code or throws an exception, the execution status feedback subunit generates an execution failure signal. The execution status feedback subunit transmits the execution result signal to the communication transmission unit for reporting to the cloud server for closed-loop control.

[0062] The embodiments of the present invention employ a full-duplex network connection channel to achieve efficient and real-time data interaction between the vehicle-mounted terminal subsystem and the cloud-based intelligent agent subsystem.

[0063] The communication transmission unit is configured to initiate a network connection request. The communication transmission unit initiates a handshake protocol request to the network communication interface of the cloud server to establish a long-lived connection channel based on the Transmission Control Protocol (TCP). In this embodiment, the communication transmission unit uses the WebSocket protocol or the General Remote Procedure Call (GRPC) protocol based on Hypertext Transfer Protocol version 2 to construct this long-lived connection channel. Once the connection is successfully established, the connection channel remains open until the vehicle terminal device loses power or the network is abnormally interrupted, thus avoiding the network handshake delay caused by frequent connection establishment and termination in the traditional short-connection mode.

[0064] The communication transmission unit defines the format of multimodal input stream data packets for uplink transmission. Uplink transmission refers to the process of sending data from the vehicle-mounted terminal device to the cloud server. Multimodal input stream data packets consist of a header field and a payload data field. The header field includes a message type identifier, a payload length identifier, and a timestamp sequence number. The message type identifier is used to distinguish the type of payload data, including audio data frame types, image data frame types, heartbeat detection types, and control signaling types.

[0065] When the message type identifier indicates an audio data frame type, the payload data field carries a compressed audio segment output by the speech stream acquisition unit. According to an embodiment of the invention, the communication transmission unit is configured to divide a continuous audio stream into multiple audio data frames according to a preset time slice length (e.g., 20 milliseconds or 40 milliseconds) and send them sequentially to the cloud server. This streaming mechanism allows the cloud server to begin speech recognition and semantic understanding before the user's statement has ended, thereby achieving streaming processing.

[0066] When the message type identifier indicates an image data frame type, the payload data field carries the screenshot data output by the visual stream perception unit. To maintain context consistency, the timestamp sequence number contained in the header field of each image data frame is clock-synchronized with the timestamp sequence number of the audio data frames acquired at the same time. The communication transmission unit can also be configured to append vehicle status metadata to the payload data field of the image data frame. The vehicle status metadata includes the current vehicle speed, gear status indicator, and day / night mode indicator. This metadata serves as auxiliary information for the cloud-based multimodal large model to determine the safety of operations and the mode of interface display.

[0067] The communication transmission unit defines the format of control command stream data packets used for downlink transmission. Downlink transmission refers to the process of sending data from a cloud server to an in-vehicle terminal device. Downlink data packets also include header fields and payload data fields. The payload data field of the downlink data packet mainly carries structured operation command data or system status feedback data.

[0068] The communication transmission unit supports asynchronous full-duplex communication mode. This means that while the vehicle-mounted terminal device continuously sends audio and image data frames to the cloud server, it can independently and in parallel receive downlink data packets from the cloud server. For example, when a user is uttering a long instruction, if the cloud-based multimodal big data model has already identified a clear intention to interrupt or correct based on the preceding speech segments, the cloud server can immediately issue a stop instruction or feedback instruction. The communication transmission unit receives the instruction and immediately triggers the operation execution simulation unit or audio playback component to respond, without waiting for the uplink transmission queue to be cleared.

[0069] To maintain the stability of long-lived connections, the communication transmission unit incorporates a heartbeat keep-alive mechanism. The unit sends heartbeat probe packets to the cloud server at fixed intervals. If no heartbeat response packet is received from the cloud server within a preset timeout period, the communication transmission unit determines the current connection is abnormal and triggers automatic reconnection logic. This automatic reconnection logic uses an exponential backoff algorithm to set the retry waiting time to prevent the storm effect during network congestion.

[0070] The communication transmission unit also includes a transmission queue management subunit. This subunit maintains a first-in, first-out (FIFO) transmission queue. For different types of uplink data, the subunit sets a priority strategy. Audio data frames have the highest transmission priority to ensure the continuity of voice interaction; control signaling types have the second highest priority; and image data frames have a normal priority. When network bandwidth is limited, the transmission queue management subunit is configured to actively discard old timestamped image data frames backed up in the transmission queue, retaining only the most recently generated image data frames to ensure optimal real-time visual information received by the cloud server.

[0071] The cloud-based intelligent agent subsystem includes a multimodal large-model inference module, which runs on a high-performance computing server cluster and is configured as the core decision-making hub of the system. Based on the deep learning Transformer architecture, the multimodal large-model inference module possesses the ability to jointly understand and logically reason about visual image data and natural language text data.

[0072] The multimodal large model inference module includes a multimodal input encoding subunit. This subunit is configured to receive a synchronous data stream from the multimodal data alignment unit. The multimodal input encoding subunit includes a visual encoder and a text embedding layer. The visual encoder, employing, for example, a visual Transformer structure or a convolutional neural network structure, is configured to convert the input screenshot data into a sequence of high-dimensional visual feature vectors. The visual encoder segments the screenshot into fixed-size image patches and maps each patch to a corresponding visual embedding vector.

[0073] The multimodal input encoding subunit also includes a modality alignment projection layer. This layer, such as a multilayer perceptron (MLP) or linear projection layer, is configured to map the dimensions of the visual feature vectors to the same dimensional space as the text embedding vectors, thereby achieving semantic alignment between the visual and linguistic modalities. The multimodal input encoding subunit then concatenates the mapped visual feature vector sequence with the text embedding vector sequence corresponding to the user's voice command to form a multimodal mixed input sequence.

[0074] The multimodal large model inference module includes a context construction subunit. This subunit is configured to insert a predefined system prompt (SystemPrompt) at the beginning of the multimodal mixed input sequence. The system prompt contains a strict definition of the graphical user interface operation space of the in-vehicle display component. Within the system prompt, the context construction subunit defines the role settings of the intelligent agent, the set of allowed actions, and the numerical representation of the coordinate system.

[0075] Action sets are defined as a set of standardized application interface call descriptions, such as click (CLICK), double-click (DOUBLE_CLICK), long-press (LONG_PRESS), text input (TYPE), scroll (SCROLL), and navigation back (BACK).

[0076] The numerical representation of a coordinate system can be defined as a normalized relative coordinate system. The context construction subunit specifies that screen images of arbitrary resolution are mapped to an integer range from 0 to 1000. For example, the top-left vertex of the screen is represented by coordinates (0, 0), and the bottom-right vertex is represented by coordinates (1000, 1000). This normalization process allows the multimodal large model inference module to ignore the specific hardware resolution differences of the in-vehicle terminal device and focus only on the relative positional relationships of interface elements.

[0077] The multimodal large-scale model inference module includes inference and decoding subunits. These subunits load parameters from a large-scale language model that have been fine-tuned using massive amounts of graphical user interface data. The inference and decoding subunits employ, for example, autoregressive generation methods to predict output tokens one by one based on the multimodal mixed input sequence and system prompts.

[0078] The execution process of the reasoning and decoding subunit includes a perception phase, a planning phase, and an action generation phase. In the perception phase, the reasoning and decoding subunit uses visual feature vectors to identify key user interface elements in the screenshot, including icons, buttons, navigation bars, and text labels, and uses optical character recognition (OCR) capabilities to extract text information from the interface. In the planning phase, the reasoning and decoding subunit combines the semantic intent of the user's voice commands with the visual state of the current interface to construct a logical reasoning chain. The logical reasoning chain is used to determine whether the current interface contains an operation entry point that meets the user's needs, or whether a page jump is required.

[0079] During the action generation phase, the inference and decoding subunit outputs structured operation instruction text. This operation instruction text follows preset format constraints, such as JSON format. The operation instruction text may explicitly contain the normalized bounding box coordinates or center point coordinates of the target operation object.

[0080] For example, when the user command is to play the next song and the current interface is a music player, the inference and decoding subunit identifies the position of the next song icon in the normalized coordinate system as [650, 800, 750, 900]. The inference and decoding subunit generates a command sequence that includes the click action and the center point coordinates (700, 850).

[0081] The multimodal large model inference module may also include a history context management subunit. This subunit maintains a sliding window cache queue to store recently accessed information. The interaction history data includes the user's historical voice commands, visual feature summaries of historical screenshots, and historical actions generated by the model. During the inference of the current round, the inference and decoding subunits simultaneously read data from the history context management subunit to achieve coherent processing of multi-turn continuous dialogues and long-term tasks. For example, after performing a page-down operation, the model needs to know the content of the previous page through the history context to decide whether to continue page-down.

[0082] Please see the appendix Figure 2 The cloud-based intelligent agent subsystem includes a coordinate mapping and normalization processing unit, which is configured to process spatial location information during the multimodal large model inference process, thereby decoupling the model inference space from the physical device display space.

[0083] The coordinate mapping and normalization unit constructs a virtual normalized coordinate system. This virtual normalized coordinate system is defined as a two-dimensional Cartesian coordinate system, with both the horizontal and vertical axes having fixed ranges. In this embodiment of the invention, the parameters The value is set to 1000. This virtual normalized coordinate system maps the physical screen space of any aspect ratio and resolution to a unified dimensionless metric space, allowing the multimodal large model inference unit to be independent of the specific hardware display parameters of the in-vehicle terminal device. The coordinate mapping and normalization processing unit includes a spatial discretization subunit. The spatial discretization subunit is configured to divide the continuous screen pixel space into discrete grid intervals. The spatial discretization subunit uses, for example, a linear interpolation algorithm to establish the physical pixel coordinates. Discrete coordinates of the model A two-way mapping relationship between them.

[0084] When the multimodal large model inference unit needs to identify interface elements in a screenshot, the coordinate mapping and normalization processing unit performs forward normalization processing. Assume the width of the input screenshot is... The height is For any point in the screenshot Its corresponding coordinates in the virtual normalized coordinate system Calculated using the following formula: ; ; Among them, symbols This indicates the floor function. The calculated result... and It is converted into a corresponding text location token and embedded in the contextual prompts of the multimodal large model, so that the model can understand the spatial distribution of interface elements.

[0085] When the multimodal large model inference unit generates operation instructions, the coordinate mapping and normalization processing unit performs inverse coordinate analysis. The operation instructions output by the multimodal large model inference unit typically contain a sequence of bounding box coordinates representing the operation region. All values ​​are integers in a virtual normalized coordinate system. The coordinate mapping and normalization processing unit first calculates the coordinates of the geometric center point of the target operation based on the bounding box coordinate sequence. The calculation formula is as follows: ; ; The coordinate mapping and normalization unit further converts the integer coordinates of the geometric center point into standardized floating-point relative coordinates. This is to facilitate transmission to the vehicle-side subsystem. The normalized floating-point relative coordinates range from [0.0, 1.0]. The conversion formula is as follows: ; ; The coordinate mapping and normalization processing unit may also include an image padding adaptive subunit. This subunit is configured to address the mismatch between the aspect ratio of the image generated by the in-vehicle widescreen display and the aspect ratio of the pre-trained input of the multimodal large model. This is useful when the screenshot from the in-vehicle terminal device is resized to fit the input resolution of the multimodal large model. If a strategy of scaling while maintaining aspect ratio and filling with black borders is adopted, the image filling adaptive sub-unit needs to perform offset correction on the output coordinates.

[0086] Scaling factor is The horizontal fill offset is The vertical fill offset is For the original coordinates output by the model The image filling adaptive subunit restores the coordinates within the effective image region through the following linear transformation. : ; ; Corrected coordinates It is then remapped back to a virtual normalized coordinate system or directly converted to standardized floating-point relative coordinates, ensuring the accuracy of click operations in irregular or ultra-wide screen scenarios.

[0087] The coordinate mapping and normalization processing unit will ultimately generate normalized floating-point relative coordinates. The parameters are encapsulated into the downlink control command data packet and sent to the vehicle-side subsystem via the network communication interface. After receiving the standardized floating-point relative coordinates, the vehicle-side subsystem performs the final physical coordinate reconstruction based on its own physical resolution, thus completing the coordinate data flow from the cloud inference space to the vehicle-side execution space.

[0088] Please see the appendix Figure 3 This embodiment takes the user's voice command to play the first episode of the TV series "Joy of Life" on Tencent Video as an example to illustrate in detail the entire process of end-to-cloud collaborative processing from task initiation to task completion.

[0089] The vehicle-side subsystem is in standby monitoring mode. When the voice activity detection subunit in the voice stream acquisition unit detects a human voice signal with energy exceeding a threshold, the system enters the interactive session cycle (time T0). The voice stream acquisition unit begins recording the user's voice commands and segments them into audio data frames with a duration of, for example, 200 milliseconds. Simultaneously, the visual stream perception unit is triggered, capturing image data of the current vehicle central control screen at a frequency of, for example, 5 frames per second. The first screenshot displays the vehicle system's desktop launcher interface, which includes the icon of the Tencent Video application.

[0090] The communication transmission unit pushes audio data frame sequences and screenshot data frame sequences to the cloud server in real time through the established full-duplex network connection channel. The audio data frames and screenshot data frames carry the same timestamp to ensure modal alignment.

[0091] The cloud-based intelligent agent subsystem receives the data stream. The multimodal data alignment unit converts the audio data frames into a text stream in real time: playing in Tencent Video...; simultaneously, the multimodal large model inference unit receives the first frame screenshot.

[0092] At time T1, the multimodal large model inference unit performs initial inference by combining partially recognized textual intent with the first frame screenshot. The multimodal large model inference unit recognizes the Tencent Video icon in the desktop launcher interface and calculates the center coordinates of the icon in the virtual normalized coordinate system. Although the voice command has not yet fully finished, the multimodal large model inference unit generates the first operation command based on probabilistic prediction: a click operation (CLICK), with target coordinates. .

[0093] The instruction serialization generation unit encapsulates and sends out the first operation instruction. The operation execution simulation unit of the vehicle-side subsystem parses the instruction, calls the underlying accessibility service interface, and executes a click action at the corresponding location on the physical screen. This action triggers the vehicle operating system to launch the Tencent Video application.

[0094] At time T2, the application startup process caused a change in the screen display. The visual stream perception unit captured an updated screenshot, which displayed the Tencent Video homepage. The new screenshot was immediately uploaded to the cloud server.

[0095] The multimodal large-scale model inference unit receives a new screenshot, combines it with the complete voice command text to play the first episode of the TV series "Joy of Life" on Tencent Video, and updates the contextual inference state. The multimodal large-scale model inference unit analyzes the new screenshot, locates the search box area at the top of the interface using optical character recognition technology or feature matching technology, and calculates the center coordinates of the search box. The multimodal large model inference unit generates a second operation instruction: a click operation (CLICK), with target coordinates. .

[0096] The vehicle-mounted subsystem executes a click on the search box. This action triggers the vehicle system to display a virtual soft keyboard, and the interface enters search input mode.

[0097] At time T3, the visual flow perception unit uploads a screenshot containing the virtual soft keyboard. The multimodal large model inference unit recognizes that the input focus has been activated and generates a third operation command based on the entity word "Joy of Life" in the voice command: text input operation (TYPE), with the parameter value being "Joy of Life".

[0098] The underlying simulation unit of the vehicle-mounted terminal subsystem receives the instruction and fills the characters into the search box by directly setting text attributes or simulating keyboard key sequences, and automatically triggers the search confirmation action (such as simulating clicking the Enter key on the keyboard).

[0099] At time T4, the screen refreshes to the search results list interface. The visual flow perception unit uploads a screenshot of this interface. The multimodal large model inference unit performs semantic matching on the video cover and title shown in the screenshot, identifies the video card element with the highest matching degree with "Joy of Life", locates the first episode sub-option button below the video card, and calculates the coordinates. The multimodal large model inference unit generates the fourth operation instruction: click operation (CLICK), target coordinates. .

[0100] The vehicle-mounted subsystem executes a click operation. The system then enters the video playback page.

[0101] At time T5, the visual flow perception unit uploads a screenshot of the video playback interface. The multimodal large model inference unit analyzes the screenshot and detects that the video player control has been loaded and the screen content is dynamically changing (or a playback progress bar is displayed). The multimodal large model inference unit compares the current visual state with the user's original intention to play...the first episode. After confirming that the visual state meets the expected goal, the multimodal large model inference unit generates a task end signal.

[0102] The communication transmission unit receives the task completion signal, and the voice playback component synthesizes and plays the voice feedback that is playing the first episode of "Joy of Life" for you, marking the end of the closed loop of this multimodal interaction process. Throughout the process, the cloud-based intelligent agent subsystem dynamically adjusts the operation strategy at each step by continuously receiving the latest visual feedback. Even if an unexpected pop-up ad appears in the middle, the model can generate a command to close the pop-up ad in the next frame of inference, thereby ensuring the continuity of the task chain.

[0103] Please see the appendix Figure 4 The embodiments of the present invention can not only execute linear task instructions, but also have the ability to perform dynamic error detection and real-time correction based on visual feedback, so as to deal with non-deterministic events in the vehicle software environment.

[0104] After each operation execution simulation unit injects a physical action, the vehicle-mounted subsystem does not immediately recognize that the step has been successfully completed. The visual flow perception unit is configured to capture one or more frames of verification images within a preset delay time window (e.g., 500 to 1000 milliseconds) after the action is executed. This verification image is transmitted to the cloud-based intelligent agent subsystem as a basis for status verification.

[0105] The multimodal large model inference unit receives the verification image and performs semantic comparison with the expected state description generated in the previous round. Internally, the multimodal large model inference unit maintains a task state machine. When the inference unit generates an operation command, it simultaneously predicts the changes in interface features after the command is executed. For example, for clicking the search box, expected state features include the keyboard popping up or the cursor appearing; for clicking play, expected state features include video loading or the play icon changing to a pause icon.

[0106] When the visual features of the verification image do not match the expected state features, the multimodal large model inference unit triggers the exception handling logic. Exceptions mainly include interference pop-up occlusion, page load timeout, and click non-response.

[0107] To address scenarios where pop-up windows obstruct the view, assuming the user intends to play a video, but after clicking the playlist, a modal dialog box pops up in the center of the screen offering membership renewal or advertising, blocking subsequent steps. The multimodal large model inference unit uses visual recognition to detect a higher-level floating window on the current interface and identifies key text information (such as "Activate Now" and "Skip") and close controls (such as an X-shaped icon and a close button) within that window.

[0108] The multimodal large model inference unit pauses the execution path of the current main task and dynamically inserts a sub-task sequence. The goal of this sub-task sequence is to eliminate visual interference. The multimodal large model inference unit calculates the position of the close control in a virtual normalized coordinate system and generates click operation instructions for the close control.

[0109] The cloud-based intelligent agent subsystem sends the error correction command to the vehicle-side subsystem. The vehicle-side subsystem then closes the pop-up window. Subsequently, the visual flow perception unit uploads a new screenshot. After confirming that the interfering elements have disappeared and the interface has returned to the main task context, the multimodal large model inference unit regenerates the interrupted original operation command (e.g., trying to click the video playback item again), thereby achieving self-healing of the task flow.

[0110] In scenarios involving page load timeouts or network latency, if the verification image display remains in a loading animation state or a blank state for an extended period, the multimodal large model inference unit identifies this state and determines it to be a non-interactive ready state. The multimodal large model inference unit generates a waiting instruction, instructing the vehicle-side subsystem to remain silent without performing any physical actions, and then triggers the screenshot upload again after a delay.

[0111] In scenarios where clicks fail to respond, if the verification image display state is exactly the same as before the operation, and there is no visual feedback (such as a button being pressed or a page redirect), the multimodal large model inference unit determines that the previous operation failed. This is caused by touch coordinate deviation or system lag. The multimodal large model inference unit initiates a retry mechanism, fine-tunes the coordinate range of the target control (e.g., adds random jitter offset) or recalculates the center point, and then issues the same operation type instruction again.

[0112] The multimodal large model inference unit has a maximum self-healing attempt threshold (e.g., 3 times). If the verification image still does not meet the expected state after reaching this threshold consecutively, or if an unresolved system-level error message appears on the interface, the multimodal large model inference unit will terminate the current task loop. At this time, the multimodal large model inference unit generates an error report text and sends it to the vehicle-side subsystem via the communication transmission unit. The audio playback component of the vehicle-side subsystem then plays the corresponding voice prompt (e.g., poor network connection, unable to play, or the corresponding content not found), providing feedback to the user on the reason for the task failure, thus completing a smooth degradation from automatic execution to manual intervention.

[0113] In a preferred embodiment of the invention, the vehicle-mounted end-side subsystem further integrates a lightweight edge inference unit to construct an edge-cloud hybrid computing architecture. This architecture is configured to reduce system latency and ensure the availability of basic functions when network communication quality is unstable or when processing high-frequency, simple commands.

[0114] The vehicle-mounted end-side subsystem may also include a task complexity assessment unit. This unit is located between the voice stream acquisition unit and the communication transmission unit. The task complexity assessment unit receives the text-to-text conversion results or raw voice features of the user's voice commands. The unit incorporates a classifier based on a lightweight neural network or rule engine. This classifier categorizes user commands into sets of low-complexity commands and sets of high-complexity commands.

[0115] The low-complexity instruction set contains atomic operations that can be explicitly executed without relying on complex visual context reasoning, or system-predefined shortcut control instructions. Examples of low-complexity instructions include increasing volume, pausing playback, skipping to the next track, turning on the air conditioner, and returning to the home page. The high-complexity instruction set contains instructions that require multi-step visual search, semantic understanding, or cross-application navigation, such as finding and playing a movie, setting navigation to avoid traffic jams, or adjusting screen brightness to match ambient light.

[0116] When the task complexity assessment unit determines that the current instruction belongs to the low-complexity instruction set, it routes the instruction to the lightweight edge inference unit. The lightweight edge inference unit runs on the local neural network processor or digital signal processor of the in-vehicle terminal device. The lightweight edge inference unit loads a small edge model that has undergone model quantization pruning. This edge model does not rely on a cloud server and directly generates corresponding local control signals based on the voice command. The lightweight edge inference unit directly calls the control interface of the in-vehicle operating system through the local system bus to execute corresponding media control or body control operations.

[0117] When the task complexity assessment unit determines that the current instruction belongs to the high-complexity instruction set, or when the confidence level output by the lightweight edge inference unit is lower than a preset threshold, the task complexity assessment unit routes the instruction to the communication transmission unit. Subsequently, the communication transmission unit initiates the full-duplex streaming communication process described in the aforementioned embodiments, sending voice data and screenshot data to the cloud server, where it is handled by the multimodal large-model inference unit of the cloud intelligent agent subsystem.

[0118] The lightweight edge inference unit also features offline fallback functionality. When the communication transmission unit detects a network connection interruption between the in-vehicle terminal and the cloud server, the task complexity assessment unit is configured to forcibly route all received instructions to the lightweight edge inference unit. The lightweight edge inference unit performs its best to parse user intent, and for complex instructions that it cannot process, it generates local voice feedback to inform the user that the current network is unavailable.

[0119] In modern smart cockpit environments, in-vehicle terminal equipment is connected to multiple display components, corresponding to the driver's central control screen, the passenger's entertainment screen, and the rear passenger entertainment screen.

[0120] The audio acquisition component in this embodiment includes a multi-zone sound source localization unit. Based on microphone array beamforming technology and time-of-arrival algorithm, the multi-zone sound source localization unit calculates the physical location of the voice command sender. The multi-zone sound source localization unit divides the vehicle interior into a driver's seat zone, a passenger's seat zone, and a rear seat zone. The multi-zone sound source localization unit appends sound source zone identifiers to the output voice data packets.

[0121] The multimodal large model inference unit in the cloud-based intelligent agent subsystem is configured to receive sound source region identifiers and parse the user's referential intent accordingly.

[0122] When a user issues an implicit instruction, such as turning off the screen or playing music, the multimodal large model inference unit, based on the sound source's audio region identifier, defaults to targeting the display screen corresponding to the user's location. For example, when voice is detected to be coming from the passenger's audio region, the multimodal large model inference unit instructs the visual flow perception unit to capture image data from the passenger's entertainment screen and generate an operation command for that screen.

[0123] When a user issues an explicit cross-screen control command, such as the driver saying to play an animation on the rear screen, the multimodal large model inference unit identifies the source audio region as the driver's audio region and the target audio region as the rear audio region through semantic analysis. The multimodal large model inference unit generates a cross-screen control request. At this time, the visual flow perception unit of the vehicle-side subsystem captures and uploads a screenshot of the current interface of the rear passenger entertainment screen according to the request. The multimodal large model inference unit performs inference based on the visual state of the rear screen, and the generated click or swipe operation command contains the target screen identification code.

[0124] The operation execution simulation unit of the vehicle-mounted subsystem is configured to support multi-instance distribution. Based on the target screen identification code, the operation execution simulation unit injects the underlying touch simulation events into the corresponding display device driver pipeline. In this way, the present invention achieves one-person multi-screen control and full-vehicle interactive collaboration based on natural language, enabling the driver to assist passengers in other seats with interface operation management without leaving the driver's seat or turning their head.

Claims

1. An in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction, characterized in that: include: The vehicle-mounted terminal subsystem is deployed on the vehicle-mounted terminal equipment and integrates a voice stream acquisition unit, a visual stream perception unit, a communication transmission unit, and an operation execution simulation unit. The cloud-based intelligent agent subsystem is deployed on a cloud server and runs a multimodal large model. The vehicle-mounted terminal subsystem uses the voice stream acquisition unit to capture digital audio streams and generate voice data packets, uses the visual stream perception unit to capture image data of the graphical user interface and generate screenshot data packets, and encapsulates the voice data packets and the screenshot data packets into a multimodal input stream through the communication transmission unit and sends them to the cloud intelligent agent subsystem. The cloud-based intelligent agent subsystem receives the multimodal input stream, uses the multimodal large model to perform joint semantic analysis and control element recognition to generate an operation action sequence, and encapsulates the operation action sequence into a control command stream and returns it to the vehicle-mounted terminal subsystem. The operation execution simulation unit is responsible for parsing the control command stream, converting it into low-level input events that the vehicle operating system can recognize, and injecting the low-level input events at a specified location in the graphical user interface to complete the interactive operation.

2. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The cloud-based intelligent agent subsystem includes: The multimodal data alignment unit converts the voice data packets into text sequences and the screenshot data packets into visual feature vectors. The multimodal large model inference unit inputs the text sequence and the visual feature vector into the multimodal large model and outputs the operation action sequence; The instruction serialization generation unit serializes the operation action sequence into the control instruction stream of structured data.

3. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The communication transmission unit establishes a full-duplex long connection channel based on the transmission control protocol and supports asynchronous parallel execution of data sending and receiving operations; The communication transmission unit includes a transmission queue management subunit. The transmission queue management subunit sets the highest transmission priority for the voice data packets and actively discards the old timestamp screenshot data packets that are backed up in the transmission queue when the network bandwidth is limited, retaining only the newly generated screenshot data packets.

4. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The cloud-based intelligent agent subsystem is equipped with a coordinate mapping and normalization processing unit, and the operation execution simulation unit of the vehicle-mounted terminal subsystem includes a coordinate mapping subunit. The coordinate mapping and normalization processing unit constructs a virtual normalized coordinate system, converts the normalized bounding box coordinates or center point coordinates generated by the multimodal large model into standardized floating-point relative coordinates, and encapsulates the standardized floating-point relative coordinates into the control command stream; The coordinate mapping subunit reads the horizontal and vertical physical pixel counts of the vehicle display component, and calculates the target physical horizontal and vertical coordinates by multiplying the standardized floating-point relative coordinates with the horizontal and vertical physical pixel counts respectively.

5. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The vehicle-mounted end-side subsystem integrates a task complexity evaluation unit and a lightweight edge inference unit. The task complexity evaluation unit receives user voice commands and uses a classifier to determine whether the user voice commands belong to a low-complexity command set or a high-complexity command set. The lightweight edge inference unit, deployed on the local processor of the vehicle terminal device, is used to directly generate local control signals and call the vehicle operating system interface based on the user voice command when the user voice command belongs to the low-complexity command set. The task complexity evaluation unit is used to route the user voice command to the communication transmission unit when the user voice command belongs to the high complexity command set.

6. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The vehicle-mounted terminal subsystem is equipped with an anomaly monitoring unit. The anomaly monitoring unit is used to trigger the visual flow perception unit to collect verification image data containing the execution result after the operation execution simulation unit completes an atomic operation instruction, and to feed the verification image data back to the cloud intelligent agent subsystem through the communication transmission unit. The cloud-based intelligent agent subsystem is used to receive the verification image data, compare it with the expected state features, and generate an error correction operation instruction or a retry operation instruction when the comparison result does not match.

7. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The voice stream acquisition unit has a built-in voice activity detection subunit, and the visual stream perception unit includes a screenshot triggering logic subunit. The voice activity detection subunit determines whether there is valid human voice input based on the short-time energy value and outputs a voice start status signal or a voice end status signal. The screenshot triggering logic subunit responds to the voice start state signal and periodically triggers screen capture operation at a first preset frequency; In response to the voice end status signal and without receiving an operation execution instruction, the screen capture operation is stopped or triggered at a second preset frequency less than the first preset frequency.

8. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The vehicle-mounted terminal device is connected to multiple display components that correspond to different cabin sound zones, and the voice stream acquisition unit includes a multi-zone sound source localization unit. The multi-zone sound source localization unit calculates the sound source zone identifier of the speaker of the voice command and appends it to the voice data packet; The cloud-based intelligent agent subsystem parses the referential intent based on the sound source zone identifier, determines the target display component, and includes the target screen identification code in the generated control command stream; The operation execution simulation unit injects the underlying input event into the corresponding display component driver pipeline based on the target screen identification code.

9. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 2, characterized in that, The multimodal data alignment unit consists of a visual encoder and a modal alignment projection layer; The visual encoder segments the screenshot data packet into image blocks and maps the image blocks into visual embedding vectors; The modal alignment projection layer maps the dimension of the visual embedding vector to the same dimension space as the text embedding vector corresponding to the text sequence, and concatenates the mapped visual embedding vector sequence with the text embedding vector sequence to form a multimodal mixed input sequence.

10. The in-vehicle graphical interface intelligent agent system based on multimodal large model and voice interaction according to claim 1, characterized in that, The operation execution simulation unit implements event injection based on the accessibility service interface or debug bridge interface of the vehicle operating system; The operation types in the control command stream include click operations, swipe operations, and text input operations; When the operation type is a text input operation, the operation execution simulation unit directly fills the text to be input into the user interface element that has gained focus through the text setting interface, or sends the key codes corresponding to the character codes one by one by simulating keyboard key events.

Images