Vehicle multi-screen interaction method, device, equipment and medium

Abstract

This application relates to the field of vehicle communication technology, and discloses a method, apparatus, and medium for multi-screen interaction in vehicles. The multi-screen interaction method provided by this application includes: establishing a connection mapping table for connected devices in response to the establishment of a communication connection with the vehicle's infotainment system; receiving playback data; determining the target device to which the playback data needs to be routed based on the type of the playback data; and sending the playback data to the target device according to the connection mapping table, so that the playback data can be played through the target device. This application, by establishing a connection mapping table and dynamically routing based on the type of playback data, achieves unified data distribution across heterogeneous operating systems, eliminates the interaction isolation between external devices and the instrument system, avoids duplicate data source requests during multi-screen synchronization, reduces system load, improves resource utilization efficiency, and enhances the user interaction experience.

Description

Technical Field

[0001] This application relates to the field of vehicle communication technology, specifically to a method, device, and medium for multi-screen interaction in vehicles. Background Technology

[0002] Current smart cockpits adopt a heterogeneous architecture with multiple operating systems and multiple domain controllers. Among them, the vehicle system is responsible for the display and interaction of the central control screen, the instrument system is responsible for the safety display of the instrument screen and head-up display (HUD), and external mobile phones, tablets and other devices can be connected wirelessly.

[0003] In related technologies, mobile phones cannot send custom content to instrument panels or head-up displays, which limits data connectivity and scene collaboration between multiple devices and makes it difficult to meet users' expectations for a seamless flow of digital experiences across all scenarios. Summary of the Invention

[0004] This application provides a method, device, and medium for multi-screen interaction in vehicles, which has the advantages of realizing a unified data routing mechanism between the vehicle-mounted system, external devices, and internal devices, avoiding repeated encoding and transmission of data sources, and reducing system processing load and network resource consumption.

[0005] In a first aspect, this application provides a vehicle multi-screen interaction method, the method comprising: in response to the establishment of a communication connection with a vehicle-mounted infotainment system, establishing a connection mapping table for connected devices, the connected devices including a first connected device and a second connected device, the first connected device being an external device of the vehicle-mounted infotainment system, the second connected device being an internal device of the vehicle-mounted infotainment system, and the vehicle-mounted infotainment system, the first connected device and the second connected device each deploying an independent operating system; receiving playback data, wherein the playback data originates from the vehicle-mounted infotainment system, the first connected device or the second connected device; determining the target device to which the playback data needs to be routed based on the type of the playback data, the target device including at least one of the vehicle-mounted infotainment system, the first connected device and the second connected device; and sending the playback data to the target device according to the connection mapping table, so that the playback data can be played through the target device.

[0006] Based on the aforementioned technical methods, by establishing a unified connection mapping table and a data-based intelligent routing mechanism, the interaction barriers caused by heterogeneous operating systems between the vehicle's infotainment system, internal devices, and external devices in the smart cockpit are effectively resolved. As a result, external devices can flexibly project custom content onto internal display devices such as the instrument panel or head-up display, improving the user experience.

[0007] In one optional implementation, the playback data is encapsulated into an encapsulated message when it is sent. The encapsulated message includes common fields, data body and message type. The common fields include protocol version number, message generation timestamp, session identifier, source device identifier, target device identifier and message sequence number. The message type is used to characterize the type of data body and includes at least one of video frame, audio frame and control command.

[0008] Based on the aforementioned technical means, this application provides a unified communication protocol foundation for complex data interaction between heterogeneous systems. By introducing a content routing table mechanism, this application achieves intelligent addressing for data transmission across heterogeneous devices. By using common fields in the encapsulated message, such as source device identifier and session identifier, along with the message type (e.g., video, audio, control command), as a query index, it can accurately match target devices that support specific data types.

[0009] In one optional implementation, determining the target device to which the playback data needs to be routed based on the type of playback data includes: querying a content routing table based on the common fields and message type of the encapsulated message. The content routing table includes the source device identifier, the target device identifier, and the supported message types. The content routing table is established and maintained by the vehicle-mounted system. The target device is determined based on the query results returned by the content routing table.

[0010] Based on the above technical means, by dynamically determining the target device based on the query results, not only is the bandwidth waste caused by blindly broadcasting data avoided, but the playback data can also be accurately routed to the corresponding display or processing terminal according to the device's capabilities. This achieves precise control of the data flow in multi-screen interactive scenarios, thereby effectively solving the problem of chaotic data distribution or inability to achieve efficient cross-device collaborative display due to the lack of a unified and dynamic routing addressing logic. In one optional implementation, before sending the playback data to the target device according to the connection mapping table, the method further includes: determining whether the playback data contains video frames based on the message type of the encapsulated message; if video frames are included, determining whether the target device supports the encoding format of the video frames; if the target device supports the encoding format, then performing the step of sending the playback data to the target device according to the connection mapping table; if the target device does not support the encoding format, then transcoding the video frames before performing the step of sending the playback data to the target device according to the connection mapping table.

[0011] Based on the above technical means, by introducing video encoding format compatibility detection and dynamic transcoding, seamless video stream transmission and display across devices and systems are achieved, improving the user experience in multi-screen interactive scenarios in smart cockpits, avoiding playback failures or display abnormalities caused by format incompatibility, thereby enhancing the robustness and user satisfaction of the entire multi-screen interactive system.

[0012] In one optional implementation, sending playback data to the target device according to the connection mapping table includes: if the target device is an in-vehicle infotainment system, writing the transcoded playback data into the display buffer of the in-vehicle infotainment system for playback on the playback device of the in-vehicle infotainment system; if the target device is a first connection device and / or a second connection device, re-encapsulating the transcoded playback data into an encapsulated message and sending it to the first connection device and / or the second connection device for playback.

[0013] Based on the aforementioned technical methods, by re-encapsulating the transcoded data into a unified encapsulated message format, protocol consistency during data transmission is ensured, enabling external or internal devices to parse and play the data based on a unified communication protocol. This not only solves the problem of encoding format mismatch but also ensures the compatibility and stability of playback data across different hardware platforms through targeted data delivery methods. This achieves seamless multi-screen interaction across devices and systems, significantly improving the user experience. In one optional implementation, playing the playback data through the target device further includes: determining whether the playback data contains a control frame based on the message type; when a control frame is contained, converting the first touch coordinates on the display screen in the source device to the second touch coordinates on the target display plane based on the differences between the display screens of the source device and the target device, wherein the differences in the display screens include resolution differences and aspect ratio differences; and displaying a touch animation in response to the second touch coordinates.

[0014] Based on the aforementioned technical means, the intuitiveness and accuracy of user operation in the multi-screen environment of the smart cockpit are improved, avoiding misoperation or operation failure caused by coordinate mismatch. Furthermore, on the basis of the basic playback data routing mechanism, the user experience at the interaction level is further improved, and a more complete and efficient multi-screen interactive ecosystem for the smart cockpit is built. In one optional implementation, the playback data further includes a device capability report, a heartbeat message, and an acknowledgment message. The method further includes: updating the hardware parameter description for the target device in the content routing table based on the device capability report; determining the connection status of the first connected device and / or the second connected device based on the heartbeat message; and determining whether the playback data received by the first connected device and / or the second connected device has been lost based on the acknowledgment message.

[0015] Based on the above technical means, the problem of the lack of real-time perception and feedback mechanism for device status in the system is effectively solved, dynamic resource scheduling and fault self-healing are realized, and the reliability and user experience of multi-screen interaction are significantly improved. Secondly, this application provides a vehicle multi-screen interactive device, comprising: an establishment module, configured to establish a connection mapping table for connected devices in response to the establishment of a communication connection with a vehicle-mounted system, the connected devices including a first connected device and a second connected device, the first connected device being an external device of the vehicle-mounted system and the second connected device being an internal device of the vehicle-mounted system, the vehicle-mounted system, the first connected device, and the second connected device all deploying independent operating systems; a receiving module, configured to receive playback data, wherein the playback data originates from the vehicle-mounted system, the first connected device, or the second connected device; a determining module, configured to determine the target device to which the playback data needs to be routed based on the type of the playback data, the target device including at least one of the vehicle-mounted system, the first connected device, and the second connected device; and a sending module, configured to send the playback data to the target device according to the connection mapping table, so that the playback data can be played through the target device.

[0016] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle multi-screen interaction method described in the first aspect or any corresponding embodiment.

[0017] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the vehicle multi-screen interaction method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of a first method for multi-screen interaction in a vehicle according to an embodiment of this application; Figure 2 This is a schematic diagram of the encapsulated message structure according to an embodiment of this application; Figure 3 This is a second flowchart illustrating a vehicle multi-screen interaction method according to an embodiment of this application; Figure 4 This is a schematic diagram of the third process of the vehicle multi-screen interaction method according to the embodiments of this application; Figure 5 This is an example schematic diagram of a vehicle multi-screen interaction method according to an embodiment of this application; Figure 6This is a structural block diagram of a vehicle multi-screen interactive device according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] Traditional smart cockpits generally employ a heterogeneous architecture with multiple operating systems and domain controllers, creating interaction barriers between the vehicle's infotainment system, instrument cluster, and external devices such as users' smartphones and tablets. Specifically, external devices cannot directly send custom content to the instrument cluster or head-up display. Furthermore, when the same content needs to be displayed on multiple screens across different systems, each screen must independently request data from the source device. This results in repeated encoding and transmission by the source device, wasting bandwidth and increasing system load, ultimately leading to low data transmission efficiency.

[0024] To address this, this application proposes a multi-screen interaction method for vehicles, applied to in-vehicle infotainment systems. For example... Figure 1 As shown, the method includes: Step S101: In response to the establishment of a communication connection with the vehicle infotainment system, a connection mapping table is established for the connected devices. The connected devices include a first connected device and a second connected device. The first connected device is an external device of the vehicle infotainment system, and the second connected device is an internal device of the vehicle infotainment system. The vehicle infotainment system, the first connected device, and the second connected device all deploy independent operating systems.

[0025] Step S102: Receive playback data, wherein the playback data originates from the vehicle system, the first connected device, or the second connected device; Step S103: Determine the target device to which the playback data needs to be routed based on the type of playback data. The target device includes at least one of the vehicle system, the first connection device, and the second connection device. Step S104: Send the playback data to the target device according to the connection mapping table so that the playback data can be played through the target device.

[0026] For ease of understanding, the following explains some key terms in this embodiment: The in-vehicle infotainment system, typically referred to as the infotainment system on the center console of a vehicle, is the core of the vehicle's multi-screen interactive system. It manages and coordinates data transmission and display between various connected devices. In-vehicle infotainment systems usually have their own operating system, such as Android, to provide rich application functions and user interfaces.

[0027] Connectivity devices refer to various devices capable of establishing communication connections with the vehicle's infotainment system and participating in multi-screen interactions. These devices are categorized into primary connectivity devices and secondary connectivity devices based on their physical location and functional attributes.

[0028] The first connected device refers to a device external to the vehicle's infotainment system, such as a user's smartphone, tablet, or other portable smart terminal. These devices establish a wireless communication connection with the vehicle's infotainment system and typically have their own operating system, such as iOS or Android.

[0029] The second connection device refers to the equipment inside the vehicle's infotainment system, such as the vehicle's instrument panel display, passenger entertainment screen, or head-up display (HUD). These devices are typically connected to the vehicle's infotainment system via wired or internal bus connections and have their own independent operating systems, such as the QNX system or a real-time operating system (RTOS), to meet specific functional and safety requirements.

[0030] The connection mapping table is a data structure maintained by the vehicle's infotainment system for each successfully established communication connection. This table records the unique identifier of the connected device, its connection status, device type (first or second connected device), and other necessary connection parameters for subsequent data routing and management.

[0031] Playback data refers to various types of data that need to be displayed or processed on different devices in multi-screen interaction scenarios. It can originate from the vehicle's own infotainment system, the first connected device, or the second connected device. Playback data can include various types such as video streams, audio streams, images, text information, and control commands.

[0032] The target device refers to the device that is determined to receive, play, or process the playback data based on the type of playback data and user needs. The target device can be the vehicle-mounted system itself, one or more first-connection devices, and one or more second-connection devices.

[0033] An independent operating system refers to a software system that runs independently of the vehicle's infotainment system, the first connection device, and the second connection device. This heterogeneous operating system environment is one of the core problems that this application aims to solve, and it is also the technical foundation for achieving cross-system interaction.

[0034] Specifically, this embodiment provides a method for multi-screen interaction in a vehicle, the implementation of which may include the following steps: First, in response to the establishment of a communication connection with the vehicle's infotainment system, a connection mapping table is created for the connected devices. When a device (whether it's the first or second connected device) attempts to communicate with the infotainment system, the system responds to its connection request. For example, the infotainment system can listen to a specific communication port or broadcast signal. Once the communication link is successfully established, the infotainment system assigns a unique identifier to the newly connected device and records its device type (e.g., whether it's an external device like a mobile phone or an internal device like the instrument cluster) and current connection status in a connection mapping table. This connection mapping table can be a simple list or hash table used to store the identifier of each connected device and its corresponding connection information. For example, when a user's mobile phone successfully pairs with the infotainment system via Bluetooth or Wi-Fi, the system records the phone's Media Access Control (MAC) address or Internet Protocol (IP) address and marks it as the first connected device. Similarly, when the vehicle's instrument cluster system starts up and establishes an internal communication link with the infotainment system, the system records the instrument cluster system's device ID and marks it as the second connected device. The vehicle infotainment system, the first connection device, and the second connection device all have independent operating systems, which makes them independent in terms of functionality and resource management.

[0035] Secondly, the system receives playback data. After the connection mapping table is established, the vehicle's infotainment system can receive playback data from different sources. This playback data can be generated by the infotainment system itself, such as map information generated by a navigation application running on the system or videos played by a multimedia player. Playback data can also originate from a first connected device, such as music or video streams playing on a user's mobile phone, or content from third-party applications on the phone. Furthermore, playback data can also originate from a second connected device, such as vehicle status information or warning prompts generated by the instrument cluster system. The infotainment system continuously listens for and receives this playback data from different sources through its communication modules, such as a wireless communication module or a wired communication interface. For example, the infotainment system can be configured with a universal data receiving interface through which all connected devices send playback data.

[0036] Secondly, the target device to which the playback data needs to be routed is determined based on the type of playback data. After receiving playback data, the vehicle's infotainment system needs to identify the type of playback data, such as video data, audio data, image data, or control commands. Based on the identification of the playback data type, the system can determine which devices are suitable or need to receive and play the data. For example, if the playback data is navigation map information, it may need to be routed to the vehicle's display screen and instrument panel display screen; if the playback data is music, it may need to be routed to the vehicle's audio output system or a connected Bluetooth speaker (as a first connected device). The system can preset some simple rules, such as "video data is sent to the large display screen first" and "audio data is sent to the audio system," to make initial routing decisions. Target devices may include the vehicle's infotainment system itself, one or more first connected devices, and one or more second connected devices.

[0037] Finally, the playback data is sent to the target device according to the connection mapping table, so that the playback data can be played through the target device. Once the target device is identified, the vehicle's infotainment system queries the previously established connection mapping table to obtain the specific connection information of the target device, such as its network address or communication port. Subsequently, the vehicle's infotainment system sends the playback data to the target device through the corresponding communication link. For example, if the target device is the vehicle's own infotainment system, the playback data is directly sent to the infotainment system's display or audio processing module. If the target device is the first connected device (such as a mobile phone), the vehicle's infotainment system sends the data to the mobile phone wirelessly. If the target device is the second connected device (such as the instrument panel), the vehicle's infotainment system sends the data to the instrument system through the internal bus or a dedicated interface. After receiving the playback data, the target device uses its own playback capabilities to play the data, such as displaying video, playing audio, or executing control commands.

[0038] It is understood that the multi-screen interaction method for vehicles proposed in this application effectively solves the interaction barriers caused by heterogeneous operating systems between the vehicle system, internal devices, and external devices in a smart cockpit by establishing a unified connection mapping table and a data type-based intelligent routing mechanism. As a result, external devices can flexibly project custom content onto internal display devices such as the instrument panel or head-up display, improving the user experience. Simultaneously, this method avoids the repeated encoding and transmission of source devices caused by independent data requests from each screen during multi-screen display, reducing bandwidth consumption and system load, thereby improving the data transmission efficiency and resource utilization of the entire system and achieving collaborative display and efficient distribution of multi-source data. In some of the solutions mentioned above in this application, a mechanism for routing playback data between multiple devices is proposed through a connection mapping table. However, in a heterogeneous system environment, the types of data transmitted between different devices are diverse and complex. If there is a lack of a unified data interaction format and metadata definition, the receiving device will have difficulty identifying the data content, source and transmission sequence, resulting in poor compatibility and low parsing efficiency in cross-system communication, and it is impossible to perform differentiated processing for different types of data (such as video, audio and control commands).

[0039] In response, this application further proposes that playback data be encapsulated into encapsulated messages during transmission. For example... Figure 2 As shown, the encapsulated message includes common fields, data ontology, and message type. Common fields include the protocol version number, message generation timestamp, session identifier, source device identifier, target device identifier, and message sequence number. The message type characterizes the type of data ontology and includes at least one of video frames, audio frames, and control commands; it may also include capability reports, heartbeats, and acknowledgment messages.

[0040] Specifically, when playback data is sent, it undergoes encapsulation into encapsulated messages. This means wrapping the original playback data (such as video streams, audio streams, control commands, etc.) in a unified structure to form a standardized data packet, i.e., an encapsulated message. This is to establish a unified communication protocol and data format between heterogeneous systems, enabling devices with different operating systems and hardware architectures to understand and parse the transmitted data. For example, application-layer protocols based on protocol stacks (such as the TCP / IP protocol stack) can be used for encapsulation, with the original data as the payload and protocol header information added; alternatively, message queues (MQTT) or remote procedure calls (RPC) frameworks can be used for encapsulation, with the playback data as the message body, and the message middleware or RPC framework automatically adding necessary metadata and transmission control information.

[0041] Encapsulating a message involves common fields, a data ontology, and a message type. The aim is to separate the metadata (common fields, message type) from the actual content (data ontology), thereby improving data transmission efficiency and parsing flexibility. Common fields provide global transmission and management information, the data ontology carries the actual playback content, and the message type defines the semantics of the data ontology. For example, in a message structure, common fields can serve as the message header, the data ontology as the message body, and the message type as a specific field within the message header; alternatively, common fields and the message type can be packaged as independent metadata blocks along with the data ontology, distinguished by specific delimiters or length fields.

[0042] The common fields include the protocol version number, message timestamp, session identifier, source device identifier, destination device identifier, and message sequence number. These common fields are the core metadata encapsulating the message, providing contextual information for data transmission, ensuring the reliability and orderliness of data transmission, and supporting precise addressing and state management between multiple devices. Specifically, the protocol version number ensures that both communicating parties use compatible protocol specifications; the message timestamp is used for data timing verification, delay calculation, and handling of expired messages; the session identifier associates all messages belonging to the same logical session, facilitating session management and state tracking; the source device identifier identifies the data sender and is used for authorization verification, backtracking, and routing decisions; the destination device identifier identifies the data receiver and is used for precise routing and multicast / broadcast control; and the message sequence number ensures the orderly transmission and integrity of messages and is used for detecting packet loss and retransmission. These fields can be fixed-length or variable-length fields, arranged in a predefined order at specific offsets in the message header; alternatively, they can be used as key-value pairs, encoding each common field and its value within the message header, providing greater flexibility and scalability.

[0043] The message type is used to characterize the type of the data ontology, and must include at least one of video frames, audio frames, or control commands. The message type field is crucial for the receiving end to parse the data ontology; it indicates the specific content format and semantics carried by the data ontology, enabling the receiving end to call the corresponding decoder or processing logic for processing. For example, the message type can be an enumeration value or an integer code, with each value corresponding to a specific data type, such as 0 representing a video frame, 1 representing an audio frame, and 2 representing a control command; alternatively, it can use string identifiers, such as "video_frame", "audio_frame", and "control_command", to improve readability and extensibility.

[0044] It is understood that, through the above technical solutions, this application provides a unified communication protocol foundation for complex data interaction between heterogeneous systems. Specifically, by encapsulating playback data into a structure containing common fields, data body, and message type, data transmission standardization is achieved. The protocol version number in the common fields ensures compatibility between different systems; even if the protocol is upgraded in the future, compatibility can be determined through the version number. The timestamp of message generation and the message sequence number together guarantee the timing and integrity of data transmission. The receiving end can use this to determine the message order, detect packet loss, and request retransmission, thereby improving the reliability of data transmission. The session identifier, source device identifier, and target device identifier clarify the logical link of data flow, solving the addressing and state management problems when multiple devices communicate concurrently, enabling the vehicle system to accurately route data to the correct device. The introduction of the message type field allows the receiving end to perform targeted processing based on the attributes of the data body, such as distinguishing between video frames, audio frames, or control commands, thereby achieving accurate scheduling and parsing of different media streams and avoiding blind guessing or complex parsing of data content by the receiving end. The encapsulation mechanism not only reduces the complexity of direct interfacing between heterogeneous systems, but also improves the system's parsing efficiency and processing flexibility for multimedia data streams by separating metadata and payload. This provides reliable and efficient data support for subsequent routing decisions and content distribution based on connection mapping tables, and significantly improves data interaction compatibility and processing efficiency in heterogeneous multi-screen environments. In some of the solutions mentioned above in this application, a routing mechanism for determining the target device based on the type of playback data is proposed. However, in this process, due to the heterogeneous system differences between the vehicle-mounted system, the first connected device, and the second connected device, and the diverse sources of playback data, if there is a lack of unified and dynamic routing addressing logic, the system will be unable to accurately identify the best transmission path for different types of data in a complex multi-device environment, which will result in chaotic data distribution or failure to achieve efficient collaborative display across devices.

[0045] In response, this application further proposes a method for multi-screen interaction in vehicles, such as... Figure 3 As shown, the method includes: Step S301: In response to the establishment of a communication connection with the vehicle's infotainment system, a connection mapping table is created for the connected devices. See details... Figure 1 Step S101 in the embodiment will not be described again here. Step S302: Receive playback data, wherein the playback data originates from the vehicle infotainment system, the first connected device, or the second connected device; see details below. Figure 1 Step S102 in the embodiment will not be described again here.

[0046] Step S303: Determine the target device to which the playback data needs to be routed based on the type of playback data. The target device includes at least one of the vehicle infotainment system, the first connection device, and the second connection device. Specifically, this includes: Step S3031: Query the content routing table based on the common fields and message type of the encapsulated message. The content routing table includes the source device identifier, the target device identifier, and the supported message types. The content routing table is established and maintained by the vehicle system.

[0047] Step S3032: Determine the target device based on the query results returned by the content routing table.

[0048] Specifically, a content routing table is a data structure used to store and manage routing rules for data transmission between different devices. The core of this table is recording which source devices can send which types of messages to which target devices. For example, it can be a database table stored inside the vehicle's infotainment system, containing fields such as "source device identifier," "target device identifier," and "supported message types." The "source device identifier" uniquely identifies the source device of the playback data, such as the vehicle's infotainment system, the first connected device, or the second connected device; the "target device identifier" uniquely identifies the receiving device of the playback data, such as the vehicle's infotainment system, the first connected device, or the second connected device; and the "supported message types" specify the types of playback data that the target device can handle, such as video frames, audio frames, or control commands. The content routing table can be stored in the vehicle's local storage as a relational database or as an in-memory hash table structure for fast lookup.

[0049] The content routing table is established and maintained by the vehicle-mounted infotainment system (V2X), meaning that the V2X, as the core controller of the entire multi-screen interactive system, is responsible for collecting and managing the routing capability information of all connected devices. When a connected device (such as the first and second connected devices) establishes a communication connection with the V2X, the V2X can proactively query the capability information of these devices, such as the types of playback data they support, encoding formats, and display resolutions, and record this information in the content routing table. Furthermore, the V2X can dynamically update the content routing table by receiving device capability reports sent by connected devices, ensuring the real-time nature and accuracy of the routing information.

[0050] Upon receiving a packaged message, the system queries the content routing table based on the message's common fields and message type. The common fields include information such as the source device identifier and target device identifier, while the message type specifies the nature of the data. The system can extract the source device identifier and message type from the packaged message as query conditions to find all matching records in the content routing table. For example, if the source device identifier is "Mobile Phone A" and the message type is "video frame," the system will query the content routing table to find all target devices capable of receiving "video frames" sent by "Mobile Phone A."

[0051] Subsequently, the system determines the target device based on the query results returned by the content routing table. If the query results return multiple target devices that meet the criteria, the vehicle's infotainment system can select one or more of the most suitable target devices based on preset strategies, such as user preferences, device load, and display priority. For example, for video frames, it may prioritize target devices with high-resolution display capabilities; for audio frames, it may select all devices that support audio playback. In this way, the system can intelligently select the target device most suitable for playing data, achieving precise routing.

[0052] Step S304: The playback data is sent to the target device according to the connection mapping table, so that the playback data can be played through the target device. See details... Figure 1 Step S104 in the embodiment will not be described again here.

[0053] It is understood that, through the above technical solution, this application achieves intelligent addressing of data transmission between heterogeneous devices by introducing a content routing table mechanism. By using common fields in the encapsulated message, such as source device identifier and session identifier, along with message type, such as video, audio, and control commands, as a query index, it can accurately match target devices that support specific data types. The content routing table is uniformly established and maintained by the vehicle system, ensuring that the capabilities and status information of each device can be centrally managed in a multi-operating system environment, thereby solving the routing blind spot problem caused by the lack of a unified communication mechanism between heterogeneous systems. By dynamically determining the target device based on the query results, it not only avoids the bandwidth waste caused by blindly broadcasting data, but also ensures that playback data can be accurately routed to the corresponding display or processing terminal according to the device's capabilities, realizing precise control of data flow in multi-screen interaction scenarios, thereby effectively solving the problem of chaotic data distribution or inability to achieve efficient cross-device collaborative display caused by the lack of a unified and dynamic routing addressing logic. In some of the solutions mentioned above in this application, playback data is sent to the target device based on the content routing table to achieve multi-screen interaction. However, in the process of transmitting video streams across heterogeneous systems, due to the differences in hardware capabilities of different devices (vehicle system, external device, internal device), if the target device does not support the video encoding format sent by the source device, the target device will be unable to decode and play the video normally, resulting in failure of multi-screen interaction or abnormal display.

[0054] In this regard, this application further proposes that before sending playback data to the target device according to the connection mapping table, it also includes: Step a1: Determine whether the playback data contains video frames based on the message type of the encapsulated message.

[0055] Step a2: When video frames are included, determine whether the target device supports the video frame encoding format.

[0056] Step a3: If the target device supports the encoding format, then execute the step of sending playback data to the target device according to the connection mapping table.

[0057] Step a4: If the target device does not support the encoding format, the video frames will be transcoded before the step of sending the playback data to the target device according to the connection mapping table is executed.

[0058] Specifically, upon receiving playback data, the system first determines whether the playback data contains video frames based on the message type of the encapsulated message. This step aims to perform preliminary screening of the received playback data to identify data streams that require video encoding format compatibility processing. The encapsulated message contains a message type field, which characterizes the type of the data body. By parsing this message type field, the system can quickly determine whether the current data body is a video frame, audio frame, control command, or other types of data. This determination is a prerequisite for subsequent video format adaptation processing, avoiding unnecessary processing of non-video data and thus improving processing efficiency. For example, when the message type indicates a video frame, the system will further perform a video format compatibility check; if it is an audio frame or a control command, the video format processing step can be skipped directly.

[0059] When the system determines that the playback data contains video frames, it further determines whether the target device supports the video frame's encoding format. This step is crucial for achieving seamless video streaming between heterogeneous devices. After confirming that the playback data contains video frames, the system needs to understand the target device's video decoding capabilities. One implementation is that during the connection establishment or device capability reporting phase, the target device reports a list of supported video encoding formats to the vehicle's infotainment system, such as H.264, H.265, VP9, ​​and AV1. The vehicle's infotainment system maintains a device capability database and queries this database to obtain the encoding formats supported by the target device when it needs to send video frames to a specific target device. Another implementation is that the system can determine the supported encoding formats based on the target device's model, operating system version, and other information, using a preset compatibility configuration file or an online query service. This determination ensures that subsequently sent video streams can be correctly decoded and played by the target device.

[0060] If the target device is determined to support the encoding format of the current video frame, the system does not need to perform any additional processing and can directly execute the step of sending playback data to the target device according to the connection mapping table. This direct transmission method minimizes system overhead and transmission latency, maintains the original quality of the video stream, and optimizes resource utilization.

[0061] If the target device determines that it does not support the encoding format of the current video frame, the system transcodes the video frame before sending the playback data to the target device according to the connection mapping table to ensure normal video playback. Transcoding converts the video frame from one encoding format to another that the target device supports. One implementation is that the vehicle's infotainment system integrates a software transcoding module that can parse the original video frame and re-encode it into a format supported by the target device. Another implementation is that the vehicle's infotainment system can utilize its built-in hardware acceleration unit, such as a graphics processing unit (GPU) or a dedicated video encoder, to perform efficient hardware transcoding, reducing the CPU load and increasing transcoding speed. After transcoding is complete, the system sends the transcoded video frame to the target device through the connection mapping table. This step effectively solves the problem of mismatched encoding and decoding capabilities between heterogeneous devices, ensuring broad compatibility of video content.

[0062] It is understood that, through the above technical solution, this application effectively solves the playback compatibility problem caused by differences in video encoding and decoding capabilities between heterogeneous devices in multi-screen interaction scenarios. Specifically, before sending playback data to the target device, the system first accurately identifies the video frame based on the message type of the encapsulated message, avoiding unnecessary processing of non-video data. Subsequently, the system intelligently determines whether the target device supports the encoding format of the current video frame. If it does, it sends the data directly, thereby reducing unnecessary processing overhead and ensuring transmission efficiency and video quality. If the target device does not support the data, the system automatically transcodes the video frame, converting it into a format that the target device can recognize and decode, ensuring that the video content can be played normally and smoothly on target devices with different hardware capabilities and operating system environments. This mechanism, by introducing video encoding format compatibility detection and dynamic transcoding, achieves seamless video stream transmission and display across devices and systems, improving the user experience in multi-screen interaction scenarios in smart cockpits, avoiding playback failures or display abnormalities caused by format incompatibility, thereby enhancing the robustness and user satisfaction of the entire multi-screen interaction system. In some of the embodiments described above in this application, a scheme for determining the target device and routing based on the playback data type is proposed. However, when transmitting data across heterogeneous systems, due to the differences in display processing mechanisms among vehicle-mounted systems, external devices, and internal devices, simply sending data to the target device may result in the target device being unable to directly compatibility with or correctly render the received data. In particular, when video frame transmission is involved, different devices have different adaptability to encoding formats, and the lack of a targeted processing mechanism may lead to playback failure or display abnormalities.

[0063] In response, this application further proposes a method for multi-screen interaction in vehicles, such as... Figure 4 As shown, the method includes: Step S401: In response to the establishment of a communication connection with the vehicle's infotainment system, a connection mapping table is created for the connected devices. See details... Figure 1 Step S101 in the embodiment will not be described again here. Step S402: Receive playback data, wherein the playback data originates from the vehicle infotainment system, the first connected device, or the second connected device; see details below. Figure 1 Step S102 in the embodiment will not be described again here.

[0064] Step S403: Determine the target device to which the playback data needs to be routed based on the type of playback data. The target device includes at least one of the vehicle infotainment system, the first connection device, and the second connection device. Specifically, this includes: Step S4031: Query the content routing table based on the common fields and message type of the encapsulated message. The content routing table includes the source device identifier, the target device identifier, and the supported message types. The content routing table is established and maintained by the vehicle-mounted system. (See details...) Figure 3 Step S3031 in the embodiment will not be repeated here.

[0065] Step S4032: Determine the target device based on the query results returned by the content routing table. See details in [link / reference]. Figure 3 Step S3032 in the embodiment will not be described again here.

[0066] Step S404: The playback data is sent to the target device according to the connection mapping table, so that the playback data can be played through the target device. Specifically, this includes: Step S4041: If the target device is a vehicle infotainment system, the transcoded playback data is written into the display buffer of the vehicle infotainment system for playback on the vehicle infotainment system's playback device.

[0067] In step S4042, if the target device is the first connection device and / or the second connection device, the transcoded playback data is re-encapsulated into an encapsulated message and sent to the first connection device and / or the second connection device for playback.

[0068] "Writing the transcoded playback data into the vehicle's display buffer" refers to directly storing the converted playback data into a specific memory area of ​​the vehicle's infotainment system for display output. The display buffer is typically a memory space directly accessed by the graphics processing unit or display controller, used to temporarily store image frame data that will be displayed on the screen. By directly writing to the display buffer, high-level application processing and additional rendering pipelines can be bypassed, achieving efficient, low-latency data transmission and display. For example, the display memory can be directly manipulated by calling the low-level graphics application programming interface (API) provided by the vehicle's operating system, or the transcoded video frame data can be directly sent to its input queue using the vehicle's dedicated display driver or hardware acceleration module, with the driver responsible for writing it into the display buffer.

[0069] "Playing data on the vehicle's infotainment system's playback device" refers to ensuring that data is correctly received and displayed on the physical display screens connected to the infotainment system, such as the central control screen or the passenger-side screen, by writing data to the display buffer. The playback device in the vehicle typically refers to the physical display screen within the infotainment system. These screens connect to the main control chip of the vehicle's infotainment system through specific display interfaces, such as Mobile Industry Processor Display Serial Interface (MIPI DSI), Low Voltage Differential Signaling (LVDS), and DisplayPort. The display controller in the vehicle's infotainment system periodically reads data from the display buffer and transmits it to the physical display screen for display through these interfaces.

[0070] "Re-encapsulating transcoded playback data into an encapsulated message" refers to reconstructing the playback data, after encoding conversion, into a complete encapsulated message according to a predefined communication protocol format. This encapsulated message typically includes common fields such as the protocol version number, message generation timestamp, session identifier, source device identifier, target device identifier, message sequence number, and data body (e.g., video frames, audio frames, control commands), as well as a message type to characterize the data body type. The purpose of re-encapsulation is to ensure that data conforms to the target device's parsing standards during cross-device transmission, maintaining protocol consistency and enabling different devices to parse and process data based on a unified communication protocol. For example, transcoded video frames, audio frames, etc., can be used as the data body, combined with necessary metadata such as source device identifier, target device identifier, and message type, to reconstruct a complete message packet according to the system's internally defined protocol structure, potentially adding checksums, message headers, and other information.

[0071] "And send it to the first connected device and / or the second connected device for playback" means transmitting the re-encapsulated playback data to the target first connected device and / or the second connected device through the established connection channel, enabling them to receive, parse, and play it. The transmission process can utilize connection information recorded in the connection mapping table, such as IP address, port, and Bluetooth MAC address, and send the encapsulated message through appropriate network protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), or Bluetooth.

[0072] It is understood that, through the above technical solutions, this application achieves differentiated distribution processing of transcoded playback data, effectively solving the problem of display capability mismatch between heterogeneous devices. For the target device being an in-vehicle infotainment system, by directly writing the transcoded data into the display buffer, the underlying display driver mechanism of the in-vehicle infotainment system can be fully utilized, ensuring efficient and low-latency rendering of the data on the in-vehicle playback device, thereby resolving display obstacles caused by data format incompatibility between different systems. For the target device being an external or internal connected device, by re-encapsulating the transcoded data into a unified encapsulation message format, protocol consistency during data transmission is ensured, enabling external or internal devices to parse and play the data based on a unified communication protocol. This not only solves the problem of encoding format mismatch but also ensures the compatibility and stability of playback data on different hardware platforms through targeted data delivery methods, thereby achieving seamless multi-screen interaction across devices and systems and significantly improving the user experience. In some of the solutions mentioned above in this application, a routing mechanism was proposed to realize cross-device and cross-system playback data transmission. However, in multi-screen interaction scenarios, the display screens of different devices (such as car infotainment systems, mobile phones, and tablets) often have significant differences in physical attributes such as resolution and aspect ratio. When the source device sends control commands to the target device, if the touch coordinates of the source device are directly used, the target device will not be able to accurately identify the user's interaction intent, thereby causing touch offset or operation failure, and failing to achieve accurate interaction between devices.

[0073] In response, this application further proposes a method for optimizing multi-screen interactive experience, which involves playing playback data through a target device, specifically including the following steps: Step b1: Determine whether the playback data contains control frames based on the message type.

[0074] Step b2: When a control frame is included, the first touch coordinates of the source device to the display screen are converted into the second touch coordinates of the target display plane based on the differences between the display screens of the source device and the target device. The differences between the display screens include resolution differences and aspect ratio differences. Step b3: Respond to the second touch coordinate to display the touch animation.

[0075] First, the system determines whether the playback data contains control frames based on the message type. Specifically, the message type field can predefine one or more specific values ​​to identify control frames. For example, an enumeration value can be set to explicitly indicate that the data body is a control command. Additionally, a specific bit flag can be set in the message type field; when this bit flag is activated, it indicates that the playback data contains control frames. In this way, the system can accurately identify data packets that require interactive coordinate transformation, thereby avoiding unnecessary processing of non-interactive data such as video or audio streams.

[0076] Based on this, when the system confirms that the playback data contains control frames, it will initiate a touch coordinate conversion mechanism. According to the differences between the display screens of the source device and the target device, the first touch coordinates on the source device's display screen are converted into second touch coordinates on the target display plane. These differences include resolution and aspect ratio differences. The source device refers to the device that sends touch commands, such as a user's mobile phone or in-vehicle infotainment system, while the target device is the device that receives touch commands and displays them, such as an instrument panel or passenger-side screen. The first touch coordinates are the original coordinates generated when the user performs a touch operation on the source device's display screen. Since the display screens of the source and target devices may have significant differences in resolution and aspect ratio, directly using the first touch coordinates will lead to inaccurate interaction. Therefore, this application converts the first touch coordinates into second touch coordinates suitable for the target display plane by calculating the differences between the display screens of the source and target devices. For example, a scaling conversion can be used. If the source device resolution is (W1, H1) and the target device resolution is (W2, H2), the X coordinate can be converted using X2 = X1 x (W2 / W1), and the Y coordinate using Y2 = Y1 x (H2 / H1). For aspect ratio differences, adjustments can be made according to preset strategies, such as centering, scaling proportionally with black borders, or stretching to ensure consistency between the touch area and the displayed content. Furthermore, devices can dynamically negotiate and exchange their screen parameters during connection, thereby calculating the optimal conversion parameters in real time, or a set of mapping rules for common device combinations can be preset to achieve precise touch coordinate mapping.

[0077] Finally, the system displays a touch animation in response to the second touch coordinates. To provide intuitive user feedback, on the target device, the system will display a corresponding touch animation in response to the converted second touch coordinates. For example, a brief circular ripple animation, a click effect, or a highlighted area can be displayed at the second touch coordinate location to visually confirm that the user's action has been received and processed. Depending on the type of touch operation, such as click, long press, or swipe, different animation effects can also be displayed to further enhance the user experience and the accuracy of the interaction.

[0078] It is understood that, through the above technical solution, this application effectively solves the problem of inaccurate touch interaction caused by inconsistent physical parameters of heterogeneous device display screens, such as differences in resolution and aspect ratio, in multi-screen interaction scenarios. By accurately identifying control frames and transforming touch coordinates based on screen differences, it ensures that the interaction intent of the source device can be accurately parsed and executed by the target device, thereby achieving a precise and unified touch interaction experience across heterogeneous devices. This improves the intuitiveness and accuracy of user operation in the multi-screen environment of the smart cockpit, avoids misoperations or operation failures caused by coordinate mismatch, and further improves the user experience at the interaction level on the basis of the basic playback data routing mechanism, building a more complete and efficient multi-screen interaction ecosystem for smart cockpits. In some of the solutions mentioned above in this application, a method for multi-screen interaction by encapsulating messages was proposed. However, in this process, the system lacks a real-time perception and feedback mechanism for the device status, which makes it impossible for the vehicle system to know in a timely manner the changes in the hardware capabilities of the connected devices, the stability of the connection, and the integrity of data transmission. As a result, it is difficult to achieve dynamic resource scheduling and fault self-healing, which affects the reliability of multi-screen interaction and user experience.

[0079] In this regard, this application further proposes that the playback data also include device capability reports, heartbeat messages, and confirmation messages, and the method further includes: Step c1: Update the hardware parameter description of the target device in the content routing table based on the device capability report.

[0080] Step c2: Determine the connection status of the first connected device and / or the second connected device based on the heartbeat message.

[0081] Step c3: Based on the confirmation message, determine whether the playback data received by the first connected device and / or the second connected device has been lost.

[0082] Specifically, a device capability report is structured data containing information about the hardware and software capabilities of connected devices. This information may include, but is not limited to, the device's screen resolution, supported video encoding formats, audio decoding capabilities, processor performance, memory size, operating system version, supported input methods, and network connection type. Its purpose is to provide the vehicle's infotainment system with detailed technical specifications of the connected devices, enabling the system to make intelligent resource allocation and data routing decisions based on this information. In one implementation, the connected device proactively generates and sends a complete device capability report when it first connects to the vehicle's infotainment system; subsequently, when the device's capabilities change, the device sends an updated capability report. In another implementation, the vehicle's infotainment system can periodically send capability query requests to the connected device. Upon receiving the request, the connected device collects and generates the current device capability report in real time and sends it back to the vehicle's infotainment system. The device capability report can be encoded using standard data formats such as JSON and XML for easy parsing and storage by the vehicle's infotainment system.

[0083] A heartbeat message is a periodically sent short message used to indicate the active status of the sending device and the persistence of the connection. Its function is to maintain connection activity detection, ensuring that the vehicle's infotainment system can monitor the online status of the first and / or second connected devices in real time. In one implementation, the first and / or second connected devices send a predefined heartbeat packet to the vehicle's infotainment system at fixed time intervals; if the vehicle's infotainment system does not receive a heartbeat packet within a preset timeout period, it considers the device offline or the connection abnormal. In another implementation, the vehicle's infotainment system can send a heartbeat request to the first and / or second connected devices and wait for a heartbeat response. This request-response mechanism can more accurately detect the health status of the bidirectional connection. Heartbeat messages can contain only a simple identifier and timestamp to minimize network overhead.

[0084] An acknowledgment message is a feedback mechanism used to indicate that a data packet or message has been successfully received and processed by the receiver. Its function is to ensure the integrity and reliability of playback data during transmission, preventing playback abnormalities caused by data loss. In one implementation, when the first and / or second connected devices receive playback data sent by the vehicle-mounted system, they send an acknowledgment message back to the system. This message contains the sequence number or hash value of the received data for the system to verify. In another implementation, the system can set a timeout retransmission mechanism for each data packet when sending playback data; if no corresponding acknowledgment message is received within the set time, the system considers the data packet lost and retransmits it. Acknowledgment messages can use a cumulative acknowledgment method, meaning one acknowledgment message can acknowledge multiple consecutively received data packets, thereby reducing the number of acknowledgment messages.

[0085] This approach, which updates the content routing table based on device capability reports to describe the hardware parameters of the target device, aims to dynamically update the content routing table maintained internally by the vehicle's infotainment system (V2S) based on the information in the device capability report, ensuring the accuracy of routing decisions. In one implementation, after receiving the device capability report, the V2S parses the report content, extracts key hardware parameters such as screen resolution and supported encoding formats, and associates or overwrites these parameters with the corresponding device entries in the content routing table. In another implementation, the update operation can trigger a re-evaluation of the routing policy. For example, if a device reports that it supports H.265 encoding, the V2S can prioritize H.265 encoding when sending video streams to that device to improve transmission efficiency.

[0086] The connection status of the first and / or second connected devices is determined based on heartbeat messages, enabling real-time monitoring of the online status of connected devices so that the vehicle's infotainment system can adjust data transmission strategies in a timely manner. In one implementation, the vehicle's infotainment system maintains a list of connected device statuses. When a heartbeat message is received from a device, the system updates the device's "last active timestamp." Simultaneously, the system runs a background process that periodically checks the "last active timestamps" of all devices. If a device's timestamp exceeds a preset threshold, it is marked as "offline" or "connection abnormal." In another implementation, when a device is detected to be offline, the vehicle's infotainment system can automatically remove the device's entry from the connection mapping table and content routing table, and notify relevant applications to stop sending data to that device.

[0087] The system uses acknowledgment messages to determine whether playback data received by the first and / or second connected devices has been lost. This is used to detect the integrity of data transmission and take appropriate corrective measures when data loss is detected. In one implementation, the vehicle-mounted system assigns a unique sequence number to each encapsulated message when sending playback data. When an acknowledgment message is received, the system checks the sequence number contained in the acknowledgment message and compares it with previously sent but unacknowledged sequence numbers. If a sequence number is missing, the corresponding playback data is considered lost. In another implementation, once data loss is detected, the vehicle-mounted system can trigger a retransmission mechanism to resend the lost data packets. Alternatively, if the lost data is streaming media with high real-time requirements, the system can choose to skip the lost data to ensure playback continuity.

[0088] It is understood that, through the above technical solutions, this application constructs a closed-loop device management and data transmission monitoring system, thereby improving the robustness of the multi-screen interaction system. Specifically, based on the hardware parameter descriptions of the target device in the updated content routing table of the device capability report, the vehicle-mounted system can keep track of the hardware specification changes of each connected device in real time, such as screen resolution and supported encoding formats. This allows for dynamic matching of the most suitable device during routing decisions, avoiding display abnormalities or performance bottlenecks caused by hardware capability mismatches and optimizing resource utilization. Simultaneously, the connection status of the first and / or second connected devices is determined based on heartbeat messages. Through periodic status checks, the vehicle-mounted system can promptly detect offline or abnormal disconnections of devices, quickly removing invalid devices from the connection mapping table to prevent data from being sent to disconnected devices. This effectively reduces the unnecessary occupation of system resources and improves system response speed and stability. Furthermore, the system determines whether playback data received by the first and / or second connected devices has been lost based on confirmation messages. Through a feedback confirmation mechanism, the vehicle-mounted system can accurately locate packet loss during data transmission, providing a basis for subsequent data retransmission or error handling. This ensures the integrity and synchronization of playback content in multi-screen interaction and significantly improves the user experience. The introduction of these mechanisms effectively solves the problem of the system lacking a real-time perception and feedback mechanism for device status, realizes dynamic resource scheduling and fault self-healing, and significantly improves the reliability and user experience of multi-screen interaction. In one example, such as Figure 5 As shown, the above technical solution will be explained in more detail below through a more specific example: In a smart cockpit environment, the vehicle's infotainment system serves as the core control unit and has its own independent operating system. User A uses their smartphone as the first connection device, which also has its own independent operating system. The vehicle's instrument cluster serves as the second connection device, and similarly, it also has its own independent operating system.

[0089] When user A enters the vehicle and starts the infotainment system, the system establishes a wireless communication connection with the smartphone. The system then creates a connection mapping table for the smartphone and instrument cluster. This table records information such as the network address, device type, and current connection status of the smartphone and instrument cluster, ensuring the system can recognize and address these connected devices.

[0090] User A wants to project route information from a third-party navigation app running on their smartphone onto the vehicle's instrument cluster for display. In this case, the smartphone, as the source device for the playback data, encapsulates the video frames of the navigation route, the audio frames of the voice navigation, and the control commands generated by the user's map zooming and panning operations on the phone into encapsulated messages. Each encapsulated message includes common fields, a data body, and a message type. The common fields include the protocol version number, the timestamp of the message generation, the session identifier, the source device identifier (smartphone), the target device identifier (instrument cluster), and the message sequence number. The message type explicitly represents the type of data body, such as "video frame," "audio frame," or "control command."

[0091] The vehicle's infotainment system receives playback data in the form of encapsulated messages from smartphones. Based on the common fields and message type of the encapsulated messages, the system queries its internally maintained content routing table. This table is pre-configured with mappings between different source device identifiers, target device identifiers, and the message types they support. For example, the content routing table indicates that navigation video frames from a smartphone should be routed to the instrument cluster, while navigation audio frames should be routed to the infotainment system's own speaker system. Control commands are routed to either the instrument cluster or the infotainment system based on their specific function. Through the query results, the infotainment system determines that the target device for navigation video frames is the instrument cluster, the target device for navigation audio frames is the infotainment system, and the target device for control commands is the instrument cluster.

[0092] Before sending navigation video frames to the instrument cluster, the vehicle's infotainment system determines whether the encapsulated message contains video frames based on its message type. If a video frame is confirmed, the system further checks if the instrument cluster supports its encoding format. If the instrument cluster does not support the encoding format, it transcodes the video frame to a format supported by the instrument cluster, such as converting from H.265 to H.264. This avoids issues where the instrument cluster cannot play the video due to encoding incompatibility.

[0093] Subsequently, the vehicle's infotainment system sends the processed playback data to the corresponding target device according to the connection mapping table. For the transcoded navigation video frames and control commands, the system re-encapsulates them into encapsulated messages and sends them to the instrument cluster via an internal communication interface. Upon receiving these encapsulated messages, the instrument cluster parses them and displays the navigation map and related information. For navigation audio frames, the system writes them to its audio playback buffer and plays them through the speakers.

[0094] During navigation, if user A zooms in or out of the navigation map on their smartphone, the smartphone generates an encapsulated message containing a control frame and sends it to the vehicle's infotainment system. Upon receiving this control frame, the system identifies it as such based on its message type. Because of the differences in resolution and aspect ratio between the smartphone and the instrument cluster screens, the system converts the first touch coordinates from the smartphone's touch input into second touch coordinates on the instrument cluster display. For example, if the smartphone screen resolution is 1920x1080 and the instrument cluster screen resolution is 1280x720, the system adjusts the touch coordinates accordingly. Upon receiving the converted second touch coordinates, the instrument cluster responsively displays a corresponding touch animation on the navigation map, such as a map zoom animation, thus achieving a seamless cross-screen interactive experience.

[0095] In addition, the smartphone periodically sends device capability reports to the vehicle's infotainment system, detailing its supported video encoding formats, audio formats, and other hardware parameters. Based on these reports, the infotainment system updates the hardware parameter descriptions for the smartphone in its content routing table, enabling more accurate judgments in subsequent data routing and transcoding decisions. The smartphone and instrument cluster also periodically send heartbeat messages, which the infotainment system receives to determine their connection status. If a connection anomaly is detected, it can be addressed promptly. Furthermore, after sending playback data, the infotainment system waits for confirmation messages from the smartphone or instrument cluster to confirm successful data reception and detect any data loss during transmission.

[0096] Using the above method, User A's smartphone, as an external device, can directly and efficiently project its customized navigation content onto the instrument cluster, an internal device, solving the problem of the lack of a direct interconnection channel between external devices and the instrument cluster in existing technologies. The vehicle's infotainment system, acting as a unified communication hub, establishes connection mapping tables and content routing tables, and combines data encapsulation, transcoding, and touch coordinate conversion mechanisms to achieve unified communication and multi-screen interaction across heterogeneous systems. This avoids redundant encoding and transmission caused by each screen independently requesting data from the source device, effectively saving bandwidth and reducing system load.

[0097] This embodiment also provides a vehicle multi-screen interaction device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0098] This embodiment provides a vehicle multi-screen interactive device, such as... Figure 6 As shown, it includes: Module 601 is established to establish a connection mapping table for the connected devices in response to the establishment of a communication connection with the vehicle infotainment system. The connected devices include a first connected device and a second connected device. The first connected device is an external device of the vehicle infotainment system, and the second connected device is an internal device of the vehicle infotainment system. The vehicle infotainment system, the first connected device, and the second connected device all have independent operating systems deployed.

[0099] The receiving module 602 is used to receive playback data, wherein the playback data originates from the vehicle-mounted system, the first connected device, or the second connected device.

[0100] The determination module 603 is used to determine the target device to which the playback data needs to be routed based on the type of the playback data. The target device includes at least one of the vehicle-mounted system, the first connection device, and the second connection device.

[0101] The sending module 604 is used to send playback data to the target device according to the connection mapping table, so that the playback data can be played through the target device.

[0102] In some alternative implementations, the determining module 603 includes: The first unit is used to query the content routing table based on the common fields and message types of the encapsulated message. The content routing table includes the source device identifier, the target device identifier, and the supported message types. The content routing table is established and maintained by the vehicle's infotainment system.

[0103] The second unit is used to determine the target device based on the query results returned by the content routing table.

[0104] The vehicle multi-screen interaction device provided in this application can execute the vehicle multi-screen interaction method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0105] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0106] The following is a detailed reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from memory 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0107] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0108] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from memory 708, or installed from ROM 702. When the computer program is executed by processor 701, it performs the functions defined in the vehicle multi-screen interaction method of embodiments of this application.

[0109] Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0110] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the vehicle multi-screen interaction method shown in the above embodiments is implemented.

[0111] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0112] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for multi-screen interaction in a vehicle, characterized in that, Applied to in-vehicle infotainment systems, the method includes: In response to the establishment of a communication connection with the vehicle infotainment system, a connection mapping table is established for the connected devices. The connected devices include a first connected device and a second connected device. The first connected device is an external device of the vehicle infotainment system, and the second connected device is an internal device of the vehicle infotainment system. The vehicle infotainment system, the first connected device, and the second connected device all have independent operating systems deployed. Receive playback data, wherein the playback data originates from the vehicle infotainment system, the first connection device, or the second connection device; The target device to which the playback data needs to be routed is determined based on the type of the playback data, and the target device includes at least one of the vehicle-mounted system, the first connection device, and the second connection device. The playback data is sent to the target device according to the connection mapping table, so that the playback data can be played through the target device.

2. The method according to claim 1, characterized in that, The playback data is encapsulated into an encapsulated message when it is sent. The encapsulated message includes common fields, data body and message type. The common fields include protocol version number, message generation timestamp, session identifier, source device identifier, target device identifier and message sequence number. The message type is used to characterize the type of the data body and includes at least one of video frame, audio frame and control command.

3. The method according to claim 2, characterized in that, Determining the target device to which the playback data needs to be routed based on the type of the playback data includes: The content routing table is queried based on the common fields of the encapsulated message and the message type. The content routing table includes the source device identifier, the target device identifier, and the supported message types. The content routing table is established and maintained by the vehicle system. The target device is determined based on the query results returned by the content routing table.

4. The method according to claim 3, characterized in that, Before sending the playback data to the target device according to the connection mapping table, the method further includes: Determine whether the playback data contains video frames based on the message type of the encapsulated message; When the video frame is included, determine whether the target device supports the encoding format of the video frame; If the target device supports the encoding format, then the step of sending the playback data to the target device according to the connection mapping table is executed; If the target device does not support the encoding format, the video frame will be transcoded before the step of sending the playback data to the target device according to the connection mapping table is executed.

5. The method according to claim 4, characterized in that, Sending the playback data to the target device according to the connection mapping table includes: If the target device is an in-vehicle infotainment system, the transcoded playback data is written into the display buffer of the in-vehicle infotainment system for playback on the playback device of the in-vehicle infotainment system; If the target device is the first connecting device and / or the second connecting device, the transcoded playback data is re-encapsulated into the encapsulated message and sent to the first connecting device and / or the second connecting device for playback.

6. The method according to claim 3, characterized in that, The step of playing the playback data through the target device further includes: Determine whether the playback data contains control frames based on the message type; When the control frame is included, the first touch coordinates of the source device to the display screen are converted into the second touch coordinates of the target display plane according to the differences between the display screens of the source device and the target device. The differences between the display screens include resolution differences and aspect ratio differences. The touch animation is displayed in response to the second touch coordinate.

7. The method according to claim 2, characterized in that, The playback data also includes device capability reports, heartbeat messages, and confirmation messages; the method further includes: The hardware parameter description of the target device is updated in the routing table based on the updated device capability report. The connection status of the first connection device and / or the second connection device is determined based on the heartbeat message; Based on the confirmation message, it is determined whether the playback data received by the first connected device and / or the second connected device has been lost.

8. A vehicle multi-screen interactive device, characterized in that, The device includes: A connection establishment module is used to establish a connection mapping table for the connected devices in response to the establishment of a communication connection with the vehicle infotainment system. The connected devices include a first connected device and a second connected device. The first connected device is an external device of the vehicle infotainment system, and the second connected device is an internal device of the vehicle infotainment system. The vehicle infotainment system, the first connected device, and the second connected device are all deployed with independent operating systems. A receiving module is used to receive playback data, wherein the playback data originates from the vehicle infotainment system, the first connecting device, or the second connecting device; The determination module is used to determine the target device to which the playback data needs to be routed based on the type of the playback data, wherein the target device includes at least one of the vehicle-mounted system, a first connection device, and a second connection device; The sending module is used to send the playback data to the target device according to the connection mapping table, so that the playback data can be played through the target device.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle multi-screen interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the vehicle multi-screen interaction method according to any one of claims 1 to 7.

Images