Vehicle-machine multi-screen cooperative display and interaction method and system, and storage medium

Abstract

The application relates to the technical field of Internet of Vehicles, and discloses a car-machine multi-screen cooperative display and interaction method, which comprises the following steps: maintaining a global session and a member view through a centralized session arranger, realizing session access and dynamic member management of multi-screen devices, and determining multi-screen related parameters; constructing a full-order execution rule based on a logical clock, the multi-screen related parameters and an operation identifier, and assigning a global serial number to each multi-screen operation; obtaining a focus of an operation request, and determining feedback parameters through a soft lock and a priority preemption strategy; periodically hashing and checking based on a shadow state and a domain Merkle tree, positioning state differences according to the feedback parameters, and performing incremental repair through an incremental patch; and adopting WebRTC layered coding and a weighted bandwidth allocation strategy, preferentially guaranteeing control signaling and a main driving screen experience, and dynamically adapting in a bandwidth-limited scene. The application can solve the problems of multi-screen interaction inconsistency and insufficient real-time performance, thereby improving the multi-screen cooperative experience in a vehicle.

Description

Technical Field

[0001] This invention relates to the field of vehicle networking technology, and in particular to a method, system and storage medium for multi-screen collaborative display and interaction in a vehicle. Background Technology

[0002] With the rapid development of smart cockpit technology, in-vehicle infotainment systems have evolved from traditional single display terminals to interactive systems encompassing multiple screen types, including the driver's screen, passenger screen, central control screen, and rear entertainment screen. Existing multi-screen collaboration solutions mainly employ either mirroring or independent control modes. Mirroring mode suffers from high interaction latency and inconsistent screen states; independent control mode lacks unified operating logic, resulting in inconsistencies in content, asynchronous states, and broken interactive experiences when users use multiple devices simultaneously, as the main screen's operations cannot be synchronized with other devices in real time.

[0003] Furthermore, existing technologies lack a consistent quality control mechanism for multi-screen interaction, which cannot guarantee that multiple screens meet the same standards in terms of resolution adaptation, interaction latency, and response speed. This seriously affects the practicality and reliability of in-vehicle multi-screen collaboration and makes it difficult to meet users' needs for smoothness, security, and consistency in multi-screen collaborative operation.

[0004] Therefore, how to achieve multi-screen collaborative display and interaction, and improve the multi-screen collaborative experience of smart cockpits, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention provides a method, system, and storage medium for multi-screen collaborative display and interaction in vehicles, in order to solve problems such as inconsistent interaction, insufficient real-time performance, lack of security, and low resource utilization in existing smart cockpit multi-screen collaborative solutions.

[0006] To achieve a multi-screen collaborative display and interactive control method that enables orderly execution of multi-screen operations, rapid status synchronization, dynamic permission adaptation, and stable and controllable quality, and to comprehensively improve the multi-screen collaborative experience in the smart cockpit, this invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a method for multi-screen collaborative display and interaction in a vehicle-mounted system, applied to an electronic device, the method comprising: A centralized session orchestrator is used to maintain a global session and member view, enabling session access and dynamic member management for multi-screen devices, and determining multi-screen related parameters; Based on the logical clock, the multi-screen related parameters, and the operation identifier, a total order execution rule is constructed, and a global sequence number is assigned to each multi-screen operation so that the multi-screen operations are executed in a unified order. Based on the global serial number, the multi-screen related parameters, and the operation request, the focus of the operation request is obtained, and the feedback parameters are determined through a soft lock and priority preemption strategy. Periodic hash reconciliation is performed based on shadow state and domain Merkle tree. State differences are located according to the feedback parameters and incremental repair is performed through incremental patching to ensure consistency of multi-screen state. We adopt WebRTC layered encoding and weighted bandwidth allocation strategies to prioritize control signaling and the user experience of the main control screen, and perform dynamic adaptation in bandwidth-constrained scenarios.

[0007] In addition, optional technical solutions include: multi-screen time synchronization and frame alignment, hierarchical permission and contextualized strategies, and quality standardization and compliance control; among which, The multi-screen time synchronization and frame alignment includes using the session clock as a reference, combined with the playback timestamp and jitter buffering mechanism, to achieve millisecond-level synchronization and alignment of multi-screen video and audio; The permission hierarchy and contextualization strategy includes a combination of RBAC and ABAC models, combined with the multi-screen related parameters and vehicle condition dynamic control operation permissions. The aforementioned quality standardization and compliance control includes setting hard QoE indicators and implementing a closed-loop mechanism through monitoring, self-healing, alarms, and degradation.

[0008] In addition, an optional technical solution is that the multi-screen related parameters include roles, permissions, and priorities; A centralized session orchestrator maintains the global session and member view, enabling session access and dynamic member management for multi-screen devices, and determining multi-screen related parameters, including: The terminal member requests to join the global session, carrying their account identifier and page identifier. The account identifier is verified by the session orchestrator. If the verification passes, the terminal member is written into the session table and assigned the corresponding role, permission and priority. The session orchestrator broadcasts a member view with new members; wherein, when any member leaves, becomes disconnected, or expires, the session orchestrator updates the member view and broadcasts it.

[0009] In addition, an optional technical solution is to construct a total-order execution rule based on the logical clock, the multi-screen related parameters, and the operation identifier, and assign a global sequence number to each multi-screen operation so that the multi-screen operations are executed in a unified order, including: The terminal generates a local operation identifier and a logical clock, and submits them to the session orchestrator, which determines the global sequence number of the operation. The session orchestrator writes the global sequence number into the global operation log and broadcasts it. Each terminal will put the operation into the local execution queue and report it for confirmation.

[0010] In addition, an optional technical solution is that the feedback parameters include focus granting, rejection, preemption notification and the corresponding soft lease; Based on the global sequence number, the multi-screen related parameters, and the operation request, the focus of the operation request is obtained, and feedback parameters are determined through a soft lock and priority preemption strategy, including: Get the focus of the terminal operation request; The session orchestrator determines the focus conflicts and priorities; Based on the aforementioned conflicts and priorities, change operations are issued. During the lease renewal period, if other terminals attempt to modify the service, the request will be rejected or the terminal will be queued for further processing. The request is processed according to the stated priority, the feedback parameters are determined, and the announcement is published through the session orchestrator.

[0011] In addition, an optional technical solution is to perform periodic hash reconciliation based on shadow state and domain-specific Merkle trees, locate state differences according to the feedback parameters, and perform incremental repair through incremental patching to ensure consistency of multi-screen states, including: Perform the operation for each global sequence number, and update the local shadow state simultaneously; The segmented hash of each domain is calculated every preset period and reported to the session orchestrator. The hash results of different screens are then compared through the session orchestrator. If the hashes of different screens are consistent, skip them; otherwise, if they are inconsistent, locate the difference segment, specify the reference source through the session orchestrator, and generate an incremental patch broadcast. Each terminal applies the incremental patch and then sends back the new hash until the hashes of all terminals are consistent.

[0012] In addition, an alternative technical solution is to use the session clock as a reference, combined with playback timestamps and a jitter buffering mechanism, to achieve millisecond-level synchronization alignment of multi-screen video and audio, including: All screens are clock-calibrated with the session orchestrator to obtain a local clock offset, so that the terminal corrects its local clock according to the local clock offset. The jitter buffer determines whether to drop frames, wait for frames, or repeat frames, ensuring that the screen display is aligned with the target playback time; if the jitter exceeds the threshold, small step frequency correction is enabled. Regularly report audio and video time differences and screen-to-screen time differences, and adjust the corresponding alignment parameters through the session orchestrator; Based on the alignment parameters, the video and audio of each screen are aligned and adjusted.

[0013] In addition, an optional technical solution is that the QoE hard metrics include latency, frame rate, and synchronization accuracy.

[0014] Secondly, embodiments of the present invention also provide a vehicle-mounted multi-screen collaborative display and interaction system, comprising: The member management unit is used to maintain the global session and member view through a centralized session orchestrator, realize session access and dynamic member management for multi-screen devices, and determine multi-screen related parameters; The sequence number allocation unit is used to construct a full-order execution rule based on the logical clock, the multi-screen related parameters and the operation identifier, and allocate a global sequence number to each multi-screen operation so that the multi-screen operations are executed in a unified order. The parameter determination unit is used to obtain the focus of the operation request based on the global sequence number, the multi-screen related parameters and the operation request, and determine the feedback parameters through a soft lock and priority preemption strategy. The repair unit is used to perform periodic hash reconciliation based on shadow state and domain Merkle tree, locate state differences according to the feedback parameters, and perform incremental repair through incremental patching to make the multi-screen state consistent. The dynamic adaptation unit is used to adopt WebRTC layered encoding and weighted bandwidth allocation strategies to prioritize control signaling and the main screen experience, and to perform dynamic adaptation in bandwidth-constrained scenarios.

[0015] Thirdly, the present invention also provides a computer storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the vehicle-mounted multi-screen collaborative display and interaction method as described above.

[0016] By utilizing the in-vehicle multi-screen collaborative display and interaction method, system, and storage medium provided by the present invention, it is possible to ensure that multiple screens achieve a unified standard in terms of resolution, latency, and interactive response, thereby improving the practicality and reliability of in-vehicle multi-screen collaboration.

[0017] It is understood that the electronic device described in the second aspect and the computer storage medium described in the third aspect are both used to execute the corresponding method in the first aspect provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding method provided above, and will not be repeated here. Attached Figure Description

[0018] Figure 1 This is a flowchart of the in-vehicle multi-screen collaborative display and interaction method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the multi-screen session and account binding algorithm of an embodiment of this application; Figure 3 This is a flowchart of the operational consistency and total order arbitration algorithm in an embodiment of this application; Figure 4 This is a flowchart of the conflict detection, focus, and soft-lock lease algorithm in an embodiment of this application; Figure 5 This is a flowchart of the state synchronization and incremental repair algorithm according to an embodiment of this application; Figure 6 This is a flowchart of the conflict detection and focus lease algorithm according to an embodiment of this application; Figure 7 This is a flowchart illustrating multi-screen time synchronization and frame alignment according to an embodiment of the present invention; Figure 8 This is a flowchart illustrating the permission hierarchy and contextualization strategy of an embodiment of the present invention; Figure 9 This is a flowchart illustrating the quality standardization and compliance control in an embodiment of the present invention; Figure 10 This is a logical block diagram of the multi-screen collaborative display and interaction of the vehicle system according to an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0021] In the embodiments of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0022] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0023] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention provides a method for multi-screen collaborative display and interaction in a vehicle infotainment system. (Refer to...) Figure 1 The diagram shown is a flowchart illustrating the in-vehicle multi-screen collaborative display and interaction method provided in Embodiment 1 of the present invention. This method can be executed by an electronic device, which can be implemented in software and / or hardware.

[0025] In this embodiment, the in-vehicle multi-screen collaborative display and interaction method includes: S100: Maintains global session and member views through a centralized session orchestrator, enabling session access and dynamic member management for multi-screen devices, and determining multi-screen related parameters.

[0026] The multi-screen related parameters may include roles, permissions, and priorities. During the application process, terminal members request to join the global session by carrying account identifiers and page identifiers. The session orchestrator verifies the account identifier. If the verification is successful, the terminal member is written into the session table and assigned corresponding roles, permissions, and priorities. The session orchestrator broadcasts the member view with the new member. When any member leaves, becomes disconnected, or expires, the session orchestrator updates the member view and broadcasts it. It can set a limit on concurrent logins of the same account to prevent abuse. Furthermore, cross-vehicle device migration triggers epoch switching, which can avoid the pollution of old session messages.

[0027] Specifically, this step is implemented based on a multi-screen session and account binding algorithm. Figure 2 The detailed process of the multi-screen session and account binding algorithm according to an embodiment of the present invention is shown.

[0028] like Figure 2 It can be seen that the multi-screen session and account binding algorithm involved in step S100 includes: S11: The terminal initiates a join request. After starting or accessing the cockpit network, the terminal, such as the driver's screen, passenger's screen, rear screen, external Pad, mobile phone, etc., sends a join session request to SO (Session Orchestrator) with AccountID, ScreenID and Capabilities. S12: Account verification and member configuration. After receiving the request in SO, verify the validity of the account token corresponding to AccountID. If the verification is successful, write the terminal into the session table and assign the "driver" role according to the terminal type, such as the screen connected to the driver's seat, and assign the role and priority according to the user's preset configuration. S13: After SO updates the global member view, it broadcasts the new view to all terminals that have joined the session. The broadcast content includes Epoch (session processing cycle), StateVersion (global increment), and the complete member list. S14: After receiving the view, the terminal sends a heartbeat packet to the SO at a preset period (e.g., 100ms) to update HeartbeatAt (heartbeat timestamp); the SO periodically checks the HeartbeatAt of each terminal. If the time difference exceeds the LeaseTTL (validity period, e.g., 300ms), the terminal is determined to be disconnected and removed from the session table, thus realizing heartbeat and lease management. S15: When any terminal actively exits (e.g., the user disconnects the external Pad connection) or is determined to be disconnected, SO immediately updates the member view and rebroadcasts; if the session restarts (e.g., the system restarts) or the master terminal switches (e.g., the master screen malfunctions and the passenger screen temporarily becomes the master), the Epoch is incremented by 1. After all terminals receive the new Epoch, they discard all messages corresponding to the old Epoch and complete the dynamic adjustment of members. S16: The terminal caches the last received consistent member view locally. When reconnecting to SO after a brief network disconnection, the cooperative state can be quickly restored based on the cached view without having to re-execute the complete joining process, thus achieving optimization of instantaneous disconnection and reconnection.

[0029] In one specific embodiment of the present invention, the member view convergence time (the time from when the terminal joins / exits to when all screens synchronously update the view) is ≤200ms; the member removal detection time (the time from when the terminal loses connection to when the SO removes it from the session table) is ≤2 × heartbeat cycle (e.g., if the heartbeat cycle is 100ms, the removal detection is ≤200ms). In practical applications, if the co-pilot Pad connects to the cockpit network, the UI of all screens will display "co-pilot has joined" within 150ms, and the Pad will be assigned "secondary master control" permissions, which can assist the driver in performing some operations, such as searching for navigation destinations.

[0030] S200: Construct a full-order execution rule based on the logical clock, the multi-screen related parameters, and the operation identifier, and assign a global sequence number to each multi-screen operation so that the multi-screen operations are executed in a unified order.

[0031] This step assigns a globally unique and ordered sequence number (GSeq) to each terminal-initiated operation. A total order relationship is constructed based on the logical clock (LamportClock, used to record the relative time of the operation) and role priority. All terminals execute operations strictly in ascending order of GSeq. Operation-related fields include: OpID (unique operation identifier), ScreenID (screen identifier initiating the operation), OpType (operation type, such as play, pause, seek, source switching, destination switching, etc.), Payload (operation parameters, such as the song ID when switching songs, destination coordinates during navigation, etc.), and Idempotent (idempotent flag, used to avoid repeatedly executing the same operation).

[0032] Specifically, this step can be implemented based on operational consistency and total order arbitration algorithms. Figure 3 The detailed process of the operational consistency and total order arbitration algorithm according to an embodiment of the present invention is shown.

[0033] like Figure 3 It can be seen that the operational consistency and total order arbitration algorithm in the embodiments of the present invention may include: S21: After the terminal user initiates an operation (such as the driver clicking "pause" media), the terminal generates a unique OpID (operation identifier) ​​locally and updates the local LamportClock (Lamport clock, with an initial value of 0, which increments by 1 each time an operation is initiated / received), generating a local operation; S22: The terminal submits an operation request containing OpID, ScreenID, LamportClock, OpType, and Payload to the SO; after receiving the request, the SO calculates the global sequence number by combining the terminal's RolePriority with the function GSeq=f(LamportClock,RolePriority,ScreenID), for example, GSeq=LamportClock×1000+RolePriority×10+the last three digits of ScreenID, to ensure that high-priority operations have a smaller GSeq and are executed earlier; S23: SO writes the operation request and the corresponding GSeq to the global operation log (WAL, used for replay after disconnection) and broadcasts the GSeq and operation information to all terminals; S24: After receiving the broadcast, each terminal adds the operations to the local execution queue in strictly ascending order of GSeq, and executes the operations in the queue in sequence; after the operation is completed, the terminal reports the execution confirmation to SO; S25: If the terminal fails to receive the operation corresponding to a certain GSeq due to network packet loss (i.e., there is a "gap" in the execution queue), it requests the SO to resend the operation information of the GSeq; at the same time, the terminal maintains the OpID idempotency table and skips the OpID operation that has been executed to avoid repeated execution. S26: If the operation timeout occurs, such as if it is not completed within 50ms or if a conflict is found during the execution process, such as if the resources on which the operation depends are not ready, the terminal triggers the rollback mechanism to restore the state before the operation was executed and requests the SO to reissue the GSeq of the operation and re-execute it in the new order.

[0034] It can be seen that this step can achieve SO momentary disconnection handling, that is, when the SO experiences a brief disconnection, the terminal temporarily caches the locally generated operations to an ordered queue. After the SO resumes connection, the terminal submits the cached operation requests to the SO, the SO recalculates the GSeq and broadcasts it, and the terminal reorders the execution based on the GSeq; and operation conflict arbitration, when the driver and other roles (such as the passenger) initiate operations at the same time, because the RolePriority (driver) is higher than other roles, the driver's operation has a smaller GSeq and is executed first; if terminals of the same priority (such as two rear screens) initiate operations at the same time, they are sorted according to the lexicographical order of the ScreenID to ensure the stability of the sorting result.

[0035] In one specific embodiment of the present invention, the total sequence delay (the time from the terminal submitting the operation to the SO broadcasting the GSeq) is ≤25ms; the gap repair time (the time from discovering the queue gap to obtaining and executing the resend operation) is ≤80ms. As a specific example, when the driver clicks "pause" on the media and the passenger clicks "next track" simultaneously, the SO calculates a smaller GSeq for the driver's operation, so all screens execute the "pause" operation first, and then execute the "next track" operation, avoiding the chaotic state of the media "playing and switching tracks at the same time".

[0036] S300: Based on the global serial number, the multi-screen related parameters, and the operation request, obtain the focus of the operation request, and determine the feedback parameters through a soft lock and priority preemption strategy.

[0037] Specifically, this step is implemented based on conflict detection, focus, and soft-lock lease algorithms. Figure 4 The detailed process of conflict detection, focus, and soft-lock lease algorithms according to embodiments of the present invention is illustrated. Figure 4 It can be seen that the conflict detection, focus, and soft-lock lease algorithms involved in step S300 include: S31: When the terminal needs to modify the content of a resource domain (such as adjusting the navigation destination, corresponding to the navigation B resource domain), it sends a focus request to SO. The request includes ScreenID, DomainID (such as player A, navigation B, video source C, etc.) and operation type, read / write. S32: After receiving the request, SO queries the current focus status of the DomainID: 1. If the focus is idle and no terminal holds it, the focus is directly granted to the terminal, a FocusToken is generated and returned, and the OwnerScreenID, Priority, LeaseTTL, and LastRenew are recorded. 2. If the focus is already held by another terminal, and the requesting terminal has a higher priority than the holder, and the PreemptPolicy of the DomainID can be set to "preemptible", then the SO sends a preemption notice to the holder (the notice duration is ≥200ms). If the holder does not respond within the notice period, such as by not initiating a renewal or release request, then the SO revokes the focus of the original holder and grants it to the requesting terminal. 3. If the focus is already held and the requesting terminal has a lower priority than the holder, or if PreemptPolicy is set to "non-preemptible", then the SO will reject the request or put the requesting terminal into the queue. S33: Only after the terminal acquires focus can it issue change-type operations to the corresponding resource domain, such as modifying the navigation destination; query-type operations, such as viewing the navigation route, can be executed directly without acquiring focus. S34: The focus holder needs to initiate a renewal request to the SO every 1 / 2 of the LeaseTTL (e.g., LeaseTTL=300ms, every 150ms) and update LastRenew; when the operation is completed or the terminal actively relinquishes the focus, the terminal sends a focus release request to the SO, the SO sets the focus of the resource domain to idle and notifies the first terminal in the queue to acquire the focus; S35: SO writes the focus granting, preemption, and release events to WAL. The focus event record facilitates the replay of the focus state after disconnection, ensuring focus consistency across multiple screens.

[0038] In this step, when the vehicle speed exceeds a preset threshold (e.g., 60km / h), SO automatically sets the PreemptPolicy of the navigation domain to "cannot be taken over by the rear seats" to prevent rear passengers from modifying the navigation and distracting the driver's attention; Human collaboration mechanism: When the passenger terminal requests the focus of the navigation domain, SO sends a pop-up prompt to the driver's screen: "Passenger requests to modify navigation, do you agree?" Only after the driver confirms and agrees will SO grant the focus to the passenger terminal, realizing collaborative operation.

[0039] As a specific example, the focus granting latency (the time from when the terminal initiates a request to when it acquires focus) is ≤30ms; the preemption visibility notification duration (the time from when the SO sends a preemption notice to the original holder to when the focus is revoked) is ≥200ms, ensuring that the original holder has sufficient time to save the operation state. For example, when a rear passenger tries to change a song, the driver simultaneously initiates navigation modification. Since the navigation domain cannot be preempted by the rear passenger, the rear passenger can normally acquire the media domain focus and change the song, while the navigation domain focus remains held by the driver, and the two operations do not interfere with each other.

[0040] S400: Performs periodic hash reconciliation based on shadow state and domain Merkle tree, locates state differences according to the feedback parameters, and performs incremental repair through incremental patching to ensure consistency of multi-screen state.

[0041] Specifically, this step is implemented based on state synchronization and incremental repair algorithms, which can maintain a "shadow state" corresponding to the global operation for each terminal. The shadow state is a local mirror of the global state on the terminal and is updated in real time with each GSeq operation. The shadow state is fragmented by StateDomain (such as UI tree, player state, navigation information, system settings), and a Merkle tree (hash tree) is built for each fragment. By comparing the MerkleRoot (root hash) of the same StateDomain on different terminals, state differences are quickly located, and then DeltaPatch (incremental patch, which only includes the data of the difference) is used to achieve efficient repair and reduce bandwidth consumption.

[0042] Figure 5 The detailed process of the state synchronization and incremental repair algorithm according to an embodiment of the present invention is illustrated. Figure 5 It can be seen that the state synchronization and incremental repair algorithms involved in step S400 include: When each S41 terminal executes a GSeq operation, it synchronously updates the corresponding shadow state locally. For example, after executing the "pause" operation, the "playback status" field of the player's StateDomain is changed from "playing" to "pause" to complete the shadow state update. S42: The terminal calculates the SegmentHash and MerkleRoot of each StateDomain according to a preset period τ, which can be set to 100-300ms and can be dynamically adjusted according to the network conditions, and reports the calculation results to SO. S43: After receiving the hash data from each terminal, SO compares the data grouped by StateDomain to achieve hash comparison and difference location: If all terminals under the same StateDomain have the same MerkleRoot, the state of that domain is considered to be synchronized and no further processing is required. If there are inconsistencies in MerkleRoot, the SO will locate the specific difference segment by comparing the SegmentHash of each terminal, such as the different segment hashes of the "current song ID" segment of the player's StateDomain. S44: SO selects the reference source for the StateDomain, usually the focus-holding terminal or the terminal that has recently performed operations on the domain, ensures that the state is up-to-date, and sends a patch generation request to the reference source; the reference source generates DeltaPatch containing only the difference data based on the local shadow state and difference shards. S45: After receiving DeltaPatch, SO broadcasts the patch to terminals with inconsistent states; after receiving the patch, the terminal applies it to the corresponding local StateDomain and updates the shadow state. S46: After the terminal applies the patch, the MerkleRoot of the StateDomain is recalculated and reported to the SO. The SO is then compared again to confirm that the status is consistent, and the repair is completed.

[0043] In step S400, the optimization process for high frame rate scenarios includes: for terminals such as the driver's screen that require high frame rate display, separating the UI state (such as button highlighting, progress bar position) from the media frame state (such as video frame image) to avoid frequent UI state reconciliation caused by high frequency changes in media frames, thereby reducing unnecessary computation and bandwidth consumption; handling extreme differences, including when the StateDomain state difference of a terminal is too large (such as more than 50% of the fragment hashes are inconsistent), directly generating DeltaPatch to repair is too inefficient, SO triggers a hot reload mechanism, notifying the terminal to directly obtain the complete StateDomain data from the reference source, replace the local old data, and quickly restore synchronization.

[0044] As a specific example, the reconciliation period τ can be configured to 100-300ms; the average DeltaPatch size is ≤2KB (containing only difference data, much smaller than the tens or even hundreds of KB of complete state data); and the repair convergence time (the time from discovering the difference to all terminals being in the same state) is ≤120ms. For example, if the passenger screen does not receive the "update playlist" operation due to network packet loss, resulting in the displayed playlist being the old version, during hash reconciliation, SO will find that the player's StateDomainMerkleRoot is inconsistent between the passenger screen and the primary screen. After locating the difference fragment, the primary screen generates a DeltaPatch containing only the new playlist. After the passenger screen applies the patch, the playlist is completely consistent with the primary screen.

[0045] S500: It adopts WebRTC layered encoding and weighted bandwidth allocation strategy to prioritize control signaling and the main screen experience, and dynamically adapts in bandwidth-constrained scenarios.

[0046] This step, based on conflict detection and focus lease algorithms, enables the construction of a transmission system that prioritizes control signaling, layers media transmission, and allocates bandwidth across screens using WebRTC technology. Control signaling (such as operation requests, focus requests, and heartbeats) uses independent data channels with the highest transmission priority and minimum retransmission queue to ensure immediacy. Media content (such as video, audio, and UI animations) employs SVC (Scalable Video Coding) technology, dividing the media into a base layer (ensuring content is viewable / audioable) and an enhancement layer (improving picture / sound quality). Different screens can subscribe to different layers based on network conditions and priorities. The total bandwidth B_total is weighted across multiple screens according to Weight(Role, ScreenSize, DistanceToDriver). The weighting rule is: the driver's screen has the highest weight (e.g., weight value 5), followed by the passenger screen (e.g., weight value 3), and then the rear screens and external devices (e.g., weight values ​​1-2), ensuring that the bandwidth requirements of the core screens are prioritized.

[0047] Specifically, Figure 6 A schematic flowchart of the conflict detection and focus lease algorithm according to an embodiment of the present invention is shown. Figure 6 As shown, the conflict detection and focus lease algorithm includes: S51: Each terminal periodically (e.g., every 50ms) checks the local network status, including RTT (round-trip time), packet loss rate, and estimated available bandwidth, and reports the monitoring data to SO to achieve network status monitoring. S52: SO calculates the total session bandwidth B_total based on the network data of each terminal and the number of terminals currently connected in the cockpit; then, based on the weight of each terminal, it calculates the allocated bandwidth B_i for each terminal according to the formula B_i=B_total×(Weight_i / ΣWeight_i). S53: SO sends transmission parameter configuration commands to each terminal and media publishing / forwarding end (such as SFU / MCU, used for multi-screen media distribution): This includes configuring an independent high-priority queue, setting the minimum number of retransmissions (e.g., 3 times) and the shortest retransmission interval (e.g., 10ms), notifying the media publishing end to generate media streams of corresponding levels for different terminals (e.g., the driver's screen subscribes to the basic layer combined with all enhancement layers, while the rear screens only subscribe to the basic layer), and the terminal adjusts the media layer subscription strategy according to the allocated bandwidth B_i (e.g., unsubscribe from enhancement layers when bandwidth is insufficient). S54: When SO detects that the packet loss rate of a terminal is greater than a preset threshold (e.g., 5%) or the RTT is greater than a preset threshold (e.g., 100ms), the pre-degradation mechanism is triggered: First, reduce the terminal's media enhancement layer subscription (e.g., from 1080p to 720p); if the downgrade is still insufficient, reduce the media frame rate (e.g., from 30fps to 15fps); in extreme cases, enter interactive-only mode, retaining only control signaling and UI vector graphics transmission, and suspending media enhancement layer transmission. S55: When the network condition improves (e.g., packet loss rate <2%, RTT <50ms), SO gradually improves the terminal's transmission quality in the order of "first restore frame rate, then restore resolution, and finally restore enhancement layer" to avoid network jitter caused by a sudden increase in bandwidth. S56: SO writes all bandwidth allocation, tier adjustment, and degradation / recovery events to the QoE (Quality of Experience) log for subsequent analysis and optimization.

[0048] The boundary condition control involved in this step includes: extreme network scenarios: when the vehicle enters a tunnel, underground parking garage or other environment with no or weak network, the system automatically switches to interactive-only mode, transmitting only control signals and small UI vector graphics (such as buttons and text) to ensure that the driver's core operations (such as navigation route confirmation and emergency calls) are not affected; local direct connection optimization: for multi-screen connections to the in-vehicle local area network (such as driver's screen, center console screen, and rear screens connected via in-vehicle Ethernet), local direct connection transmission is prioritized, without going through an external network; when external devices (such as mobile phones and tablets) are connected via Wi-Fi, the data is first transmitted to the edge SFU, and then distributed to each screen in the vehicle by the SFU, reducing network latency.

[0049] During application, the end-to-end latency of control signaling, P95 (95% of the latency data is less than this value), is ≤60ms; the frame rate of the media screen on the driver's side, P95, is ≥30fps; the frame rate of the passenger screen, P95, is ≥24fps; and the number of network jitters during degradation and recovery is <2 times / minute (to avoid frequent switching affecting the experience). For example, when the driver's side screen, passenger screen, two rear screens, and an external pad are simultaneously connected in the vehicle, and the network bandwidth suddenly drops, SO prioritizes ensuring the 1080p / 30fps transmission of the driver's side screen, reduces the passenger screen to 720p / 24fps, and reduces the rear screens and external pad to 720p / 15fps. The control channel always maintains low latency, and after the driver clicks "confirm navigation," all screens respond to the operation within 50ms.

[0050] In one specific embodiment of the present invention, the vehicle-mounted multi-screen collaborative display and interaction method may further include: multi-screen time synchronization and frame alignment, permission hierarchical and contextualized strategies, and quality standardization and compliance control; wherein, the multi-screen time synchronization and frame alignment includes using the session clock as a reference, combined with playback timestamps and jitter buffering mechanisms, to achieve millisecond-level synchronization alignment of multi-screen video and audio; the permission hierarchical and contextualized strategies include using a combination model of RBAC and ABAC, combined with the multi-screen related parameters and vehicle condition dynamic adjustment of operation permissions; the quality standardization and compliance control includes setting QoE hard indicators, and implementing a closed-loop mechanism through monitoring, self-healing, alarms, and degradation.

[0051] Specifically, the multi-screen time synchronization and frame alignment algorithm can achieve millisecond-level time alignment of multi-screen media (video, audio) and animations (navigation animations, UI transition animations), avoiding "mistimed" events and improving the audiovisual consistency of multi-screen collaboration. It establishes a globally unified session clock (T_anchor) using the SO as the time anchor (or employing PTP / NTP network time protocols). Each terminal periodically calibrates its clock with the SO, calculating the deviation (Skew) between its local clock and the T_anchor. Media / animation frames carry a PTS (playback timestamp, identifying the frame's playback time) and a TargetPlayAt (target playback time, TargetPlayAt = T_anchor + Δ, where Δ is the frame's delay offset). The terminal corrects its local clock based on the Skew and fine-tunes the frame's display time through a jitter buffer, ensuring that multiple screens play the same frame at the same TargetPlayAt time, achieving frame alignment. Simultaneously, it monitors clock drift between multiple screens, enabling small-step frequency correction when it exceeds a threshold to avoid abrupt "fast forward / rewind" effects.

[0052] Specifically, Figure 7 The flowchart illustrating the multi-screen time synchronization and frame alignment algorithm of an embodiment of the present invention is shown. Figure 7 As shown, the process of the multi-screen time synchronization and frame alignment algorithm includes: S61: After each terminal starts up, it sends a clock calibration request to the SO. The SO returns the current T_anchor and the calibration timestamp. The terminal calculates the Skew (Skew = local clock - T_anchor) based on the difference between the local clock and the timestamp, and recalibrates and updates the Skew periodically (e.g., every 1 second). S62: The media publishing end (such as video server, local media library) assigns PTS to each media frame (video frame, audio frame) and calculates TargetPlayAt=T_anchor+Δ based on T_anchor (Δ is preset according to network latency, such as Δ=200ms, to ensure that the frame has enough time to be transmitted to the terminal); the animation frame (such as navigation route animation) is uniformly assigned TargetPlayAt by SO; S63: After receiving frame data, the terminal corrects the local clock according to the current Skew (corrected local clock = local clock - Skew) to ensure that the local clock is synchronized with T_anchor and realize local clock correction. S64: The terminal stores the received frames into a jitter buffer. The buffer determines the frame scheduling strategy based on the frame's TargetPlayAt and the current corrected local clock. 1. If the current time is earlier than the TargetPlayAt buffer threshold (e.g., 20ms), wait until it is close to TargetPlayAt before displaying; 2. If the current time is later than TargetPlayAt + buffer threshold, discard the frame (to avoid screen delay) or repeat the previous frame (to avoid screen stuttering). 3. If the current time is within the TargetPlayAt± buffer threshold, the frame will be displayed normally; S65: The terminal periodically calculates the drift between the locally calibrated clock and T_anchor (Drift = |calibrated local clock - T_anchor|). If the drift exceeds the threshold (e.g., 50ms), small-step frequency correction is enabled (e.g., adjusting the local clock by 1ms every 100ms) to gradually reduce the drift and avoid large-scale adjustments at once that could cause "frame skipping" in the screen, thus achieving clock drift correction. S66: The terminal periodically reports two key synchronization indicators to the SO: A / V synchronization error (the playback time difference between audio and video) and Inter-screen synchronization error (the playback time difference of the same frame between different screens), which can realize synchronization status detection. S67: SO dynamically adjusts synchronization parameters based on the synchronization error data of each terminal: if the Inter-screen error is too large, increase the jitter buffer threshold (e.g., from 20ms to 30ms); if the A / V error is too large, adjust the Δ value (e.g., from 200ms to 250ms) to ensure synchronization accuracy.

[0053] In this step, when users perform high-frequency interactive operations (such as the driver quickly switching navigation routes or the passenger frequently adjusting media volume), the system allows for slight A / V synchronization errors (such as ≤±60ms) to prioritize the immediacy of operation input and avoid operation delays caused by strict synchronization. In addition, when a terminal (such as the rear screen) enters sleep / screen-off state, logical clock synchronization is still maintained (Skew is periodically calibrated with SO). When the terminal wakes up, it can immediately align to the global frame playback progress based on the current Skew and the cached TargetPlayAt without reloading the media.

[0054] During application, the alignment error of the same frame between multiple screens is ≤ ±20ms (the difference is imperceptible to the naked eye); the synchronization error between audio and video is ≤ ±40ms (compliant with industry audiovisual synchronization standards). Practical application example: The driver's screen and passenger's screen simultaneously play the same movie, while the rear screens play the same audio. The picture and sound on all screens are completely synchronized. The drumbeats in the movie sound simultaneously on all screens, and the lip movements of the characters on screen perfectly match the audio, with no "mismatched" moments.

[0055] The permission hierarchy and contextualized strategy process dynamically adjusts multi-screen operation permissions based on user roles and vehicle conditions. This maximizes the operational convenience for each user role while ensuring driving safety, representing the core balance between security and usability in multi-screen collaboration. It employs a hybrid model combining RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control). RBAC defines basic permissions for roles (driver, co-driver, rear, guest) (e.g., the driver has navigation modification and emergency call permissions, while rear passengers have media control permissions). ABAC dynamically adjusts permissions based on vehicle conditions (speed, gear, driverPresent (driver's presence), and driving mode (e.g., sport mode, eco mode)). The policy engine outputs three decisions—"allow," "deny," and "convert to confirmation request"—based on the combination of roles, vehicle conditions, and operation domains (e.g., media domain, navigation domain, system settings domain). This is linked to operation consistency arbitration and focus lease mechanisms to ensure effective permission control.

[0056] Figure 8 A schematic flowchart of the permission hierarchical and contextualized strategy algorithm according to an embodiment of the present invention is shown. Figure 8 As shown, the process of the permission leveling and contextualization strategy algorithm includes: S71: The system defaults to three types of core permission policies: Driver-only policy: Critical safety-related operations (such as changing navigation destinations and adjusting driving modes) are only allowed to be performed by the driver; other roles have no permission under any vehicle conditions. Joint-control policy: Collaborative operations (such as the passenger searching for a navigation destination or the rear passengers requesting to play a specific song) require confirmation from the driver before execution; Entertainment-free strategy: Entertainment-related operations (such as adjusting media volume and switching video sources) are fully controllable by rear passengers, but cannot affect key areas such as navigation and driving modes; S72: After the terminal initiates an operation request, it first submits the request to the policy engine. The engine extracts the following information and completes the operation request evaluation.

[0057] The extracted information includes: Role information: the terminal role that initiates the operation (e.g., the passenger seat); Vehicle status information: the current vehicle speed, gear (e.g., D gear, P gear), and driverPresent status provided by SO in real time; Operation domain information: the DomainID corresponding to the operation (e.g., navigation domain, media domain); Risk level: the preset operation risk (e.g., modifying navigation is high risk, adjusting volume is low risk).

[0058] S73: The strategy engine evaluates the above information: if it is a high-risk operation (such as modifying navigation) and the character is not the driver, the vehicle speed is greater than the threshold (such as 30km / h), and driverPresent=true, then it outputs "deny" or "request confirmation" (if initiated by the passenger, it will be converted to request confirmation from the driver); if it is a low-risk operation (such as adjusting volume) and the character is in the back seat, the vehicle speed is less than the threshold, or the gear is in P, then it outputs "allow"; if the vehicle condition changes (such as the vehicle speed increases from 20km / h to 70km / h), the strategy engine updates the decision in real time, such as changing "allow back seat to modify navigation" to "deny", and outputs the decision strategy; S74: Decision Execution and Feedback, Output "Allow": The operation request enters the operation consistency and total sequence arbitration process (step S200), executed according to GSeq; Output "Reject": The terminal provides feedback to the user on the reason for rejection, such as "Navigation modification is prohibited while the vehicle is in motion"; Output "Convert to Request Confirmation": SO sends a confirmation request to the driver's screen (such as a pop-up window "Passenger requests to modify navigation destination to XX, do you agree?"), after the driver clicks "Agree", the operation request enters the arbitration process; after clicking "Reject", the terminal provides feedback to the initiator "Driver has rejected"; S75: The policy engine writes all permission decisions (allow / deny / confirm) and decision basis (role, vehicle status) into the audit log for subsequent security analysis and policy optimization.

[0059] During this process, when driverPresent=false (e.g., the driver temporarily gets out of the car and the vehicle is in P gear), the passenger or rear passenger is allowed to temporarily obtain some key permissions (e.g., start the air conditioning, play media), but the highest permissions such as modifying the driving mode and emergency calls are still prohibited. When the terminal initiates an emergency call (e.g., SOS), regardless of the role or vehicle condition, the policy engine will directly output "allow" and prioritize the allocation of bandwidth and system resources to ensure that emergency requests are responded to in a timely manner.

[0060] Specifically, during application, the policy engine's evaluation latency (the time from receiving an operation request to outputting a decision) is ≤5ms; the false interception rate of permission decisions (operations that should be allowed are incorrectly denied) is <0.5%, ensuring that normal user operations are not affected. For example, while the vehicle is in motion, if a rear passenger attempts to change the navigation destination, the policy engine outputs "Deny" and prompts "Navigation modification is prohibited while the vehicle is in motion"; if the front passenger initiates a navigation modification request, the policy engine outputs "Confirmation request will be forwarded," a pop-up message appears on the driver's screen, and after the driver clicks "Agree," the operation is executed according to GSeq, and all screens are updated with the navigation destination synchronously.

[0061] The quality standardization and compliance control process serves as the central control hub of the multi-screen collaboration system. By defining and quantifying quality indicators, monitoring operational status in real time, and triggering self-healing and degradation strategies, it forms a closed-loop quality control system, ensuring the long-term stable operation of multi-screen collaboration and ultimately guaranteeing system reliability. It defines a QoE hard indicator system for multi-screen collaboration, covering dimensions such as latency, frame rate, synchronization accuracy, state consistency, and network quality. SO uses a sliding window (e.g., a 10-second window) to statistically analyze the KPIs (Key Performance Indicators) of each terminal and compare them with preset thresholds. If the KPIs meet the standards, the current configuration is maintained; if the KPIs do not meet the standards, control actions are triggered according to the self-healing, alarm, degradation, and recovery process. All actions apply to the first seven algorithm modules, forming a closed-loop adjustment.

[0062] Figure 9 A schematic flowchart of a quality standardization and compliance control algorithm according to an embodiment of the present invention is shown. Figure 9 As shown, the process of quality standardization and compliance control includes: S81: Receives KPI data reported by the first seven modules every SO period (e.g., 100ms), including: Operation Consistency Module: Total Sequence Delay, Gap Repair Time; Status Synchronization Module: Reconciliation Convergence Time, Patch Size; Adaptive Transmission Module: Control Delay, Frame Rate of Each Screen, Packet Loss Rate; Time Synchronization Module: Frame Alignment Error, A / V Synchronization Error; Permission Policy Module: Policy Evaluation Delay, False Interception Rate. S82: SO uses a sliding window (e.g., window size of 100 data points, approximately 10 seconds) to calculate the P50 (median) and P95 values ​​of KPIs and compares them with preset thresholds: if all KPIs meet the target, the current system configuration is maintained; if a certain KPI does not meet the target (e.g., driver's screen frame rate P95 = 25fps < 30fps), proceed to the next step; S83: SO sends self-healing instructions to the corresponding modules based on the type of non-compliant KPI: If the frame rate is non-compliant: send instructions to the adaptive transmission module to "increase the bandwidth allocation weight of the driver's screen" and "reduce the enhancement layer bit rate of the passenger's screen"; if the frame alignment error exceeds the standard: send instructions to the time synchronization module to "increase the jitter buffer threshold to 30ms"; if the reconciliation convergence time is too long: send instructions to the status synchronization module to "shorten the reconciliation cycle to 100ms". S84: If, after executing the self-healing action, the KPI still fails to meet the standard for N consecutive sliding windows (e.g., N=3, approximately 30 seconds), then SO is triggered, including alarms and degradation. The alarm mainly sends alarm information to the cockpit system management platform, including the type of KPI that fails to meet the standard and the current operating status (e.g., the number of connected terminals and network bandwidth). The degradation mainly sends degradation instructions to relevant modules, such as "maintain 1080p / 30fps for the driver's screen, reduce the passenger's screen to 720p / 20fps, and reduce the rear screen to 480p / 15fps", "disable the media enhancement layer transmission of all terminals", etc. S85: If the KPI is met (e.g., network bandwidth is restored, main screen frame rate P95=32fps≥30fps) and the system remains stable for M consecutive sliding windows (e.g., M=5, approximately 50s), then SO sends recovery commands to the relevant modules in the order of "first restore frame rate, then restore resolution, and finally restore enhancement layer" to gradually restore the system configuration. S86: SO writes all KPI statistics, self-healing actions, alarms, and degradation / recovery events to the QoE log, supporting subsequent offline analysis and optimizing threshold configuration and self-healing strategies.

[0063] During the algorithm process, if the KPI of a certain terminal (such as the passenger screen) is consistently below the standard (e.g., the frame rate is always <15fps), SO will only perform a downgrade on that terminal (e.g., downgrade to 480p / 10fps), without affecting the normal operation of other terminals. When the number of terminals connected in the cabin exceeds the limit (e.g., more than 8), causing the system's CPU and memory usage to be too high, SO will trigger "overload protection downgrade", refuse new terminals to join the session, and reduce the bandwidth allocation of some non-core terminals (e.g., external Pads), prioritizing the core functions of the driver's screen and the central control screen.

[0064] In practical applications: QoEGate's KPI statistics latency is ≤100ms; the effective time of self-healing actions (from sending the command to KPI improvement) is ≤1s; the system meets the core KPIs (control latency, driver's screen frame rate, and frame alignment error) in 99% of operating conditions. For example, when the in-vehicle network packet loss rate soars to 12% and the driver's screen frame rate drops to 28fps < 30fps, the SO first sends a self-healing command to the adaptive transmission module to "increase the driver's screen bandwidth weight to 6 and turn off the passenger screen enhancement layer." One second later, the driver's screen frame rate recovers to 31fps. If the packet loss rate continues to rise, the SO triggers an alarm and reduces the rear screen to 480p / 15fps to ensure that the driver's core experience is not affected.

[0065] Corresponding to the above-mentioned vehicle-mounted multi-screen collaborative display and interaction method, the present invention also provides a vehicle-mounted multi-screen collaborative display and interaction system. Figure 10 A logical block diagram of a vehicle-mounted multi-screen collaborative display and interaction system according to an embodiment of the present invention is shown.

[0066] like Figure 10 As shown, the in-vehicle multi-screen collaborative display and interaction system of this invention includes: Member management unit 11 is used to maintain the global session and member view through a centralized session orchestrator, realize session access and dynamic member management of multi-screen devices, and determine multi-screen related parameters; The sequence number allocation unit 12 is used to construct a full-order execution rule based on the logical clock, the multi-screen related parameters and the operation identifier, and allocate a global sequence number to each multi-screen operation so that the multi-screen operations are executed in a unified order. The parameter determination unit 13 is used to obtain the focus of the operation request based on the global sequence number, the multi-screen related parameters and the operation request, and determine the feedback parameters through a soft lock and priority preemption strategy. Repair unit 14 is used to perform periodic hash reconciliation based on shadow state and domain Merkle tree, locate state differences according to the feedback parameters, and perform incremental repair through incremental patching to make the multi-screen state consistent. The dynamic adaptation unit 15 is used to adopt WebRTC layered encoding and weighted bandwidth allocation strategies to prioritize the control signaling and the main screen experience, and to perform dynamic adaptation in bandwidth-constrained scenarios.

[0067] It should be noted that the embodiments of the above-mentioned vehicle-mounted multi-screen collaborative display and interaction system and the embodiments of the vehicle-mounted multi-screen collaborative display and interaction method can be described in detail and will not be repeated here.

[0068] As can be seen from the above embodiments, the in-vehicle multi-screen collaborative display and interaction method and system of the present invention, through the global sequence number (GSeq) arbitration mechanism, ensures that all screen operations are executed in the same order, avoiding conflicts in concurrent multi-screen operations. For example, when the driver's "pause" and the passenger's "next track" are initiated simultaneously, the system can clearly determine the execution order without chaotic response. Based on the hash reconciliation mechanism of shadow state and Merkle tree, it can quickly detect and repair differences in multi-screen states, with a state convergence time of ≤120ms, completely eliminating the state drift problem of the driver's screen displaying playback and the passenger's screen displaying pause. Based on the session orchestrator, through the Epoch and heartbeat mechanism, it ensures that after a member joins / leaks, all screens synchronously update the member view within 200ms, allowing users to instantly perceive changes in the multi-screen collaborative state.

[0069] It is understood that the in-vehicle multi-screen collaborative display and interaction method and system of the present invention has the following beneficial effects: It can combine role and vehicle condition permission policies to ensure that while the vehicle is in motion (speed > 30km / h), the rear seats cannot modify the navigation, and the passenger seat needs the driver's confirmation to modify the navigation, effectively avoiding distraction of the driver's attention and reducing safety hazards; and when the vehicle condition changes (such as the speed increasing from 20km / h to 70km / h), the permission policy is adjusted in real time, changing from "allow" to "deny" or "confirm," balancing safety and user experience; the bandwidth allocation strategy based on role weight ensures that the driver's screen receives the highest bandwidth resources, while non-core screens automatically degrade when bandwidth is insufficient, avoiding resource waste; the SVC layered encoding technology allows different screens to subscribe to different media levels, transmitting only necessary data, with an average DeltaPatch size ≤ 2KB, significantly reducing bandwidth consumption, and is applicable to weak network environments such as tunnels and underground parking garages. The system automatically switches to interactive-only mode, retaining only core control and UI transmission, ensuring uninterrupted critical functions and a stable and reliable user experience.

[0070] Figure 11 Only electronic devices with components are shown, including in-vehicle multi-screen collaborative display and interactive programs. These electronic devices can be installed in electronic devices, as will be understood by those skilled in the art. Figure 11 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0071] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0072] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0073] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0074] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0075] Furthermore, if the modules / units of the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0076] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0077] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for multi-screen collaborative display and interaction in a vehicle infotainment system, characterized in that, include: A centralized session orchestrator is used to maintain a global session and member view, enabling session access and dynamic member management for multi-screen devices, and determining multi-screen related parameters; Based on the logical clock, the multi-screen related parameters, and the operation identifier, a total order execution rule is constructed, and a global sequence number is assigned to each multi-screen operation so that the multi-screen operations are executed in a unified order. Based on the global serial number, the multi-screen related parameters, and the operation request, the focus of the operation request is obtained, and the feedback parameters are determined through a soft lock and priority preemption strategy. Periodic hash reconciliation is performed based on shadow state and domain Merkle tree. State differences are located according to the feedback parameters and incremental repair is performed through incremental patching to ensure consistency of multi-screen state. We adopt WebRTC layered encoding and weighted bandwidth allocation strategies to prioritize control signaling and the user experience of the main control screen, and perform dynamic adaptation in bandwidth-constrained scenarios.

2. The in-vehicle multi-screen collaborative display and interaction method as described in claim 1, characterized in that, Also includes: Multi-screen time synchronization and frame alignment, hierarchical permission and contextualized strategies, and quality standardization and compliance control; among them, The multi-screen time synchronization and frame alignment includes using the session clock as a reference, combined with the playback timestamp and jitter buffering mechanism, to achieve millisecond-level synchronization and alignment of multi-screen video and audio; The permission hierarchy and contextualization strategy includes a combination of RBAC and ABAC models, combined with the multi-screen related parameters and vehicle condition dynamic control operation permissions. The aforementioned quality standardization and compliance control includes setting hard QoE indicators and implementing a closed-loop mechanism through monitoring, self-healing, alarms, and degradation.

3. The in-vehicle multi-screen collaborative display and interaction method as described in claim 1, characterized in that, The multi-screen related parameters include roles, permissions, and priorities; A centralized session orchestrator maintains the global session and member view, enabling session access and dynamic member management for multi-screen devices, and determining multi-screen related parameters, including: The terminal member requests to join the global session, carrying their account identifier and page identifier. The account identifier is verified by the session orchestrator. If the verification passes, the terminal member is written into the session table and assigned the corresponding role, permission and priority. The session orchestrator broadcasts a member view with new members; wherein, when any member leaves, becomes disconnected, or expires, the session orchestrator updates the member view and broadcasts it.

4. The in-vehicle multi-screen collaborative display and interaction method as described in claim 3, characterized in that, A total-order execution rule is constructed based on the logical clock, the multi-screen related parameters, and the operation identifier. A global sequence number is assigned to each multi-screen operation to ensure that the multi-screen operations are executed in a unified order, including: The terminal generates a local operation identifier and a logical clock, and submits them to the session orchestrator, which determines the global sequence number of the operation. The session orchestrator writes the global sequence number into the global operation log and broadcasts it. Each terminal will put the operation into the local execution queue and report it for confirmation.

5. The in-vehicle multi-screen collaborative display and interaction method as described in claim 3, characterized in that, The feedback parameters include focus granting, rejection, preemption notification, and the corresponding soft lease; Based on the global sequence number, the multi-screen related parameters, and the operation request, the focus of the operation request is obtained, and feedback parameters are determined through a soft lock and priority preemption strategy, including: Get the focus of the terminal operation request; The session orchestrator determines the focus conflicts and priorities; Based on the aforementioned conflicts and priorities, change operations are issued. During the lease renewal period, if other terminals attempt to modify the service, the request will be rejected or the terminal will be queued for further processing. The request is processed according to the stated priority, the feedback parameters are determined, and the announcement is published through the session orchestrator.

6. The in-vehicle multi-screen collaborative display and interaction method as described in claim 3, characterized in that, Periodic hash reconciliation is performed based on shadow state and domain-specific Merkle trees. State differences are located according to the feedback parameters, and incremental repairs are performed through incremental patches to ensure consistency across multiple screens, including: Perform the operation for each global sequence number, and update the local shadow state simultaneously; The segmented hash of each domain is calculated every preset period and reported to the session orchestrator. The hash results of different screens are then compared through the session orchestrator. If the hashes of different screens are consistent, skip them; otherwise, if they are inconsistent, locate the difference segment, specify the reference source through the session orchestrator, and generate an incremental patch broadcast. Each terminal applies the incremental patch and then sends back the new hash until the hashes of all terminals are consistent.

7. The in-vehicle multi-screen collaborative display and interaction method as described in claim 3, characterized in that, Based on the session clock, and combined with playback timestamps and a jitter buffering mechanism, millisecond-level synchronization alignment of multi-screen video and audio is achieved, including: All screens are clock-calibrated with the session orchestrator to obtain a local clock offset, so that the terminal corrects its local clock according to the local clock offset. The jitter buffer determines whether to drop frames, wait for frames, or repeat frames, ensuring that the screen display is aligned with the target playback time; if the jitter exceeds the threshold, small step frequency correction is enabled. Regularly report audio and video time differences and screen-to-screen time differences, and adjust the corresponding alignment parameters through the session orchestrator; Based on the alignment parameters, the video and audio of each screen are aligned and adjusted.

8. The in-vehicle multi-screen collaborative display and interaction method as described in claim 2, characterized in that, The QoE hard metrics include latency, frame rate, and synchronization accuracy.

9. A vehicle-mounted multi-screen collaborative display and interaction system, characterized in that, include: The member management unit is used to maintain the global session and member view through a centralized session orchestrator, realize session access and dynamic member management for multi-screen devices, and determine multi-screen related parameters; The sequence number allocation unit is used to construct a full-order execution rule based on the logical clock, the multi-screen related parameters and the operation identifier, and allocate a global sequence number to each multi-screen operation so that the multi-screen operations are executed in a unified order. The parameter determination unit is used to obtain the focus of the operation request based on the global sequence number, the multi-screen related parameters and the operation request, and determine the feedback parameters through a soft lock and priority preemption strategy. The repair unit is used to perform periodic hash reconciliation based on shadow state and domain Merkle tree, locate state differences according to the feedback parameters, and perform incremental repair through incremental patching to make the multi-screen state consistent. The dynamic adaptation unit is used to adopt WebRTC layered encoding and weighted bandwidth allocation strategies to prioritize control signaling and the main screen experience, and to perform dynamic adaptation in bandwidth-constrained scenarios.

10. A computer storage medium, wherein the computer-readable storage medium stores instructions, characterized in that, When the instruction is executed on the electronic device, the electronic device performs the vehicle-mounted multi-screen collaborative display and interaction method as described in any one of claims 1-8.

Images