Multi-screen interaction system and multi-screen interaction method

Abstract

A multi-screen interaction system and a multi-screen interaction method are disclosed. The multi-screen interaction system comprises: a first display; a second display; an electronic control unit configured to: manage a main operating system runtime and a first containerized operating system runtime, and perform rendering pipeline management for the first display and the second display, wherein the electronic control unit allocates the main operating system runtime to the first display, and allocates the first containerized operating system runtime to the second display.

Description

Technical Field

[0001] This disclosure relates to the field of electronic device technology, and more specifically, to a multi-screen interaction system and a method for multi-screen interaction. Background Technology

[0002] Currently, infotainment systems are rapidly developing towards multi-screen and multi-functionality, with the number of displays within a single system constantly increasing, and users' demand for interaction between multiple display screens also growing.

[0003] Multi-screen systems not only support independent use by multiple users (individuals), but also support multi-screen interaction among multiple users. How to achieve efficient and convenient multi-user interaction with relatively low hardware costs is a problem that needs to be solved. Summary of the Invention

[0004] The present invention is provided in a brief form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or / or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] According to one aspect of this disclosure, a multi-screen interaction system is provided, the multi-screen interaction system comprising: a first display; a second display; and an electronic control unit configured to: manage a main operating system runtime and a first containerized operating system runtime, and perform rendering pipeline management for the first display and the second display, wherein the electronic control unit allocates the main operating system runtime to the first display and allocates the first containerized operating system runtime to the second display.

[0006] The electronic control unit can create a first rendering pipeline and a second rendering pipeline, wherein the first rendering pipeline realizes the mapping between the main operating system runtime and the first display, and the second rendering pipeline realizes the mapping between the first containerized operating system runtime and the second display.

[0007] The main operating system and the first containerized operating system can be of the same or different types of operating systems.

[0008] The electronic control unit may also be configured to receive screen interaction requests from a user, wherein the screen interaction requests may include at least one of the following: copying an application, copying a screen, sharing a screen, and transferring a screen.

[0009] The electronic control unit can also be configured to: display a first instance of the first application on a first display via a first rendering pipeline based on the main operating system runtime, wherein the electronic control unit can also be configured to: in response to a screen interaction request including copying the application from the first display to a second display, reuse the first containerized operating system runtime, or create a second containerized operating system runtime and assign the second containerized operating system runtime to the second display, such that a second instance of the first application is displayed on the second display based on the first containerized operating system runtime or the second containerized operating system runtime, wherein the first containerized operating system runtime is reused in response to the first containerized operating system runtime meeting the running conditions of the first application, and wherein the second containerized operating system runtime is created in response to the first containerized operating system runtime not meeting the running conditions of the first application.

[0010] The electronic control unit can also be configured to: in response to the first containerized operating system running and meeting the operating conditions of the first application, to enable the second display to operate as a receiver of the second rendering pipeline and to enable the second rendering pipeline to receive input from the second display; and in response to the first containerized operating system running and not meeting the operating conditions of the first application, to create a third rendering pipeline, wherein the third rendering pipeline implements the mapping between the second containerized operating system running and the second display, enables the second display to operate as a receiver of the third rendering pipeline, and enables the third rendering pipeline to receive input from the second display.

[0011] The multi-screen interaction system may further include a third display, wherein the electronic control unit is further configured to: in response to a screen interaction request, include copying an application from the first display to the third display, creating a third containerized operating system runtime, and assigning the third containerized operating system runtime to the third display, such that a third instance of the first application is displayed on the third display based on the third containerized operating system runtime.

[0012] The electronic control unit can also be configured to: create a fourth rendering pipeline, wherein the fourth rendering pipeline implements a mapping between the third containerized operating system runtime and the third display, enables the third display to operate as a receiver of the fourth rendering pipeline, and enables the fourth rendering pipeline to receive input from the third display.

[0013] The electronic control unit can also be configured to: in response to a screen interaction request, copy the screen from the first display to the second display, add the second display as a receiver for the first rendering pipeline, and prevent the first rendering pipeline from receiving input from the second display, so that the second display displays the first content displayed on the first display without controlling the first content.

[0014] The electronic control unit can also be configured to: in response to a screen interaction request that the first display and the second display share a screen, add the second display as a receiver for the first rendering pipeline, and enable the first rendering pipeline to receive input from the second display, so that the second display displays the first content displayed on the first display, and is able to control the first content.

[0015] The electronic control unit can also be configured to: in response to a screen interaction request, transfer the screen, prevent the first display from receiving image frames from the first rendering pipeline, so that the first display does not display the first content; identify the second display as the receiver of the first rendering pipeline, and enable the first rendering pipeline to receive input from the second display, so that the second display displays the first content, and be able to control the first content.

[0016] The electronic control unit can also be configured to: receive a first input from a user, the first input including an action acting on a source display for entering multi-screen interaction; in response to the received first input, display first content and a display layout thumbnail on the source display, wherein the first content is content displayed on the source display prior to receiving the first input, and wherein the display layout thumbnail includes a thumbnail of the source display and a thumbnail of the target display; receive a second input from the user, the second input including an action acting on the source display for displaying a mode selection interface for a screen interaction request; in response to the received second input, display a mode selection interface for a screen interaction request on the source display; receive a third input from the user, the third input including an action acting on the source display for selecting a corresponding mode; and in response to the received third input, determine the user's screen interaction request, wherein the source display is one of a first display and a second display, and the target display is the other of the first display and the second display.

[0017] The electronic control unit is also configured to: display a list of applications on a source display in response to determining that a user’s screen interaction request is to copy an application; receive a fourth input from the user, the fourth input including an action performed on the source display to select the application to be copied; and determine the application to be copied in response to the received fourth input.

[0018] The first input may include a multi-finger long press action on the source display; the second input may include a drag action on the source display to drag the first content to a thumbnail on the target display; the third input may include a click action on the source display to select the corresponding mode; and the fourth input may include a click action on the source display to select the application to be copied.

[0019] According to one aspect of this disclosure, a method for multi-screen interaction is provided, the method comprising: managing a main operating system runtime and a first containerized operating system runtime through an electronic control unit; and performing rendering pipeline management for a first display and a second display through the electronic control unit, wherein the electronic control unit assigns the main operating system runtime to the first display and the first containerized operating system runtime to the second display.

[0020] According to one aspect of this disclosure, a vehicle is provided that includes the multi-screen interaction system.

[0021] Further aspects and / or advantages of the inventive concept will be set forth in part in the description which follows, and in part will be obvious from the description and / or may be learned by practice of various exemplary embodiments. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings.

[0023] Figure 1A and Figure 1B This is a diagram illustrating a rendering pipeline according to an example embodiment of the present disclosure.

[0024] Figure 2 This is a diagram illustrating a multi-screen interactive system according to an example embodiment of the present disclosure.

[0025] Figure 3 This is a diagram illustrating the receipt of a screen interaction request according to an example embodiment of the present disclosure.

[0026] Figure 4 This is a diagram illustrating the receipt of a screen interaction request according to an example embodiment of the present disclosure.

[0027] Figure 5 This is a diagram illustrating rendering pipeline management according to an example embodiment of the present disclosure.

[0028] Figure 6 This is a diagram illustrating rendering pipeline management according to an example embodiment of the present disclosure.

[0029] Figure 7 This is a flowchart illustrating a method for multi-screen interaction according to an example embodiment of the present disclosure. Detailed Implementation

[0030] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, except for those that must occur in a specific order, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0031] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0032] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0033] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0034] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0035] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0036] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0037] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0038] With the development of information technology, the configuration of multiple displays (or multiple physical screens) within a single system has become increasingly common. For example, in automotive systems, the number of displays in the cabin continues to increase, and the functions and application scenarios of each display are becoming increasingly diverse. In smart home systems, multiple appliances with screens can share content and interact through control. The following description uses an automotive system as an example to illustrate the inventive concept of this application; however, this is merely an example, and the inventive concept of this application can be applied to any electronic system with multiple displays.

[0039] For ease of understanding, the following descriptions of concepts related to this application are provided as examples for reference.

[0040] Container technology is a lightweight virtualization technology that allows developers to package applications and their dependencies into a portable container. This container contains all the environment required for the application to run, enabling the application to run seamlessly on any platform that supports container technology.

[0041] The main operating system is the underlying operating system that provides computing resources, data isolation, and toolchain compatibility for running pipelines, containers, and other software workloads.

[0042] A containerized operating system is a streamlined operating system that runs on a host operating system (e.g., the main operating system) based on standard container technology. A containerized operating system can provide an isolated execution environment and data separation mechanism for each application instance, ensuring the independence and data security between different application instances.

[0043] Runtime refers to an instance of a containerized operating system or a host operating system; that is, the collection of environments and related components that a program depends on during execution. In this disclosure, runtime can include both host operating system runtime and containerized operating system runtime (also referred to as a "containerized runtime"). For example, a containerized operating system runtime can represent an instance of a containerized operating system, which includes the collection of environments and related components that a program running on container technology depends on during execution.

[0044] The main operating system can start upon system power-on and shut down upon system power-off. Containerized operating system runtimes can be created, allocated, or destroyed as needed.

[0045] "Creating" a containerized runtime can mean starting a container runtime environment and initializing, configuring, and starting a container instance within that environment. For example, when a user needs to copy an application from the current display (i.e., the source display) to another display (i.e., the target display), a new runtime might be created on the target display to host the application. Idle containerized runtimes can be destroyed to optimize system resource usage when they have been idle for an extended period and system resource utilization has reached a predetermined threshold.

[0046] The role of a rendering pipeline is to map the graphics output from the operating system runtime (e.g., the main operating system runtime or a containerized operating system runtime) to one or more target physical screens, and to feed back input events from the physical screens to the corresponding operating system runtime. In other words, the rendering pipeline is the collective term for a series of processes that receive input from the physical screen (i.e., the monitor) and output graphics to the physical screen.

[0047] The Electronic Control Unit (ECU) is the core component of a vehicle's electronic system, responsible for monitoring, control, and communication with various subsystems. Vehicles typically contain one or more ECUs, forming a distributed or centralized electronic network.

[0048] In most vehicle designs, each display (e.g., dashboard, center console, rear-seat screens, head-up display (HUD), etc.) is driven by a dedicated ECU and a separate I / O controller. However, as the number of cockpit displays increases, a distributed approach with a separate ECU for each display leads to a significant increase in hardware costs and a serious waste of system resources. Because each display is driven by a separate ECU or dedicated I / O controller, the content on different displays is often isolated, making convenient interaction between multiple users difficult. Furthermore, each additional ECU increases the total system power consumption by approximately 15-20%, thus reducing the vehicle's driving range.

[0049] Figure 1A and Figure 1B This is a diagram illustrating a rendering pipeline according to an example embodiment of the present disclosure.

[0050] The role of the rendering pipeline is to output or map image frames from the operating system runtime (e.g., the main operating system runtime or the containerized operating system runtime) to one or more target displays, and to feed back input events from the displays (e.g., touch events) to the corresponding operating system runtime.

[0051] Reference Figure 1AIf the application is running on the main operating system 110 (e.g., when the main operating system is running), the main operating system 110 outputs image frames directly to the corresponding displays 130 and 140 through the rendering pipeline 120.

[0052] Reference Figure 1B If the application is running on a containerized operating system 150 (e.g., a containerized operating system runtime), the host graphics card will execute display instructions from the virtual graphics card and output the generated image frames to the display 130 corresponding to the containerized operating system runtime.

[0053] Figure 2 This is a diagram illustrating a multi-screen interactive system according to an example embodiment of the present disclosure.

[0054] Reference Figure 2 A multi-screen interactive system 2000 according to an example embodiment of this disclosure may include: an electronic control unit (ECU) 200, a first display 240, and a second display 250. Although Figure 2 The image shows two displays, but this is just an example; the number of displays (or physical screens) included in a multi-screen interactive system can be three or more.

[0055] According to exemplary embodiments of this disclosure, applications in a multi-screen interactive system can run on either a host operating system or a containerized operating system. A containerized operating system refers to a streamlined operating system that runs on a host operating system based on container technology. A containerized operating system contains all the environment required for an application to run, enabling the application to function correctly on the containerized operating system. Unlike virtual machines, a containerized operating system contains only the application and the environment components for running the application, and the containerized operating system shares the kernel of the host operating system with the host operating system.

[0056] The ECU 200 can be used as the overall controller of the multi-screen interactive system 2000, and performs display interaction functions, runtime management functions, and rendering pipeline management functions.

[0057] In the example embodiment, the above-described display interaction functions, runtime management functions, and rendering pipeline management functions can be implemented by executing software or instructions in the ECU 200. For example, the ECU 200 may include a display interaction manager 210, a runtime manager 220, and a rendering pipeline manager 230.

[0058] ECU 200 (e.g., display interaction manager 210) controls the interaction between all displays and coordinates the scheduling of resources between applications, rendering pipelines, and displays.

[0059] ECU 200 can manage the main operating system runtime and all containerized operating system runtimes, and perform rendering pipeline management for the first display 240 and the second display 250.

[0060] In an example embodiment, ECU 200 (e.g., runtime manager 220) may assign a main operating system runtime to a main display and create one or more containerized operating system runtimes based on user needs, and assign one or more containerized operating system runtimes to other displays in the multi-screen interaction system 2000. For example, ECU 200 may assign a main operating system runtime to a first display 240 and assign a first containerized operating system runtime to a second display 250.

[0061] ECU 200 (e.g., rendering pipeline manager 230) can manage rendering pipelines (e.g., perform rendering pipeline creation, switching, and allocation) and transmit user input events through the rendering pipeline to the corresponding runtime.

[0062] The first display 240 and the second display 250 can be used as receivers in a rendering pipeline to display image frames on the screen. Furthermore, the first display 240 and the second display 250 can feed back user touch events to the operating system.

[0063] In an example embodiment, ECU 200 can perform rendering pipeline management for a first display and a second display. ECU 200 can create a first rendering pipeline and a second rendering pipeline, wherein the first rendering pipeline implements the mapping between the main operating system runtime and the first display, and the second rendering pipeline implements the mapping between the containerized operating system runtime and the second display.

[0064] In the example embodiments, the main operating system runtime and the containerized operating system runtime may be of the same type of operating system or of different types. For example, when both the main operating system and the containerized operating system are based on the same kernel (e.g., the Linux kernel), the main operating system runtime and the containerized operating system runtime can be of different types of operating systems. Therefore, the multi-screen interaction system according to the example embodiments of this disclosure can support multi-operating system compatibility using a single ECU based on container technology.

[0065] ECU 200 (e.g., display interaction manager 210) may receive screen interaction requests from a user. In an example embodiment, a screen interaction request may include at least one of the following: copying an application, copying a screen, sharing a screen, and transferring a screen.

[0066] Users can input screen interaction requests by touching the display, but this is only an example, and this disclosure is not limited thereto. In other examples, users can input screen interaction requests in other ways (e.g., voice, gestures, etc.). Figure 3 This is a diagram illustrating the receipt of a screen interaction request according to an example embodiment of the present disclosure. Figure 3 The image shows the interface displayed on the source display for receiving screen interaction requests.

[0067] A user can perform a first action on the source display, and the ECU 200 can receive a first input corresponding to the first action from the user. The first input may include an action performed on the source display to initiate multi-screen interaction. In an example embodiment, the source display is any one of multiple displays in a multi-screen interaction system. For example, the source display may be... Figure 2 The first display 240 or the second display 250 is used. In an example embodiment, the first input may include a multi-finger long press action acting on the source display. See also Figure 3 A multi-finger long press action can represent a three-finger long press action, but this is just an example and this disclosure is not limited to this.

[0068] In response to a received first input, ECU 200 may display first content and a display layout thumbnail (also known as a screen layout thumbnail) on a source display. The first content may be content displayed on the source display prior to receiving the first input. The display layout thumbnail may be a diagram showing all displays within the multi-screen interaction system in a thumbnail format. In other words, the display layout thumbnail may include thumbnails of the source display and thumbnails of the target display. For example, in an example where the multi-screen interaction system includes a first display and a second display, the display layout thumbnail may display thumbnails of both the first and second displays.

[0069] The user can perform a second action on the source display, and the ECU 200 can receive a second input from the user corresponding to the second action. The second input may include an action performed on the source display to display a mode selection interface for a screen interaction request. In an example embodiment, the second input may include a dragging action performed on the source display to drag the first content to a thumbnail on the target display.

[0070] In the example embodiment, the target display can be any one of multiple displays in a multi-screen interaction system. For example, when the source display is... Figure 2 In the case of one of the first display 240 and the second display 250, the target display can be the other of the first display 240 and the second display 240.

[0071] In response to the received second input, the ECU 200 can display a mode selection interface for the screen interaction request on the source display.

[0072] Reference Figure 3 The screen interaction request mode selection interface can list several screen interaction modes: copy application, copy screen, share screen, and transfer screen.

[0073] The user can perform a third action on the source display, and the ECU 200 can receive a third input corresponding to the third action from the user. The third input may include an action performed on the source display to select a corresponding mode. In an example embodiment, the third input may include a click action performed on the source display to select a corresponding mode.

[0074] In response to the received third input, ECU 200 can determine the user's screen interaction request. For example, ECU 200 can determine the source display (e.g., source display ID), the target display (e.g., target display ID), and the mode of the screen interaction request (e.g., copying an application, copying a screen, sharing a screen, and transferring a screen) based on the first to the third input.

[0075] In response to a user's screen interaction request to copy an application, the ECU 200 can display a list of applications on the source display.

[0076] The user can perform a fourth action on the source display, and the ECU 200 can receive a fourth input from the user corresponding to the fourth action. The fourth input may include an action performed on the source display to select the application to be copied. In an example embodiment, the fourth input may include a click action performed on the source display to select the application to be copied.

[0077] Figure 4 This is a diagram illustrating the receipt of a screen interaction request according to an example embodiment of the present disclosure. Figure 4 The image shows the interface displayed on the target display for receiving screen interaction requests.

[0078] After determining the user's screen interaction request, the ECU 200 can display an interface on the target display indicating whether to accept the screen interaction request from the source display. Based on a predetermined action performed by the user on the target display (e.g., clicking the "Accept" option), the ECU 200 can display corresponding screen-sharing content on the target display.

[0079] Taking in-vehicle systems as an example, in order to save hardware resources and achieve control over other displays in the cabin, most applications can be installed only on the main operating system and displayed on the main display (e.g., the central control screen). For example, users may not want rear passengers who are minors to have any applications installed on the rear displays.

[0080] According to an example embodiment of this disclosure, when a user other than the driver wishes to install an application, the application can be installed and run on a containerized operating system simply by initiating a screen interaction request to "copy the application".

[0081] According to example embodiments of this disclosure, containerization technology can be used to deploy multiple independent runtimes through a single ECU to enable application dual-opening (or application multi-opening) functionality on multiple displays within a multi-screen interactive system.

[0082] In an example embodiment, ECU 200 may cause a first application (e.g., a first instance of the first application) to be displayed on a first display 240 based on a host operating system runtime via a first rendering pipeline. When a user receives a screen interaction request to "copy the first application from the first display to a second display", ECU 200 may reuse the first containerized operating system runtime previously allocated to the second display, causing a second instance of the first application to be displayed on the second display based on the first containerized operating system runtime.

[0083] For example, when a screen interaction request for a duplicate application is received, the ECU 200 can determine whether the source and target displays are assigned different operating system runtimes (or runtime environments). In other words, it can determine whether the runtime environment for the dual application instances is ready.

[0084] If the target display uses the same runtime as the source display, an additional containerized operating system runtime needs to be rebuilt for the target display. If the first containerized operating system runtime previously assigned to the target display meets the runtime requirements of the first application, the first containerized operating system runtime can be reused. If the first containerized operating system runtime does not meet the runtime requirements of the first application, an additional second containerized operating system runtime can be created and assigned to the second display, so that a second instance of the first application is displayed on the second display based on the second containerized operating system runtime.

[0085] Specifically, after assigning a reused first containerized operating system runtime or a created second containerized operating system runtime to the second display, it can be determined whether the first application is already installed on the assigned runtime. If the first application is not installed on the assigned runtime, the installation process is performed on the assigned runtime. If the first application is already installed on the assigned runtime, a second instance of the first application can be run on the assigned runtime.

[0086] In an example embodiment, the conditions under which the first containerized operating system runtime can be reused may include: the first containerized operating system runtime is in an idle state, the graphics output dimension of the first containerized operating system runtime meets the requirements of the first application, and the operating system type of the first containerized operating system runtime meets the requirements of the first application.

[0087] In the example embodiment, when the first containerized operating system runtime is not reusable, it can be determined whether an additional second containerized operating system runtime can be created based on the system resource usage. For example, an additional second containerized operating system runtime can be created when the following conditions are met: the first containerized runtime is not reusable; the current total number of runtimes is less than a predetermined runtime quantity threshold; and the system resource availability percentage is greater than a system resource threshold.

[0088] In an example embodiment, idle containerized operating system runtimes can be deleted to improve system resource utilization efficiency. For example, when the system resource utilization rate is higher than a first predetermined threshold, or a specific containerized operating system runtime is idle for a long period of time (e.g., the idle time is greater than a second predetermined threshold), or the hit rate of a specific containerized operating system runtime is lower than a third threshold, that specific containerized operating system runtime can be deleted.

[0089] After a reused first containerized operating system runtime or a created second containerized operating system runtime is assigned to the second display, the ECU 200 can perform rendering pipeline management for screen interaction requests from the copied application.

[0090] Figure 5 This is a diagram illustrating rendering pipeline management according to an example embodiment of the present disclosure.

[0091] exist Figure 5 In this context, assume that a first application (App) is displayed on a first display (512) via a first rendering pipeline (511) based on a first runtime (510) (e.g., the main operating system runtime). When a screen interaction request to "copy the application" is received, the ECU (200) can perform runtime management and rendering pipeline management, causing a second instance of the first application (App) to be displayed on a second display (522) via a second rendering pipeline (521) based on a second runtime (520) (e.g., a reused or created containerized operating system runtime).

[0092] In an example embodiment, in response to the first containerized operating system runtime meeting the operating conditions of the first application, the ECU 200 can reuse the first containerized operating system runtime and enable the second display to operate as a receiver of the second rendering pipeline, and enable (or allow) the second rendering pipeline to receive input from the second display.

[0093] In an example embodiment, in response to the first containerized operating system runtime failing to meet the operating conditions of the first application, ECU 200 may create a second containerized operating system runtime and a third rendering pipeline. The third rendering pipeline enables mapping between the second containerized operating system runtime and the second display, allowing the second display to operate as a receiver of the third rendering pipeline, and enabling (or allowing) the third rendering pipeline to receive input from the second display.

[0094] although Figure 5 The illustration shows two instances of a first application displayed on two monitors in a multi-screen interactive system, but this is merely an example and the disclosure is not limited thereto. Multiple instances of the first application can be displayed on more than two monitors as needed by the user to achieve application multi-instance functionality.

[0095] In an example embodiment, the multi-screen interaction system 2000 may further include a third display. When the user also wishes to run a third instance of the first application, the ECU 200 may create a third containerized operating system runtime and assign the third containerized operating system runtime to the third display, so that the third instance of the first application is displayed on the third display based on the third containerized operating system runtime.

[0096] Accordingly, ECU 200 can create a fourth rendering pipeline. The fourth rendering pipeline enables mapping between the third containerized operating system runtime and the third display, allowing the third display to operate as a receiver for the fourth rendering pipeline, and enabling (or allowing) the fourth rendering pipeline to receive input from the third display.

[0097] In one example embodiment, a user can use the "Duplicate App" screen interaction mode to display different instances of the same application on different displays. Because different instances of the same application run on different operating systems, data isolation between different users is achieved. For example, different users can run the same game simultaneously without interfering with each other. Similarly, different users can run the same video application simultaneously to watch different videos.

[0098] As mentioned above, when the screen interaction request is to copy the application, it may be necessary to create an additional containerized operating system runtime to host additional instances of the application. In contrast, when the screen interaction request is to copy, share, or transfer the screen, it may not be necessary to create an additional containerized operating system runtime.

[0099] Figure 6 This is a diagram illustrating rendering pipeline management according to an example embodiment of the present disclosure. Figure 6 The diagram illustrates the rendering pipeline management in scenarios where screen interaction requests are screen copying, screen sharing, and screen transfer.

[0100] ECU 200 creates a first rendering pipeline 611. In an example embodiment, the first rendering pipeline 611 may implement a mapping between the host operating system runtime and the first display 610. However, this is merely an example, and this disclosure is not limited thereto. In another example embodiment, the first rendering pipeline 611 may implement a mapping between a containerized operating system runtime and the first display 610.

[0101] In response to a screen interaction request to copy the screen from the first display 610 to the second display 620, the ECU 200 may add the second display 620 as a receiver for the first rendering pipeline 611 and disable the first rendering pipeline 611 from receiving input from the second display 620, so that the second display 620 displays the first content displayed on the first display 610 without controlling the first content.

[0102] In response to a screen interaction request that the first display 610 and the second display 620 share a screen, the ECU 200 may add the second display 620 as a receiver for the first rendering pipeline 611 and enable (or allow) the first rendering pipeline 611 to receive input from the second display 620, so that the second display 620 displays the first content displayed on the first display and is able to control the first content.

[0103] In response to a screen interaction request that involves screen transfer (e.g., transferring the screen from the first display 610 to the second display 620), the ECU 200 may prevent the first display 610 from receiving image frames from the first rendering pipeline 611, causing the first display 610 not to display the first content. Furthermore, the ECU 200 may identify the second display 620 as a receiver of the first rendering pipeline 611 and enable (or allow) the first rendering pipeline 611 to receive input from the second display 620, causing the second display 620 to display the first content and enabling control over the first content.

[0104] In one example embodiment, a user can use the "Duplicate Screen" screen interaction mode to prevent other users from controlling the display.

[0105] For example, if a child wants to watch a cartoon but the father doesn't want his son to control the rear-seat monitor, the father can set up a "copy screen" interaction mode to copy the content from the main display or center console screen to a sub-display for rear passengers. This allows the child to view the content only through the sub-display and prevents them from performing any other operations, thus achieving safe content sharing. The father can also terminate the "copy screen" mode at any time during this process.

[0106] In another example embodiment, the sharer can disable the recipient's input function through the "copy screen" screen interaction mode to prevent the leakage of privacy information and ensure that the sharing process is secure and controllable.

[0107] In one example embodiment, a user can use a "screen sharing" screen interaction mode to allow multiple users to receive the same content and control it. The sharer can send content displayed on their own monitor to other users' monitors via "screen sharing" mode, allowing other users to control the content displayed on their monitors (e.g., through touch, voice, gestures, etc.).

[0108] Figure 7 This is a flowchart illustrating a method for multi-screen interaction according to an example embodiment of the present disclosure.

[0109] According to the example embodiments of this disclosure, a single ECU can be used to achieve multi-screen interaction compatible with multiple operating systems and to realize multi-screen interaction in multiple modes.

[0110] The ECU can receive screen interaction requests from the user. According to an example embodiment of this disclosure, the screen interaction request includes at least one of the following: copying an application, copying the screen, sharing the screen, and transferring the screen.

[0111] Reference Figure 7 In step S710, a single ECU can manage the main operating system runtime and one or more containerized operating system runtimes. In an example embodiment, a single ECU can deploy and manage the main operating system runtime and a first containerized operating system runtime among the one or more containerized operating system runtimes. The single ECU can assign the main operating system runtime to a first display and assign the first containerized operating system runtime among the one or more containerized operating system runtimes to a second display.

[0112] The main operating system and the first containerized operating system can be of the same or different types of operating systems.

[0113] In step S720, rendering pipeline management can be performed for the first and second displays via the ECU.

[0114] In an example embodiment, a first rendering pipeline and a second rendering pipeline can be created using a single ECU. The first rendering pipeline implements the mapping between the main operating system runtime and the first display, and the second rendering pipeline implements the mapping between the first containerized operating system runtime and the second display.

[0115] This application discloses a runtime-based multi-screen interaction system that utilizes container technology to deploy multiple independent runtimes on a single ECU, enabling application dual-instance (or application multi-instance) functionality across multiple displays. Furthermore, it allows compatibility with multiple operating systems on a single ECU and achieves multi-user isolation, providing efficient and convenient multi-screen interaction. Specifically, by deploying a main operating system (or main operating system runtime) and (one or more) containerized operating systems (or containerized operating system runtimes) on a single ECU, different instances of the same application can run on different physical screens or displays, ensuring resource isolation between different instances. This achieves multi-screen interaction with reduced hardware costs, reduced resource consumption, and a simplified system architecture. Moreover, the main operating system and containerized operating systems can be of the same or different types. Therefore, a simplified system architecture can support compatibility with multiple operating systems, improving the convenience of application deployment and porting. Furthermore, the runtime-based multi-screen interaction system can support various interaction modes between multiple users through rendering pipeline management, greatly enhancing the user experience.

[0116] The solutions provided in the exemplary embodiments of this disclosure can be applied to scenarios involving multi-screen interaction between multiple displays. Multi-screen interaction can refer to multiple users performing functions such as screen duplication, screen sharing, screen transfer, and application duplication between multiple displays, but is not limited to these. Multiple displays can be located in the same space or different spatial areas, and can belong to the same electronic device or different electronic devices within the same system.

[0117] In one example embodiment, the multiple displays may be multiple displays configured in a vehicle cabin, and the multi-screen interaction system according to an example embodiment of the present disclosure may be included in a vehicle (e.g., a vehicle control system).

[0118] In another example embodiment, the multiple displays may be displays for multiple electronic devices (e.g., televisions, computers, monitoring equipment) within a smart home system. However, this is merely an example scenario of the application of the inventive concept of this disclosure, which can be applied to any electronic system with multiple screens.

[0119] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0120] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0121] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions and / or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art or programmers can readily write the instructions and / or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0122] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drive (HDD), solid-state drive (SSD), flash memory, card memory (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and at least one of any other device, any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0123] Although various exemplary embodiments have been described, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A multi-screen interactive system, comprising: First display; Second display; The electronic control unit is configured as follows: Manage the main operating system runtime and the first containerized operating system runtime, and Render pipeline management is performed for both the first and second displays. The electronic control unit assigns the main operating system runtime to the first display and the first containerized operating system runtime to the second display.

2. The multi-screen interaction system according to claim 1, in, The electronic control unit creates the first and second rendering pipelines. The first rendering pipeline implements the mapping between the main operating system and the first display during runtime. The second rendering pipeline implements the mapping between the first containerized operating system runtime and the second display.

3. The multi-screen interaction system according to claim 1, wherein, The main operating system and the first containerized operating system may be of the same or different types of operating systems.

4. The multi-screen interaction system according to claim 1, in, The electronic control unit is also configured to receive screen interaction requests from the user. The screen interaction request includes at least one of the following: Copy application, copy screen, share screen, and transfer screen.

5. The multi-screen interaction system according to claim 2, in, The electronic control unit is also configured to display a first instance of the first application on a first display via a first rendering pipeline when the first application is running on the main operating system. The electronic control unit is further configured to: in response to a screen interaction request, including copying an application from a first display to a second display, reusing a first containerized operating system runtime, or creating a second containerized operating system runtime and assigning the second containerized operating system runtime to the second display, such that a second instance of the first application is displayed on the second display based on either the first containerized operating system runtime or the second containerized operating system runtime. Specifically, in response to the first containerized operating system runtime meeting the running conditions of the first application, the first containerized operating system runtime is reused. In response to the first containerized operating system runtime not meeting the running conditions of the first application, a second containerized operating system runtime is created.

6. The multi-screen interaction system according to claim 5, wherein, The electronic control unit is also configured as follows: In response to the first containerized operating system meeting the runtime conditions of the first application, the second display is enabled to operate as a receiver of the second rendering pipeline, and the second rendering pipeline is enabled to receive input from the second display. In response to the first containerized operating system not meeting the runtime conditions of the first application, a third rendering pipeline is created. The third rendering pipeline implements the mapping between the second containerized operating system runtime and the second display, enabling the second display to operate as a receiver of the third rendering pipeline, and enabling the third rendering pipeline to receive input from the second display.

7. The multi-screen interaction system according to claim 5 further includes: Third display, The electronic control unit is also configured to: in response to a screen interaction request, include copying an application from the first display to the third display. A third containerized operating system runtime is created and assigned to a third display, so that a third instance of the first application is displayed on the third display based on the third containerized operating system runtime.

8. The multi-screen interaction system according to claim 7, wherein, The electronic control unit is also configured as follows: Create a fourth rendering pipeline, which implements the mapping between the third containerized operating system runtime and the third display, enabling the third display to operate as the receiver of the fourth rendering pipeline, and enabling the fourth rendering pipeline to receive input from the third display.

9. The multi-screen interactive system according to claim 2, wherein, The electronic control unit is also configured as follows: In response to a screen interaction request, the screen is copied from the first display to the second display. The second display is added as a receiver for the first rendering pipeline, and the first rendering pipeline is prevented from receiving input from the second display, so that the second display displays the first content displayed on the first display without controlling the first content.

10. The multi-screen interaction system according to claim 2, wherein, The electronic control unit is also configured as follows: In response to a screen interaction request, the first and second displays share the screen. The second display is added as a receiver for the first rendering pipeline, and the first rendering pipeline is enabled to receive input from the second display, so that the second display displays the first content displayed on the first display and can control the first content.

11. The multi-screen interaction system according to claim 2, wherein, The electronic control unit is also configured to: transfer the screen in response to a screen interaction request. The first display is prevented from receiving image frames from the first rendering pipeline, so that the first display does not display the first content; The second display is designated as the receiver of the first rendering pipeline, and the first rendering pipeline is enabled to receive input from the second display, so that the second display displays the first content and can control the first content.

12. The multi-screen interaction system according to claim 4, wherein, The electronic control unit is also configured as follows: Receive first input from the user, the first input including an action performed on the source display for entering multi-screen interaction; In response to a received first input, first content and a display layout thumbnail are displayed on a source display, wherein the first content is content displayed on the source display prior to receiving the first input, and the display layout thumbnail includes a thumbnail of the source display and a thumbnail of the target display. The user receives a second input, which includes an action performed on the source display to display a mode selection interface for screen interaction requests. In response to the received second input, a mode selection interface for the screen interaction request is displayed on the source display; Receive a third input from the user, which includes an action performed on the source display to select the appropriate mode; In response to the received third input, determine the user's screen interaction request. The source display is one of the first display and the second display, and the target display is the other of the first display and the second display.

13. The multi-screen interaction system according to claim 12, wherein, The electronic control unit is also configured as follows: In response to a user's screen interaction request, the application is copied, and a list of applications is displayed on the source display; A fourth input is received from the user, which includes an action performed on the source display to select the application to be copied. In response to the received fourth input, the application to be copied is determined.

14. The multi-screen interaction system according to claim 12 or 13, wherein, The first input includes a multi-finger long press action applied to the source display. The second input includes a dragging action performed on the source display to drag the first content to a thumbnail on the target display. The third input includes a click action performed on the source display to select the corresponding mode. The fourth input includes a click action performed on the source display to select the application to be copied.

15. A method for multi-screen interaction, the method comprising: The main operating system runtime and the first containerized operating system runtime are managed by an electronic control unit. The electronic control unit performs rendering pipeline management for the first and second displays. The electronic control unit assigns the main operating system runtime to the first display and the first containerized operating system runtime to the second display.

16. A vehicle comprising the multi-screen interactive system according to any one of claims 1 to 14.

Images