Multi-screen interaction method, device, system, car machine system and storage medium

Abstract

A multi-screen interaction method, device, system and storage medium solve the problem that private customization cannot be realized and display efficiency is low when multi-screen display in the multi-screen interaction process in the prior art. The multi-screen interaction method is used for a multi-screen interaction system including a display chip, a display memory, a main display screen and a secondary display screen. The display memory includes a plurality of layer display memory areas, a main display memory area and a virtual display memory area. The plurality of layer display memory areas respectively store graphical data of a plurality of application programs. The main display memory area stores synthesis data of the plurality of application programs. The plurality of application programs include a target application program. The multi-screen interaction method includes: determining a secondary display screen selected by a user and screen projection content based on a screen projection operation of the user; when the screen projection content is the target application program, storing graphical data of the target application program in the virtual display memory area; and sending the graphical data in the virtual display memory area and the synthesis data in the main display memory area to the secondary display screen and the main display screen respectively for display.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-screen interaction, and in particular to a multi-screen interaction method, device, system, car machine system and storage medium. BACKGROUND

[0002] With the progress of technology, Android terminal products are applied more and more widely, and consumers have higher and higher experience requirements for terminal products. Multi-screen interaction and multi-screen display have become a fashion and a necessity. In related technologies, multiple display screens display the same content, that is, multi-screen display, which cannot realize private customization. At the same time, the multi-screen display process is lengthy, the display efficiency is low, and in serious cases, it even causes different display synchronization. SUMMARY

[0003] Therefore, the embodiments of the present application provide a multi-screen interaction method, device, system and storage medium to solve the problems that the multi-screen display cannot realize private customization and the display efficiency is low in the multi-screen interaction process of the prior art.

[0004] The first aspect of the present application provides a multi-screen interaction method for a multi-screen interaction system, the multi-screen interaction system including a display chip, a display memory, a main display screen and a secondary display screen. The display memory includes a plurality of layer display memory areas, a main display memory area and a virtual display memory area. The plurality of layer display memory areas respectively store graphic data of a plurality of application programs currently running on the display chip. The main display memory area stores synthesis data of the graphic data of the plurality of application programs. The plurality of application programs include a target application program. The virtual display memory area corresponds to the secondary display screen. The multi-screen interaction method includes: determining a secondary display screen selected by a user and a screen projection content based on a screen projection operation of the user; when the screen projection content is the target application program, storing the graphic data of the target application program in the virtual display memory area; sending the graphic data in the virtual display memory area to the secondary display screen for display; and sending the synthesis data in the main display memory area to the main display screen for display.

[0005] In one embodiment, determining the secondary display screen selected by the user and the screen projection content based on the screen projection operation of the user includes: determining the secondary display screen selected by the user and the screen projection content based on a screen projection operation of the user on the main display screen.

[0006] In one embodiment, the main display screen is provided with a screen projection control cascade menu. Determining the secondary display screen selected by the user and the screen projection content based on the screen projection operation of the user on the main display screen includes: determining the secondary display screen selected by the user and the screen projection content based on a click operation of the user on the screen projection control cascade menu.

[0007] In an embodiment, when the screen projection content is the target application, storing the graphic data of the target application into the virtual display memory region comprises: when the screen projection content is the target application, determining the virtual display memory region corresponding to the secondary display screen and a target layer memory region where the graphic data of the target application is located; copying the graphic data in the target layer memory region into the virtual display memory region.

[0008] In an embodiment, sending the graphic data in the virtual display memory region to the secondary display screen comprises: calling a refresh function interface to send the graphic data in the virtual display memory region to the secondary display screen.

[0009] In an embodiment, after determining the secondary display screen selected by the user and the screen projection content based on the screen projection operation of the user, the method further comprises: when the screen projection content is a plurality of applications, storing the synthesized data into the virtual display memory region.

[0010] In an embodiment, the multi-screen interaction method further comprises the following steps executed in parallel with the step of determining the secondary display screen selected by the user and the screen projection content based on the screen projection operation of the user: synthesizing the graphic data in the plurality of layer memory regions, and storing the synthesized data into the main display memory region.

[0011] The second aspect of the present application provides a multi-screen interaction device for a multi-screen interaction system, the multi-screen interaction system comprising a display chip, a display memory, a main display screen and a secondary display screen; the display memory comprising a plurality of layer memory regions, a main display memory region and a virtual display memory region, the plurality of layer memory regions respectively storing graphic data of a plurality of applications currently running on the display chip, the main display memory region storing synthesized data of the graphic data of the plurality of applications, the plurality of applications comprising a target application, the virtual display memory region corresponding to the secondary display screen. The multi-screen interaction device comprises: a determination module configured to determine a secondary display screen selected by a user and screen projection content based on a screen projection operation of the user; a storage module configured to store graphic data of the target application into the virtual display memory region when the screen projection content is the target application; a first sending module configured to send the graphic data in the virtual display memory region to the secondary display screen for display; and a second sending module configured to send the synthesized data in the main display memory region to the main display screen for display.

[0012] The third aspect of the present application provides a multi-screen interaction system, comprising: a display chip, a display memory, a main display screen and a secondary display screen, the display memory, the main display screen and the secondary display screen being communicatively connected to the display chip; wherein the display chip comprises a processor and an internal memory, the internal memory storing a computer program, the computer program being executed by the processor to implement the multi-screen interaction method provided in any of the above embodiments.

[0013] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-screen interaction method provided in any of the above embodiments.

[0014] The multi-screen interaction method, device, system and storage medium provided by the embodiments of the present application are used in a multi-screen interaction system, which includes a display chip, a display memory, a main display screen and a secondary display screen. The display memory includes a plurality of layer display memory areas, a main display memory area and a virtual display memory area. The plurality of layer display memory areas respectively store graphic data of a plurality of application programs currently running on the display chip. The main display memory area stores synthesis data of the graphic data of the plurality of application programs. The plurality of application programs include a target application program. The virtual display memory area corresponds to the secondary display screen. The multi-screen interaction method includes: determining the secondary display screen selected by a user and screen projection content based on a screen projection operation of the user; when the screen projection content is the target application program, storing graphic data of the target application program into the virtual display memory area; sending the graphic data in the virtual display memory area to the secondary display screen for display; and sending the synthesis data in the main display memory area to the main display screen for display. It can be seen that, according to the multi-screen interaction method, device, system and storage medium provided by the embodiments of the present application, the user can selectively project any one of the plurality of currently running application programs, i.e., the target application program, onto the secondary display screen. On the one hand, private customization is achieved, and user experience satisfaction is improved. On the other hand, compared with a conventional scheme in which synthesis data displayed on the main display screen is all projected onto the secondary display screen, the graphic data of the target application program is directly projected onto the secondary display screen before synthesis, so that display efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart of a multi-screen interaction process in the related art.

[0016] Figure 2 A structure block diagram of a multi-screen interaction system provided by an embodiment of the present application.

[0017] Figure 3 A logic architecture diagram of the multi-screen interaction system shown in Figure 2

[0018] Figure 4 A flowchart of a multi-screen interaction method provided by a first embodiment of the present application.

[0019] Figure 5 A data flow diagram in a multi-screen interaction process provided by an embodiment of the present application.

[0020] ​Figure 6 A flowchart of a multi-screen interaction method provided for a second embodiment of the present application is shown.

[0021] Figure 7 A flowchart of a multi-screen interaction method provided for a third embodiment of the present application is shown.

[0022] Figure 8 A flowchart of a multi-screen interaction method provided for a fourth embodiment of the present application is shown.

[0023] Figure 9 A flowchart of a multi-screen interaction method provided for a fifth embodiment of the present application is shown.

[0024] Figure 10 A cockpit sound area distribution diagram provided for an embodiment of the present application is shown.

[0025] Figure 11 A structural block diagram of a multi-screen interaction device provided for a first embodiment of the present application is shown.

[0026] Figure 12 A structural block diagram of a multi-screen interaction device provided for a second embodiment of the present application is shown.

[0027] Figure 13 A structural block diagram of a multi-screen interaction device provided for a third embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] A virtual machine monitor, also known as a Hypervisor, is the core of all virtualization technologies. The Hypervisor is an intermediate layer software running between a physical server and an operating system, which can allow multiple operating systems and applications to share a set of underlying physical hardware. When the Hypervisor runs on the hardware or the operating system, the Hypervisor allows the creation of multiple virtual machines, and allocates an appropriate amount of memory, CPU, network and disk resources to each virtual machine.

[0029] In the related art, multi-screen display is usually implemented based on virtual display technology. Figure 1 A flowchart of a process for implementing multi-screen display in the related art is shown. As shown in Figure 1 The multi-screen display process 100 includes the following steps.

[0030] In step S110, a virtual machine is created based on the Hypervisor, and a virtual display canvas (surface) and a virtual display buffer corresponding to the virtual machine are obtained. The virtual machine is used for a secondary display screen, i.e., the address of the virtual machine is bound to the secondary display screen, so as to establish a corresponding relationship among the secondary display screen, the virtual display surface and the virtual display buffer.

[0031] The virtual surface is an abstract block of memory used as a canvas for image drawing. The virtual buffer is an area in the graphics card specifically for use by the secondary display.

[0032] Step S120 ( Figure 1 (Not shown in the image), it writes the graphics data of multiple currently running applications to the main display memory (buffer). The main display buffer is an area in the graphics card specifically for use by the main display screen.

[0033] Step S130: Store the data in the main display buffer into the virtual display buffer.

[0034] In step S140, the graphic data in the main display buffer is displayed on the main display screen, and the graphic data in the virtual display buffer is displayed on the secondary display screen.

[0035] for Figure 1 The multi-screen display process shown has several drawbacks. First, the main display buffer and the virtual display buffer store the same graphics data, resulting in identical content displayed on both the main and secondary displays, making it impossible to customize the display to meet the user's specific needs. Second, the execution of step S130 requires the participation of the Window Manager Service (WMS), Display Manager Service (DMS), and Surface Flinger (SF), making the process lengthy. Furthermore, step S130 necessitates the compositing of graphics data from multiple applications before copying the composite data to the virtual display buffer. This leads to low display efficiency.

[0036] To address at least one of the aforementioned two technical problems, embodiments of this application provide a multi-screen interaction method, apparatus, system, and storage medium for a multi-screen interaction system. The multi-screen interaction system includes a display chip, video memory, a main display screen, and a secondary display screen. The video memory includes multiple layered video memory areas, a main video memory area, and a virtual video memory area. The multiple layered video memory areas respectively store graphics data of multiple applications currently running on the display chip. The main video memory area stores composite data of the graphics data of the multiple applications. The multiple applications include a target application. The virtual video memory area corresponds to the secondary display screen. The multi-screen interaction method includes: determining the secondary display screen and the projected content selected by the user based on the user's screen projection operation; when the projected content is the target application, storing the graphics data of the target application in the virtual video memory area; sending the graphics data in the virtual video memory area to the secondary display screen for display; and sending the composite data in the main video memory area to the main display screen for display. As can be seen, according to the multi-screen interaction method, apparatus, system and storage medium provided in the embodiments of this application, users can selectively project any one of the currently running multiple applications, i.e., the target application, onto the secondary display screen. On the one hand, this achieves private customization and improves user experience satisfaction; on the other hand, compared with the conventional solution that requires all the composite data displayed on the main display screen to be projected onto the secondary display screen, there is no need to wait for the graphic data to be composited before projection. Instead, the graphic data of the target application can be projected onto the secondary display screen directly before composited, thereby improving display efficiency.

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Figure 2 This is a structural block diagram of a multi-screen interactive system provided in an embodiment of this application. Figure 3 for Figure 2 The diagram shows the logical architecture of a multi-screen interaction system. This multi-screen interaction system is an application scenario of the multi-screen interaction method provided in the embodiments of this application, such as an in-vehicle infotainment system. Figure 2 As shown, the multi-screen interactive system 20 includes a display chip 21, video memory 22, a main display screen 23, and at least one secondary display screen 24. The video memory 22, the main display screen 23, and the at least one secondary display screen 24 are all communicatively connected to the display chip 21. The video memory 22 can be integrated into the display chip 21 or it can be set up separately.

[0039] Referring to Figure 3 The display memory 22 includes a plurality of layer buffers 221. The layer buffer 221, i.e., a layer display memory region, is a predetermined storage region in the display memory 22. The layer buffer 221 corresponds to each of the plurality of applications currently running on the display chip 21. The layer buffer 221 is used to store the graphics data of the application. When the application is opened, a layer buffer 221 is allocated to the application, and the address of the layer buffer 221 does not change until the application is destroyed.

[0040] The logical architecture of the multi-screen interaction system 20 includes a main display unit 25 and at least one virtual display unit 26. The main display unit is used to drive the main display screen 23 to display the synthesized data in the main display buffer 222. The main display buffer 222 is a display memory region in the display memory 22, which is a predetermined storage region in the display memory 22 and is used by the main display unit 25 to store the synthesized data of the graphics data of the plurality of applications currently running on the display chip 21, i.e., a frame of picture displayed by the main display screen 23. The virtual display unit 26 corresponds to the virtual machine running on the display chip 21. The virtual display unit 26 is used to drive the secondary display screen 24 to display the graphics data in the virtual display buffer 223. The virtual display buffer 223, i.e., a virtual display memory region, is a predetermined storage region in the display memory 22 and is used by the secondary display screen 24 to store the synthesized data of the graphics data of the selected screen projection content, including any one of the plurality of applications currently running, denoted as a target application, or all of the applications currently running.

[0041] The display chip 21 includes a processor 211 and an internal memory 212. The processor 211 can be a processing unit having data processing capability and / or instruction execution capability, such as a central processing unit (CPU). The internal memory 212 stores a computer program. When the processor 211 executes the computer program, the multi-screen interaction method provided by the embodiments of the present application can be implemented. Referring to Figure 3 According to the actual needs of the user, the graphics data of the selected target application is copied from the layer buffer 221 to the virtual display buffer 223, so that the secondary display screen 24 corresponding to the virtual display buffer 223 displays the target application.

[0042] At least one of the internal memory 212 and the display memory 22 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory includes random access memory (RAM) and / or cache memory, etc. The non-volatile memory includes read-only memory (ROM), hard disk, flash memory, etc.

[0043] Figure 4 A flowchart of a multi-screen interaction method provided for a first embodiment of the present application is shown. Figure 5 A data flow diagram in a multi-screen interaction process provided for an embodiment of the present application is shown. In combination with the description of the multi-screen interaction method 400 shown in Figure 4 and Figure 5 The multi-screen interaction method 400 includes the following steps:

[0044] In step S410, the user-selected secondary display screen and the screen casting content are determined based on the user's screen casting operation.

[0045] The user-selected secondary display screen is any secondary display screen 24 in the multi-screen interaction system 20. The screen casting content can be any of the currently running application programs, denoted as the target application program, or all the application programs. As shown in Figure 5 The application programs displayed on the primary display screen can include the status bar, the navigation bar, the notification, the floating window, the iQiyi, the Tencent news, the Douyin segment video, etc. When the screen casting content is the target application program, the display contents of the secondary display screen 23 and the primary display screen 24 are different, realizing multi-screen different display. When the screen casting content is all the application programs currently displayed on the primary display screen, the display contents of the secondary display screen 23 and the primary display screen 24 are the same, realizing multi-screen same display.

[0046] The screen casting operation can be performed on the primary display screen 23 or the secondary display screen 24. In an embodiment, the user-selected secondary display screen and the screen casting content are determined based on the user's screen casting operation on the primary display screen 23. For example, a screen casting control cascading menu is provided on the primary display screen 23, which includes the identification options of multiple secondary display screens 24 and the identification options of multiple application programs. The user clicks the identification option of the secondary display screen to be cast and the identification option of the target application program on the screen casting control cascading menu, triggering the primary display screen 23 to issue a screen casting instruction. The display chip 21 determines the selected secondary display screen and the target application program based on the screen casting instruction. By performing the screen casting operation on the primary display screen 23, the primary control identity of the primary display screen 23 is consistent, which is more in line with the actual demand.

[0047] In step S420, when the screen casting content is the target application program, the graphical data of the target application program is stored in the virtual display memory area.

[0048] In one embodiment, step S420 is executed by the screen layer delivery service, i.e. the graphic data of the target application is stored into the virtual display memory region 223 within the screen layer delivery service process. Specifically, the corresponding relationship between the secondary display screen 24 and the virtual display buffer 223, and the corresponding relationship between the application and the layer buffer 221 are stored in the internal memory 212. The corresponding relationship between the secondary display screen 24 and the virtual display buffer 223 is obtained when the virtual machine is created, and the corresponding relationship between the application and the layer buffer 221 is obtained when the application is opened on the primary display screen 23. When the secondary display screen and the target application are determined, the screen layer delivery service can directly obtain the virtual display buffer 223 corresponding to the secondary display screen and the layer buffer 221 corresponding to the target application from the internal memory, and then copy the graphic data in the layer buffer 221 into the virtual display buffer 223. It can be seen that the execution process of step S420 is executed within the screen layer delivery service, without the participation of other services such as the window management service and the display management service, i.e. without cross-service execution, thereby improving the processing speed.

[0049] Step S430, the graphic data in the virtual display buffer 223 is sent to the secondary display screen 23 for display.

[0050] In one embodiment, after step S420 is executed, the secondary display screen 23 is immediately refreshed, i.e. a refresh function interface is called to send the graphic data in the virtual display buffer 223 to the secondary display screen 23, and the secondary display screen is actively refreshed without waiting for the preset refresh time to come. In this way, the display efficiency can be further improved.

[0051] Step S440, the synthesized data in the primary display buffer 222 is sent to the primary display screen 24 for display.

[0052] According to the multi-screen interaction method provided in the embodiment, the secondary display screen selected by the user and the screen casting content are determined based on the screen casting operation of the user; when the screen casting content is a target application, the graphic data of the target application is stored into a virtual display memory region; the graphic data in the virtual display memory region is sent to the secondary display screen for display; and the synthesized data in the primary display memory region is sent to the primary display screen for display. It can be seen that the user can selectively cast any one of the plurality of currently running applications, i.e. a target application, onto the secondary display screen. On the one hand, private customization is achieved, and the user experience satisfaction is improved. On the other hand, compared with the conventional scheme of casting all the synthesized data displayed on the primary display screen to the secondary display screen, the graphic data of the target application is directly cast to the secondary display screen before synthesis, thereby improving the display efficiency.

[0053] Figure 6 A flowchart of the multi-screen interaction method provided by the second embodiment of the present application is shown in FIG. 6. In this embodiment, the multi-screen interaction method 600 is combined with the multi-screen interaction method 400 provided by the first embodiment of the present application. Figure 5 As shown in FIG. 6, the multi-screen interaction method 600 further includes the following steps performed before step S410:

[0054] In step S610, the graphic data of the plurality of application programs obtained is respectively stored in the plurality of layer buffers 211. The screen layer delivery service synthesizes and renders the graphic interfaces of the plurality of application programs on the canvas to obtain graphic data that can be used for display of the display screen, and stores the graphic data in the plurality of layer buffers 211. It should be noted that the synthesis referred to herein means synthesizing graphics in the same application program to obtain the graphic interface of the application program.

[0055] In this case, the multi-screen interaction method 600 further includes the following steps performed before step S440:

[0056] In step S610, the graphic data obtained from the plurality of layer buffers 221 is synthesized to obtain synthesis data.

[0057] The graphic data of the plurality of application programs currently running has been respectively stored in the plurality of layer buffers 221. At this time, the graphic data of the plurality of application programs can be obtained from the plurality of layer buffers 221. Then, the graphic data of the plurality of application programs is synthesized to obtain a frame of image.

[0058] In step S620, the synthesis data is stored in the main display buffer 222 for display of the main display screen 23.

[0059] In one embodiment, after step S610, two parallel threads are set, one thread executes step S620, step S630 and step S440, and the other thread executes step S410, step S420 and step S430. By setting two threads to execute in parallel, the display efficiency is maximized as much as possible.

[0060] Figure 7 A flowchart of the multi-screen interaction method provided by the third embodiment of the present application is shown in FIG. 7. As shown in FIG. 7, in this embodiment, the multi-screen interaction method 700 is based on the multi-screen interaction method provided by any of the above embodiments (for example, the multi-screen interaction method 400), and further includes: Figure 7

[0061] In step S710, when the screen projection content is a plurality of application programs, the synthesis data is stored in the virtual display buffer 222. When the screen projection content is a plurality of application programs, i.e., when the multi-screen same display is desired, the synthesis data needs to be stored in the virtual display buffer 222. ​

[0062] According to the multi-screen interaction method provided in this embodiment, multi-screen display and multi-screen simultaneous display can be realized according to the user's actual needs.

[0063] Figure 8 This is a flowchart illustrating the multi-screen interaction method provided in the fourth embodiment of this application. Figure 8 As shown, the multi-screen interaction method 800 is a multi-screen interaction method provided in any of the above embodiments ( Figure 8 Based on the multi-screen interaction method 400 (as an example), it also includes the following steps performed before step S410:

[0064] Step S810: Create a virtual machine based on the specifications of the main graphics memory. The specifications of the virtual machine's virtual graphics buffer are consistent with those of the main graphics buffer. This ensures that the storage space of the virtual graphics buffer is sufficient to enable multi-screen simultaneous display, that is, the secondary display and the main display show the same content.

[0065] The multi-screen interaction method provided in any of the above embodiments enables video data interaction between multiple displays, including simultaneous display and asynchronous display. In practical applications, the multi-screen interaction method may also include an audio interaction step.

[0066] Figure 9 This is a flowchart illustrating the multi-screen interaction method provided in the fifth embodiment of this application. Figure 9 As shown, the multi-screen interaction method 900, based on the multi-screen interaction method provided in any of the above embodiments, further includes:

[0067] Step S910: Determine the target audio playback mode based on the user's selection of the audio playback mode. The target audio playback mode includes the correspondence between the audio channels and sound zones of the secondary display screen.

[0068] Each display screen corresponds to one audio channel. The audio playback mode includes the correspondence between audio channels and sound zones. When a user selects an audio playback mode, they are essentially assigning the audio channels of the secondary display screen to different sound zones according to their actual needs.

[0069] Taking the vehicle infotainment system as an example, Figure 10 This is a schematic diagram of the cockpit sound zone distribution provided in one embodiment of this application. Figure 10 As shown, a typical in-vehicle infotainment system includes a driver's side display, a passenger side display, a left rear seat display, and a right rear seat display. Each of these four displays corresponds to an audio channel, and each audio channel corresponds to a sound player 100. The in-vehicle infotainment system includes four-zone mode, full-zone mode, front and rear zone mode, and left and right zone mode. Taking four-zone mode as an example... Figure 10As shown, when the user selects the four sound area mode, the four display screens correspond to the four sound areas of the cabin, denoted as the main driver sound area a, the co-driver sound area b, the left rear seat sound area c and the right rear seat sound area d. Similarly, when the user selects the front and rear sound area mode, the main driver display screen and the co-driver display screen correspond to the front sound area, and the left rear seat display screen and the right rear seat display screen correspond to the rear sound area. When the user selects the left and right sound area mode, the main driver display screen and the left rear seat display screen correspond to the left sound area, and the co-driver display screen and the right rear seat display screen correspond to the right sound area.

[0070] Step S920, while executing step S430, sends the audio data of the target application program to all audio channels corresponding to the sound area of the audio channel corresponding to the auxiliary display screen, so as to play the audio data in the sound area.

[0071] Continuing the above example, for example, when the driver display screen plays music and the co-driver display screen plays a TV series, the user can select the four sound area mode. In this case, the audio data of the music is only played in the main driver sound area a, and the audio data of the TV series is only played in the co-driver sound area b, avoiding interference between different sound areas. Of course, the user can also select the left and right play mode. In this case, the audio data of the music is only played in the left sound area, and the audio data of the TV series is only played in the right sound area, which can also avoid interference between different sound areas. For example, when the driver display screen plays music and the left rear seat display screen plays an animation, the user can select the front and rear sound area mode. In this case, the audio data of the music is only played in the front sound area, and the audio data of the animation is only played in the rear sound area, thereby avoiding interference between different sound areas.

[0072] The application also provides a multi-screen interactive device. Figure 11 The structural block diagram of the multi-screen interactive device provided by the first embodiment of the application is shown in the figure. Figure 11 As shown, the multi-screen interactive device 1100 includes a determination module 1110, a storage module 1120, a first sending module 1130 and a second sending module 1140. The determination module 1110 is configured to determine the auxiliary display screen selected by the user and the screen projection content based on the screen projection operation of the user. The storage module 1120 is configured to store the graphic data of the target application program in the virtual display memory area when the screen projection content is the target application program. The first sending module 1130 is configured to send the graphic data in the virtual display buffer 223 to the auxiliary display screen 23 for display. The second sending module 1140 is configured to send the synthesized data in the main display buffer 222 to the main display screen 24 for display.

[0073] According to the multi-screen interaction device provided in the embodiment, the secondary display screen selected by the user and the screen projection content are determined based on the screen projection operation of the user; when the screen projection content is a target application program, the graphic data of the target application program is stored in the virtual display buffer area; the graphic data in the virtual display buffer area is sent to the secondary display screen for display; and the composite data in the main display buffer area is sent to the main display screen for display. As can be seen, the user can selectively project any one of the currently running application programs, i.e., a target application program, onto the secondary display screen. On the one hand, this realizes private customization and improves user experience satisfaction. On the other hand, compared with the conventional scheme of projecting all the composite data displayed on the main display screen to the secondary display screen, the graphic data of the target application program can be directly projected to the secondary display screen before the synthesis, thereby improving the display efficiency.

[0074] In one embodiment, the determining module 1110 is specifically configured to determine the secondary display screen selected by the user and the screen projection content based on the screen projection operation of the user on the main display screen 23. Specifically, the screen projection control cascade menu is arranged on the main display screen, and the secondary display screen selected by the user and the screen projection content are determined based on the click operation of the user on the screen projection control cascade menu. The screen projection operation is performed on the main display screen 23, which is consistent with the master control identity of the main display screen 23 and is more in line with the actual demand.

[0075] In one embodiment, the storing module 1120 is specifically configured to store the graphic data of the target application program into the virtual display buffer area in the screen layer delivery service process when the screen projection content is the target application program. Specifically, the virtual display buffer area corresponding to the secondary display screen and the target layer buffer area where the graphic data of the target application program is located are determined in the screen layer delivery service process; and the graphic data in the target layer buffer area is copied into the virtual display buffer area. In this case, the determining module 1110 is a functional module in the same service, and the implementation of the function of the determining module 1110 does not need to be executed across services, thereby improving the processing speed.

[0076] In one embodiment, the first sending module 1130 is specifically configured to call a refresh function interface and send the graphic data in the virtual display buffer 223 to the secondary display screen 23. That is, the secondary display screen is actively refreshed without waiting for the preset refresh time to come. In this way, the display efficiency can be further improved.

[0077] In one embodiment, the storing module 1120 is further configured to store the composite data into the virtual display buffer 222 when the screen projection content is the plurality of application programs. In this case, the multi-screen heterogeneous display and the multi-screen homogeneous display can be realized according to the actual demand of the user.

[0078] Figure 12 A structural block diagram of the multi-screen interaction device provided in the second embodiment is shown in FIG. 2. As shown in FIG. 2, the multi-screen interaction device comprises a main display screen 23, a secondary display screen 24, a main display buffer 221, a virtual display buffer 222, a screen layer delivery service process, a virtual display buffer area, a target layer buffer area, a first sending module 1130, a second sending module 1140, and a third sending module 1150.Figure 10 As shown in the multi-screen interactive device 1200, the storage module 1120 is further configured to store the acquired graphical data of the plurality of application programs into a plurality of layer buffers 211 respectively. In this case, the multi-screen interactive device 1200 further comprises a synthesis module 1210 configured to synthesize the graphical data acquired from the plurality of layer buffers 221 to obtain synthesis data. The storage module 1120 is further configured to store the synthesis data into a main display buffer 222 for display on the main display screen 23.

[0079] Figure 13 A structural block diagram of a multi-screen interactive device according to a third embodiment of the present application is provided. As shown in the multi-screen interactive device 1300, the multi-screen interactive device 1300 is based on any of the above real-time provided multi-screen interactive devices, and further comprises a creation module 1310 configured to create a virtual machine based on the specification of the main display memory, and the specification of a virtual display buffer corresponding to the virtual machine is consistent with that of the main display buffer. In this way, the storage space of the virtual display buffer is ensured to be sufficient for multi-screen same display, i.e., the secondary display screen and the main display screen display the same content. Figure 13

[0080] The embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the multi-screen interactive method provided by any of the above embodiments.

[0081] The computer readable storage medium can adopt any combination of one or more readable media. The readable storage medium can be in any form of electric, magnetic, optical, electromagnetic, infrared, semiconductor, or a combination thereof. For example, the readable storage medium includes a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, etc.

[0082] The above description has been given for the purpose of illustration and description. Furthermore, this description does not intend to limit the embodiments of the present application to the forms disclosed herein. Although a plurality of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.​

Claims

1. A multi-screen interaction method, characterized in that, A multi-screen interaction system comprises a display chip, a display memory, a main display screen and a secondary display screen. The display memory comprises a plurality of layer display memory areas, a main display memory area and a virtual display memory area. The plurality of layer display memory areas respectively store graphic data of a plurality of application programs currently running on the display chip. The main display memory area stores synthesis data of the graphic data of the plurality of application programs. The plurality of application programs comprise a target application program. The virtual display memory area corresponds to the secondary display screen. The multi-screen interaction method comprises: determining the secondary display screen and the screen projection content selected by a user based on a screen projection operation of the user; when the screen projection content is the target application program, determining the virtual display memory area corresponding to the secondary display screen and the target layer display memory area where the graphic data of the target application program is located through a screen layer delivery service process, and storing the graphic data in the target layer display memory area into the virtual display memory area corresponding to the secondary display screen; sending the graphic data in the virtual display memory area to the secondary display screen for display; sending the synthesis data in the main display memory area to the main display screen for display.

2. The multi-screen interaction method of claim 1, wherein, The method for determining the secondary display screen and the screen projection content selected by the user based on the screen projection operation of the user comprises: determining the secondary display screen and the screen projection content selected by the user based on a screen projection operation of the user on the main display screen.

3. The multi-screen interaction method of claim 1, wherein, The method for sending the graphic data in the virtual display memory area to the secondary display screen comprises: calling a refresh function interface to send the graphic data in the virtual display memory area to the secondary display screen.

4. The method of any of claims 1-3, wherein, After the method for determining the secondary display screen and the screen projection content selected by the user based on the screen projection operation of the user, the method further comprises: when the screen projection content is the plurality of application programs, storing the synthesis data into the virtual display memory area.

5. The method of any of claims 1-3, wherein, The method further comprises the following steps performed in parallel with the method for determining the secondary display screen and the screen projection content selected by the user based on the screen projection operation of the user: synthesizing the graphic data in the plurality of layer display memory areas and storing the synthesis data into the main display memory area.

6. The method of claim 1-3, wherein, The method further comprises: determining a target audio playing mode based on a selection operation of the user on the audio playing mode, the target audio playing mode comprising a correspondence between an audio channel and an audio area of the secondary display screen; simultaneously with the method for sending the graphic data in the virtual display memory area to the secondary display screen for display, sending audio data of the target application program to all the audio channels corresponding to the audio area to play the audio data in the audio area.

7. A multi-screen interaction device, characterized in that, A multi-screen interaction system comprises a display chip, a display memory, a main display screen and a secondary display screen. The display memory comprises a plurality of layer display memory areas, a main display memory area and a virtual display memory area. The plurality of layer display memory areas respectively store graphic data of a plurality of application programs currently running on the display chip. The main display memory area stores composite data of the graphic data of the plurality of application programs. The plurality of application programs comprise a target application program. The virtual display memory area and the secondary display screen correspond to each other. The multi-screen interaction system comprises: A determining module configured to determine the secondary display screen and the screen projection content selected by a user based on a screen projection operation of the user. A storage module configured to, when the screen projection content is the target application program, determine a virtual display memory area corresponding to the secondary display screen and a target layer display memory area in which the graphic data of the target application program is located through a screen layer delivery service process, and store the graphic data in the target layer display memory area into the virtual display memory area corresponding to the secondary display screen. A first sending module configured to send the graphic data in the virtual display memory area to the secondary display screen for display. A second sending module configured to send the composite data in the main display memory area to the main display screen for display.

8. A multi-screen interaction system, characterized in that, The multi-screen interaction system comprises: A display chip, a display memory, a main display screen and a secondary display screen. The display memory, the main display screen and the secondary display screen are respectively communicatively connected to the display chip. The display chip comprises a processor and an internal memory. The internal memory stores a computer program. The computer program is executed by the processor to implement the multi-screen interaction method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-screen interaction method of any one of claims 1-6.

Images