Screen Expansion Method, Device, Terminal Device, and Computer Readable Storage Medium

Abstract

This application is applicable to the field of image processing, and provides a screen extension method, device, terminal device, and computer-readable storage medium. In the screen extension method of this application, the first terminal device renders and synthesizes the image data of each application program to obtain screen display data that can be directly displayed on the screen. Then, the first terminal device splits the screen display data into first screen display data and second screen display data, displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device. Since the second screen display data is data that has been rendered and synthesized, when the second terminal device can directly perform a screen operation on the second screen display data, there is no need to re-render and synthesize the second screen display data, reducing the occurrence of screen refresh lag and reducing the latency of the screen image displayed on the second display screen.

Description

Technical Field

[0001] This application belongs to the field of image processing, and particularly relates to a screen extension method, device, terminal device, and computer-readable storage medium. Background Art

[0002] Screen extension technology allows users to use multiple display screens for collaborative work, entertainment, gaming, etc. simultaneously, which can greatly improve the user experience.

[0003] In the current screen extension solution, the main screen device sends the display information that the extended screen needs to display to the terminal device where the extended screen is located. After receiving the display information, the terminal device where the extended screen is located needs to perform rendering and composition operations on the display information, and it is difficult to quickly display the display information on the screen, resulting in easy refresh lag on the extended screen. Moreover, compared with the screen image displayed on the main screen, there is a large time delay in the screen image displayed on the extended screen. Summary of the Invention

[0004] The embodiments of this application provide a screen extension method, device, terminal device, and computer-readable storage medium, which can solve the problems that in the current extended screen solution, the extended screen is prone to refresh lag when displaying the screen image, and there is a large time delay between the screen image displayed on the extended screen and the screen image displayed on the main screen.

[0005] The first aspect of the embodiments of this application provides a screen extension method, including:

[0006] The first terminal device obtains the graphic data of each application program, renders the graphic data of each application program, and synthesizes the rendered graphic data corresponding to each application program into screen display data;

[0007] The first terminal device splits the screen display data into first screen display data and second screen display data;

[0008] The first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

[0009] It should be noted that after the first terminal device obtains the graphic data of each application program, it renders the graphic data of each application program to obtain the window data corresponding to each application program, and synthesizes the window data corresponding to each application program into screen display data.

[0010] Then, the first terminal device further splits the screen display data into first screen display data and second screen display data.

[0011] Since both the first screen display data and the second display data are rendered and synthesized data, the first terminal device can directly display the first screen display data on the first display screen (i.e., the display screen of the first terminal device), and send the second screen display data to the second terminal device.

[0012] After the first terminal device sends the second screen display data to the second terminal device, the second terminal device can directly display the second screen display data on the second display screen (i.e., the display screen of the second terminal device), without performing rendering and synthesizing operations, improving the screen display speed of the second screen display data, reducing the occurrence of screen refresh lag, and reducing the latency of the screen image displayed on the second display screen.

[0013] In a possible implementation manner of the first aspect, before the first terminal device acquires the graphic data of each application, it further includes:

[0014] When the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device determines the terminal device corresponding to the screen event as the second terminal device.

[0015] It should be noted that when the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device enters the extended screen mode, determines the terminal device corresponding to the screen event as the second terminal device, and determines the display screen of the second terminal device as the second display screen.

[0016] In a possible implementation manner of the first aspect, the step of when the first terminal device monitors a screen event that meets a preset trigger condition and the first terminal device determines the terminal device corresponding to the screen event as the second terminal device includes:

[0017] When the first terminal device monitors a screen movement event, the first terminal device determines the terminal device corresponding to the screen movement event as a candidate terminal device;

[0018] If the distance between the display screen of the candidate terminal device and the display screen of the first terminal device is less than a preset distance threshold, the first terminal device determines the candidate terminal device as the second terminal device.

[0019] It should be noted that the above screen event may be a screen movement event. When the first terminal device monitors a screen movement event, it determines the terminal device corresponding to the screen movement event as a candidate terminal device.

[0020] At this time, the above preset trigger condition may be that the distance between the display screen of the candidate terminal device and the first display screen is less than a preset distance threshold. When the preset trigger condition is met, the first terminal device determines the candidate terminal device that meets the preset trigger condition as the second terminal device, and determines the display screen of the second terminal device as the second display screen.

[0021] In another possible implementation manner of the first aspect, when the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device determines the terminal device corresponding to the screen event as the second terminal device, including:

[0022] When the first terminal device monitors a screen touch event, the first terminal device determines the terminal device corresponding to the screen touch event as the second terminal device.

[0023] It should be noted that the above screen event may also be a screen touch event.

[0024] When the first terminal device monitors a screen touch event, the first terminal device may determine the terminal device corresponding to the screen touch event as the second terminal device, and determine the display screen of the second terminal device as the second display screen.

[0025] In another possible implementation manner of the first aspect, before the first terminal device acquires the graphic data of each application program, it further includes:

[0026] When the first terminal device detects a Near Field Communication (NFC) signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold, the first terminal device determines the terminal device corresponding to the NFC signal as the second terminal device.

[0027] It should be noted that the trigger condition for the first terminal device to enter the extended screen mode may also be that the first terminal device detects a Near Field Communication (NFC) signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold.

[0028] When the first terminal device detects an NFC signal and the NFC signal meets the above trigger condition, the first terminal device may determine the terminal device that generates the NFC signal as the second terminal device, and determine the display screen of the second terminal device as the second display screen.

[0029] In another possible implementation manner of the first aspect, before the first terminal device acquires the graphic data of each application program, it further includes:

[0030] When the first terminal device obtains a screen expansion instruction, the first terminal device determines the terminal device indicated in the screen expansion instruction as the second terminal device.

[0031] It should be noted that the trigger condition for the first terminal device to enter the extended screen mode can also be that the first terminal device receives a screen expansion instruction.

[0032] When the user wants the first terminal device to enter the extended screen mode, the user can trigger a screen expansion instruction by operating the first terminal device. The screen expansion instruction includes a screen expansion object, and the screen expansion object can be the identifier of the second terminal device.

[0033] The first terminal device can determine the terminal device indicated in the screen expansion instruction as the second terminal device, and determine the display screen of the second terminal device as the second display screen.

[0034] In a possible implementation manner of the first aspect, the first terminal device renders the graphic data of each application program, including:

[0035] The first terminal device obtains a target display resolution;

[0036] The first terminal device renders the graphic data of each application program according to the target display resolution to obtain window data corresponding to each application program.

[0037] It should be noted that the first terminal device can virtualize the first display resolution of the first display screen and the second display resolution of the second display screen to the target display resolution, and render the graphic data of each application program according to the target display resolution, reducing the rendering difficulty of the graphic data.

[0038] In a possible implementation manner of the first aspect, the first terminal device obtains a target display resolution, including:

[0039] The first terminal device obtains the first display resolution corresponding to the first terminal device and the second display resolution corresponding to the second terminal device, and determines the maximum value of the first display resolution and the second display resolution as the target display resolution.

[0040] It should be noted that since it is easy to have problems such as picture distortion or picture tearing when displaying low-resolution images on a high-resolution display screen. Therefore, when determining the target display resolution, the first terminal device can obtain the first display resolution corresponding to the first terminal device and the second display resolution corresponding to the second terminal device, and take the larger value of the first display resolution and the second display resolution as the target display resolution.

[0041] Then, the first terminal device renders the graphic data of each application using the target display resolution to avoid problems such as picture distortion or picture tearing.

[0042] In a possible implementation manner of the first aspect, the first terminal device splits the screen display data into first screen display data and second screen display data, including:

[0043] The first terminal device obtains the total number of the display screens of the first terminal device and the second terminal device, and equally splits the screen display data according to the total number to obtain first screen display data and second screen display data.

[0044] It should be noted that when the first terminal device splits the screen display data, it can first obtain the total number of the first display screen and the second display screen, and then split according to the total number. For example, assuming there is 1 first display screen and 2 second display screens, the total number is 3, and the screen display data is equally split into 3 parts to obtain 1 part of first screen display data and 2 parts of second screen display data.

[0045] In a possible implementation manner of the first aspect, the first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device, including:

[0046] The first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device to instruct the second terminal device not to render the second screen display data and display the second screen display data on the display screen of the second terminal device.

[0047] It should be noted that since the second screen display data is data that has undergone rendering and composition operations, the second terminal device does not need to render the second screen display data, and can directly perform the operation of displaying the second screen display data on the second display screen.

[0048] A second aspect of the embodiments of the present application provides a screen expansion device, including:

[0049] A rendering and composition module, configured to obtain the graphic data of each application, render the graphic data of each application, and synthesize the rendered graphic data corresponding to each application into screen display data;

[0050] A data splitting module, configured to split the screen display data into first screen display data and second screen display data;

[0051] A screen expansion module, configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to a second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

[0052] In a possible implementation manner of the second aspect, the apparatus further includes:

[0053] An event monitoring module, configured to determine the terminal device corresponding to the screen event as the second terminal device when a screen event that meets a preset trigger condition is monitored.

[0054] In a possible implementation manner of the second aspect, the event monitoring module includes:

[0055] A movement candidate sub-module, configured to determine the terminal device corresponding to the screen movement event as a candidate terminal device when a screen movement event is monitored;

[0056] A distance determination sub-module, configured to determine the candidate terminal device as the second terminal device if the distance between the display screen of the candidate terminal device and the display screen of the first terminal device is less than a preset distance threshold.

[0057] In another possible implementation manner of the second aspect, the event monitoring module includes:

[0058] A collision determination sub-module, configured to determine the terminal device corresponding to the screen touch event as the second terminal device when a screen touch event is monitored.

[0059] In another possible implementation manner of the second aspect, the apparatus further includes:

[0060] A near-field communication module, configured to determine the terminal device corresponding to the NFC signal as the second terminal device when the first terminal device detects an NFC signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold.

[0061] In another possible implementation manner of the second aspect, the apparatus further includes:

[0062] An instruction determination module, configured to determine the terminal device indicated in the screen expansion instruction as the second terminal device when a screen expansion instruction is obtained.

[0063] In a possible implementation manner of the second aspect, the rendering and composition module includes:

[0064] A resolution sub-module, configured to obtain a target display resolution;

[0065] A graphics rendering sub-module, configured to render the graphic data of each application according to the target display resolution.

[0066] In a possible implementation manner of the second aspect, the resolution sub-module is specifically configured to obtain a first display resolution corresponding to a first terminal device and a second display resolution corresponding to the second terminal device, and determine the maximum value of the first display resolution and the second display resolution as the target display resolution.

[0067] In a possible implementation manner of the second aspect, the data splitting module is specifically configured to obtain the total number of the display screens of the first terminal device and the number of the display screens of the second terminal device, and equally split the screen display data according to the total number to obtain first screen display data and second screen display data.

[0068] In a possible implementation manner of the second aspect, the screen expansion module is specifically configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to the second terminal device to instruct the second terminal device not to render the second screen display data, and display the second screen display data on the display screen of the second terminal device.

[0069] A third aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the terminal device implements the steps of the method as described above.

[0070] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the terminal device implements the steps of the method as described above.

[0071] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device implements the steps of the method as described above.

[0072] The beneficial effects of the embodiments of the present application compared with the prior art are:

[0073] In the screen extension method of this application, the first terminal device renders and synthesizes the image data of each application to obtain screen display data that can be directly displayed on the screen. Then, the first terminal device splits the screen display data into first screen display data and second screen display data, displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device. Since the second screen display data is data that has been rendered and synthesized, when the second terminal device receives the second screen display data, it can directly perform the screen display operation on the second screen display data without re-rendering and synthesizing the second screen display data, reducing the occurrence of screen refresh lag and reducing the latency of the screen image displayed on the second display screen. Description of the Drawings

[0074] Figure 1 is a schematic flowchart of a screen extension method provided by an embodiment of this application;

[0075] Figure 2 is a schematic diagram of a screen movement event provided by an embodiment of this application;

[0076] Figure 3 is a schematic diagram of a screen touch event provided by an embodiment of this application;

[0077] Figure 4 is a schematic diagram of cross-screen display of an application provided by an embodiment of this application;

[0078] Figure 5 is a schematic diagram of a screen display data splitting method provided by an embodiment of this application;

[0079] Figure 6 is a schematic diagram of a local area network provided by an embodiment of this application;

[0080] Figure 7 is a system architecture diagram of an Android operating system provided by an embodiment of this application;

[0081] Figure 8 is a schematic diagram of a producer-consumer model provided by an embodiment of this application;

[0082] Figure 9 is another system architecture diagram of an Android operating system provided by an embodiment of this application;

[0083] Figure 10 is another schematic diagram of a producer-consumer model provided by an embodiment of this application;

[0084] Figure 11 is a schematic structural diagram of a screen extension device provided by an embodiment of this application;

[0085] Figure 12 It is a schematic diagram of the terminal device provided by the embodiments of the present application. Specific embodiments

[0086] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0087] In order to illustrate the technical solutions described in the present application, specific embodiments are used for illustration below.

[0088] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0089] It should also be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0090] It should be further understood that the term "and / or" used in this specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0091] As used in this specification and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0092] The screen extension method provided by the embodiments of this application can be applied to terminal devices. The terminal device can be any device with a display screen and image processing capabilities, including but not limited to smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), artificial intelligence (AI) terminals, and other terminal devices. The embodiments of this application do not impose any restrictions on the specific types of terminal devices.

[0093] For example, the terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device, or other processing devices connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a customer premise equipment (CPE), and / or other devices for communicating on a wireless system, as well as next-generation communication systems, such as mobile terminals in a 5G network or mobile terminals in a future evolved Public Land Mobile Network (PLMN) network.

[0094] In addition, in the description of this application, the terms "first", "second", "third", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0095] Screen extension technology allows users to use multiple display screens simultaneously for collaborative work, entertainment, gaming, etc., which can greatly enhance the user experience.

[0096] On the current Android system, multi-window technology can be implemented in one screen, such as split-screen mode, picture-in-picture mode, FreeFrom mode, etc.

[0097] The core idea of the multi-window technology is stack division and stack boundary setting. The implementation of the multi-window function mainly depends on the ActivityManagerService (AMS) and the WindowManagerService (WMS). AMS mainly manages parameters such as the lifecycle of windows, and WMS mainly manages parameters such as the position and size of windows.

[0098] The multi-window technology is based on the data structure of the stack. A stack contains multiple tasks, and a task contains multiple window activities (Activities).

[0099] The data structures of the stacks and tasks in AMS and WMS correspond one by one. The ActivityStack in AMS corresponds to the TaskStack in WMS, and the ActivityRecord in AMS corresponds to the Task in WMS.

[0100] The above multi-window modes can be achieved by setting the stack ID and stack boundary under different multi-window modes. For example, currently, 5 stack IDs are defined in the Android system: (1) Stack ID = 0. This stack is the Home Stack, which is the stack where the Android system's desktop launcher is located. In addition to the desktop launcher, some system interfaces also run on this stack, such as the recent tasks, etc.; (2) Stack ID = 1. This stack is the FullScreen Stack, where ordinary Activities are located. However, in the split-screen mode, this stack only occupies half of the screen; (3) Stack ID = 2. This stack is the stack where Activities in the free mode are located; (4) Stack ID = 3. This stack is the Docked Stack. In the split-screen mode, a fixed application runs on the other half of the screen, and the Activity of this application is in the Docked Stack; (5) Stack ID = 4. This stack is the Pinned Stack, which is the stack where the Activity in the picture-in-picture mode is located.

[0101] Different Activities may belong to different stacks but may have the same stack ID. By adjusting the boundaries of the above stacks, the size of each task in the stack can be adjusted, and ultimately the size of the task determines the size of the window. However, since the stack boundary can only be the current screen range, the above multi-window technology solves the problem of cutting the screen of a single terminal device into multiple windows, such as cutting the screen into a two-screen, three-screen, four-screen, etc., but it cannot achieve multi-screen expansion. Moreover, the functional scalability of the above multi-window technology is not good and can only be used on applications that support split-screen.

[0102] In addition, some current screen mirroring technologies can achieve multi-screen interaction, such as the push mode and the mirror mode. In the push mode, the mobile phone and the TV are connected to the same local area network, and the mobile phone and the TV transmit data through the screen mirroring protocol (the channel of the screen mirroring protocol is similar to the dedicated transmission channel of Bluetooth). After the user clicks the screen mirroring button, the mobile phone searches for devices in the local area network that can provide the screen mirroring broadcast service for applications. When the mobile phone finds the TV, the mobile phone will send a video address to the TV. After receiving the address, the TV accesses the video resources in the address for playback.

[0103] In the mirror mode, the screen display content on the current terminal device can be projected onto the screen of other terminal devices, that is, the same screen display. For example, the content such as files, desktops, game screens, software applications, etc. displayed on the current terminal device can be projected onto the screen of other terminal devices. The functions such as screen mirroring, multi-screen interaction, and mirror switch in current various mobile phones all belong to the mirror mode.

[0104] The principle of the mirror mode is to record the screen of the current terminal device and send it to other terminal devices. The recording speed can usually reach more than 60 frames per second. After other terminal devices receive the recorded screen, they display it, thus realizing the mirror mode. For example, when the mobile phone and the TV are in the same local area network, after the mobile phone finds the TV and turns on the mirror mode, the mobile phone will continuously take screenshots (i.e., record the screen). The mobile phone takes screenshots at a specific frequency and sends the screenshot images to the TV at the same time. After receiving them, the TV displays them, so that the TV can be used to view the screen of the mobile phone.

[0105] The push mode and the mirror mode can use the screen mirroring protocol for data transmission. The screen mirroring protocol includes the Digital Living Network Alliance (DLNA) protocol, the airplay push protocol, the lelink protocol, etc.

[0106] Through the push mode and the mirror mode, the screen can be extended and displayed to a certain extent. However, neither the push mode nor the mirror mode is a true screen extension, and it is difficult for users to operate the extended screen to achieve collaborative work or entertainment.

[0107] In this regard, there are some software or services that can achieve the function of an extended screen. For example, the spacedesk software and the extended screen service in Microsoft's Windows system.

[0108] However, in the spacedesk and the extended screen service in the Windows system, after the main screen device sends the display information of the extended screen to the terminal device where the extended screen is located, the terminal device where the extended screen is located needs to perform rendering and composition operations on the display information of the extended screen, making it difficult to quickly display on the screen, resulting in a phenomenon of screen refresh lag. As a result, there is a large time delay between the screen image displayed on the extended screen and the screen image displayed on the main screen.

[0109] In summary, in the current extended screen solution, when the extended screen displays the screen image, the screen will experience refresh lag, and there is a large time delay between the screen image displayed on the extended screen and the screen image displayed on the main screen.

[0110] In view of this, the embodiments of the present application provide a screen extension method. When the first terminal device uses the second display screen of the second terminal device as an extended screen, it will render and compose the graphic data of each application to obtain screen display data that can be directly displayed on the screen. Then, the first terminal device splits the screen display data into first screen display data and second screen display data, performs a screen display operation on the first screen display data, and displays the first screen display data on the first display screen. Moreover, the second screen display data is sent to the second terminal device. Since the second screen display data is data that has been rendered and composed, when the second terminal device receives the second screen display data, it can directly perform a screen display operation on the second screen display data without re-rendering and composing the second screen display data, reducing the screen refresh lag, thereby reducing the time delay of the screen image displayed on the second display screen, and solving the problem that in the current extended screen solution, when the extended screen displays the screen image, the screen will experience refresh lag, and there is a large time delay between the screen image displayed on the extended screen and the screen image displayed on the main screen.

[0111] Next, from the perspective of the first terminal device, the screen extension method provided in this embodiment will be described. Please refer to Figure 1 the flowchart of the screen extension method shown, and this method includes:

[0112] S101. The first terminal device obtains the graphic data of each application, renders the graphic data of each application to obtain the window data corresponding to each application, and synthesizes the window data corresponding to each application into screen display data;

[0113] In this embodiment, the first terminal device is the terminal device where the main display screen (i.e., the first display screen) is located, and the second terminal device is the terminal device where the extended screen (i.e., the second display screen) is located.

[0114] The first terminal device and the second terminal device can perform data transmission through wired communication and / or wireless communication. For example, the first terminal device and the second terminal device can be communicatively connected through wired communication methods such as HDMI cable connection, RJ45 interface connection, and wifi connection and / or wireless communication methods. The number of second terminal devices is one or more, and the specific number is determined according to the actual situation.

[0115] The user can set the trigger condition for enabling the extended screen mode in the first terminal device. When the trigger condition for the extended screen mode is met, the first terminal device can detect whether there is any other terminal device communicatively connected to the first terminal device. If there is no other terminal device communicatively connected to the first terminal device, the first terminal device cannot enable the extended screen mode.

[0116] When there is any other terminal device communicatively connected to the first terminal device, the first terminal device can determine whether the application corresponding to the screen image currently displayed on the first display supports the extended screen mode.

[0117] If the application corresponding to the screen image currently displayed on the first display supports the extended screen mode, the first terminal device enables the extended screen mode, selects the second terminal device from the terminal devices communicatively connected to the first terminal device, and uses the second display of the second terminal device as the extended screen.

[0118] If the application corresponding to the screen image currently displayed on the first display does not support the extended screen mode, the first terminal device prompts the user that there is an application that does not support the extended screen mode in the screen image currently displayed on the current display.

[0119] In some possible implementation manners, in the local area network where the first terminal device is located, in addition to the first terminal, there is at least one terminal device. The first terminal device monitors the screen events of each terminal device in the local area network. When a screen event that meets the preset trigger condition is detected, the terminal device corresponding to the screen event that meets the preset trigger condition is used as the second terminal device, and the display of the second terminal device is used as the second display.

[0120] The screen event can include a screen movement event and a screen touch event. The screen movement event refers to an event in which the display of the terminal device is moved. The screen touch event refers to an event in which the display of the terminal device collides.

[0121] In some embodiments, the trigger condition for enabling the extended screen mode may be that a screen movement event occurs on a certain terminal device, and the distance between the display screen of the terminal device and the first display screen is less than a preset distance threshold. Therefore, when the first terminal device detects a screen movement event, it can use the terminal device corresponding to the screen movement event as a candidate terminal device. If the distance between the display screen of the candidate terminal device and the first display screen of the first terminal device is less than the preset distance threshold, the extended screen mode can be started, using the candidate terminal device as the second terminal device and the display screen of the second terminal device as the second display screen.

[0122] For example, as Figure 2 shown, the user has terminal device 1 and terminal device 2, and the user sets the display screen 201 of terminal device 1 as the main display screen. When the user wants to extend the display of the content of display screen 1 to the display screen 202 of terminal device 2, the user can move the display screen 202 from position A to near display screen 1 (i.e., position B). When terminal device 1 detects a screen movement event of the display screen 202 of terminal device 2 and the distance between display screen 201 and display screen 202 is less than the preset distance threshold, terminal device 1 enables the extended screen mode, using terminal device 2 as the second terminal device and display screen 202 as the second display screen (i.e., the extended screen).

[0123] In some other embodiments, the trigger condition for enabling the extended screen mode may be that a screen touch event is emitted from the display screen of a certain terminal device. Therefore, when the first terminal device detects a screen touch event in the local area network, it can use the terminal device corresponding to the screen touch event as the second terminal device and the display screen of the second terminal device as the second display screen.

[0124] For example, as Figure 3 shown, the user has terminal device 3 and terminal device 4, and the user sets the display screen 301 of terminal device 3 as the main display screen. When the user wants to extend the display of the content of screen 3 to the display screen 302 of terminal device 4, the user can move the display screen 302 from position C to position D to make display screen 302 and display screen 301 touch. Terminal device 3 detects the sensor data of the motion sensor of terminal device 4 in the local area network. When terminal device 3 detects a touch event of the display screen 302 of terminal device 4 through the sensor data of the motion sensor of terminal device 4, terminal device 3 uses terminal device 4 as the second terminal device and display screen 302 as the second display screen (i.e., the extended screen).

[0125] Alternatively, the user can also tap the display screen 302 with their hand or other object, causing a screen touch event to occur on the display screen 302. When the terminal device 3 detects, via the local area network, that a touch event has occurred on the display screen 302 of the terminal device 4, the terminal device 4 is taken as the second terminal device, and the display screen 302 is taken as the second display screen (i.e., the extended screen).

[0126] It should be noted that the above-mentioned screen movement event, screen touch event, and the distance between the first terminal device and other terminal devices can be obtained by analyzing the sensor data collected by the sensors. The aforementioned sensors can be provided on the first terminal device and / or other terminal devices. The type of sensor is selected according to the actual situation. For example, an acceleration sensor can be provided on the display screen, and the screen movement event of the display screen can be detected by the acceleration sensor. The type of sensor and the location where the sensor is provided are not limited in this embodiment.

[0127] In some other possible implementation manners, an NFC tag can be provided on the first terminal device and the NFC function is enabled. When the first terminal device detects the NFC signal of another NFC tag and the signal strength of the NFC signal is greater than or equal to a preset strength threshold, the first terminal device performs a pairing operation with the terminal device corresponding to the NFC signal. After successful pairing, the first terminal device takes the terminal device corresponding to the NFC signal as the second terminal device and takes the display screen of the second terminal device as the second display screen.

[0128] For example, the user enables the NFC functions of the terminal device 5 and the terminal device 6. Then, the terminal device moves the terminal device 6 so that the NFC tag of the terminal device 6 contacts the NFC tag of the terminal device 5. At this time, the terminal device 5 detects the NFC signal of the terminal device 6 and the signal strength of the NFC signal is greater than the preset strength threshold. Therefore, the terminal device 5 sends a pairing request to the terminal device 6, and the user agrees to the pairing request on the terminal device 6, then the terminal device 5 and the terminal device 6 are successfully paired. After successful pairing, the terminal device 5 takes the terminal device 6 as the second terminal device and takes the display screen of the terminal device 6 as the second display screen (i.e., the extended screen).

[0129] In some other possible implementation manners, the trigger condition for the first terminal device to enable the extended screen mode can also be a screen extension instruction triggered by the user operating the first terminal device. When the first terminal device detects the screen extension instruction, the terminal device corresponding to the screen extension instruction is taken as the second terminal device, and the display screen of the second terminal device is taken as the second display screen.

[0130] For example, the user sets the display screen of the terminal device 7 as the main display screen and connects the terminal device 7 and the terminal device 8 with an HDMI cable. At this time, the user performs corresponding operations on the terminal device 7. For example, the user can drag the application program of the terminal device 7 outside the screen display range of the terminal device 7 to trigger a screen expansion instruction and specify the operation object of the screen expansion instruction as the terminal device 8. When the terminal device 7 detects the screen expansion instruction, it takes the terminal device 8 corresponding to the screen expansion instruction as the second terminal device and the display screen of the terminal device 8 as the second display screen.

[0131] After the first terminal device determines the second terminal device and the second display screen, it can render and synthesize the graphic data of each application program.

[0132] It can be understood that the screen image displayed on the display screen is usually composed of windows corresponding to multiple application programs. The first terminal device usually renders the graphic data corresponding to each application program separately to obtain the window data corresponding to each application program. The window data is the rendered graphic data. Then, the first terminal device synthesizes the window data corresponding to each application program into screen display data. The screen display data is the data that can be directly displayed on the screen and does not need to perform a rendering operation again during the process of being displayed on the screen.

[0133] Before the first terminal device renders the graphic data of each application program, it needs to virtualize the display resolution of each display screen as the target display resolution and perform the rendering operation according to the target display resolution.

[0134] Take Figure 4 as an example. Suppose the display resolution of the display screen 401 is 2560×1600, and the display resolution of the display screen 402 is 1280×720. Part of the window interface of the application program A is on the display screen 401, and part of the window interface is on the display screen 402.

[0135] During the process of rendering the graphic data of application A, if the display resolutions of display screen 401 and display screen 402 are not unified and virtualized to the target display resolution, the first terminal device needs to first distinguish which part of the graphic data of application A is located on display screen 401 and needs to be rendered according to the display resolution of display screen 401, and which part of the graphic data is located on display screen 402 and needs to be rendered according to the display resolution of display screen 402, so as to split the graphic data of application A into graphic data 1 and graphic data 2. Then, the first terminal device makes graphic data 1 be rendered according to the display resolution of 2560×1600 of display screen 401, and makes graphic data 2 be rendered according to the display resolution of 1280×720 of display screen 402. The rendering process is complex and requires splitting the graphic data of the application first and then rendering it according to the corresponding display resolution.

[0136] If the display resolutions of display screen 401 and display screen 402 are unified and virtualized to the target display resolution of 2560×1600, then during the rendering process, whether it is the graphic data corresponding to display screen 401 or the graphic data corresponding to display screen 402, they are all rendered according to the target display resolution of 2560×1600, and there is no need to split the graphic data of the application before rendering, which simplifies the rendering process.

[0137] The target display resolution can be set according to the actual situation. In some possible implementation manners, the target display resolution can be a value preset by the user. For example, the user can preset the target display resolution to 1280×720. Then, during the rendering process, regardless of what the first display resolution of the first display screen and the second display resolution of the second display screen are, the first terminal device will virtualize both the first display resolution of the first display screen and the second display resolution of the second display screen to 1280×720, and then perform the rendering operation.

[0138] In addition, during the process of the screen display data being displayed on the screen, if the target display resolution is inconsistent with the display resolution of the display screen, it is necessary to perform a resolution adaptation operation on the screen display data to adapt the screen display data from the target display resolution to the first display resolution or the second display resolution.

[0139] When the target display resolution is a value preset by the user, it is possible that the target display resolution is higher than the first display resolution and the second display resolution. At this time, the screen display data after resolution adaptation can be normally displayed on the first display screen and the second display screen.

[0140] However, when the target display resolution is the value preset by the user, it is also possible that the target display resolution is lower than the first display resolution or the target display resolution is lower than the second display resolution. At this time, the screen display data needs to be adapted from the lower-resolution target display resolution to the higher-resolution first display resolution or second display resolution. When the resolution-adapted screen display data is displayed on the display screen, the screen image displayed on the display screen is prone to problems such as image distortion or image tearing.

[0141] Therefore, in some other possible implementation manners, the first display resolution of the first terminal device and the second display resolution of the second terminal device can be obtained, and the maximum value of the first display resolution and the second display resolution can be used as the target display resolution. At this time, the target display resolution is greater than or equal to the display resolution of each display screen. After resolution adaptation, the screen image displayed on the display screen will not have problems such as image distortion or image tearing.

[0142] For example, assume that the first display resolution of the first display screen is 2560×1600, and the second display resolution of the second display screen is 1280×720. Since the first display resolution is greater than the second display resolution, the first display resolution is selected as the target display resolution. When the screen display data is displayed on the first display screen, the first display resolution is the same as the target display resolution, and the screen display data does not need to be resolution-adapted and can be directly displayed on the screen. When the screen display data is displayed on the second display screen, the second display resolution is lower than the target display resolution, and the screen display data is adapted from the target display resolution of 2560×1600 to the second display resolution of 1280×720. The screen display data is adapted from a higher display resolution to a lower display resolution, which can avoid problems such as image distortion or image tearing in the screen image displayed on the second display screen.

[0143] Or, assume that the first display resolution of the first display screen is 720×480, and the second display resolution of the second display screen is 1280×720. Since the second display resolution is greater than the first display resolution, the second display resolution is selected as the target display resolution. When the screen display data is displayed on the second display screen, the second display resolution is the same as the target display resolution, and the screen display data does not need to be resolution-adapted and can be directly displayed on the screen. When the screen display data is displayed on the first display screen, the first display resolution is lower than the target display resolution, and the screen display data is adapted from the target display resolution of 1280×720 to the first display resolution of 720×480. The screen display data is adapted from a higher display resolution to a lower display resolution, which can avoid problems such as image distortion or image tearing in the screen image displayed on the first display screen.

[0144] The method of resolution adaptation can be selected according to the actual situation. For example, in some embodiments, the resolution of the screen display data can be adapted from a high resolution to a low resolution by downsampling.

[0145] The above process of rendering graphic data can be executed by components with graphic rendering capabilities such as a central processing unit (CPU) and / or a graphic processing unit (GPU).

[0146] The screen image displayed on the display screen is usually composed of windows corresponding to multiple application programs. After the first terminal device finishes rendering the graphic data of each application program, window data corresponding to each application program can be obtained, and the first terminal device synthesizes the window data corresponding to each application program into screen display data.

[0147] S102. The first terminal device splits the screen display data into first screen display data and second screen display data;

[0148] After the first terminal device obtains the screen display data, it can split the screen display data into first screen display data and second screen display data. The first screen display data is the data to be displayed on the first display screen, and the second screen display data is the data to be displayed on the second display screen. The number of copies of the second screen display data is greater than or equal to 1, and the number of copies of the second screen display data is the same as the number of second display screens. One second display screen corresponds to one copy of the second screen display data.

[0149] Since the display resolution of each display screen is virtualized to the target display resolution during the process of rendering graphic data, after rendering, the sizes of the screen display data corresponding to the screen images of each display screen are equal. When splitting the screen display data, the total number of the first display screen and the second display screens can be obtained, and then the screen display data can be equally split according to the aforementioned total number, so as to obtain the first screen display data corresponding to the first display screen and the second screen display data corresponding to the second display screen.

[0150] For example, as Figure 5As shown in the figure, assume that there is 1 first display screen (display screen 501) and 2 second display screens (display screen 502 and display screen 503). Then the total number of the first display screen and the second display screens is 3. The screen display data is stored in Arrays 1 to 9. When the first terminal device splits the screen display data, it can split the screen display data into three equal parts, divide Arrays 1 to 3 into the first screen display data corresponding to display screen 501, divide Arrays 4 to 6 into the second screen display data corresponding to display screen 502, and divide Arrays 7 to 9 into the second screen display data corresponding to display screen 503. Thus, 1 portion of the first screen display data and 2 portions of the second screen display data are obtained.

[0151] When splitting the first screen display data and the second screen display data, the application program information related to the first screen display data and the application program information related to the second screen display data can also be marked. When the first terminal device needs to know which application program windows are included in the first screen display data and which application program windows are included in the second screen display data, it can directly call the application program information corresponding to the first screen display data and the application program information corresponding to the second screen display data.

[0152] S103. The first terminal device displays the first screen display data on the first display screen of the first terminal device, and sends the second screen display data to a second terminal device to instruct the second terminal device to display the second screen display data on the second display screen of the second terminal device.

[0153] After the first terminal device splits the screen display data into the first screen display data and the second screen display data, it can perform an on-screen operation on the first screen display data and display the first screen display data on the first display screen.

[0154] Moreover, the first terminal device can also send the second screen display data to the corresponding second terminal device. After receiving the second screen display data, the second terminal device does not need to perform the processing procedures of re-rendering and compositing, and can directly perform an on-screen operation on the second screen display data and display the second screen display data on the second display screen.

[0155] The following describes the screen extension method provided in this embodiment in combination with a specific application program scenario:

[0156] As Figure 6 shown, there are multiple terminal devices in the local area network, such as terminal device 601, terminal device 602, and terminal device 603. Each terminal device in the local area network is connected to each other by means of wired communication and / or wireless communication.

[0157] The operating system of each terminal device is the Android operating system. Please refer to Figure 7 , Figure 7 Figure Figure 7 is the system architecture diagram of the Android operating system of terminal device 601. The system architecture of the Android operating system can include the software (Application, APP) layer, the framework layer, the native layer, the kernel layer, and the hardware layer.

[0158] At least one application 701 is set in the APP layer. The application 701 can be a system application or a third-party application.

[0159] In the Framework layer, there are a Window Manager Service (WMS) module 702, an Activity Manager Service (AMS) module 703, a Display Manager Service (DMS) module 704, a Multi-Display Policy Service module 705, and an Input Manager module 706.

[0160] The WMS module 702 is responsible for allocating windows (surfaces) for applications and managing the attributes of the surfaces (such as the size, display position, display area, and hidden area of the surface).

[0161] The AMS module 703 is responsible for managing the life cycle and handling matters of applications.

[0162] The DMS module 704 is responsible for managing the currently available display screens of the system. In the non-expanded screen mode, the DMS module 704 registers the display screen of the current terminal device as an available display screen; in the expanded screen mode, the DMS module 704 registers the display screen of the current terminal device and the expanded screen as available display screens.

[0163] The Multi-Display Policy Service module 705 is responsible for making a decision on whether to enable the expanded screen mode according to the preset expanded screen trigger conditions.

[0164] The Input Manager module 706 is responsible for transmitting data.

[0165] In the Native layer, there are a graphics compositor (SurfaceFlinger) module 707, a hardware graphics composer (HWC) module 709, an OnFrameAvailable module 708, and a display channel (Display) 710.

[0166] The SurfaceFlinger module 707 is responsible for synthesizing the window data rendered by each application into screen display data.

[0167] The HWC module 709 is responsible for displaying the screen display data on the screen.

[0168] The OnFrameAvailable module 708 is responsible for splitting the screen display data.

[0169] The Display 710 is responsible for transmitting the split screen display data.

[0170] In the Kernel layer, there are a digital rights management (DRM) module 711 and a sensor driver (Sensor Driver) module 712.

[0171] The DRM module 711 is responsible for performing copyright verification on the content displayed on the terminal device.

[0172] The Sensor Driver module 712 is responsible for receiving and transmitting the sensor data collected by the sensor.

[0173] In the Hardware layer, there is a display screen (Screen) 713 and a local area network service (Local Area Network, LAN) module 714.

[0174] The Screen 713 is responsible for displaying the screen image.

[0175] The LAN module 714 is responsible for creating and maintaining a local area network connection between the terminal device and other terminal devices, and this local area network connection can be a wired communication connection and / or a wireless communication connection.

[0176] The user can specify the terminal device 601 as the first terminal device, set the display screen 713 of the terminal device 601 as the first display screen, and set the trigger condition for the terminal device 601 to enable the extended screen mode as a screen touch event occurring on the display screen of a certain terminal device within the local area network.

[0177] At this time, the terminal device 601 receives the sensor data sent by other terminal devices within the local area network through the LAN module 714 and monitors the screen touch events occurring on other terminal devices within the local area network.

[0178] When the user wants to use the display screen of the terminal device 602 as an extended screen of the terminal device 601, the user can move the display screen of the terminal device 602 and touch the display screen 713 of the terminal device 601 with the display screen of the terminal device 602.

[0179] The sensors on the display screen of the terminal device 602 collect corresponding sensor data, and the terminal device 602 transmits the above sensor data to the terminal device 601 through the local area network.

[0180] The LAN module 714 of the terminal device 601 passes the received sensor data to the Sensor Driver module 712, and the Sensor Driver module 712 passes the sensor data to the Multi-DisplayPolicy Service module 705 through the Input Manager module 706.

[0181] The Multi-Display Policy Service module 705 detects a screen touch event of the terminal device 602 according to the sensor data, enables the extended screen mode of the terminal device 601, and uses the display screen of the terminal device 602 as the extended screen object.

[0182] After the terminal device 601 enables the extended screen mode, the DMS module 704 performs a redirection operation to register the display screen of the terminal device 602 as an available display screen.

[0183] In the graphics pipeline rendering framework of the Android operating system, graphics rendering components with graphics rendering functions such as the GPU are responsible for rendering the graphics data of each application. During the rendering process, the graphics rendering components obtain the display resolution of the display screen of the terminal device 601 and the display resolution of the display screen of the terminal device 602, select the maximum value as the target display resolution, and render the graphics data of each application according to the target display resolution.

[0184] After the graphics data of each application is rendered, the window data of each application is obtained, and the graphics rendering components pass the rendered window data of each application into the buffer queue.

[0185] The SurfaceFlinger module 707 retrieves the rendered window data of each application from the data buffer queue.

[0186] Among them, the buffer queue adopts the producer-consumer mode and can adjust the fixed cycle of the buffer from the production queue to the consumption queue. Its working principle is as Figure 8 shown.

[0187] When the producer needs to cache data, the producer can create a bufferqueue cache queue through a dequeue instruction, or the producer can also wake up the buffer queue in the dequeue state.

[0188] Before the buffer queue is successfully created or woken up, the buffers in the buffer queue are in the Free state; when the buffer queue is successfully created or woken up, the buffer queue passes the buffer to the producer, and at this time the state of the buffer becomes the dequeue state.

[0189] The producer stores the window data rendered by each application in the buffer and passes the buffer to the buffer queue through a queue instruction, and the state of the buffer becomes the queue state.

[0190] When the consumer needs to process the data cached in the buffer, the consumer can send an acquire instruction to the buffer queue.

[0191] After the buffer queue receives the acquire instruction, it passes the buffer to the consumer, and at this time the state of the buffer becomes the acquired state.

[0192] When the consumer finishes processing the data in the buffer, it clears the data cached in the buffer and releases the buffer back to the buffer queue. At this time, the state of the buffer becomes the Free state, and the buffer queue completes the caching and calling of this data.

[0193] As described above, in the producer-consumer model adopted by the buffer queue of the terminal device 601, the producer is a component with graphics rendering capabilities such as the GPU. After the producer finishes rendering the graphic data of each application, it obtains the window data of each application and passes the window data of each application after rendering into the buffer of the buffer queue, waiting to be called by the consumer. The consumer is a component such as the SurfaceFlinger module 707 that needs to operate on the data cached in the buffer. When the producer passes the window data of each application after rendering into the buffer, the SurfaceFlinger module 707 calls the buffer of each buffer queue to obtain the window data of each application and performs a composition operation on the window data of each application.

[0194] In the SurfaceFlinger module 707, the window data of each application exists in the form of a layer. The SurfaceFlinger module 707 fuses the multi-layer window data into one layer of screen display data according to the surface attributes corresponding to each layer of window data.

[0195] After that, the SurfaceFlinger module 707 passes the screen display data to the OnFrameAvailable module 708 in the form of a buffer. The screen display data exists in the form of an array in the buffer.

[0196] The total number of screens of the terminal device 601 and the terminal device 602 is 2. Therefore, the OnFrameAvailable module 708 divides the screen display data in the buffer into two equal parts, takes the first half of the screen display data in the buffer as the first screen display data, and passes it to the Display 710, and takes the second half of the screen display data in the buffer as the second screen display data, and passes it to the terminal device 602 through the LAN module 714.

[0197] For example, the screen display data exists in the form of an array in the buffer. Assuming that there are 8 data blocks in the array, numbered 000 to 007 in sequence, the OnFrameAvailable module 708 divides the array in the buffer into two equal parts, takes the data blocks numbered 000 to 003 as the first screen display data, and passes it to the Display 710, and takes the data blocks numbered 004 to 007 as the second screen display data, and passes it to the terminal device 602 through the LAN module 714.

[0198] The Display 710 transfers the first screen display data to the DRM module 711 for copyright verification. After the copyright verification, the DRM module 711 transfers the first screen display data to the HWC module 709, and the HWC module 709 is responsible for adapting the resolution of the first screen display data and displaying the first screen display data with the adapted resolution on the display screen 713 of the terminal 601.

[0199] Please refer to Figure 9 , Figure 9 which is the system architecture diagram of the Android operating system of the terminal device 602. The functions of each module in the Android operating system of the terminal device 602 are the same as those of the corresponding modules in the Android operating system of the terminal device 601, and will not be repeated here.

[0200] After receiving the second screen display data sent by the terminal device 601 through the LAN module 901, the terminal device 602 stores the second screen display data in the buffer of the buffer queue of the terminal device 602, and the buffer queue is responsible for caching and transferring the second screen display data.

[0201] Please refer to Figure 10 , the buffer queue of the terminal device 602 also adopts the producer-consumer model. At this time, in the producer-consumer model of the buffer queue of the terminal device 602, the producer is the LAN module 901, and the consumers are components such as the Display module 902, the DRM module 903, and the HWC module 904 that need to process the second screen display data.

[0202] The producer creates the buffer queue through the dequeue instruction or wakes up the buffer queue in the dequeue state. After successfully creating or waking up the buffer queue, the buffer queue transfers the buffer to the producer, and at this time the state of the buffer becomes the dequeue state.

[0203] The producer stores the second screen display data in the buffer and transfers the buffer to the buffer queue through the enqueue instruction, and the state of the buffer becomes the enqueue state.

[0204] When the consumer needs to process the data in the buffer, the consumer can send an acquire instruction to the buffer queue.

[0205] The buffer queue passes the buffer to the consumer, and at this time, the state of the buffer becomes the acquired state.

[0206] Combined with Figure 9 and Figure 10 When the LAN module stores the second screen display data in the buffer, it passes the buffer to the buffer queue. The Display module 902 obtains the second screen display data from the buffer queue, and the Display module 902 passes the second screen display data to the DRM module 903 for copyright verification. After the copyright verification, the DRM module 903 passes the second screen display data to the HWC module 904. The HWC module 904 performs resolution adaptation on the second screen display data and displays the resolution-adapted second screen display data on the display screen 905 of the terminal device 602.

[0207] It can be understood that in the existing solution, usually after determining the screen content displayed on each display screen during the virtual display process, the main screen device sends the unrendered graphic data corresponding to the extended screen to the terminal device where the extended screen is located. After receiving the unrendered graphic data, the extended screen device needs to perform a rendering operation and a composition operation, and then display the composed screen display data on the screen. Therefore, when the extended screen displays the screen content, it is easy to have a screen refresh lag phenomenon. Compared with the screen picture displayed on the main screen, there is a large time delay in the screen picture displayed on the extended screen.

[0208] In the screen extension method of this embodiment, the first terminal device first performs a rendering operation and a composition operation on the graphic data of each application program to obtain screen display data. Then, the first terminal device divides the screen display data to obtain the first screen display data and the second screen display data, displays the first screen display data on the first display screen, and sends the second screen display data to the second terminal device. The second screen display data is the data after rendering and composition. That is to say, after the second terminal device receives the second screen display data, it can directly perform the screen display operation and display the second screen display data on the second display screen. There is no need to perform the rendering and composition operations on the second screen display data again during the screen display process, thereby improving the data screen display speed, reducing the occurrence of the screen refresh lag phenomenon to a certain extent, reducing the time delay between the screen picture displayed on the extended screen and the screen picture displayed on the main screen, and solving the problem that in the current extended screen solution, when the extended screen displays the screen picture, the screen will have a refresh lag situation, and there is a large time delay between the screen picture displayed on the extended screen and the screen picture displayed on the main screen.

[0209] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0210] Please refer to Figure 11 , an embodiment of the present application provides a screen expansion device. For the convenience of description, only the parts related to the present application are shown, such as Figure 11 shown, the screen expansion device includes

[0211] A rendering and composition module 1101, configured to obtain the graphic data of each application program, render the graphic data of each application program, and compose the rendered graphic data corresponding to each application program into screen display data;

[0212] A data splitting module 1102, configured to split the screen display data into first screen display data and second screen display data;

[0213] A screen expansion module 1103, configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to a second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

[0214] Optionally, the device further includes:

[0215] An event monitoring module, configured to determine the terminal device corresponding to the screen event as the second terminal device when detecting a screen event that meets a preset trigger condition.

[0216] Optionally, the event monitoring module includes:

[0217] A movement candidate sub-module, configured to determine the terminal device corresponding to the screen movement event as a candidate terminal device when detecting a screen movement event;

[0218] A distance determination sub-module, configured to determine the candidate terminal device as the second terminal device if the distance between the display screen of the candidate terminal device and the display screen of the first terminal device is less than a preset distance threshold.

[0219] Optionally, the event monitoring module includes:

[0220] A collision determination sub-module, configured to determine the terminal device corresponding to the screen touch event as the second terminal device when detecting a screen touch event.

[0221] Optionally, the device further includes:

[0222] A near - field communication module, configured to determine the terminal device corresponding to the NFC signal as a second terminal device when the first terminal device detects an NFC signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold.

[0223] Optionally, the apparatus further includes:

[0224] An instruction determination module, configured to determine the terminal device indicated in the screen expansion instruction as the second terminal device when the screen expansion instruction is obtained.

[0225] Optionally, the rendering and composition module 1101 includes:

[0226] A resolution sub - module, configured to obtain a target display resolution;

[0227] A graphics rendering sub - module, configured to render the graphic data of each application program according to the target display resolution.

[0228] Optionally, the resolution sub - module is specifically configured to obtain a first display resolution corresponding to the first terminal device and a second display resolution corresponding to the second terminal device, and determine the maximum value of the first display resolution and the second display resolution as the target display resolution.

[0229] Optionally, the data splitting module 1102 is specifically configured to obtain the total number of the display screens of the first terminal device and the second terminal device, and equally split the screen display data according to the total number to obtain first screen display data and second screen display data.

[0230] Optionally, the screen expansion module 1103 is specifically configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to the second terminal device to instruct the second terminal device not to render the second screen display data and display the second screen display data on the display screen of the second terminal device.

[0231] It should be noted that, for the information interaction, execution process, etc. between the above - mentioned device / unit, since it is based on the same concept as the method embodiment of the present application, for its specific functions and the technical effects brought, please refer to the method embodiment part for details, and will not be elaborated here.

[0232] Please refer to Figure 12 , this embodiment of the present application further provides a terminal device. As Figure 12As shown, the terminal device 12 of this embodiment includes: a processor 120, a memory 121, and a computer program 122 stored in the memory 121 and executable on the processor 120. When the processor 120 executes the computer program 122, the steps in the above-described embodiment of the screen expansion method are implemented, such as Figure 1 the steps S101 to S103 shown. Alternatively, when the processor 120 executes the computer program 122, the functions of each module / unit in the above-described device embodiments are implemented, such as Figure 11 the functions of the modules 1101 to 1103 shown.

[0233] Exemplarily, the computer program 122 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 121 and executed by the processor 120 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 122 in the terminal device 12. For example, the computer program 122 can be divided into a rendering synthesis module, a data splitting module, and a screen expansion module. The specific functions of each module are as follows:

[0234] The rendering synthesis module is configured to obtain the graphic data of each application program, render the graphic data of each application program, and synthesize the rendered graphic data corresponding to each application program into screen display data;

[0235] The data splitting module is configured to split the screen display data into first screen display data and second screen display data;

[0236] The screen expansion module is configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to a second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

[0237] The terminal device 12 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor 120 and a memory 121. Those skilled in the art can understand that Figure 12 merely examples of the terminal device 12, which do not constitute a limitation on the terminal device 12, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the terminal device may further include input / output devices, network access devices, a bus, etc.

[0238] The so-called processor 120 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0239] The memory 121 may be an internal storage unit of the terminal device 12, such as the hard disk or memory of the terminal device 12. The memory 121 may also be an external storage device of the terminal device 12, such as a plug-in hard disk equipped on the terminal device 12, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 121 may also include both the internal storage unit of the terminal device 12 and the external storage device. The memory 121 is used to store the computer program and other programs and data required by the terminal device. The memory 121 may also be used to temporarily store the data that has been output or will be output.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual application programs, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0241] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0242] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0243] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0244] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0246] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0247] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A screen extension method, characterized in that Including: The first terminal device obtains the target display resolution; The first terminal device obtains the graphic data of each application program, renders the graphic data of each application program according to the target display resolution, and synthesizes the rendered graphic data corresponding to each application program into screen display data, and each application program supports the extended screen mode; The first terminal device splits the screen display data into first screen display data and second screen display data; The first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

2. The screen expansion method according to claim 1, wherein Before the first terminal device obtains the graphic data of each application program, it further includes: When the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device determines the terminal device corresponding to the screen event as the second terminal device.

3. The screen expansion method according to claim 2, wherein The step that when the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device determines the terminal device corresponding to the screen event as the second terminal device includes: When the first terminal device monitors a screen movement event, the first terminal device determines the terminal device corresponding to the screen movement event as a candidate terminal device; If the distance between the display screen of the candidate terminal device and the display screen of the first terminal device is less than a preset distance threshold, the first terminal device determines the candidate terminal device as the second terminal device.

4. The screen expansion method according to claim 2, characterized in that, The step that when the first terminal device monitors a screen event that meets a preset trigger condition, the first terminal device determines the terminal device corresponding to the screen event as the second terminal device includes: When the first terminal device monitors a screen touch event, the first terminal device determines the terminal device corresponding to the screen touch event as the second terminal device.

5. The screen expansion method according to claim 1, characterized in that Before the first terminal device obtains the graphic data of each application program, it further includes: When the first terminal device detects a near field communication (NFC) signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold, the first terminal device determines the terminal device corresponding to the NFC signal as the second terminal device.

6. The screen expansion method according to claim 1, wherein Before the first terminal device obtains the graphic data of each application program, it further includes: When the first terminal device obtains a screen extension instruction, the first terminal device determines the terminal device indicated in the screen extension instruction as the second terminal device.

7. The screen expansion method according to claim 1, wherein, The first terminal device obtains the target display resolution, including: The first terminal device obtains the first display resolution corresponding to the first terminal device and the second display resolution corresponding to the second terminal device, and determines the maximum value of the first display resolution and the second display resolution as the target display resolution.

8. The screen expansion method according to claim 1, wherein The first terminal device splits the screen display data into first screen display data and second screen display data, including: The first terminal device obtains the total number of the display screens of the first terminal device and the second terminal device, and equally divides and splits the screen display data according to the total number, so as to obtain first screen display data and second screen display data.

9. The screen expansion method according to claim 1, wherein The first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device, so as to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device, including: The first terminal device displays the first screen display data on the display screen of the first terminal device, and sends the second screen display data to the second terminal device, so as to instruct the second terminal device not to render the second screen display data and display the second screen display data on the display screen of the second terminal device.

10. A screen expansion device, characterized in that, Including: A rendering and synthesizing module, configured to obtain a target display resolution; Obtain the graphic data of each application program, render the graphic data of each application program according to the target display resolution, and synthesize the rendered graphic data corresponding to each application program into screen display data, and each application program supports an extended screen mode; A data splitting module, configured to split the screen display data into first screen display data and second screen display data; A screen extension module, configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to the second terminal device, so as to instruct the second terminal device to display the second screen display data on the display screen of the second terminal device.

11. The screen expansion device according to claim 10, characterized in that, The device further includes: An event monitoring module, configured to determine the terminal device corresponding to the screen event as the second terminal device when detecting a screen event that meets a preset trigger condition.

12. The screen expansion device according to claim 11, wherein, The event monitoring module includes: A movement candidate sub-module, configured to determine the terminal device corresponding to the screen movement event as a candidate terminal device when detecting a screen movement event; A distance determination sub-module, configured to determine the candidate terminal device as the second terminal device if the distance between the display screen of the candidate terminal device and the display screen of the first terminal device is less than a preset distance threshold.

13. The screen expansion device according to claim 11, wherein, The event monitoring module includes: A collision determination sub-module, configured to determine the terminal device corresponding to the screen touch event as the second terminal device when detecting a screen touch event.

14. The screen expansion device according to claim 12, characterized in that The device further includes: A near-field communication module, configured to determine the terminal device corresponding to the NFC signal as the second terminal device when the first terminal device detects an NFC signal and the signal strength of the NFC signal is greater than or equal to a preset strength threshold.

15. The screen expansion device according to claim 12, characterized in that, The device further includes: An instruction determination module, configured to determine the terminal device indicated in the screen extension instruction as the second terminal device when obtaining a screen extension instruction.

16. The screen expansion device according to claim 10, wherein The rendering and composition module is specifically configured to obtain the first display resolution corresponding to the first terminal device and the second display resolution corresponding to the second terminal device, and determine the maximum value of the first display resolution and the second display resolution as the target display resolution.

17. The screen expansion device according to claim 10, characterized in that, The data splitting module is specifically configured to obtain the total number of the display screens of the first terminal device and the second terminal device, and equally split the screen display data according to the total number to obtain first screen display data and second screen display data.

18. The screen expansion device according to claim 10, wherein The screen extension module is specifically configured to display the first screen display data on the display screen of the first terminal device, and send the second screen display data to the second terminal device to instruct the second terminal device not to render the second screen display data and display the second screen display data on the display screen of the second terminal device.

19. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the terminal device implements the steps of the method according to any one of claims 1 to 9.

20. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the terminal device implements the steps of the method according to any one of claims 1 to 9.

Images