A display control method, a chip, an electronic device, and a readable storage medium

Abstract

This application provides a display control method, relating to the field of terminals. A screen image displayed on a second electronic device is projected onto a first electronic device. The first electronic device responds to a first operation by a stylus pen, sending first information associated with the first operation to the second electronic device. When the second electronic device receives the first information, it can acquire an image frame corresponding to the first information and generate a first image based on the image frame and its previous image frame. The second electronic device then transmits the first image to the first electronic device for display. This application also provides an electronic device, a chip, and a computer-readable storage medium. This application can effectively reduce the latency of using a stylus pen to control a second electronic device on a first electronic device, improving the user experience.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to a display control method, a chip, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of consumer electronics, tablet computers can now be used as a second monitor for a personal computer (PC). Content displayed on the PC can be projected onto the tablet. When the content from the PC is displayed on the tablet, the user can control the tablet using the PC's mouse and keyboard. In some scenarios, such as drawing and writing, the drawing interface of drawing software displayed on the PC can be shown on the tablet. The user can then use a stylus to draw on the tablet. The user's drawing is loaded into the drawing software, which updates its interface in response. Both the tablet and PC sides can display the updated drawing interface, thus allowing the user to use a stylus on the tablet instead of the keyboard / mouse on the PC for drawing, resulting in a better user experience.

[0003] Compared to a mouse and keyboard, users are more likely to perceive latency when using a stylus for writing / drawing. For example, drawing a horizontal line on a tablet with a stylus may take a considerable amount of time for the tablet to display the line. Existing tablet-PC collaborative control solutions (controlling the tablet from the PC side or the PC from the tablet side) generally have long latency, resulting in a noticeable delay for users and a poor experience in drawing and writing scenarios. Summary of the Invention

[0004] In view of this, it is necessary to provide a display control method, chip, electronic device and computer-readable storage medium that can reduce the latency of reverse control and improve the user experience.

[0005] The first aspect of this application discloses a display control method applied to a first electronic device and a second electronic device. The screen image displayed by the second electronic device is projected onto the first electronic device for display. The display control method includes: the first electronic device responding to a first operation of a stylus pen by sending first information associated with the first operation to the second electronic device; when the second electronic device receives the first information, the second electronic device acquires an image frame corresponding to the first information; the second electronic device generates a first image based on the image frame and the previous image frame of the image frame; and the second electronic device transmits the first image to the first electronic device for display.

[0006] By adopting this technical solution, the second electronic device can generate a first image based on image change information and transmit it to the first electronic device for display. This can reduce the latency of the user controlling the second electronic device with a stylus on the first electronic device and improve the user experience.

[0007] In one possible implementation, the first information includes operation location information and image frame information associated with the first operation.

[0008] In one possible implementation, generating a first image based on an image frame and the previous image frame includes: a second electronic device acquiring image change information of the image frame relative to the previous image frame; the second electronic device obtaining an image change area based on the image change information; and the second electronic device encoding the image change area to obtain the first image.

[0009] By adopting this technical solution, the second electronic device can determine the image change area based on the acquired image change information, and then encode the image change area to obtain the first image, thereby realizing local refresh of the image frame and reducing the latency of reverse control by the user using a stylus.

[0010] In one possible implementation, encoding the image change area to obtain a first image includes: a second electronic device calculating the area of ​​the image change area; if the area of ​​the image change area is less than or equal to a first preset area, the second electronic device encoding the image change area to obtain the first image.

[0011] By adopting this technical solution, it is possible to avoid processing the entire screen image when the image change area is small, and only the change area is captured and encoded, thereby reducing the latency of reverse control by the user using a stylus and improving the user experience.

[0012] In one possible implementation, calculating the area of ​​the image change region includes: a second electronic device obtaining at least one image difference based on image change information; the second electronic device defining the local region where each image difference is located as an image change sub-region, wherein the local region is a box region that can at least contain the image difference; the second electronic device obtaining the corner coordinates of each image change sub-region and calculating the area of ​​each image change sub-region based on the corner coordinates; and the second electronic device summing the areas of each image change sub-region to obtain the area of ​​the image change region.

[0013] By adopting this technical solution, the second electronic device defines the smallest box region that can contain the image difference as the image change sub-region, and calculates the area of ​​the image change region by summing the areas of each image change sub-region.

[0014] In one possible implementation, the display control method further includes: if the area of ​​the image change region is greater than a first preset area, the second electronic device performs a merging operation on multiple image change sub-regions; the second electronic device recalculates the area of ​​the image change region based on the merged image change sub-regions; if the recalculated area of ​​the image change region is less than or equal to the first preset area, the second electronic device encodes the image change region to obtain a first image.

[0015] By adopting this technical solution, when the area of ​​the image change region is greater than the first preset area, the second electronic device attempts to perform a merging operation on each image change sub-region, and recalculates the area of ​​the image change region based on the merged image change sub-region, in order to determine whether the image encoding can be performed using the local refresh method.

[0016] In one possible implementation, a merging operation is performed on multiple image change sub-regions, including: a second electronic device binarizes the multiple image change sub-regions to obtain image grayscale values ​​corresponding to the multiple image change sub-regions; if the difference between the image grayscale values ​​of two image change sub-regions is less than a preset difference, the second electronic device ignores the area of ​​one of the two image change sub-regions.

[0017] By adopting this technical solution, the second electronic device performs binarization processing on multiple image change sub-regions. If the difference between the image grayscale values ​​of two image change sub-regions is less than a preset difference, the area of ​​any one of the two image change sub-regions is ignored and the area of ​​the image change region is recalculated to determine whether the image encoding can be performed using a local refresh method.

[0018] In one possible implementation, performing a merging operation on multiple image change sub-regions includes: a second electronic device selecting two image change sub-regions from the multiple image change sub-regions; the second electronic device calculating the area of ​​the graphic region of the two selected image change sub-regions using a minimum bounding box; if the area of ​​the graphic region is less than or equal to a second preset area, the second electronic device defining the graphic region as the region obtained by performing a merging operation on the two selected image change sub-regions.

[0019] By adopting this technical solution, if the graphic regions of the two image change sub-regions selected by the minimum energy frame are less than or equal to the second preset area, the second electronic device defines the graphic region as the region obtained by performing a merging operation on the two selected image change sub-regions, so as to recalculate the area of ​​the image change region and determine whether the image encoding can be performed using the local refresh method.

[0020] In one possible implementation, the first image includes multiple sub-images corresponding to multiple image change sub-regions, and timestamps, position information, and size information of each sub-image. Transmitting the first image to a first electronic device for display includes: when the first electronic device receives the first image, it adds a layer corresponding to the first image; the first electronic device adds multiple sub-images to the layer based on the position information and size information; and the first electronic device merges and displays multiple layers based on the timestamp order of the layers, wherein the timestamp of the layer corresponds to the timestamp of the sub-image added to the layer.

[0021] By adopting this technical solution, the first electronic device can continuously merge and display the received first image, so that the image display effect changes with the user's stylus operation.

[0022] In one possible implementation, the display control method further includes: the first electronic device comparing the timestamps of multiple layers with the timestamp of the layer currently displayed by the first electronic device, and filtering out layers whose timestamps are later than the timestamp of the currently displayed layer.

[0023] By adopting this technical solution, it is possible to avoid the first electronic device displaying a layer whose timestamp is later than the timestamp of the currently displayed layer, thus achieving the display of layers in the order they appear.

[0024] In one possible implementation, generating a first image based on first information includes: when a second electronic device receives the first information, the second electronic device simulates performing a first operation based on the first information to generate the first image.

[0025] By adopting this technical solution, for a specific writing scenario, the second electronic device can directly simulate and execute the first operation based on the first information to generate the first image, which can further reduce the reverse control latency.

[0026] In one possible implementation, the display control method further includes: if the area of ​​the recalculated image change area is greater than a first preset area, the second electronic device performs a global refresh of the screen image based on the first information.

[0027] By adopting this technical solution, if the image change area is large, and the local refresh efficiency is lower than the global refresh efficiency, the screen image can be refreshed globally.

[0028] The second aspect of this application discloses a display control method applied to a second electronic device, wherein the screen image displayed by the second electronic device is projected onto a first electronic device for display. The display control method includes: receiving first information sent by the first electronic device, wherein the first information is associated with a first operation of a stylus; acquiring an image frame corresponding to the first information; generating the first image based on the image frame and the previous image frame of the image frame; and transmitting the first image to the first electronic device for display.

[0029] By adopting this technical solution, the second electronic device can generate a first image based on image change information and transmit it to the first electronic device for display. This can reduce the latency of the user controlling the second electronic device with a stylus on the first electronic device and improve the user experience.

[0030] In one possible implementation, the first information includes operation position information associated with the first operation and image frame information. Generating a first image based on the image frame and the previous image frame includes: obtaining image change information of the image frame relative to the previous image frame; obtaining an image change area based on the image change information; and encoding the image change area to obtain the first image.

[0031] By adopting this technical solution, the second electronic device can determine the image change area based on the acquired image change information, and then encode the image change area to obtain the first image, thereby realizing local refresh of the image frame and reducing the latency of reverse control by the user using a stylus.

[0032] In one possible implementation, encoding the image change region to obtain a first image includes: calculating the area of ​​the image change region; if the area of ​​the image change region is less than or equal to a first preset area, encoding the image change region to obtain the first image.

[0033] By adopting this technical solution, it is possible to avoid processing the entire screen image when the image change area is small, and only the change area is captured and encoded, thereby reducing the latency of reverse control by the user using a stylus and improving the user experience.

[0034] In one possible implementation, calculating the area of ​​the image change region includes: obtaining at least one image difference based on image change information; defining the local region where each image difference is located as an image change sub-region, wherein the local region is a box region that can contain the image difference at the smallest size; obtaining the corner coordinates of each image change sub-region, and calculating the area of ​​each image change sub-region based on the corner coordinates; and summing the areas of each image change sub-region to obtain the area of ​​the image change region.

[0035] By adopting this technical solution, the second electronic device defines the smallest box region that can contain the image difference as the image change sub-region, and calculates the area of ​​the image change region by summing the areas of each image change sub-region.

[0036] In one possible implementation, the display control method further includes: if the area of ​​the image change region is greater than a first preset area, performing a merging operation on multiple image change sub-regions; recalculating the area of ​​the image change region based on the merged image change sub-regions; if the recalculated area of ​​the image change region is less than or equal to the first preset area, encoding the image change region to obtain a first image.

[0037] By adopting this technical solution, when the area of ​​the image change region is greater than the first preset area, the second electronic device attempts to perform a merging operation on each image change sub-region, and recalculates the area of ​​the image change region based on the merged image change sub-region, in order to determine whether the image encoding can be performed using the local refresh method.

[0038] In one possible implementation, a merging operation is performed on multiple image change sub-regions, including: binarizing the multiple image change sub-regions to obtain image grayscale values ​​corresponding to the multiple image change sub-regions; if the difference between the image grayscale values ​​of two image change sub-regions is less than a preset difference, the area of ​​one of the two image change sub-regions is ignored.

[0039] By adopting this technical solution, the second electronic device performs binarization processing on multiple image change sub-regions. If the difference between the image grayscale values ​​of two image change sub-regions is less than a preset difference, the area of ​​any one of the two image change sub-regions is ignored and the area of ​​the image change region is recalculated to determine whether the image encoding can be performed using a local refresh method.

[0040] In one possible implementation, a merging operation is performed on multiple image change sub-regions, including: selecting two image change sub-regions from the multiple image change sub-regions; calculating the area of ​​the graphic region of the two selected image change sub-regions using a minimum bounding box; and defining the graphic region as the region obtained by performing the merging operation on the two selected image change sub-regions if the area of ​​the graphic region is less than or equal to a second preset area.

[0041] By adopting this technical solution, if the graphic regions of the two image change sub-regions selected by the minimum energy frame are less than or equal to the second preset area, the second electronic device defines the graphic region as the region obtained by performing a merging operation on the two selected image change sub-regions, so as to recalculate the area of ​​the image change region and determine whether the image encoding can be performed using the local refresh method.

[0042] Thirdly, embodiments of this application provide a display control method applied to a first electronic device, wherein a screen image displayed by a second electronic device is projected onto the first electronic device for display. The display control method includes: receiving a first image sent by the second electronic device; adding a layer corresponding to the first image, wherein the first image includes multiple sub-images corresponding to multiple image change sub-regions and timestamps, position information, and size information of each sub-image; adding multiple sub-images to the layer based on the position information and size information of each sub-image; and merging and displaying multiple layers based on the timestamp order of the layers, wherein the timestamps of the layers correspond to the timestamps of the sub-images added to the layers.

[0043] By adopting this technical solution, the first electronic device can continuously merge and display the received first image, so that the image display effect changes with the user's stylus operation.

[0044] In one possible implementation, the display control method further includes: comparing the timestamps of multiple layers with the timestamp of the currently displayed layer, and filtering out layers whose timestamps are later than the timestamp of the currently displayed layer.

[0045] By adopting this technical solution, it is possible to avoid the first electronic device displaying a layer whose timestamp is later than the timestamp of the currently displayed layer, thus achieving the display of layers in the order they appear.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the display control method as described in the second or third aspect.

[0047] Fifthly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the display control method as described in the second or third aspect.

[0048] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to execute the display control method as described in the second or third aspect.

[0049] A seventh aspect provides an apparatus having the function of implementing the electronic device behavior described in the methods provided in the second or third aspects above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functions.

[0050] It is understood that the computer-readable storage medium described in the fourth aspect, the electronic device described in the fifth aspect, the computer program product described in the sixth aspect, and the device described in the seventh aspect all correspond to the methods described in the second or third aspects above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0051] Figure 1 A schematic diagram illustrating a multi-device collaborative interaction scenario provided in an embodiment of this application;

[0052] Figure 2 A schematic diagram illustrating the interaction between a first electronic device and a second electronic device according to an embodiment of this application;

[0053] Figure 3 A schematic flowchart illustrating a display control method provided in an embodiment of this application;

[0054] Figure 4 for Figure 3 A detailed flowchart illustrating the one-step process of the display control method;

[0055] Figure 5 A schematic diagram illustrating the merging process of multiple image change sub-regions provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram illustrating the merging and mapping of multiple layers according to an embodiment of this application;

[0057] Figure 7 A flowchart illustrating a display control method provided in another embodiment of this application;

[0058] Figure 8 A flowchart illustrating a display control method provided in yet another embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the structure of a possible first electronic device provided in an embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the structure of a possible second electronic device provided in an embodiment of this application. Detailed Implementation

[0061] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0062] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.

[0063] A personal computer (PC) can connect to a tablet computer via a communication network. This network can be wired or wireless. For example, it can be a local area network (LAN) or a wide area network (WAN), such as the Internet. The communication network can be implemented using any known network communication protocol, which can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth, Wireless Fidelity (Wi-Fi), NFC, Voice Over Internet Protocol (VoIP), communication protocols supporting network slicing architecture, or any other suitable communication protocol. Exemplarily, in some embodiments, a PC can establish a Wi-Fi connection with a tablet computer via the Wi-Fi protocol.

[0064] A PC can act as the source device, and a tablet can act as the destination device for the PC. Content displayed on the PC can be projected onto the tablet. For example, the PC can encode its content, and the resulting video stream can be sent to the tablet. The tablet then decodes and displays the received video stream, mirroring the PC desktop. Users can control the tablet using peripherals connected to the PC, such as a mouse and keyboard, effectively using the tablet as a second display for the PC.

[0065] The existing process of using a stylus to control a PC on a tablet involves three parts: a1. When the tablet detects the user using the stylus, it receives the stylus input, converts it into a stylus event, and sends it to the PC. The PC then responds to the stylus event. a2. The PC acquires a first-area image of the screen (which can be the entire screen or a specified area), encodes it, and sends it to the tablet. a3. After receiving the encoded data, the tablet decodes and displays it (sending the decoded image data to the graphics card to drive the tablet screen to display the image). This existing process is time-consuming, resulting in noticeable user latency and a poor user experience. In stylus scenarios, if the area of ​​image change is small, the existing solution still encodes the entire screen image, further contributing to the long processing time.

[0066] The following is combined Figure 1 An exemplary schematic diagram illustrating a scenario of collaborative interaction among multiple electronic devices provided in an embodiment of the present invention is presented.

[0067] The first electronic device 100 and the second electronic device 200 can be located on the same local area network. The first electronic device 100 and the second electronic device 200 can establish a communication connection via Wi-Fi. The first electronic device 100 includes a first display screen 1001, and the second electronic device 200 includes a second display screen 2001. The screen image (the entire screen image or a specified area of ​​the screen image) displayed on the second display screen 2001 can be projected onto the first display screen 1001 for display. For example, the first electronic device 100 can mirror the screen displayed by the second electronic device 200. The interface of a certain application APP1 running on the second electronic device 200 can be projected onto the first electronic device 100 for display.

[0068] The first electronic device 100 can be a handheld device such as a mobile phone or tablet computer. The second electronic device 200 can be a device such as a laptop computer or PC. The following example illustrates this with the first electronic device 100 being a tablet computer and the second electronic device 200 being a laptop computer.

[0069] Users can use the stylus 300 to operate the first electronic device 100, such as writing on the first display screen 1001 or performing air gestures. The stylus 300 may not be connected to the first electronic device 100, or it may establish a wireless communication connection with the first electronic device 100, such as connecting via Bluetooth or Wi-Fi.

[0070] When the first electronic device 100 detects the operation of the stylus 300, the first electronic device 100 can respond to the operation by generating an operation event associated with the operation, and the first electronic device 100 sends the operation event associated with the operation to the second electronic device 200.

[0071] When the second electronic device 200 receives an operation event, it can respond to the event, acquire image change information, and determine the image change area based on the information. If the area of ​​the image change area is less than or equal to a first preset area, the second electronic device 200 uses a partial refresh method to encode the image change area, and sends the encoded data stream to the first electronic device 100 for display. The partial refresh method can refer to refreshing the image change area. If the area of ​​the image change area is greater than the first preset area, the image change area is merged. If the area of ​​the merged image change area is still greater than the first preset area, the second electronic device 200 uses a global refresh method to encode a first region image of the screen (which can be the entire screen image or a specified area image), and sends the encoded data stream to the first electronic device 100 for display. The area of ​​the first region image can be larger than the area of ​​the image change area. If the area of ​​the merged image change area is less than or equal to the first preset area, the second electronic device 200 uses a partial refresh method to encode the image change area, and sends the encoded data stream to the first electronic device 100 for display. The first preset area can be set according to actual needs, and this application does not limit it.

[0072] In some embodiments, the second electronic device 200 is provided with an icon for enabling the collaborative optimization function. When the collaborative optimization function is enabled, the second electronic device 200 can determine whether to use a local refresh method or a global refresh method to refresh the displayed content based on the area of ​​the image change region. When the collaborative optimization function is not enabled, the second electronic device 200 uses a global refresh method to refresh the displayed content.

[0073] In some embodiments, when the second electronic device 200 detects that the network latency is greater than a preset latency, or when the user enables the collaborative optimization function on the second electronic device 200, the second electronic device 200 selects either a local refresh method or a global refresh method to refresh the image based on the area of ​​the image change region. For example, the preset latency is set to 10ms or 15ms. When the second electronic device 200 detects that the network latency is not greater than (less than) the preset latency, or when the second electronic device 200 detects that the collaborative optimization function is not enabled, the second electronic device 200 uses a global refresh method to refresh the image, that is, after the second electronic device 200 responds to the operation event, a preset area of ​​the screen (e.g., the entire screen display area) is encoded.

[0074] The following is combined Figure 2 An exemplary schematic diagram illustrating the interaction between the first electronic device 100 and the second electronic device 200 provided in an embodiment of the present invention is presented.

[0075] For ease of description below, in the application scenario described below, the second electronic device 200 is pre-installed with application APP1. Application APP1 can be selected according to actual needs and is not limited here. For example, application APP1 is a drawing application. The first electronic device 100 and the second electronic device 200 are on the same local area network, and application APP1 displayed on the second electronic device 200 is projected onto the first display screen 100 for display. For example... Figure 2 As shown, the process of interaction between the first electronic device 100 and the second electronic device 200 may specifically include:

[0076] 20. The second electronic device 200 opens the application APP1 and enters the interface of the application APP1, so that the user can use the hardware modules (mouse, keyboard, etc.) of the second electronic device 200 to draw on the application APP1.

[0077] 21. The second electronic device 200 selects the first electronic device 100 as the target device, and the second electronic device 200 and the first electronic device 100 establish a communication connection via Wi-Fi. The interface of the application APP1 on the second electronic device 200 is projected onto the first electronic device 100.

[0078] In some embodiments, the interaction process 21 can also be set before the interaction process 20, that is, the second electronic device 200 can first select the first electronic device 100 as the target device for screen projection, and then open the application APP1.

[0079] In some embodiments, the example of a communication connection established between a first electronic device 100 and a second electronic device 200 via Wi-Fi is described. The first electronic device 100 includes a video decoder, and the second electronic device 200 includes an image frame capture device and a video encoder. The image frame capture device can capture images of the application APP1 in real time and transmit them to the video encoder for encoding. The encoded video stream can be transmitted to the first electronic device 100 via the Wi-Fi channel. The first electronic device 100 can receive the encoded video stream sent by the second electronic device 200 via the Wi-Fi channel, decode it using the video decoder, and play the video stream, thereby displaying the application APP1 on the second electronic device 200 on the first electronic device 100.

[0080] 22. The first electronic device 100 detects the first operation of the stylus 300, and the first electronic device 100 generates first information associated with the first operation in response to the first operation of the stylus 300.

[0081] In some embodiments, taking the application APP1 as a drawing application as an example, the first operation may refer to a drawing operation performed by the user on the first electronic device 100, and the first electronic device 100 can detect the drawing operation. The first information may include operation location information and image frame information associated with the first operation.

[0082] 23. The first electronic device 100 sends first information associated with the first operation to the second electronic device 100.

[0083] In some embodiments, the first electronic device 100 may send first information associated with the first operation to the second electronic device 100 via a Wi-Fi channel.

[0084] 24. The second electronic device 200 acquires image change information based on the first information.

[0085] In some embodiments, image change information may refer to the image difference information between the current image frame and the previous image frame. When the application APP1 installed on the second electronic device 200 receives the first information, the application APP1 processes the first information, and then the second electronic device 200 can obtain the image change information through a preset application programming interface (API).

[0086] 25. The second electronic device 200 obtains the image change area based on the image change information.

[0087] In some embodiments, image change information may include multiple image differences. The second electronic device 200 can acquire the local region where each image difference is located. For example, the second electronic device 200 can obtain a bounding box based on the image difference. The area contained in the bounding box is defined as the local region where the image difference is located. The bounding box is a rectangle that can enclose the image difference and has the smallest area. The image change area may be composed of multiple local regions.

[0088] 26. When the area of ​​the image change region is less than or equal to the first preset area, the second electronic device 200 encodes the image change region to obtain the first image.

[0089] In some embodiments, the image change area is in RGB format, and the second electronic device 200 can encode the image change area in JPEG format to obtain the first image.

[0090] 27. The second electronic device 200 transmits the first image to the first electronic device 100 for display.

[0091] In some embodiments, when the area of ​​the image change region is greater than a first preset area, the second electronic device 200 can merge the image change regions. If the area of ​​the merged image change region is still greater than the first preset area, the second electronic device 200 uses a global refresh method to encode the first area image of the screen (which may be the entire screen image or a specified area image of the screen, such as the display interface of application APP1) to obtain a first image, and transmits it to the first electronic device 100 for display. If the area of ​​the merged image change region is less than or equal to the first preset area, the second electronic device 200 can encode only the image change region to obtain the first image.

[0092] 28. The first electronic device 100 decodes and displays the first image.

[0093] This application allows the second electronic device to not process the entire screen image when the first electronic device is operated with a stylus, but only to encode and transmit the image change area to the first electronic device for display. This can effectively reduce the latency of reverse control and improve the user experience.

[0094] Reference Figure 3 As shown in the illustration, this application provides a display control method applied to a first electronic device 100 and a second electronic device 200. The first electronic device 100 and the second electronic device 200 are located on the same local area network. The following description uses the example of the first electronic device 100 and the second electronic device 200 establishing a communication connection via Wi-Fi. The application APP1 displayed on the second electronic device 200 can be projected onto the first electronic device 100 for display. In this embodiment, the display control method includes:

[0095] Step 30: In response to the first operation of the stylus 300, the first electronic device 100 sends first information associated with the first operation to the second electronic device 200.

[0096] In some embodiments, the first electronic device 100 includes a first display screen 1001, and the second electronic device 200 includes a second display screen 2001. Both the first display screen 1001 and the second display screen 2001 can each have a custom coordinate system. When a user performs a first operation on the first electronic device 100 using a stylus 300, the first electronic device 100 can detect operation information of the stylus 300 on the first display screen 1001. For example, the operation information includes coordinate point information involved in the first operation. The first information may include operation location information and image frame information associated with the first operation.

[0097] In some embodiments, the first operation may refer to a user writing directly on the first display screen 1001 using a stylus 300, or a user manipulating the content displayed on the first display screen 1001 remotely using a stylus 300. The first electronic device 100 may send first information associated with the first operation to the second electronic device 200 via a Wi-Fi channel.

[0098] Step 31: When the second electronic device 200 receives the first information, the second electronic device 200 acquires the image frame corresponding to the first information.

[0099] Step 32: The second electronic device 200 generates a first image based on the image frame and the previous image frame of the image frame.

[0100] In some embodiments, when the second electronic device 200 receives the first information, the application APP1 can process the first information, and the second electronic device 200 can capture the image frame after responding to the first information and the previous image frame of the image frame, and then generate the first image based on the image frame and the previous image frame of the image frame.

[0101] In some embodiments, such as Figure 4 As shown, step 32 may include:

[0102] Step 320: The second electronic device 200 acquires image change information of the image frame relative to the previous image frame;

[0103] Step 321: The second electronic device 200 obtains the image change area based on the image change information;

[0104] Step 322: The second electronic device 200 calculates the area of ​​the image change region;

[0105] Step 323: If the area of ​​the image change region is less than or equal to the first preset area, the second electronic device 200 encodes the image change region to obtain the first image.

[0106] In some embodiments, the second electronic device 200 can obtain image change information of an image frame relative to the previous image frame through a preset API. The image change information may include information such as the location and size of the image differences between the two frames. The second electronic device 200 can obtain image change regions based on the image change information. For example, if the image change information includes multiple image differences, the second electronic device 200 can obtain the local region where each image difference is located and define each local region as an image change sub-region. An image change region can be composed of multiple image change sub-regions, with each sub-region corresponding to one image difference. For example, the region with the smallest area that can frame the image difference is defined as the local region where that image difference is located; each frame region is an image change sub-region.

[0107] In some embodiments, after determining the image change region, the second electronic device 200 can obtain the corner coordinates of each image change sub-region and calculate the area of ​​each image change sub-region based on the corner coordinates. Since each image change sub-region is a rectangular area, the corner coordinates of the image change sub-region can refer to the coordinates of the four corner points of the rectangular area. When the coordinates of the four corner points are obtained, the second electronic device 200 can obtain the length and width of the image change sub-region based on the coordinates of the four corner points, and then calculate the area of ​​the image change sub-region based on the length and width of the image change sub-region. When the second electronic device 200 calculates the area of ​​each image change sub-region, it can sum up the areas of each image change sub-region to obtain the area of ​​the image change region.

[0108] If the area of ​​the image change region is less than or equal to the first preset area, the second electronic device 200 can encode the image change region to obtain the first image. For example, if the image change region is in RGB format, the second electronic device 200 can encode the RGB format image change region in JPEG format to obtain the first image. The first preset area can be set according to actual needs, and this application does not limit it. For example, the first preset area is one-third of the area of ​​the second display screen 2001.

[0109] If the area of ​​the image change region is greater than a first preset area, the second electronic device 200 can perform a merging operation on multiple image change sub-regions, and then recalculate the area of ​​the image change region based on the merged image change sub-regions. The merging operation on multiple image change sub-regions can refer to filtering the multiple image change sub-regions. For example, the second electronic device 200 can use OpenCV to binarize multiple image change sub-regions to obtain the image grayscale value corresponding to each image change sub-region. If the difference between the grayscale values ​​of two image change sub-regions is less than a preset difference, it indicates that the image difference between the two image change sub-regions is small, and the image processing time of the two image change sub-regions can be approximately equal to the image processing time of one image change sub-region. When recalculating the area of ​​the image change region, the second electronic device 200 can ignore the area of ​​one of the two image change sub-regions. If the difference between two image grayscale values ​​is not less than the preset difference, then the areas of the two image change sub-regions need to be counted when calculating the area of ​​the image change region. The second electronic device 200 can also binarize any two image change sub-regions from multiple image change sub-regions to obtain two image grayscale values ​​corresponding to the two image change sub-regions. If the difference between the two image grayscale values ​​is less than a preset difference, the second electronic device 200 can ignore the area of ​​one of the image change sub-regions in the arbitrary two image change sub-regions. The preset difference can be set according to actual needs, for example, the preset difference is 30.

[0110] For example, if the image grayscale value is between 0 and 255, and the image variation region includes four image variation sub-regions Ia1, Ia2, Ia3, and Ia4, the second electronic device 200 uses OpenCV to binarize the four image variation sub-regions Ia1 to Ia4 to obtain the image grayscale values ​​gv1 to gv4 corresponding to each image variation sub-region Ia1 to Ia4. If the difference between image grayscale value gv1 and image grayscale value gv2 is less than 30, and the difference between image grayscale value gv3 and image grayscale value gv4 is less than 30, the second electronic device 200 can ignore the area of ​​image variation sub-region Ia1 (or ignore the area of ​​image variation sub-region Ia2), and ignore the area of ​​image variation sub-region Ia3 (or ignore the area of ​​image variation sub-region Ia4). Assuming the second electronic device 200 performs filtering on multiple image change sub-regions Ia1 to Ia4, and ignores the area of ​​two image change sub-regions Ia1 and Ia3, the second electronic device 200 can recalculate the area of ​​the image change region based on the areas of image change sub-regions Ia2 and Ia4. If the recalculated area of ​​the image change region is less than or equal to a first preset area, the second electronic device 200 can encode the image change regions Ia1 to Ia4 to obtain the first image. If the recalculated area of ​​the image change region is still greater than the first preset area, it indicates that the efficiency of local refresh may be less than that of global refresh. In this case, the second electronic device 200 uses a global refresh method to encode the first area image of the screen (which can be the entire screen image or a specified area image of the screen) to obtain the first image.

[0111] In some embodiments, if both sets of image change sub-regions (each set includes two image change sub-regions) satisfy the condition that the difference between the image grayscale values ​​is less than a preset difference, and the two sets of image change sub-regions intersect, the second electronic device 200 may retain only the intersecting image change sub-regions. For example, if the image change region includes four image change sub-regions Ia1, Ia2, Ia3, and Ia4, and the difference between image grayscale values ​​gv1 and gv2 is less than 30, and the difference between image grayscale values ​​gv2 and gv3 is less than 30, the second electronic device 200 may ignore the area of ​​the image change sub-regions Ia1 and Ia3.

[0112] In some embodiments, the second electronic device 200 performing a merging operation on multiple image change sub-regions may also refer to performing a merging process on multiple image change sub-regions. For example, the second electronic device 200 selects two image change sub-regions from multiple image change sub-regions, and calculates the area of ​​the graphic region of the two selected image change sub-regions using a minimum bounding box. If the area of ​​the graphic region is less than or equal to a second preset area, the second electronic device 200 defines the graphic region as the region obtained by performing the merging process on the two selected image change sub-regions. The second electronic device 200 may repeat the above merging process until the area of ​​the graphic region of the two image change sub-regions selected by the minimum bounding box is not less than the second preset area. The second preset area can be set according to actual needs, and this application does not limit it.

[0113] like Figure 5As shown, assuming the image change region includes four image change sub-regions Ia1, Ia2, Ia3, and Ia4, with overlapping portions between Ia1 and Ia2, the second electronic device 200 arbitrarily selects two image change sub-regions Ia1 and Ia2 from the multiple image change sub-regions Ia1 to Ia4. It then obtains the coordinates of the four corner points of the bounding box b1 containing the selected two image change sub-regions Ia1 and Ia2, and can calculate the area of ​​the bounding box b1. If the area of ​​the bounding box b1 is less than a second preset area, the second electronic device 200 defines the area covered by the bounding box b1 as the image region obtained by merging the image change sub-regions Ia1 and Ia2. At this time, the image change region changes to include the image change sub-regions Ia3 and Ia4 and the bounding box b1. If the area of ​​the bounding box b1 is greater than the second preset area, the second electronic device 200 abandons the merging process for the image change sub-regions Ia1 and Ia2. The second electronic device 200 can again attempt to merge the image change sub-regions Ia3 and Ia4 and the region of box b1 using the above-described merging processing method. For example, the second electronic device 200 obtains the coordinates of the four corner points of box b2 of the two image change sub-regions Ia3 and Ia4 that can be bounded by the minimum bounding capacity. If the calculated area of ​​box b2 is less than a second preset area, the second electronic device 200 defines the region covered by box b2 as the image region obtained by merging the image change sub-regions Ia3 and Ia4. At this time, the image change region changes again to include the region of box b1 and the region of box b2. The second electronic device 200 then obtains the coordinates of the four corner points of box b3, which is the region of box b1 and the region of box b2 that can be bounded by the minimum bounding capacity. If the calculated area of ​​box b3 is greater than the second preset area, the second electronic device 200 abandons the merging processing of the region of box b1 and the region of box b2. That is, the second electronic device 200 performs merging processing on multiple image change sub-regions Ia1 to Ia4 to obtain the region of box b1 and the region of box b2. The second electronic device 200 combines the areas of box b1 and box b2 to recalculate the area of ​​the image change region. If the recalculated area of ​​the image change region is less than or equal to the first preset area, the second electronic device 200 can encode the image change regions Ia1 to Ia4 to obtain the first image. If the recalculated area of ​​the image change region is still greater than the first preset area, the second electronic device 200 uses a global refresh method to encode the first area image of the screen (which can be the entire screen image or a specified area image of the screen) to obtain the first image.

[0114] In some embodiments, the second electronic device 200 may first filter multiple image change sub-regions, and then merge the filtered image change sub-regions. Alternatively, the second electronic device 200 may first merge multiple image change sub-regions, and then filter the merged image change sub-regions.

[0115] Step 33: The second electronic device 200 transmits the first image to the first electronic device 100 for display.

[0116] In some embodiments, the first image may include multiple sub-images and timestamps, location information, and size information associated with each sub-image, and each sub-image may correspond to an image change area. The second electronic device 200 may transmit the multiple sub-images corresponding to the multiple image change areas, as well as the timestamps, location information, and size information of each sub-image, to the first electronic device 100 via a Wi-Fi channel.

[0117] Assume that the second electronic device 200 is based on image frame I n and image frame I n Previous image frame I n-1 Generate the first image I m Image frame I n With timestamp T n Image frame I n-1 With timestamp T n-1 At this time, the first image I m The timestamps of the multiple sub-images included are all T. n If the second electronic device 200 is based on image frame I n+1 and image frame I n+1 Previous image frame I n Generate the first image I m+1 Image frame I n+1 With timestamp T n+1 At this time, the first image I m+1 The timestamps of the multiple sub-images included are all T. n+1 If the second electronic device 200 is based on image frame I n+2 and image frame I n+2 Previous image frame I n+1 Generate the first image I m+2 Image frame I n+2 With timestamp T n+2 At this time, the first image I m+2 The timestamps of the multiple sub-images included are all T. n+2 .

[0118] When the first electronic device 100 receives the first image, it can add a new layer corresponding to the first image. The first electronic device 100 can add multiple sub-images to the layer based on the position and area information of each sub-image. The first electronic device 100 can merge and display multiple layers based on the order of their timestamps, where the timestamp of a layer corresponds to the timestamp of the sub-image added to that layer.

[0119] For example, the first electronic device 100 receives the first image I m At that time, a new image can be added that is related to the first image I. m Corresponding layer L m Layer L m The timestamp is T n The first electronic device 100 can interact with the first image I m Add the corresponding sub-images to layer L m The first electronic device 100 receives the first image I m+1 At that time, a new image can be added that is related to the first image I. m+1 Corresponding layer L m+1 Layer L m+1 The timestamp is T n+1 The first electronic device 100 can interact with the first image I m+1 Add the corresponding sub-images to layer L m+1 The first electronic device 100 receives the first image I m+2 At that time, a new image can be added that is related to the first image I. m+2 Corresponding layer L m+2 Layer L m+2 The timestamp is T n+2 The first electronic device 100 can interact with the first image I m+2 Add the corresponding sub-images to layer L m+2 The first electronic device 100 can be based on multiple layers L m ~L m+2 The order of timestamps for multiple layers L m ~L m+2 Merge textures for display. For example, the first electronic device 100 uses SurfaceFlinger to display multiple layers L. m ~L m+2 Merge and display textures.

[0120] like Figure 6 As shown, the first electronic device 100 is based on 3 layers L m ~L m+2 The order of timestamps for the three layers L m ~L m+2 Merge and display textures.

[0121] In some embodiments, during the process of merging and displaying multiple layers, the first electronic device 100 may only merge and display layers whose timestamps are after the currently displayed layer, while layers whose timestamps are earlier than the currently displayed layer are not displayed. The first electronic device 100 may compare the timestamps of the multiple layers with the timestamp of the currently displayed layer to filter out layers whose timestamps are later than the timestamp of the currently displayed layer.

[0122] In some embodiments, the second electronic device 200 further includes a simplified refresh mode. The second electronic device 200 may also have an icon for enabling the simplified refresh function. When the simplified refresh function is enabled, the second electronic device 200 can use the simplified refresh mode to refresh the displayed content. When the simplified refresh function is not enabled, the second electronic device 200 uses a global refresh mode to refresh the displayed content. The second electronic device 200 can also use the simplified refresh mode to refresh the displayed content when it detects that the current stylus usage scenario is a whiteboard writing scenario. A whiteboard writing scenario can refer to a drawing / writing scenario using black lines on a white background with fixed line width, requiring no other tools. When the second electronic device 200 receives the first information sent by the first electronic device 100, the first information may include information such as line color and line width. The second electronic device 200 can simulate and execute the first operation based on the first information to generate the first image, which can further reduce the latency of the first electronic device 100 controlling the second electronic device 200. The second electronic device 200 does not need to capture the screen to obtain the image difference between the current image frame and the previous image frame.

[0123] Reference Figure 7 As shown in the embodiment of this application, a display control method is applied to a second electronic device 200. The first electronic device 100 and the second electronic device 200 are located on the same local area network. The application APP1 displayed on the second electronic device 200 is projected onto the first electronic device 100 for display. In this embodiment, the display control method includes:

[0124] Step 71: Receive the first information sent by the first electronic device 100.

[0125] In some embodiments, when a user performs a first operation on the first electronic device 100 using a stylus 300, the first electronic device 100 can sense the first operation of the stylus 300. The first information may refer to information associated with the first operation of the stylus 300. For example, the first information may include operation location information and image frame information associated with the first operation.

[0126] In some embodiments, when the second electronic device 200 receives the first information sent by the first electronic device 100, the application APP1 can respond to the first message, and the second electronic device 200 enters a partial refresh state.

[0127] Step 72: Obtain the image frame corresponding to the first information.

[0128] Step 73: Generate a first image based on the image frame and the previous image frame.

[0129] In some embodiments, after application APP1 responds to the first information, the second electronic device 200 can obtain image change information of the currently displayed image frame relative to the previous image frame through a preset API. The image change information may include information such as the position and size of the image differences between the two frames. The second electronic device 200 can obtain the image change area based on the image change information. For example, if the image change information includes multiple image differences, the second electronic device 200 can obtain the local region where each image difference is located and define each local region where the image difference is located as an image change sub-region. The image change area can be composed of multiple image change sub-regions, and each image change sub-region corresponds to one image difference. For example, the area with the smallest area that can frame the image difference is defined as the local region where the image difference is located, and each frame area is an image change sub-region.

[0130] After determining the image change area, the second electronic device 200 can obtain the corner coordinates of each image change sub-region and calculate the area of ​​each image change sub-region based on the corner coordinates. Since each image change sub-region is a rectangular area, the corner coordinates of the image change sub-region can refer to the coordinates of the four corner points of the rectangular area. When the coordinates of the four corner points are obtained, the second electronic device 200 can obtain the length and width of the image change sub-region based on the coordinates of the four corner points, and then calculate the area of ​​the image change sub-region based on the length and width of the image change sub-region. When the second electronic device 200 calculates the area of ​​each image change sub-region, it can sum the areas of each image change sub-region to obtain the area of ​​the image change region. If the area of ​​the image change region is less than or equal to a first preset area, the second electronic device 200 can encode the image change region to obtain the first image. If the area of ​​the image change region is greater than the first preset area, the second electronic device 200 can perform a merging operation on multiple image change sub-regions, and then recalculate the area of ​​the image change region based on the merged image change sub-region. If the recalculated area of ​​the image change region is less than or equal to the first preset area, the second electronic device 200 can encode the image change region to obtain the first image. If the area of ​​the image change region obtained by recalculation is still greater than the first preset area, the second electronic device 200 adopts a global refresh method to encode the first area image of the screen (which may be the entire screen image or a specified area image of the screen) to obtain the first image.

[0131] Step 74: Transmit the first image to the first electronic device 100 for display.

[0132] In some embodiments, the first image may include multiple sub-images and timestamps, location information, and size information associated with each sub-image, and each sub-image may correspond to an image change area. The second electronic device 200 may transmit the multiple sub-images corresponding to the multiple image change areas, as well as the timestamps, location information, and size information of each sub-image, to the first electronic device 100 via a Wi-Fi channel.

[0133] Reference Figure 8 As shown in the embodiment of this application, a display control method is applied to a first electronic device 100. The first electronic device 100 and a second electronic device 200 are located on the same local area network. The application APP1 displayed on the second electronic device 200 is projected onto the first electronic device 100 for display. In this embodiment, the display control method includes:

[0134] Step 81: Receive the first image sent by the second electronic device 200 and add a new layer corresponding to the first image.

[0135] In some embodiments, the first image includes multiple sub-images corresponding to multiple image change sub-regions, and timestamps, location information, and size information of each sub-image. When the first electronic device 100 receives the first image sent by the second electronic device 200, it can add a layer corresponding to the first image.

[0136] Step 82: Add multiple sub-images to the layer based on the position and size information of each sub-image.

[0137] Step 83: Based on the timestamp order of the layers, merge and display the textures of multiple layers.

[0138] In some embodiments, the first electronic device 100 can add multiple sub-images to a layer based on the position and area information of each sub-image. The first electronic device 100 can merge and display multiple layers based on the order of their timestamps, wherein the timestamp of a layer corresponds to the timestamp of the sub-image added to that layer.

[0139] In some embodiments, during the process of merging and displaying multiple layers, the first electronic device 100 may only merge and display layers whose timestamps are after the currently displayed layer, while layers whose timestamps are earlier than the currently displayed layer are not displayed. The first electronic device 100 may compare the timestamps of the multiple layers with the timestamp of the currently displayed layer to filter out layers whose timestamps are later than the timestamp of the currently displayed layer.

[0140] refer to Figure 9 This is a schematic diagram of the hardware structure of the first electronic device 100 provided in an embodiment of this application. Figure 9 As shown, the first electronic device 100 may include a first display screen 1001, a first processor 1002, a first memory 1003, and a first communication bus 1004. The first memory 1003 stores one or more first computer programs 1005. The one or more first computer programs 1005 are configured to be executed by the first processor 1002. The one or more first computer programs 1005 include instructions that can be used to perform actions such as... in the first electronic device 100. Figure 8 The aforementioned display control method.

[0141] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the first electronic device 100. In other embodiments, the first electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.

[0142] refer to Figure 10 This is a schematic diagram of the hardware structure of the second electronic device 200 provided in an embodiment of this application. Figure 10 As shown, the second electronic device 200 may include a second display screen 2001, a second processor 2002, a second memory 2003, and a second communication bus 2004. The second memory 2003 stores one or more second computer programs 2005. The one or more second computer programs 2005 are configured to be executed by the second processor 2002. The one or more second computer programs 2005 include instructions that can be used to perform actions such as... in the second electronic device 200. Figure 7 The aforementioned display control method.

[0143] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the second electronic device 200. In other embodiments, the second electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.

[0144] Both the first processor 1002 and the second processor 2002 may include one or more processing units, such as an application processor (AP), modem, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0145] Both the first processor 1002 and the second processor 2002 may also include a memory for storing instructions and data. In some embodiments, the memory in the first processor 1002 and the second processor 2002 is a cache memory. This memory can store instructions or data that the first processor 1002 (second processor 2002) has just used or is repeatedly used. If the first processor 1002 (second processor 2002) needs to use the instruction or data again, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the first processor 1002 (second processor 2002), and thus improves the efficiency of the system.

[0146] In some embodiments, both the first processor 1002 and the second processor 2002 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, SIM interfaces, and / or USB interfaces, etc.

[0147] In some embodiments, both the first memory 1003 and the second memory 2003 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0148] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the display control method in the above embodiment.

[0149] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the display control method in the above embodiment.

[0150] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the display control methods in the above-described method embodiments.

[0151] In this embodiment, the first electronic device, the second electronic device, the computer storage medium, the computer program product, or the chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are illustrative. For instance, the division of modules or units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0154] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A display control method, applied to a first electronic device and a second electronic device, wherein a screen image displayed by the second electronic device is projected onto the first electronic device for display, characterized in that, The method includes: In response to a first operation of the stylus, the first electronic device sends first information associated with the first operation to the second electronic device; When the second electronic device receives the first information, the second electronic device acquires the image frame corresponding to the first information; The second electronic device obtains image change information of the image frame relative to the previous image frame based on the image frame and the previous image frame, and obtains the image change area based on the image change information; The second electronic device obtains at least one image difference based on the image change information; The second electronic device defines the local region where each image difference is located as an image change sub-region, wherein the local region is a box region that can contain the image difference the smallest; The second electronic device encodes the image change area to generate a first image; The second electronic device transmits the first image to the first electronic device for display; If the area of ​​the image change region is greater than the first preset area, the second electronic device performs a merging operation on the multiple image change sub-regions; The second electronic device recalculates the area of ​​the image change region based on the image change sub-region after the merging operation; If the area of ​​the recalculated image change region is less than or equal to the first preset area, the second electronic device encodes the image change region to obtain the first image; The second electronic device performs a merging operation on multiple image change sub-regions, including: The second electronic device performs binarization processing on the plurality of image change sub-regions to obtain image grayscale values ​​corresponding to the plurality of image change sub-regions; If the difference between the grayscale values ​​of two image change sub-regions is less than a preset difference, the second electronic device ignores the area of ​​one of the two image change sub-regions.

2. The display control method as described in claim 1, characterized in that, The first information includes operation location information and image frame information associated with the first operation.

3. The display control method as described in claim 1, characterized in that, Encoding the image change region to generate the first image includes: The second electronic device calculates the area of ​​the image change region; If the area of ​​the image change region is less than or equal to the first preset area, the second electronic device encodes the image change region to obtain the first image.

4. The display control method as described in claim 3, characterized in that, The calculation of the area of ​​the image change region includes: The second electronic device acquires the corner coordinates of each image change sub-region and calculates the area of ​​each image change sub-region based on the corner coordinates; The second electronic device obtains the area of ​​the image change region based on the sum of the areas of each image change sub-region.

5. The display control method as described in claim 1, characterized in that, The merging operation on multiple image change sub-regions includes: The second electronic device selects two image change sub-regions from the plurality of image change sub-regions; The second electronic device calculates the area of ​​the graphic region of the two image variation sub-regions selected using the minimum energy frame; If the area of ​​the graphic region is less than or equal to the second preset area, the second electronic device defines the graphic region as the region obtained by performing a merging operation on the two selected image change sub-regions.

6. The display control method as described in claim 1, characterized in that, The first image includes multiple sub-images corresponding to multiple image change sub-regions, and timestamps, location information, and size information of each sub-image. The step of transmitting the first image to the first electronic device for display includes: When the first electronic device receives the first image, it adds a layer corresponding to the first image. The first electronic device adds the plurality of sub-images to the layer based on the location information and size information; The first electronic device merges and displays multiple layers based on the timestamp order of the layers, wherein the timestamp of the layer corresponds to the timestamp of the sub-image added to the layer.

7. The display control method as described in claim 6, characterized in that, The method further includes: The first electronic device compares the timestamps of the multiple layers with the timestamp of the layer currently displayed by the first electronic device, and filters out layers whose timestamps are later than the timestamp of the currently displayed layer.

8. The display control method as described in claim 1 or 2, characterized in that, The step of generating the first image based on the first information includes: When the second electronic device receives the first information, the second electronic device simulates and executes the first operation based on the first information to generate the first image.

9. The display control method as described in claim 1, characterized in that, The method further includes: If the area of ​​the image change region obtained by recalculation is greater than the first preset area, the second electronic device performs a global refresh of the screen image based on the first information.

10. A display control method, applied to a second electronic device, wherein a screen image displayed by the second electronic device is projected onto a first electronic device for display, characterized in that, The method includes: Receive first information sent by the first electronic device, wherein the first information is associated with a first operation of the stylus; Obtain the image frame corresponding to the first information; Based on the image frame and the previous image frame, obtain image change information of the image frame relative to the previous image frame, and obtain the image change area based on the image change information; At least one image difference is obtained based on the image change information; Each local region containing an image difference is defined as an image change sub-region, wherein the local region is the smallest box region that can enclose the image difference; The image change region is encoded to generate a first image; The first image is transmitted to the first electronic device for display. If the area of ​​the image change region is greater than the first preset area, a merging operation is performed on the plurality of image change sub-regions; The area of ​​the image change region is recalculated based on the image change sub-region after the merging operation; If the area of ​​the recalculated image change region is less than or equal to the first preset area, the image change region is encoded to obtain the first image. The merging operation performed on the multiple image change sub-regions includes: Binarize the plurality of image change sub-regions to obtain image grayscale values ​​corresponding to the plurality of image change sub-regions; If the difference between the grayscale values ​​of two image change sub-regions is less than a preset difference, the area of ​​one of the two image change sub-regions is ignored and statistically analyzed.

11. The display control method as described in claim 10, characterized in that, Encoding the image change region to generate the first image includes: Calculate the area of ​​the image change region; If the area of ​​the image change region is less than or equal to a first preset area, the image change region is encoded to obtain the first image.

12. The display control method as described in claim 11, characterized in that, The calculation of the area of ​​the image change region includes: Obtain the corner coordinates of each image change sub-region, and calculate the area of ​​each image change sub-region based on the corner coordinates; The area of ​​the image change region is obtained by summing the areas of each of the image change sub-regions.

13. The display control method as described in claim 10, characterized in that, The merging operation on multiple image change sub-regions includes: Select two image change sub-regions from the plurality of image change sub-regions; Calculate the area of ​​the graphic region of the two image variation sub-regions selected using the minimum energy frame; If the area of ​​the graphic region is less than or equal to the second preset area, the graphic region is defined as the region obtained by performing a merging operation on the two selected image change sub-regions.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the display control method as described in any one of claims 10 to 13.

15. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store instructions, and the processor being used to invoke the instructions in the memory to cause the electronic device to perform the display control method as described in any one of claims 10 to 13.

16. A chip coupled to a memory in an electronic device, characterized in that, The chip is used to control the electronic device to perform the display control method as described in any one of claims 10 to 13.

Images