An Adaptive Screen Mirroring Method and a Tablet Computer

Abstract

The present invention provides an adaptive screen mirroring method and a tablet computer. By receiving a screen mirroring instruction input by a user, the screen mirroring instruction includes the network name and network address of a target device in a screen mirroring network, establishing a communication connection with the target device in the screen mirroring network, and exchanging screen mirroring parameters with the target device. The screen mirroring parameters include the I / O device type of the target device. According to the I / O device type of the target device, the display screen of the tablet computer is converted into a screen mirroring screen of the target device, and the screen mirroring screen is transmitted to the target device through the screen mirroring network for display, which can effectively improve the user interaction operation experience of screen mirroring.

Description

Technical Field

[0001] The present invention relates to the technical field of display control, and particularly relates to an adaptive screen mirroring method and a tablet computer. Background Art

[0002] Screen mirroring refers to projecting the display content of one device onto the display screen of another device, so as to utilize the hardware advantages of the other device to achieve a better viewing experience or operation experience. The reason why users need to perform projection is often that the device running the application has certain hardware deficiencies, such as a small screen, low resolution, narrow color gamut, or inconvenient interaction operations, while another device that does not run the application has relative hardware advantages, such as a larger screen size, higher resolution, wider color gamut, or more convenient interaction operations. Screen mirroring technology is usually applied in fields such as entertainment, office, or online education. A typical application method is to project the display content of a small-screen device onto a large-screen device to enhance the viewing experience. However, whether in the field of entertainment applications or office applications, it usually involves frequent user interaction operations. After the application is projected from a small-screen device to a large-screen device, its interaction method often changes greatly, and if the display content still adopts the original layout method, it will bring great inconvenience to the interaction operation, resulting in a poor user operation experience. Summary of the Invention

[0003] Based on the above problems, the present invention proposes an adaptive screen mirroring method and a tablet computer, which can effectively improve the user interaction operation experience of the screen mirroring.

[0004] In view of this, the first aspect of the present invention proposes an adaptive screen mirroring method, including:

[0005] Receiving a screen mirroring instruction input by a user, where the screen mirroring instruction includes the network name and network address of a target device in a screen mirroring network;

[0006] Establishing a communication connection with the target device in the screen mirroring network;

[0007] Exchanging screen mirroring parameters with the target device, where the screen mirroring parameters include the I / O device type of the target device;

[0008] Converting the display screen of the tablet computer into a screen mirroring screen of the target device according to the I / O device type of the target device;

[0009] Transmitting the screen mirroring screen to the target device through the screen mirroring network for display;

[0010] The steps of converting the display screen of the tablet computer into the screen mirroring screen of the target device according to the I / O device type of the target device specifically include:

[0011] Obtain the display screen of the tablet computer in real time;

[0012] Perform interactive element parsing on the display screen to obtain display - type interactive elements and operation - type interactive elements on the display screen;

[0013] Adjust the positions of the operation - type interactive elements on the display screen according to the I / O device type of the target device to generate the screen mirroring screen of the target device.

[0014] Further, the steps of performing interactive element parsing on the display screen to obtain display - type interactive elements and operation - type interactive elements on the display screen specifically include:

[0015] Generate an interactive element list on the display screen, and the interactive elements in the interactive element list are visible elements on the display screen;

[0016] Obtain the layout size attribute and interactive event attribute of each interactive element in the interactive element list;

[0017] Determine whether the corresponding interactive element is a display - type interactive element or an operation - type interactive element according to the layout size attribute and the interactive event attribute.

[0018] Further, before the step of generating the interactive element list on the display screen, it further includes:

[0019] Determine the foreground application corresponding to the display screen, and the foreground application is the application to which the activity (activity) at the top of the task stack of the tablet computer belongs;

[0020] Determine the activity at the top of the task stack of the tablet computer as the target activity;

[0021] Parse the resource file of the foreground application;

[0022] Determine the target layout file corresponding to the target activity in the resource file, and the target layout file is a resource file in XML (Extensible Markup Language) format.

[0023] Further, the steps of determining whether the corresponding interactive element is a display - type interactive element or an operation - type interactive element according to the layout size attribute and the interactive event attribute specifically include:

[0024] Calculate the screen coverage rate of each interactive element in the list of interactive elements , where , is the number of interactive elements in the list of interactive elements;

[0025] Generate an interactive event sequence for each interactive element in the list of interactive elements , where is the pre-configured number of standard interactive events;

[0026] Map the screen coverage rate ratio and the interactive event sequence to the input space of the interactive element classification model to obtain the input vector of the interactive element classification model ;

[0027] Input the input vector into the interactive element classification model to identify the type of the interactive element, and the type of the interactive element includes display - type interactive elements and operation - type interactive elements.

[0028] Furthermore, the step of generating an interactive event sequence for each interactive element in the list of interactive elements specifically includes:

[0029] Obtain the interactive event operation record of the interactive element, where the interactive event operation record is the record of the listening functions corresponding to the respective interactive event attributes of the interactive element being called and executed when the response conditions are met;

[0030] Construct the original interactive event sequence of the interactive element according to the interactive event operation record, and the length of the original interactive event sequence is equal to the number of interactive event attributes of the interactive element ;

[0031] Judge whether the length of the original interactive event sequence is greater than the standard number of interactive events ;

[0032] When the length of the original interactive event sequence is greater than the standard number of interactive events , intercept a part with a length equal to the standard number of interactive events from the original interactive event sequence to form the interactive event sequence of the interactive element ;

[0033] When the length of the original interactive event sequence is less than the standard number of interactive events , supplement in the original interactive event sequence padding bits to form an interaction event sequence of the interaction element .

[0034] Further, the step of adjusting the positions of the operation-based interaction elements on the display screen according to the I / O device type of the target device to generate the screen mirroring image of the target device specifically includes:[[]]

[0035] Determining the quantization operation characteristics of the target device according to the I / O device type of the target device, where the quantization operation characteristics include the operation accuracy and the focus change rate corresponding to the I / O device type;

[0036] Generating the layout characteristics of the operation-based interaction elements according to the quantization operation characteristics of the target device, where the layout characteristics include the interaction element scaling ratio and the interaction element layout spacing of the operation-based interaction elements;

[0037] Adjusting the positions of the operation-based interaction elements on the display screen according to the layout characteristics of the operation-based interaction elements to generate the screen mirroring image of the target device.

[0038] Further, the I / O devices of the target device include a main I / O device and a secondary I / O device, and the screen mirroring parameters include the main I / O device identifier and the secondary I / O device identifier of the target device. The step of determining the quantization operation characteristics of the target device according to the I / O device type of the target device specifically includes:[[]]

[0039] Determining the main I / O device and the secondary I / O device of the target device according to the main I / O device identifier and the secondary I / O device identifier;

[0040] Determining the first quantization operation characteristics of the target device based on the type of the main I / O device of the target device, where the first quantization operation characteristics of the target device include the operation accuracy of the main I / O device of the target device and the focus change rate of the main I / O device of the target device ;

[0041] Determining the second quantization operation characteristics of the target device based on the type of the secondary I / O device of the target device, where the second quantization operation characteristics of the target device include the operation accuracy of the th secondary I / O device of the target device and the focus change rate of the th secondary I / O device of the target device , where , is the number of secondary I / O devices of the target device.

[0042] Further, the step of generating the layout features of the operation - type interaction elements according to the quantization operation characteristics of the target device specifically includes:

[0043] Calculate the original scaling ratio:

[0044] ,

[0045] where is the original horizontal scaling ratio, is the original vertical scaling ratio, is the picture width of the screen - mirroring picture, is the picture height of the screen - mirroring picture;

[0046] Calculate the first correction coefficient of the main I / O device according to the quantization operation characteristics of the target device:

[0047] ;

[0048] Calculate the second correction coefficient of the secondary I / O device according to the quantization operation characteristics of the target device:

[0049] ,

[0050] Based on the first correction coefficient and the second correction coefficient calculate the interaction - element scaling ratio:

[0051] ,

[0052] where is the horizontal scaling ratio of the interaction element, is the vertical scaling ratio of the interaction element, is the correction weight of the main I / O device, is the correction weight of the secondary I / O device.

[0053] Further, the step of generating the layout features of the operation - type interaction elements according to the quantization operation characteristics of the target device further includes:

[0054] Obtain the layout width and layout height in the layout dimensions of the th operation - type interaction element on the display screen, where , is the number of operation - type interaction elements on the display screen;

[0055] According to the interaction - element scaling ratio, calculate the layout dimensions of the th operation - type interaction element on the screen - mirroring picture:

[0056] ,

[0057] Among them, is the layout width of the th operation - type interaction element on the screen mirroring screen, is the layout height of the th operation - type interaction element on the screen mirroring screen;

[0058] Determine the minimum layout spacing of the operation - type interaction element on the screen mirroring screen:

[0059] ;

[0060] Calculate the interaction element layout spacing based on the minimum layout spacing:

[0061] .

[0062] A second aspect of the present invention provides a tablet computer, including a memory and a processor, and the processor executes the computer program stored in the memory to implement the adaptive screen mirroring method according to any one of the first aspects of the present invention.

[0063] The present invention provides an adaptive screen mirroring method and a tablet computer. By receiving a screen mirroring instruction input by a user, the screen mirroring instruction includes the network name and network address of a target device in a screen mirroring network, establishing a communication connection with the target device in the screen mirroring network, exchanging screen mirroring parameters with the target device, the screen mirroring parameters including the I / O device type of the target device, converting the display screen of the tablet computer into the screen mirroring screen of the target device according to the I / O device type of the target device, and transmitting the screen mirroring screen to the target device through the screen mirroring network for display, it can effectively improve the user interaction operation experience of the screen mirroring. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of an adaptive screen mirroring method provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] In order to more clearly understand the above - mentioned objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0066] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention may be practiced in other ways than those specifically described herein, and thus, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0067] In the description of the present invention, the term "a plurality of" refers to two or more, unless otherwise clearly defined. The orientation or positional relationship indicated by terms such as "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present invention. Terms such as "connection", "installation", "fixation", etc. should all be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the technical features indicated. Thus, features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0068] In the description of this specification, the descriptions of terms such as "an embodiment", "some embodiments", "specific embodiments", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or instance. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0069] Next, a method for adaptive screen mirroring and a tablet computer according to some embodiments of the present invention will be described with reference to the accompanying drawings.

[0070] As Figure 1 shown, a first aspect of the present invention provides a method for adaptive screen mirroring, including:

[0071] Receiving a screen mirroring instruction input by a user, where the screen mirroring instruction includes the network name and network address of a target device in a screen mirroring network;

[0072] Establishing a communication connection with the target device in the screen mirroring network;

[0073] Exchange screen mirroring parameters with the target device, where the screen mirroring parameters include the I / O (Input / Output) device type of the target device;

[0074] Convert the display screen of the tablet computer into the screen mirroring image of the target device according to the I / O device type of the target device;

[0075] Transmit the screen mirroring image to the target device for display through the screen mirroring network.

[0076] Specifically, the screen mirroring instruction is used to project the display content of the tablet computer onto the display screen of the target device. The screen mirroring instruction is triggered by the user's operation on the tablet computer. For example, the user scans the screen mirroring devices in the screen mirroring network through the tablet computer, and a list of screen mirroring devices is displayed on the screen of the tablet computer. When the user selects one of the screen mirroring devices in the list for screen mirroring, the selected screen mirroring device is determined as the target device, and at the same time, the screen mirroring instruction is generated.

[0077] The screen mirroring network can be a dedicated network formed by the tablet computer and the target device through wired or wireless connections, or a local area network provided by the same gateway device that the tablet computer and the target device join through wired or wireless connections. The network name is the name used by the target device to display in the screen mirroring network, and the network address is the communication address of the target device in the screen mirroring network.

[0078] The I / O devices of the target device include input / output control devices integrally installed on the target device body, such as touch panels, display panels, mechanical buttons, etc., and also include peripheral input / output control devices connected to the target device body through wired or wireless means to provide input / output control capabilities for the target device, such as mice, keyboards, remote controls, etc. The I / O device is the interaction medium between the user and the target device, and is the carrier for the user to input control instructions to the target device or receive input information from the target device.

[0079] Further, the step of converting the display screen of the tablet computer into the screen mirroring image of the target device according to the I / O device type of the target device specifically includes:

[0080] Obtain the display screen of the tablet computer in real time;

[0081] Perform interactive element parsing on the display screen to obtain display-type interactive elements and operation-type interactive elements on the display screen;

[0082] Adjust the positions of the operation - type interactive elements on the display screen according to the I / O device type of the target device to generate the screen mirroring image of the target device.

[0083] Specifically, the interactive elements refer to each operable or inoperable graphic element on the display screen. These image elements can be presented on the display screen in the form of controls such as icons, buttons, text boxes, pictures, or videos. Some of these interactive elements are mainly used to implement the function of content presentation, such as text box controls, picture controls, or video controls, etc. These interactive elements are collectively referred to as display - type interactive elements. While some other interactive elements are mainly used to implement user interaction, especially for receiving and responding to user input operations, such as icon controls and button controls, etc. These interactive elements are collectively referred to as operation - type interactive elements.

[0084] Since the display - type interactive elements are mainly used to implement the function of content presentation, they usually have the characteristics of high screen coverage rate and few interactive event attributes. In order to avoid distortion of the main content during screen mirroring, during the screen mirroring process, for display - type interactive elements, they are usually processed in the form of proportional scaling according to the screen sizes between the source device and the target device of the screen - mirroring content. The main function of the operation - type interactive elements is to provide user interaction. Therefore, their layout in the picture is mainly for convenience of operation. Since screen mirroring often occurs between two different types of devices, for example, projecting from a source device such as a tablet computer to a target device such as a personal computer or a television, etc. Since the I / O devices of these target devices usually have significant differences from those of a tablet computer, the interactive interface designed for touch operation on a tablet computer is often not suitable for the I / O devices such as personal computers or televisions, etc., resulting in time - consuming and laborious input operations. It is necessary to adjust the layout of the operation - type interactive elements according to the I / O devices of the target device to improve this problem.

[0085] Further, the steps of performing interactive element parsing on the display screen to obtain the display - type interactive elements and operation - type interactive elements on the display screen specifically include:

[0086] Generate an interactive element list on the display screen, and the interactive elements in the interactive element list are visible elements on the display screen;

[0087] Obtain the layout size attributes and interactive event attributes of each interactive element in the interactive element list;

[0088] Determine whether the corresponding interactive element is a display - type interactive element or an operation - type interactive element according to the layout size attributes and the interactive event attributes.

[0089] Further, before the step of generating the interactive element list on the display screen, it also includes:

[0090] Determine the foreground application corresponding to the display screen, where the foreground application is the application to which the activity at the top of the task stack of the tablet belongs;

[0091] Determine the activity at the top of the task stack of the tablet as the target activity;

[0092] Parse the resource file of the foreground application;

[0093] Determine the target layout file corresponding to the target activity in the resource file, where the target layout file is a resource file in XML (Extensible Markup Language) format.

[0094] Further, the step of generating the list of interactive elements on the display screen specifically includes:

[0095] Parse the XML tags in the target layout file, where the XML tags are composed of a tag name, tag attributes, and a tag value;

[0096] Identify the interactive element tags according to the tag name, tag attributes, and / or tag value of the XML tags;

[0097] Extract the tag name, tag attributes, and tag value of the interactive element tags to form the list of interactive elements.

[0098] Specifically, in the layout resource file, its XML tags include layout tags and interactive element tags. Usually, the layout tags and interactive element tags can be identified according to the tag names. For example, LinearLayout (linear layout), RelativeLayout (relative layout), ConstraintLayout (constraint layout), FrameLayout (frame layout), GridLayout (grid layout), etc. are layout tags, and TextView (text view), Button (button), EditText (text box), ImageView (image view), Spinner (drop-down list), CheckBox (checkbox), RadioButton (radio button), ProgressBar (progress bar), etc. are interactive element tags. The layout size attributes such as layout_width (layout width), layout_height (layout height), etc. are usually located in the tag attributes of the XML tags and can be obtained by parsing the target layout file. The interactive event attributes such as click event, long click event, touch event, key event, focus change event, etc. are usually defined by listeners and can be obtained by parsing listeners such as OnClickListener (click event listener), OnLongClickListener (long click event listener), OnTouchListener (touch event listener), OnKeyListener (key event listener), OnFocusChangeListener (focus change event listener), etc.

[0099] It should be known that since the display screen is usually dynamically changing, the layout file of an activity often contains not only the tags of visible elements but also the tags of invisible elements. There are two situations for invisible elements. One is that it is directly set to invisible in the visibility attribute of the interactive element, and the other is that it is invisible due to the occlusion of the upper-level interactive element. When performing the step of generating the list of interactive elements on the display screen, it also includes identifying the visibility of the elements through the visibility attributes of each interactive element or the size and position of each interactive element.

[0100] Further, the steps of determining that the corresponding interactive element is a display-type interactive element or an operation-type interactive element according to the layout size attribute and the interactive event attribute specifically include:

[0101] Calculate the screen coverage rate of each interactive element in the interactive element list , where , is the number of interactive elements in the list of interactive elements;

[0102] Generate an interaction event sequence for each interactive element in the list of interactive elements , where is the pre-configured number of standard interaction events;

[0103] Map the screen coverage ratio and the interaction event sequence to the input space of the interactive element classification model to obtain the input vector of the interactive element classification model ;

[0104] Input the input vector into the interactive element classification model to identify the type of the interactive element, and the types of the interactive elements include display-type interactive elements and operation-type interactive elements.

[0105] In the technical solution of the above embodiment, the interactive elements in the list of interactive elements are visible elements on the display screen. In the step of generating the list of interactive elements on the display screen, when any interactive element is completely covered by other interactive elements on the display screen, then this interactive element is not added to the list of interactive elements. Therefore, the number of interactive elements in the list of interactive elements is the number of visible elements on the display screen.

[0106] Further, the step of calculating the screen coverage ratio of each interactive element in the list of interactive elements specifically includes:

[0107] Obtain the layout width and layout height of the th interactive element, as well as the screen width and screen height of the display screen, where the layout width , the layout height , the screen width and the screen height are in pixels;

[0108] Calculate the screen coverage ratio of the th interactive element:

[0109] .

[0110] The interaction event sequence is an important data element that constitutes the input vector. Since classification models usually require input vectors to have the same dimension, in the technical solution of the present invention, for the sake of convenience in processing, a standard number of interaction events is pre-configured as the length of the interaction event sequence, so that when the screen coverage rate ratio and the interaction event sequence are mapped to the input space of the interaction element classification model to obtain the input vector of the interaction element classification model , each data element in the screen coverage rate and the interaction event sequence can be conveniently mapped to each dimension of the input vector .

[0111] The interaction element classification model is a deep learning model trained based on a large number of classification sample data. More specifically, developers collect and organize the screen coverage data and interaction event data of a large number of display-type interaction elements and operation-type interaction elements to form training sample data, associate classification labels of display-type interaction elements or operation-type interaction elements with these sample data, and input these sample data into a designed neural network structure for iterative training to obtain the interaction element classification model.

[0112] Further, the steps of generating the interaction event sequence of each interaction element in the interaction element list specifically include:

[0113] Obtain the interaction event operation record of the interaction element, where the interaction event operation record is the record of the listening functions corresponding to the respective interaction event attributes of the interaction element being called and executed when the response conditions are satisfied;

[0114] Construct the original interaction event sequence of the interaction element according to the interaction event operation record, and the length of the original interaction event sequence is equal to the number of interaction event attributes of the interaction element ;

[0115] Judge whether the length of the original interaction event sequence is greater than the standard number of interaction events ;

[0116] When the length of the original interaction event sequence is greater than the standard number of interaction events , intercept a part with a length equal to the standard number of interaction events from the original interaction event sequence to form the interaction event sequence of the interaction element;

[0117] When the length of the original interaction event sequence is less than the standard number of interaction events fill in padding bits in the original interaction event sequence to form the interaction event sequence of the interaction element .

[0118] Furthermore, before the step of generating the interaction event sequence of each interaction element in the interaction element list , it further includes constructing an interaction event dictionary, and the interaction event dictionary includes the numerical encoding of each interaction event attribute, which is used to quantify the interaction event attributes of the interaction element into numerical values, so that when mapping the screen coverage rate ratio and the interaction event sequence to the input space of the interaction element classification model to obtain the input vector of the interaction element classification model, it is convenient to map each data element in the screen coverage rate and the interaction event sequence to each dimension of the input vector .

[0119] Furthermore, the step of constructing the original interaction event sequence of the interaction element according to the interaction event operation record specifically includes:

[0120] Obtain the number of operations of each interaction event attribute in the interaction element;

[0121] Write the numerical encoding of the interaction event attribute into the original interaction event sequence in the order from more to less according to the number of operations;

[0122] The step of intercepting a part with a length equal to the standard number of interaction events from the original interaction event sequence to form the interaction event sequence of the interaction element specifically includes:

[0123] Intercept the first bits of data from the original interaction event sequence to form the interaction event sequence of the interaction element

[0124] Furthermore, the padding bit can be a pre-configured specific value, and the value of the padding bit is other values except the numerical encoding used to represent the interaction event attribute in the interaction event dictionary. For example, 0 or -1 can be used as the value of the padding bit. Fill in padding bits in the original interaction event sequence to form the interaction event sequence The steps specifically include supplementing at the end of the original interaction event sequence padding bits to form the interaction event sequence of the interaction element .

[0125] Furthermore, the step of adjusting the positions of the operation-based interaction elements on the display screen according to the I / O device type of the target device to generate the screen mirroring image of the target device specifically includes:

[0126] Determining the quantization operation characteristics of the target device according to the I / O device type of the target device, where the quantization operation characteristics include the operation accuracy and the focus switching rate corresponding to the I / O device type;

[0127] Generating the layout characteristics of the operation-based interaction elements according to the quantization operation characteristics of the target device, where the layout characteristics include the interaction element scaling ratio and the interaction element layout spacing of the operation-based interaction elements;

[0128] Adjusting the positions of the operation-based interaction elements on the display screen according to the layout characteristics of the operation-based interaction elements to generate the screen mirroring image of the target device.

[0129] The operation accuracy is a quantization operation characteristic reflecting the operation accuracy of the I / O device corresponding to the I / O device type, and its specific manifestation is the screen coverage rate of the minimum operation range of a single operation of the I / O device corresponding to the I / O device type. The minimum operation range of a single operation of various types of I / O devices can be obtained through manual configuration or by identifying the coverage range of a single operation of various types of I / O devices. The focus switching rate is a quantization operation characteristic reflecting the switching speed of the focus element of the I / O device corresponding to the I / O device type on the display screen, and it can be an empirical value obtained by analyzing a large amount of measurement data and manually configured into the system by developers, or it can be automatically statistically analyzed by the system according to various operation data.

[0130] The interaction element scaling ratio is the scaling ratio of the element size when the operation-based interaction element is projected from the display screen of the tablet computer to the display screen of the target device (i.e., the screen mirroring image), and the element size includes the layout width and layout height of the operation-based interaction element. That is, the interaction element scaling ratio is used to control the size scaling ratio of the operation-based interaction element projected from the display screen of the tablet computer to the display screen of the target device. The interaction element layout spacing is a characteristic reflecting the layout spacing of the operation-based interaction elements on the screen mirroring image, and the layout spacing is the maximum spacing between each operation-based interaction element and the adjacent operation-based interaction element on the screen mirroring image.

[0131] Further, the I / O devices of the target device include a main I / O device and a secondary I / O device, and the screen mirroring parameters include the main I / O device identifier and the secondary I / O device identifier of the target device. The steps of determining the quantization operation characteristics of the target device according to the I / O device type of the target device specifically include:

[0132] Determine the main I / O device and the secondary I / O device of the target device according to the main I / O device identifier and the secondary I / O device identifier;

[0133] Based on the type of the main I / O device of the target device, determine the first quantization operation characteristic of the target device. The first quantization operation characteristic of the target device includes the operation accuracy of the main I / O device of the target device and the focus change rate of the main I / O device of the target device ;

[0134] Based on the type of the secondary I / O device of the target device, determine the second quantization operation characteristic of the target device. The second quantization operation characteristic of the target device includes the operation accuracy of the th secondary I / O device of the target device and the focus change rate of the th secondary I / O device of the target device , where , is the number of secondary I / O devices of the target device.

[0135] Specifically, the main I / O device is the I / O device that is used more frequently in the target device, and the secondary I / O device is the I / O device that is used relatively less frequently in the target device. It can be specifically determined according to the comparison of the operation times of the I / O devices on the target device. For example, an I / O device with an average operation time ratio per unit time greater than a preset threshold is determined as the main I / O device, and the remaining I / O devices are determined as secondary I / O devices, etc.

[0136] Further, the screen mirroring parameters also include the main I / O device identifier and the secondary I / O device identifier of the target device. Before the target device exchanges the screen mirroring parameters, it records and counts the operation times of each I / O device, distinguishes the main I / O device and the secondary I / O device according to the operation times of each I / O device, and when exchanging the screen mirroring parameters, packs the main I / O device identifier and the secondary I / O device identifier that identify the I / O device as the main I / O device and the secondary I / O device into the screen mirroring parameters, so that the tablet computer can determine the main I / O device and the secondary I / O device in the target device according to the main I / O device identifier and the secondary I / O device identifier.

[0137] Further, the step of generating the layout feature of the operation class interaction element according to the quantization operation feature of the target device specifically includes:

[0138] Calculate the original scaling ratio:

[0139] ,

[0140] wherein is the original horizontal scaling ratio, is the original vertical scaling ratio, is the screen width of the screen mirroring image, is the screen height of the screen mirroring image;

[0141] Calculate the first correction coefficient of the main I / O device according to the quantization operation feature of the target device:

[0142] ;

[0143] Calculate the second correction coefficient of the secondary I / O device according to the quantization operation feature of the target device:

[0144] ,

[0145] Based on the first correction coefficient and the second correction coefficient calculate the interaction element scaling ratio:

[0146] ,

[0147] wherein is the interaction element horizontal scaling ratio, is the interaction element vertical scaling ratio, is the correction weight of the main I / O device, is the correction weight of the secondary I / O device.

[0148] Specifically, the original scaling ratio is the scaling ratio for projecting the display screen onto the display screen of the target device in equal proportion, and the display class interaction elements on the display screen are scaled and then displayed in the screen mirroring image at the original scaling ratio. To keep the display class interaction elements from being deformed, usually the original horizontal scaling ratio is configured to be equal to the original vertical scaling ratio In this case, when the aspect ratio of the screen of the target device is inconsistent with the aspect ratio of the screen of the tablet computer that is the source device of the display content, there will be black borders on the upper and lower or left and right sides when the screen mirroring image is displayed on the screen of the target device.

[0149] In the technical solution of the above embodiment, there is only one main I / O device of the target device, while there can be multiple secondary I / O devices. The first correction coefficient is based on the operation accuracy of the main I / O device of the target device and the focus switching rate and is calculated. The second correction coefficient is related to the operation accuracy of the secondary I / O device of the target device and the focus switching rate . The correction weights and are empirically configured values respectively used to correct the layout characteristics of the main I / O device and the secondary I / O device of the target device. By adjusting the magnitudes of the correction weights and , the influence of the quantization operation characteristics of the main I / O device and the secondary I / O device of the target device on the layout characteristics of its screen mirroring can be corrected, so as to obtain a better layout effect of the operation type interaction elements

[0150] Further, the step of generating the layout characteristics of the operation type interaction elements according to the quantization operation characteristics of the target device further includes

[0151] Obtaining the layout width and the layout height in the layout dimensions of the th operation type interaction element on the display screen, where , is the number of operation type interaction elements on the display screen

[0152] Calculating the layout dimensions of the th operation type interaction element on the screen mirroring according to the interaction element scaling ratio

[0153] ,

[0154] where is the layout width of the th operation type interaction element on the screen mirroring, is the layout height of the th operation type interaction element on the screen mirroring

[0155] Determining the minimum layout spacing of the operation type interaction elements on the screen mirroring

[0156] ;

[0157] Calculating the interaction element layout spacing based on the minimum layout spacing

[0158] 。

[0159] Further, the step of adjusting the positions of the operation-type interaction elements on the display screen according to the layout characteristics of the operation-type interaction elements to generate the screen mirroring image of the target device specifically includes arranging the operation-type interaction elements at equal intervals according to the interaction element layout spacing and placing them in a specified area of the screen mirroring image.

[0160] Specifically, after the operation-type interaction elements are scaled according to the interaction element scaling ratio and in order to avoid overlapping of their placement positions, the interaction element layout spacing should be greater than the minimum layout spacing 。

[0161] The second aspect of the present invention provides a tablet computer, including a memory and a processor, and the processor executes the computer program stored in the memory to implement the adaptive screen mirroring method according to any one of the first aspects of the present invention.

[0162] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0163] According to the embodiments of the present invention as described above, these embodiments do not describe all details in detail, nor limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the above description. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and its modifications based on the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. An adaptive screen projection method, characterized in that: include: Receive a screen projection instruction input by a user, wherein the screen projection instruction includes a network name and a network address of a target device in a screen projection network; Establishing a communication connection with the target device in the screen projection network; Exchanging screen projection parameters with the target device, wherein the screen projection parameters include an I / O device type of the target device; Converting the display screen of the tablet computer to the projection screen of the target device according to the I / O device type of the target device; Transmitting the projected screen image to the target device for display via the projected screen network; The step of converting the display screen of the tablet computer into the projection screen of the target device according to the I / O device type of the target device specifically includes: Get the display screen of the tablet in real time; Performing interactive element parsing on the display screen to obtain display-type interactive elements and operation-type interactive elements on the display screen; Adjusting the position of the operation-related interactive elements on the display screen according to the I / O device type of the target device to generate a projection screen of the target device; The step of adjusting the position of the operation-related interactive element on the display screen according to the I / O device type of the target device to generate a projection screen of the target device specifically includes: Determining a quantitative operation feature of the target device according to an I / O device type of the target device, wherein the quantitative operation feature includes an operation accuracy and a focus change rate corresponding to the I / O device type; Generating layout features of the operation-related interactive elements according to the quantized operation features of the target device, wherein the layout features include an interactive element scaling ratio and an interactive element layout spacing of the operation-related interactive elements; Adjusting the position of the operation-related interactive elements on the display screen according to the layout characteristics of the operation-related interactive elements to generate a projection screen of the target device; The I / O device of the target device includes a primary I / O device and a secondary I / O device, the screen projection parameter includes a primary I / O device identifier and a secondary I / O device identifier of the target device, and the step of determining the quantitative operation characteristics of the target device according to the I / O device type of the target device specifically includes: Determine the primary I / O device and the secondary I / O device of the target device according to the primary I / O device identifier and the secondary I / O device identifier; Determine a first quantitative operation characteristic of the target device based on the type of the main I / O device of the target device, wherein the first quantitative operation characteristic of the target device includes an operation precision p of the main I / O device of the target device m and the focus change rate tf of the main I / O device of the target device m ; Determine a second quantitative operation characteristic of the target device based on the type of the secondary I / O device of the target device, wherein the second quantitative operation characteristic of the target device includes an operation precision p of the kth secondary I / O device of the target device. k and the focus change rate tf of the kth secondary I / O device of the target device k , where k∈[1,n se ],n se The number of the secondary I / O device for the target device.

2. The adaptive screen projection method according to claim 1, characterized in that: The step of performing interactive element parsing on the display screen to obtain display-type interactive elements and operation-type interactive elements on the display screen specifically includes: Generate an interactive element list on the display screen, where the interactive elements in the interactive element list are visible elements on the display screen; Obtaining the layout size attribute and the interaction event attribute of each interactive element in the interactive element list; According to the layout size attribute and the interaction event attribute, it is determined that the corresponding interaction element is a display interaction element or an operation interaction element.

3. The adaptive screen projection method according to claim 2, characterized in that: Before the step of generating the interactive element list on the display screen, the method further includes: Determine a foreground application corresponding to the display screen, where the foreground application is an application to which an activity located at the top of a task stack of the tablet computer belongs; Determine the activity at the top of the task stack of the tablet computer as the target activity; Parsing the resource file of the foreground application; A target layout file corresponding to the target activity is determined in the resource file, and the target layout file is a resource file in XML (Extensible Markup Language) format.

4. The adaptive screen projection method according to claim 2, characterized in that: The step of determining whether the corresponding interactive element is a display interactive element or an operation interactive element according to the layout size attribute and the interactive event attribute specifically includes: Calculate the screen coverage r of each interactive element in the interactive element list i , where i∈[1,n elem ],n elem is the number of interactive elements in the interactive element list; Generate an interactive event sequence for each interactive element in the interactive element list Where n in The number of pre-configured standard interaction events; The screen coverage ratio r i And the interaction event sequence Mapped to the input space of the interactive element classification model to obtain the input vector of the interactive element classification model The input vector The interactive element classification model is input to identify the type of the interactive element, where the type of the interactive element includes a display-type interactive element and an operation-type interactive element.

5. The adaptive screen projection method according to claim 4, characterized in that: Generate an interactive event sequence for each interactive element in the interactive element list The steps specifically include: Acquire the interactive event operation record of the interactive element, where the interactive event operation record is a record of a listening function corresponding to each interactive event attribute of the interactive element being called and executed when a response condition is met; An original interaction event sequence of the interaction element is constructed according to the interaction event operation record, wherein the length of the original interaction event sequence is equal to the number n of interaction event attributes of the interaction element. ev ; Determine the length n of the original interaction event sequence ev Is it greater than the standard number of interaction events n? in ; When the length of the original interaction event sequence n ev Greater than the standard number of interaction events n in When the length of the standard interaction event number n is intercepted from the original interaction event sequence in The interactive event sequence of the interactive element When the length of the original interaction event sequence n ev Less than the standard number of interaction events n in When n is added to the original interaction event sequence in ―n ev Filling bits to form the interactive event sequence of the interactive element 6. The adaptive screen projection method according to claim 1, characterized in that: The step of generating the layout features of the operation-related interactive elements according to the quantitative operation features of the target device specifically includes: Calculate the original scaling: Where R w0 is the original horizontal scaling ratio, R h0 is the original vertical scaling ratio, w t is the screen width of the projection screen, h t is the screen height of the projection screen, w0 is the screen width of the display screen, and h0 is the screen height of the display screen; Calculate the first correction coefficient of the primary I / O device according to the quantitative operation characteristics of the target device: Calculate a second correction factor for the secondary I / O device based on the quantified operating characteristics of the target device: The interactive element scaling ratio is calculated based on the first correction coefficient c1 and the second correction coefficient c2: Where R w is the horizontal scaling ratio of the interactive element, R h is the vertical scaling ratio of the interactive element, α is the correction weight of the primary I / O device, and β is the correction weight of the secondary I / O device.

7. The adaptive screen projection method according to claim 6, characterized in that: The step of generating the layout features of the operation-related interactive elements according to the quantified operation features of the target device also includes: Get the layout width w_b of the layout size of the jth operation-type interactive element on the display screen j and layout height h_b j , where j∈[1,n opel ],n opel is the number of operation-related interactive elements on the display screen; Calculate the layout size of the jth operation-type interactive element on the projection screen according to the interactive element scaling ratio: Among them, w_a j is the layout width of the jth operation-type interactive element on the projection screen, h_a j is the layout height of the jth operation-type interactive element on the projection screen; Determine the minimum layout spacing of the operation-type interactive elements on the projection screen: Calculate the interactive element layout spacing based on the minimum layout spacing: d=d min ·(1+α·c1+β·c2).

8. A tablet computer, characterized in that: It includes a memory and a processor, and the processor executes a computer program stored in the memory to implement the adaptive screen projection method as described in any one of claims 1-7.

Images