INTERFACE LAYOUT METHOD, APPARATUS, AND SYSTEM
The interface layout method addresses the challenge of controlling projected interfaces across devices with varying screen sizes by using pre-trained models to rearrange elements, ensuring compatibility and consistent user experience.
Patent Information
- Application Number
- JP2022550007
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-20
- Filing Date
- 2020-10-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-10-30
AI Technical Summary
Users face difficulties in conveniently controlling projected interfaces across terminal devices with different screen sizes, leading to inconsistent user experiences.
An interface layout method that involves a first terminal device generating a second interface tailored to the screen size and state of a second terminal device using pre-trained interface recognition models to rearrange interface elements, ensuring compatibility and ease of control.
Enables users to conveniently control projected interfaces on various terminal devices, maintaining consistent control actions and improving user experience by adapting interfaces to the specific screen characteristics of the target device.
Smart Images

Figure 0007584526000002 
Figure 0007584526000003 
Figure 0007584526000004
Abstract
Description
[Technical field]
[0001] This application claims priority to Chinese Patent Application No. 2020101068011, entitled “INTERFACE LAYOUT METHOD, APPARATUS AND SYSTEM,” filed with the State Intellectual Property Office of China on February 10, 2020, which is incorporated herein by reference in its entirety.
[0002] This application belongs to the field of artificial intelligence recognition technology, in particular to an interface layout method, apparatus and system. [Background technology]
[0003] Based on the continuous development of terminal devices, when a terminal device loads an application, the terminal device can not only display the interface of the application, but also project the interface of the application onto another terminal device, so that a user can control the application to perform different functions via another terminal device, and the user can experience seamless services that enable consistent operations on different terminal devices.
[0004] In a related technology, when a first terminal device loads an application, if the first terminal device detects a screen projection action triggered by a user, the first terminal device can project a currently displayed interface of the application indicated by the screen projection action to a second terminal device based on the screen projection action, and the second terminal device can display the interface of the application displayed on the first terminal device.
[0005] However, different terminal devices have different screen sizes and different degrees of convenience for users to control the terminal devices, so that after an interface displayed on a first terminal device is projected onto a second terminal device, the user cannot conveniently control the projected interface via the second terminal device. Summary of the Invention [Means for solving the problem]
[0006] The embodiments of this application provide an interface layout method, apparatus and system to solve the problem that after a first terminal device projects a displayed interface onto a second terminal device, a user cannot conveniently control the projected interface via the second terminal device.
[0007] According to a first aspect, an embodiment of the present application provides an interface layout method, the method being applied to a first terminal device, the first terminal device being connected to a second terminal device, and the method comprising: receiving a screen projection command, the screen projection command being used to instruct the first terminal device to perform screen projection to a second terminal device; generating a second interface to be displayed on a second terminal device based on interface information of the first interface and second device information, where the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; Includes.
[0008] In a first possible implementation of the first aspect, the step of generating a second interface to be displayed on a second terminal device based on interface information of the first interface and second device information includes: Obtaining interface information of a first interface and second device information, where the interface information of the first interface includes element information of at least one interface element in the first interface, and the element information is used to indicate a name and a type of the interface element and a position of the interface element in the first interface; performing recognition based on element information of at least one interface element by using a pre-trained interface recognition model to determine an interface category; arranging at least one interface element based on the interface category and the second device information to obtain a second interface; Includes.
[0009] In relation to the first possible implementation of the first aspect, in a second possible implementation of the first aspect, the interface information of the first interface further includes interface attributes, and the interface attributes are used to indicate an interface size and an interface direction of the first interface; The step of performing recognition based on element information of at least one interface element by using a pre-trained interface recognition model to determine an interface category includes: performing feature extraction on at least one element information based on the interface attributes to obtain interface feature data; inputting interface feature data into an interface recognition model and recognizing the interface feature data by using the interface recognition model to obtain an interface category output from the interface recognition model; Includes.
[0010] In relation to the first possible implementation of the first aspect, in a third possible implementation of the first aspect, the step of arranging at least one interface element based on the interface category and the second device information to obtain the second interface includes: Dividing a display area of the second terminal device according to an interface category to obtain a plurality of sub-areas, the display areas being indicated by second device information; determining interface elements to be placed in each sub-area; adjusting each interface element in each sub-area based on a size of the display area indicated by the second device information and a quantity of the interface elements disposed in each sub-area to obtain a second interface; Includes.
[0011] In relation to the third possible implementation of the first aspect, in a fourth possible implementation of the first aspect, the step of adjusting each interface element in each sub-area based on a size of the display area indicated by the second device information and a quantity of interface elements to be arranged in each sub-area to obtain the second interface includes: determining a quantity of interface elements in each sub-area; adjusting a size and an orientation of each interface element in each sub-area based on a size of the display area, a preset arrangement rule, and a quantity of elements corresponding to the sub-area to obtain adjusted interface elements; adjusting, in each sub-area, a position of the adjusted interface element in the sub-area based on a quantity of elements corresponding to the sub-area to obtain a second interface; Includes.
[0012] In relation to any one of the first to fourth possible implementations of the first aspect, in a fifth possible implementation of the first aspect, after the step of generating a second interface to be displayed on the second terminal device based on the interface information of the first interface and the second device information, the method includes: transmitting the second interface to the second terminal device such that the second terminal device displays the second interface; Further includes:
[0013] In relation to the fifth possible implementation of the first aspect, in a sixth possible implementation of the first aspect, after the step of transmitting the second interface to the second terminal device, the method further comprises: obtaining feedback information, the feedback information being information fed back by a user regarding a second interface displayed on a second terminal device; updating the interface recognition model based on the feedback information when the feedback information satisfies a preset update condition; Further includes:
[0014] In relation to any one of the first to fourth possible implementations of the first aspect, in a seventh possible implementation of the first aspect, before the step of generating a second interface to be displayed on the second terminal device based on interface information of the first interface and second device information, the method includes: performing an interface element extraction in the first interface based on an extraction action triggered by a user to obtain a plurality of interface elements; generating element information for a plurality of interface elements based on the supplemental actions triggered by a user; Further includes:
[0015] In relation to any one of the first to fourth possible implementations of the first aspect, in an eighth possible implementation of the first aspect, after the step of generating a second interface to be displayed on the second terminal device based on the interface information of the first interface and the second device information, the method includes: recording a user-triggered adjustment action on at least one interface element in the second interface; adjusting the placement rules based on the adjustment action; Further includes:
[0016] According to a second aspect, an embodiment of the present application provides an interface layout apparatus, the apparatus being applied to a first terminal device, the first terminal device being connected to a second terminal device, and the apparatus comprising: A receiving module configured to receive a screen projection command, the screen projection command being used to instruct a first terminal device to perform screen projection to a second terminal device; A generating module configured to generate a second interface to be displayed on a second terminal device based on interface information of the first interface and second device information, where the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; Includes.
[0017] In a first possible implementation of the second aspect, the generation module is particularly configured to obtain interface information of a first interface and second device information, the interface information of the first interface including element information of at least one interface element in the first interface, the element information being used to indicate a name and type of the interface element and a position of the interface element in the first interface, perform recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category, and arrange the at least one interface element based on the interface category and the second device information to obtain a second interface.
[0018] In relation to the first possible implementation of the second aspect, in a second possible implementation of the second aspect, the interface information of the first interface further includes interface attributes, and the interface attributes are used to indicate an interface size and an interface direction of the first interface; The generation module is further specifically configured to perform feature extraction on at least one element information based on the interface attributes to obtain interface feature data, input the interface feature data into an interface recognition model, and recognize the interface feature data by using the interface recognition model to obtain an interface category output from the interface recognition model.
[0019] In relation to the first possible implementation of the second aspect, in a third possible implementation of the second aspect, the generation module is further configured to divide the display area of the second terminal device based on an interface category to obtain a plurality of sub-areas, the display area being indicated by the second device information, determine interface elements to be placed in each sub-area, and adjust each interface element in each sub-area based on the size of the display area indicated by the second device information and the quantity of interface elements to be placed in each sub-area to obtain a second interface.
[0020] In relation to the third possible implementation of the second aspect, in a fourth possible implementation of the second aspect, the generation module is further configured to determine a quantity of interface elements in each sub-area, adjust a size and orientation of each interface element in each sub-area based on the size of the display area, the preset placement rule, and the quantity of elements corresponding to the sub-area to obtain an adjusted interface element, and adjust a position of the adjusted interface element in the sub-area based on the quantity of elements corresponding to the sub-area, in each sub-area, to obtain a second interface.
[0021] In relation to any one of the first to fourth possible implementations of the second aspect, in a fifth possible implementation of the second aspect, the apparatus comprises: a transmitting module configured to transmit the second interface to the second terminal device such that the second terminal device displays the second interface; Further includes:
[0022] In relation to a fifth possible implementation of the second aspect, the apparatus comprises: an acquisition module configured to acquire feedback information, the feedback information being information fed back by a user regarding a second interface displayed on a second terminal device; an update module configured to update the interface recognition model based on the feedback information when the feedback information satisfies a preset update condition; Further includes:
[0023] In relation to any one of the first to fourth possible implementations of the second aspect, the apparatus comprises: an extraction module configured to perform an interface element extraction in the first interface based on an extraction action triggered by a user to obtain a plurality of interface elements; a supplemental module configured to generate element information for a plurality of interface elements based on a supplemental action triggered by a user; Further includes:
[0024] In relation to any one of the first to fourth possible implementations of the second aspect, in an eighth possible implementation of the second aspect, the apparatus comprises: a recording module configured to record an adjustment action triggered by a user on at least one interface element in the second interface; an adjustment module configured to adjust the placement rules based on the adjustment operation; Further includes:
[0025] According to a third aspect, an embodiment of the present application provides an interface layout system including a first terminal device and a second terminal device, the first terminal device being connected to the second terminal device; A first terminal device receives a screen projection command, and the screen projection command is used to instruct the first terminal device to perform screen projection to a second terminal device; The first terminal device generates a second interface to be displayed on the second terminal device according to interface information of the first interface and the second device information, the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; the first terminal device transmits the second interface to the second terminal device; A second terminal device receives and displays the second interface.
[0026] According to a fourth aspect, an embodiment of the present application provides a terminal device, the terminal device including a memory, a processor, and a computer program stored in the memory and executable by the processor, the computer program, when executed, causing the processor to perform an interface layout method according to the first aspect or any one of the possible implementations of the first aspect.
[0027] According to a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, which, when executed by a processor, performs an interface layout method according to the first aspect or any one of the possible implementations of the first aspect.
[0028] According to a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to perform an interface layout method according to the first aspect and any one of the possible implementations of the first aspect.
[0029] Compared with the prior art, the embodiments of this application have the following beneficial effects:
[0030] In an embodiment of this application, the first terminal device receives a projection command instructing the first terminal device to perform screen projection to the second terminal device, and generates a second interface to be displayed on the second terminal device based on the second device information and the interface information of the first interface displayed on the first terminal device, and the second device information is used to indicate the screen size and screen status of the second terminal device. In this way, the second terminal device can display a second interface compatible with the second terminal device, and the user can conveniently control the second interface through the second terminal device. This avoids the problem that the user cannot conveniently control the screen projection interface, improves the convenience of the user to control the second interface through the second terminal device, and improves the consistency between the control operations performed by the user on different terminal devices. [Brief description of the drawings]
[0031] [Figure 1] FIG. 1 is a diagram of a system architecture of an interface layout system related to an interface layout method according to an embodiment of the present application. [Diagram 2] FIG. 1 is a schematic diagram of the structure of a mobile phone according to an embodiment of the present application. [Diagram 3] FIG. 1 is a schematic diagram of a layered architecture of a software system according to an embodiment of the present application. [Figure 4] 1 is a schematic flowchart of an interface layout method according to an embodiment of the present application; [Diagram 5] FIG. 2 is a schematic diagram of a first interface of a player according to an embodiment of the present application. [Figure 6] FIG. 1 is a schematic diagram of an interface that falls into interface category 1 according to an embodiment of the present application. [Figure 7] FIG. 2 is a schematic diagram of an interface that falls into interface category 2 according to an embodiment of the present application. [Figure 8-a]FIG. 1 is a schematic diagram of an interface that falls into interface category 3 according to an embodiment of the present application. [Figure 8-b] FIG. 1 is a schematic diagram of an interface falling into another interface category 3 according to an embodiment of the present application. [Figure 9-a] FIG. 1 is a schematic diagram of an interface that falls into interface category 4 according to an embodiment of the present application. [Figure 9-b] FIG. 1 is a schematic diagram of an interface falling into another interface category 4 according to an embodiment of the present application. [Figure 10] FIG. 1 is a schematic diagram of an interface falling into interface category 5 according to an embodiment of the present application. [Figure 11] FIG. 1 is a schematic diagram of an interface that falls into interface category 6 according to an embodiment of the present application. [Figure 12] FIG. 1 is a schematic diagram of an interface that falls into interface category 7 according to an embodiment of the present application. [Figure 13] FIG. 1 is a schematic diagram of an interface that falls into interface category 8 according to an embodiment of the present application. [Figure 14A] FIG. 2 is a schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 14B] FIG. 2 is a schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 15A] FIG. 2 is another schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 15B] FIG. 2 is another schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 16A] FIG. 2 is yet another schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 16B] FIG. 2 is yet another schematic diagram of interfaces on different terminal devices according to an embodiment of the present application; [Figure 17] FIG. 2 is a schematic diagram of a first interface according to an embodiment of the present application. [Figure 18] FIG. 2 is a schematic diagram of an IDE interface according to an embodiment of the present application. [Figure 19] FIG. 1 is a structural block diagram of an interface layout device according to an embodiment of the present application; [Figure 20] FIG. 2 is a structural block diagram of another interface layout device according to an embodiment of the present application; [Figure 21] FIG. 2 is a schematic diagram of the structure of a terminal device according to an embodiment of this application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0032] In the following description, for purposes of illustration and not limitation, specific details are provided, such as specific system structures and techniques, to thoroughly understand the embodiments of this application. However, those skilled in the art should know that this application can be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so that the application will be described without obscuring it with unnecessary detail.
[0033] The terms used in the following embodiments are merely intended to describe certain embodiments and are not intended to limit this application. As used in this specification and the appended claims of this application, the singular terms "one," "a," "the," "the aforementioned," "this," and "the one" are intended to include the plural terms such as "one or more," unless the context clearly dictates otherwise. In the embodiments of this application, it should be further understood that "one or more" means one, two, or more. Furthermore, "and / or" describes an association relationship between associated objects and indicates that three relationships may exist. For example, A and / or B may indicate that only A is present, both A and B are present, and only B is present, and A and B may be singular or plural. The symbol " / " generally indicates an "or" relationship between associated objects.
[0034] The interface layout method provided in the embodiments of this application may be applied to terminal devices such as mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR) devices / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs). The specific types of terminal devices are not limited in the embodiments of this application.
[0035] For example, the terminal device may be a station (STATION, ST) in a WLAN, a mobile phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, an in-vehicle device, an Internet of Vehicle terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, or a satellite radio device.
[0036] By way of example and not of limitation, if the terminal device is a wearable device, the wearable device may instead be a collective term for wearable devices such as glasses, gloves, watches, clothes, and shoes that are developed based on intelligent design to be worn daily by using wearable technology. A wearable device is a portable device that can be worn directly by a user or can be incorporated into a user's clothes or accessories. A wearable device is not only a hardware device, but also implements powerful functions through software support, data exchange, and cloud interaction. Typical wearable intelligent devices include full-featured large devices that can implement full or partial functions without relying on a smartphone, such as smart watches and smart glasses, and devices that focus only on one type of application function and need to cooperate with other devices such as smartphones or various smart bands and smart jewelry for monitoring physical signs.
[0037] 1 is a diagram of a system architecture of an interface layout system related to an interface layout method according to an embodiment of this application. As shown in FIG. 1, the interface layout system may include a first terminal device 101 and at least one second terminal device 102, and the first terminal device may be connected to each second terminal device.
[0038] The first terminal device may be a terminal device that is convenient for the user to perform an input operation, and the second terminal device may be a terminal device that is commonly used by the user but is inconvenient for the user to perform an input operation. For example, the first terminal device may be a mobile phone or a tablet computer, and the second terminal device may be a television, a sound box, a headset, an in-car device, etc., and the input operation performed by the user may include inputting text information and a tap operation triggered on each interface element in the interface. The tap operation may be a single tap operation, a double tap operation, or other forms of operation.
[0039] The first terminal device can load a different application and display a first interface corresponding to the application on the screen of the first terminal device. If the first terminal device detects a screen projection command triggered by a user, it indicates that the user expects to project the first interface to the second terminal device and expects to display the interface executed by the application through the second terminal device. In this case, the first terminal device can obtain interface information of the first interface and second device information of the second terminal device, and generate a rearranged second interface based on the interface information and the second device information. Then, the first terminal device can send the rearranged second interface to the second terminal device, and the second terminal device can display the rearranged second interface.
[0040] The interface information of the first interface may include element information of interface elements in the first interface that may be displayed on the second terminal device. For example, the element information may include a position of the interface element in the first interface, an element type to which the interface element belongs, a name of the interface element, etc. In addition, the second device information may include information such as a screen size, a screen orientation, and a screen resolution of the second terminal device. For example, the second device information may indicate that the resolution of the second terminal device is 2244*1080 and the screen is in landscape mode.
[0041] In addition, in the process of generating a rearranged second interface based on the interface information and the second device information, the first terminal device can analyze the pre-processed interface information by using a pre-trained interface recognition model to determine the type of interface, and then the first terminal device can arrange each interface element in the interface information based on the interface type, the screen size and screen orientation of the second terminal device indicated by the second device information, and the screen of the second terminal device to obtain the rearranged second interface.
[0042] It should be noted that in practical application, the first terminal device can perform interface layout for one first interface, or can perform interface layout for multiple first interfaces simultaneously. Correspondingly, each first interface can correspond to one interface category. When there are multiple first interfaces, each first interface can correspond to one interface category. In the embodiment of this application, one first interface and one interface category are merely used as an example for explanation, and the quantity of the first interfaces and the quantity of the interface categories are not limited.
[0043] Furthermore, the embodiments of the present application mainly relate to the field of artificial intelligence (AI) recognition, and in particular to the field of machine learning and / or neural network technology. For example, the interface recognition model in the embodiments of the present application is obtained by training by using AI recognition and machine learning technology.
[0044] In the following, the description is given by using an example in which the first terminal device is a mobile phone. Figure 2 is a schematic diagram of the structure of a mobile phone 200 according to one embodiment of this application.
[0045] The mobile phone 200 may include a processor 210, an external memory interface 220, an internal memory 221, a USB port 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 251, a wireless communication module 252, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, a SIM card interface 295, etc. The sensor module 280 may include a gyroscope sensor 280A, an acceleration sensor 280B, an optical proximity sensor 280G, a fingerprint sensor 280H, and a touch sensor 280K (of course, the mobile phone 200 may further include other sensors such as a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, an air pressure sensor, and a bone conduction sensor, which are not shown in the figure).
[0046] It may be understood that the structure shown in this embodiment of the invention does not constitute a specific limitation of the mobile phone 200. In some other embodiments of this application, the mobile phone 200 may include more or fewer components than those shown in the figures, some components may be combined, some components may be divided, or different component configurations may be used. The components shown in the figures may be implemented using hardware, software, or a combination of software and hardware.
[0047] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent components or may be integrated into one or more processors. The controller may be the brain and command center of the mobile phone 200. The controller may generate operation control signals based on the instruction operation code and the time series signal to control instruction reading and execution.
[0048] The memory may further be disposed in the processor 210 and is configured to store instructions and data. In some embodiments, the memory of the processor 210 is a cache memory. The memory may store instructions or data that have just been used or are used periodically by the processor 210. When the processor 210 needs to use the instructions or data again, the processor 210 may directly call the instructions or data from the memory. This avoids repeated access and reduces the latency of the processor 210. Thus, the system efficiency is improved. For example, the memory may store interface attributes of the first terminal device, such as the interface size and interface direction of the first interface.
[0049] The processor 210 can execute the interface layout method provided in the embodiment of this application to improve the convenience of controlling the second interface through the second terminal device by the user, and improve the consistency between the control operations performed by the user on different terminal devices. The processor 210 may include different components. For example, when the CPU and the GPU are integrated, the CPU and the GPU can cooperate to execute the interface layout method provided in the embodiment of this application. For example, in the interface layout method, some algorithms are executed by the CPU and other algorithms are executed by the GPU to obtain higher processing efficiency. For example, the CPU can obtain interface information of the currently displayed first interface and device information of the screen projection terminal device according to the received screen projection command, and the GPU can generate a second interface suitable for the screen projection terminal device based on the interface information and the device information.
[0050] The display 294 is configured to display images, videos, and the like. The display 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), and the like. In some embodiments, the mobile phone 200 may include one or N displays 294, where N is a positive integer greater than 1. The display 294 may be configured to display information entered by or provided to a user, and various graphical user interfaces (GUIs). For example, the display 294 may display photos, videos, web pages, files, and the like. In another example, the display 294 may display a graphical user interface. The graphical user interface may include a status bar, a hideable navigation bar, time and weather widgets, and application icons, such as a browser icon. The status bar includes an operator name (e.g., China Mobile), a mobile network (e.g., 4G), time, and battery level. The navigation bar includes a back button icon, a home button icon, and a forward button icon. It may further be understood that in some embodiments, the status bar may further include a Bluetooth icon, a Wi-Fi icon, an external connection device icon, and the like.It may be further understood that in some other embodiments, the graphical user interface may further include a dock bar, and the dock bar may include icons of frequently used applications, etc. After the processor 210 detects a user's touch event on an application icon by using a finger (such as a stylus), in response to the touch event, the processor 210 initiates a user interface of the application corresponding to the application icon and displays the user interface of the application on the display 294.
[0051] In this embodiment of the application, the display 294 may be one integrated flexible display, or may be a splice display including two rigid screens and one flexible screen located between the two rigid screens. After the processor 210 executes the interface layout method provided in the embodiment of the application, the processor 210 may control the GPU to generate a second interface to be displayed on the second terminal device.
[0052] The camera 293 (which may be a front-facing camera, a rear-facing camera, or a camera that can function as both a front-facing camera and a rear-facing camera) is configured to capture still images or videos. Typically, the camera 293 may include a lens group and a light-sensitive element such as an image sensor. The lens group includes a number of lenses (convex or concave lenses) and is configured to collect light signals reflected by the object to be photographed and transfer the collected light signals to the image sensor. The image sensor generates an original image of the object based on the light signals.
[0053] The internal memory 221 may be configured to store computer executable program code. The executable program code includes instructions. The processor 210 executes the instructions stored in the internal memory 221 to perform various functional applications and data processing of the mobile phone 200. The internal memory 221 may include a program storage area and a data storage area. The program storage area can store an operating system, codes of applications (e.g., a camera application or a WeChat application), etc. The data storage area can store data created during the use of the mobile phone 200 (e.g., images or videos collected by a camera application), etc.
[0054] The internal memory 221 may further store one or more computer programs corresponding to the interface layout method provided in the embodiment of this application. The one or more computer programs are stored in the memory 221 and configured to be executed by one or more processors 210. The one or more computer programs include instructions, which may be used to execute steps of the corresponding embodiments of Figures 4 to 18. The computer programs may include a receiving module and a generating module. The receiving module is configured to receive a screen projection command, and the screen projection command is used to instruct the first terminal device to perform screen projection to the second terminal device. The generating module is configured to generate a second interface to be displayed on the second terminal device based on the interface information of the first interface and the second device information, where the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate the screen size and screen status of the second terminal device.
[0055] In addition, the internal memory 221 may include high-speed random access memory or may include at least one magnetic disk storage device, flash memory, universal flash storage (UFS), or other non-volatile memory.
[0056] Of course, the code corresponding to the interface layout method provided in the embodiment of this application may alternatively be stored in an external memory. In this case, the processor 210 can execute the code stored in the external memory corresponding to the interface layout method through the external memory interface 220, and the processor 210 can control the GPU to generate a second interface to be displayed on the second terminal device.
[0057] The function of the sensor module 280 will now be described.
[0058] The gyroscope sensor 280A may be configured to determine a motion attitude of the mobile phone 200. In some embodiments, the gyroscope sensor 280A may be used to determine the angular velocity of the mobile phone 200 about three axes (i.e., the x-axis, the y-axis, and the z-axis). In other words, the gyroscope sensor 280A may be configured to detect the current motion state of the mobile phone 200, for example, a vibration state or a stationary state.
[0059] If the display in this embodiment of the application is a folding screen, the gyroscope sensor 280A may be configured to detect a folding or unfolding action performed on the display 294. The gyroscope sensor 280A may report the detected folding or unfolding action as an event to the processor 210 to determine whether the display 294 is in a folded or unfolded state.
[0060] The acceleration sensor 280B can detect the magnitude of acceleration in various directions (usually on three axes) of the mobile phone 200. In other words, the gyroscope sensor 280A may be configured to detect the current operating state of the mobile phone 200, for example, a vibrating state or a stationary state. If the display in this embodiment of the application is a folding screen, the acceleration sensor 280B may be configured to detect a folding or unfolding operation performed on the display 294. The acceleration sensor 280B can report the detected folding or unfolding operation as an event to the processor 210 to determine whether the display 294 is in a folded or unfolded state.
[0061] The optical proximity sensor 280G may include, for example, a light emitting diode (LED) and a photodetector, for example, a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone emits infrared light by using the light emitting diode. The mobile phone detects infrared reflected light from a nearby object by using the photodiode. When sufficient reflected light is detected, the mobile phone may determine that there is an object near the mobile phone. When insufficient reflected light is detected, the mobile phone may determine that there is no object near the mobile phone. When the display in this embodiment of the application is a folding screen, the optical proximity sensor 280G may be disposed on the first screen of the folding display 294, and the optical proximity sensor 280G may detect the magnitude of the angle between the first screen and the second screen in the folded or unfolded state based on the optical path difference between the infrared signals.
[0062] The gyroscope sensor 280A (or the acceleration sensor 280B) can transmit the detected motion state information (e.g., angular velocity) to the processor 210. The processor 210 determines whether the mobile phone is currently in a handheld state or a tripod state based on the motion state information (e.g., if the angular velocity is not 0, it indicates that the mobile phone 200 is in a handheld state).
[0063] The fingerprint sensor 280H is configured to capture a fingerprint. The mobile phone 200 can use the captured fingerprint characteristics to perform fingerprint unlocking, application lock access, fingerprint-based photographing, fingerprint-based phone answering, etc.
[0064] The touch sensor 280K is also referred to as a "touch panel." The touch sensor 280K may be disposed on the display 294. The touch sensor 280K and the display 294 constitute a touch screen, also referred to as a "touch screen." The touch sensor 280K is configured to detect touch operations performed on or near the touch sensor 280K. The touch sensor can communicate the detected touch operations to an application processor to determine a type of touch event. The display 294 can provide a visual output associated with the touch operation. In some other embodiments, the touch sensor 280K may alternatively be disposed on the surface of the mobile phone 200 at a location different from the location of the display 294.
[0065] For example, the display 294 of the mobile phone 200 displays a home screen, and the home screen includes icons of multiple applications (e.g., a camera application and a WeChat application). A user taps the icon of the camera application on the home screen via the touch sensor 280K, triggering the processor 210 to launch the camera application and turn on the camera 293. The display 294 displays an interface of the camera application, for example, an interface of a viewfinder.
[0066] The wireless communication functions of the mobile phone 200 may be implemented via antenna 1, antenna 2, mobile communication module 251, wireless communication module 252, a modem processor, a baseband processor, and the like.
[0067] Antenna 1 and Antenna 2 are each configured to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 200 may be configured to cover one or more communication bands. Different antennas may be further multiplexed to improve antenna utilization. For example, Antenna 1 may be multiplexed as a diversity antenna in a wireless local area network. In some other embodiments, the antennas may be used in combination with a tuning switch.
[0068] The mobile communication module 251 may provide wireless communication solutions including 2G, 3G, 4G, 5G, etc. and applied to the mobile phone 200. The mobile communication module 251 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 251 may receive electromagnetic waves via the antenna 1, process the received electromagnetic waves such as filtering and amplifying, and send the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 251 may further amplify the signal modulated by the modem processor and convert the signal to electromagnetic waves for emission via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 251 may be located within the processor 210. In some embodiments, at least some functional modules of the mobile communication module 251 may be located within the same device as at least some modules of the processor 210. In this embodiment of the application, the mobile communication module 251 may be further configured to exchange information with another terminal device, for example, to send an audio output request to another terminal device, or the mobile communication module 251 may be configured to receive an audio output request and encapsulate the received audio output request into a message in a specified format.
[0069] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium or high-frequency signal. The demodulator is configured to demodulate a received electromagnetic signal into a low-frequency baseband signal. The demodulator then transmits the low-frequency baseband signal obtained by demodulation to the baseband processor for processing. The baseband processor processes the low-frequency baseband signal and then transfers the resulting signal to the application processor. The application processor outputs a sound signal by using an audio device (such as but not limited to the speaker 270A or the receiver 270B) or displays an image or video by using the display 294. In some embodiments, the modem processor may be an independent component. In some other embodiments, the modem processor may be independent from the processor 210 and be located in the same device as the mobile communication module 251 or another functional module.
[0070] The wireless communication module 252 may provide wireless communication solutions including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc., and applied to the mobile phone 200. The wireless communication module 252 may be one or more components integrating at least one communication processing module. The wireless communication module 252 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and sends the processed signal to the processor 210. The wireless communication module 252 can further receive a transmission signal from the processor 210, perform frequency modulation and amplification on the signal, and convert the signal into an electromagnetic wave for radiation via the antenna 2. In this embodiment of the application, the wireless communication module 252 is configured to establish a connection to an audio output device and output an audio signal through the audio output device. Alternatively, the wireless communication module 252 may be configured to access an access point device and send a message corresponding to an audio output request to another terminal device or receive a message corresponding to an audio output request sent by another terminal device. Optionally, the wireless communication module 252 may be further configured to receive audio data from another terminal device.
[0071] The mobile phone 200 may also realize audio functions such as playing and recording music by using the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headset jack 270D, the application processor, etc. The mobile phone 200 may receive input from the button 290 and generate button signal inputs related to user settings and function control of the mobile phone 200. The mobile phone 200 may generate vibration prompts (e.g., incoming call vibration prompts) via the motor 291. The indicator 292 of the mobile phone 200 may be an indicator light, may be configured to indicate a charging status and a power change, or may be configured to indicate a message, a missed call, a notification, etc. The SIM card interface 295 in the mobile phone 200 is configured to connect to a SIM card. The SIM card may be inserted into or removed from the SIM card interface 295 to realize contact with or separation from the mobile phone 200.
[0072] It should be understood that in actual applications, the mobile phone 200 may include more or less components than those shown in FIG. 2. This is not limited in this embodiment of the application. The mobile phone 200 shown in the figure is merely an example, and the mobile phone 200 may have more or less components than those shown in the figure, two or more components may be combined, or different component forms may be used. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0073] The software system of the terminal device can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present invention, the Android system having a layered architecture is used as an example to describe the software structure of the terminal device. Figure 3 is a block diagram of the software structure of the terminal device according to an embodiment of the present invention.
[0074] In a layered architecture, software is divided into several layers, each with a clear role and task. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer.
[0075] The application layer can include a series of application packages.
[0076] As shown in FIG. 3, the application package may include applications such as phone, camera, gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, message, and projection.
[0077] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predetermined functions.
[0078] As shown in FIG. 3, the application framework layer can include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.
[0079] The window manager is configured to manage window programs. The window manager may obtain a size of a display, determine whether there is a status bar, perform a screen lock, take a screen shot, etc. For example, the window manager may obtain interface attributes of the first interface, such as an interface size and an interface orientation of the first interface.
[0080] Content providers are configured to store and retrieve data and make the data accessible by applications. Data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0081] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system may be configured to build applications. A display interface may include one or more views. For example, a display interface that includes a message notification icon may include a text display view and an image display view.
[0082] The Telephone Manager is configured to manage the communication functions of the terminal device, for example, the call status (including answer, reject, etc.).
[0083] The resource manager provides various resources for an application, such as localized strings, icons, images, layout files, and video files.
[0084] The notification manager may be configured to allow applications to display notification information in the status bar and to convey notification messages. The notification manager may disappear automatically after a short pause without user interaction. For example, the notification manager may be configured to provide download completion notifications, message notifications, etc. Alternatively, the notification manager may be notifications that appear in the form of graphs or scrollbar text in the status bar at the top of the system, notifications of applications running in the background, or notifications that appear on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt tone is generated, the terminal device vibrates, or an indicator light flashes.
[0085] The Android runtime includes the kernel library and the virtual machine, and is responsible for scheduling and managing the Android system.
[0086] The kernel library includes two parts: the functions that need to be called in the Java language and the Android kernel library.
[0087] The application layer and the application framework layer are run on a virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is configured to implement functions such as object life cycle management, stack management, thread management, security and exception management, garbage collection, etc.
[0088] The system library may include multiple functional modules, for example, a surface manager, media libraries, a 3D graphics processing library (eg, OpenGL ES), and a 2D graphics engine (eg, SGL).
[0089] The surface manager is configured to manage the display subsystem and provide a blend of 2D and 3D layers for multiple applications.
[0090] The media library supports the playback and recording of multiple commonly used audio and video formats, still image files, etc. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.
[0091] The 3D graphics processing library is configured to perform 3D graphics drawing, image rendering, image compositing, layer processing, and the like.
[0092] The 2D graphics engine is a drawing engine for 2D drawing.
[0093] The kernel layer is a layer between the hardware and the software, and includes at least a display driver, a camera driver, an audio driver, and a sensor driver.
[0094] 4 is a schematic flowchart of an interface layout method according to an embodiment of this application. By way of example and not limitation, the method can be applied to the above-mentioned first terminal device. As shown in FIG. 4, the method includes the following steps:
[0095] Step 401: A screen projection command is received.
[0096] The screen projection command is used to instruct the first terminal device to perform screen projection to the second terminal device. For example, the screen projection command may include a second device identifier used to indicate the second terminal device. In this case, the first terminal device may determine to perform screen projection to the second terminal device based on the second device identifier.
[0097] In the process of loading the application, the first terminal device can display the interface of the application. If the network in which the first terminal device is located also includes another terminal device, for example, a second terminal device, the first terminal device can detect a screen projection command triggered by a user. If the first terminal device detects a screen projection command triggered to perform screen projection to the second terminal device, the first terminal device can receive the screen projection command, so that the first terminal device can generate a second interface compatible with the second terminal device in a subsequent step.
[0098] For example, the first terminal device may be a mobile phone, and the second terminal device may be a television. The first terminal device may load a fitness application, and the interface displayed by the first terminal device may be a fitness video. However, when the user is keeping fit, it is inconvenient for the user to hold the mobile phone, and therefore the screen of the mobile phone is relatively small. In this case, the first terminal device can detect a screen projection command triggered by the user, and the screen projection command instructs the first terminal device to project the interface of the fitness application onto the television, so that the user can conveniently watch the fitness video via the television.
[0099] Step 402: Obtain interface information of a first interface and device information of a second interface.
[0100] After the first terminal device receives the screen projection command, it indicates that the user expects to project the interface displayed by the first terminal device to the second terminal device and expects to display the interface displayed by the first terminal device via the second terminal device, so that the user can conveniently control the projected interface via the second terminal device. The first terminal device can obtain interface information of the first interface and second device information, so that in a subsequent step, the first terminal device can generate a second interface that is compatible with the second terminal device and should be displayed on the second terminal device based on the interface information of the first interface and the second device information.
[0101] Specifically, for different types of second terminal devices, the user needs to control the second terminal device by using different actions. Based on this, in the process of performing screen projection to different second terminal devices, the first terminal device can adjust the first interface displayed by the first terminal device to obtain a second interface respectively compatible with the second terminal device.
[0102] The first interface is an interface displayed on the first terminal device. The interface information may include interface attributes and element information of at least one interface element in the first interface. The interface attributes are used to indicate an interface size and an interface direction of the first interface. The element information of the interface element is used to indicate a name and a type of the interface element, as well as a position of the interface element in the first interface. For example, the first terminal device may recognize each interface element in the first interface in a preset element recognition manner, and determine a plurality of interface elements in the first interface and element information of each interface element.
[0103] 5 shows a first interface of a player displayed by a first terminal device. The first interface may include multiple interface elements such as a title 501, a cover 502, a seek bar 503, a repeat control 504, a pre control 505, a play 506 control, a next control 507, and a menu control 508.
[0104] In addition, the first terminal device may further obtain element information of each interface element. The element information of the interface elements may include: [{"label":0,"labelName":"title","uiRect":{"bottom":170,"left":168,"right":571,"top":102},"viewId":684},{"label":1,"labelName":"seek","uiRect":{"bottom":1992,"left":0,"right":1080,"top":1924},"viewId":670},{"label":2,"labelName":"repeat","uiRect":{"bottom":2167,"left":84,"right":204,"top":2047},"viewId":675},{"label":3,"labelName":"pre","uiRect":{"bottom":2167,"left":279,"right":399,"top":2047},"viewId":676},{"label":4,"labelName":"play","uiRect":{"bottom":2212,"left":435,"right":645,"top":2002},"viewId":677},{"label":5,"labelName":"next","uiRect":{"bottom":2167,"left":681,"right":801,"top":2047},"viewId":678},{"label":6,"labelName":"menu","uiRect":{"bottom":2167,"left":876,"right":996,"top":2047},"viewId":679},{"label":7,"labelName":"cover","uiRect":{"bottom":1255,"left":0,"right":1080,"top":451},"viewId":618}]. can be cited.
[0105] The label is used to represent an identifier of each interface element, which may be, for example, a sequence number of each interface element, the labelname is used to represent a name of each interface element, the uiRect is used to represent an area corresponding to each interface element in the first interface, and the viewID is a view identifier used to represent an identification information of an image corresponding to an interface element. Furthermore, the uiRect may include four parameters, namely bottom, top, left, and right, where the bottom is used to represent the bottom border of the interface element, the top is used to represent the top border of the interface element, the left is used to represent the left border of the interface element, and the right is used to represent the right border of the interface element. Also, each parameter in the element information may be in pixels. For example, the area corresponding to the song name has a top border of 102 pixels, a bottom border of 170 pixels, a left border of 168 pixels, and a right border of 571 pixels.
[0106] Note that the parameters shown in the element information of each interface element are merely examples, and the element information of the interface elements is not limited here.
[0107] It should be further noted that the interface elements recognized by the first terminal device are interface elements that can be displayed on the second terminal device. In the process of recognizing the interface elements, the first terminal device can first recognize all interface elements in the first interface, and then compare and match each recognized interface element with the acquired second device information according to a preset recommendation algorithm. If the first terminal device determines that the interface element can be displayed on the second terminal device, the first terminal device can extract the interface element to obtain element information of the interface element. If the first terminal device determines that the interface element cannot be displayed on the second terminal device, the first terminal device can ignore the interface element and not extract the interface element.
[0108] In addition, in the process of obtaining the second device information, the first terminal device can first request the second device information from the second terminal device based on the second device identifier carried in the screen projection command; after receiving the request sent by the first terminal device, the second terminal device can obtain the screen size and screen status of the second terminal device by extraction based on the preset configuration information, and feed back the second device information including the screen size and screen status to the first terminal device; and then the first terminal device completes the obtaining of the second device information.
[0109] For example, the second device information of the second terminal device may include (dst_width:2244,dst_height:1080,2), which indicates that the resolution of the second terminal device is 2244*1080 and the screen state of the second terminal device is landscape mode, as indicated by 2.
[0110] Step 403: Perform recognition based on element information of at least one interface element by using a pre-trained interface recognition model to determine an interface category.
[0111] After obtaining the interface information, the first terminal device can analyze element information in the interface information based on interface attributes included in the interface information by using the interface recognition model obtained by pre-training to determine an interface category corresponding to the first interface, so that the first terminal device can arrange each interface element based on the interface category in a subsequent step.
[0112] Different first terminal devices have different screen resolutions. Therefore, the first terminal device can pre-process the element information to reduce the amount of calculation of the first terminal device. Specifically, the first terminal device maps each interface element into a mapping area of a relatively small size, performs feature extraction in the mapping area to obtain interface feature data, and further determines an interface category according to the position of the interface element indicated by the interface feature data.
[0113] Optionally, the first terminal device may perform feature extraction on element information of the multiple interface elements based on the interface attributes, obtain interface feature data, input the interface feature data into an interface recognition model, recognize the interface feature data by using the interface recognition model, and obtain an interface category output from the interface recognition model.
[0114] In a possible implementation, to obtain the position of each interface element in the mapping area, the first terminal device can first obtain the position of each interface element based on the multiple element information, and perform calculations based on the interface attributes in the interface information by using a preset mapping formula. Then, the first terminal device can perform feature extraction in the mapping area based on whether there is an interface element at each position in the mapping area to obtain interface feature data indicating the position of the interface element. Then, the first terminal device can input the interface feature data into a pre-trained interface recognition model, analyze the interface feature data indicating the position of the interface element by using the interface recognition model, and finally recognize an interface category of the first interface based on the position of each interface element in the first interface.
[0115] For example, the mapping expression is
number
[0116] x=(xt,xb.xl,xr), ftop=top*dsth / src_height, fbot=bottom*dsth / src_height, fleft=left*dstw / src_width, fright=right*dstw / src_width, dsth represents the height of the mapping area, dstw represents the width of the mapping area, src_height represents the height of the first interface, and src_width represents the width of the first interface.
[0117] It should be noted that in practical applications, the interfaces of all applications can be classified into multiple interface types, and the quantity of interface types is not limited in this embodiment of this application. For example, eight interface categories may be pre-set. Figures 6 to 13 are schematic diagrams corresponding to each interface category.
[0118] FIG. 6 is a schematic diagram of interface category 1. Multiple interface elements in an interface may be located in the same layer, and the interface elements are not overlaid. For example, interface category 1 may be applied to a music playback interface. FIG. 7 is a schematic diagram of interface category 2. Multiple interface elements in an interface may be located in the same layer, but the interface elements are overlaid. For example, interface category 2 may be applied to a video playback interface. FIG. 8-a and FIG. 8-b are schematic diagrams of interface category 3 in portrait mode and landscape mode. Multiple interface elements in an interface may be located in the same layer, and the extension items in the interface can be overlaid. For example, interface category 3 may be applied to a music playback interface with a popup playlist or a video playback page with a popup episode, and the playlist and the video episode belong to the slideable part. FIG. 9-a and FIG. 9-b are schematic diagrams of interface category 4 in portrait mode. All interface elements in the interface are located in different layers, and an upward or downward slide operation or a slide operation in any direction can be performed in the view area in the interface. For example, interface category 4 may be applied to a page where multiple videos are displayed, such as a home page or a navigation page of a video application. Fig. 10 is a schematic diagram of interface category 5. Multiple interface elements in the interface may be located on different layers, an information bar is located at both the top and bottom of the interface, and a view area in the interface is slidable. For example, interface category 5 may be applied to a chat interface or an email interface of a social software. Fig. 11 is a schematic diagram of interface category 6. Multiple interface elements in the interface may be located on different layers, a bar is located at the top of the interface, and a view area in the interface is slidable.For example, interface category 6 may be applied to a home page of an email application or a search interface of a shopping application. FIG. 12 is a schematic diagram of interface category 7. Interface elements in the interface may be located in different layers, and the upper and lower parts in the interface are view areas, and the upper view area is fixed and the lower view area is slidable. For example, interface category 7 may be applied to a live streaming interface. FIG. 13 is a schematic diagram of interface category 8. Interface elements in the interface may be located in different layers, and the upper and lower parts are a bar, a photo, a tab bar, a view area, and a bar from top to bottom, and the view area may be slidable. For example, interface category 8 may be applied to a product details interface of a shopping application.
[0119] Step 404: Arrange at least one interface element based on the interface category and the second device information to obtain a second interface.
[0120] After determining the interface category, the first terminal device can arrange at least one interface element based on the determined interface category, second device information of the second terminal device, and the screen size and screen orientation of the second terminal device indicated by the second device information to obtain a second interface compatible with the second terminal device.
[0121] Optionally, the first terminal device may also divide the display area of the second terminal device based on interface categories to obtain multiple sub-areas, where the display areas are indicated by the second device information, and the first terminal device may determine interface elements arranged in each sub-area, and then the first terminal device may adjust each interface element in each sub-area based on the size of the display area indicated by the second device information and the quantity of interface elements arranged in each sub-area to obtain a second interface.
[0122] In a possible implementation, the first terminal device may determine interface elements that may be arranged in each subarea based on the multiple subareas obtained by division. Then, for all interface elements in all subareas, the first terminal device may adjust the size, position, and orientation of each interface element in each subarea based on the size of the display area and the quantity of interface elements that may be arranged in the subarea, the quantity of elements corresponding to the subarea, and the importance of each interface element to obtain a second interface.
[0123] Furthermore, in the process of adjusting the size, position and orientation of the interface elements, the first terminal device can first collect statistics on the interface elements arranged in each subarea to determine the quantity of the interface elements in each subarea, and adjust the size and orientation of each interface element in the subarea based on the size of the display area, the preset arrangement rule and the quantity of elements corresponding to the subarea to obtain the adjusted interface element, so that the adjusted interface element is better suited to the second terminal device. Finally, the first terminal device can adjust the position of the adjusted interface element in the subarea based on the quantity of elements corresponding to the subarea in each subarea to obtain the second interface.
[0124] In addition, in the process of adjusting the adjusted interface elements, the first terminal device can further obtain the importance of each adjusted interface element, and arrange the adjusted interface element whose importance parameter has a maximum value in the central area of the sub-area based on the importance of each adjusted interface element.
[0125] The first terminal device can perform multiple adjustment operations on the interface element, such as scaling, rotation, and translation, etc. The adjustment operations are not limited in this embodiment of this application.
[0126] For example, as shown in FIG. 5, when the interface category shown in FIG. 5 is category 1, the display area of the second terminal device may be divided into three subareas, namely, upper, middle, and lower subareas. The upper subarea occupies 17% of the display area, the middle subarea occupies 50% of the display area, and the lower subarea occupies 33% of the display area. The song title and / or singer name may be arranged in the upper subarea, the cover and / or lyrics may be arranged in the middle subarea, and multiple interface elements including playback control, menu control, previous control, next control, repeat playback control, and a seek bar may be arranged in the lower subarea, i.e., the control area. The interface elements other than the seek bar may all be arranged below the seek bar, or may be arranged above and below the seek bar based on the quantity of interface elements in the lower subarea.
[0127] For example, if the amount of interface elements in the lower sub-area is less than the element threshold, the interface elements may be equally spaced below the seek bar, and if the amount of interface elements in the lower sub-area is equal to or greater than the element threshold, the interface elements may be separately spaced above and below the seek bar.
[0128] Assume that the preset element threshold is 6, and the amount of interface elements other than the seek bar in the lower sub-area shown in FIG. 5 is 5. In this case, the amount of interface elements is less than the element threshold, and the interface elements other than the seek bar may be arranged at equal intervals below the seek bar. Also, in the arrangement process, the most important playback control may be arranged in the center. Then, the second most important previous and next controls may be arranged respectively to the left and right of the playback control, and finally, the repeat playback control may be arranged at the far left and the menu control at the far right.
[0129] It should be noted that the size of the area occupied by each sub-area in the display area may be set according to a preset arrangement rule, and the element threshold for each sub-area may be obtained through learning the user's usage habits. Similarly, the importance of each interface element may also be obtained based on how frequently the user triggers the interface element. For example, a higher trigger frequency indicates a higher importance of the interface element. The method of determining the size of the area occupied by each sub-area in the display area, the element threshold for each sub-area, and the importance of each interface element is not limited in this embodiment of this application.
[0130] In addition, in practical applications, there may be multiple types of second terminal devices, and the interface layouts of the terminal devices are different.
[0131] For example, as shown in FIG. 14A and FIG. 14B, a non-overlay layout is used as an example. A top-middle-bottom layout can be used for a television, a notebook computer, and a tablet computer. A left-right layout may be used for a vehicle-mounted terminal device. A layer-distinguishing layout can be used for a clock. For example, a view area is arranged in the bottom layer, and a top-bottom floating layout is used. As shown in FIG. 15A and FIG. 15B, an overlay layout is used as an example. For a television, a notebook computer, a tablet computer, a vehicle-mounted terminal device, or a clock, a view area is arranged in the bottom layer, and a top-bottom floating layout is arranged in the top layer. For example, an overlay layout is used for a map application loaded by a vehicle-mounted terminal device. Furthermore, as shown in FIG. 16A and FIG. 16B, an overlay scroll layout is used as an example. A top-bottom layout method may be used for a television, and a left-right layout method may be used for a notebook computer, a tablet computer, and a vehicle-mounted terminal device.
[0132] Step 405: Send the second interface to the second terminal device, so that the second terminal device displays the second interface.
[0133] After generating the second interface, the first terminal device can transmit the second interface to the second terminal device so that the second terminal device can display the second interface and present the second interface to the user that is compatible with the screen of the second terminal device.
[0134] In practical application, step 403 and step 404 may be performed by a first terminal device, i.e., the first terminal device may arrange interface elements based on an interface category to obtain a second interface, or step 403 and step 404 may be performed by a second terminal device, i.e., the second terminal device may receive the interface category and interface elements sent by the first terminal device and arrange the interface elements based on the interface category and second device information to generate and display the second interface. The process of the second terminal device generating the second interface is similar to the process of step 403. Details are not repeated here.
[0135] Step 406: Update the interface recognition model based on the obtained feedback information.
[0136] After the first terminal device transmits the second interface to the second terminal device so that the second terminal device displays the second interface, the first terminal device may detect an action triggered by a user and obtain feedback information input by the user about the second interface, so that the first terminal device may update an interface recognition model based on the obtained feedback information.
[0137] In a possible implementation, after generating the second interface, the first terminal device can first display a feedback interface to the user and detect an input operation triggered by the user. If an input operation is detected, the first terminal device can obtain feedback information input by the user. After the feedback information is recorded, if the currently recorded feedback information and the previously recorded feedback information meet a preset update condition, the first terminal device can update the interface recognition model based on the recorded feedback information.
[0138] Furthermore, in the process of determining whether the feedback information satisfies the update condition, the first terminal device can obtain an amount of feedback time in the recorded plurality of feedback information, and compare the amount of feedback time with a preset feedback threshold. If the feedback time is equal to or greater than the feedback threshold, the first terminal device can update an interface recognition model based on the recorded plurality of feedback information, and more accurately determine an interface category by using the updated interface recognition model.
[0139] It should be noted that the interface layout method provided in this embodiment of the present application can be applied not only to an interface projection scenario, but also to an interface development scenario. Correspondingly, when the interface layout method is applied to an interface development scenario, manual interface element extraction may be performed in the first interface before step 401.
[0140] Optionally, the first terminal device may perform interface element extraction in the first interface based on an extraction operation triggered by the user to obtain a plurality of interface elements, and then generate element information for the plurality of interface elements based on a supplementary operation triggered by the user, such that the first terminal device can perform interface layout in a subsequent step based on the generated element information.
[0141] For example, the first terminal device may load an Integrated Development Environment (IDE) and input an image corresponding to the first interface and an interface attribute of the first interface into the IDE based on an input operation triggered by a user, i.e., input a first interface image and a resolution corresponding to the first interface into the IDE. Then, as shown in Fig. 17, the first terminal device may detect a box selection action triggered by a user on interface elements, and select multiple interface elements in the first interface by using a box (shown as a dashed box in Fig. 17) based on the box selection action to obtain multiple interface elements.
[0142] The area occupied by each interface element can be determined based on a box used to select the interface element, for example, coordinates corresponding to the four edges of the box can be determined as the corresponding top, bottom, left, and right coordinates of the interface element in the first interface based on the four edges of the box.
[0143] In addition, the first terminal device can further remind the user to supplement each interface element and generate element information for each interface element based on a preset table. For example, the first terminal device can obtain multiple pieces of information, such as the name and element type of each interface element, based on an input operation triggered by the user, generate element information for the interface elements, and generate a whole element list based on the element information for the multiple interface elements.
[0144] In addition, the first terminal device may further acquire second device information of the second terminal device input by the user based on the operation triggered by the user. For example, the second device information may include the name, screen resolution, and landscape / portrait mode of the second terminal device.
[0145] After obtaining the interface elements, the first terminal device may perform operations similar to steps 402 and 403 to generate a second interface, and then may detect an adjustment action triggered by a user to adjust the size and position of each interface element in the second interface, and record the adjustment action triggered by the user. In this way, the first terminal device can adjust the pre-set arrangement rule based on the recorded adjustment action. Specifically, the first terminal device can record the adjustment action triggered by the user on at least one interface element in the second interface, and adjust the arrangement rule based on the adjustment action.
[0146] For example, FIG. 18 shows an IDE interface displayed on a first terminal device. As shown in FIG. 18, the left side shows the first interface in which an interface element is selected by using a box, the upper part of the right side records attribute information of each interface element, such as name, location, type, etc., the middle part of the right side shows the name of the first terminal device "mobile phone", the name of the second terminal device "television", the screen resolution of the first terminal device "720*1080", the screen resolution of the second terminal device "2244*1080", the landscape / portrait mode of the first terminal device "1" (indicating a portrait screen), and the landscape / portrait mode of the second terminal device "2" (indicating landscape), and the lower part of the right side shows the generated second interface. The first terminal device can further adjust each interface element in the second interface based on the adjustment operation triggered by the user.
[0147] In conclusion, according to the interface layout method provided in this embodiment of the application, a first terminal device receives a projection command instructing the first terminal device to perform screen projection to a second terminal device, and generates a second interface to be displayed on the second terminal device based on the second device information and the interface information of the first interface displayed on the first terminal device, and the second device information is used to indicate the screen size and screen status of the second terminal device. In this way, the second terminal device can display a second interface compatible with the second terminal device, and the user can conveniently control the second interface through the second terminal device. This avoids the problem that the user cannot conveniently control the screen projection interface, improves the convenience of the user to control the second interface through the second terminal device, and improves the consistency between the control operations performed by the user on different terminal devices.
[0148] In addition, for a user performing interface development, a rearranged second interface can be provided to the user, and each interface element in the second interface is rearranged based on an operation triggered by the user so that the user can obtain the second interface without manual operation, thereby reducing the time the user spends on interface development and improving the user's interface development efficiency.
[0149] In addition, in the process of determining an interface category using an interface recognition model, first, feature extraction is performed on element information of an interface element to obtain interface feature data, and by using the interface feature data, the amount of calculation required to determine an interface category can be reduced, thereby improving the efficiency of determining an interface category.
[0150] Furthermore, feedback information is obtained and the interface recognition model is updated based on the feedback information, thereby improving the accuracy of recognizing the interface type using the interface recognition model.
[0151] Finally, all interface elements in the first interface are filtered and rearranged to extract only interface elements that can be displayed on the second terminal device, and the extracted interface elements are arranged based on the screen size and screen orientation of the second terminal device so that the generated second interface is better suited to the second terminal device, thereby improving the aesthetic visibility of the second interface.
[0152] It should be understood that the sequence numbers of steps do not refer to the execution order in the above embodiments. The execution order of the processes should be determined based on the functions and internal logic of the processes, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0153] Corresponding to the interface layout method in the above embodiment, Figure 19 is a structural block diagram of an interface layout device according to an embodiment of this application. For ease of explanation, only parts related to the embodiment of this application are shown in the figure.
[0154] As shown in FIG. 19, the device A receiving module 1901 configured to receive a screen projection command, the screen projection command being used to instruct the first terminal device to perform screen projection to a second terminal device; A generating module 1902 configured to generate a second interface to be displayed on a second terminal device based on interface information of a first interface and second device information, where the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; Includes.
[0155] Optionally, the generation module 1902 is specifically configured to obtain interface information of a first interface and second device information, the interface information of the first interface including element information of at least one interface element in the first interface, the element information being used to indicate a name and a type of the interface element and a position of the interface element in the first interface, perform recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category, and arrange the at least one interface element based on the interface category and the second device information to obtain a second interface.
[0156] Optionally, the interface information of the first interface further includes interface attributes, where the interface attributes are used to indicate an interface size and an interface direction of the first interface.
[0157] The generation module 1902 is further specifically configured to recognize the interface feature data by performing feature extraction on at least one element information based on the interface attributes to obtain interface feature data, inputting the interface feature data into an interface recognition model, and using the interface recognition model to obtain an interface category output from the interface recognition model.
[0158] Optionally, the generation module 1902 is further configured to divide the display area of the second terminal device based on an interface category to obtain a plurality of sub-areas, the display area being indicated by the second device information, determine interface elements to be placed in each sub-area, and adjust each interface element in each sub-area based on the size of the display area indicated by the second device information and the quantity of interface elements to be placed in each sub-area to obtain a second interface.
[0159] Optionally, the generation module 1902 is further configured to determine a quantity of interface elements in each sub-area, adjust a size and an orientation of each interface element in each sub-area based on the size of the display area, the preset arrangement rule, and the quantity of elements corresponding to the sub-area to obtain adjusted interface elements, and adjust a position of the adjusted interface elements in the sub-area based on the quantity of elements corresponding to the sub-area, in each sub-area, to obtain a second interface.
[0160] Optionally, as shown in FIG. 20 , the apparatus further comprises: a transmitting module 1903 configured to transmit the second interface to a second terminal device such that the second terminal device displays the second interface; Further includes:
[0161] Optionally, as shown in FIG. 20 , the apparatus further comprises: an acquiring module 1904 configured to acquire feedback information, the feedback information being information fed back by a user regarding a second interface displayed on a second terminal device; an update module 1905 configured to update the interface recognition model based on the feedback information when the feedback information satisfies a preset update condition; Further includes:
[0162] Optionally, as shown in FIG. 20 , the apparatus further comprises: an extraction module 1906 configured to perform an interface element extraction in the first interface based on an extraction operation triggered by a user to obtain a plurality of interface elements; A supplementary module 1907 configured to generate element information for a plurality of interface elements based on a supplementary action triggered by a user; Further includes:
[0163] The device is a recording module 1908 configured to record the adjustment action triggered by the user to at least one interface element in the second interface; an adjustment module 1909 configured to adjust the placement rules based on the adjustment operation; Further includes:
[0164] In conclusion, according to the interface layout device provided in this embodiment of the application, a first terminal device receives a projection command instructing the first terminal device to perform screen projection to a second terminal device, and generates a second interface to be displayed on the second terminal device based on the second device information and the interface information of the first interface displayed on the first terminal device, and the second device information is used to indicate the screen size and screen state of the second terminal device. In this way, the second terminal device can display a second interface compatible with the second terminal device, and the user can conveniently control the second interface through the second terminal device. This avoids the problem that the user cannot conveniently control the screen projection interface, improves the convenience of the user to control the second interface through the second terminal device, and improves the consistency between the control operations performed by the user on different terminal devices.
[0165] An embodiment of the present application further provides a terminal device, which includes a memory, a processor, and a computer program that can be stored in the memory and executed by the processor, and when the computer program is executed, the processor performs any one of the steps of the embodiments of the interface layout method described above.
[0166] An embodiment of this application further provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, performs the steps of any one of the embodiments of the interface layout method described above.
[0167] Figure 21 is a schematic diagram of the structure of a terminal device according to an embodiment of this application. As shown in Figure 21, the terminal device 21 of this embodiment includes at least one processor 211 (only one processor is shown in Figure 21), a memory 212, and a computer program 212 that can be stored in the memory 212 and executed by the at least one processor 211. When executing the computer program 212, the processor 211 performs any one of the steps of the embodiments of the interface layout method described above.
[0168] The terminal device 21 may be a computing device such as a desktop computer, a notebook computer, a palmtop computer, or a cloud server. The terminal device may include, but is not limited to, a processor 211 and a memory 212. As will be appreciated by those skilled in the art, FIG. 21 is merely an example of the terminal device 21 and does not constitute a limitation on the terminal device 21. The terminal device may include more or less components than those shown in the figure, or some components may be combined, or different components may be used. For example, the terminal device may further include an input / output device, a network access device, etc.
[0169] Processor 211 may be a Central Processing Unit (CPU). Processor 211 may alternatively be another general-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or another programmable logic device, discrete gate or transistor logic device, or discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.
[0170] In some embodiments, the memory 212 may be an internal storage unit of the terminal device 21, such as a hard disk or memory of the terminal device 21. In some other embodiments, the memory 212 may instead be an external storage device of the terminal device 21, such as a removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., that is equipped in the terminal device 21. Furthermore, the memory 212 may instead include both an internal storage unit and an external storage device of the terminal device 21. The memory 212 is configured to store an operating system, applications, a boot loader, data, and another program, such as program code of a computer program. The memory 212 may further be configured to temporarily store output data or data to be output.
[0171] As will be clearly understood by those skilled in the art, for the sake of convenience and concise description, the division into the above-mentioned functional units or modules is only used as an example for description. In actual application, the above-mentioned functions may be allocated to different functional units or modules for implementation based on requirements. That is, the internal structure of the device is divided into different functional units or modules for implementing all or part of the above-mentioned functions. The functional units or modules in the embodiments may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit. Furthermore, the specific names of the functional units or modules are merely used for ease of distinction, and are not intended to limit the scope of protection of this application. For the specific operation processes of the units or modules in the above-mentioned system, please refer to the corresponding processes in the above-mentioned method embodiments. Details will not be repeated here.
[0172] The above embodiments are described from their respective focuses. For parts that are not described or recorded in detail in one embodiment, please refer to the relevant descriptions of other embodiments.
[0173] As can be recognized by those skilled in the art, the units and algorithm steps in the examples described in connection with the embodiments disclosed in this specification can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of this application.
[0174] In the embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the division into modules or units is merely a division of logical functions, and may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. Indirect couplings or communication connections between apparatuses or units may be implemented in electrical, mechanical, or other forms.
[0175] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions in the embodiments.
[0176] In addition, the functional units in the embodiments of this application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.
[0177] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, all or part of the steps of the method in the embodiments of this application may be implemented by a computer program instructing relevant hardware. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned method embodiments may be performed. The computer program includes a computer program code. The computer program code may be in source code form, object code form, executable file form, some intermediate form, etc. The computer-readable medium may include at least any entity or apparatus that can convey the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a removable hard disk, a magnetic disk, or an optical disk. In some jurisdictions, the computer-readable medium may not be an electrical carrier signal or a telecommunication signal according to the law and patent practice.
[0178] The above embodiments are merely intended to describe the technical solutions of this application, and are not intended to limit this application. Although this application is described in detail in conjunction with the above embodiments, those skilled in the art should understand that the above embodiments may further modify the technical solutions described in the above embodiments or make equivalent substitutions for some technical features of the above embodiments without departing from the spirit and scope of the technical solutions of the embodiments of this application. [Explanation of symbols]
[0179] 21 Terminal Devices 101 Terminal Device 102 Terminal Devices 200 Mobile Phones 210 Processor 211 Processor 212 Memory 220 External Memory Interface 221 Internal Memory 230 USB ports 240 Charging Management Module 241 Power Management Module 242 Battery 251 Mobile Communication Module 252 Wireless communication module 270 Audio Module 270A Speaker 270B Receiver 270C Microphone 270D Headset Jack 280 Sensor Module 280A Gyroscope Sensor 280B Acceleration Sensor 280G Optical Proximity Sensor 280H Fingerprint Sensor 280K touch sensor 290 Button 291 Motor 292 Indicator 293 Camera 294 Folding Display 295 SIM card interface 503 Bar 504 Playback Control 505 Control 507 Control 508 Control 1901 Receiver module 1902 Generation Module 1903 Transmitting Module 1904 Acquisition Module 1905 Update Module 1906 Extraction Module 1907 Supplementary Module 1908 Recording Module 1909 Adjustment Module
Claims
1. 1. An interface layout method applied to a first terminal device, the first terminal device being connected to a second terminal device, the method comprising: receiving a screen projection command, the screen projection command being used to instruct the first terminal device to perform screen projection to the second terminal device; generating a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, wherein the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; Including, The step of generating a second interface to be displayed on the second terminal device based on interface information of the first interface and second device information includes: obtaining the interface information of the first interface and the second device information, the interface information of the first interface including element information of at least one interface element in the first interface, the element information being used to indicate a name and type of the interface element and a position of the interface element in the first interface; performing recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category; arranging the at least one interface element based on the interface category and the second device information to obtain the second interface; An interface layout method comprising:
2. The interface information of the first interface further includes interface attributes, the interface attributes being used to indicate an interface size and an interface direction of the first interface; The step of performing recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category includes: performing feature extraction on at least one element information based on the interface attributes to obtain interface feature data; inputting the interface feature data into the interface recognition model and recognizing the interface feature data by using the interface recognition model to obtain the interface category output from the interface recognition model; The interface layout method of claim 1 , comprising:
3. The step of arranging the at least one interface element based on the interface category and the second device information to obtain the second interface includes: Dividing a display area of the second terminal device according to the interface category to obtain a plurality of sub-areas, the display areas being indicated by the second device information; determining interface elements to be placed in each sub-area; adjusting each interface element in each sub-area based on a size of the display area and a quantity of interface elements disposed in each sub-area indicated by the second device information to obtain the second interface; 3. The interface layout method according to claim 1 or 2, comprising:
4. The step of adjusting each interface element in each sub-area based on the size of the display area and the number of interface elements to be arranged in each sub-area indicated by the second device information to obtain the second interface includes: determining a quantity of interface elements in each sub-area; adjusting a size and an orientation of each interface element in each sub-area based on a size of the display area, a preset arrangement rule, and a quantity of interface elements in the sub-area to obtain adjusted interface elements; adjusting, in each subarea, a position of the adjusted interface element in the subarea based on a quantity of interface elements in the subarea to obtain the second interface; The interface layout method of claim 3, comprising:
5. After the step of generating a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, the method further comprises: transmitting the second interface to the second terminal device such that the second terminal device displays the second interface; The interface layout method of claim 1 , further comprising:
6. After the step of transmitting the second interface to the second terminal device, the method further comprises: obtaining feedback information, the feedback information being information fed back by a user regarding the second interface displayed on the second terminal device; and updating the interface recognition model based on the feedback information when the feedback information satisfies a preset update condition; The interface layout method of claim 5 further comprising:
7. Before the step of generating a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, the method further comprises: performing an interface element extraction in the first interface based on an extraction action triggered by a user to obtain a plurality of interface elements; generating element information for the plurality of interface elements based on a supplemental action triggered by the user; The interface layout method of claim 1 , further comprising:
8. After the step of generating a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, the method further comprises: recording a user-triggered adjustment action on at least one interface element in the second interface; adjusting the preset placement rules based on the adjustment action; The interface layout method of claim 1 , further comprising:
9. An interface layout apparatus applied to a first terminal device, the first terminal device being connected to a second terminal device, the interface layout apparatus comprising: A receiving module configured to receive a screen projection command, the screen projection command being used to instruct the first terminal device to perform screen projection to the second terminal device; A generating module configured to generate a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, the first interface being an interface displayed on the first terminal device, and the second device information being used to indicate a screen size and a screen state of the second terminal device; Equipped with The generating module includes: obtaining the interface information of the first interface and the second device information, the interface information of the first interface including element information of at least one interface element in the first interface, the element information being used to indicate a name and type of the interface element and a position of the interface element in the first interface; performing recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category; arranging the at least one interface element based on the interface category and the second device information to obtain the second interface; The interface layout device is particularly configured to:
10. An interface layout system comprising a first terminal device and a second terminal device, the first terminal device being connected to the second terminal device, the first terminal device receiving a screen projection command, the screen projection command being used to instruct the first terminal device to perform a screen projection to the second terminal device; The first terminal device generates a second interface to be displayed on the second terminal device based on interface information of a first interface and second device information, the first interface is an interface displayed on the first terminal device, and the second device information is used to indicate a screen size and a screen state of the second terminal device; the first terminal device transmitting the second interface to the second terminal device; the second terminal device receiving and displaying the second interface; The first terminal device generates a second interface to be displayed on the second terminal device based on interface information of the first interface and second device information; obtaining the interface information of the first interface and the second device information, the interface information of the first interface including element information of at least one interface element in the first interface, the element information being used to indicate a name and type of the interface element and a position of the interface element in the first interface; performing recognition based on the element information of the at least one interface element by using a pre-trained interface recognition model to determine an interface category; arranging the at least one interface element based on the interface category and the second device information to obtain the second interface; An interface layout system, including:
11. A terminal device comprising a memory, a processor and a computer program stored in the memory and executed by the processor, the processor implementing the method according to any one of claims 1 to 8 when executing the program.
12. 9. A computer readable storage medium having a computer program stored therein, the computer program causing, when executed by a processor, to perform the method according to any one of claims 1 to 8.
13. A chip comprising instructions which, when executed on a computer, enable the computer to carry out the method of any one of claims 1 to 8.
Citation Information
Patent Citations
Layout template processing method, device, and computer-readable storage medium
CN109508189A
Information processing system and gateway server and information terminal
JP2002063108A
Server and method for providing information
JP2003281030A
Information processor and control method
JP2015090570A
Information processing apparatus and control method
US20150128065A1
