A screen projection method, apparatus, storage medium, and electronic device

Abstract

This application discloses a screen projection method, apparatus, storage medium, and electronic device, relating to the field of image processing technology. The method includes: responding to a first terminal device projecting a screen onto a second terminal device; receiving target projection elements and user profile information of a target user corresponding to the first terminal device; the target projection elements being elements adapted to the second terminal device generated based on elements to be projected in the display interface of the first terminal device; generating a target display interface template based on the target projection elements and user profile information; and generating a target display interface adapted to the second terminal device according to the target display interface template, thereby completing the screen projection from the first terminal device to the second terminal device. Compared with existing technologies, the target display interface generated in this application is adapted to the second terminal device, improving the smoothness of screen projection and thus enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a screen projection method, apparatus, storage medium and electronic device. Background Technology

[0002] Screen projection (also known as screen mirroring) is a technology that allows a target device (such as a mobile phone, tablet, or computer) to display the content of the source device in real time on the target screen.

[0003] Currently, screen projection is mainly achieved by using a local area network protocol to send the display interface of the first terminal device to the second terminal device via a single video link, and then playing the display interface of the first terminal device on the screen of the second terminal device, thereby realizing the screen projection of the display interface of the first terminal device onto the second terminal device.

[0004] However, using this method, it is impossible to control the content projected on the screen using the control device of the second terminal device. Every time the user selects new content, they need to operate through the first terminal device and re-project the screen. This makes the operation process of operating the projected screen image cumbersome. Furthermore, the screen projection relies on the device performance of the first terminal device, resulting in low projection smoothness and affecting the user experience. Summary of the Invention

[0005] In view of this, this application provides a screen projection method, apparatus, storage medium, and electronic device. The main purpose is to improve the existing technology, which cannot use the control device of the second terminal device to control the content projected on the screen. Each time the user selects new content, they need to operate through the first terminal device and re-project the screen. This makes the operation process of operating the projected screen image cumbersome. In addition, the screen projection process depends on the device performance of the first terminal device, resulting in low projection smoothness and affecting the user experience.

[0006] In a first aspect, this application provides a screen projection method, including: In response to the first terminal device projecting the screen onto the second terminal device, the system receives the target projection element and the user profile information of the target user corresponding to the first terminal device. The target projection element is an element adapted to the second terminal device based on the element to be projected in the display interface of the first terminal device. A target display interface template is generated based on the target projection elements and the user profile information; Generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device.

[0007] Secondly, this application provides a screen projection device, comprising: The receiving module is configured to receive a target projection element and user profile information of the target user corresponding to the first terminal device in response to the first terminal device projecting the screen onto the second terminal device. The target projection element is an element adapted to the second terminal device based on the element to be projected in the display interface of the first terminal device. The generation module is configured to generate a target display interface template based on the target projection element and the user profile information; The generation module is further configured to generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device. In a third aspect, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the screen projection method described in the first aspect.

[0008] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the screen projection method described in the first aspect.

[0009] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the screen projection method described in the first aspect.

[0010] By means of the above technical solutions, this application provides a screen projection method, apparatus, storage medium, and electronic device, comprising: responding to a first terminal device projecting a screen onto a second terminal device, receiving target projection elements and user profile information of a target user corresponding to the first terminal device, wherein the target projection elements are elements adapted to the second terminal device based on elements to be projected in the display interface of the first terminal device; generating a target display interface template based on the target projection elements and user profile information; and generating a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection from the first terminal device to the second terminal device. Compared with existing technologies, this application, when using a first terminal device to project the screen onto a second terminal device, can generate a target display interface template adapted to the user and the second terminal device based on target projection elements and user profile information. Then, based on the target display interface template, the target display interface of the second terminal device is generated, thus completing the screen projection. This means that the target display interface generated by this application is regenerated within the target display interface template based on the target projection elements and user profile information, ensuring that the generated target display interface is adapted to the second terminal device. This allows the screen projection display to rely on the device performance of the second terminal device, enabling the user to directly control the target display interface from the second terminal device, improving the smoothness of screen projection and enhancing the user experience. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart of a screen projection method provided in an embodiment of this application is shown; Figure 2 A schematic flowchart of a screen projection method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating an example provided in an embodiment of this application is shown; Figure 4 A schematic diagram illustrating an example provided in an embodiment of this application is shown; Figure 5 An interactive schematic diagram of an example provided in an embodiment of this application is shown; Figure 6An interactive schematic diagram of an example provided in an embodiment of this application is shown; Figure 7 This paper shows a schematic diagram of the structure of a screen projection device provided in an embodiment of this application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0014] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0015] To address the technical problem that existing technologies cannot control the content projected onto the screen using the control device of a second terminal device, requiring users to operate through the first terminal device and re-project the screen each time they select new content, resulting in a cumbersome process for manipulating the projected image, and relying on the performance of the first terminal device for projection, leading to low projection smoothness and negatively impacting the user experience, this embodiment provides a screen projection method, such as... Figure 1 As shown, the method includes: Step 101: In response to the first terminal device projecting the screen onto the second terminal device, receive the target projection element and the user profile information of the target user corresponding to the first terminal device.

[0016] Among them, the target projection element is an element adapted to the second terminal device, generated based on the element to be projected in the display interface of the first terminal device.

[0017] In this application embodiment, the first terminal device can be the device that triggers screen projection, and correspondingly, the second terminal device can be the device being projected onto. For example, if it is necessary to project the screen of a mobile phone onto a tablet computer, the mobile phone can be the first terminal device in this application embodiment, and the tablet computer can be the second terminal device in this application embodiment. If it is necessary to project the screen of a tablet computer onto a television, the tablet computer can be the first terminal device in this application embodiment, and the television can be the second terminal device in this application embodiment, and so on. No further examples will be given here.

[0018] In some examples, screen projection can be the process of transmitting interface content (such as EPG pages, web pages, application interfaces, etc.) from a first terminal device to a second terminal device for display. This process includes not only image transmission but also interaction adaptation and interface reconstruction.

[0019] For example, if it is necessary to project the mobile phone screen onto a tablet computer, the mobile phone can be the first terminal device in this application embodiment, and the tablet computer can be the second terminal device in this application embodiment. When the user clicks the "project" button on the mobile phone, the display screen on the mobile phone will be projected onto the tablet computer.

[0020] As an alternative, if it is necessary to project the tablet screen onto a TV, the tablet can be the first terminal device in this application embodiment, and the TV can be the second terminal device in this application embodiment. When the user clicks the "cast" button on the tablet, the display screen on the mobile phone is projected onto the TV.

[0021] In this embodiment, the target projection element can be a set of interface elements extracted from the first terminal device and adapted to the second terminal device. For example, the target projection element in this embodiment may include, but is not limited to: layout structure (such as control tree, group information), visual elements (such as pictures, text, buttons), interactive elements (such as focus area, operation hotspot) in the first terminal device, etc.

[0022] For example, if it is necessary to project the mobile phone screen onto a tablet computer, the mobile phone can be the first terminal device in this application embodiment, and the tablet computer can be the second terminal device in this application embodiment. When the user clicks the "project" button on the mobile phone, at least one element 1 can be extracted from the mobile phone. Then, the extracted at least one element 1 is processed to generate at least one element 2 adapted to the tablet computer. At least one element 1 can be the element to be projected in this application embodiment, and at least one element 2 can be the target projection element in this application embodiment. Then, based on at least one element 2, the display screen on the mobile phone can be projected onto the tablet computer.

[0023] As an alternative approach, if it is necessary to project the tablet screen onto a television, the tablet can be the first terminal device in this application embodiment, and the television can be the second terminal device in this application embodiment. When the user clicks the "cast" button on the tablet, at least one element 3 can be extracted from the tablet. The extracted at least one element 3 is then processed to generate at least one element 4 adapted to the television. At least one element 3 can be the element to be projected in this application embodiment, and at least one element 4 can be the target projection element in this application embodiment. Thus, based on at least one element 2, the display screen on the tablet can be projected onto the television.

[0024] In this embodiment of the application, the user profile information can be a user feature model constructed based on user historical behavior, preferences, device usage habits, and other data. In this embodiment of the application, the user profile information can be used for personalized interface adaptation in a second terminal device.

[0025] Step 102: Generate a target display interface template based on the target projection elements and user profile information.

[0026] In this embodiment of the application, the target display interface template can be an interface layout and interaction template dynamically generated based on user profile and terminal device characteristics. For example, the target display interface template can be used to guide the second terminal device on how to display the projected content.

[0027] As an alternative approach, when it is necessary to project the mobile phone screen onto a tablet computer, if the user is determined to be a sports enthusiast based on the user profile information, a display interface template 1 that includes displaying sports events can be selected during the screen projection process onto the tablet computer. In this case, the display interface template 1 can be the target display interface template in the embodiments of this application.

[0028] As an alternative approach, when it is necessary to project the tablet screen onto a TV, if the user is determined to be a child based on the user profile information, a display interface template 2 with larger interface elements, more vivid colors, and simpler interaction can be selected during the screen projection process. The display interface template 2 can then be the target display interface template in the embodiments of this application.

[0029] Step 103: Generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device.

[0030] In this embodiment, the target display interface can be a personalized and device-adapted user interface displayed on the second terminal device. It should be noted that the target display interface in this embodiment can support display interfaces with multiple interaction methods such as remote control, mobile phone touch, and motion sensing.

[0031] As an alternative, when it is necessary to project the mobile phone screen onto a tablet, the target display interface can be a personalized and device-adapted user interface displayed on the tablet; when it is necessary to project the tablet screen onto a TV, the target display interface can be a personalized and device-adapted user interface displayed on the TV, and so on. Examples will not be listed here.

[0032] Compared with existing technologies, this embodiment, when using a first terminal device to project the screen onto a second terminal device, can generate a target display interface template adapted to the user and the second terminal device based on target projection elements and user profile information. Then, based on the target display interface template, the target display interface of the second terminal device is generated, thus completing the screen projection. This means that the target display interface generated in this embodiment is regenerated within the target display interface template based on the target projection elements and user profile information, ensuring that the generated target display interface is adapted to the second terminal device. This allows the screen projection display to rely on the device performance of the second terminal device, enabling the user to directly control the target display interface from the second terminal device, improving the smoothness of screen projection and enhancing the user experience.

[0033] As a refinement and extension of the above embodiments, this embodiment provides an instruction determination method, such as... Figure 2 As shown, the method includes: Step 201: In response to the first terminal device projecting the screen onto the second terminal device, receive the target projection element and the user profile information of the target user corresponding to the first terminal device.

[0034] Among them, the target projection element is an element adapted to the second terminal device, generated based on the element to be projected in the display interface of the first terminal device.

[0035] For example, such as Figure 3 As shown, during the interaction between the large and small screens (i.e., during the process of the first terminal device projecting its screen onto the second terminal device in this embodiment), the small screen (i.e., the first terminal device in this embodiment) and the large screen (i.e., the second terminal device in this embodiment) will fully understand the characteristics of the device by reading parameters such as the device's resolution parameters, planar interaction mode, and device ID.

[0036] In some examples, resolution parameters determine the screen's display clarity and image size. Different resolutions require different adaptation strategies. For example, high-resolution devices may require higher-quality images and finer fonts, while low-resolution devices require appropriate compression and adjustment to ensure content is clearly displayed on the screen. The planar interaction mode determines the way users interact with the device. For example, mobile phone UIs typically use touch operation, while TV UIs use remote control operation. In this embodiment, by reading the planar interaction mode parameters, the system can make corresponding adjustments according to different interaction methods to provide users with a more natural and convenient operating experience. For example, when the small screen (i.e., the first terminal device in this embodiment) is in touch operation mode, the system can automatically convert it to an interactive interface suitable for remote control operation after casting to the large screen (i.e., the second terminal device in this embodiment), such as increasing button size and simplifying operation processes.

[0037] For example, the device ID parameter can be used to uniquely identify each device, ensuring accurate connection establishment and data transmission in a multi-device environment. For instance, when the small screen (i.e., the first terminal device in this embodiment) establishes a binding relationship with the large screen (i.e., the second terminal device in this embodiment) by scanning a code, the device ID can be used for verification and confirmation, ensuring that only qualified devices can connect. At the same time, the established message channel also depends on the device ID, ensuring the accuracy and security of data transmission. In this way, the small screen (i.e., the first terminal device in this embodiment) and the large screen (i.e., the second terminal device in this embodiment) can achieve a fast and stable connection to guarantee subsequent data transmission.

[0038] As an optional approach, after the small-screen terminal (i.e., the first terminal device in this embodiment) connects to the server, it can obtain page templates and page data, and display them according to the screen size and interaction mode of the small-screen terminal (i.e., the first terminal device in this embodiment). During this process, the server will send page templates and data suitable for small-screen display to the small-screen terminal (i.e., the first terminal device in this embodiment) according to its request. The page template may include design elements such as layout, color, and font, while the page data may include specific information such as text content, images, and videos. The small-screen terminal (i.e., the first terminal device in this embodiment) will adjust and optimize the obtained page templates and data according to its own screen size and interaction mode to ensure that the best effect can be presented on the small screen (i.e., the first terminal device in this embodiment).

[0039] Step 202: Generate a target display interface template based on the target projection elements and user profile information.

[0040] For example, based on the example in step 201, such as Figure 3 As shown, when a user clicks the one-click screen mirroring button, the page template and page data of the small screen (i.e., the first terminal device in this embodiment) are sent to the large screen (i.e., the second terminal device in this embodiment) through a message channel. After receiving the data, the large screen (i.e., the second terminal device in this embodiment) will dynamically adapt to the screen resolution and interaction method of the large screen (i.e., the second terminal device in this embodiment), thereby displaying a suitable UI page on the large screen. In this process, the system can adjust the images, fonts, and other elements in the page template according to the resolution parameters of the large screen to adapt to the display requirements of the large screen. For example, if the resolution of the large screen is high, the system will automatically improve the clarity and quality of the image to ensure that a more delicate picture can be presented on the large screen. At the same time, the system will also optimize the operation buttons, menus, etc. of the page according to the interaction method of the large screen to make them more suitable for remote control operation. For example, the button size will be increased and the operation process will be simplified so that users can operate the remote control more conveniently.

[0041] It should be noted that, through the data transmission and adaptation process, the embodiments of this application can realize the connection and content sharing between the small screen terminal (i.e., the first terminal device in this application embodiment) and the large screen terminal (i.e., the second terminal device in this application embodiment), providing users with a more convenient and efficient screen casting experience.

[0042] For example, when the small screen (i.e., the first terminal device in this application embodiment) is a mobile phone, the mobile phone can send the screen casting command along with various elements on the UI interface to the message server; the message server can forward the user profile and the adapted large screen template elements to the large screen (i.e., the second terminal device in this application embodiment), for example, a tablet computer, a desktop computer, a TV, etc.; after receiving the message, the large screen (i.e., the second terminal device in this application embodiment) can identify the elements in the template through the built-in SDK component, and automatically adapt the UI template corresponding to the user mode according to the user profile sent by the server, such as: elderly mode, movie lover mode, sports lover mode, children's mode, etc.

[0043] In some examples, Software Development Kit (SDK) event tracking is a technique that uses an application to automatically or semi-automatically collect data such as user behavior, performance metrics, and business events by integrating a data collection SDK into the application.

[0044] It should be noted that the embodiments of this application have significant advantages in automatically adjusting and optimizing content presentation and interaction. Firstly, the embodiments of this application can accurately adapt content based on the size, resolution, and display characteristics of different screens by reading the device parameters of the small screen (i.e., the first terminal device in this application embodiment) and the large screen (i.e., the second terminal device in this application embodiment). For example, when casting from a small-screen mobile phone to a large-screen TV, the system automatically analyzes that the mobile phone's resolution is 1080×2400 and the TV's resolution is 1920×1080, and then adapts it according to the calculated ratio. For the mobile phone, assuming the width of the design drawing is 360dp, the density = (float)1080 / 360 = 3, and the new dpi = density * 160 = 480. The same calculation is performed for the TV to find suitable density and dpi values, ensuring that the content displayed on different devices has the same ratio. In this way, images, text, and other content can be displayed optimally on the TV screen. Images can be scaled proportionally while maintaining clarity; text can automatically adjust font size and line spacing according to screen size and resolution to ensure readability.

[0045] In some examples, when content from a small screen (i.e., the first terminal device in this embodiment) is projected onto a large screen (i.e., the second terminal device in this embodiment), the system will re-plan the content layout according to the aspect ratio of the large screen (i.e., the first terminal device in this embodiment). For example, for text paragraphs in a webpage, an algorithm based on the golden ratio can be used to determine the number of characters per line and the paragraph spacing. If the aspect ratio of the large screen is close to 16:9, the system will appropriately lengthen the more compact paragraphs on the small screen, increasing the number of characters per line and reducing the number of paragraphs, so as to make full use of the horizontal space of the large screen while ensuring the smoothness and aesthetics of reading.

[0046] In some examples, in addition to simply adjusting the font size according to the screen size ratio, the viewing distance factor can also be considered. Specifically, the appropriate font size can be calculated using the perspective principle in visual science, based on the distance sensor data (if available) between the small and large screens or the preset average viewing distance parameters. For example, for a viewing scenario of about 3 meters away from the large screen, the minimum readable angle of the font at that distance can be used to ensure that the title font size is clearly distinguishable on the large screen without being too abrupt, and that the font size of the body text is also easy for users to read, avoiding visual fatigue caused by fonts that are too small or affecting the aesthetics of the page due to fonts that are too large.

[0047] In some examples, image super-resolution algorithms and content-aware scaling techniques can also be used. Specifically, for images sent from small screens, the image type (such as landscape or portrait) and its important elements can be determined first. If it is a landscape image, the focus is on preserving the details of the distant scenery. If it is a portrait image, the clarity of key areas such as the face of the person is ensured. This allows the embodiments of this application to improve the resolution while avoiding image distortion caused by simple stretching, so that the image can be displayed in high quality on a large screen and is consistent with the layout of the entire page.

[0048] For example, converting a mobile webpage UI to a TV set-top box UI may include, but is not limited to, the following processing steps: First, the Hypertext Markup Language (HTML) and Cascading Style Sheets (CSS) structures of the mobile webpage are analyzed to identify various page elements (such as text blocks, images, buttons, input boxes, etc.). These elements are then grouped according to their function and visual relevance on the page; for example, the menu buttons in the navigation bar are grouped together, while the titles, body text, and images in the article content are grouped together. For each element, its coordinate position on the mobile device (with the top left corner of the screen as the origin), size (width and height), and CSS style properties (such as display, float, position, etc.) are recorded.

[0049] Furthermore, the aspect ratio of the mobile phone screen to the TV screen can be calculated. Let the width of the mobile phone screen be... The height is The width of the TV screen is The height is The width scaling factor can be determined using Formula 1, and the height scaling factor can be determined using Formula 2. Formulas 1 and 2 are shown below: (Formula 1) (Formula 2) Furthermore, the page can be divided into different areas based on the landscape orientation of the TV screen. For example, the screen can typically be divided into a top navigation area, a main content area, and a bottom operation area. The space ratio of each area can be allocated according to its importance and user visual habits. For example, the top navigation area occupies 10% of the total height, the bottom operation area occupies 10%, and the main content area occupies 80%. For each element group, its position and size on the TV screen can be recalculated based on its relative position on the mobile device and the new space allocation rules.

[0050] As an optional method, for important text such as titles, the font size and line spacing can be adjusted according to the new layout area and scaling factor. If a horizontal stretching effect is more important, the font size can be adjusted using Formula 3. If the changes in width and height are taken into account, the font size can be adjusted using Formula 4, and the line spacing will also be increased accordingly to adapt to large screen displays. This can be done by setting a fixed multiplier (such as 1.5 times) or adjusting proportionally according to the new font size. Formulas 3 and 4 are shown below: (Formula 3) (Formula 4) As an alternative, image scaling algorithms can be used for image elements. Specifically, for simple images, bilinear interpolation can be used to scale the image according to the new size requirements. If it is necessary to keep the key content of the image from being distorted, the main subject area of ​​the image can be detected first (such as using image recognition algorithms to find the main parts such as people and objects), and then the non-subject areas can be stretched or compressed to ensure that the proportion of the main subject area remains basically unchanged. The scaling ratio is determined based on the layout ratio of the image on mobile devices and televisions, as well as the size of the area in which it is located.

[0051] As an alternative, the size of buttons and action elements can be increased to make them easier to select using a remote control. Specifically, Formula 5 can be used to adjust the width of the button, and Formula 6 can be used to adjust the height of the button. Formulas 5 and 6 are shown below: (Formula 5) (Formula 6) In Formulas 5 and 6, f can be a scaling factor (such as 1.2-1.5). At the same time, the spacing between buttons is adjusted to ensure the aesthetics of the layout and the convenience of operation. The position of the button in the new layout is determined according to its element group and the new area division, so as to maintain visual consistency and operational logic as much as possible.

[0052] Step 203: Determine the target user's operational behavior data on the first terminal device.

[0053] For example, based on the example in step 202, such as Figure 3 As shown, the second terminal device can collect user operation behavior data through the SDK. Specifically, it can embed points in the SDK of the mobile phone (i.e., the first terminal device in this application embodiment) to collect the user's operation behavior on the mobile phone interface in real time. For example, it can include, but is not limited to, the coordinate sequence of touch events, the duration of focus stay, and the swiping speed, etc.

[0054] Step 204: Based on the operation behavior data, determine the first operation focus area corresponding to the target user and the focus area transfer information corresponding to the first operation focus area in the target display interface template.

[0055] In this embodiment, the first operation focus area can be the area in the target display interface template that the user is most likely to focus on or operate first, determined based on user operation behavior data. For example, if the user frequently clicks on the recently viewed area, the recently viewed area can be the first operation focus area in this embodiment; if the user frequently browses the sports live broadcast entrance before screen casting, the sports live broadcast entrance can be the first operation focus area in this embodiment, and so on. No further examples will be given here.

[0056] In this embodiment, the focus area transfer information can be the pattern or probability information of the user switching between different operation focus areas. For example, if the user usually views the recommendation area first and then switches to the category area, the transfer information corresponding to the recommendation area to the category area can be the focus area transfer information in this embodiment. If the user returns to the homepage after viewing the details page, the transfer information corresponding to the details page to the homepage can be the focus area transfer information in this embodiment, and so on. No further examples will be given here.

[0057] For example, based on the example in step 203, such as Figure 3 As shown, in this embodiment of the application, based on the user's operation trajectory, a time decay-based clustering algorithm can be used to cluster the user's operation coordinates on the first terminal device in the operation behavior data, and the dwell time is introduced as a weight factor. For example, a long press operation will generate a high weight point, optimize the calculation of the cluster center point, and then generate a hot zone for each cluster center according to the ellipse model. The major and minor axes of the ellipse are determined by the standard deviation of the coordinate points, and a time decay factor is superimposed to give higher weight to recent operations.

[0058] As an alternative approach, embodiments of this application can train a lightweight long short-term memory network (LSTM) to learn user operation sequences (i.e., operation data in embodiments of this application), for example, after continuously clicking on area A, the user usually switches to area B, and establish a hot zone transfer prediction model. Then, the hot zone transfer prediction model can be used to identify the focus area transfer information.

[0059] Step 205: Adjust the target display interface template according to the first operation focus area and focus area transfer information to generate a target display interface adapted to the second terminal device.

[0060] Optionally, when performing the step of "adjusting the target display interface template according to the first operation focus area and focus area transfer information to generate a target display interface adapted to the second terminal device", this embodiment may use the following method, but is not limited thereto, including: parsing the target display interface to obtain the recombined control tree corresponding to the target display interface; recombining the recombined control tree according to the first operation focus area and focus area transfer information to obtain at least one first operation function area corresponding to the target display interface template; and generating the target display interface according to at least one first operation function area.

[0061] As an alternative approach, based on the example in step 204, such as Figure 3 As shown, the embodiments of this application can further obtain a UI template adapted to the user mode (i.e., the target display interface template in the embodiments of this application), and then parse the UI template adapted to the user mode (i.e., the target display interface template in the embodiments of this application) into a reconfigurable control tree, and dynamically reconstruct the control tree according to the determined hot zone (i.e., the first operation focus area in the embodiments of this application) and the predicted hot zone transfer (i.e., the focus area transfer information in the embodiments of this application). For example, the controls corresponding to the high-frequency operation hot zone can be moved up 1-2 levels, and related controls in adjacent hot zones can be merged to form an operation function area, and the control size can be automatically scaled according to the activity of the hot zone.

[0062] It should be noted that the embodiments of this application can calculate the hot zone of the focus (i.e., the first operation focus area in the embodiments of this application) and display the projected template according to the hot zone to improve the user's click rate.

[0063] Optionally, after executing "generating a target display interface adapted to the second terminal device according to the target display interface template to complete the screen projection of the first terminal device onto the second terminal device", this embodiment may adopt the following method, but not limited thereto, including: dividing the target display interface into at least one second operation focus area; determining the target operation focus area selected by the remote control command of the remote control device corresponding to the second terminal device from the at least one second operation focus area; and switching the target operation focus area in the at least one second operation focus area in response to the focus area switching command of the remote control device.

[0064] For example, based on the above example, such as Figure 3As shown, in this embodiment of the application, the page can be divided into multiple focus areas (i.e., at least one second operation focus area in this embodiment) based on the rearranged page elements on the TV screen (i.e., the second terminal device in this embodiment of the application). Each focus area contains one or more related operation elements (such as a button group, an input box and its related prompt text, etc.). Then, a unique identifier can be assigned to each focus area, and its coordinate range (top left and bottom right coordinates) on the TV screen can be recorded.

[0065] As an optional approach, such as Figure 4 As shown, once the page has finished loading on the TV (i.e., the second terminal device in this application embodiment), the focus can be set on the main operation area of ​​the page or the element that the user is most likely to operate first (such as the first menu button in the top navigation bar or the play button on the video playback page).

[0066] For example, in this embodiment of the application, the directional key operations of the remote control (i.e., the remote control device in this embodiment of the application) can also be mapped to the movement of the focus between different focus areas; when the up, down, left, and right directional keys are pressed on the remote control (i.e., the remote control device in this embodiment of the application), adjacent focus areas can be found according to the current focus area and the target direction. For example, if the current focus is in a vertically arranged button group, pressing the up or down directional key will move the focus between the buttons in the button group; if the left or right directional key is pressed, and the current button group has horizontally adjacent focus areas in the page layout, the focus will move to the adjacent area.

[0067] As an alternative approach, the movement of focus between different focus areas in this embodiment can follow the shortest path principle and user visual habits. For example, in a form page, the order in which focus moves from one input box to the next should be from left to right and from top to bottom, conforming to the general order in which users fill out forms.

[0068] Optionally, after executing "generating a target display interface adapted to the second terminal device according to the target display interface template to complete the screen projection of the first terminal device onto the second terminal device", this embodiment may adopt the following methods, but not limited thereto, including: displaying the target operation focus area according to the target display method; and controlling the target operation focus area to execute operation commands in response to the operation commands of the remote control device on the target operation focus area.

[0069] For example, based on the above example, such as Figure 3As shown, when the focus moves to a certain element (i.e., the target operation focus area in this embodiment), the element provides visual feedback by changing its appearance (such as changing color, enlarging, adding borders, etc.) so that the user can clearly know the position of the focus. When the user presses the confirmation button on the remote control, the operation corresponding to the current focus element is triggered. For button elements, the button click event is executed (such as submitting a form, playing a video, etc.); for input box elements, the input mode is activated (if remote control input is supported) or the user is prompted to use other input devices to input.

[0070] It should be noted that the embodiments of this application can provide users with a good screen casting experience. In addition, considering further improving the user experience, the embodiments of this application can also control the UI interaction of the large screen through the touch screen sensor, motion sensing, and voice interaction of the mobile phone, through a multi-screen interaction platform, and in combination with adaptive large screen UI adaptation.

[0071] Optionally, after executing "generating a target display interface adapted to the second terminal device according to the target display interface template to complete the screen projection of the first terminal device onto the second terminal device", this embodiment may adopt the following method, but not limited thereto, including: receiving a control command from the target user to the target display interface, wherein the control command is input by the target user through the first terminal device; determining the operation to be executed corresponding to the target display interface based on the control command, and controlling the target display interface to execute the operation to be executed.

[0072] As an alternative approach, this application embodiment can utilize the touchscreen of a mobile phone (i.e., the first terminal device in this application embodiment) to respond to the user's pressing operation. When the pressing time reaches a certain duration, the screen position pressed by the user corresponds to the focus position of the large screen. When the user presses and holds on the screen and moves the position, the screen focus controlled by the user also moves in the corresponding direction until the user releases the hand, at which point the movement of the focus stops.

[0073] As an optional approach, this embodiment of the application can also sense the user's pressing operation through the screen of a mobile phone (i.e., the first terminal device in this embodiment). When a certain duration is reached (e.g., the duration exceeds 1 second), the simulated "mouse" operation function is activated. The mobile phone screen calculates the movement speed of the user's touch point and monitors the displacement distance of the user's touch point at a certain frequency (e.g., 10Hz, sampling 10 times per second). The displacement distance / time is used to obtain the speed. The displacement speed of the mobile phone pressing the screen is calculated according to a certain ratio algorithm (e.g., 10 pixels / s corresponds to 1 grid / s of focus movement on the large screen). If the user's "mouse" movement speed is faster, the focus movement speed on the large screen will also increase synchronously. The mobile phone screen senses the user's pressing operation in the up / down and left / right directions, which is consistent with the focus movement direction of the large screen UI. After releasing the hand, a single touch on the screen is received, and the mobile phone screen responds to the single-click operation, converts it into a click command, and transmits it to the large screen UI as an "OK" click command.

[0074] As an alternative approach, in this embodiment of the application, if the mobile phone screen senses that the user's touch operation is a rapid horizontal swipe across the screen, the operation can be converted into a page-turning command on the mobile phone (i.e., the first terminal device in this embodiment of the application) and sent to the UI on the large screen to respond to the page-turning command; if the swipe across the screen is vertical, it can be converted into an up-down page-turning command and sent to the UI on the large screen to respond to the page-turning command.

[0075] As an alternative approach, when the phone detects the deflection signal from its built-in gyroscope, it can convert the deflection direction and angle into control signals, which are then transmitted to the large-screen UI via a multi-screen interaction messaging platform to control the UI to switch focus accordingly. The size of the deflection angle controls the speed of focus movement.

[0076] For example, the embodiments of this application can optimize the interaction according to the interaction mode of different devices. Specifically, when the small screen terminal (i.e., the first terminal device in this application embodiment) uses touch operation, while the large screen terminal (i.e., the second terminal device in this application embodiment) uses remote control operation, the system will automatically switch the interaction mode during the screen projection process. For example, when the small screen terminal (i.e., the first terminal device in this application embodiment) connects to the server to obtain page data and displays it, the user browses through touch operation. When the user clicks one-click screen projection, the large screen terminal (i.e., the second terminal device in this application embodiment) receives the data and performs dynamic adaptation. At this time, the system will adjust the interactive interface to a mode suitable for remote control operation, increase the button size, and simplify the operation process, so that the user can more conveniently use the remote control to perform UI interaction operations such as page focus control.

[0077] As an optional approach, this application also provides an interaction example between a small-screen terminal (i.e., the first terminal device in this application embodiment) and a large-screen terminal (i.e., the second terminal device in this application embodiment), such as... Figure 5 and Figure 6 As shown, this includes: On both small and large screens, by reading parameters such as device resolution, interactive mode, and device ID, the small screen can establish a binding relationship with the large screen via QR code scanning and establish a message channel. The small screen connects to the server to obtain page templates and page data, and displays them according to its screen size and interaction mode. Clicking "cast" sends the page templates and page data from the small screen to the large screen via the message channel. The large screen dynamically adapts the obtained page templates and page data according to its screen resolution and interaction method, thus displaying a suitable UI page. By incorporating a lightweight AI model SDK on both small and large screens, and combining it with user playback habits, the UI layout is adapted to the user's personalized preferences. After the page is displayed on the large screen, users can directly control the UI interaction using their mobile phones and button remote controls. The unique touch screen interaction, motion-sensing interaction, and voice interaction of the mobile phone screen are converted by the server into interaction methods such as fast page turning and voice command control on the large screen.

[0078] In related technologies, the commonly used EPG pages for large and small screens adopt a separate and independent operation model, requiring independent production of display templates and image materials, which brings many drawbacks. The UI for mobile phones and TVs requires completely independent design and development investment. Designers need to repeatedly conceive layouts and visual effects for different screen sizes, resulting in a large workload and difficulty in maintaining a consistent design style; independent production leads to repetitive work, such as requiring multiple copies of image materials, making resource sharing impossible. At the same time, updating program information requires separate operations on both ends, easily causing data asynchrony and increasing management difficulty. Furthermore, existing screen mirroring technologies also have many limitations in terms of interaction. Functionally, they only support casting a single video link from the mobile phone to the TV via a local area network protocol, making it impossible to switch content on the TV. Users need to return to the mobile phone and re-cast each time they select new content, leading to viewing interruptions and cumbersome operation; while mirroring mode can completely copy the mobile phone screen to the TV, it is essentially just screen projection—all interactive controls still need to be performed on the mobile phone, rendering the TV remote control ineffective, unable to directly fast forward, pause, etc. This not only sacrifices the operational convenience of the large screen but also depends on the mobile phone's performance; insufficient device performance can easily lead to screen mirroring lag and unsmooth playback.

[0079] It should be noted that the embodiments of this application can automatically adjust and optimize the content presentation according to different screen sizes, resolutions, display characteristics, etc. At the same time, it can be controlled collaboratively by a mobile phone (i.e., the first terminal device in this application embodiment) and a remote control (i.e., the remote control device in this application embodiment) to realize the control of the EPG page. Specifically, when the user casts the web page content on the mobile phone to the TV, the TV can automatically rearrange, adjust the font size and image resolution to adapt to the display of the TV screen, and also supports direct control of the EPG page using a mobile phone remote control and a button remote control.

[0080] In this embodiment, when a user needs to cast their screen, they can simply click one-click casting on the small screen (i.e., the first terminal device in this embodiment) to send the page template and data of the small screen (i.e., the first terminal device in this embodiment) to the large screen (i.e., the second device in this embodiment) via a message channel. This eliminates the need for complex setup and pairing, greatly improving operational convenience. Furthermore, this embodiment can utilize the linkage mode between the large and small screens, overlaying UI casting technology, to fully leverage the diversity of mobile screen interactions and compensate for the limited interaction of large screen UI focus control, providing a more efficient, convenient, and personalized UI interaction experience. This embodiment can also automatically adapt... The UI template corresponding to the user mode is matched, and combined with the collected user operation behavior data, the focus hot zone is calculated according to the user's operation trajectory. The template projected to the screen is then displayed according to the hot zone. In this embodiment, the touch screen sensor, motion sensing, and voice interaction of the mobile phone (i.e., the first terminal device in this embodiment) can also be used to coordinate the UI interaction of the large screen (i.e., the second device in this embodiment) based on a multi-screen interaction platform and combined with adaptive large screen UI adaptation. This enables the remote control to select a focus on the large screen, and when the user presses and holds the mobile phone screen and moves the position, the focus on the large screen controlled by the remote control will also move in the corresponding direction until the user releases the hand, at which point the focus movement stops, and the remote control can then select the corresponding function item.

[0081] Compared with existing technologies, this embodiment, when using a first terminal device to project the screen onto a second terminal device, can generate a target display interface template adapted to the user and the second terminal device based on target projection elements and user profile information. Then, based on the target display interface template, the target display interface of the second terminal device is generated, thus completing the screen projection. This means that the target display interface generated in this embodiment is regenerated within the target display interface template based on the target projection elements and user profile information, ensuring that the generated target display interface is adapted to the second terminal device. This allows the screen projection display to rely on the device performance of the second terminal device, enabling the user to directly control the target display interface from the second terminal device, improving the smoothness of screen projection and enhancing the user experience.

[0082] Furthermore, as Figure 1 and Figure 2 To illustrate the specific implementation of the method shown, this embodiment provides a screen projection device, such as... Figure 7 As shown, the device includes: a receiving module 31 and a generating module 32.

[0083] The receiving module 31 is configured to receive a target projection element and user profile information of the target user corresponding to the first terminal device in response to the first terminal device projecting the screen onto the second terminal device. The target projection element is an element adapted to the second terminal device based on the element to be projected in the display interface of the first terminal device. The generation module 32 is configured to generate a target display interface template based on the target projection element and the user profile information; The generation module 32 is also configured to generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device.

[0084] In some examples of this embodiment, the generation module 32 is specifically configured to: determine the target user's operation behavior data on the first terminal device; determine the first operation focus area corresponding to the target user and the focus area transfer information corresponding to the first operation focus area in the target display interface template based on the operation behavior data; adjust the target display interface template according to the first operation focus area and the focus area transfer information to generate a target display interface adapted to the second terminal device.

[0085] In some examples of this embodiment, the generation module 32 is further configured to parse the target display interface to obtain a recombined control tree corresponding to the target display interface; recombine the recombined control tree according to the first operation focus area and the focus area transfer information to obtain at least one first operation function area corresponding to the target display interface template; and generate the target display interface according to the at least one first operation function area.

[0086] In some examples of this embodiment, the generation module 32 is further configured to divide the target display interface into at least one second operation focus area; determine the target operation focus area selected by the remote control command of the remote control device corresponding to the second terminal device from the at least one second operation focus area; and switch the target operation focus area in the at least one second operation focus area in response to the focus area switching command of the remote control device.

[0087] In some examples of this embodiment, the generation module 32 is further configured to display the target operation focus area according to the target display mode; and to control the target operation focus area to execute the operation command in response to the operation command of the remote control device on the target operation focus area.

[0088] In some examples of this embodiment, the generation module 32 is further configured to receive control instructions from the target user on the target display interface, the control instructions being input by the target user through the first terminal device; determine the operation to be performed corresponding to the target display interface based on the control instructions, and control the target display interface to perform the operation to be performed.

[0089] It should be noted that other corresponding descriptions of the functional units involved in the screen projection device provided in this embodiment can be found in [reference needed]. Figure 1 and Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0090] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The method shown.

[0091] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0092] like Figure 8 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 401; and, A memory 402 is communicatively connected to at least one of the processors 401; wherein, The memory 402 stores instructions that can be executed by at least one of the processors to enable at least one of the processors to perform the screen projection method as described above.

[0093] Figure 8 Take a processor 401 as an example.

[0094] The electronic device may also include an input device 403 and a display device 404.

[0095] The processor 401, memory 402, input device 403, and display device 404 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0096] The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the screen projection method in the embodiments of this application, for example, Figure 1 and Figure 2 The method flow is shown. The processor 401 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 402, thereby implementing the screen projection method in the above embodiments.

[0097] The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the screen projection method. Furthermore, the memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include memory remotely located relative to the processor 401, and these remote memories may be connected to the apparatus performing the screen projection method via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0098] The input device 403 can receive user clicks and generate signal inputs related to user settings and function control of the screen projection method. The display device 404 may include a display screen or other display device.

[0099] When one or more modules are stored in the memory 402, and are run by one or more processors 401, the screen projection method in any of the above method embodiments is executed.

[0100] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0101] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0102] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms, or it can be implemented by hardware. By applying the solution of this embodiment, compared with the prior art, this embodiment, when using a first terminal device to project the screen onto a second terminal device, can generate a target display interface template adapted to the user and the second terminal device based on the target projection elements and user profile information. Then, based on the target display interface template, the target display interface of the second terminal device is generated to complete the screen projection. This means that the target display interface generated in this embodiment is regenerated in the target display interface template based on the target projection elements and user profile information, so that the generated target display interface is adapted to the second terminal device. Thus, the screen projection display relies on the device performance of the second terminal device, allowing the user to directly control the second terminal device to control the target display interface, improving the smoothness of screen projection, and thus enhancing the user experience.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A screen projection method, characterized in that, include: In response to the first terminal device projecting the screen onto the second terminal device, the system receives the target projection element and the user profile information of the target user corresponding to the first terminal device. The target projection element is an element adapted to the second terminal device based on the element to be projected in the display interface of the first terminal device. A target display interface template is generated based on the target projection elements and the user profile information; Generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device.

2. The method according to claim 1, characterized in that, Before generating the target display interface adapted to the second terminal device according to the target display interface template, the method further includes: Determine the target user's operational behavior data on the first terminal device; The step of generating a target display interface adapted to the second terminal device according to the target display interface template includes: Based on the operation behavior data, determine the first operation focus area corresponding to the target user in the target display interface template and the focus area transfer information corresponding to the first operation focus area; The target display interface template is adjusted according to the first operation focus area and the focus area transfer information to generate a target display interface adapted to the second terminal device.

3. The method according to claim 2, characterized in that, The step of adjusting the target display interface template according to the first operation focus area and the focus area transfer information to generate a target display interface adapted to the second terminal device includes: The target display interface is parsed to obtain the recombined control tree corresponding to the target display interface; The reorganized control tree is reorganized based on the first operation focus area and the focus area transfer information to obtain at least one first operation function area corresponding to the target display interface template; The target display interface is generated based on the at least one first operation function area.

4. The method according to claim 1, characterized in that, After generating a target display interface adapted to the second terminal device according to the target display interface template to complete the screen projection of the first terminal device onto the second terminal device, the method further includes: The target display interface is divided into at least one second operation focus area; Determine the target operation focus area selected by the remote control command of the remote control device corresponding to the second terminal device from the at least one second operation focus area; In response to a focus area switching command from the remote control device, the target operating focus area is switched within the at least one second operating focus area.

5. The method according to claim 4, characterized in that, After determining the target operating focus area selected by the remote control command of the remote control device corresponding to the second terminal device from the at least one second operating focus area, the method further includes: The target operation focus area is displayed according to the target display method; In response to the operation command of the remote control device to the target operation focus area, the target operation focus area is controlled to execute the operation command.

6. The method according to claim 4 or 5, characterized in that, After generating a target display interface adapted to the second terminal device according to the target display interface template to complete the screen projection of the first terminal device onto the second terminal device, the method further includes: Receive control commands from the target user to the target display interface, wherein the control commands are input by the target user through the first terminal device; Based on the control instructions, the operation to be performed corresponding to the target display interface is determined, and the target display interface is controlled to perform the operation to be performed.

7. A screen projection device, characterized in that, include: The receiving module is configured to receive a target projection element and user profile information of the target user corresponding to the first terminal device in response to the first terminal device projecting the screen onto the second terminal device. The target projection element is an element adapted to the second terminal device based on the element to be projected in the display interface of the first terminal device. The generation module is configured to generate a target display interface template based on the target projection element and the user profile information; The generation module is also configured to generate a target display interface adapted to the second terminal device according to the target display interface template, so as to complete the screen projection of the first terminal device onto the second terminal device.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images