A screen projection method

Abstract

The application relates to a screen projection method applied to a first device, comprising the following steps: acquiring screen proportion information of a second device; acquiring a first interface displayed by the first device; acquiring data of at least one second interface by segmenting or cropping the first interface according to the screen proportion information of the second device; and sending the data of the at least one second interface to the second device; wherein the data of the at least one second interface is used for full-screen display of a third interface corresponding to the first interface on the second device. Through the method provided in the application, the display proportion of the content of the first interface in the display screen of the second device can be improved, and the user experience can be improved.

Description

Technical Field

[0001] This invention relates to the field of smart terminal technology, and in particular to a screen projection method. Background Technology

[0002] In existing synchronous screen mirroring methods, according to the protocol, after the first device (such as a mobile phone) obtains the resolution of the second device (such as a TV), it fills the interface displayed on the first device with a black background according to the resolution of the second device to obtain an interface that conforms to the resolution of the second device, and then sends the processed interface data that conforms to the resolution of the second device to the second device.

[0003] Because the resolutions of the first and second devices are different, using the existing screen mirroring method, only a small portion of the first device's interface is displayed on the second device's screen, resulting in a small display area. This does not meet the user's original intention for screen mirroring and affects the user experience. Summary of the Invention

[0004] This application provides a screen mirroring method that increases the display area of ​​the interface displayed on the first device on the screen of the second device, thereby improving the user experience.

[0005] In a first aspect, a screen mirroring method for a first device is provided, comprising: acquiring screen ratio information of a second device; acquiring a first interface displayed by the first device; acquiring data of at least one second interface by segmenting or cropping the first interface based on the screen ratio information of the second device; and sending the data of the at least one second interface to the second device; wherein the data of the at least one second interface is used by the second device to display a third interface corresponding to the first interface in full screen.

[0006] The screen ratio information is used to indicate the resolution ratio of the second device. This information may include the screen's aspect ratio, width-to-height ratio, resolution, and indicators indicating the screen resolution ratio. The ratio information can also be used to indicate the resolution ratio of a region or interface. This information may include the interface's or region's aspect ratio, width-to-height ratio, resolution, and indicators indicating the screen resolution ratio.

[0007] The above-described screen mirroring method can project the content of the first interface on the first device onto the display screen of the second device, thereby increasing the display area of ​​the content of the first interface on the display screen of the second device and improving the user experience.

[0008] The above-mentioned method of obtaining data for at least one second interface by segmenting the first interface includes: obtaining data for at least two second interfaces by segmenting the first interface based on the screen ratio information of the second device and the ratio information of the first interface.

[0009] The number of at least two second interfaces can be determined based on the screen ratio information of the second device and the ratio information of the first interface. The content displayed on the first interface does not include video content.

[0010] By using a segmentation method, the content of the first interface can be projected onto the screen of the second device with a larger display area, thus improving the user experience.

[0011] The above-mentioned method of obtaining data for at least one second interface by cropping the first interface includes: identifying a first region in the first interface; obtaining data for at least one second interface by cropping the first interface based on the screen ratio information of the second device and the ratio information of the first region; wherein the at least one second interface includes the display content of the first region.

[0012] By using the cropping method, the content of the first area in the first interface can be primarily projected onto the screen, increasing the display ratio of the content of the first area on the screen of the second device and improving the user experience. At the same time, the cropping method increases the proportion of the target content to the amount of data sent by the first device, thereby improving data transmission efficiency.

[0013] Wherein, the at least one second interface includes all the display content of the first area; or, the at least one second interface includes part of the display content of the first area.

[0014] Depending on the ratio between the second interface and the first area, the content displayed on the second interface can vary.

[0015] In one possible implementation, the screen ratio information of the second device is the same as the ratio information of the first area, and the second interface is the entire display content of the first area.

[0016] In another possible implementation, the screen ratio information of the second device indicates a different ratio than the ratio information of the first area, and the at least one second interface includes video content displayed in the first area and non-video content displayed in the second area; wherein the second area is located within the first interface.

[0017] The second area can display text and images, or it can display a background without text and images, such as a black background. The position of the second area can be set by the first device or by the user.

[0018] In another possible implementation, the screen ratio information of the second device is different from the ratio information of the first area, and the at least one second interface is a part of the display content of the first area.

[0019] The position of the third area, where part of the content displayed in the first area is located, can be adjusted in different ways.

[0020] In one possible embodiment, the position of the third region can be adjusted by the first device. For example, the first device can receive user input, and the gyroscope of the first device can respond and adjust the position of the third region; or, the first device can receive user taps on the display screen and adjust the position of the third region accordingly.

[0021] In another possible embodiment, the first device can adjust the position of the first area in response to a first signal received from the second device. The first signal can be sent by a remote control operated by a user, indicating a change in the position of the third area, thereby indicating a change in the interface content on the second device.

[0022] The content displayed on the first interface mentioned above includes video content.

[0023] Optionally, when at least two second interfaces are obtained from the first interface displayed on the first device, the first device can set the priority of the second interfaces according to the user's usage and send the second interfaces in order of priority. When the network is poor, one or more high-priority second interfaces can be sent to the second device first, thereby improving the smoothness of screen mirroring.

[0024] Optionally, the method further includes determining whether the displayed content of the first interface includes video content. This can be determined by analyzing the pixel information of the first interface.

[0025] Depending on whether the content displayed on the first interface includes video content, different methods can be used to process the first interface, thereby obtaining at least one second interface data.

[0026] In one possible implementation, if the content displayed on the first interface does not include video content, data from at least two second interfaces are obtained by segmenting the first interface based on the screen ratio information of the second device.

[0027] The data of the aforementioned at least two second interfaces includes: image data of each of the aforementioned at least two second interfaces.

[0028] Optionally, the data of the at least two second interfaces may further include one or more of the following: a quantity identifier, which indicates the quantity of the second interfaces; a sequence identifier, which indicates the arrangement order of each of the at least two second interfaces; and a direction identifier, which indicates the arrangement direction of each of the at least two second interfaces, or, indicates the arrangement direction of each of the at least two second interfaces in the display on the second device.

[0029] In another possible implementation, if the content displayed on the first interface includes video content, data of at least one second interface is obtained by cropping the first interface based on the screen ratio information of the second device.

[0030] The data of the at least one second interface mentioned above includes: image data of each of the at least one second interface mentioned above.

[0031] Optionally, the data of the at least one second interface may further include one or more of the following: a quantity identifier for indicating the quantity of the second interface; a sequence identifier for indicating the arrangement order of each of the at least one second interface; a direction identifier for indicating the arrangement direction of each of the at least one second interface, or for indicating the arrangement direction of each of the at least one second interface in the display of the second device.

[0032] Optionally, before sending data from at least two second interfaces, at least two second interfaces can be spliced ​​together and the background filled.

[0033] When the first device sends at least two second interfaces to the second device in sequence, it can send them in order according to the sequence identifier, direction identifier, and other information of the second interfaces, or it can send them according to the priority of the second interfaces.

[0034] There are several possibilities regarding the aspect ratio of the first interface and the aspect ratio of the second device's screen.

[0035] In one possible implementation, the aspect ratio of the first interface is greater than 1, and the aspect ratio of the screen of the second device is less than 1.

[0036] In another possible implementation, the aspect ratio of the first interface is less than 1, and the aspect ratio of the screen of the second device is greater than 1.

[0037] The aspect ratio of the first region is greater than 1, and the aspect ratio of the screen of the second device is greater than 1.

[0038] The first device mentioned above includes a mobile phone or a tablet computer; and / or, the second device mentioned above includes a television.

[0039] In a second aspect, a screen mirroring method for a second device is provided, comprising: sending screen ratio information of the second device to a first device; receiving data of at least two second interfaces sent by the first device; wherein the at least two second interfaces are obtained by dividing a first interface displayed by the first device; and splicing the at least two second interfaces together according to the data of the at least two second interfaces to display a third interface in full screen; wherein the third interface corresponds to the first interface displayed by the first device.

[0040] The above-mentioned data from at least two second interfaces are combined to display a third interface in full screen. Specifically, the above-mentioned data from at least two second interfaces and the above-mentioned filling area are combined to obtain the third interface; the third interface is then displayed in full screen.

[0041] Optionally, before splicing the at least two second interfaces and the filling area to obtain the third interface, the method further includes: determining the size of the filling area based on the data of the at least two second interfaces and the screen ratio information of the second device.

[0042] The data of the aforementioned at least two second interfaces includes: image data of each of the aforementioned at least two second interfaces.

[0043] Optionally, the data of the at least two second interfaces may also include one or more of the following: a quantity identifier for indicating the quantity of the second interfaces; a sequence identifier for indicating the arrangement order of each of the at least two second interfaces; a direction identifier for indicating the arrangement direction of the at least two second interfaces, or for indicating the arrangement direction of the at least two second interfaces in the display of the second device.

[0044] Specifically, splicing the at least two second interfaces based on the data of the at least two second interfaces includes: splicing the at least two second interfaces according to the arrangement direction indicated by the direction identifier; and / or splicing the at least two second interfaces according to the arrangement order indicated by the sequence identifier.

[0045] The content displayed on the first interface mentioned above does not include video content.

[0046] There are several possibilities regarding the aspect ratio of the first interface and the aspect ratio of the second device's screen.

[0047] In one possible implementation, the aspect ratio of the first interface is greater than 1, and the aspect ratio of the screen of the second device is less than 1.

[0048] In another possible implementation, the aspect ratio of the first interface is less than 1, and the aspect ratio of the screen of the second device is greater than 1.

[0049] The aspect ratio of the first region is greater than 1, and the aspect ratio of the screen of the second device is greater than 1.

[0050] The first device mentioned above includes a mobile phone or a tablet computer; the second device mentioned above includes a television.

[0051] Thirdly, a screen projection method for a first device is provided, comprising: displaying a first interface; wherein the first interface includes at least two second interfaces arranged in a first manner; sending data related to the first interface to a second device; wherein the data related to the first interface is used by the second device to display a third interface corresponding to the first interface in full screen, the third interface including at least two second interfaces arranged in a second manner; the first manner and the second manner are different.

[0052] Wherein, the arrangement direction of the at least two second interfaces arranged in the first manner is different from that of the at least two second interfaces arranged in the second manner, and the arrangement order of the at least two second interfaces arranged in the first manner is the same as that of the at least two second interfaces arranged in the second manner.

[0053] Depending on the different aspect ratios of the first interface and the second device's screen, there are different possible layout variations.

[0054] In one possible implementation, the aspect ratio of the first interface is greater than the aspect ratio of the screen of the second device, and the at least two second interfaces arranged in the first manner are arranged vertically and horizontally.

[0055] The aspect ratio of the first interface can be greater than 1, and the aspect ratio of the screen of the second device can be less than 1.

[0056] In another possible implementation, the aspect ratio of the first interface is smaller than that of the screen of the second device, and the at least two second interfaces arranged in the first manner are arranged horizontally, and the at least two second interfaces arranged in the second manner are arranged vertically.

[0057] The aspect ratio of the first interface may be less than 1, while the aspect ratio of the screen of the second device may be greater than 1.

[0058] Each of the at least two second interfaces mentioned above has the same size.

[0059] The relationship between the aspect ratios of the first interface, the second interface, and the second device's screen can be represented in different ways.

[0060] In one possible implementation, the difference between the aspect ratio of the first interface and the aspect ratio of the second device screen is greater than the difference between the aspect ratio of the second interface and the aspect ratio of the second device screen.

[0061] In another possible implementation, the difference between the first aspect ratio and the second aspect ratio is greater than the difference between the third aspect ratio and the second aspect ratio; wherein, the first aspect ratio is the aspect ratio of the at least two second interfaces arranged in the first manner, the second aspect ratio is the aspect ratio of the second device screen, and the third aspect ratio is the aspect ratio of the at least two second interfaces arranged in the second manner.

[0062] By changing the orientation of at least two secondary interfaces, the display area of ​​at least two secondary interfaces on the screen of the secondary device is increased, thereby improving the user experience; at the same time, by making the arrangement order of at least two secondary interfaces the same, the continuity between at least two secondary interfaces is improved, thereby improving the user experience.

[0063] Fourthly, a screen mirroring method for a second device is provided, comprising: receiving data related to a first interface displayed on the first device sent by a first device; wherein the first interface includes at least two second interfaces arranged in a first manner; displaying a third interface in full screen; wherein the third interface includes at least two second interfaces arranged in a second manner; the first manner and the second manner are different.

[0064] Wherein, the arrangement direction of the at least two second interfaces arranged in the first manner is different from that of the at least two second interfaces arranged in the second manner, and the arrangement order of the at least two second interfaces arranged in the first manner is the same as that of the at least two second interfaces arranged in the second manner.

[0065] Depending on the different aspect ratios of the first interface and the second device's screen, there are different possible layout variations.

[0066] In one possible implementation, the aspect ratio of the first interface is greater than the aspect ratio of the screen of the second device, and the at least two second interfaces arranged in the first manner are arranged vertically and horizontally.

[0067] The aspect ratio of the first interface can be greater than 1, and the aspect ratio of the screen of the second device can be less than 1.

[0068] In another possible implementation, the aspect ratio of the first interface is smaller than that of the screen of the second device, and the at least two second interfaces arranged in the first manner are arranged horizontally, and the at least two second interfaces arranged in the second manner are arranged vertically.

[0069] The aspect ratio of the first interface may be less than 1, while the aspect ratio of the screen of the second device may be greater than 1.

[0070] In this case, each of the at least two second interfaces mentioned above has the same size.

[0071] The relationship between the aspect ratios of the first interface, the second interface, and the second device's screen can be represented in different ways.

[0072] In one possible implementation, the difference between the aspect ratio of the first interface and the aspect ratio of the second device screen is greater than the difference between the aspect ratio of the second interface and the aspect ratio of the second device screen.

[0073] In another possible implementation, the difference between the first aspect ratio and the second aspect ratio is greater than the difference between the third aspect ratio and the second aspect ratio; wherein, the first aspect ratio is the aspect ratio of the at least two second interfaces arranged in the first manner, the second aspect ratio is the aspect ratio of the second device screen, and the third aspect ratio is the aspect ratio of the at least two second interfaces arranged in the second manner.

[0074] Fifthly, a screen projection system is provided, comprising: a first device capable of performing the first aspect described above; or, a second device capable of performing the second aspect described above; or, a first device capable of performing the third aspect described above; or, a second device capable of performing the fourth aspect described above.

[0075] Sixthly, a first device is provided. The first device is, for example, a mobile phone or a tablet computer, wherein the mobile phone or tablet computer may be a foldable screen mobile phone or tablet computer. The first device includes a display screen, a memory, one or more processors, multiple applications, and one or more programs; wherein the one or more programs are stored in the memory; when the one or more processors execute the one or more programs, the first device performs the method and any possible technical solution described in either the first or third aspect.

[0076] A seventh aspect provides a second device. The second device is, for example, a television. The second device includes a display screen, a memory, one or more processors, multiple application programs, and one or more programs; wherein the one or more programs are stored in the memory; and when the one or more processors execute the one or more programs, the electronic device performs the methods and any possible technical solutions described in either the second or fourth aspect.

[0077] Eighthly, a communication device is provided. The communication device may be a first device as described in any of the first and third aspects above. The communication device possesses the functions of the first device. The communication device may be, for example, the first device, a larger device including the first device, or a functional module within the first device, such as a chip system. In an optional implementation, the communication device includes a functional module for implementing the method described in any of the first and third aspects above.

[0078] A ninth aspect provides a communication device. The communication device may be a second device as described in any of the second and fourth aspects above. The communication device possesses the functions of the second device. The communication device may be, for example, a second device, a larger device including a second device, or a functional module within a second device, such as a chip system. In an optional implementation, the communication device includes a functional module for implementing the method described in any of the second and fourth aspects above.

[0079] In a tenth aspect, a computer-readable storage medium is provided, including instructions that, when executed on an electronic device, cause the electronic device to perform any of the methods and any possible technical solutions described in the first, second, third, or fourth aspects.

[0080] Eleventhly, a chip system is provided, the chip system including logic circuitry for coupling with an input / output interface to transmit data through the input / output interface, so as to perform the methods and any possible technical solutions described in any of the first, second, third, or fourth aspects.

[0081] In a twelfth aspect, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the methods and any possible technical solutions described in any of the first, second, third, or fourth aspects. Attached Figure Description

[0082] Figure 1This is a schematic diagram of the architecture of a screen mirroring system using existing technology;

[0083] Figure 2 This is a flowchart illustrating the existing synchronous screen projection method.

[0084] Figures 3A to 3B This is a schematic diagram of a scenario in the existing synchronous screen projection method;

[0085] Figure 4A A simplified schematic diagram of the system architecture of a screen projection method provided in an embodiment of the present invention;

[0086] Figure 4B This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0087] Figure 4C A software structure block diagram of an electronic device provided in an embodiment of the present invention;

[0088] Figure 5 This is a flowchart illustrating a screen projection method provided in an embodiment of the present invention;

[0089] Figure 6 A schematic diagram of the first boundary line provided in an embodiment of the present invention;

[0090] Figure 7 This is a schematic diagram of a method for sending a second interface according to priority provided in an embodiment of the present invention;

[0091] Figure 8 A schematic diagram illustrating a screen projection method provided in an embodiment of the present invention;

[0092] Figure 9 A schematic diagram illustrating the background layout method provided in an embodiment of the present invention;

[0093] Figure 10 A schematic diagram of a first interface with a height smaller than its width and a second device with a height larger than its width, provided for an embodiment of the present invention;

[0094] Figures 11 to 13C A schematic diagram illustrating another screen projection method provided in an embodiment of the present invention;

[0095] Figure 14 A schematic diagram of the interface of a first device including multiple subjects is provided for an embodiment of the present invention;

[0096] Figure 15 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0097] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention are within the scope of protection of this invention.

[0098] The terms "first," "second," and "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0099] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0100] In this application's embodiments, the term "A and / or B" merely describes the relationship between related objects, indicating that three relationships can exist: A alone, A and B simultaneously, and B alone. Additionally, in this application's embodiments, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0101] In some of the processes described in the embodiments of this application, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0102] Screen mirroring can include asynchronous screen mirroring and synchronous screen mirroring.

[0103] The asynchronous screen projection allows the first device to send the data to be projected to a second device with display capabilities, so that the user can see the content projected by the first device through the second device, and the projected content is not affected by the interface switching or other operations of the first device.

[0104] The synchronized screen projection allows a first device to send data from its displayed interface to a second device with display capabilities, enabling the user to see either the interface displayed on the first device or a processed version of the first device's interface through the second device. The content displayed on the second device changes as the interface displayed on the first device changes.

[0105] The screen mirroring method provided in this application is applicable to synchronous screen mirroring.

[0106] An interface may consist of the display content of one area on the display screen, or an interface may consist of the display content of multiple areas on the display screen, or an interface may consist of the display content of all areas on the display screen. It is understood that an interface may include multiple areas, and the display content of one or more of these multiple areas may also be considered as an interface; that is, an interface may include multiple interfaces.

[0107] For example, if interface A is displayed in regions X and Y, and the interface displayed in region X is called interface B, and the interface displayed in region Y is called interface C, then interface A can be described as including interface B and interface C. Interface B can be understood as the content displayed in region X, and interface C can be understood as the content displayed in region Y.

[0108] The first device can communicate with the second device through different methods, sending interface data to the second device. This interface data can be used by the second device to display an image of the first device's interface. The second device receives the interface data and displays the same interface as the first device at the same time. This "same time" can be the same instant or two instants with a time difference within a specific threshold.

[0109] In one implementation, when both the first device and the second device are connected to the server, the first device can send the interface data that needs to be projected to the server, and the second device can then retrieve this interface data from the server.

[0110] For example, in Figure 1 In the screen mirroring system shown, the first device 101 and the second device 102 are respectively communicatively connected to the cloud server 103. The first device 101 and the second device 102 indirectly communicate with each other through the cloud server 103. During screen mirroring, the first device 101 can send a request to the cloud server 103, requesting the cloud server 103 to send the interface data to be mirrored to the second device 102. After receiving this interface data, the second device 102 can process and display the interface to be mirrored.

[0111] In another implementation, when the first device and the second device are connected via short-range communication, the first device can directly send the interface data that needs to be projected to the second device.

[0112] For example, the first device 101 and the second device 102 can achieve direct communication through near-field communication methods, such as Bluetooth, Wi-Fi Director, or direct connection followed by screen projection based on the MiraCast protocol. The first device 101 can directly send the interface data to be projected to the second device 102. After receiving this interface data, the second device 102 can display the interface to be projected.

[0113] Figure 2 A flowchart of an existing synchronous screen mirroring method is illustrated as an example. In the existing synchronous screen mirroring method, according to the protocol, the first device obtains the resolution of the second device (S201) and the interface displayed by the first device (S202). When the resolution of the second device is different from that of the first device, the first device fills the interface displayed by the first device with a black background according to the resolution of the second device, so that the resolution of the filled interface is the same as that of the second device (S203), and sends the filled interface to the second device after encoding (S204). The second device receives and decodes the filled interface, performs black border detection on the filled interface, and performs black border cropping according to the resolution of the second device. Subsequently, the second device displays the processed interface that conforms to the resolution of the second device (S205). Typically, the second device scales up the height of the first device's interface proportionally to make the height of the first device's interface consistent with the height of the second device, and scales up the width of the first device's interface proportionally.

[0114] In existing screen mirroring methods, due to the different resolutions of the first and second devices, only a portion of the second device's screen displays the interface of the first device. For example, if the first device is a mobile phone and the second device is a television, the mobile phone's screen resolution is typically higher in height than in width, while the television's screen resolution is lower in height than in width. Due to the limitations of the display ratio, after screen mirroring, the mobile phone's interface occupies a relatively small portion of the television screen, resulting in a low actual utilization rate of the television screen.

[0115] Figure 3AAn exemplary screen mirroring scenario is illustrated, in which a mobile phone 301 and a television 302 establish a direct communication connection via near-field communication (NFC). The mobile phone 301 obtains the resolution of the television 302. The interface on the mobile phone 301 that the user wants to mirror is the entire screen, i.e., interface 305. The mobile phone 301 fills interface 305 with a black background to obtain interface 306 that meets the resolution of the television 302, and sends it to the television 302. The television 302 receives and displays interface 306 that meets the aspect ratio of the television 302. Here, interface 303 is the interface of the mobile phone 301, and interface 304 is the filled black background. Figure 3A It is evident that, using the existing screen mirroring method, the display area of ​​interface 303 on the TV 302 is relatively low, which affects the user experience interface.

[0116] Figure 3B Another screen mirroring scenario is illustrated, where a mobile phone 301 and a television 302 establish a direct communication connection via near-field communication. On the mobile phone 301, interface 303 represents the interface the user wants to mirror, and interface 304 represents the background or another interface. Specifically, the mobile phone 301 obtains the resolution of the television 302, fills its interface with a black background, obtains an interface 308 that meets the resolution of the television 302, and sends it to the television 302. The television displays interface 308, which includes interface 305 corresponding to the mobile phone interface 303, interface 306 corresponding to the mobile phone interface 304, and interface 307 with a filled black background. Therefore, the interface 305 that the user wants to view on the screen of the mirrored phone occupies a relatively small portion of the screen on the television 302, and the user cannot see the mirrored interface of the mobile phone more clearly on the television, affecting the user experience.

[0117] To address the aforementioned issues with simultaneous screen mirroring, this application provides a screen mirroring method in which a first device segments or crops the displayed interface and sends it to a second device, thereby increasing the display ratio of the interface displayed on the first device when mirrored to the screen of the second device, improving the user experience, and in some scenarios, improving data transmission efficiency.

[0118] Figure 4A A simplified schematic diagram of the system architecture for a screen projection method is shown in Figure 4. As shown, the system architecture may include at least: a first device 401 and a second device 402.

[0119] There are several possibilities regarding the aspect ratio of the screens of the first device and the second device. In one possible scenario, the aspect ratio of the first device's screen, or the aspect ratio of the displayed interface, is greater than 1, while the aspect ratio of the second device's screen is less than 1. In another possible scenario, the aspect ratio of the first device's screen, or the aspect ratio of the displayed interface, is less than 1, while the aspect ratio of the second device's screen is greater than 1. It is understood that an aspect ratio of the screen, or the aspect ratio of the displayed interface, greater than 1 can also be expressed as an aspect ratio of the screen's width and height, or the aspect ratio of the displayed interface, less than 1; conversely, an aspect ratio of the screen's width and height, or the aspect ratio of the displayed interface, less than 1 can also be expressed as an aspect ratio of the screen's width and height, or the aspect ratio of the displayed interface, greater than 1.

[0120] The first or second device can be a mobile phone, tablet computer, handheld computer, PC, cellular phone, personal digital assistant (PDA), wearable device (such as smartwatch), in-vehicle computer, game console, and augmented reality (AR) / virtual reality (VR) device, etc. This embodiment does not impose any special restrictions on the specific form of the terminal.

[0121] refer to Figure 4B The present application provides a schematic diagram of the structure of an electronic device 100.

[0122] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0123] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0124] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0125] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0126] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0127] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0128] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0129] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0130] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0131] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0132] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0133] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0134] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0135] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0136] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0137] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0138] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0139] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0140] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0141] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0142] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0143] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0144] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0145] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays screens 194, where N is a positive integer greater than 1. Display screen 194 can also be referred to as a screen.

[0146] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0147] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0148] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0149] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0150] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0151] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0152] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0153] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0154] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0155] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0156] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0157] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0158] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0159] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0160] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194. Pressure sensor 180A

[0161] There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may consist of at least two parallel plates with conductive material. When force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal from the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0162] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0163] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0164] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0165] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0166] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0167] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0168] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0169] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0170] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0171] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0172] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0173] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0174] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0175] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0176] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0177] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0178] Figure 4C This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0179] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0180] The application layer can include a series of application packages.

[0181] like Figure 4C As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0182] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0183] like Figure 4C As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0184] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0185] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0186] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0187] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0188] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0189] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0190] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0191] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0192] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0193] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0194] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0195] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0196] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0197] A 2D graphics engine is a graphics engine for 2D drawing.

[0198] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0199] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100.

[0200] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0201] Figure 5 A screen mirroring method is shown. Step S503 is optional.

[0202] S501, The first device obtains the screen ratio information of the second device.

[0203] The first device has multiple ways to obtain the screen ratio information of the second device.

[0204] In one possible implementation, the first device can connect to the second device via a first network. The first device and the second device establish a screen mirroring connection via a screen mirroring protocol, such as Miracast. Through the connection with the second device, the first device obtains the screen ratio information of the second device.

[0205] In another possible implementation, if the first device stores the screen ratio information of the second device, the first device can directly obtain the stored screen ratio information of the second device.

[0206] Screen ratio information is used to indicate the resolution ratio of the second device. This information may include the screen's resolution ratio, the ratio of its height to width (height-to-width ratio), the ratio of its width to its height (width-to-height ratio), and an identifier indicating the screen resolution ratio. For example, a resolution of 1920:1080 corresponds to a width-to-height ratio of 16:9 and a height-to-width ratio of 9:16. The identifier indicating this resolution ratio can be any set identifier or a commonly used identifier such as 1080P. For instance, identifier 01 can indicate a height-to-width ratio of 3:4, and identifier 02 can indicate a height-to-width ratio of 16:9.

[0207] This application embodiment uses the screen aspect ratio as an example (hereinafter referred to as aspect ratio). That is, the first device obtains the aspect ratio of the second device as (T... y / T x ).

[0208] S502, The first device acquires the first interface displayed by the first device.

[0209] In other words, the first interface is the interface currently displayed on the first device. The first interface may or may not include a status bar. The first interface can be an application interface or a system interface.

[0210] It is understandable that S502 may occur before S501, or after S501, or simultaneously with S501.

[0211] S503, The first device determines whether the content displayed on the first interface includes video content.

[0212] If the content displayed on the first screen includes video content, then users usually mainly watch the video content, and the video content needs to be cast as the core content in the casting process; if the content displayed on the first screen does not include video content, then users tend to watch all the content displayed on the first screen, and the entire content displayed on the first screen needs to be cast in the casting process.

[0213] By determining whether the content displayed on the first interface includes video content using the first device, the target content to be projected onto the first device can be identified. Different projection methods can then be used for different target content, thereby increasing the display ratio of the target content on the second device's screen and improving the user experience.

[0214] For example, if the content displayed on the first interface includes video content, the content to be projected to the target screen may include the video content, wherein the video content may refer to all video content or part of the video content; if the content displayed on the first interface does not include video content, the content to be projected to the target screen may include all the content displayed on the first interface.

[0215] The first device first detects whether the displayed content of the first interface has a first boundary line. The first boundary line is the boundary between the video content and non-video content. If the first interface does not have a first boundary line, then the displayed content of the first interface does not include video content. If the first interface has a first boundary line, then based on the number of first boundary lines and the displayed content of the area enclosed by the first boundary lines, it is determined whether the displayed content of the enclosed area is video content.

[0216] Figure 6 The method for detecting the first boundary line is illustrated using the top and bottom boundary lines of the video content as an example. It is understood that the first boundary line can also be the left and right boundary lines. This application does not limit the method for detecting the first boundary line.

[0217] For example, this application provides a possible first boundary line detection method. A first device compares the parameter values ​​of two adjacent rows of pixels in a first interface. The parameter values ​​can be RGB color values. A first parameter threshold c and a first quantity threshold n are set. For any pixel p located on row l0 in the first interface... o Its RGB values ​​are (r0, g0, b0). For a pixel located on row l1 and at point p... o For adjacent pixel p1, whose RGB values ​​are (r1, g1, b1), calculate the RGB values ​​of pixel p1. o The parameter comparison value c with pixel p1 01 Among them, the parameter comparison value c 01 =|r0-r1|+|g0-g1|+|b0-b1|. If pixel p o The parameter comparison value c with pixel p1 01 If the value is greater than the first parameter threshold c, then the parameter comparison value c is considered to be greater than the first parameter threshold c. 01The first parameter threshold c is exceeded. Among all pixels in row l0, a proportion of n1 pixels have parameter comparison values ​​exceeding the first parameter threshold with their adjacent pixels. If n1 is greater than the first quantity threshold n, then l0 is considered a boundary line.

[0218] Taking a first parameter threshold c = 120 and a first quantity threshold n = 80% as an example. Assume a pixel p on row l0. o The RGB values ​​of a pixel are (230, 100, 50), and the RGB values ​​of its adjacent pixel p1 in the adjacent row l1 are (0, 0, 50). Therefore, pixel p... o The parameter comparison value c with pixel p1 01 =230+100+0=330, parameter comparison value c 01 If the comparison value of a pixel in row l0 exceeds the first parameter threshold c, then the comparison value of that pixel exceeds the first parameter threshold. If the proportion of pixels in row l0 whose comparison value exceeds the first parameter threshold is n1 = 90%, then row l0 is considered the first boundary line because n1 > n. It can be understood that row l0 can be either the upper boundary line or the lower boundary line.

[0219] Similarly, the left or right boundary line can be detected. The detection of the first boundary line can be performed line by line, or multiple lines can be detected in parallel on the first interface.

[0220] When the first boundary line is not detected, it is assumed that the content displayed on the first interface does not include video content; when the first boundary line is detected, it is necessary to further judge the content displayed in the judgment area enclosed by the first boundary line to determine whether the content displayed on the first interface includes video content.

[0221] Further determine whether the displayed content in the judgment area is video content. Set a first time threshold t, and periodically judge the image in the judgment area according to the first time threshold t.

[0222] Within a first time threshold t, acquire M frames of the judgment region and calculate the parameter value of each frame in the M frames. The parameter value of each frame can be the average of the parameter values ​​of all pixels in the judgment region. The parameter value can be an RGB color value.

[0223] Compare the parameter values ​​of pixels in two adjacent frames to obtain parameter comparison values. For M frames of pixels, obtain (M-1) parameter comparison values. Record the number of parameter comparison values ​​greater than a set second parameter threshold as r0. If the number of parameter comparison values ​​r is greater than the set second threshold r, then the displayed content of the judgment area is considered to be video content.

[0224] For example, within a first time threshold t = 1 second, 10 frames of the judgment area are acquired, and a second quantity threshold r = 6. The parameter values ​​for each frame are calculated, and the parameter values ​​of adjacent frames are compared to obtain 9 parameter comparison values. The number of parameter comparison values ​​exceeding the set second parameter threshold, r0 = 7, is determined. Since r0 > r, and the number of parameter comparison values ​​exceeds the set second quantity threshold, the displayed content of the judgment area is considered to be video content, and therefore the displayed content of the first interface includes video content.

[0225] S504. Based on the screen ratio information of the second device, segment or crop the first interface to obtain data of at least one second interface.

[0226] The first device determines the area within the first interface that needs to be projected based on its content. By segmenting or cropping the first interface, it obtains data for at least one second interface. This second interface is then sent to the second device for processing and display.

[0227] When the content displayed on the first screen does not include video content, such as Figure 3A As shown, the content displayed on the first interface of the first device 301 is mostly static content such as images and text. The first device 301 can use the content of the first interface as the content to be projected and displayed.

[0228] The first device can divide the first interface into multiple second interfaces based on the screen ratio information of the second device. These multiple second interfaces are located within the first interface, and their aspect ratios may be the same or different. This will be discussed in conjunction with... Figure 8 The method of splitting the first interface to obtain multiple second interfaces will not be elaborated here.

[0229] When the content displayed on the first screen includes video content, such as Figure 3B As shown, the first device 301 can use the video content 303 as the content to be projected.

[0230] The first device can crop the first interface and obtain the data for the second interface based on the screen ratio information of the second device and the ratio information of the first area where the video content is located. The following will combine... Figures 11 to 13C The method for cropping the first interface to obtain the second interface will be introduced, and will not be repeated here.

[0231] S505, the first device sends data from at least one second interface to the second device.

[0232] The first device encodes data from at least one second interface and sends it to the second device.

[0233] It is understandable that when the content displayed on the first interface does not include video content, at least one second interface can refer to one second interface or at least two second interfaces; when the content displayed on the first interface does not include video content, at least one second interface can refer to at least two second interfaces.

[0234] The data of at least one second interface may include image data of each of the at least one second interface.

[0235] When the first device segments the first interface to obtain data for at least two second interfaces based on the screen ratio information of the second device, the data for the at least two second interfaces may include image data for each of the at least two second interfaces, and may also include a quantity identifier, a sequence identifier, and a direction identifier. The quantity identifier indicates the number of second interfaces; the sequence identifier indicates the arrangement order of each of the at least two second interfaces; and the direction identifier indicates the arrangement direction of each of the at least two second interfaces in the first interface, or, more specifically, the arrangement direction of each of the at least two second interfaces in the display on the second device.

[0236] For example, the first device divides the first interface into two second interfaces, upper and lower, based on the screen ratio information of the second device. The quantity identifier can be 2, the sequence identifier of the upper second interface can be 1, and the direction identifier can be "up" indicating the position of the second interface within the first interface, or "left" indicating the position of the second interface in the display on the second device. Similarly, the sequence identifier of the lower second interface can be 2, and the direction identifier can be "down" indicating the position of the second interface within the first interface, or "right" indicating the position of the second interface in the display on the second device.

[0237] When the first device obtains data for a second interface by cropping the first interface based on the screen ratio information of the second device, the data for the second interface may include image data of the second interface. When the first device obtains data for at least two second interfaces by cropping the first interface based on the screen ratio information of the second device, the data for at least two second interfaces may include image data of each of the at least two second interfaces, and may also include quantity identifiers, order identifiers, and direction identifiers.

[0238] When at least two second interfaces are obtained from the first interface displayed by the first device, the first device can send the at least two second interfaces to the second device simultaneously, or send them to the second device sequentially.

[0239] When the first device sends at least two second interfaces to the second device in sequence, it can send them in order according to the sequence identifier, direction identifier, and other information of the second interfaces, or it can send them according to the priority of the second interfaces.

[0240] The first device can prioritize the second interface based on user usage and send the second interfaces sequentially according to priority. For example, the first device can set the second interface showing user usage as a high priority and set the rest of the third interfaces as low priority. When the network is poor, one or more high-priority second interfaces can be sent to the second device first, thereby improving the smoothness of screen mirroring. The "user usage" second interface can include, for example, a second interface for user interaction.

[0241] For example, such as Figure 7 As shown, for the first interface 701 displayed by the first device 70, the first device obtains the second interface 702 and the second interface 703 from the first interface. In the first interface 701, the user uses the virtual keyboard in the second interface 703 to input text and views the input text through the second interface 702. At this time, the first device can set the second interface 702 as high priority and the second interface 703 as low priority. The first device sends the second interface 702 and the second interface 703 to the second device in sequence, so that when the network is poor, the user can see the content of the second interface 702 first and then view the input text, improving the smoothness of screen projection and user experience.

[0242] S506, The second device receives data from at least one second interface, and acquires and displays the third interface in full screen.

[0243] The second device receives and decodes data from at least one second interface. Based on the data from the at least one second interface, the second device can acquire a third interface with the same aspect ratio as the screen of the second device. The third interface includes the at least one second interface, and the content of the third interface corresponds to the content of the first interface.

[0244] When the second device receives data from a second interface, the second device can scale the second interface to obtain a third interface that can be displayed in full screen on the second device, and then the second device will display the third interface in full screen. Figure 8 (c), Figure 11 (c), Figure 12A (d) etc.

[0245] When the second device receives data from at least two second interfaces, the second device can stitch the at least two second interfaces together.

[0246] Optionally, the second device can splice the at least two second interfaces according to the direction identifiers in the data of the at least two second interfaces, in accordance with the direction identifiers indicating the direction of each second interface.

[0247] Optionally, the second interface can be assembled according to the order identifier in the data of the at least two second interfaces, in the order indicated by the order identifier of each second interface.

[0248] The arrangement direction of at least two second interfaces in the third interface can be different from the arrangement direction in the first interface. This ensures that the difference between the aspect ratio of the first interface and the aspect ratio of the second device screen is greater than the difference between the aspect ratio of the second interface and the aspect ratio of the second device screen; or, the difference between the first aspect ratio and the second aspect ratio is greater than the difference between the third aspect ratio and the second aspect ratio, thereby increasing the display area of ​​the two second interfaces on the second device.

[0249] Wherein, the first aspect ratio is the aspect ratio of the at least two second interfaces laid out in the first manner, the second aspect ratio is the aspect ratio of the second device screen, and the third aspect ratio is the aspect ratio of the at least two second interfaces laid out in the second manner.

[0250] At the same time, the layout order of at least two second interfaces in the third interface can be the same as the layout order in the first interface, which improves the continuity between the second interfaces and enhances the user experience.

[0251] For example, such as Figure 8 As shown, the aspect ratio of the first interface 801 is greater than 1, while the aspect ratio of the second device's screen is less than 1. Dividing the first interface 801 into a second interface 802 and a second interface 803, it can be understood that within the first interface 801, interfaces 802 and 803 are arranged vertically, specifically, the second interface 802 is located on the top and the second interface 803 is located on the bottom. The second device can arrange the two second interfaces sequentially according to their order identifiers; alternatively, it can arrange the second interface 802 on the left and the second interface 803 on the right according to their direction identifiers, so that the arrangement direction of the two second interfaces in the third interface differs from their arrangement direction in the first interface, thereby increasing the display area of ​​the two second interfaces on the second device.

[0252] Similarly, for cases where the aspect ratio of the first interface is less than 1, and the aspect ratio of the second device's screen is greater than 1, the first interface can be divided into two second interfaces arranged horizontally. The second device can then arrange the two second interfaces in sequence according to their order identifiers, resulting in two second interfaces arranged vertically within the third interface.

[0253] The second device can compare the aspect ratios of at least two spliced ​​second interfaces with the aspect ratio of the second device's screen.

[0254] If the aspect ratios of at least two spliced ​​second interfaces are the same as those of the second device's screen, the spliced ​​at least two second interfaces can be scaled to obtain a third interface that can be displayed in full screen on the second device, and then the second device can display the third interface in full screen.

[0255] If the aspect ratios of at least two spliced ​​second interfaces are different from those of the screen of the second device, the at least two spliced ​​second interfaces can be scaled so that the width of the at least two spliced ​​second interfaces is the same as the width of the screen of the second device, or the height of the at least two spliced ​​second interfaces is the same as the height of the screen of the second device.

[0256] Based on the aspect ratio of the second device's screen, fill the background of at least two spliced ​​second interfaces with a background, such as a black background, to obtain a third interface that can be displayed in full screen on the second device, and then display the third interface in full screen on the second device.

[0257] By using a second device to stitch together at least two second interfaces and fill the background, the amount of data sent from the first device to the second device can be reduced, the proportion of data related to the second interface in the total amount of data sent can be increased, and the data transmission efficiency in screen projection can be improved.

[0258] For example, such as Figure 8 As shown in (c), the aspect ratio of the combined third interface 803 and third interface 804 is different from that of the second device's screen. Therefore, the second device fills the black background 804 according to the aspect ratio of its own screen, resulting in a third interface 805 with the same aspect ratio as the second device's screen, which can be displayed in full screen on the second device. The third interface 805 maximizes the display of the two second interfaces in the first interface displayed on the first device.

[0259] In the above embodiments, the example of splicing at least two second interfaces obtained by dividing the first interface on the second device and filling the background is used for illustration. It can be understood that at least two interfaces obtained by splicing the first interface on the first device and filling the background can also be used to obtain a second interface and then sent to the second device for display. In this case, it can be understood that the second interface is the third interface.

[0260] Among them, the background filling method is as follows: Figure 9 As shown. Figure 8 Taking the second interface 802 and the second interface 803 as examples, the second device fills the black background 804 according to the aspect ratio of the screen of the second device. The black background 804 can be located on the top, bottom, or both sides of the second interface 802 and the second interface 803. It is understood that for second interfaces with different aspect ratios, the black background can also be located on the left, right, or both sides of the second interface, and this application embodiment does not limit this.

[0261] It is understood that the above embodiments use a black background as an example for illustration, but it is also understood that the background can be other colors or patterns.

[0262] When the content displayed on the first interface does not include video content, the content displayed on the first interface by the first device is mostly static content such as images and text. The first device can segment the first interface according to the screen ratio information of the second device to obtain multiple second interfaces. This application embodiment provides a possible segmentation method.

[0263] The aspect ratio of the first interface displayed on the first device is (S) y / S x The aspect ratio of the screen of the second device is (T) y / T x For example, in one possible calculation method, the aspect ratio of the screens of the first interface and the second device is used to calculate the result.

[0264] When k′>1, the aspect ratio of the first interface is greater than the aspect ratio of the second device's screen, for example, as Figure 4A As shown, the aspect ratio of the first interface is greater than 1, while the aspect ratio of the second device's screen is less than 1. The first device can segment the first interface based on its height to obtain multiple second interfaces. When k′<1, the aspect ratio of the first interface is less than the aspect ratio of the second device's screen, for example, as... Figure 10 As shown, the aspect ratio of the first interface is less than 1, while the aspect ratio of the second device's screen is greater than 1. The first device can divide the first interface by its width to obtain multiple second interfaces.

[0265] When k′>1, round k′ up to obtain the number of segments to be divided, k1; round k′ down to obtain the number of segments to be divided, k2. Here, k1 and k2 are positive integers greater than or equal to 1. If the second interface is divided using the number of segments to be divided, k1 second interfaces are obtained. The width of each second interface displayed on the second device is... If the first interface is divided into k2 segments, k2 second interfaces are obtained. The width of each second interface displayed on the second device is...

[0266] To increase the display area of ​​the second interface on the second device, the larger width between width L1 and width L2 is selected as the width L of the resulting second interface, and the corresponding number of segments is determined as k. Here, the number of segments k is the number of segments used to divide the second interface; after segmentation, k second interfaces can be obtained.

[0267] It is understandable that if L1 ≥ L2, then the number of partitions k = k1, dividing the second interface into k1 second interfaces, each with a width of... If L1 ≤ L2, then the number of partitions k = k2, dividing the first interface into k2 second interfaces, each with a width of [missing value].

[0268] For example, such as Figure 8 As shown in (a), the first device 80 displays a first interface 801, the aspect ratio of which is (S). y / S x =16 / 9, the aspect ratio of the second device's screen is (T) y / T x When k = 9 / 16, the formula calculates k′ = 16 / 9. Rounding k′ up gives the number of segments to be divided, k1 = 2, corresponding to the width... Rounding k′ down gives the number of segments to be divided, k2 = 1, corresponding to the width. Since L1 ≥ L2, the partition number k = k1 = 2. For example... Figure 8 As shown in (b), the first interface is divided into two second interfaces, namely second interface 802 and second interface 803, and the width of each second interface is...

[0269] Similarly, when k′<1, for Round up to the nearest integer to obtain the number of segments to be divided, k1. Rounding down, we get the number of segments to be divided, k2. The height of the third interface corresponding to the number of segments to be divided, k1, is... The height of the third interface corresponding to the number of segments k2 is To increase the display area of ​​the second interface on the second device, the larger height between height H1 and height H2 is selected as the height H of the resulting second interface, and the corresponding number of segments is determined as k. The specific derivation process will not be elaborated here.

[0270] It is understood that, based on the determined number of segments k, the first interface can be divided uniformly or non-uniformly. That is, the height of each second interface can be the same or different, and the width of each second interface can be the same or different. This application embodiment does not impose any restrictions.

[0271] When the content displayed on the first interface includes video content, the first device can crop the first interface and obtain the data for the second interface based on the screen ratio information of the second device and the ratio information of the first area where the video content is located. Figure 11 As shown.

[0272] The area in the first interface that displays the video content is the first area. When the aspect ratio of the first area is greater than 1 and the aspect ratio of the screen of the second device is greater than 1; or when the aspect ratio of the first area is less than 1 and the aspect ratio of the screen of the second device is less than 1, the first device can project the video content onto the screen of the second device with a larger display ratio.

[0273] The second interface may include all the content displayed in the first area, or the second interface may include part of the content displayed in the first area.

[0274] With the aspect ratio of the second device's screen as (T) y / T x For example, the aspect ratio of the second interface is the same as that of the second device's screen; that is, the aspect ratio of the second interface is (T...). y / T x ).

[0275] For an aspect ratio of (T) y / T x The second interface and aspect ratio are (S) y / S x The first area of ​​the first area. This application provides a possible determination method, and processes the interface according to this method to increase the display ratio of the content in the first area on the screen of the second device.

[0276] like The aspect ratio of the first area is the same as that of the second device's screen. The second interface displays all the content of the first area, and the data of the first area is used as the data for the second interface. For example... Figure 11 As shown, the first device 110 acquires data from the first area 1102 as data for the second interface and sends it to the second device for processing and display.

[0277] like This application provides various possible processing methods in its embodiments.

[0278] In one possible implementation, when the width of the first area and the width of the second interface are the same, the height of the second interface is greater than the height of the first area. The second interface may include the video content displayed in the first area and the content displayed in the second area. The second area is located within the first interface, and the content displayed in the second area may be text and images, background, etc.

[0279] like Figure 12A As shown, in the first interface 1201 displayed by the first device 120, area 1202 is a schematic area with the same aspect ratio as the second interface, and the second interface may include the display content of the first area 1203.

[0280] The first device can be based on the aspect ratio (V) of the first region. y / V x ) and the aspect ratio of the second interface (T) y / T x Determine the aspect ratio (B) of the second region. y / B x The method for calculating the aspect ratio of the second region is not limited in this application embodiment. This application embodiment provides a possible calculation method: the method is based on the fact that the width of the first region is the same as the width of the second interface, then...

[0281] The location of the second area can be set by default by the first device or by the user; this embodiment of the application does not limit this. For example, the second area can be set by default by the first device to be a text and image area located within the first interface but outside the first area. If the aspect ratio of the second area is greater than that of the text and image area, then the second area includes the content of the text and image area; if the aspect ratio of the second area is less than that of the text and image area, then the second area includes a portion of the content of the text and image area. The location of the second area within the text and image area is determined based on the aspect ratio of the second area.

[0282] The image and text area can be determined by detecting the parameter values ​​of the pixels on the first interface. These parameter values ​​can be RGB color values.

[0283] The method for detecting text and image regions can refer to the method for detecting the first boundary line, and will not be elaborated here. For example, Figure 12A As shown, region 1202 may include a first region 1103 and a region 1104. The aspect ratio of region 1202 is the same as that of the second interface, and the aspect ratio of region 1204 is the same as that of the second region. Let the aspect ratio of the second interface be (T... y / T x =11 / 16, the aspect ratio of the first region is (V) y / V x Taking 9 / 16 as an example, the aspect ratio of the second region is calculated using the formula (B). y / B x = 2 / 16. According to the default settings of the first device, the first device detects the graphic and text area in the first interface and obtains the area 1205, which includes the graphic and text area content, as the second area.

[0284] The first device can scale and stitch together the video content displayed in the first area and the non-video content displayed in the second area to obtain a second interface 1206 with the same aspect ratio as the second device, and send it to the second device for display, such as... Figure 12A As shown in (c)(d).

[0285] Optionally, to facilitate user access or to determine the location and size of the second area, it can be labeled. For example, a border or color can be used to identify the second area.

[0286] In another possible implementation, when the height of the first region and the height of the second interface are the same, the width of the second interface is smaller than the width of the first region. The second interface may include a portion of the display content of the first region. The first device can obtain the display content of a third region within the first region as the content of the second interface based on the aspect ratio of the second interface. Figure 12B As shown, the third region 1202 in the first region 1201 is determined according to the aspect ratio of the second interface. The display content of the third region 1202 is cropped by the first device to be the display content of the second interface 1204. That is to say, the second interface 1204 does not include the display content of region 1203.

[0287] The size of the third region can be determined based on the height or width of the first region. The user can adjust the position of the third region according to the determined position of the first region, keeping the size of the third region unchanged, or, while maintaining the aspect ratio of the third region, scale the size of the third region.

[0288] Optionally, the location of the third region can be indicated by color or borders.

[0289] The third region can be adjusted by the first device according to the user's operation, or by the first device receiving a first signal from the second device.

[0290] The third area can be adjusted by the first device based on user input. In one possible implementation, based on the user adjusting the posture of the first device, the first device can detect changes in posture using a gyroscope and adjust the position of the third area. The first device can detect the user tilting the device backward using a gyroscope and control the third area to move downward within the first area; the first device can detect the user tilting the device forward using a gyroscope and control the third area to move upward within the first area. In another possible implementation, the position of the third area can be adjusted based on the user tapping the screen of the first device.

[0291] The third area can be adjusted by the first device receiving a first signal from the second device. The first device can adjust the position of the first area in response to the received first signal from the second device, wherein the first signal can be sent by the second device at the instruction of the user to operate the second device remote control. The first signal indicates the change in the position of the third area, thereby indicating the change in the interface content in the second device.

[0292] In another possible implementation, the width of the first region and the width of the second interface are different, and the height of the first region and the height of the second interface are also different. For example... Figure 12C As shown, the second interface may include part of the content displayed in the first area and the content displayed in the second area.

[0293] The area 1203 with the same aspect ratio as the second interface is determined based on the aspect ratio of the second interface. The aspect ratio of the second area is determined based on the size and position of area 1203 and the first area 1202. The position of the second area can be the graphic area 1205 in the first interface. The display content of the clipping area 1206 of the first device is the content of the second interface 1206, and the display content of the second area 1205 is the display content of the second interface. That is to say, the second interface 1206 does not include the display content of area 1207. The specific clipping method is similar. Figure 12A and Figure 12B The methods will not be elaborated here.

[0294] like This application provides various possible processing methods in its embodiments.

[0295] In one possible implementation, when the width of the first region and the width of the second interface are the same, the height of the second interface is less than the height of the first region. The second interface may include a portion of the display content of the first region. The first device can obtain the display content of a third region within the first region as the content of the second interface based on the aspect ratio of the second interface. Figure 13A As shown, the third region 1302 in the first region 1303 is determined according to the aspect ratio of the second interface, and the first device can obtain the display content of the third region 1302 as the display content of the second interface 1304.

[0296] The description of the third region is the same as hour, Figure 12B Description of the third region.

[0297] In another possible implementation, when the height of the first area and the height of the second interface are the same, the width of the second interface is greater than the width of the first area. The second interface may include the video content displayed in the first area and the content displayed in the second area. The second area is located within the first interface, and the content displayed in the second area may be text and images, background, etc.

[0298] like Figure 13B As shown, in the first interface 1301 displayed by the first device 130, area 1302 is a schematic area with the same aspect ratio as the second interface, and the second interface may include the display content of the first area 1303.

[0299] The first device can be based on the aspect ratio (V) of the first region.y / V x ) and the aspect ratio of the second interface (T) y / T x Determine the aspect ratio (B) of the second region. y / B x The method for calculating the aspect ratio of the second region is not limited in this application embodiment. This application embodiment provides a possible calculation method: the method is based on the fact that the height of the first region is the same as the height of the second interface, then... The method for determining the second region is similar to... hour, Figure 12A The methods described above will not be elaborated here.

[0300] The first device stitches together the video content displayed in the first area 1303 and the graphic content displayed in the second area 1305 to obtain a second interface 1306 with the same aspect ratio as the second device, and sends it to the second device for display, such as... Figure 13B (c)(d).

[0301] In another possible implementation, the width of the first region and the width of the second interface are different, and the height of the first region and the height of the second interface are also different. For example... Figure 13C As shown, the second interface may include part of the display content of the first area and the display content of the second area. A region 1304 with the same aspect ratio as the second interface is determined based on the aspect ratio of the second interface. The aspect ratio of the second region is determined based on the size and position of region 1304 and the first region 1302. The position of the second region can be the graphic area 1305 in the first interface. The display content of the first device clipping region 1303 becomes the content of the second interface 1306, and the display content of the second region 1305 becomes the display content of the second interface. That is, the second interface 1306 does not include the display content of region 1307. The specific clipping method is similar. Figure 13A and Figure 13B The methods will not be elaborated here.

[0302] The above embodiments illustrate the division of the first interface when the displayed content of the first interface does not include video content, and the cropping of the first interface when the displayed content of the first interface includes video content. It can be understood that the first interface can also be cropped when the displayed content of the first interface does not include video content, and the first interface can also be divided when the displayed content of the first interface includes video content. This application embodiment does not impose any limitations.

[0303] The first device may include one or more main bodies, which may be equipped with a screen. Figures 8 to 13C Taking the first device, which includes a main body, as an example for illustration, Figure 14 The following explanation will be based on the example of a first device comprising multiple main components.

[0304] For example, taking a first device comprising two main bodies, a first main body and a second main body, the first main body can be bent relative to the second main body. When the first device is in a folded state 1401, that is, when the first main body is bent relative to the second main body, the screens of the first main body and the second main body can be regarded as two independent screens or two interconnected screens. The screen projection method of the first device is the same as that of the first device comprising a single main body.

[0305] When the first device is in the unfolded state 1402, that is, when the first body is not bent relative to the second body, the screen of the first body and the screen of the second body can be regarded as "two" screens and the method described herein can be applied, or they can be regarded as one screen and the method described herein can be applied.

[0306] When the screens of the first and second main bodies are treated as "two" screens, the content displayed in area 1403 can be projected as the first interface, or the content displayed in area 1404 can be projected as the first interface.

[0307] When the screens of the first subject and the second subject are treated as one screen, the display content of area 1402, which includes area 1403 and area 104, can be regarded as the first interface. In this case, the screen projection method can be the same as that of the first device which includes one subject.

[0308] The above embodiments use a method of segmenting or cropping the first interface to obtain data for at least one second interface. It is understood that the same method can be used to obtain data for at least one second interface. When the first interface is segmented and cropped, the display content in areas 1402 and 1403 of the first interface can be segmented or cropped to obtain data for at least one second interface, which is then sent to the second device for screen mirroring.

[0309] This application embodiment can divide an electronic device including a first device and a second device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. For example, in the case of dividing each functional module according to its own function, Figure 15A schematic diagram of an electronic device is shown. The electronic device 1500 may be the first device or the second device involved in the above-described method embodiments, or it may be a chip in the first device or the second device. The electronic device 1500 includes a transmitting unit 1501, a processing unit 1502, and a receiving unit 1503.

[0310] It should be understood that the electronic device 1500 can be used to implement the steps performed by the first device or the second device in the method of the embodiments of this application. The relevant features can be referred to the various embodiments above, and will not be repeated here.

[0311] Optional, Figure 15 The functions / implementation processes of the transmitting unit 1501, receiving unit 1503, and processing unit 1502 can be understood through... Figure 4B The processor 110 calls computer execution instructions stored in memory 121 to implement the function. Alternatively, Figure 15 The function / implementation process of the processing unit 1502 can be achieved through... Figure 4B The processor 110 in the memory calls computer execution instructions stored in the memory 121 to implement this. Figure 15 The functions / implementation process of the transmitting unit 1501 and the receiving unit 1503 can be obtained through Figure 4B This is achieved through the mobile communication module 150 or the wireless communication module 160.

[0312] Optionally, when the electronic device 1500 is a chip or circuit, the functions / implementation of the transmitting unit 1501 and the receiving unit 1503 can also be implemented through pins or circuits, etc.

[0313] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the methods performed by the first or second device in the aforementioned method embodiments. Thus, the functions described in the above embodiments can be implemented as software functional units and sold or used as independent products. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to it, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0314] This application also provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method executed by the first device or the second device in any of the foregoing method embodiments.

[0315] This application also provides a processing apparatus, including a processor and an interface; the processor is used to execute the method executed by the first device or the second device involved in any of the above method embodiments.

[0316] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0317] The various illustrative logic units and circuits described in the embodiments of this application can be implemented or operate the described functions using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0318] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be disposed in an ASIC, which can be disposed in the terminal device. Optionally, the processor and storage medium can also be disposed in different components of the terminal device.

[0319] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0320] Although embodiments of this application have been described in conjunction with specific features and examples, it is obvious that various modifications and combinations can be made thereto without departing from the scope of the embodiments of this application. Accordingly, the embodiments and drawings of this application are merely exemplary illustrations of the embodiments of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the embodiments of this application. Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the scope of the embodiments of this application. Thus, if these modifications and variations of the embodiments of this application fall within the scope of the claims of the embodiments of this application and their equivalents, then the embodiments of this application are also intended to include these modifications and variations.

Claims

1. A screen projection method, applied to a first device, characterized in that, include: Obtain the screen ratio information of the second device; Obtain the first interface displayed by the first device; Based on the screen ratio information of the second device, data of at least one second interface is obtained by segmenting or cropping the first interface; Send data of at least one second interface to the second device; wherein, the data of at least one second interface is used by the second device to display a third interface corresponding to the first interface in full screen. The step of obtaining data for at least one second interface by segmenting or cropping the first interface based on the screen ratio information of the second device includes: if the display content of the first interface includes video content, obtaining data for at least one second interface by cropping the first interface based on the screen ratio information of the second device; if the display content of the first interface does not include video content, obtaining data for at least one second interface by segmenting the first interface based on the screen ratio information of the second device.

2. The method according to claim 1, characterized in that, The step of obtaining data for at least one second interface by segmenting the first interface based on the screen ratio information of the second device includes: Based on the screen ratio information of the second device and the ratio information of the first interface, data of at least two second interfaces are obtained by segmenting the first interface.

3. The method according to claim 2, characterized in that, The number of at least two second interfaces is determined based on the screen ratio information of the second device and the ratio information of the first interface.

4. The method according to claim 1, characterized in that, The step of obtaining data for at least one second interface by cropping the first interface based on the screen ratio information of the second device includes: Identify the first region in the first interface; Based on the screen ratio information of the second device and the ratio information of the first area, data of at least one second interface is obtained by cropping the first interface; wherein, the at least one second interface includes the display content of the first area.

5. The method according to claim 4, characterized in that: The at least one second interface includes all the display content of the first area; or, The at least one second interface includes a portion of the content displayed in the first area.

6. The method according to claim 4, characterized in that: The screen ratio information of the second device indicates the same ratio as the ratio information of the first area, and the at least one second interface is the entire display content of the first area.

7. The method according to claim 4, characterized in that: The screen ratio information of the second device indicates a different ratio than the ratio information of the first area. The at least one second interface includes video content displayed in the first area and non-video content displayed in the second area; wherein the second area is located within the first interface.

8. The method according to claim 4, characterized in that: The screen ratio information of the second device indicates a different ratio than the ratio information of the first area, and the at least one second interface is a partial display of the content of the first area.

9. The method according to any one of claims 1-3 and 4-8, characterized in that, The data of the at least one second interface includes: Image data of each of the at least one second interface.

10. The method according to claim 9, characterized in that, The data of the at least one second interface also includes one or more of the following: A quantity identifier is used to indicate the quantity of the at least one second interface; Sequence identifier, used to indicate the arrangement order of each of the at least one second interface; A direction indicator is used to indicate the arrangement direction of the at least one second interface in the first interface, or to indicate the arrangement direction of the at least one second interface in the display of the second device.

11. The method according to any one of claims 1-3 and 4-8, characterized in that: The aspect ratio of the first interface is greater than 1, and the aspect ratio of the screen of the second device is less than 1; or, The aspect ratio of the first interface is less than 1, while the aspect ratio of the screen of the second device is greater than 1.

12. The method according to any one of claims 4-8, characterized in that: The aspect ratio of the first region is greater than 1, and the aspect ratio of the screen of the second device is greater than 1.

13. The method according to any one of claims 1-3 and 4-8, characterized in that: The first device includes: a mobile phone or a tablet computer; and / or, The second device includes: a television.

14. A screen projection method, applied to a second device, characterized in that, include: Send the screen ratio information of the second device to the first device; The system receives data from at least two second interfaces sent by the first device; wherein the at least two second interfaces are obtained by segmenting or cropping the first interface displayed by the first device; the data of the second interfaces are obtained by the first device by cropping the first interface according to the screen ratio information of the second device when the displayed content of the first interface includes video content; and by the first device by segmenting the first interface according to the screen ratio information of the second device when the displayed content of the first interface does not include video content. Based on the data from the at least two second interfaces, the at least two second interfaces are spliced ​​together to display a third interface in full screen; wherein the third interface corresponds to the first interface displayed by the first device.

15. The method according to claim 14, characterized in that, The step of splicing the at least two second interfaces together based on the data from the at least two second interfaces to display the third interface in full screen includes: The third interface is obtained by splicing together the at least two second interfaces and the filled area; The third interface is displayed in full screen.

16. The method according to claim 15, characterized in that, Before splicing the at least two second interfaces and the filled area to obtain the third interface, the method further includes: The size of the filling area is determined based on the data from the at least two second interfaces and the screen ratio information of the second device.

17. The method according to any one of claims 14-16, characterized in that, The data from the at least two second interfaces includes: Image data for each of the at least two second interfaces.

18. The method according to claim 17, characterized in that, The data from the at least two second interfaces also include one or more of the following: Quantity identifier, used to indicate the quantity of the second interface; Sequence identifier, used to indicate the arrangement order of each of the at least two second interfaces; A direction indicator is used to indicate the arrangement direction of the at least two second interfaces, or to indicate the arrangement direction of the at least two second interfaces in the display of the second device.

19. The method according to claim 18, characterized in that, The step of splicing the at least two second interfaces based on the data from the at least two second interfaces includes: According to the arrangement direction indicated by the direction markers, the at least two second interfaces are spliced ​​together; and / or, The at least two second interfaces are assembled according to the arrangement order indicated by the sequence identifier.

20. The method according to any one of claims 14-16, characterized in that: The aspect ratio of the first interface is greater than 1, and the aspect ratio of the screen of the second device is less than 1; or, The aspect ratio of the first interface is less than 1, while the aspect ratio of the screen of the second device is greater than 1.

21. The method according to any one of claims 14-16, characterized in that: The first device includes: a mobile phone or a tablet computer; The second device includes: a television.

22. A screen projection method, applied to a first device, characterized in that, include: Display a first interface; wherein the first interface includes at least two second interfaces laid out in a first manner; Obtaining data related to the first interface includes: when the displayed content of the first interface includes video content, obtaining it by cropping the first interface according to the screen ratio information of the second device; when the displayed content of the first interface does not include video content, obtaining it by segmenting the first interface according to the screen ratio information of the second device. Send data related to the first interface to the second device; wherein the data related to the first interface is used by the second device to display a third interface corresponding to the first interface in full screen, the third interface including the at least two second interfaces laid out in a second manner; the first manner and the second manner are different.

23. A screen projection method, applied to a second device, characterized in that, include: The system receives data related to a first interface displayed on the first device, sent by the first device. The first interface includes at least two second interfaces laid out in a first manner. The data related to the first interface is obtained by the first device by cropping the first interface according to the screen ratio information of the second device when the displayed content of the first interface includes video content. When the displayed content of the first interface does not include video content, the first device obtains the data by segmenting the first interface according to the screen ratio information of the second device. A third interface is displayed in full screen; wherein the third interface includes at least two second interfaces laid out in a second manner; the first manner and the second manner are different.

24. The method according to claim 22 or 23, characterized in that: The arrangement direction of the at least two second interfaces laid out in the first manner is different from the arrangement direction of the at least two second interfaces laid out in the second manner; The arrangement order of the at least two second interfaces laid out in the first manner is the same as the arrangement order of the at least two second interfaces laid out in the second manner.

25. The method according to claim 24, characterized in that: The aspect ratio of the first interface is greater than that of the screen of the second device. The at least two second interfaces laid out in the first manner are arranged vertically, and the at least two second interfaces laid out in the second manner are arranged horizontally.

26. The method according to claim 25, characterized in that: The aspect ratio of the first interface is greater than 1, while the aspect ratio of the screen of the second device is less than 1.

27. The method according to claim 24, characterized in that: The aspect ratio of the first interface is smaller than that of the screen of the second device. The at least two second interfaces laid out in the first manner are arranged horizontally, and the at least two second interfaces laid out in the second manner are arranged vertically.

28. The method according to claim 27, characterized in that: The aspect ratio of the first interface is less than 1, while the aspect ratio of the screen of the second device is greater than 1.

29. The method according to claim 22 or 23, characterized in that: Each of the at least two second interfaces has the same size.

30. The method according to claim 29, characterized in that: The difference between the aspect ratio of the first interface and the aspect ratio of the second device screen is greater than the difference between the aspect ratio of the second interface and the aspect ratio of the second device screen.

31. The method according to claim 29, characterized in that: The difference between the first aspect ratio and the second aspect ratio is greater than the difference between the third aspect ratio and the second aspect ratio; Wherein, the first aspect ratio is the aspect ratio of the at least two second interfaces laid out in the first manner, the second aspect ratio is the aspect ratio of the second device screen, and the third aspect ratio is the aspect ratio of the at least two second interfaces laid out in the second manner.

32. A screen projection system, characterized in that, Including the first device and the second device: The first device is configured to perform the method as described in any one of claims 1-13; or, The second device is used to perform the method as described in any one of claims 14-21; or, The first device is configured to perform the method as described in any one of claims 22, 24-31; or, The second device is used to perform the method as described in any one of claims 23-31.

33. A first device, comprising a screen, a memory, one or more processors, and one or more programs; wherein the one or more programs are stored in the memory; characterized in that, When the one or more processors execute the one or more programs, they cause the first device to perform the method as described in any one of claims 1-13, or to perform the method as described in any one of claims 22, 24-31.

34. A second device comprising a screen, a memory, one or more processors, and one or more programs; wherein the one or more programs are stored in the memory; characterized in that, When the one or more processors execute the one or more programs, they cause the second device to perform the method as described in any one of claims 14-21, or to perform the method as described in any one of claims 23-31.

35. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device performs the method as described in any one of claims 1-13, or performs the method as described in any one of claims 14-21, or performs the method as described in any one of claims 22, 24-31, or performs the method as described in any one of claims 23-31.

36. A computer program product containing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device performs the method as described in any one of claims 1-13, or performs the method as described in any one of claims 14-21, or performs the method as described in any one of claims 22, 24-31, or performs the method as described in any one of claims 23-31.

Images