Screen projection display method, screen projection display system, electronic device and storage medium

Abstract

The present application relates to the field of screen projection technology, and in particular to a screen projection display method, a screen projection display system, an electronic device and a storage medium. The screen projection display method includes the following steps: receiving screen projection content sent by at least one source device; obtaining the number of screen projection interfaces corresponding to the screen projection content and / or the content information of the screen projection content; obtaining the number of blank areas on the display interface of the target device; determining the number of virtual control groups to be displayed based on the number of screen projection interfaces and / or the content information of the screen projection content; automatically adjusting the virtual control groups to be displayed in the blank areas based on the number of virtual control groups and the number of blank areas; and displaying the virtual control groups to be displayed in the blank areas. The screen projection display method of the present application can generate and adjust virtual control groups in blank areas according to the number of blank areas and the number of virtual control groups, so as to facilitate users to adjust the screen projection content and improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of screen projection technology, and in particular to a screen projection display method, a screen projection display system, an electronic device and a storage medium. Background Art

[0002] With the development of technology, existing electronic devices can now switch and display multimedia data between multiple electronic devices through methods such as screen projection.

[0003] However, due to the different sizes and aspect ratios of different electronic devices, after the screen of an electronic device (i.e., the source device) is projected onto the target device, a wider blank area, i.e., a black frame edge, will generally be left on the side of the target device. For example, when projecting content played on a TV onto a mobile phone, users will usually adjust the interface on the mobile phone to which the projected content belongs so that the ratio between the interface to which the projected content belongs and the screen reaches the ratio that the user normally watches. At this time, since the size of the TV is completely different from that of the mobile phone, the interface to which the projected content belongs cannot fill the entire screen, which can easily leave a wider blank area on the side of the target device, resulting in a huge waste of the screen and affecting the user experience.

[0004] In addition, if the user casts the screens of multiple source devices to a target device, and then adjusts the size of the interface to which the cast content belongs so that the interface to which the cast content belongs reaches the normal viewing size; at this time, more blank areas may appear on the screen of the target device.

[0005] In summary, there is an idle blank area on the target device during screen projection, which makes the screen of the target device look unattractive and also leads to a poor user experience. Summary of the Invention

[0006] The present application provides a screen projection display method, a screen projection display system, an electronic device and a storage medium, which can generate virtual controls in the blank areas according to the number of blank areas and the number of projection interfaces, thereby overcoming the problem of unattractiveness caused by the blank areas and improving the user experience.

[0007] In a first aspect, an embodiment of the present application provides a screen projection display method, the screen projection display method comprising:

[0008] Receive projection content sent by at least one source device;

[0009] Obtain the number of projection screens corresponding to the projection content and / or the content information of the projection content;

[0010] Obtaining the number of blank areas on the display interface of the target device;

[0011] Determining the number of virtual control groups to be displayed based on the number of projection interfaces and / or content information of the projection content;

[0012] Automatically adjusting the virtual control groups to be displayed in the blank areas based on the number of the virtual control groups and the number of the blank areas; and

[0013] The virtual control group to be displayed is displayed in the blank area.

[0014] The target device can determine the number of virtual control groups and automatically adjust the virtual control groups to be generated in the blank areas according to the number of blank areas and the number of virtual control groups, thereby avoiding the screen in the blank areas being idle and unusable, and improving the user experience.

[0015] In the screen projection display method provided in the present application, automatically adjusting the virtual control group to be displayed in the blank area based on the number of the virtual control groups and the number of the blank areas includes:

[0016] When the number of blank areas is greater than the number of virtual control groups, blank areas corresponding to the number of virtual control groups are selected, and a group of virtual control groups is allocated in each selected blank area; in the remaining unselected blank areas, parameter adjustment control groups for adjusting the display parameters of the target device are allocated.

[0017] In the screen projection display method provided in the present application, automatically adjusting the virtual controls to be displayed in the blank area based on the number of virtual control groups and the number of blank areas includes:

[0018] When the number of the blank areas is equal to the number of the virtual control groups, one virtual control group is allocated to each of the blank areas.

[0019] In the screen projection display method provided in the present application, automatically adjusting the virtual controls to be displayed in the blank area based on the number of virtual control groups and the number of blank areas includes:

[0020] When the number of blank areas is less than the number of virtual control groups, at least two virtual control groups of projection contents are allocated to at least one blank area, and the blank area will switch to allocate the corresponding virtual control group based on the user's interface selection of the corresponding projection content.

[0021] In the screen projection display method provided in the present application, automatically adjusting the virtual control group to be displayed in the blank area based on the number of the virtual control groups and the number of the blank areas includes:

[0022] When the number of the blank areas is less than the number of the virtual control groups, at least one selection control is allocated in at least one of the blank areas and / or the projection interface to allow the user to switch the virtual control group displayed in the blank area.

[0023] In the screen projection display method provided in the present application, automatically adjusting the virtual controls to be displayed in the blank area based on the number of virtual control groups and the number of blank areas includes:

[0024] When the number of blank areas is less than the number of virtual control groups, at least two virtual control groups of projection contents are allocated in at least one of the blank areas. When a first preset time is interval, at least two virtual control groups corresponding to the projection interfaces will be displayed in turn in the at least one blank area.

[0025] In the screen projection display method provided in this application, the method further includes:

[0026] Receiving the projection content and / or function control information corresponding to the projection content sent by the source device;

[0027] According to the projection content or the function control information, select a virtual control corresponding to the projection content or the function control information from a pre-stored virtual control library;

[0028] The virtual controls are grouped into the virtual control group and displayed in the blank area.

[0029] In a second aspect, the present application also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the above-mentioned screen projection display method are implemented.

[0030] In a third aspect, the present application also provides a screen projection display system, comprising at least one source device and a target device, wherein the target device is used to execute the steps of the above-mentioned screen projection display method.

[0031] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned screen projection display method are implemented.

[0032] Compared with the existing technology, in the screen projection display method provided in the embodiment of the present application, the target device displays the projection content sent by the source device, and determines the number of virtual control groups to be displayed based on the number of projection interfaces and / or the content information of the projection content. When there are blank areas on the screen of the target device, virtual control groups are generated and adjusted in the blank areas based on the number of blank areas and the number of virtual control groups, so as to facilitate users to adjust the projection content and improve the user experience.

[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 This is a structural diagram for projecting the screen of a source device to a target device.

[0036] Figure 2 A flowchart of the first screen projection display method provided in the embodiment of the present application.

[0037] Figure 3 A structural diagram of the target device displaying the first type of projection content provided in the embodiment of the present application.

[0038] Figure 4 A structural diagram of the target device displaying the second type of projection content provided in the embodiment of the present application.

[0039] Figure 5a A structural diagram of the target device displaying the third type of projection content provided in the embodiment of the present application.

[0040] Figure 5b A structural diagram of the target device displaying the fourth type of projection content provided in the embodiment of the present application.

[0041] Figure 6a A structural diagram of the target device displaying the fifth type of projection content provided in the embodiment of the present application.

[0042] Figure 6b A schematic diagram of the first state of the target device displaying the sixth type of projection content provided in the embodiment of the present application.

[0043] Figure 6c A schematic diagram of the second state of the target device displaying the sixth type of projection content provided in the embodiment of the present application.

[0044] Figure 6d A structural diagram of the target device displaying the seventh type of projection content provided in the implementation manner of this application.

[0045] Figure 7 A flowchart of the second screen projection display method provided in an embodiment of the present application.

[0046] Figure 8 A structural diagram of an electronic device suitable for implementing the screen projection display method according to the embodiment of the present application.

[0047] Description of main components and symbols:

[0048] 101. Source device; 102. Target device; 103. Select control. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0051] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first groove and the second groove are merely used to distinguish different grooves and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0053] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] See also Figure 1 As shown, Figure 1 The inventors of this application have found that when projecting the source device to the target device, there may be some blank areas on the screen of the target device (i.e. Figure 1 This is because in most cases, the source device and the target device are different sizes. After the target device receives the projected content, the interface of the projected content cannot fill the entire target device screen, so there will be several blank areas on the target device screen.

[0055] It is understood that the screen projection content may include but is not limited to: the screen interface of the source device, or the video, audio, image, document, game, etc. played by the source device. After receiving the screen projection, the target device can display these contents on its screen.

[0056] In addition, the source device may be an electronic device, such as a television, a mobile phone, a tablet computer, a laptop computer (Laptop), a wearable device (such as a smart watch), etc. It should be noted that the embodiments of the present application do not limit the physical form of the electronic device. For example, the electronic device may be a foldable device, a straight-screen device, a flip-top device, etc. It should also be noted that a smart protective cover may also be installed on the electronic device in the embodiments of the present application. In other embodiments of the present application, the source device may also be an all-in-one machine, a desktop computer, a smart speaker, etc.

[0057] The target device is mainly a device that the user can control with touch screen or keyboard and mouse, such as a tablet computer, all-in-one computer, desktop computer, touch-screen smart TV, smart phone, etc. The target device is an electronic device that can receive and present or display the projected content.

[0058] To address the issue of useless and unsightly blank areas on the target device's screen when displaying projected content, which reduces the user experience, the present invention provides a projection display method, which is applied to the target device. This projection display method can display virtual controls with different functions in the blank areas on the target device's screen, thereby increasing the utilization of the blank areas and overcoming the unsightly appearance issue. Users can control the corresponding projected content by touching the virtual controls for the corresponding functions, thereby improving the user experience.

[0059] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0060] See also Figure 2 As shown, Figure 2This is a flowchart of a first screen projection display method provided in an embodiment of the present application. The screen projection display method may include steps S101-S106.

[0061] Step S101: The target device receives the projection content sent by at least one source device.

[0062] Before receiving the screen projection, the target device will establish a connection with the at least one source device and communicate with it. The source device and the target device can be connected via a communication network so that the source device and the target device are on the same network.

[0063] The communication network may be a local area network (LAN) or a wide area network (WAN), such as the Internet. The communication network may be implemented using any known network communication protocol, and the network communication protocol may be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), FireWire, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), a communication protocol supporting a network slicing architecture, or any other suitable communication protocol. For example, in some implementations, the source device may establish a Wi-Fi connection with the target device via a Wi-Fi protocol.

[0064] The user can first turn on the source device, and then map the interface displayed on the source device screen to the target device, and finally the screen of the target device can display the projected content.

[0065] In some implementations, a target device can be connected to a source device. For example, the target device can be a mobile phone, and the source device can be a smart TV. After the smart TV's screen content is projected onto the mobile phone, the user can view and control the projected content through the mobile phone.

[0066] In other implementations, the target device can also be connected to two source devices. For example, the target device can be a mobile phone, one source device can be a smart TV, and the other source device can be a tablet. In this case, the mobile phone can obtain the screen content of the smart TV and the screen content of the tablet, and then display the screen content mapped from the smart TV and the tablet simultaneously.

[0067] Alternatively, the user can first connect the mobile phone to the smart TV. At this point, the mobile phone can obtain the screen content of the smart TV, and then watch and operate the screen content of the smart TV through the mobile phone. After watching the screen content for a while, or adjusting the screen content, the user can connect the mobile phone to the tablet. The tablet interface is then mapped to the mobile phone. At this time, the screen content mapped by the smart TV and the screen content mapped by the tablet can be displayed on the mobile phone.

[0068] It is understandable that the projection content of a smart TV can be the desktop of the smart TV, or the music, video, etc. played by the smart TV; the projection content of a tablet computer can be the desktop of the tablet computer, or the music, video, etc. played by the tablet computer.

[0069] Step S102: Obtain the number of projection interfaces corresponding to the projection content and / or the content information of the projection content.

[0070] Among them, the number of projection interfaces is an integer greater than 0.

[0071] To facilitate understanding of the number of projection interfaces, the following examples are provided with the accompanying drawings:

[0072] When there is only one source device, see Figure 3 or Figure 4 As shown, at this time, the number of projection interfaces corresponding to the projection content obtained by the target device is 1.

[0073] When there are two source devices, see Figure 5a As shown, at this time, the target device obtains 2 projection interfaces corresponding to the projection content.

[0074] When there are three source devices, see Figure 6a As shown, at this time, the target device obtains 3 projection interfaces corresponding to the projection content.

[0075] The content information may be one of a game operation interface, audio and video content, picture or document content, input box content, and desktop content.

[0076] In addition, this application classifies content information. To facilitate understanding of the classification rules of this application, two projection screen interfaces are used as examples for explanation:

[0077] Considering that audio content and video content can have the same control controls, audio content and video content are classified into the same category. If the content information displayed by multiple screen projections on the screen is one of video content and video content, video content and audio content, or audio content and audio content, then the content information is defined as audio and video content. Among them, the virtual control group corresponding to the audio and video content includes at least the following two virtual controls: volume adjustment control and progress adjustment control.

[0078] If the content information is image content and document content, similarly, the corresponding virtual control group includes at least virtual controls such as an up control and a down control;

[0079] If the content information is video content and picture content, the video content and the picture content are classified into different categories, so there are two corresponding virtual control groups. One virtual control group may include at least virtual controls such as a volume adjustment control and a progress adjustment control, and the other virtual control group may include at least virtual controls such as an up control and a down control.

[0080] If the content information is a text input box, the corresponding virtual control group includes at least letter key controls, number and symbol key controls.

[0081] Step S103, the target device obtains the number of blank areas on the display interface of its screen. The number of blank areas is an integer greater than 0. The blank area can be a closed space formed by multiple continuous and uninterrupted boundaries (the boundary includes the boundary of the projection interface and the boundary of the screen display interface), specifically including: the boundary of the screen display interface and the closed blank space between the projection interface, and the closed blank space between two adjacent projection interfaces. Of course, in other implementations, there is no order for the steps of obtaining the number of projection interfaces, the content information of the projection content, and the number of blank areas. You can choose according to the actual situation and will not elaborate on them here.

[0082] Among them, when the target device obtains the number of projection screen interfaces corresponding to the projection content is 1, see Figure 3 or Figure 4 As shown, the number of blank areas obtained by the target device is 2 at this time;

[0083] When the target device obtains 2 screen projection interfaces corresponding to the projection content, see Figure 5aAs shown in the figure, the number of blank areas obtained by the target device is also 2. Figure 5a Using the perspective of as a reference, the upper and lower edges of the video and audio interfaces do not overlap with the upper and lower borders of the screen. There is one border that overlaps between the two interfaces, so the two projection interfaces can be considered as a single, integrated projection interface. The left and right borders of the overall projection interface essentially overlap with the left and right borders of the screen, respectively. The gap between the top border of the overall projection interface and the top border of the screen forms a blank area, and the gap between the bottom border of the overall projection interface and the bottom border of the screen forms a blank area. Therefore, the two existing gaps correspond to two blank areas.

[0084] When the target device obtains 3 screen projection interfaces corresponding to the projection content, see Figure 6a As shown in the figure, the number of blank areas obtained by the target device is 2. Figure 6a As a reference, Figure 6a In the example, only the gaps between the top and bottom borders of the image's interface and the top and bottom borders of the screen form a blank area. The two gaps correspond to two blank area quantities.

[0085] Step S104: Based on the number of projection screens and / or the content information of the projection screen content, determine the number of virtual control groups to be displayed in the blank area. The following describes the above content in different situations:

[0086] In the first case, the number of virtual control groups is determined based on the number of projection interfaces. Figure 3 and Figure 4 As shown, at this time, the number of projection screen interfaces is one, and the corresponding number of virtual control groups is determined to be one. Of course, when the number of projection screen interfaces is two, the corresponding number of virtual control groups is determined to be two, and so on. In the second case, the number of virtual control groups is determined according to the content information of the projection screen content. Figure 5a As shown, the content information of the screen projection content is audio content and video content of the same category, so the content information should be defined as one type, and the corresponding number of virtual control groups should be determined to be one. Of course, when the content information of the screen projection content includes audio and video content, as well as pictures or documents, it should be defined as two categories, and the corresponding number of virtual control groups should be determined to be two, and so on.

[0087] In the third case, the number of virtual control groups is determined based on the number of projection interfaces and the content information of the projection content. Figure 5aAs shown, at this time, the number of screen projection interfaces obtained is two, and then the content information of the two screen projection interfaces is obtained, which are audio content and video content respectively. Since audio content and video content are of the same category, both are audio and video content, the content information should be defined as one, and then the corresponding number of virtual control groups is determined to be one. Of course, when the number of screen projection interfaces obtained is more, the above method can also be used to determine the number of virtual control groups.

[0088] Step S105 : Automatically adjust the virtual control groups to be displayed in the blank areas based on the determined number of virtual control groups and the number of blank areas.

[0089] It should be emphasized that the automatic adjustment of the virtual control groups to be displayed in the blank area is mainly based on the allocation principle, and the virtual control groups to be displayed are automatically allocated in the blank area. The following content mainly focuses on the first scenario: determining the number of virtual control groups based on the number of projection interfaces, and explains step S105 in detail.

[0090] In addition, it should be noted that when the number of source devices is more than two, the number of blank areas and the number of projection screen interfaces may be multiple. Therefore, when the number of projection screen interfaces is more than two, the corresponding virtual control group can be displayed preferentially in the blank area close to each projection screen interface according to the allocation principle. Through this allocation method, it can be ensured that the projection content and the corresponding virtual control group are close to each other, which is convenient for users to identify the projection content corresponding to each virtual control group, and it is convenient for user operation and can also avoid misoperation.

[0091] Alternatively, a sequence identifier can be set for each interface (including the projection interface and the blank area interface) in advance according to the first preset direction, so that the target device can match the virtual control group one by one according to the set sequence identifier.

[0092] When at least two or more interfaces (including the projection interface and the blank area interface) cannot be marked according to the first preset direction, the at least two or more interfaces are marked according to the second preset direction.

[0093] For example, refer to Figure 3 As shown, the first preset direction may include from left to right. In this case, the interface to which the video belongs is marked as interface number one, the blank area on the left is marked as blank area number one, and the blank area on the right is marked as blank area number two. Therefore, the blank area on the left displays the virtual control group corresponding to the interface to which the video belongs. The blank area on the right may display a parameter adjustment control group.

[0094] When two or more blank areas or projection screens have the same length along the first preset direction, it is impossible to mark the blank areas or projection screens according to the first preset direction. Figure 5b Direction shown, the first preset direction includes from left to right, Figure 5b The lengths of the two blank areas in the horizontal direction (from left to right) are the same; when marking the blank areas according to the first preset direction, the two blank areas cannot be marked because there is no order between them. At this time, the target device can mark the at least two blank areas according to the second preset direction (from top to bottom). Therefore, the target device can mark Figure 5b The video and audio interfaces are marked as Interface 1 and Interface 2, respectively, while the blank areas are marked as Blank Area 1 and Blank Area 2, from top to bottom. In this case, the target device can prioritize allocating the virtual controls corresponding to Interface 1 to Blank Area 1 and the virtual controls corresponding to Interface 2 to Blank Area 2.

[0095] Specifically, based on the determined number of virtual control groups and the number of blank areas, automatically adjusting the virtual control groups to be displayed in the blank areas includes: when the number of blank areas is greater than the number of virtual control groups, selecting blank areas corresponding to the number of virtual control groups, and assigning a virtual control group to each selected blank area. In the redundant blank areas that are not assigned virtual controls, priority is given to assigning a parameter adjustment control group for adjusting display parameters of the target device, or assigning at least one parameter adjustment control from the parameter adjustment control group. Of course, other controls may also be assigned, which will not be listed here.

[0096] For example, there is one virtual control group and two blank areas. The target device automatically enables this allocation principle, randomly selects a blank area close to the projection interface, and randomly allocates the virtual control group to the blank area. If each blank area is equidistant from the projection interface, the virtual control group can be allocated in any blank area. In the redundant blank areas that are not allocated with virtual controls, controls for adjusting the display parameters of the target device, i.e., parameter adjustment control groups, including brightness adjustment controls and contrast adjustment controls for adjusting brightness and / or contrast, can be used to ensure that all blank areas can be fully utilized.

[0097] For ease of understanding, the following example uses a smart TV as the source device and a mobile phone as the target device. On the target phone screen, there is one projection interface and two blank areas. In this case, a virtual control group can be displayed in any blank area, allowing the user to adjust the projection content.

[0098] When the projection content displayed on the mobile phone screen is a video, the virtual control group may include: a progress adjustment control and a volume adjustment control. Figure 3 As shown, the virtual control group is displayed on one side and the parameter adjustment control group is assigned on the other side.

[0099] Specifically, Figure 3 For example, the progress and volume control are located on the left side of the video interface. The progress control allows you to advance or rewind the video, while the volume control allows you to adjust the volume. The parameter control group can be located on the right side of the video interface. This group allows you to adjust the brightness and contrast of the video interface.

[0100] Furthermore, given that each virtual control group may contain multiple virtual controls, in another embodiment, when the number of blank areas exceeds the number of virtual control groups, at least one virtual control group can be split and displayed in different blank areas. For example, if a virtual control group includes a progress control and a volume control, and there is one projection interface and two blank areas, the progress control can be displayed in the blank area on one side of the projection interface, and the volume control can be displayed in the blank area on the other side, thereby fully utilizing the blank area and avoiding waste.

[0101] After the virtual control group is split, Figure 4 When the projection interface displayed on the mobile phone screen is a text input box, the virtual control group corresponding to the projection interface may include two virtual controls: a letter button control and a number and symbol button control.

[0102] Specifically, the letter button control can be located on one side of the interface to which the text input box belongs, and the number and symbol button control can be located on the other side of the interface to which the text input box belongs. Figure 4 As a reference, the upper or lower side of the interface to which the text input box belongs displays letter button controls, or displays number and symbol button controls.

[0103] Specifically, based on the determined number of virtual control groups and the number of blank areas, automatically adjusting the virtual control groups to be displayed in the blank areas includes, that is, the allocation principle also includes: when the number of blank areas is equal to the number of virtual control groups, randomly allocating a group of virtual control groups to each blank area.

[0104] It is worth mentioning that when the number of blank areas is equal to the number of virtual control groups, the target device may no longer display the parameter adjustment control group or parameter adjustment control.

[0105] See also Figure 5bAs shown, the following explanation is based on the assumption that there are two source devices and one target device. The two source devices are a smart TV and a smart bracelet, and the target device is a mobile phone. At this time, the user can map the video interface of the smart TV and the audio interface of the smart bracelet to the mobile phone respectively. Given that the number of virtual control groups in this embodiment is determined according to the number of projection interfaces, there are two virtual control groups at this time, and the number of blank areas is also two.

[0106] At this time, a virtual control group corresponding to the interface to which the video belongs can be allocated in at least one blank area near the position of the interface to which the video belongs, including a progress adjustment control and a volume adjustment control; and a virtual control group corresponding to the interface to which the audio belongs can be allocated in at least one blank area near the position of the interface to which the audio belongs, including a progress adjustment control and a volume adjustment control.

[0107] Specifically, Figure 5b Taking the viewing angle as an example, the progress adjustment control and the volume adjustment control can be displayed in the upper (or lower) blank area near the position of the video interface. Correspondingly, the progress adjustment control and the volume adjustment control can be displayed in the upper (or lower) blank area near the position of the audio interface.

[0108] Specifically, based on the determined number of virtual control groups and the number of blank areas, the virtual control groups to be displayed in the blank areas are automatically adjusted, that is, the allocation principle also includes: when the number of blank areas is less than the number of virtual control groups, preferably, at least one blank area is assigned at least two virtual control groups corresponding to the projection contents, and the blank area will switch to the corresponding virtual control group based on the user's interface selection of the corresponding projection content.

[0109] That is, at least two or more virtual control groups may be allocated to at least one blank area, and the two or more virtual control groups will be switched and displayed in the blank area according to the user's interface selection of the projection content.

[0110] For example, consider two projection screens and a blank area. The two projection screens are a video screen and an image screen, respectively, with two corresponding virtual control groups. When a user taps a video screen in the projection screen, the video screen is selected, and the blank area displays the corresponding virtual control group for the video screen, making it easier for the user to operate.

[0111] See also Figure 6aAs shown, the three source devices can be two smart TVs and a smart bracelet, and the target device is a tablet. After the user maps the screens of the two smart TVs and the smart bracelet to the tablet, three screen projection content interfaces are formed on the tablet screen, namely video, audio, and picture. At the same time, two blank areas are also displayed on the tablet screen. Since the number of virtual control groups in this embodiment is determined according to the number of screen projection interfaces, the number of virtual control groups is three. In this case, the number of blank areas is less than the number of virtual control groups.

[0112] At this time, the target device allocates at least two virtual control groups of the projection content to at least one blank area, and the blank area will switch to allocate the corresponding virtual control group based on the user's interface selection of the corresponding projection content. Figure 6a As shown, in Figure 6a The blank area in the upper center can simultaneously display the virtual control groups corresponding to the video interface and the audio interface. When the user clicks the video interface, the video interface is selected, and the blank area will center the virtual control group corresponding to the video interface. When the user clicks the audio interface, the audio interface is selected, and the blank area will switch to displaying the virtual control group corresponding to the audio interface, thereby switching the display of the two virtual control groups.

[0113] Of course, in the above solution, when the target device assigns two or more virtual control groups to a blank area, it is also possible that not all virtual control groups are displayed in the blank area by default. When the user clicks on the projection interface, the virtual control group corresponding to the projection interface is displayed in the blank area. For example, when the user clicks on the audio interface, the virtual control group corresponding to the audio interface is displayed in the blank area.

[0114] In another embodiment, when the number of blank areas is less than the number of virtual control groups, at least one selection control is allocated in at least one blank area or projection interface, and the blank area / or projection interface will switch and display the corresponding virtual control group based on the user clicking the selection control.

[0115] That is, in response to the user triggering the operation of selecting a control, the target device can determine the projection content selected by the user and display a virtual control group corresponding to the projection content selected by the user in at least one blank area.

[0116] For example, consider two projection screens and a blank area. The two projection screens are a video screen and an image screen. Within this blank area, a selection control can be displayed. When the user selects the video screen using the selection control, the blank area switches to the virtual control group corresponding to the video screen. When the user selects the image screen, the blank area switches to the virtual control group corresponding to the image screen.

[0117] Further, refer to Figure 6b-6d As shown, the screen projection display method further includes:

[0118] The target device responds to the user triggering the selection control operation to determine the screen projection interface selected by the user;

[0119] And in at least one blank area, a virtual control group corresponding to the projection interface selected by the user is displayed.

[0120] like Figure 6b As shown, a selection control is displayed in the blank area above. When the user clicks the selection control, a pop-up window appears, as shown in Figure 6c As shown, the pop-up window displays screen projection 1, screen projection 2, and so on, where n is a natural number. In this case, when the user clicks any screen projection x, where x is any integer from 0 to n, the virtual control group corresponding to screen projection x is displayed in at least one blank area.

[0121] In another embodiment, a selection control can be displayed at the corner of each projection screen. When a user touches a selection control with a finger, a virtual control group of the projection screen corresponding to the selection control touched by the user is displayed in at least one blank area. Furthermore, because the projection screen displays a selection control, the selection control can be displayed relatively transparently to improve the user experience.

[0122] In another embodiment, in at least one blank area, at least two virtual control groups corresponding to the projection information are displayed in turn.

[0123] See also Figure 6d As shown, in a blank area of ​​the target device screen, a virtual control group corresponding to the projection content of the first source device is displayed, and in another blank area, at intervals of a first preset time, virtual control groups corresponding to the projection content of the other two second source devices are displayed in turn.

[0124] The first preset time may be one minute. At this time, in another blank area, the virtual control group corresponding to the screen projection content of the first second source end device is first displayed, and after an interval of one minute, the virtual control group corresponding to the screen projection content of the second second source end device is displayed.

[0125] In addition, if there is a larger blank area among multiple blank areas, multiple virtual control groups corresponding to the projection content can be displayed in the larger blank area. For example, when there are three virtual control groups and two blank areas, the virtual control groups corresponding to two projection screens can be displayed in one larger blank area, and all virtual control groups corresponding to the third projection screen can be displayed in another blank area.

[0126] The method for determining a larger blank area may be as follows: the target device obtains the minimum projection interface area and the area of ​​the maximum blank area. When the area of ​​the maximum blank area is larger than the minimum projection interface area, the blank area is a larger blank area.

[0127] Of course, the criterion for determining a larger blank area may also be: the area of ​​the larger blank area is between 1 / 7 and 1 / 9 of the screen area of ​​the target device.

[0128] Displaying multiple virtual control groups corresponding to the projected content in a larger blank area can overcome the problem of not being able to display all virtual controls due to a smaller blank area.

[0129] For example, take a tablet screen with two blank areas and three groups of virtual controls as an example:

[0130] There is a larger blank area between the two blank areas. This larger blank area can display the virtual control groups corresponding to the two projection screens, and the remaining blank area can display the virtual control group corresponding to the third projection screen.

[0131] The above solution is mainly applicable to the situation where the number of virtual control groups is determined according to the number of projection interfaces.

[0132] If the number of virtual control groups is determined by the second scenario: the number of virtual control groups is determined based on the content information of the projected content; or the third scenario: the number of virtual control groups is determined based on the number of projection screens and the content information of the projected content. Once the number of virtual control groups is determined, the target device can allocate the virtual control groups to the corresponding blank areas based on the above content allocation principles. The blank area below will display parameter adjustment controls.

[0133] The following example illustrates this:

[0134] See also Figure 5a As shown, the screen projection content can include video information and audio information, and there are two blank areas. Because video information and audio information are classified as the same category, the corresponding number of virtual control groups is one. The target device, based on the above content allocation principle, allocates the virtual control group to the blank area above, and the blank area below displays the parameter adjustment controls.

[0135] Furthermore, when the user needs to adjust the projection content through the virtual control group, the user can associate the virtual control group with the corresponding projection interface (such as the interface to which the video belongs, the interface to which the audio belongs), and then control the corresponding projection interface (such as the interface to which the video belongs, the interface to which the audio belongs) by clicking the virtual control group. For ease of understanding, the following scenarios are explained:

[0136] The first case: See Figure 5a As shown, the user can touch any virtual control in the virtual control group, such as the progress adjustment control, with a finger or stylus, and then hold it down and drag it to any projection interface, such as the interface where the video belongs. The progress adjustment control will be associated with the interface where the video belongs. Then the user lifts the finger or stylus off the screen, and the progress adjustment control will automatically return to its original position. At this point, the user can adjust the video content by clicking the progress adjustment control. The parameter adjustment control can use the same association method to be associated with the interface where the video belongs or the interface where the audio belongs.

[0137] In the second scenario, the user can use a finger or stylus to press any virtual control in the virtual control group for several seconds, such as continuously pressing the progress adjustment control for more than 5 seconds; then click on any screen projection interface, such as the interface containing a video. The progress adjustment control will now be associated with the interface containing the video. At this point, the user can adjust the video content by clicking the progress adjustment control.

[0138] In addition, the target device determines and obtains the virtual controls of each virtual control group, which can be completed in the following ways:

[0139] In the first method, each source device sends the projected content to the target device, and further sends the functional control information corresponding to the projected content to the target device. The target device selects the virtual control corresponding to the above functional control information from the pre-stored virtual control library based on the functional control information, and combines the virtual controls to form a corresponding virtual control group, which is displayed in the blank area corresponding to the projected content.

[0140] In the second method, the target device's virtual control library has pre-stored virtual control groups corresponding to each type of content information. For example, the virtual control group corresponding to audio and video content includes volume adjustment controls and progress adjustment controls. When the source device sends the projected content to the target device, the target device selects virtual controls corresponding to the content information from the virtual control library based on the content information of the projected content, combines them into corresponding virtual control groups, and displays them in the blank area corresponding to the projected content.

[0141] Step S106: Display the virtual control group to be displayed in the corresponding blank area. The virtual control is preferably located in the center of the corresponding blank area, which makes it more aesthetically pleasing. After the target device displays the virtual control group, the user can adjust the screen content of the source device screen by clicking the virtual control group.

[0142] In summary, the target device displays the projection content sent by the source device. When there is a blank area on the target device screen, virtual controls are generated in the blank area based on the number of blank areas and the number of projection interfaces to facilitate users to adjust the projection content, thereby improving the user experience.

[0143] The inventors of this application also found that different users have different usage habits. Therefore, in order to improve the user experience, see Figure 7 As shown, in another embodiment of the present application, the screen projection display method also includes step S107.

[0144] Step S107: obtaining the continuous time that the user touches a virtual control in the virtual control group; when the pressing time is greater than a second preset time, activating the movable function of the virtual control; obtaining the trajectory of the user sliding the screen, and moving the virtual control to the target blank area according to the trajectory.

[0145] Specifically, the user can touch the virtual control with a finger or stylus for a long time, and then the target device can obtain the continuous time that the user touches the virtual control; when the continuous time is greater than a second preset time, the movable function of the virtual control is activated, making the virtual control become movable.

[0146] When the virtual control becomes movable, the target device can obtain the trajectory of the user sliding the screen and move the virtual control to the corresponding target blank area based on the trajectory. Therefore, in this way, the user can drag the virtual control to any blank area.

[0147] For example, the second preset time may be 2 seconds. When a user touches a virtual control for more than 2 seconds, the movable function of the virtual control is activated, that is, the virtual control becomes movable. The user can then drag the virtual control to another blank area as needed.

[0148] It should be emphasized that in the implementation of this application, the data of the source device and the target device need to be displayed in the same ratio. For example, if the resolution of a smart TV is 1920*1080, the display ratio should be 16:9. The mobile phone can display the screen content in a 16:9 ratio according to its own screen size.

[0149] It should also be emphasized that all the above operations on virtual controls can be performed not only through touch screen operations, but also through input devices such as keyboard, mouse, stylus, etc.

[0150] In addition, an embodiment of the present application also provides a screen projection display system, including at least one source device 101 and a target device 102, wherein the target device is used to execute the steps of the above-mentioned screen projection display method.

[0151] Among them, each module in the above-mentioned projection device corresponds to each step in the implementation of the above-mentioned projection display method, and its functions and implementation processes are no longer described here one by one. This application also provides an electronic device, see Figure 8 As shown, the electronic device 50 may include a processor (CPU, GPU, FPGA, etc.) 501, which can perform some or all of the processing in the embodiments shown in the above figures according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 into the random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The processor 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0152] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0153] In particular, according to embodiments of the present application, the methods described above with reference to the accompanying drawings can be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for executing the methods described in the accompanying drawings. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 509 and / or installed from the removable medium 511.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the diagram or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0155] The units or modules involved in the embodiments described in this application may be implemented by software or hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves.

[0156] As another aspect, the present application further provides a computer-readable storage medium, which may be the computer-readable storage medium included in the screen projection device in the above-described embodiment; or it may be a separate computer-readable storage medium that is not incorporated into the device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the screen projection display method described in the present application.

[0157] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A screen projection display method, characterized in that: Applied to the target device, the method comprises the following steps: Receive projection content sent by at least one source device; Obtain the number of projection screens corresponding to the projection content and / or the content information of the projection content; Obtaining the number of blank areas on the display interface of the target device; Determining the number of virtual control groups to be displayed based on the number of projection interfaces and / or content information of the projection content; Automatically adjusting the virtual control groups to be displayed in the blank areas based on the number of the virtual control groups and the number of the blank areas; and Displaying the virtual control group to be displayed in the blank area; The automatically adjusting the virtual controls to be displayed in the blank areas based on the number of virtual control groups and the number of blank areas includes: When the number of blank areas is less than the number of virtual control groups, at least two virtual control groups of projection content are allocated to at least one blank area, and the blank area will switch to allocate the corresponding virtual control group based on the user's interface selection of the corresponding projection content; When the number of blank areas is greater than the number of virtual control groups, blank areas corresponding to the number of virtual control groups are selected, and a group of virtual control groups is allocated to each selected blank area; parameter adjustment control groups for adjusting display parameters of the target device are allocated to the remaining unselected blank areas; When the number of the blank areas is equal to the number of the virtual control groups, one virtual control group is allocated to each of the blank areas.

2. The screen projection display method according to claim 1, wherein: The automatically adjusting the virtual control groups to be displayed in the blank areas based on the number of the virtual control groups and the number of the blank areas includes: When the number of the blank areas is less than the number of the virtual control groups, at least one selection control is allocated in at least one of the blank areas and / or the projection interface to allow the user to switch the virtual control group displayed in the blank area.

3. The screen projection display method according to claim 1, wherein: The automatically adjusting the virtual controls to be displayed in the blank areas based on the number of virtual control groups and the number of blank areas includes: When the number of blank areas is less than the number of virtual control groups, at least two virtual control groups of projection contents are allocated in at least one of the blank areas. When a first preset time is interval, at least two virtual control groups corresponding to the projection interfaces will be displayed in turn in the at least one blank area.

4. The screen projection display method according to any one of claims 1 to 3, characterized in that: The method further comprises: Receiving the projection content and / or function control information corresponding to the projection content sent by the source device; According to the projection content or the function control information, select a virtual control corresponding to the projection content or the function control information from a pre-stored virtual control library; The virtual controls are grouped into the virtual control group and displayed in the blank area.

5. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the screen projection display method as described in any one of claims 1 to 4 are implemented.

6. A screen projection display system, characterized in that: It includes at least one source device and a target device, wherein the target device is used to execute the steps of the screen projection display method described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the screen projection display method according to any one of claims 1 to 4 are implemented.

Images