Processing method, device, equipment and program product
By creating a virtual screen on the mobile phone and transmitting the application interface in media streaming, the problem of coding capabilities limitation in APP streaming screen projection technology is solved, and the simultaneous display and resource optimization of multiple application interfaces is realized, improving user experience and System performance.
Patent Information
- Application Number
- CN202411194865.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
AI Technical Summary
When using the APP streaming screen projection technology, due to the limited encoding capability of the mobile phone, there is an upper limit on the number of screens projections, and each screen projection can only display one application interface, resulting in poor user experience and high-resolution video Images occupy a large amount of coding resources, limiting the use of other video software.
By creating a virtual screen, draw the interactive interface of the application and transmit it to the second electronic device in the form of a media stream, the interactive interfaces of multiple applications are displayed in the same window, and the use of encoding resources is optimized, for example, by merging two The interactive interface is placed on the same virtual screen to reduce the use of encoding and transmission resources.
An interactive interface for displaying multiple applications on the second electronic device simultaneously is realized, increasing the number of displays, saving encoding and transmission resources, and improving user experience and system performance.
Smart Images

Figure CN119065623A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of electronic technology, and relate to but are not limited to a processing method, device, equipment, and program product. Background Art
[0002] When using APP streaming screen projection technology, the number of projections is limited by the encoding capabilities of the mobile phone, and each projection can only display one application interface. In actual application, a mobile phone with good performance can only create 5 projections, and a mobile phone with poor performance can only create 2. When the number of projections created exceeds the maximum number, the last screen will be overwritten, which is inconvenient for users.
[0003] Since APP streaming occupies about 35% of the encoding capacity of the mobile phone at most, encoding resources need to be reserved for other video software, etc. When using APP streaming for screen projection, due to the limitation of the encoding capacity of the mobile phone, each time an APP streaming is started, the corresponding video image needs to be encoded and transmitted to the projection device, and since the resolution of the video image is very high, it takes up a lot of encoding resources. Summary of the invention
[0004] In view of this, embodiments of the present application provide a processing method, apparatus, device, and program product.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides a processing method, including:
[0007] Establishing a connection channel with a second electronic device;
[0008] Creating a first virtual screen, where the first virtual screen is used to draw a first interactive interface of a first application program;
[0009] Obtaining a drawing image based on the first virtual screen and transmitting the image to the second electronic device in the form of a media stream through a connection channel so that the first interactive interface is displayed in a first window;
[0010] Creating a second virtual screen, where the second virtual screen is used to draw a second interactive interface of a second application program;
[0011] Obtaining a drawing image based on the second virtual screen and transmitting the image to the second electronic device in the form of a media stream through a connection channel so that the second interactive interface is displayed in a second window;
[0012] In response to the target instruction, a first interactive interface and a second interactive interface are drawn based on the first virtual screen, and at the same time, transmission of the drawn image of the second virtual screen in the form of a media stream through the connection channel is stopped, so that the second electronic device displays the first interactive interface and the second interactive interface in a first window.
[0013] In a second aspect, an embodiment of the present application provides a processing device, including:
[0014] A connection establishment module, used to establish a connection channel with a second electronic device;
[0015] A first creation module, used to create a first virtual screen, where the first virtual screen is used to draw a first interactive interface of a first application;
[0016] A first acquisition and transmission module, configured to acquire a drawing image based on a first virtual screen and transmit the image to a second electronic device in the form of a media stream through a connection channel so as to display a first interactive interface in a first window;
[0017] A second creation module, used to create a second virtual screen, where the second virtual screen is used to draw a second interactive interface of a second application;
[0018] A second acquisition and transmission module, used for acquiring a drawing image based on a second virtual screen and transmitting it to a second electronic device in the form of a media stream through a connection channel so that a second interactive interface is displayed in a second window;
[0019] A drawing module is used to draw a first interactive interface and a second interactive interface based on a first virtual screen in response to a target instruction, and at the same time, stop transmitting the drawn image of the second virtual screen in the form of a media stream through a connection channel, so that the second electronic device displays the first interactive interface and the second interactive interface in a first window.
[0020] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above method is implemented when the processor executes the program.
[0021] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.
[0022] In a fifth aspect, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1A A schematic diagram of an implementation flow of a processing method provided in an embodiment of the present application;
[0024] Figure 1BA schematic diagram of displaying five windows on a second display screen provided in an embodiment of the present application;
[0025] Figure 2 A schematic diagram of an implementation flow of generating a target instruction provided in an embodiment of the present application;
[0026] Figure 3 A schematic diagram of an implementation flow of generating a target instruction provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of the implementation process of encoding image data provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of the structure of a processing device provided in an embodiment of the present application;
[0029] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.
[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0034] The present application embodiment provides a processing method, such as Figure 1A As shown, the method includes:
[0035] Step S101, establishing a connection channel with a second electronic device;
[0036] Here, the execution subject of the processing method is a first electronic device, for example, a mobile phone, a tablet computer, a computer, etc. The second electronic device is a device that can project the content on the first device to a second electronic device (such as a TV, a projector, a display, etc.) for display via wireless or wired means.
[0037] The connection channel can be a wireless connection channel or a wired connection channel.
[0038] During the implementation process, the first electronic device and the second electronic device establish a connection channel in advance to achieve the transmission of coding resources generated by screen projection when the first electronic device performs a screen projection operation.
[0039] Step S102: creating a first virtual screen, where the first virtual screen is used to draw a first interactive interface of a first application;
[0040] Here, the virtual screen refers to a screen created in the background of the first electronic device, which can be understood as a secondary screen that is invisible to the user on the first electronic device.
[0041] Drawing an interface refers to the process of implementing a graphical user interface (GUI). The user interface (UI) of an application is a window for users to exchange information and interact with the application. Users can input information and perform operations on electronic devices through the interactive interface, and electronic devices provide information to users through the interactive interface for reading, analysis and judgment.
[0042] During the implementation process, in response to a screen projection instruction to project the first application to the second electronic device, a first virtual screen may be created first, and then a first interactive interface of the first application may be drawn on the first virtual screen.
[0043] Step S103: obtaining a drawing image based on the first virtual screen and transmitting the image to the second electronic device in the form of a media stream through the connection channel so as to display the first interactive interface in a first window;
[0044] Here, the drawn image may be an image obtained by drawing the interactive interface on the first virtual screen. Streaming media is a technology for transmitting multimedia content such as audio and video through the Internet. Streaming media technology allows the interactive interface of an application to be played in real time without transmitting the entire application. This streaming media format can compress a series of media data and send the data in segments through the established connection channel, so as to realize the instant use of the application for users to view the interactive interface and perform interactive operations on the interactive interface. This greatly improves the real-time and convenience of viewing and use.
[0045] During implementation, the drawn image of the first virtual screen may be obtained first, then the drawn image may be encoded, and the encoded data may be transmitted to the second electronic device in the form of a media stream through a connection channel, so as to further display the first interactive interface in a first window on the second electronic device.
[0046] In some embodiments, the size information of the first virtual screen may be obtained first; then the target data volume of the encoded data corresponding to the drawn image of the first virtual screen may be determined based on the size information; and finally, the encoded data corresponding to the target data volume of at least one drawn image may be sent to the second electronic device.
[0047] Step S104: creating a second virtual screen, where the second virtual screen is used to draw a second interactive interface of a second application;
[0048] During implementation, in response to a screen projection instruction to project the second application to the second electronic device, a second virtual screen may be created first, and then a second interactive interface of the second application may be drawn on the second virtual screen.
[0049] Step S105: obtaining a drawing image based on the second virtual screen and transmitting the image to the second electronic device in the form of a media stream through the connection channel so as to display the second interactive interface in a second window;
[0050] During implementation, the drawn image of the second virtual screen may be acquired first, then encoded, and the encoded data may be transmitted to the second electronic device in the form of a media stream through a connection channel, so as to further display the second interactive interface in a second window on the second electronic device.
[0051] Step S106, in response to the target instruction, draw the first interactive interface and the second interactive interface based on the first virtual screen, and at the same time, stop transmitting the drawn image of the second virtual screen in the form of a media stream through the connection channel, so that the second electronic device displays the first interactive interface and the second interactive interface in a first window.
[0052] In some embodiments, the target instruction can be generated based on the coding resources actually transmitted. For example, when it is determined based on the transmitted coding resources that the corresponding resources of 5 virtual screens can actually be transmitted, the sixth interactive interface of the sixth application needs to be transmitted, and the two applications corresponding to the two virtual screens can be determined to be merged into one virtual screen for drawing and transmission.
[0053] In some embodiments, the position information and image size information of the interactive interface of the first application and the second application displayed on the first virtual screen can be determined based on the corresponding attribute information of the first application and the second application; and the first interactive interface and the second interactive interface can be created in a split screen based on the position information and image size information of the interactive interface of the first application and the second application displayed on the first virtual screen.
[0054] In some embodiments, three or more interactive interfaces can be drawn based on the first virtual screen, and the drawn image of the first virtual screen can be transmitted in the form of a media stream through a connection channel, so that the second electronic device displays three or more interactive interfaces in the first window.
[0055] In some embodiments, the encoded data corresponding to the first interactive interface and the second interactive interface on the first virtual screen can be obtained; the encoded data is compressed and sent to the second electronic device. In this way, transmitting the compressed second encoded data can save bandwidth resources for transmission.
[0056] Figure 1B A schematic diagram of displaying five windows on a second display screen provided in an embodiment of the present application, such as Figure 1B As shown, the schematic diagram includes: a first window 11, a second window 12, a third window 13, a fourth window 14 and a fifth window 15, wherein.
[0057] The first window 11 displays the interactive interface of the first application and the interactive interface of the sixth application; the second window 12 displays the interactive interface of the second application and the interactive interface of the seventh application; the third window 13 displays the interactive interface of the third application and the interactive interface of the eighth application; the first window 14 displays the interactive interface of the fourth application and the interactive interface of the ninth application; the first window 11 displays the interactive interface of the fifth application and the interactive interface of the tenth application. In this way, the interactive interfaces of 10 applications can be displayed in 5 windows of the second electronic device.
[0058] In some embodiments, the target instruction may also be generated by a user instruction. For example, the user may trigger the target instruction by dragging or other operations to ensure that the two applications are drawn on the same virtual screen.
[0059] During the implementation process, when the target instruction indicates that the first interactive interface and the second interactive interface are drawn on the same virtual screen, the first electronic device responds to the target instruction and draws the first interactive interface and the second interactive interface based on the first virtual screen. At the same time, since the second interactive interface has been drawn on the first virtual screen, the transmission of the drawn image of the second virtual screen in the form of a media stream through the connection channel is stopped. In this way, the second electronic device can display the first interactive interface and the second interactive interface in the first window, that is, the first interactive interface and the second exchange interface are displayed simultaneously in the same window of the second electronic device. Here, there is no restriction on the layout of the display area of the first interactive interface and the second exchange interface. The first interactive interface and the second exchange interface can be displayed in an upper and lower layout, or in a left and right layout. The layout of the first interactive interface and the second exchange interface can also be set in a specific position according to user needs.
[0060] In some embodiments, the split-screen display function can also be used to draw the first interactive interface and the second interactive interface on the same virtual screen.
[0061] In the embodiment of the present application, in response to the target instruction, the first interactive interface and the second interactive interface are drawn based on the first virtual screen, and at the same time, the transmission of the drawn image of the second virtual screen in the form of a media stream through the connection channel is stopped. In this way, the first interactive interface and the second interactive interface are drawn on the first virtual screen at the same time, which can save the coding resources occupied by encoding a virtual screen, and at the same time, the first interactive interface and the second interactive interface can be displayed on the second electronic device in the first window, thereby increasing the number of interactive interfaces of the application that can be displayed on the second electronic device.
[0062] In some embodiments, before the above step S106 "responding to the target instruction", a target instruction is generated, such as Figure 2 As shown, this can be achieved by following the steps below:
[0063] Step S210, obtaining target parameters;
[0064] Here, the target parameter may be a parameter of the first electronic device acquired in real time. For example, the target parameter may include at least one of the following: the number of applications to be drawn, the encoding resources of the encoding virtual screen, and the real-time bandwidth resources of the connection channel.
[0065] Step S220: If the target parameter meets the resource threshold, generate the target instruction;
[0066] The resource threshold includes any of the following:
[0067] The threshold for creating the number of virtual screens;
[0068] Threshold of encoding resources for streaming media;
[0069] The bandwidth resource threshold of the connection channel.
[0070] In some embodiments, a target instruction may be generated when it is determined that the number of virtual screens to be created is greater than a threshold value for the number of virtual screens to be created. For example, the threshold value for the number of virtual screens to be created is 5, and each virtual screen draws an interactive interface of a corresponding application. When it is determined that the sixth application is to be drawn, a target instruction may be generated to draw the interactive interfaces of two applications among the first five applications on one virtual screen, and draw the sixth application on the fifth virtual screen.
[0071] In some embodiments, a target instruction may be generated when it is determined that the real-time coding resources are greater than a coding resource threshold. For example, a coding resource allocation scheme may be determined based on the computing power of the first electronic device, including reserving a coding resource threshold for the screen projection function of the first electronic device. When it is determined that the real-time coding resources are greater than the coding resource threshold, the two interactive interfaces may be drawn on the same virtual screen to reduce the occupied coding resources.
[0072] In some embodiments, a bandwidth resource threshold of a connection channel may be predetermined, that is, a maximum resource value that can be transmitted in the connection channel. For example, when it is determined that the bandwidth resource corresponding to the drawn image to be transmitted exceeds the bandwidth resource threshold, the two interactive interfaces may be drawn on the same virtual screen to achieve the effect of reducing the bandwidth resource used for transmission.
[0073] In the embodiment of the present application, the target parameter is first obtained; then if the target parameter meets the resource threshold, the target instruction is generated; wherein the resource threshold includes any one of the following: the number threshold of creating virtual screens; the encoding resource threshold for transmitting in the form of streaming media; the bandwidth resource threshold of the connection channel. In this way, the target instruction can be generated based on the preset resource threshold, so as to achieve the effect of reducing the resource threshold in real time by drawing two interactive interfaces on the same virtual screen.
[0074] In some embodiments, before the above step S106 "responding to the target instruction", a target instruction is generated, such as Figure 3 As shown, this can be achieved by following the steps below:
[0075] Step S310: creating an N+1th virtual screen based on the N+1th application, where N is an integer greater than or equal to 2;
[0076] For example, a third virtual screen may be created based on a third application.
[0077] Step S320: Generate the target instruction based on creating the N+1th virtual screen.
[0078] When creating the third virtual screen, a target instruction may be generated to enable the first two interactive interfaces to be drawn on the same virtual screen. The transmission of the drawn image of the second virtual screen in the form of a media stream through the connection channel is stopped, and the third interactive interface of the third application is drawn on the third virtual screen.
[0079] In the embodiment of the present application, firstly, the N+1th virtual screen is created based on the N+1th application, and then the target instruction is generated based on the creation of the N+1th virtual screen. In this way, when two virtual screens have been created and a third virtual screen needs to be created, the target instruction can be triggered to realize drawing the first two interactive interfaces on the same virtual screen, saving the coding resources for one virtual screen.
[0080] In some embodiments, it is determined that an interactive interface of a target application is newly displayed on a target virtual screen of the first electronic device; the current interactive interface on the target virtual screen is transferred and drawn to other virtual screens other than the target virtual screen; and an interactive interface corresponding to the target application is created on the target virtual screen. In this way, the interactive interface of the target application can occupy a complete target virtual screen, and the interactive interface drawn on the current target virtual screen can be drawn to other virtual screens in a thumbnail display manner.
[0081] In some embodiments, the application icon corresponding to the current interactive interface on the target virtual screen can be obtained; the application icon corresponding to the current application interface on the target virtual screen is moved and drawn to other virtual screens other than the target virtual screen; and the first coded data of the application icon is sent to the second electronic device. In this way, the application icon corresponding to the current application interface can be created in split screen on other virtual screens, and the coding resources occupied by the coded data corresponding to the application icon are less than the coding resources occupied by the coded interactive interface, so the coding resources can be greatly saved.
[0082] In some embodiments, before the above step S106 “responding to the target instruction”, generating the target instruction can be achieved by the following steps:
[0083] Step S107: obtaining an input operation, wherein the input operation is an input operation transmitted by the second electronic device through the connection channel to the first window and the second window; wherein the input operation includes trajectory information, a first operation object representing the first window, and a second operation object representing the second window;
[0084] Here, the input operation may be an operation performed by the user on the second electronic device. For example, the user may drag the second window on the second electronic device so that the second window overlaps with the first window to meet the condition.
[0085] In some embodiments, the second device can determine that the overlap condition is met, and then send the input operation to the first electronic device using the connection channel, so that the first electronic device obtains the input operation. For example, the user can drag the second window on the second electronic device with the screen projection display to overlap the first window to meet the condition. When the second electronic device determines that the window overlap condition is met, it sends an instruction corresponding to the overlap condition to the first electronic device, and the first electronic device determines that the user has performed an input operation on the second electronic device based on the instruction.
[0086] Step S108, parsing the input operation to generate the target instruction;
[0087] In some embodiments, the second electronic device can send an instruction corresponding to the overlapping condition to the first electronic device when determining that the window overlapping condition is met, and the first electronic device determines that the user has performed an input operation on the second electronic device based on parsing the instruction.
[0088] In some embodiments, the first electronic device may parse the input operation sent by the second electronic device, determine based on the input operation that the overlap between the second window and the first window satisfies a condition, and generate a target instruction to draw the two interactive interfaces on the same virtual screen.
[0089] or;
[0090] Step S109: obtaining an input event, where the input event is a merging event for the first window and the second window transmitted by the second electronic device through the connection channel;
[0091] In the implementation process, the input event may be that the user selects a merge control for the second window on the second electronic device and determines the first window as a merge object. The merge control refers to combining multiple views into a custom view component.
[0092] The first electronic device can obtain the input event sent by the second electronic device through the connection channel.
[0093] Step S110: generating the target instruction in response to the input event.
[0094] During the implementation process, the first electronic device can determine based on the input event that the user's operation on the second electronic device achieves the effect of merging the first window and the second window, and based on the input event, generate a target instruction to draw the corresponding two interactive interfaces on the same virtual screen on the first electronic device.
[0095] In the embodiment of the present application, the first electronic device analyzes the input operation for the first window and the second window to generate a target instruction; or generates a target instruction in response to a merge event for the first window and the second window. In this way, the user can send the corresponding input operation or merge event to the first electronic device through a track operation or a merge control on the second electronic device, so that the first electronic device generates a target instruction.
[0096] In some embodiments, the above step S106 “responding to the target instruction” can also be implemented by executing the following steps:
[0097] From the N virtual screen rendering applications that have been created, two application programs whose resolution difference meets the target range are determined as the first application program and the second application program.
[0098] Here, the resolution of an application refers to the pixel size of the application when it is displayed on the display device, which determines the size and layout of the application interface elements. For example, video playback and image processing applications have higher resolutions, while computer, call interface, file management and other applications have lower resolutions.
[0099] In the implementation process, the resolutions of the two applications can be subtracted to obtain a resolution difference. If the resolution difference between the two applications is determined to be small based on a preset target range, the two applications can be used as the first application and the second application that can be drawn on one virtual screen.
[0100] In some embodiments, the frame rate of the application may also be obtained, and two applications whose frame rate difference and resolution difference satisfy the target range are determined as the first application and the second application.
[0101] Here, since different applications correspond to different frame rates, two applications with similar frame rates can be used as the first application and the second application. The two applications are encoded and transmitted in the same way to achieve the optimal effect of displaying two interactive interfaces in the same window of the second electronic device.
[0102] In the embodiment of the present application, two applications whose resolution difference meets the target range are determined from the N applications that have been created for virtual screen drawing as the first application and the second application. In this way, the same encoding and transmission are performed on the two applications, so that the optimal effect of displaying two interactive interfaces in the same window of the second electronic device can be achieved.
[0103] In some embodiments, the implementation of "drawing the first interactive interface and the second interactive interface based on the first virtual screen" in the above step S106 can be achieved by the following steps:
[0104] Step 1061: drawing the first interactive interface in a first area of the first virtual screen, and drawing the second interactive interface in a second area of the first virtual screen, wherein the first area and the second area are different areas that do not overlap;
[0105] In some embodiments, the first virtual screen may be evenly divided into a first area and a second area, and the first interaction interface and the second interaction interface may be drawn on the first area and the second area, respectively. For example, the first virtual screen may be divided into two equal parts (the first area and the second area).
[0106] In some embodiments, the first area and the second area of different sizes may be divided based on different attributes of the first application and the second application. For example, when the first application is a video application and the second application is a memo application, the area of the first area allocated to the first application is larger than the area of the second area allocated to the second application.
[0107] Step 1062, encode the image data corresponding to the first interactive interface and the second interactive interface displayed on the first virtual screen, and transmit the encoded image data in the form of streaming media to the second electronic device to display the first interactive interface and the second interactive interface in a first window.
[0108] In some embodiments, an encoding method may be used to encode the image data of the first virtual screen, and the encoded image data may be transmitted to the second electronic device in the form of streaming media.
[0109] In some embodiments, different encoding methods may be used for the first interactive interface and the second interactive interface, respectively, and the two encoded image data may be transmitted to the second electronic device in the form of streaming media.
[0110] In the embodiment of the present application, the first interactive interface is first drawn in the first area of the first virtual screen, and the second interactive interface is drawn in the second area of the first virtual screen; then, the image data corresponding to the first interactive interface and the second interactive interface displayed on the first virtual screen are encoded, and the encoded image data is transmitted to the second electronic device in the form of streaming media to display the first interactive interface and the second interactive interface in the first window. In this way, different interactive interfaces are first drawn in different areas of the first virtual screen, and then the drawn image data is encoded, and the encoded image data is sent to the second electronic device, so that the first interactive interface and the second interactive interface are drawn on the same virtual screen, and the encoded data can also be sent to the second electronic device, saving encoding resources.
[0111] In some embodiments, the above step 1062 "encodes the image data corresponding to the first interactive interface and the second interactive interface displayed on the first virtual screen, and transmits the encoded image data to the second electronic device in the form of streaming media to display the first interactive interface and the second interactive interface in the first window", such as Figure 4 As shown, this can be achieved by following the steps below:
[0112] Step S410: encoding first image data of the first interactive interface displayed in the first area using a first encoding method based on a first resolution and / or a first frame rate of the first application, and transmitting the encoded first image data to the second electronic device in the form of streaming media so that the first interactive interface is displayed in the first area of the first window;
[0113] Here, the resolution of the application refers to the pixel size of the application when it is presented on the display device, that is, the length multiplied by the width of the application's interactive interface. The frame rate (FPS) of the application refers to the number of frames rendered per second, reflecting the update speed of the application screen.
[0114] Here, the first resolution of the first application and the second resolution of the second application may be greatly different. For example, the first application may be an image program, and the second application may be a clock display program. Obviously, the resolution of a game program is much greater than that of a clock display program.
[0115] The first frame rate of the first application and the second frame rate of the second application may be greatly different. For example, the first application may be a game program and the second application may be a text message display program. Obviously, the frame rate of the game program is much higher than that of the text message display program.
[0116] In the implementation process, different encoding methods can be used for applications with different resolutions and / or frame rates. For example, when the resolution and / or frame rate of a first application is much larger than that of a second application, an encoding method that causes less loss in resolution and / or frame rate of the first application and restores better quality after decoding can be used.
[0117] Step S420: based on the second resolution and / or the second frame rate of the second application, encode the second image data of the second interactive interface displayed in the second area using a second encoding method, and transmit the encoded second image data to the second electronic device in the form of streaming media so that the second interactive interface is displayed in the second area of the first window;
[0118] In the implementation process, different encoding methods can be used for applications with different resolutions and / or frame rates. For example, when the resolution and / or frame rate of the first application is much greater than that of the second application, an encoding method with a higher compression ratio can be used for the second application to effectively save encoding resources without affecting the display of the second interactive interface of the second application on the second electronic device.
[0119] During the implementation process, step S410 and step S420 may be executed simultaneously regardless of the order of execution.
[0120] In an embodiment of the present application, based on the first resolution and / or the first frame rate of the first application, the first image data of the first interactive interface displayed in the first area is encoded using a first encoding method, and the encoded first image data can be transmitted to the second electronic device in the form of streaming media to display the first interactive interface in the first area of the first window. Based on the second resolution and / or the second frame rate of the second application, the second image data of the second interactive interface displayed in the second area is encoded using a second encoding method, and the encoded second image data is transmitted to the second electronic device in the form of streaming media to display the second interactive interface in the second area of the first window. In this way, different encoding methods can be used to encode the first interactive interface and the second interactive interface, respectively, effectively saving encoding resources without affecting the presentation of the interactive interface on the second electronic device.
[0121] Based on the foregoing embodiments, the embodiments of the present application provide a processing device, which includes the modules included, each module includes each sub-module, each sub-module includes a unit, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0122] Figure 5 A schematic diagram of the structure of the processing device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:
[0123] A connection establishing module 510 is used to establish a connection channel with a second electronic device;
[0124] A first creation module 520, configured to create a first virtual screen, wherein the first virtual screen is used to draw a first interactive interface of a first application;
[0125] A first acquisition and transmission module 530, configured to acquire a drawing image based on the first virtual screen and transmit the image to the second electronic device in the form of a media stream through the connection channel so as to display the first interactive interface in a first window;
[0126] A second creation module 540, configured to create a second virtual screen, wherein the second virtual screen is used to draw a second interactive interface of a second application;
[0127] A second acquisition and transmission module 550 is used to obtain a drawing image based on the second virtual screen and transmit it to the second electronic device in the form of a media stream through the connection channel so that the second interactive interface is displayed in a second window;
[0128] The drawing and stopping transmission module 560 is used to respond to the target instruction, draw the first interactive interface and the second interactive interface based on the first virtual screen, and at the same time, stop transmitting the drawing image of the second virtual screen in the form of a media stream through the connection channel, so that the second electronic device displays the first interactive interface and the second interactive interface in a first window.
[0129] In some embodiments, the processing device also includes a first acquisition module and a first generation module, wherein the first acquisition module is used to obtain the target parameters; the first generation module is used to generate the target instruction if the target parameters meet the resource threshold; wherein the resource threshold includes any one of the following: a threshold for the number of virtual screens created; a threshold for encoding resources transmitted in the form of streaming media; a bandwidth resource threshold for the connection channel.
[0130] In some embodiments, the processing device also includes a first creation module and a second generation module, wherein the first creation module is used to create the N+1th virtual screen based on the N+1th application, wherein N is an integer greater than or equal to 2; and the second generation module is used to generate the target instruction based on the creation of the N+1th virtual screen.
[0131] In some embodiments, the device further includes a second acquisition module and a parsing module; wherein the second acquisition module is used to obtain an input operation, wherein the input operation is an input operation transmitted by the second electronic device to the first window and the second window through the connection channel; wherein the input operation includes trajectory information, representing a first operation object of the first window and a second operation object of the second window; the parsing module is used to parse the input operation and generate the target instruction; or, the device further includes a second acquisition module and a response module, wherein the second acquisition module is used to obtain an input event, wherein the input event is a merge event transmitted by the second electronic device to the first window and the second window through the connection channel; and the response module is used to generate the target instruction in response to the input event.
[0132] In some embodiments, the drawing module and the stop transmission 560 are further used to determine two applications whose resolution difference meets the target range from the N applications that have been created for virtual screen drawing as the first application and the second application.
[0133] In some embodiments, the drawing module and the stop transmission 560 include a drawing submodule and an encoding submodule, wherein the display submodule is used to draw the first interactive interface in the first area of the first virtual screen, and to draw the second interactive interface in the second area of the first virtual screen, wherein the first area and the second area are different areas that do not overlap; the encoding submodule is used to encode the drawing images corresponding to the first interactive interface and the second interactive interface drawn on the first virtual screen, and transmit the encoded image data in the form of streaming media to the second electronic device to display the first interactive interface and the second interactive interface in the first window.
[0134] In some embodiments, the encoding submodule includes a first encoding unit and a second encoding unit, wherein the first encoding unit is used to encode a first drawn image of the first interactive interface displayed in the first area using a first encoding method based on a first resolution and / or a first frame rate of the first application, and transmit the encoded first image data in the form of streaming media to the second electronic device to display the first interactive interface in the first area of the first window; the second encoding unit is used to encode a second drawn image of the second interactive interface displayed in the second area using a second encoding method based on a second resolution and / or a second frame rate of the second application, and transmit the encoded second image data in the form of streaming media to the second electronic device to display the second interactive interface in the second area of the first window.
[0135] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0136] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.
[0137] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the processing method provided in the above embodiment are implemented.
[0138] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps in the processing method provided in the above embodiment when executing the program.
[0139] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).
[0140] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0141] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0142] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0144] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0145] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0146] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0147] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0148] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0149] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0150] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0151] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A processing method, the method include: Establishing a connection channel with a second electronic device; Creating a first virtual screen, where the first virtual screen is used to draw a first interactive interface of a first application; Obtaining a drawing image based on the first virtual screen and transmitting the image to the second electronic device in the form of a media stream through the connection channel so that the first interactive interface is displayed in a first window; Creating a second virtual screen, where the second virtual screen is used to draw a second interactive interface of a second application; Obtaining a drawing image based on the second virtual screen and transmitting the image to the second electronic device in the form of a media stream through the connection channel so that the second interactive interface is displayed in a second window; In response to the target instruction, the first interactive interface and the second interactive interface are drawn based on the first virtual screen, and at the same time, the transmission of the drawn image of the second virtual screen in the form of a media stream through the connection channel is stopped, so that the second electronic device displays the first interactive interface and the second interactive interface in a first window.
2. The method according to claim 1, wherein before responding to the target instruction, the method further include: Get target parameters; If the target parameter satisfies the resource threshold, generating the target instruction; The resource threshold includes any of the following: The threshold for creating the number of virtual screens; Threshold of encoding resources for streaming media; The bandwidth resource threshold of the connection channel.
3. The method according to claim 1, wherein before responding to the target instruction, the method further comprises: include: Based on the N+1th application, create the N+1th virtual screen, where N is an integer greater than or equal to 2; Based on creating the N+1th virtual screen, the target instruction is generated.
4. The method according to claim 1, before responding to the target instruction, the method further comprises one of the following: Obtaining an input operation, where the input operation is an input operation transmitted by the second electronic device through the connection channel to the first window and the second window; in, The input operation includes track information representing a first operation object in the first window and a second operation object in the second window; Parsing the input operation and generating the target instruction; or; Obtaining an input event, where the input event is a merging event for the first window and the second window transmitted by the second electronic device through the connection channel; In response to the input event, the target instruction is generated.
5. The method according to claim 2 or 3, wherein in response to the target instruction, include: From the N virtual screen rendering applications that have been created, two application programs whose resolution difference meets the target range are determined as the first application program and the second application program.
6. The method according to claim 1, wherein the first interactive interface and the second interactive interface are drawn based on the first virtual screen. include: Drawing the first interactive interface in a first area of the first virtual screen, and drawing the second interactive interface in a second area of the first virtual screen, wherein the first area and the second area are different areas that do not overlap; Encode the drawn images corresponding to the first interactive interface and the second interactive interface drawn on the first virtual screen, and transmit the encoded image data to the second electronic device in the form of streaming media to display the first interactive interface and the second interactive interface in a first window.
7. The method of claim 6, wherein the encoding comprises encoding the drawn images corresponding to the first interactive interface and the second interactive interface drawn on the first virtual screen, transmitting the encoded image data in the form of streaming media to the second electronic device so that the first interactive interface and the second interactive interface are displayed in a first window, include: Based on the first resolution and / or the first frame rate of the first application, encode the first drawn image of the first interactive interface displayed in the first area using a first encoding method, and transmit the encoded first image data to the second electronic device in the form of streaming media so that the first interactive interface is displayed in the first area of the first window; Based on the second resolution and / or second frame rate of the second application, a second drawn image of the second interactive interface displayed in the second area is encoded using a second encoding method, and the encoded second image data is transmitted to the second electronic device in the form of streaming media to display the second interactive interface in the second area of the first window.
8. A processing device, the device include: A connection establishment module, used to establish a connection channel with a second electronic device; A first creation module, used to create a first virtual screen, where the first virtual screen is used to draw an interactive interface of a first application; A first acquisition and transmission module, configured to acquire a drawing image based on the first virtual screen and transmit the image to the second electronic device in the form of a media stream through the connection channel so as to display an interactive interface of the first application in a first window; A second creation module, used to create a second virtual screen, where the second virtual screen is used to draw an interactive interface of a second application; A second acquisition and transmission module, configured to acquire a drawing image based on the second virtual screen and transmit the image to the second electronic device in the form of a media stream through the connection channel so as to display an interactive interface of the second application in a second window; A drawing and transmission stop module is used to draw the interactive interface of the first application and the interactive interface of the second application based on the first virtual screen in response to a target instruction, and at the same time, stop transmitting the drawn image of the second virtual screen in the form of a media stream through the connection channel, so that the second electronic device displays the interactive interface of the first application and the interactive interface of the second application in a first window.
9. An electronic device, comprising a memory and a processor, wherein the memory stores an application program interconnected with a second electronic device, and the processor is used to establish a connection channel with the second electronic device; create a first virtual screen, wherein the first virtual screen is used to draw an interactive interface of a first application program; obtain a drawing image based on the first virtual screen and transmit it to the second electronic device in the form of a media stream through the connection channel so that the interactive interface of the first application program is displayed in a first window ; Creating a second virtual screen, where the second virtual screen is used to draw an interactive interface of a second application; Obtain a drawing image based on the second virtual screen and transmit it to the second electronic device in the form of a media stream through the connection channel to display the interaction interface of the second application in a second window; in response to a target instruction, draw the interaction interface of the first application and the interaction interface of the second application based on the first virtual screen, and at the same time, stop transmitting the drawing image of the second virtual screen in the form of a media stream through the connection channel, so that the second electronic device displays the interaction interface of the first application and the interaction interface of the second application in a first window.
10. A computer program product, comprising a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, a connection channel is established with a second electronic device; a first virtual screen is created, wherein the first virtual screen is used to draw an interactive interface of a first application program; an image drawn based on the first virtual screen is obtained and transmitted to the second electronic device in the form of a media stream through the connection channel so that the interactive interface of the first application program is displayed in a first window. ; Creating a second virtual screen, where the second virtual screen is used to draw an interactive interface of a second application; Obtain a drawing image based on the second virtual screen and transmit it to the second electronic device in the form of a media stream through the connection channel to display the interaction interface of the second application in a second window; in response to a target instruction, draw the interaction interface of the first application and the interaction interface of the second application based on the first virtual screen, and at the same time, stop transmitting the drawing image of the second virtual screen in the form of a media stream through the connection channel, so that the second electronic device displays the interaction interface of the first application and the interaction interface of the second application in a first window.
