Cross-device application operation method and device, electronic device and storage medium

Abstract

The invention discloses a cross-device application operation method and device, electronic equipment and a storage medium. The first equipment is connected with second equipment, virtual equipment associated with the second equipment is created in the first equipment, applications in the first equipment are added in the virtual equipment, and the second equipment comprises an application list corresponding to the applications obtained in the first equipment. The first device responds to application information of a target application sent by the second device, the target application is run in the virtual device, a virtual display associated with the target application is created in the virtual device, audio and video information of the target application in the running process is collected through the virtual display, the audio and video information is sent to the second device, and the second device sends the audio and video information to the first device. And the second device plays the audio and video information. According to the application, by creating the virtual device associated with the second device in the first device and creating the virtual display associated with the determined application in the virtual device, the second device operates the application in the first device, and the development cost can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and more specifically, to a cross-device application operation method, apparatus, electronic device, and storage medium. Background Art

[0002] Smart watches, smart home appliances, smart car machines and other devices lack corresponding applications due to large platform differences or limited device computing power. If application developers develop corresponding applications for different devices, it will result in huge workload and high development costs. Summary of the invention

[0003] In view of the above problems, the present application proposes a cross-device application operation method, apparatus, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a cross-device application operation method, which is applied to a first device, the first device is connected to a second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The method comprises: in response to application information of the target application sent by the second device, running the target application in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list; creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display; and sending the audio and video information to the second device to instruct the second device to play the audio and video information.

[0005] In a second aspect, an embodiment of the present application provides a cross-device application operation method, which is applied to a second device, the second device is connected to a first device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The method includes: in response to a selection operation on the application list, determining a target application from the applications in the first device, and obtaining application information of the target application; sending the application information to the first device to instruct the first device to run the target application in the virtual device, and creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display; receiving the audio and video information sent by the first device, and playing the audio and video information.

[0006] In the third aspect, an embodiment of the present application provides a cross-device application operation device, which is applied to a first device, the first device is connected to a second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device, the device includes: a target application running module, which is used to run the target application in the virtual device in response to application information of the target application sent by the second device, wherein the target application is determined by the second device from the applications in the first device based on the application list; an audio and video information collection module, which is used to create a virtual display associated with the target application in the virtual device, and collect audio and video information of the target application during operation through the virtual display; an audio and video information sending module, which is used to send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0007] In a fourth aspect, an embodiment of the present application provides a cross-device application operation device, which is applied to a second device, the second device is connected to a first device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The device includes: an application information acquisition module, which is used to respond to a selection operation on the application list, determine a target application from the applications in the first device, and obtain application information of the target application; an application information sending module, which is used to send the application information to the first device to instruct the first device to run the target application in the virtual device, and create a virtual display associated with the target application in the virtual device, and collect audio and video information of the target application during operation through the virtual display; an audio and video information playback module, which is used to receive the audio and video information sent by the first device and play the audio and video information.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0010] The cross-device application operation method, apparatus, electronic device, and storage medium provided in the embodiments of the present application set a first device to be connected to a second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application obtained in the first device. The first device responds to the application information of the target application sent by the second device, runs the target application in the virtual device, creates a virtual display associated with the target application in the virtual device, and collects the audio and video information of the target application during operation through the virtual display, and sends the audio and video information to the second device to instruct the second device to play the audio and video information, thereby enabling the second device to operate the application in the first device by creating a virtual device associated with the second device in the first device and creating a virtual display associated with the determined application in the virtual device, without the need to develop an application adapted to the second device, which can reduce development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of an application environment that can be used for the cross-device application operation method provided in an embodiment of the present application is shown;

[0013] Figure 2 A timing diagram of a cross-device application operation method provided by an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of an interface of a first device provided in an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing the interaction between a first device and a second device provided in an embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0017] Figure 6 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0018] Figure 7 A schematic diagram showing devices with different screen sizes provided in an embodiment of the present application;

[0019] Figure 8A schematic diagram showing an adaptive layout of applications for devices with different screen sizes provided by an embodiment of the present application;

[0020] Fig. 9 A schematic diagram showing a flexible layout of the application of the device provided in the embodiment of the present application;

[0021] Fig.10 A schematic diagram of creating a standby layout for an application of a device provided in an embodiment of the present application is shown;

[0022] Fig.11 A schematic diagram showing a layout display of a second device provided in an embodiment of the present application is shown;

[0023] Fig.12 A schematic diagram showing a layout display of a second device provided in an embodiment of the present application is shown;

[0024] Fig.13 A schematic diagram showing a layout display of a second device provided in an embodiment of the present application is shown;

[0025] Fig.14 A schematic diagram showing a layout display of a second device provided in an embodiment of the present application is shown;

[0026] Fig.15 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0027] Fig.16 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0028] Fig.17 This application shows Fig.16 The flowchart of step S550 of the cross-device application operation method shown;

[0029] Fig.18 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0030] Fig.19 This application shows Fig.18 The flowchart of step S610 of the cross-device application operation method shown;

[0031] Fig. 20 A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0032] Fig.21 A schematic diagram of cross-device application resource capability sharing provided by an embodiment of the present application is shown;

[0033] Fig. 22A schematic diagram showing a flow chart of a cross-device application operation method provided by an embodiment of the present application;

[0034] Fig.23 A module block diagram of a cross-device application operation apparatus provided by an embodiment of the present application is shown;

[0035] Fig.24 A module block diagram of a cross-device application operation apparatus provided by an embodiment of the present application is shown;

[0036] Fig.25 A block diagram of an electronic device according to an embodiment of the present application for executing a cross-device application operation method according to an embodiment of the present application is shown;

[0037] Fig.26 A storage unit for storing or carrying program codes for implementing a cross-device application operation method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0039] Among them, smart watches, smart home appliances, smart car machines, etc. are often not designed with high storage and processing capabilities due to cost considerations, making it impossible for users to install very useful applications independently. Therefore, such devices generally only have a small number of applications with relatively simple functions. If application developers want to develop corresponding applications for different devices, it will lead to high development costs.

[0040] In view of the above problems, the inventor has discovered after long-term research and proposed a cross-device application operation method, apparatus, electronic device and storage medium provided in the embodiments of the present application. By creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, the second device can operate the application in the first device, without developing an application adapted for the second device, which can reduce development costs. In a specific embodiment, the cross-device application operation method is described in detail in the subsequent embodiments.

[0041] The following describes an application environment that can be used for the cross-device application operation method provided in an embodiment of the present application.

[0042] See also Figure 1 , Figure 1 A schematic diagram of an application environment that can be used for the cross-device application operation method provided in an embodiment of the present application is shown. Figure 1As shown, it includes a first device 100 and a second device 200, wherein the first device 100 and the second device 200 are connected to perform data interaction. Optionally, the first device 100 may include a smart phone, a tablet computer, or other device with strong data processing capabilities and installed with more applications, and the second device 200 may include a smart watch, a smart home appliance, a smart car machine, or other device with smaller storage capacity, smaller data processing capabilities, and installed with fewer applications. Optionally, the connection method between the first device 100 and the second device 200 may include: Bluetooth, near field communication, WLAN direct connection, RTC protocol, MIRACAST, etc., which are not limited here.

[0043] See also Figure 2 , Figure 2 The timing diagram of the cross-device application operation method provided by an embodiment of the present application is shown. The method is used to enable the second device to operate the application in the first device by creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device. There is no need to develop an application adapted to the second device, which can reduce development costs. Figure 2 The process shown is described in detail. In this embodiment, the first device and the second device are connected, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0044] Step S110: the second device determines a target application from applications in the first device in response to a selection operation on the application list, and obtains application information of the target application.

[0045] In this embodiment, a connection can be established between a first device and a second device, and when the connection between the first device and the second device is established, a virtual device associated with the second device is created in the first device, wherein the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device.

[0046] Optionally, the virtual device may be a device virtualized by an operating system of the first device, which is usually run in the background of the first device, and the user cannot directly view the content in the virtual device on the first device.

[0047] Optionally, the first device can actively initiate a connection with the second device, and when the connection with the second device is successfully established, the first device can create a virtual device associated with the second device in the first device, and add the application in the first device to the virtual device. At the same time, the application list corresponding to the application in the first device is sent to the second device, so that the second device includes the application list corresponding to the application in the first device.

[0048] Optionally, the second device can actively initiate a connection with the first device, and when the connection with the first device is successfully established, the first device can create a virtual device associated with the second device in the first device, and add the application in the first device to the virtual device. At the same time, the application list corresponding to the application in the first device is sent to the second device, so that the second device includes the application list corresponding to the application in the first device.

[0049] In some implementations, when the first device establishes a connection with the second device, the identifier corresponding to the second device may be displayed on the first device, such as Figure 3 For example, an image corresponding to the second device may be displayed on the first device, an icon corresponding to the second device may be displayed on the first device, or a name corresponding to the second device may be displayed on the first device, etc., which are not limited here.

[0050] In this embodiment, when the second device displays an application list corresponding to an application in the first device, it can detect operations acting on the application list. If a selection operation acting on the application list is detected, it can respond to the selection operation acting on the application list, determine the target application from the applications of the first device, and obtain application information of the target application.

[0051] Optionally, the application information of the target application may include: an application name of the target application, a package name of the target application, a number of the target application, and other information used to uniquely identify the target application.

[0052] Optionally, the application list may include application icons of applications in the first device arranged in order. Based on this, the selection operation acting on the application list may include: a single-finger click operation acting on an application icon, a multi-finger click operation acting on an application icon, a single-finger press operation acting on an application icon, a multi-finger press operation acting on an application icon, a single-finger slide operation acting on an application icon, a multi-finger slide operation acting on an application icon, etc., which are not limited here.

[0053] As an example, taking the selection operation on the application list as a single-finger click operation on the application icon as an example, when the second device is displaying the application list, if it detects a single-finger click operation on a certain application icon, it can determine the application corresponding to the certain application icon from the applications of the first device as the target application, and obtain the application information of the target application.

[0054] Step S120: the second device sends the application information to the first device.

[0055] In this embodiment, when the second device obtains the application information of the target application, it can send the application information to the first device. Optionally, if the application information is an application name, when the second device obtains the application name of the target application, it can send the application name to the first device.

[0056] In some implementations, the second device may establish a message transmission channel with the first device when obtaining the application information of the target application (eg, Figure 4 As shown), based on this, the second device can send application information to the first device through the message transmission channel. Optionally, the message transmission channel can be a long connection channel established based on protocols such as Bluetooth or TCP, which has low channel power consumption and can meet the needs of the interaction of device information with less data volume between the first device and the second device.

[0057] Step S130: the first device runs the target application in the virtual device in response to the application information sent by the second device.

[0058] In this embodiment, when the first device receives the application information of the target application sent by the second device, it can run the target application in the virtual device in response to the application information.

[0059] In some implementations, when the first device receives application information of a target application sent by the second device, it can determine a target application corresponding to the application information from applications in the first device based on the application information and run the target application in the virtual device.

[0060] As an implementable manner, when the first device receives the application information of the target application sent by the second device, it can respond to the application information of the target application sent by the second device, pull up the target application in the virtual device, and establish a data transmission channel with the second device (such as Figure 4As shown), then, the target application is run in the virtual device. Optionally, the data transmission channel is created when the target application is launched, and can be used to transmit the audio and video stream and control instruction stream between the first device and the second device in real time. Optionally, the data transmission channel can be a long connection channel established based on the UDP protocol, which has good real-time performance, can transmit a large amount of data in real time, has good interactive effect, and can meet the needs of data transmission between the first device and the second device at any time.

[0061] Step S140: the first device creates a virtual display associated with the target application in the virtual device, and collects audio and video information of the target application during operation through the virtual display.

[0062] In this embodiment, when the first device receives the application information of the target application sent by the second device, it can create a virtual display (Virtual Display) associated with the target application in the virtual device, and collect the audio and video information (including pictures and audio) of the target application during operation through the virtual display. It can be understood that the application in the virtual device will run in the background of the first device and will not affect the normal use of the first device.

[0063] Optionally, the virtual screen may be a screen virtualized by an operating system of the first device, which is usually run in the background of the first device, and the user cannot directly see the image on the virtual screen on the first device.

[0064] In some implementations, when the first device receives the application information of the target application sent by the second device, it can create a virtual display and bind the created virtual display to the target application to complete the association between the target application and the virtual display.

[0065] In some embodiments, when the first device creates a virtual display, the audio and video information of the target application during its operation can be collected in real time through the virtual display. The audio and video information of the target application during its operation can be collected through the virtual display at preset time intervals. The audio and video information of the target application during its operation can be collected through the virtual display at preset time points. The audio and video information of the target application during its operation can be collected through the virtual display according to other rules, etc., which are not limited here.

[0066] Step S150: The first device sends the audio and video information to the second device.

[0067] In this embodiment, when the first device collects audio and video information, it can send the audio and video information to the second device.

[0068] In some implementations, when the first device collects audio and video information, it can transmit the information through a data transmission channel (such as Figure 4 As shown), the audio and video information is sent to the second device.

[0069] Step S160: the second device plays the audio and video information.

[0070] In this embodiment, when the second device receives the audio and video information sent by the first device, it can play the audio and video information to complete the cross-terminal operation and playback of the application of the first device by the second device, thereby eliminating the need to install the corresponding application on the second device and reducing software development costs.

[0071] In some implementations, when the second device receives the audio and video information sent by the first device, it can render the audio and video information and then play the audio and video information.

[0072] The cross-device application operation method provided by an embodiment of the present application sets a connection between a first device and a second device, creates a virtual device associated with the second device in the first device, adds the application in the first device to the virtual device, and includes an application list corresponding to the application obtained in the first device in the second device. The second device responds to the selection operation acting on the application list, determines the target application from the applications in the first device, and obtains the application information of the target application. The second device sends the application information to the first device. The first device responds to the application information sent by the second device and runs the target application in the virtual device. The first device creates a virtual display associated with the target application in the virtual device, and collects the audio and video information of the target application during operation through the virtual display. The first device sends the audio and video information to the second device, and the second device plays the audio and video information, thereby creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, so that the second device can operate the application in the first device, without developing an application adapted to the second device, which can reduce the development cost.

[0073] See also Figure 5 , Figure 5 The flowchart of the cross-device application operation method provided by an embodiment of the present application is shown. The method is used to enable the second device to operate the application in the first device by creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, without developing an application adapted to the second device, which can reduce development costs. In a specific embodiment, the method is applied to Fig.23 The cross-device application operating apparatus 300 and the first device 100 ( Figure 1 ). Figure 5 The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0074] Step S210: In response to the application information of the target application sent by the second device, the target application is run in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0075] Step S220: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0076] Step S230: Send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0077] For the detailed description of step S210 to step S230 , please refer to step S110 to step S160 , which will not be repeated here.

[0078] An embodiment of the present application provides a cross-device application operation method, which sets a connection between a first device and a second device, creates a virtual device associated with the second device in the first device, adds applications in the first device to the virtual device, and includes an application list corresponding to the applications obtained in the first device in the second device. The first device responds to the application information of the target application sent by the second device, runs the target application in the virtual device, creates a virtual display associated with the target application in the virtual device, and collects audio and video information of the target application during operation through the virtual display, and sends the audio and video information to the second device to instruct the second device to play the audio and video information, thereby enabling the second device to operate the application in the first device by creating a virtual device associated with the second device in the first device and creating a virtual display associated with the determined application in the virtual device, without the need to develop an application adapted to the second device, which can reduce development costs.

[0079] See also Figure 6 , Figure 6 The following is a flow chart of a cross-device application operation method provided by an embodiment of the present application. Figure 6The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0080] Step S310: determining a screen resolution corresponding to the second device in response to application information of a target application sent by the second device, wherein the target application is determined by the second device from applications in the first device based on the application list.

[0081] In this embodiment, when the first device receives the application information of the target application sent by the second device, it can determine the screen resolution of the second device.

[0082] As an implementable manner, when a first device and a second device are connected, the second device can send its screen resolution to the first device, and accordingly, the first device can store the screen resolution corresponding to the second device locally. Based on this, when the first device receives the application information of the target application sent by the second device, it can obtain the screen resolution corresponding to the second device locally.

[0083] As another practicable manner, when the second device sends the application information of the target application to the first device, it can also send its screen resolution to the first device. Based on this, when the first device receives the application information of the target application sent by the second device, it can also obtain its screen resolution sent by the second device.

[0084] Step S320: If the target application supports the screen resolution, the target application is run in the virtual device according to the screen resolution.

[0085] It is understandable that different devices often have different screen parameters (such as screen resolution). For example, smart phones, smart watches, smart car machines and other devices may even have special-shaped screens (such as Figure 7 In order to adapt to devices with different screen resolutions, most applications often use adaptive layout to design UI, so that users can have a better interactive experience on different devices (such as Figure 8 Common adaptive layout methods include flexible layout methods (such as Fig. 9 ) or create an alternate layout (as shown in Fig.10As shown), the flexible layout method can automatically adjust the width of the view according to the screen width, and the alternative layout method can create additional screen layouts so that the same application is displayed on screens of different sizes, using different layouts for each screen size. The adaptive layout of the application can make the application running on the first device perfectly displayed on the screen of the second device, providing users with the best UI interaction experience. However, not all applications can adapt to all screen sizes. The UI design of an application often only considers the screen sizes of common devices. If the application declares restricted screen support, for example, it declares that it only supports a specific aspect ratio or only supports a specific screen size, then the application needs to be adapted for cross-device UI.

[0086] When it is necessary to transfer an application in a first device to a second device for playback and display, since the application in the first device may or may not support the screen resolution of the second device, it is possible to detect whether the application in the first device supports the screen resolution of the second device.

[0087] In some implementations, when the screen resolution corresponding to the second device is determined, it can be determined whether the target application supports the screen resolution. In some implementations, a reference resolution supported by the target application can be obtained, and the reference resolution supported by the target application can be compared with the screen resolution, wherein if the comparison result indicates that the reference resolution supported by the target application includes the screen resolution, it can be determined that the target application supports the screen resolution, and if the comparison result indicates that the reference resolution supported by the target application does not include the screen resolution, it can be determined that the target application does not support the screen resolution.

[0088] In this embodiment, if it is determined that the target application supports the screen resolution, the target application can be run in the virtual device according to the screen resolution to ensure that the audio and video parameters corresponding to the target application can adapt to the screen resolution of the second device when played on the second device, thereby improving the display effect of the target application on the second device.

[0089] Step S330: If the target application does not support the screen resolution, determine a plurality of reference resolutions supported by the target application.

[0090] In this embodiment, if it is determined that the target application does not support the screen resolution, multiple reference resolutions supported by the target application may be determined.

[0091] In some embodiments, when installing the target application, the first device can obtain multiple reference resolutions supported by the target application and store the multiple reference resolutions supported by the target application locally. Based on this, if it is determined that the target application does not support the screen resolution, the multiple reference resolutions supported by the target application can be obtained locally.

[0092] In some implementations, when it is determined that the target application does not support the screen resolution, multiple reference resolutions supported by the target application may be acquired from a communication connected server (eg, a manufacturer server corresponding to the target application).

[0093] Step S340: determining a target resolution based on the multiple reference resolutions and the screen resolution, and running the target application in the virtual device according to the target resolution.

[0094] In this embodiment, when multiple reference resolutions supported by the target application are obtained, the target resolution can be determined based on the multiple reference resolutions and the screen resolution, and the target application can be run in the virtual device according to the target resolution to ensure that the audio and video information corresponding to the target application can be close to the screen resolution of the second device when played on the second device, thereby improving the display effect of the target application on the second device.

[0095] In some implementations, when multiple reference resolutions supported by the target application are obtained, a reference resolution closest to the screen resolution may be selected from the multiple resolutions, and the selected reference resolution closest to the screen resolution may be used as the target resolution.

[0096] As an implementable method, when multiple reference resolutions supported by the target application are obtained, the reference resolution closest to the screen resolution among the multiple reference resolutions can be calculated, and the reference resolution closest to the screen resolution can be used as the target resolution, and the target application can be run in the virtual device according to the target resolution.

[0097] Specifically, when multiple reference resolutions supported by the target application are obtained, the multiple reference resolutions can be stored in a list List to generate a supported size supportedSizes, the length of the screen resolution of the second device is set to the requested length RequestLength, and the width of the screen resolution of the second screen is set to the requested width RequestWidth, and the most suitable width and length, that is, the target resolution, is calculated through the getClosestSupportedSize algorithm.

[0098] Among them, it can be understood that the target resolution is a reference resolution supported by the target application that is closest to the screen resolution. The aspect ratio of the target resolution is inconsistent with the aspect ratio of the screen resolution, and the aspect ratio of the target resolution may be greater than the aspect ratio of the screen resolution, or the aspect ratio of the target resolution may be smaller than the aspect ratio of the screen resolution. Therefore, in this embodiment, when the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution or smaller than the aspect ratio of the screen resolution, different display modes can be set to improve the display effect of the target application on the second device.

[0099] In some implementations, if the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution, the target application can be run in a manner that the width of the application interface of the target application is kept consistent with the width of the screen resolution and the length of the application interface is truncated to be consistent with the length of the screen resolution. Fig.11 As shown, when the audio and video information corresponding to the target application is displayed on the second device, the width of the display interface of the target application is consistent with the width of the screen resolution, and the length of the application interface is truncated to the length consistent with the screen resolution. Based on this, the user can switch to display the complete interface of the target application by dragging up and down on the display interface of the second device.

[0100] In some implementations, if the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution, the target application can be run in a manner that the application interface of the target application is proportionally reduced to the length consistent with the screen resolution in the virtual device. Fig.12 As shown, when the audio and video information corresponding to the target application is displayed on the second device, the length of the application interface of the target application is consistent with the length of the screen resolution, and the width of the application interface is smaller than the width of the screen resolution, that is, there is blank space on both sides of the screen width of the second device.

[0101] In some implementations, if the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution, the target application can be run in a manner that the length of the application interface of the target application is kept consistent with the length of the screen resolution, and the width of the application interface is truncated to be consistent with the width of the screen resolution. Fig.13 As shown, when the audio and video information corresponding to the target application is displayed on the second device, the length of the application interface of the target application is consistent with the length of the screen resolution, and the width of the application interface is truncated to be consistent with the width of the screen resolution. Based on this, the full interface of the target application can be switched by dragging left and right on the display interface of the second device.

[0102] In some implementations, if the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution, the target application can be run in the virtual device by scaling down the application interface of the target application to a width that is consistent with the width of the screen resolution. Fig.14 As shown, when the audio and video information corresponding to the target application is displayed on the second device, the width of the application interface of the target application is consistent with the width of the screen resolution, and the length of the application interface is smaller than the length of the screen resolution, that is, there is blank space on both sides of the screen length of the second device.

[0103] As an implementable method, the second device can set a default layout display mode, and during use, it can switch to another layout display mode according to the switching instruction input by the user. For example, assuming that the second device is set by default to run the layout display mode of the target application in a manner that if the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution, the width of the application interface of the target application is kept consistent with the width of the screen resolution in the virtual device, and the length of the application interface is truncated to the length consistent with the screen resolution, then, when the switching instruction input by the user is detected, it can be switched to run the target application in a manner that proportionally reduces the application interface of the target application in the virtual device to the length consistent with the screen resolution.

[0104] Step S350: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0105] In some embodiments, if it is determined that the target application supports the screen resolution, a virtual display associated with the target application can be created in the virtual device based on the screen resolution, and the audio and video information of the target application during operation can be collected through the virtual display. That is, if it is determined that the target application supports the screen resolution, an associated virtual display is created based on the screen resolution, so that the virtual display is adapted to the screen resolution, so that the subsequent cross-device display of the target application can be adapted to the screen resolution of the second device.

[0106] In some embodiments, if it is determined that the target application does not support the screen resolution, a virtual display associated with the target application can be created in the virtual device based on the target resolution, and the audio and video information of the target application during operation can be collected through the virtual display. That is, if it is determined that the target application does not support the screen resolution, an associated virtual display is created based on the target resolution so that the virtual display is adapted to the target resolution, so that the subsequent cross-device display of the target application can be close to the screen resolution of the second device.

[0107] Step S360: Send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0108] For the detailed description of step S360, please refer to steps S150 to S160, which will not be repeated here.

[0109] Compared with the cross-device application operation method provided by an embodiment of the present application, Figure 5 The cross-device application operation method shown in the present embodiment further determines the screen resolution of the second device. If the target application supports the screen resolution, the target application is run in the virtual device according to the screen resolution. Alternatively, if the target application does not support the screen resolution, multiple reference resolutions supported by the target application are determined, the target resolution is determined based on the multiple reference resolutions and the screen resolution, and the target application is run in the virtual device according to the target resolution, so that the display of the application interface of the second device corresponding to the target application can be adapted, thereby improving the interface adaptability of cross-device application operations.

[0110] See also Fig.15 , Fig.15 The following is a flow chart of a cross-device application operation method provided by an embodiment of the present application. Fig.15 The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0111] Step S410: In response to the application information of the target application sent by the second device, the target application is run in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0112] Step S420: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0113] For the detailed description of step S410 to step S420, please refer to step S110 to step S140, which will not be repeated here.

[0114] Step S430: Encode the audio and video information to obtain encoded audio and video information.

[0115] In this embodiment, when the first device collects audio and video information, it can encode the audio and video information to obtain the encoded audio and video information.

[0116] As an implementable manner, the first device may be pre-set with a target encoding method, and when audio and video information is collected, the audio and video information may be encoded based on the target encoding method to obtain the encoded audio and video information.

[0117] As another practicable manner, the first device may be pre-set with multiple encoding modes, and when audio and video information is collected, a target encoding mode is determined from the multiple encoding modes based on the audio and video information, and the audio and video information is encoded based on the target encoding mode to obtain the encoded audio and video information. Optionally, determining the target encoding mode from the multiple encoding modes based on the audio and video information may include: determining the target encoding mode from the multiple encoding modes based on the data volume corresponding to the audio and video information; determining the target encoding mode from the multiple encoding modes based on the real-time requirements corresponding to the audio and video information, etc., which are not limited here.

[0118] In some implementations, when the first device collects audio and video information, it may encrypt the audio and video information to obtain the encrypted audio and video information, and then encode the encrypted audio and video information to obtain the encoded audio and video information. Optionally, it may be determined whether the first device and the second device belong to the same user. If it is determined that the first device and the second device belong to the same user, the audio and video information may not be encrypted. If it is determined that the first device and the second device do not belong to the same user, the audio and video information may be encrypted.

[0119] Step S440: Send the encoded audio and video information to the second device to instruct the second device to decode and play the encoded audio and video information.

[0120] In this embodiment, when the first device obtains the encoded audio and video information, it can send the encoded audio and video information to the second device to instruct the second device to decode and play the encoded audio and video information. Optionally, when the second device obtains the encoded audio and video information sent by the first device, it can decode and play the encoded audio and video information in a corresponding decoding manner.

[0121] In some embodiments, if the first device encrypts the audio and video information, then, upon obtaining the encrypted and encoded audio and video information, the first device can send the encrypted and encoded audio and video information to the second device to instruct the second device to decode and decrypt the encrypted and encoded audio and video and play it.

[0122] Compared with the cross-device application operation method provided by an embodiment of the present application, Figure 5The cross-device application operation method shown in this embodiment also encodes the audio and video information, obtains the encoded audio and video information, and sends the encoded audio and video information to the second device to instruct the second device to decode the encoded audio and video information and then play it. In this way, when the second device performs cross-device application operations, it only needs to have basic network and audio and video decoding capabilities to run a large number of applications, thereby reducing the hardware cost of the second device.

[0123] See also Fig.16 , Fig.16 The following is a flow chart of a cross-device application operation method provided by an embodiment of the present application. Fig.16 The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0124] Step S510: In response to the application information of the target application sent by the second device, the target application is run in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0125] Step S520: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0126] Step S530: Send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0127] For the detailed description of step S510 to step S530 , please refer to step S110 to step S160 , which will not be repeated here.

[0128] Step S540: receiving an operation instruction sent by the second device, wherein the operation instruction is generated by the second device according to an operation performed on the second device during the process of playing the audio and video information.

[0129] In the process of playing audio and video information, the second device can detect the operation acting on the second device. If the operation acting on the second device is detected in the process of playing audio and video information, an operation instruction can be generated according to the operation acting on the second device and sent to the first device. Accordingly, the first device can receive the operation instruction sent by the second device.

[0130] In some embodiments, when the second device plays audio and video information, it can display a display interface corresponding to the audio and video information. Based on this, the second device can detect the touch operation acting on the display interface in the process of displaying the display interface corresponding to the audio and video information, wherein if the touch operation acting on the display interface is detected, an operation instruction can be generated according to the touch operation acting on the display interface. Optionally, the touch operation acting on the display interface may include: a single-finger click operation acting on the display interface, a multi-finger click operation acting on the display interface, a single-finger press operation acting on the display interface, a multi-finger press operation acting on the display interface, a single-finger slide operation acting on the display interface, a multi-finger slide operation acting on the display interface, etc., which are not limited here.

[0131] In some implementations, when the second device generates an operation instruction, it may encapsulate the operation instruction and send the encapsulated operation instruction to the first device. For example, after the second device encapsulates the operation instruction, it may send the encapsulated operation instruction to the first device through a data transmission channel.

[0132] Step S550: operating the target application in the virtual device according to the operating instruction.

[0133] In this embodiment, when the first device receives the operation instruction sent by the second device, it can perform a corresponding operation on the target application in the virtual device according to the operation instruction. For example, if the operation instruction indicates to enter a certain interface of the target application, the corresponding operation can be performed on the target application in the virtual device to enter a certain interface of the target application; if the operation instruction indicates to exit a certain page of the target application, the corresponding operation can be performed on the target application in the virtual device to exit a certain interface of the target application, etc., which is not limited here.

[0134] In some embodiments, when there are multiple second devices, and each of the multiple second devices sends an operation instruction to the first device, the first device can operate the target application in the virtual device according to the operations sent by each of the multiple second devices. That is, when the first device is interconnected with multiple second devices, and each of the multiple second devices runs the target application, then each of the multiple second devices can operate the target application, and accordingly, the first device can respond to the operations of each of the multiple second devices on the target application.

[0135] As an implementable manner, when a first device is interconnected with multiple second devices, if the multiple second devices all run the target application (i.e., run the same application), the target applications running in different second devices can jointly operate the files in the first device. Optionally, assuming that the target application is a video editing application, the first device is a smart phone, and the second devices include a smart watch and a smart car machine, the smart phone, the smart watch, and the smart car machine all run the video editing application, and the user can control the video editing application through the smart watch and / or the smart car machine, and the smart phone updates the operation progress accordingly. In addition, the user can add local materials to the video editing application on the smart watch and / or the smart car machine, for example, add the video file in the smart watch to the video editing application, and then edit the added video file on the smart car machine.

[0136] See also Fig.17 , Fig.17 This application shows Fig.16 The flowchart of step S550 of the cross-device application operation method shown in FIG. Fig.17 The process shown is described in detail, and the method may specifically include the following steps:

[0137] Step S551: Process the operation instruction based on instruction mapping or coordinate conversion to obtain an operation instruction adapted to the first device.

[0138] In some implementations, when the first device receives an operation instruction sent by the second device, it may convert the operation instruction to obtain an operation instruction that the first device can understand, that is, obtain an operation instruction that is compatible with the first device.

[0139] As an implementable manner, when the first device receives an operation instruction sent by the second device, it can process the operation instruction based on the instruction mapping to obtain an operation instruction adapted to the first device. That is, when the first device receives an operation instruction sent by the second device, it can convert the operation instruction such as the key in the second device into an operation instruction that can be understood by the target application of the first device, that is, obtain an operation instruction that can be understood by the first device (an operation instruction adapted to the first device).

[0140] As another practicable manner, when the first device receives an operation instruction sent by the second device, it can process the operation instruction based on coordinate conversion to obtain an operation instruction adapted to the first device. That is, when the first device receives an operation instruction sent by the second device, it can convert the coordinates corresponding to the icon in the display interface of the second device into the corresponding coordinates in the target application in the first device, that is, obtain an operation instruction that the first device can understand (an operation instruction adapted to the first device).

[0141] Step S552: operating the target application in the virtual device according to an operation instruction adapted to the first device.

[0142] In some embodiments, when the first device obtains an operating instruction adapted for the first device, it can operate the target application in the virtual device according to the operating instruction adapted for the first device, that is, the operating instruction adapted for the first device is transferred to the virtual device to operate the target application.

[0143] Compared with the cross-device application operation method provided by an embodiment of the present application, Figure 5 The cross-device application operation method shown, this embodiment also receives an operation instruction sent by the second device, wherein the operation instruction is generated by the second device according to the operation acting on the second device during the process of playing audio and video information, and according to the operation instruction, the target application is operated in the virtual device, so that the second device can perform cross-device application operations on the first device.

[0144] See also Fig.18 , Fig.18 The following is a flow chart of a cross-device application operation method provided by an embodiment of the present application. Fig.18 The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0145] Step S610: creating a virtual device associated with the second device in the first device.

[0146] In this embodiment, when a connection is established between the first device and the second device, a virtual device associated with the second device can be created in the first device. Optionally, the device virtualization method includes but is not limited to: hardware virtualization, operating system virtualization, application layer virtualization, and container virtualization. It can be understood that the virtualized application is independent of the original application and will not interfere with the work of the original application.

[0147] See also Fig.19 , Fig.19 This application shows Fig.18 The flowchart of step S610 of the cross-device application operation method shown in FIG. Fig.19 The process shown is described in detail, and the method may specifically include the following steps:

[0148] Step S611: Acquire parameter information of the second device.

[0149] In some implementations, when a connection is established between a first device and a second device, the second device may send its parameter information to the first device, and correspondingly, the first device may obtain the parameter information of the second device.

[0150] Optionally, the parameter information may include: screen size, screen shape, screen resolution, decoder performance, audio playback capability, etc., which are not limited here.

[0151] Step S612: Based on the parameter information, create a virtual device associated with the second device in the first device.

[0152] In some implementations, when the parameter information of the second device is obtained, a virtual device associated with the second device may be created in the first device based on the parameter information.

[0153] Step S620: Add the applications in the first device to the virtual device, and send the application list corresponding to the applications in the first device to the second device.

[0154] In this embodiment, when the first device creates a virtual device associated with the second device, the application in the first device can be added to the virtual device, and at the same time, the application list corresponding to the application in the first device can be sent to the second device.

[0155] In some embodiments, when creating a virtual device associated with a second device, the first device can obtain applications installed on the first device and add the applications installed on the first device to the virtual device. At the same time, the application list corresponding to the applications installed on the first device is sent to the second device.

[0156] In some embodiments, when the first device creates a virtual device associated with the second device, it can obtain the applications installed by the first device, determine the authorized applications from the applications installed by the first device, add the authorized applications of the first device to the virtual device, and at the same time, send the application list corresponding to the authorized applications of the first device to the second device.

[0157] Step S630: In response to the application information of the target application sent by the second device, the target application is run in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0158] Step S640: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0159] Step S650: Send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0160] For the detailed description of step S630 to step S650 , please refer to step S110 to step S160 , which will not be repeated here.

[0161] Compared with the cross-device application operation method provided by an embodiment of the present application, Figure 5 The cross-device application operation method shown in the figure, this embodiment also creates a virtual device associated with the second device in the first device, adds the application in the first device to the virtual device, and sends the application list corresponding to the application in the first device to the second device, so that the second device can use the application service in the first device, effectively reducing development costs.

[0162] See also Fig. 20 , Fig. 20 The following is a flow chart of a cross-device application operation method provided by an embodiment of the present application. Fig. 20 The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0163] Step S710: In response to the application information of the target application sent by the second device, the target application is run in the virtual device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0164] Step S720: creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display.

[0165] Step S730: Send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0166] For the detailed description of step S710 to step S730, please refer to step S110 to step S160, which will not be repeated here.

[0167] Step S740: Send a request message to the second device to request to call one or more hardware of the second device to collect and / or output data.

[0168] In this embodiment, the first device can send a request message to the second device to request to call one or more hardware of the second device to collect and / or output data. Optionally, the first device runs the target application (such as a video call application) in the virtual device, and the audio and video information corresponding to the target application is sent to the second device for playback. During this process, the first device can apply to call the hardware of the second device to collect and / or output data through the hardware of the second device. For example, the first device can apply to call the camera of the second device to collect image information, the first device can apply to call the microphone of the second device to collect audio information, the first device can apply to call the speaker of the second device to output audio information, etc., which is not limited here.

[0169] As an implementable approach, Fig.21 As shown, assuming that the first device is a smart phone and the second device is a smart car computer, the device adapter module of the smart car computer can call the local hardware capabilities of the smart car computer, virtualize the hardware capabilities of the smart car computer, and generate corresponding virtualized hardware resources on the smart phone. Accordingly, the application running on the smart phone can call the hardware resources of the smart car computer just like calling the local hardware of the smart phone.

[0170] Compared with the cross-device application operation method provided by an embodiment of the present application, Figure 5 The cross-device application operation method shown in this embodiment also sends a request message to the second device to request to call one or more hardware of the second device to collect and / or output data, thereby realizing cross-device hardware resource sharing and improving the cross-device interactive experience.

[0171] See also Fig. 22 , Fig. 22 The flowchart of the cross-device application operation method provided by an embodiment of the present application is shown. The method is used to enable the second device to operate the application in the first device by creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, without developing an application adapted to the second device, which can reduce development costs. In a specific embodiment, the method is applied to Fig.24 The cross-device application operating apparatus 400 and the second device 200 ( Figure 1 ). Fig. 22The process shown is described in detail. In this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation method may specifically include the following steps:

[0172] Step S810: In response to a selection operation on the application list, a target application is determined from applications in the first device, and application information of the target application is acquired.

[0173] Step S820: Send the application information to the first device to instruct the first device to run the target application in the virtual device, create a virtual display associated with the target application in the virtual device, and collect audio and video information of the target application during operation through the virtual display.

[0174] Step S830: Receive the audio and video information sent by the first device, and play the audio and video information.

[0175] For the detailed description of step S810 to step S830, please refer to step S110 to step S160, which will not be repeated here.

[0176] As an implementable manner, when the second device plays the audio and video information of the target application, it can display a display interface corresponding to the audio and video information. In the process of displaying the display interface corresponding to the audio and video information, if a first touch operation acting on the display interface is detected, the interface content of the display interface can be enlarged or reduced, so as to realize the display interaction on the second device and enhance the user's interactive experience. Optionally, the first touch operation may include a two-finger pinch or two-finger expansion operation. For example, if a two-finger pinch operation acting on the display interface is detected, the interface content of the display interface can be reduced; if a two-finger expansion operation acting on the display interface is detected, the interface content of the display interface can be enlarged. Optionally, the first touch operation may include a quick double-click operation, that is, the interface content of the display interface can be partially enlarged by a quick double-click operation.

[0177] As an implementable manner, when the second device plays the audio and video information of the target application, it can display a display interface corresponding to the audio and video information. During the display of the display interface corresponding to the audio and video information, if a second touch operation acting on the display interface is detected, the interface content of the display interface can be moved to achieve display interaction on the second device and enhance the user's interactive experience. Optionally, the second touch operation may include a drag operation acting on the interface content.

[0178] As an implementable manner, when the second device plays the audio and video information of the target application, it can display a display interface corresponding to the audio and video information. During the process of displaying the display interface corresponding to the audio and video information, if a third touch operation acting on the display interface is detected, the display layout of the display interface can be switched. For example, the user can select a default display layout according to personal preference on the second device, and during use, the user can switch to another display layout through a third touch operation. Optionally, the third touch operation can include a two-finger pinch operation, a two-finger slide operation, etc., which are not limited here.

[0179] An embodiment of the present application provides a cross-device application operation method, which sets a connection between a first device and a second device, creates a virtual device associated with the second device in the first device, adds applications in the first device to the virtual device, and includes an application list corresponding to the applications obtained in the first device in the second device. In response to a selection operation on the application list, a target application is determined from the applications in the first device, and application information of the target application is obtained, and the application information is sent to the first device to instruct the first device to run the target application in the virtual device, and a virtual display associated with the target application is created in the virtual device, and audio and video information of the target application during operation is collected through the virtual display, and the audio and video information sent by the first device is received, and the audio and video information is played, so that the second device operates the application in the first device by creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, without the need to develop an application adapted to the second device, which can reduce development costs.

[0180] See also Fig.23 , Fig.23 The module block diagram of the cross-device application operation apparatus provided by an embodiment of the present application is shown. The cross-device application operation apparatus 300 is applied to the first device. Fig.23 The block diagram shown is used to illustrate that in this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation device 300 includes: a target application running module 310, an audio and video information collection module 320 and an audio and video information sending module 330, wherein:

[0181] The target application running module 310 is used to run the target application in the virtual device in response to the application information of the target application sent by the second device, wherein the target application is determined by the second device from the applications in the first device based on the application list.

[0182] Furthermore, the target application running module 310 includes: a screen resolution determination submodule and a first target application running submodule, wherein:

[0183] The screen resolution determination submodule is used to determine the screen resolution corresponding to the second device.

[0184] The first target application running submodule is configured to run the target application in the virtual device according to the screen resolution if the target application supports the screen resolution.

[0185] Furthermore, the target application running module 310 further includes: a reference resolution determination submodule and a second target application running submodule, wherein:

[0186] The reference resolution determination submodule is used to determine a plurality of reference resolutions supported by the target application if the target application does not support the screen resolution.

[0187] The second target application running submodule is used to determine a target resolution based on the multiple reference resolutions and the screen resolution, and run the target application in the virtual device according to the target resolution.

[0188] Furthermore, the second target application running submodule includes: a first target application running unit and a second target application running unit, wherein:

[0189] The first target application running unit is used to run the target application in the virtual device in a manner that keeps the width of the application interface of the target application consistent with the width of the screen resolution and cuts the length of the application interface to be consistent with the length of the screen resolution if the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution.

[0190] The second target application running unit is used to run the target application in the virtual device by proportionally reducing the application interface of the target application to a length consistent with the screen resolution if the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution.

[0191] Furthermore, the second target application running submodule includes: a third target application running unit and a fourth target application running unit, wherein:

[0192] The third target application running unit is used to run the target application in the virtual device in a manner that keeps the length of the application interface of the target application consistent with the length of the screen resolution and cuts the width of the application interface to be consistent with the width of the screen resolution if the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution.

[0193] The fourth target application running unit is used to run the target application in the virtual device by proportionally reducing the application interface of the target application to a width consistent with the screen resolution if the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution.

[0194] Furthermore, the target application running module 310 includes: a data transmission channel establishing submodule and a third target application running submodule, wherein:

[0195] The data transmission channel establishing submodule is used to pull up the target application in the virtual device in response to the application information of the target application sent by the second device, and establish a data transmission channel with the second device.

[0196] The third target application running submodule is used to run the target application in the virtual device.

[0197] The audio and video information collection module 320 is used to create a virtual display associated with the target application in the virtual device, and collect audio and video information of the target application during operation through the virtual display.

[0198] Furthermore, the audio and video information acquisition module 320 includes: a first virtual display creation submodule and a second virtual display creation submodule, wherein:

[0199] The first virtual display creation submodule is configured to create a virtual display associated with the target application in the virtual device based on the screen resolution if the target application supports the screen resolution.

[0200] The second virtual display creation submodule is configured to create a virtual display associated with the target application in the virtual device based on the target resolution if the target application does not support the screen resolution.

[0201] The audio and video information sending module 330 is used to send the audio and video information to the second device to instruct the second device to play the audio and video information.

[0202] Furthermore, the audio and video information sending module 330 includes: an audio and video information encoding submodule and an audio and video information sending submodule, wherein:

[0203] The audio and video information encoding submodule is used to encode the audio and video information to obtain the encoded audio and video information.

[0204] The audio and video information sending submodule is used to send the encoded audio and video information to the second device to instruct the second device to decode and play the encoded audio and video information.

[0205] Furthermore, the cross-device application operation apparatus 300 further includes: an operation instruction receiving module and an application operation module, wherein:

[0206] The operation instruction receiving module is used to receive the operation instruction sent by the second device, wherein the operation instruction is generated by the second device according to the operation acting on the second device during the process of playing the audio and video information.

[0207] The application operation module is used to operate the target application in the virtual device according to the operation instruction.

[0208] Further, when there are multiple second devices, and each of the multiple second devices sends an operation instruction to the first device, the application operation module includes:

[0209] The first application operation submodule is used to operate the target application in the virtual device according to the operation instructions respectively sent by the plurality of second devices.

[0210] Furthermore, the application operation module includes: an operation instruction conversion submodule and a second application operation submodule, wherein:

[0211] The operation instruction conversion submodule is used to process the operation instruction based on instruction mapping or coordinate conversion to obtain an operation instruction adapted to the first device.

[0212] The second application operation submodule is used to operate the target application in the virtual device according to an operation instruction adapted to the first device.

[0213] Furthermore, the cross-device application operation apparatus 300 further includes: a virtual device creation module and an application adding module, wherein:

[0214] A virtual device creation module is used to create a virtual device associated with the second device in the first device.

[0215] Furthermore, the virtual device creation module includes:

[0216] The parameter information acquisition submodule is used to acquire the parameter information of the second device.

[0217] The virtual device creation submodule is used to create a virtual device associated with the second device in the first device based on the parameter information.

[0218] The application adding module is used to add the application in the first device to the virtual device, and send the application list corresponding to the application in the first device to the second device.

[0219] Furthermore, the cross-device application operation apparatus 300 further includes: a request information sending module, wherein:

[0220] The request information sending module is used to send request information to the second device to request to call one or more hardware of the second device to collect and / or output data.

[0221] See also Fig.24 , Fig.24 The module block diagram of the cross-device application operation apparatus provided by an embodiment of the present application is shown. The cross-device application operation apparatus 400 is applied to the second device. Fig.24 The block diagram shown is used to illustrate that in this embodiment, the first device is connected to the second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device. The cross-device application operation device 400 includes: an application information acquisition module 410, an application information sending module 420 and an audio and video information playback module 430, wherein:

[0222] The application information acquisition module 410 is used to respond to a selection operation on the application list, determine a target application from the applications in the first device, and acquire application information of the target application.

[0223] The application information sending module 420 is used to send the application information to the first device to instruct the first device to run the target application in the virtual device, and to create a virtual display associated with the target application in the virtual device, and to collect audio and video information of the target application during operation through the virtual display.

[0224] The audio and video information playing module 430 is used to receive the audio and video information sent by the first device and play the audio and video information.

[0225] Furthermore, the cross-device application operation apparatus 400 further includes: a zoom module, a movement module and a switching module, wherein:

[0226] The zoom module is used to respond to a first touch operation on the display interface of the second device and to zoom in or out the interface content of the display interface.

[0227] The moving module is used to move the interface content of the display interface in response to the second touch operation applied to the display interface of the second device.

[0228] The switching module is used to switch the display layout of the display interface in response to a third touch operation applied to the display interface of the second device.

[0229] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0230] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0231] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0232] See also Fig.25 , which shows a structural block diagram of an electronic device 500 provided in an embodiment of the present application. The electronic device 500 may be an electronic device such as a smart phone, a tablet computer, an e-book, etc. that can run applications. The electronic device 500 may include the above-mentioned first device 100 or the above-mentioned second device 200, which is not limited here. The electronic device 500 in the present application may include one or more of the following components: a processor 510, a memory 520, and one or more applications, wherein one or more applications may be stored in the memory 520 and configured to be executed by one or more processors 510, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0233] Among them, the processor 510 may include one or more processing cores. The processor 510 uses various interfaces and lines to connect various parts within the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 510 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 510, but may be implemented separately through a communication chip.

[0234] The memory 520 may include a random access memory (RAM) or a read-only memory (ROM). The memory 520 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 500 during use.

[0235] See also Fig.26 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0236] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in an appropriate form.

[0237] In summary, the cross-device application operation method, apparatus, electronic device, and storage medium provided in the embodiments of the present application set a first device to be connected to a second device, a virtual device associated with the second device is created in the first device, the application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application obtained in the first device. The first device responds to the application information of the target application sent by the second device, runs the target application in the virtual device, creates a virtual display associated with the target application in the virtual device, and collects the audio and video information of the target application during operation through the virtual display, and sends the audio and video information to the second device to instruct the second device to play the audio and video information, thereby creating a virtual device associated with the second device in the first device, and creating a virtual display associated with the determined application in the virtual device, so that the second device can operate the application in the first device, without developing an application adapted to the second device, which can reduce development costs.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cross-device application operation method, characterized in that: Applied to a first device, the first device is connected to a second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device, the method comprising: In response to the application information of the target application sent by the second device, running the target application in the virtual device, wherein the target application is determined by the second device from applications in the first device based on the application list; Creating a virtual display associated with the target application in the virtual device, and collecting audio and video information of the target application during operation through the virtual display; The audio and video information is sent to the second device to instruct the second device to play the audio and video information.

2. The method according to claim 1, characterized in that The running of the target application in the virtual device includes: Determining a screen resolution corresponding to the second device; If the target application supports the screen resolution, the target application is run in the virtual device according to the screen resolution.

3. The method according to claim 2, characterized in that The method further comprises: If the target application does not support the screen resolution, determining a plurality of reference resolutions supported by the target application; A target resolution is determined based on the multiple reference resolutions and the screen resolution, and the target application is run in the virtual device according to the target resolution.

4. The method according to claim 3, characterized in that The running the target application in the virtual device according to the target resolution includes: If the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution, the target application is run in the virtual device in a manner that keeps the width of the application interface of the target application consistent with the width of the screen resolution and cuts the length of the application interface to be consistent with the length of the screen resolution; or If the aspect ratio of the target resolution is greater than the aspect ratio of the screen resolution, the target application is run in the virtual device by scaling down the application interface of the target application in proportion to a length consistent with the screen resolution.

5. The method according to claim 3, characterized in that: The running the target application in the virtual device according to the target resolution includes: If the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution, the target application is run in the virtual device in a manner that keeps the length of the application interface of the target application consistent with the length of the screen resolution and cuts the width of the application interface to be consistent with the width of the screen resolution; or If the aspect ratio of the target resolution is smaller than the aspect ratio of the screen resolution, the target application is run in the virtual device by scaling down the application interface of the target application in proportion to a width consistent with the screen resolution.

6. The method according to claim 3, characterized in that The step of creating a virtual display associated with the target application in the virtual device includes: If the target application supports the screen resolution, creating a virtual display associated with the target application in the virtual device based on the screen resolution; or If the target application does not support the screen resolution, a virtual display associated with the target application is created in the virtual device based on the target resolution.

7. The method according to claim 1, characterized in that The sending the audio and video information to the second device to instruct the second device to play the audio and video information includes: Encoding the audio and video information to obtain encoded audio and video information; The encoded audio and video information is sent to the second device to instruct the second device to decode and play the encoded audio and video information.

8. The method according to claim 1, characterized in that After sending the audio and video information to the second device to instruct the second device to play the audio and video information, the method further includes: receiving an operation instruction sent by the second device, wherein the operation instruction is generated by the second device according to an operation performed on the second device during the process of playing the audio and video information; According to the operation instruction, the target application is operated in the virtual device.

9. The method according to claim 8, characterized in that When there are multiple second devices, and each of the multiple second devices sends an operation instruction to the first device, operating the target application in the virtual device according to the operation instruction includes: The target application is operated in the virtual device according to the operation instructions respectively sent by the plurality of second devices.

10. The method according to claim 8, characterized in that According to the operation instruction, operating the target application in the virtual device includes: Processing the operation instruction based on instruction mapping or coordinate conversion to obtain an operation instruction adapted to the first device; The target application is operated in the virtual device according to the operation instruction adapted to the first device.

11. The method according to claim 1, characterized in that: The step of running the target application in the virtual device in response to the application information of the target application sent by the second device includes: In response to the application information of the target application sent by the second device, launching the target application in the virtual device and establishing a data transmission channel with the second device; The target application is run in the virtual device.

12. The method according to any one of claims 1 to 11, characterized in that: In response to the application information of the target application sent by the second device, before running the target application in the virtual device, the method further includes: Creating a virtual device associated with the second device in the first device; The applications in the first device are added to the virtual device, and an application list corresponding to the applications in the first device is sent to the second device.

13. The method according to claim 12, characterized in that The step of creating a virtual device associated with the second device in the first device includes: Acquire parameter information of the second device; Based on the parameter information, a virtual device associated with the second device is created in the first device.

14. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: Send a request message to the second device to request to call one or more hardware of the second device to collect and / or output data.

15. A cross-device application operation method, characterized in that: The method is applied to a second device, the second device is connected to a first device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device, and the method includes: In response to a selection operation on the application list, determining a target application from applications in the first device, and acquiring application information of the target application; Sending the application information to the first device to instruct the first device to run the target application in the virtual device, and to create a virtual display associated with the target application in the virtual device, and to collect audio and video information of the target application during operation through the virtual display; Receive the audio and video information sent by the first device, and play the audio and video information.

16. The method according to claim 15, characterized in that After receiving the audio and video information sent by the first device and playing the audio and video information, the method further includes: In response to a first touch operation on a display interface of the second device, zooming in or out on interface content of the display interface; or In response to a second touch operation on the display interface of the second device, moving the interface content of the display interface; or In response to a third touch operation applied to the display interface of the second device, a display layout of the display interface is switched.

17. A cross-device application operation device, characterized in that: The apparatus is applied to a first device, the first device is connected to a second device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device, and the apparatus includes: a target application running module, configured to run the target application in the virtual device in response to application information of the target application sent by the second device, wherein the target application is determined by the second device from applications in the first device based on the application list; An audio and video information acquisition module, used to create a virtual display associated with the target application in the virtual device, and to acquire audio and video information of the target application during operation through the virtual display; The audio and video information sending module is used to send the audio and video information to the second device to instruct the second device to play the audio and video information.

18. A cross-device application operation device, characterized in that: The device is applied to a second device, the second device is connected to a first device, a virtual device associated with the second device is created in the first device, an application in the first device is added to the virtual device, and the second device includes an application list corresponding to the application in the first device, the device includes: an application information acquisition module, configured to respond to a selection operation on the application list, determine a target application from applications in the first device, and acquire application information of the target application; an application information sending module, configured to send the application information to the first device to instruct the first device to run the target application in the virtual device, create a virtual display associated with the target application in the virtual device, and collect audio and video information of the target application during operation through the virtual display; The audio and video information playing module is used to receive the audio and video information sent by the first device and play the audio and video information.

19. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor performs the method according to any one of claims 1 to 14 or any one of claims 15 to 16.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 14 or any one of claims 15 to 16.

Images