Control method and device, head-mounted display device, and medium

Abstract

The method comprises: in a running process of a desktop environment, receiving a first starting instruction for starting a target application; in response to the first starting instruction, obtaining a first hash table and target package name information of the target application; wherein a key of the first hash table is package name information of any application installed in a head-mounted display device and a terminal device, and a hash value corresponding to the key is used to represent an installed device of the any application; obtaining a target hash value corresponding to the target package name information according to the first hash table and the target package name information; and determining to start the target application in the head-mounted display device or the terminal device according to the target hash value.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of wearable devices, and more particularly, to a control method, a control device, a head-mounted display device, and a computer-readable storage medium. BACKGROUND

[0002] With the continuous development of augmented reality technology, many AR products and AR applications have emerged, and multiple AR applications can be opened in an AR product.

[0003] However, most of the AR products currently used are lightweight all-in-one machines, which have small batteries and relatively weak computing capabilities. When too many AR applications are opened, there will be problems such as lag, crash, and serious heating, which affect the user experience. SUMMARY

[0004] Embodiments of the present disclosure aim to provide a control method, device, head-mounted display device, and medium.

[0005] According to a first aspect of embodiments of the present disclosure, a control method is provided, which comprises:

[0006] receiving a first starting instruction for starting a target application during the running of a desktop environment;

[0007] In response to the first starting instruction, obtaining a first hash table and target package name information of the target application; wherein the key of the first hash table is the package name information of any application installed in the head-mounted display device and the terminal device, and the hash value corresponding to the key is used to represent the installed device of the any application;

[0008] According to the first hash table and the target package name information, obtaining a target hash value corresponding to the target package name information;

[0009] According to the target hash value, determining to start the target application in the head-mounted display device or the terminal device.

[0010] Optionally, the method further comprises the step of establishing the first hash table,

[0011] The establishment of the first hash table comprises:

[0012] Obtaining the package name information of any application installed in the head-mounted display device and the terminal device;

[0013] In the case where the package name information is derived from the head-mounted display device, taking the package name information as the key, and the hash value corresponding to the key is a first numerical value used to represent that the installed device of the any application is the head-mounted display device, to establish a first hash data;

[0014] In a case where the package name information is derived from the terminal device, second hash data is established with the package name information as a keyword, and a hash value corresponding to the keyword being a second value for representing that the device installed with the arbitrary application is the terminal device;

[0015] In a case where the package name information is derived from the terminal device and the head-mounted display device, third hash data is established with the package name information as a keyword, and a hash value corresponding to the keyword being a third value for representing that the device installed with the arbitrary application is the head-mounted display device and the terminal device;

[0016] The first hash table is established based on the first hash data, the second hash data, and the third hash data.

[0017] Optionally, the determining, according to the target hash value, that the target application is started in the head-mounted display device or the terminal device includes:

[0018] In a case where the target hash value represents that the device installed with the target application is the head-mounted display device, a first virtual screen and a first canvas are created in the desktop environment;

[0019] The target application is started in the first virtual screen, and texture information of the first virtual screen is rendered and displayed to the first canvas.

[0020] Optionally, the determining, according to the target hash value, that the target application is started in the head-mounted display device or the terminal device further includes:

[0021] In a case where the target hash value represents that the device installed with the target application is the head-mounted display device and the terminal device, a first occupation rate of a resource of the head-mounted display device is acquired;

[0022] The first occupation rate of the resource and the target package name information are sent to the terminal device;

[0023] In a case where the terminal device determines that the first occupation rate of the resource and a second occupation rate of the resource satisfy a first preset condition, a second virtual screen is created in the terminal device, and the target application is started in the second virtual screen based on the target package name information;

[0024] Image data of the target application transmitted by the terminal device is received, and the image data of the target application is rendered and displayed to a second canvas in the desktop environment.

[0025] Optionally, the determining, according to the target hash value, that the target application is started in the head-mounted display device or the terminal device further includes:

[0026] determining, by the terminal device, that the first occupation ratio of the resource and the second occupation ratio of the resource satisfy a second preset condition, and sending, by the terminal device, first notification information to the head-mounted display device; wherein the first notification information is used to instruct starting the target application in the head-mounted display device;

[0027] receiving the first notification information, and creating a first virtual screen and a first canvas in the desktop environment;

[0028] starting the target application in the first virtual screen, and rendering and displaying texture information of the first virtual screen to the first canvas.

[0029] Optionally, the determining, according to the target hash value, that the target application is started in the head-mounted display device or the terminal device comprises:

[0030] in a case where the target hash value is used to represent that the target application is installed in the terminal device, sending the target package name information to the terminal device;

[0031] starting, by the terminal device, a second virtual screen, and starting the target application in the second virtual screen based on the target package name information;

[0032] receiving image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application in a second canvas in the desktop environment.

[0033] Optionally, the method further comprises:

[0034] obtaining a third occupation ratio of a resource of the head-mounted display device;

[0035] in a case where the third occupation ratio of the resource exceeds a first occupation ratio threshold, outputting first prompt information;

[0036] wherein the first prompt information is used to prompt that the number of applications started by the head-mounted display device exceeds a set number.

[0037] Optionally, after receiving the image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application in the second canvas in the desktop environment, the method further comprises:

[0038] displaying a switching control on the second canvas;

[0039] receiving first input of the switching control;

[0040] in response to the first input, controlling the terminal device to close the second virtual screen, and sending a second notification message to the head-mounted display device through the terminal device; wherein the second notification message is used to instruct the head-mounted display device to start the target application;

[0041] receiving the second notification message, creating a first virtual screen and a first canvas in the desktop environment;

[0042] starting the target application in the first virtual screen, and rendering and displaying texture information of the first virtual screen to the first canvas.

[0043] According to a second aspect of the embodiments of the present disclosure, a control device is provided, and the device comprises:

[0044] a receiving module, configured to receive a first starting instruction of starting a target application during running of a desktop environment;

[0045] a first obtaining module, configured to obtain a first hash table and target package name information of the target application in response to the first starting instruction; wherein a key of the first hash table is package name information of any application installed in a head-mounted display device and a terminal device, and a hash value corresponding to the key is used to represent an installed device of the any application;

[0046] a second obtaining module, configured to obtain a target hash value corresponding to the target package name information according to the first hash table and the target package name information;

[0047] a determining module, configured to determine to start the target application in the head-mounted display device or the terminal device according to the target hash value.

[0048] According to a third aspect of the embodiments of the present disclosure, a head-mounted display device is provided, and the head-mounted display device comprises:

[0049] a memory, configured to store executable computer instructions;

[0050] a processor, configured to execute the control method according to the first aspect above according to control of the executable computer instructions.

[0051] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, and the computer instructions are run by a processor to execute the control method according to the first aspect above.

[0052] One beneficial effect of the embodiment of the present disclosure is that, in the running process of the desktop environment of the head-mounted display device, if a first application is selected to be opened in the head-mounted display device, the head-mounted display device obtains a target hash value corresponding to the target package name information according to the target package name information of the first application and a first hash table recorded by the head-mounted display device, the first hash table is established according to the package name information of any application installed in the head-mounted display device and the terminal device, and the value used to represent the installation device of the any application is a hash value, and then the target application is determined to be started in the head-mounted display device or the terminal device according to the target hash value, so that in the application multi-opening scene, a part of the applications can be opened in the terminal device, and a part of the applications can be opened in the head-mounted display device, and then the computing capabilities of the head-mounted display device and the terminal device are reasonably allocated, and a more smooth application multi-opening scene is realized.

[0053] Other features of the present specification and its advantages will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present specification and, together with the description, serve to explain the principles of the present specification.

[0055] Figure 1 is a hardware configuration schematic diagram of a head-mounted display device according to an embodiment of the present disclosure;

[0056] Figure 2 is a flowchart of a control method according to an embodiment of the present disclosure;

[0057] Figure 3 is a principle block diagram of a control device according to an embodiment of the present disclosure;

[0058] Figure 4 is a principle block diagram of a head-mounted display device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the embodiments of the present disclosure unless otherwise specifically stated.

[0060] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the present disclosure and its applications or uses.

[0061] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0062] In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0063] It should be noted that like reference numerals and letters refer to like items throughout the drawings, and once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0064] <Hardware Configuration>

[0065] Figure 1 is a block diagram of a hardware configuration of a head-mounted display device 1000 according to an embodiment of the present disclosure.

[0066] As shown in Figure 1 , the head-mounted display device 1000 can be smart glasses, which can be AR glasses, and of course can also be other devices, which are not limited by the embodiments of the present disclosure.

[0067] In one embodiment, as shown in Figure 1 , the head-mounted display device 1000 can include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like.

[0068] The processor 1100 can include, but is not limited to, a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, and the like. The communication device 1400 can perform wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display screen, an LED display screen, an OLED (Organic Light-Emitting Diode) display screen, and the like. The input device 1600 includes, for example, a touch screen, a keyboard, a joystick, and the like. The head-mounted display device 1000 can output audio information through the speaker 1700, and can collect audio information through the microphone 1800.

[0069] Those skilled in the art will understand that although a plurality of devices of the head-mounted display device 1000 are shown in Figure 1 , the head-mounted display device 1000 of the embodiments of the present disclosure can only involve part of the devices, and can also include other devices, which are not limited herein.

[0070] In this embodiment, the memory 1200 of the head-mounted display device 1000 is configured to store instructions for controlling the processor 1100 to operate to implement or support implementation of the control method according to any embodiment. The skilled person can design the instructions according to the solutions disclosed in the specification. How the instructions control the processor to operate is known in the art, and thus will not be described in detail herein.

[0071] In the above description, the skilled person can design the instructions according to the solutions provided in the disclosure. How the instructions control the processor to operate is known in the art, and thus will not be described in detail herein.

[0072] Figure 1 The head-mounted display device shown is merely illustrative and is by no means intended to limit the disclosure, its application or use.

[0073] In the following, various embodiments and examples according to the disclosure are described with reference to the accompanying drawings.

[0074] <Method Embodiment>

[0075] Figure 2 A control method of one embodiment of the disclosure is shown, which can be implemented by a head-mounted display device, or can be implemented by a control device independent of the head-mounted display device and the head-mounted display device together, or can be implemented by a cloud server, a terminal device and the head-mounted display device together, wherein the head-mounted display device can be AR glasses, and the terminal device can be a mobile phone.

[0076] As shown, the control method of this embodiment can include the following steps S2100-S2400: Figure 2

[0077] Step S2100, during the running of the desktop environment, a first start instruction for starting a target application is received.

[0078] In this embodiment, the desktop launcher of the head-mounted display device is called AR Launcher, which is automatically started when the head-mounted display device is powered on, and the desktop environment of the head-mounted display device starts to run. After the user wears the head-mounted display device, the head-mounted display device displays the desktop environment in the user's line of sight. It can be understood that the desktop environment is usually a 3D desktop environment.

[0079] Optionally, the first start instruction can be a touch input to the icon of the target application.

[0080] Optionally, the first start instruction can also be a ray event sent by an interactive device to the icon of the target application, and the interactive device can be a handle, a mouse, a mobile phone or the like.

[0081] ​Optionally, the first starting instruction can also be a gesture event of a user (i.e., a wearer) on an icon of the target application.

[0082] In one specific embodiment, during the running of the desktop environment, on one hand, the desktop environment of the head-mounted display device can display icons of applications installed in the head-mounted display device, and on the other hand, the desktop environment of the head-mounted display device can also display icons of applications transmitted by the terminal device through wireless streaming, where the user can click on the icon of the target application displayed in the desktop environment of the head-mounted display device to start the target application.

[0083] In step S2200, in response to the first starting instruction, a first hash table and target package name information of the target application are obtained.

[0084] The key of the first hash table is package name information of any application installed in the head-mounted display device and the terminal device, and the hash value corresponding to the key is used to represent the installation device of the any application. Based on this, the control method of the embodiment of the present disclosure further includes the step of establishing a first hash table, which can specifically include steps S3100-S3500 as follows:

[0085] In step S3100, package name information of any application installed in the head-mounted display device and the terminal device is obtained.

[0086] In this step S3100, the terminal device and the head-mounted display device establish a P2P wireless communication connection, the terminal device can obtain the package name information of the application installed locally according to the label "android.intent.category.LAUNCHER", and trigger a callback function to send the obtained package name information of the application installed locally to the head-mounted display device. At the same time, the head-mounted display device can also obtain the package name information of the application installed locally according to the label "android.intent.category.LAUNCHER".

[0087] In step S3200, in the case where the package name information is from the head-mounted display device, a first hash data is established with the package name information as the key and a first value corresponding to the key, which is used to represent that the installation device of the any application is the head-mounted display device.

[0088] In step S3300, in the case where the package name information is from the terminal device, a second hash data is established with the package name information as the key and a second value corresponding to the key, which is used to represent that the installation device of the any application is the terminal device.

[0089] Step S3400, in a case where the package name information is derived from the terminal device and the head-mounted display device, establishing third hash data with the package name information as a key, and a hash value corresponding to the key being a third numerical value representing that the device installed with the arbitrary application is the head-mounted display device and the terminal device.

[0090] Step S3500, establishing the first hash table based on the first hash data, the second hash data, and the third hash data.

[0091] Next, the steps S3100-S3500 will be described in detail. The head-mounted display device obtains the package name information of the application installed locally, and the package name information of the application installed locally in the terminal device transmitted by the terminal device. The package name information is used as a key of the first hash table. If the package name information is unique to the head-mounted display device, a hash value corresponding to the key is a first numerical value, for example, 0. If the package name information is common to the head-mounted display device and the terminal device, the hash value corresponding to the key is a second numerical value, for example, 1. If the package name information is unique to the terminal device, the hash value corresponding to the key is a third numerical value, for example, 2.

[0092] In one specific embodiment, after the head-mounted display device receives a first start instruction for starting a target application, the head-mounted display device can respond to the first start instruction. On one hand, the head-mounted display device obtains target package name information of the target application. On the other hand, the head-mounted display device also obtains the first hash table.

[0093] Step S2300, obtaining a target hash value corresponding to the target package name information according to the first hash table and the target package name information.

[0094] In this embodiment, after the head-mounted display device obtains the first hash table and the package name information of the target application, the head-mounted display device queries the first hash table with the package name information of the target application as a key, and obtains a hash value corresponding to the package name information of the target application as a target hash value.

[0095] Step S2400, determining to start the target application in the head-mounted display device or the terminal device according to the target hash value.

[0096] Optionally, the step S2400 of determining to start the target application in the head-mounted display device or the terminal device according to the target hash value can further include the following steps S3410a-S3420a.

[0097] Step S3410a, in the case that the target hash value is used to represent that the device on which the target application is installed is the head-mounted display device, a first virtual screen and a first canvas are created in the desktop environment.

[0098] In this step S3410a, when the hash value corresponding to the package name information of the target application is 0, it indicates that the package name information of the target application is unique to the head-mounted display device, i.e., the target application is installed only on the head-mounted display device. At this time, the target application needs to be started on the head-mounted display device. Here, the head-mounted display device creates a first virtual screen and a first canvas in the desktop environment.

[0099] Step S3420a, the target application is started in the first virtual screen, and the texture information of the first virtual screen is rendered and displayed to the first canvas.

[0100] In this step S3420a, after the head-mounted display device creates the first virtual screen and the first canvas, the target application can be started in the first virtual screen, and the texture information of the first virtual screen is rendered and displayed to the first canvas.

[0101] Optionally, the step S2400 of determining to start the target application in the head-mounted display device or the terminal device according to the target hash value can further include steps S3410b-S3440b.

[0102] Step S3410b, in the case that the target hash value is used to represent that the device on which the target application is installed is the head-mounted display device and the terminal device, a first occupation rate of resources of the head-mounted display device is obtained.

[0103] The resources can be at least one of CPU, memory usage, graphics processing unit (GPU), and FPGA. It can be understood that the occupation rate of the resources can be determined according to the resources required when the target application is running. For example, the resources required when the target application is running are mainly CPU, and the head-mounted display device can obtain the occupation rate of the CPU.

[0104] It can be understood that in the case that the resources are multiple, the first occupation rate of the resources can be the sum of the occupation rates of the multiple resources.

[0105] In this step S3410b, when the hash value corresponding to the package name information of the target application is 1, it indicates that the package name information of the target application is common to the head-mounted display device and the terminal device, i.e., the target application is installed not only on the head-mounted display device but also on the terminal device. At this time, the head-mounted display device can obtain, for example, the first occupation rate of its CPU.

[0106] Step S3420b, sending the first occupancy of the resource and the target package name information to the terminal device.

[0107] In this step S3420b, after the head-mounted display device obtains the first occupancy of the CPU thereof, for example, the head-mounted display device can send the first occupancy of the CPU and the target package name information of the target application to the terminal device.

[0108] Step S3430b, when the terminal device determines that the first occupancy of the resource and the second occupancy of the resource satisfy a first preset condition, creating a second virtual screen in the terminal device, and starting the target application in the second virtual screen based on the target package name information.

[0109] The first preset condition can be that the second occupancy of the resource is less than or equal to a second occupancy threshold, or that the second occupancy of the resource and the first occupancy of the resource are both greater than the second occupancy threshold. The second occupancy threshold can be designed according to the scene.

[0110] In this step S3430b, when the terminal device determines that the second occupancy of the CPU thereof is less than or equal to a second occupancy threshold (for example, less than 50%), or that the second occupancy of the CPU of the terminal device and the first occupancy of the CPU of the head-mounted display device are both greater than the second occupancy threshold (for example, greater than 50%), it is determined that the target application is started in the terminal device. Here, the terminal device can create a second virtual screen, and start the target application in the second virtual screen based on the target package name information of the target application.

[0111] Step S3440b, receiving the image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application in a second canvas in the desktop environment.

[0112] In this step S3440b, after the second virtual screen of the terminal device starts the target application, the terminal device transmits the image data of the target application to the head-mounted display device in a wireless streaming manner, the head-mounted display device creates a second canvas, and renders and displays the image data in the second canvas.

[0113] According to the above steps S3410b to S3440b, if the target application is installed in both the terminal device and the head-mounted display device, and the resources of the terminal device are sufficient to run the target application, the target application can be run in the terminal device.

[0114] Optionally, the step S2400 of determining to start the target application in the head-mounted display device or the terminal device according to the target hash value can further include the following steps S3450b to S3470b:

[0115] Step S3450b, when the terminal device determines that the first occupation rate of the resource and the second occupation rate of the resource satisfy a second preset condition, the terminal device sends first notification information to the head-mounted display device.

[0116] The first notification information is used to instruct to start the target application in the head-mounted display device.

[0117] The second preset condition can be that the second occupation rate of the resource is greater than a second occupation rate threshold, and the first occupation rate of the resource is less than or equal to the second occupation rate threshold. The second preset condition can be specifically opposite to the first preset condition.

[0118] In this step S3450b, if the terminal device determines that the second occupation rate of the CPU of the terminal device is greater than a second occupation rate threshold (for example, greater than 50%), and the first occupation rate of the CPU transmitted by the head-mounted display device is less than or equal to the second occupation rate threshold (for example, less than 50%), it is determined that the target application is started in the head-mounted display device. Here, the terminal device needs to send the first notification information to the head-mounted display device, which is used to instruct the head-mounted display device to start the target application.

[0119] Step S3460b, receiving the first notification information, creating a first virtual screen and a first canvas in the desktop environment.

[0120] In this step S3460b, after the head-mounted display device receives the first notification information, a first virtual screen and a first canvas are created in the running desktop environment.

[0121] Step S3470b, starting the target application in the first virtual screen, and rendering and displaying the texture information of the first virtual screen to the first canvas.

[0122] In this step S3470b, after the head-mounted display device creates a first virtual screen and a first canvas, the target application can be started in the first virtual screen, and the texture information of the first virtual screen is rendered and displayed to the first canvas.

[0123] According to the above steps S3450b-S3470b, if the target application is installed in the terminal device and the head-mounted display device at the same time, and the resource of the head-mounted display device is sufficient to run the target application, the target application can be run in the head-mounted display device.

[0124] Optionally, the step S2400 of determining to start the target application in the head-mounted display device or the terminal device according to the target hash value can further include the following steps S3410c-S3430c:

[0125] Step S3410c, in the case that the target hash value is used to represent that the target application is installed in the terminal device, the target package name information is sent to the terminal device.

[0126] In this step S3410c, when the hash value corresponding to the package name information of the target application is 2, it indicates that the package name information of the target application is unique to the terminal device, i.e., the target application is only installed in the terminal device. At this time, the target application needs to be started in the terminal device. Here, the head-mounted display device sends the target package name information of the target application to the terminal device.

[0127] Step S3420c, the terminal device opens a second virtual screen, and starts the target application in the second virtual screen based on the target package name information.

[0128] In this step S3420c, the terminal device can create a second virtual screen, and start the target application in the second virtual screen based on the target package name information of the target application.

[0129] Step S3430c, receiving the image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application in the second canvas in the desktop environment.

[0130] In this step S3430c, after the second virtual screen of the terminal device starts the target application, the terminal device transmits the image data of the target application to the head-mounted display device in a wireless streaming manner. The head-mounted display device creates a second canvas, and renders and displays the image data in the second canvas.

[0131] According to the embodiments of the present disclosure, during the running of the desktop environment of the head-mounted display device, if a first application is selected to be opened in the head-mounted display device, the head-mounted display device can obtain a target hash value corresponding to the target package name information of the first application according to the target package name information of the first application and a first hash table recorded by the head-mounted display device, the first hash table taking the package name information of any application installed in the head-mounted display device and the terminal device as a key and taking a value used to represent the device on which the any application is installed as a hash value. Then, the head-mounted display device determines whether to start the target application in the head-mounted display device or in the terminal device according to the target hash value. In this way, in the application multi-opening scenario, a part of the applications can be opened in the terminal device, and a part of the applications can be opened in the head-mounted display device. Thus, the computing capability of the head-mounted display device and the terminal device can be reasonably allocated, and a more smooth application multi-opening scenario can be realized.

[0132] In one embodiment, the control method of the embodiments of the present disclosure further includes steps S4100-S4200:

[0133] Step S4100, obtaining a third occupation rate of the resources of the head-mounted display device.

[0134] In step S4100, the head-mounted display device detects the resource occupancy rate in real time or at a set time interval. For example, the head-mounted display device detects the CPU occupancy rate in real time.

[0135] In step S4200, the first prompt information is output when the third occupancy rate of the resource exceeds the first occupancy rate threshold.

[0136] The first prompt information is used to prompt that the number of applications launched by the head-mounted display device exceeds a set number.

[0137] The first occupancy rate threshold can be designed according to a scene, and the first occupancy rate threshold can be 80%.

[0138] For example, when the head-mounted display device detects that the CPU occupancy rate exceeds the first occupancy rate threshold (for example, exceeds 80%) at any time, the head-mounted display device outputs the prompt information “The load of the head-mounted display device is too high. Please clean up the applications in time” to prompt that the number of applications launched by the head-mounted display device exceeds a set number. At this time, all applications launched on the head-mounted display device side cannot be opened.

[0139] It can be understood that the terminal device can also detect the CPU occupancy rate in real time or at a set time interval. If the CPU occupancy rate exceeds the first occupancy rate threshold (for example, exceeds 80%), a callback function is triggered to send a first notification message to the head-mounted display device. At this time, the head-mounted display device opens the application launched on the terminal device side and having a hash value of 1 in the first hash table on the head-mounted display device. When the user selects to open the application having a hash value of 2 in the first hash table, the terminal device outputs the prompt information “The load of the terminal device is too high. Please close the streaming application and then launch the application”.

[0140] In one embodiment, the control method of the present disclosure further includes steps S5100-S5500.

[0141] In step S5100, a switching control is displayed on the second canvas.

[0142] The switching control is used for the user to switch the launch mode of the target application. For example, after the second virtual screen of the terminal device launches the target application, the terminal device transmits image data of the target application to the second canvas of the head-mounted display device for rendering and display in a wireless streaming manner. At this time, the switching control is displayed on the second canvas of the head-mounted display device. The user can switch the launch mode of the target application from the terminal device to the head-mounted display device through the switching control.

[0143] In step S5200, a first input to the switching control is received.

[0144] Optionally, the first input can be a touch input for the switching control.

[0145] Optionally, the first input can also be a ray event sent by an interactive device for the switching control, the interactive device can be a handle, a mouse, a mobile phone or the like.

[0146] Optionally, the first input can also be a gesture event of a user, i.e., a wearer, for the switching control.

[0147] Step S5300, in response to the first input, controlling the terminal device to close the second virtual screen and sending a second notification message through the head-mounted display device; wherein the second notification message is used to instruct the head-mounted display device to start the target application.

[0148] In this embodiment, after the head-mounted display device receives the first input of the user for the switching control, it can respond to the first input and send a message to the terminal device, and the terminal device can close the second virtual screen, stop running the target application, release resources, and then the terminal device will send a second notification message for instructing the head-mounted display device to start the target application.

[0149] Step S5400, receiving the second notification message, and creating a first virtual screen and a first canvas in the desktop environment.

[0150] In this embodiment, after the head-mounted display device receives the second notification message, it can create a first virtual screen and a first canvas in the running desktop environment.

[0151] Step S5500, starting the target application in the first virtual screen, and rendering and displaying the texture information of the first virtual screen to the first canvas.

[0152] In this embodiment, after the head-mounted display device creates a first virtual screen and a first canvas, it can start the target application in the first virtual screen, and render and display the texture information of the first virtual screen to the first canvas.

[0153] According to the above steps S5100-S5500, the head-mounted display device side provides a switching control for the user to switch the starting mode of the target application, and the user can manually switch the starting mode of the target application through the switching control.

[0154] <Device Embodiment>

[0155] Figure 3 is a schematic diagram of the principle of a control device according to an embodiment, referring to Figure 3 As shown, the control device 300 includes a receiving module 310, a first obtaining module 320, a second obtaining module 330, and a determining module 340.

[0156] The receiving module 310 is configured to receive a first starting instruction for starting a target application during running of a desktop environment.

[0157] The first obtaining module 320 is configured to obtain a first hash table and target package name information of the target application in response to the first starting instruction, wherein a key of the first hash table is package name information of any application installed in a head-mounted display device and a terminal device, and a hash value corresponding to the key is used to represent an installation device of the any application.

[0158] The second obtaining module 330 is configured to obtain a target hash value corresponding to the target package name information according to the first hash table and the target package name information.

[0159] The determining module 340 is configured to determine to start the target application in the head-mounted display device or the terminal device according to the target hash value.

[0160] In an embodiment, the apparatus 300 further includes a establishing module (not shown in the figure) configured to: obtain package name information of any application installed in the head-mounted display device and the terminal device; in a case where the package name information is from the head-mounted display device, establish first hash data by taking the package name information as a key and taking a hash value corresponding to the key as a first numerical value used to represent that an installation device of the any application is the head-mounted display device; in a case where the package name information is from the terminal device, establish second hash data by taking the package name information as a key and taking a hash value corresponding to the key as a second numerical value used to represent that the installation device of the any application is the terminal device; in a case where the package name information is from the terminal device and the head-mounted display device, establish third hash data by taking the package name information as a key and taking a hash value corresponding to the key as a third numerical value used to represent that the installation device of the any application is the head-mounted display device and the terminal device; and establish the first hash table based on the first hash data, the second hash data and the third hash data.

[0161] In an embodiment, the determining module 340 is configured to create a first virtual screen and a first canvas in the desktop environment in a case where the target hash value is used to represent that the installation device of the target application is the head-mounted display device; start the target application in the first virtual screen, and render and display texture information of the first virtual screen to the first canvas.

[0162] In an embodiment, the determining module 340 is configured to, in a case where the target hash value is used to represent that the target application is installed on the head-mounted display device and the terminal device, acquire a first occupation rate of a resource of the head-mounted display device; send the first occupation rate of the resource and the target package name information to the terminal device; in a case where the terminal device determines that the first occupation rate of the resource and / or a second occupation rate of the resource satisfy a first preset condition, create a second virtual screen in the terminal device, and start the target application in the second virtual screen based on the target package name information; receive image data of the target application transmitted by the terminal device, and render and display the image data of the target application in a second canvas in the desktop environment.

[0163] In an embodiment, the determining module 340 is further configured to, in a case where the terminal device determines that the first occupation rate of the resource and the second occupation rate of the resource satisfy a second preset condition, send first notification information to the head-mounted display device through the terminal device; the first notification information is used to instruct to start the target application in the head-mounted display device; receive the first notification information, create a first virtual screen and a first canvas in the desktop environment; start the target application in the first virtual screen, and render and display texture information of the first virtual screen to the first canvas.

[0164] In an embodiment, the determining module 340 is configured to, in a case where the target hash value is used to represent that the target application is installed on the terminal device, send the target package name information to the terminal device; start the target application in a second virtual screen of the terminal device based on the target package name information; receive image data of the target application transmitted by the terminal device, and render and display the image data of the target application in a second canvas in the desktop environment.

[0165] In an embodiment, the apparatus 300 further includes a third acquisition module and an output module (none of which is shown in the figure).

[0166] The third acquisition module is configured to acquire a third occupation rate of a resource of the head-mounted display device.

[0167] The output module is configured to, in a case where the third occupation rate of the resource exceeds a first occupation rate threshold, output first prompt information.

[0168] The first prompt information is used to prompt that a number of applications started by the head-mounted display device exceeds a set number.

[0169] In an embodiment, the apparatus 300 further includes a display module and a sending module (none of which is shown in the figure).

[0170] a display module, configured to display a switching control on the second canvas;

[0171] a receiving module 310, configured to receive a first input to the switching control;

[0172] a sending module, configured to, in response to the first input, control the terminal device to close the second virtual screen, and send a second notification message to the head-mounted display device through the terminal device; wherein the second notification message is used to instruct the head-mounted display device to start the target application;

[0173] the receiving module 310, configured to receive the second notification message, and create a first virtual screen and a first canvas in the desktop environment;

[0174] a display module, configured to start the target application in the first virtual screen, and render and display texture information of the first virtual screen to the first canvas.

[0175] According to the embodiment, during the running of the desktop environment of the head-mounted display device, if a first application is selected to be opened in the head-mounted display device, the head-mounted display device can obtain a target hash value corresponding to target package name information of the first application according to the target package name information and a first hash table recorded by the head-mounted display device, the first hash table taking the package name information of any application installed in the head-mounted display device and the terminal device as a key and taking a hash value representing the installation device of the any application as a value, and then determine to start the target application in the head-mounted display device or the terminal device according to the target hash value. In this way, in the application multi-opening scenario, a part of applications can be opened in the terminal device and a part of applications can be opened in the head-mounted display device, and the computing capacity of the head-mounted display device and the terminal device can be reasonably allocated to realize a more smooth application multi-opening scenario.

[0176] <Device Embodiment>

[0177] Figure 4 is a hardware structure schematic diagram of a head-mounted display device according to an embodiment. As shown in Figure 4 , the head-mounted display device 400 includes a processor 410 and a memory 420.

[0178] The memory 420 can be used to store executable computer instructions.

[0179] The processor 410 can be used to control the execution of the control method according to the embodiments of the method of the present disclosure according to the executable computer instructions.

[0180] The head-mounted display device 400 can be a head-mounted display device 1000 as shown in Figure 1 , or can be a device with other hardware structures, which is not limited here.

[0181] In another embodiment, the head-mounted display device 400 can include the above control device 300.

[0182] In one embodiment, the modules of the above control device 300 can be implemented by the processor 410 executing computer instructions stored in the memory 420.

[0183] <Computer readable storage medium>

[0184] The embodiments of the present disclosure further provide a computer readable storage medium, having stored thereon computer instructions, which when executed by a processor, perform the control method provided by the embodiments of the present disclosure.

[0185] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0186] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical or other optical medium, and / or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0187] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0188] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0189] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0190] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0192] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0193] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of the terms to be used in the description is not intended to limit the scope of the embodiments described herein, but rather to best describe the principles of the embodiments in the context of the specific application or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A control method characterized by, The method comprises: During running of a desktop environment, a first starting instruction for starting a target application is received; In response to the first starting instruction, a first hash table and target package name information of the target application are acquired; wherein a key of the first hash table is package name information of any application installed in a head-mounted display device and a terminal device, and a hash value corresponding to the key is used to represent an installation device of the any application; According to the first hash table and the target package name information, a target hash value corresponding to the target package name information is obtained; According to the target hash value, it is determined to start the target application in the head-mounted display device or the terminal device.

2. The method of claim 1, wherein, The method further comprises a step of establishing the first hash table, The step of establishing the first hash table comprises: Acquiring package name information of any application installed in the head-mounted display device and the terminal device; In a case where the package name information is from the head-mounted display device, first hash data is established by taking the package name information as a key and a hash value corresponding to the key as a first numerical value used to represent that an installation device of the any application is the head-mounted display device; In a case where the package name information is from the terminal device, second hash data is established by taking the package name information as a key and a hash value corresponding to the key as a second numerical value used to represent that the installation device of the any application is the terminal device; In a case where the package name information is from the terminal device and the head-mounted display device, third hash data is established by taking the package name information as a key and a hash value corresponding to the key as a third numerical value used to represent that the installation device of the any application is the head-mounted display device and the terminal device; The first hash table is established based on the first hash data, the second hash data and the third hash data.

3. The method of claim 1, wherein, The step of determining, according to the target hash value, to start the target application in the head-mounted display device or the terminal device comprises: In a case where the target hash value is used to represent that an installation device of the target application is the head-mounted display device, a first virtual screen and a first canvas are created in the desktop environment; The target application is started in the first virtual screen, and texture information of the first virtual screen is rendered and displayed to the first canvas.

4. The method of claim 1, wherein, The step of determining, according to the target hash value, to start the target application in the head-mounted display device or the terminal device further comprises: In a case where the target hash value is used to represent that the installation device of the target application is the head-mounted display device and the terminal device, a first occupancy rate of a resource of the head-mounted display device is acquired; The first occupancy rate of the resource and the target package name information are sent to the terminal device; When the terminal device determines that the first occupancy rate of the resource and / or a second occupancy rate of a resource of the terminal device satisfy a first preset condition, a second virtual screen is created in the terminal device, and the target application is started in the second virtual screen based on the target package name information. receive the image data of the target application transmitted by the terminal device, and render and display the image data of the target application on a second canvas in the desktop environment.

5. The method of claim 4, wherein, The determining, according to the target hash value, whether to start the target application in the head-mounted display device or the terminal device further includes: when the terminal device determines that the first occupancy rate of the resource and the second occupancy rate of the resource of the terminal device satisfy a second preset condition, sending, by the terminal device, first notification information to the head-mounted display device; the first notification information is used to instruct starting the target application in the head-mounted display device; receiving the first notification information, and creating a first virtual screen and a first canvas in the desktop environment; starting the target application in the first virtual screen, and rendering and displaying texture information of the first virtual screen to the first canvas.

6. The method of claim 1, wherein, The determining, according to the target hash value, whether to start the target application in the head-mounted display device or the terminal device further includes: when the target hash value is used to represent that the device on which the target application is installed is the terminal device, sending the target package name information to the terminal device; starting, by the terminal device, a second virtual screen, and starting the target application in the second virtual screen based on the target package name information; receiving the image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application on a second canvas in the desktop environment.

7. The method of claim 1, wherein, The method further includes: obtaining a third occupancy rate of a resource of the head-mounted display device; when the third occupancy rate of the resource exceeds a first occupancy rate threshold, outputting first prompt information; The first prompt information is used to prompt that the number of applications started by the head-mounted display device exceeds a set number.

8. The method of claim 4, wherein, After receiving the image data of the target application transmitted by the terminal device, and rendering and displaying the image data of the target application on a second canvas in the desktop environment, the method further includes: displaying a switching control on the second canvas; receiving a first input to the switching control; in response to the first input, controlling the terminal device to close the second virtual screen, and sending, by the terminal device, second notification information to the head-mounted display device; the second notification information is used to instruct starting the target application in the head-mounted display device; receiving the second notification information, and creating a first virtual screen and a first canvas in the desktop environment; starting the target application in the first virtual screen, and rendering and displaying texture information of the first virtual screen to the first canvas.

9. A display device comprising: The apparatus includes: a receiving module, configured to receive, in a running process of a desktop environment, a first starting instruction for starting a target application; a first obtaining module, configured to, in response to the first starting instruction, obtain a first hash table and target package name information of the target application; a key of the first hash table is package name information of any application installed in a head-mounted display device and a terminal device, and a hash value corresponding to the key is used to represent a device on which the any application is installed; A second obtaining module, configured to obtain a target hash value corresponding to the target package name information according to the first hash table and the target package name information; A determining module, configured to determine to start the target application in the head-mounted display device or the terminal device according to the target hash value.

10. A head-mounted display device, comprising: The head-mounted display device comprises: A memory, configured to store executable computer instructions; A processor, configured to execute the control method according to any one of claims 1-8 according to control of the executable computer instructions.

11. A computer readable storage medium, having computer instructions stored thereon, wherein the computer instructions are run by a processor to execute the control method according to any one of claims 1-8.

Images