Control method and apparatus, and head-mounted display device and medium

Abstract

Disclosed in the present invention are a control method and apparatus, and a head-mounted display device and a medium. The method comprises: launching a first virtual screen to run a first 3D application, and creating in a rendering scene of a 3D desktop application a first canvas corresponding to the first virtual screen; and creating a first mask on the first canvas, and filling the first mask with an application image of the first 3D application.

Description

Control method and device, head-mounted display device, and medium

[0001] The present application claims priority to the Chinese patent application No. 202410852275.1, filed on June 27, 2024, and entitled "Control method and device, head-mounted display device, and medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of head-mounted display devices, and more particularly, to a control method, a control device, a head-mounted display device, and a computer-readable storage medium. BACKGROUND

[0003] In the user experience of augmented reality, application multi-opening in the AR Launcher is an important use scenario. However, in the current AR products such as AR glasses, when application multi-opening is implemented in the AR glasses, since the AR Launcher itself is a 3D application, if another 3D application is opened, the other 3D application cannot be normally displayed in the AR Launcher scene, so the effect of multi-opening cannot be achieved. SUMMARY

[0004] Embodiments of the present application aim to provide a control method, a control device, a head-mounted display device, and a computer-readable storage medium.

[0005] According to a first aspect of embodiments of the present application, a control method is provided, the method comprising:

[0006] opening a first virtual screen to run a first 3D application, and creating a first canvas corresponding to the first virtual screen in a rendering scene of a 3D desktop application;

[0007] creating a first mask in the first canvas, and filling an application picture of the first 3D application in the first mask.

[0008] Optionally, the creating the first mask in the first canvas and filling the application picture of the first 3D application in the first mask comprises:

[0009] obtaining a first pixel region of the first canvas in the rendering scene of the 3D desktop application;

[0010] creating the first mask in the first pixel region, and filling the application picture of the first 3D application in the first mask.

[0011] Optionally, the obtaining the first pixel region of the first canvas in the rendering scene of the 3D desktop application comprises:

[0012] obtaining canvas information of the first canvas;

[0013] obtaining freedom information of the head-mounted display device;

[0014] obtaining parameter information of a camera of the head-mounted display device;

[0015] determining, according to the freedom information, the parameter information of the camera, and the canvas information, a first pixel region of the first canvas in a rendering scene of the 3D desktop application.

[0016] Optionally, the determining, according to the freedom information, the parameter information of the camera, and the canvas information, a first pixel region of the first canvas in a rendering scene of the 3D desktop application, comprises:

[0017] clipping, according to the freedom information, the parameter information of the camera, and the canvas information, a primitive of the first canvas within a frustum range;

[0018] converting coordinates of the primitive of the first canvas after clipping to a screen coordinate system;

[0019] determining, according to vertex information of the primitive of the first canvas after conversion, a first pixel region of the first canvas in a rendering scene of the 3D desktop application.

[0020] Optionally, the canvas information relates to at least one of the following: position information, rotation information, and size information.

[0021] Optionally, after the first canvas creates a first mask and fills an application screen of the first 3D application in the first mask, the method further comprises:

[0022] starting a second virtual screen to run a second 3D application, and creating a second canvas corresponding to the second virtual screen in a rendering scene of the 3D desktop application;

[0023] creating a second mask in the second canvas, and filling an application screen of the second 3D application in the second mask.

[0024] Optionally, after the first canvas creates a first mask and fills an application screen of the first 3D application in the first mask, the method further comprises:

[0025] receiving an enlargement input to the first mask;

[0026] In response to the amplified input, in a case where the first mask is full-screen displayed on a display area of a head-mounted display device, a third canvas corresponding to each third 3D application is respectively created, and an application picture of the corresponding third 3D application is filled in the third mask;

[0027] The third 3D application is a 3D application other than the first 3D application and the 3D desktop application, and the 3D desktop application is a desktop application of the head-mounted display device.

[0028] According to a second aspect of the embodiment of the present application, a control device is provided, which comprises:

[0029] A first creating module is configured to start a first virtual screen to run a first 3D application, and create a first canvas corresponding to the first virtual screen in a rendering scene of a 3D desktop application;

[0030] The first creating module is configured to create a first mask in the first canvas, and fill an application picture of the first 3D application in the first mask.

[0031] Optionally, the second creating module is specifically configured to:

[0032] Obtain a first pixel area of the first canvas in the rendering scene of the 3D desktop application;

[0033] Create the first mask in the first pixel area, and fill the application picture of the first 3D application in the first mask.

[0034] Optionally, the second creating module is specifically configured to:

[0035] Obtain canvas information of the first canvas;

[0036] Obtain degree-of-freedom information of a head-mounted display device;

[0037] Obtain parameter information of a camera of the head-mounted display device;

[0038] According to the degree-of-freedom information, the parameter information of the camera and the canvas information, determine a first pixel area of the first canvas in the rendering scene of the 3D desktop application.

[0039] According to a third aspect of the embodiment of the present application, a head-mounted display device is provided, which comprises:

[0040] A memory is configured to store executable computer instructions;

[0041] A processor is configured to execute the control method according to the first aspect above according to control of the executable computer instructions.

[0042] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to perform the control method according to the first aspect.

[0043] An advantage of the embodiments of the present application is that, during the running of the 3D desktop application, if the first 3D application needs to be started, the first 3D application can be run on the first virtual screen, a first canvas corresponding to the first virtual screen can be created in the rendering scene of the 3D desktop application, a first mask can be created in the first canvas, and the application screen of the first 3D application can be filled in the first mask. In this way, during the running of the 3D desktop application, the first mask is added in the rendering scene of the 3D desktop application, and the display content of the first mask is the application screen of the first 3D application, so that the effect of starting another 3D application in the scene of the 3D desktop application is realized, and the multi-opening of the 3D application is realized.

[0044] Other features of the present application, and their advantages, will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0046] Fig. 1 is a schematic diagram of a hardware configuration of a head-mounted display device according to an embodiment of the present application;

[0047] Fig. 2 is a schematic diagram of a flow of a control method according to an embodiment of the present application;

[0048] Fig. 3a is a schematic diagram of a display area of a head-mounted display device according to an embodiment of the present application;

[0049] Fig. 3b is a schematic diagram of a display area of a head-mounted display device according to an embodiment of the present application;

[0050] Fig. 4 is a schematic diagram of a principle block of a control device according to an embodiment of the present application;

[0051] Fig. 5 is a schematic diagram of a principle block of a head-mounted display device according to an embodiment of the present application.

[0052] The implementation, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0053] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0054] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0055] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. For example, "A and / or B" includes A scheme, or B scheme, or A and B simultaneously meet the scheme. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person of ordinary skill in the art, and when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection claimed by the present application.

[0056] <Hardware configuration>

[0057] FIG. 1 is a block diagram of a hardware configuration of a head-mounted display device 1000 according to an embodiment of the present application.

[0058] As shown in FIG. 1, the head-mounted display device 1000 can be smart glasses, which can be AR glasses, and of course can be other devices, and the embodiments of the present application do not limit this.

[0059] In one embodiment, as shown in FIG. 1, the head-mounted display device 1000 can include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc.

[0060] The processor 1100 can include, but is not limited to, a central processing unit (CPU), a microprocessor (MCU), and the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, and the like. The communication device 1400 can perform wired or wireless communication. The display device 1500 is, for example, a liquid crystal display, an LED display, an OLED (Organic Light-Emitting Diode) display, and the like. The input device 1600 includes, for example, a touch screen, a keyboard, a joystick, and the like. The head-mounted display device 1000 can output audio information through the speaker 1700 and can collect audio information through the microphone 1800.

[0061] Those skilled in the art should understand that although a plurality of devices of the head-mounted display device 1000 are shown in FIG. 1, the head-mounted display device 1000 of the embodiments of the present specification can only involve part of the devices, and can also include other devices, which are not limited herein.

[0062] In the present embodiment, the memory 1200 of the head-mounted display device 1000 is configured to store instructions for controlling the processor 1100 to operate to implement or support the implementation of the control method according to any embodiment. Those skilled in the art can design the instructions according to the solutions disclosed in the present specification. How the instructions control the processor to operate is known in the art, and thus will not be described in detail herein.

[0063] In the above description, those skilled in the art can design the instructions according to the solutions provided by the present application. How the instructions control the processor to operate is known in the art, and thus will not be described in detail herein.

[0064] The head-mounted display device shown in FIG. 1 is only illustrative and is by no means intended to limit the present application, its application or use.

[0065] In the following, various embodiments and examples according to the present application are described with reference to the accompanying drawings.

[0066] <Method Embodiment>

[0067] FIG. 2 shows a control method of one embodiment of the present application, which can be implemented by a head-mounted display device, or can be implemented by a control device independent of the head-mounted display device and the head-mounted display device together, or can be implemented by a cloud server, an interactive device and a head-mounted display device together, wherein the head-mounted display device can be an AR glasses, which can be a split type AR glasses or an integrated type AR glasses, and the interactive device can be a joystick, a mouse, a mobile phone, and the like.

[0068] As shown in FIG. 2, the control method of this embodiment can include the following steps S2100-S2200.

[0069] In step S2100, a first virtual screen is started to run a first 3D application, and a first canvas corresponding to the first virtual screen is created in a rendering scene of a 3D desktop application.

[0070] In this embodiment, the step S2100 of starting the first virtual screen to run the first 3D application can further include: during running of the 3D desktop application, if a first starting instruction for starting the first 3D application is received, the first virtual screen is started to run the first 3D application in response to the first starting instruction.

[0071] The first starting instruction can be a touch input to an icon of the first 3D application, or a ray event sent by an interactive device to the icon of the first 3D application. The interactive device can be a handle, a mouse, a mobile phone, or the like, or a gesture event of a user to the icon of the first 3D application.

[0072] In this embodiment, in the eye movement interaction mode, a gaze state of a wearer of the head-mounted display device is acquired, and an operation matched with the gaze state is performed. The ray event of the interactive device to the icon of the first 3D application is a ray event in the eye movement and ray double interaction mode, a gaze position of the wearer of the head-mounted display device is acquired, and a projection position of the ray event in a fusion image is acquired, and an operation matched with the gaze position and the projection position is performed. Generally, eye movement interaction needs to be tracked, and the head-mounted display device captures three-dimensional information of the eye, such as three-dimensional coordinates of the eye, and then starts the first 3D application according to the three-dimensional information of the eye. That is, the 3D application can be started by using the three-dimensional information of the eye, so as to improve the starting accuracy of the 3D application.

[0073] It should be noted that in the eye movement tracking mode, the gaze position of the wearer on the fusion image and the gaze time period corresponding to the gaze position can be determined by an eye movement tracking algorithm. The gaze position and the gaze time period can be recorded as a gaze state. In the eye movement tracking mode, different operations are performed by the control device for different gaze states. For example, in the case where the gaze state indicates that the gaze position does not change and the gaze time period corresponding to the gaze position reaches a preset time length, it is determined that the wearer is in a state of gazing at the gaze position, and an operation matched with the state of gazing at the gaze position is performed. For example, the operation matched with the state of gazing at the gaze position can be a confirmation operation.

[0074] Of course, the position of the ray event on the fusion image can also be determined to coincide with the gaze position of the wearer in a case where the difference between the position of the ray event on the fusion image and the gaze position of the wearer is less than a preset threshold, and the operation corresponding to the gaze position and / or the ray event at the projection position is performed. For example, in a case where the gaze position and / or the ray event at the projection position correspond to an icon of a first 3D application, the first 3D application is started.

[0075] In the embodiment, the desktop application of the head-mounted display device is referred to as an AR Launcher application, which is a 3D application and is automatically started when the head-mounted display device is powered on. During the running of the AR Launcher application, if the head-mounted display device receives a first starting instruction for starting a first 3D application, the first 3D application can be started by the AR Launcher application through a first virtual screen, and a first canvas corresponding to the first virtual screen is newly created in the rendering scene of the AR Launcher application. Generally, the created first canvas can include a left canvas and a right canvas, the left canvas is located in the left part of the display area, and the right canvas is located in the right part of the display area.

[0076] After the above step S2100 of starting the first virtual screen to run the first 3D application and creating the first canvas corresponding to the first virtual screen in the rendering scene of the 3D desktop application, the following step S2200 is entered:

[0077] Step S2200, creating a first mask in the first canvas and filling an application screen of the first 3D application in the first mask.

[0078] In the embodiment, the step S2200 of creating a first mask in the first canvas and filling an application screen of the first 3D application in the first mask can further include the following steps S2210-S2220:

[0079] Step S2210, obtaining a first pixel area of the first canvas in the rendering scene of the 3D desktop application.

[0080] It should be noted that the embodiment of the application adds a mask calculation module to the OpenXR Runtime rendering process, which can be used to calculate the pixel area occupied by the first canvas in the rendering scene of the AR Launcher application, so as to create a corresponding first mask in the pixel area based on subsequent steps. The pixel area includes a plurality of pixel points, and since the first canvas generally includes a left canvas and a right canvas, the created first mask generally also includes two masks, that is, the first mask includes a mask corresponding to the left canvas and a mask corresponding to the right canvas.

[0081] Specifically, the step S2210 of obtaining the first pixel region of the first canvas in the rendering scene of the 3D desktop application can further include the following steps S2211-S2214:

[0082] Step S2211, obtaining the canvas information of the first canvas.

[0083] The canvas information relates to at least one of the following: position information, rotation information, and size information.

[0084] Specifically, in the case of newly creating a first canvas corresponding to a first virtual screen in the rendering scene of the AR Launcher application, the head-mounted display device transmits the canvas information of the first canvas to the OpenXR Runtime rendering process.

[0085] Step S2212, obtaining the degree of freedom information of the head-mounted display device.

[0086] The degree of freedom information of the head-mounted display device refers to six-degree-of-freedom information (hereinafter referred to as 6DoF information) of the head-mounted display device.

[0087] Specifically, the degree of freedom information of the head-mounted display device can be obtained by the OpenXR Runtime rendering process.

[0088] Step S2213, obtaining the parameter information of the camera of the head-mounted display device.

[0089] The camera of the head-mounted display device includes a left camera and a right camera.

[0090] Specifically, the parameter information of the camera of the head-mounted display device can be obtained by the OpenXR Runtime rendering process.

[0091] Step S2214, determining the first pixel region of the first canvas in the rendering scene of the 3D desktop application according to the degree of freedom information, the parameter information of the camera, and the canvas information.

[0092] Specifically, the step S2214 usually specifically includes: clipping the primitives of the first canvas within the frustum range according to the degree of freedom information, the parameter information of the camera, and the canvas information; converting the coordinates of the primitives constituting the first canvas after clipping to a screen coordinate system; and determining the first pixel region of the first canvas in the rendering scene of the 3D desktop application according to the vertex information of the converted primitives of the first canvas. The coordinates of the primitives constituting the first canvas after clipping can be x coordinates and y coordinates. Generally, the first pixel region is determined according to the vertex information of the converted primitives of the first canvas.

[0093] According to the above steps S2211-S2214, the first pixel region of the first canvas in the rendering scene of the 3D desktop application can be determined, and the first mask based on the first pixel region can be created in the first canvas.

[0094] Step S2220, creating the first mask in the first pixel region and filling the application picture of the first 3D application in the first mask.

[0095] It should be noted that creating the first mask in the first pixel region can be understood as erasing the first pixel region in the rendering scene of the AR Launcher application. After creating the first mask in the first pixel region, the application picture of the first 3D application is filled into the first mask, so that the application picture of the AR Launcher application can be used as the foreground texture, i.e., the pixels other than the first mask position are visible, and the application picture of the first 3D application can be used as the background texture, i.e., only the pixels of the first mask position are visible, and the fused texture is directly displayed on the display area of the head-mounted display device.

[0096] Exemplarily, during the running of the AR Launcher application, the user opens the 3D application 1, the AR Launcher application opens the virtual screen 1 to run the 3D application 1, and a new canvas 1 corresponding to the virtual screen 1 is created in the rendering scene of the AR Launcher application, and the canvas information of the canvas 1 is transmitted to the OpenXR Runtime rendering process. According to the canvas information of the canvas 1, the 6DoF information of the head-mounted display device, and the camera parameter information, the OpenXR Runtime rendering process determines the corresponding pixel points occupied by the canvas 1 in the rendering scene of the AR Launcher application as the pixel points corresponding to the mask 1. Then, referring to FIG. 3a, based on the corresponding pixel points occupied by the canvas 1 in the rendering scene of the AR Launcher application, the mask 1 (including the mask 1 corresponding to the left canvas and the mask 1 corresponding to the right canvas) is created, and the application picture of the 3D application 1 is filled into the mask 1, so that the application picture of the AR Launcher application can be used as the foreground texture, i.e., the pixels other than the mask 1 position are visible, and the application picture of the 3D application 1 can be used as the background texture, i.e., only the pixels of the mask 1 position are visible, and the foreground texture and the background texture are displayed on the display area of the head-mounted display device. Referring to FIG. 3a, the effect of opening a “window” of the 3D application 1 in the scene of the AR Launcher application can be realized.

[0097] According to the embodiment of the present application, during the running of the 3D desktop application, if the first 3D application needs to be started, the first virtual screen is started to run the first 3D application, a first canvas corresponding to the first virtual screen is created in the rendering scene of the 3D desktop application, a first mask is created in the first canvas, and the application picture of the first 3D application is filled in the first mask. In this way, during the running of the 3D desktop application, the first mask is added in the rendering scene of the 3D desktop application, and the display content of the first mask is the application picture of the first 3D application, so that the effect of starting another 3D application in the scene of the 3D desktop application is realized, and the multi-opening of the 3D application is realized.

[0098] In one embodiment, after the above step S2200 of creating the first mask in the first canvas and filling the application picture of the first 3D application in the first mask, the control method of the embodiment of the present application further includes the following steps S3100-S3200.

[0099] Step S3100, starting a second virtual screen to run a second 3D application, and creating a second canvas corresponding to the second virtual screen in the rendering scene of the 3D desktop application.

[0100] In the embodiment, the step S3100 of starting the second virtual screen to run the second 3D application can further include: during the running of the 3D desktop application and the first 3D application, if a second starting instruction for starting the second 3D application is received, the second virtual screen is started to run the second 3D application in response to the second starting instruction.

[0101] The second starting instruction can be a touch input to the icon of the second 3D application, or a ray event sent by an interactive device to the icon of the second 3D application. The interactive device can be a handle, a mouse, a mobile phone, or the like. The second starting instruction can also be a gesture event of a user to the icon of the second 3D application.

[0102] As described above, in the eye tracking mode, the gaze position of the wearer on the fusion image and the gaze time period corresponding to the gaze position can be determined by the eye tracking algorithm. The gaze position and the gaze time period can be recorded as a gaze state. In the eye tracking mode, the control device performs different operations for different gaze states. For example, in the case where the gaze state indicates that the gaze position does not change and the gaze time period corresponding to the gaze position reaches a preset time length, it is determined that the wearer is in a state of gazing at the gaze position, and an operation matched with the state of gazing at the gaze position is performed. For example, the operation matched with the state of gazing at the gaze position can be a confirmation operation.

[0103] Of course, the position of the ray event on the fusion image can also be determined to coincide with the gaze position of the wearer in a case where a difference between the position of the ray event on the fusion image and the gaze position of the wearer is less than a preset threshold, and the operation corresponding to the gaze position and / or the ray event at the projection position is performed. For example, in a case where the gaze position and / or the ray event at the projection position corresponds to an icon of a second 3D application, the second 3D application is started.

[0104] In this embodiment, in a case where the AR Launcher application and the first 3D application are both in a running state, if the head-mounted display device receives a second starting instruction for starting a second 3D application, the AR Launcher application can be used to start a second virtual screen, and a second canvas corresponding to the second virtual screen can be newly created in the rendering scene of the AR Launcher application. Generally, the created second canvas can include a left canvas and a right canvas, the left canvas is located in the left part of the display area, and the right canvas is located in the right part of the display area.

[0105] Step S3200, a second mask is created in the second canvas, and an application screen of the second 3D application is filled in the second mask.

[0106] In this embodiment, the step S3200 of creating a second mask in the second canvas and filling an application screen of the second 3D application in the second mask can further include the following steps S3210-S3220:

[0107] Step S3210, a second pixel region of the second canvas in the rendering scene of the 3D desktop application is obtained.

[0108] It should be noted that the mask calculation module of the OpenXR Runtime rendering process can be used to calculate the pixel region occupied by the second canvas in the rendering scene of the AR Launcher application, so as to create a second mask corresponding to the pixel region based on subsequent steps. The pixel region includes a plurality of pixel points, and since the second canvas generally includes a left canvas and a right canvas, the created second mask generally also includes two masks, that is, the second mask includes a mask corresponding to the left canvas and a mask corresponding to the right canvas.

[0109] Specifically, the OpenXR Runtime rendering process can obtain canvas information of the second canvas, degree of freedom information of the head-mounted display device, and parameter information of the camera of the head-mounted display device, and determine the second pixel region of the second canvas in the rendering scene of the AR Launcher application according to the canvas information of the second canvas, the degree of freedom information of the head-mounted display device, and the parameter information of the camera of the head-mounted display device.

[0110] The determining the second pixel region of the second canvas in the rendering scene of the AR Launcher application according to the canvas information of the second canvas, the degree-of-freedom information of the head-mounted display device, and the parameter information of the camera of the head-mounted display device can specifically include: clipping the primitives of the second canvas within the frustum range according to the degree-of-freedom information, the parameter information of the camera, and the canvas information; converting the coordinates of the primitives of the second canvas after clipping to a screen coordinate system; and determining the second pixel region of the second canvas in the rendering scene of the 3D desktop application according to the vertex information of the primitives of the second canvas after conversion.

[0111] At step S3220, the second mask is created in the second pixel region, and the application screen of the second 3D application is filled in the second mask.

[0112] It should be noted that creating the second mask in the second pixel region can be understood as erasing the second pixel region in the rendering scene of the AR Launcher application. After creating the second mask in the second pixel region, the application screen of the second 3D application is filled in the second mask, so that the application screen of the AR Launcher application can be used as foreground texture, the application screen of the second 3D application can be used as background texture, and the fused texture can be directly displayed on the display area of the head-mounted display device.

[0113] Exemplarily, during the running of the AR Launcher application and the 3D application 1, the user opens the 3D application 2, the AR Launcher application opens the virtual screen 2 to run the 3D application 2, and a new canvas 2 corresponding to the virtual screen 2 is created in the rendering scene of the AR Launcher application, and the canvas information of the canvas 2 is transmitted to the OpenXR Runtime rendering process. The OpenXR Runtime rendering process determines the corresponding pixel points occupied by the canvas 2 in the rendering scene of the AR Launcher application as the pixel points corresponding to the mask 2 according to the canvas information of the canvas 2, the 6DoF information of the head-mounted display device, and the camera parameter information. Then, referring to FIG. 3b, the mask 2 (including the mask 2 corresponding to the left canvas and the mask 2 corresponding to the right canvas in the canvas 2) is created based on the corresponding pixel points occupied by the canvas 2 in the rendering scene of the AR Launcher application, and the application screen of the 3D application 2 is filled in the mask 2. Referring to FIG. 3b, the scene “window” effect of the 3D application 2 can be realized on the basis of the scene “window” effect of the 3D application 1 in the scene of the AR Launcher application.

[0114] It should be noted that the user can continue to open 3D applications 3, 3D application 4, etc. to open the "windows" of multiple scenes, so that multiple scenes with 6DoF effect coexist.

[0115] In one embodiment, after the above step S2200, the control method of the embodiment of the present application further comprises the following steps S4100-S4200:

[0116] Step S4100, receiving a zoom-in input on the first mask.

[0117] Step S4200, in response to the zoom-in input, when the first mask is displayed full screen on the display area of the display device, creating a corresponding third mask in each third canvas corresponding to each third 3D application respectively, and filling the application screen of the corresponding third 3D application in the third mask.

[0118] The third 3D application is other 3D application run by the head-mounted display device in addition to the first 3D application, and the other 3D application includes the 3D desktop application.

[0119] Specifically, in response to the zoom-in input, when the first mask is displayed full screen on the display area of the head-mounted display device, the third pixel area of each third canvas corresponding to each third 3D application in the rendering scene of the first 3D application is obtained, and for each third pixel area, a corresponding third mask is created in the third pixel area, and the application screen of the corresponding third 3D application is filled in the third mask.

[0120] It should be noted that, based on the implementation of opening the "window" of the scene of 3D application 1 and the "window" of the scene of 3D application 2 in the scene of AR Launcher application, if the user looks at 3D application 1, the "window" of the scene of 3D application 1 is displayed full screen, at this time, the effect of opening the "window" of the scene of AR Launcher application and the "window" of the scene of 3D application 2 in the scene of 3D application 1 can be realized.

[0121] <Device Embodiment>

[0122] Fig. 4 is a schematic diagram of a control device according to an embodiment. Referring to Fig. 4, the control device 400 comprises a first creating module 410 and a second creating module 420.

[0123] The first creating module 410 is configured to start a first 3D application on a first virtual screen, and create a first canvas corresponding to the first virtual screen in a rendering scene of a 3D desktop application.

[0124] The second creating module 420 is configured to create a first mask in the first canvas, and fill the application picture of the first 3D application in the first mask.

[0125] In an embodiment, the second creating module 420 is specifically configured to: acquire a first pixel area of the first canvas in a rendering scene of the 3D desktop application; create the first mask in the first pixel area, and fill the application picture of the first 3D application in the first mask.

[0126] In an embodiment, the second creating module 420 is specifically configured to: acquire canvas information of the first canvas; acquire degree-of-freedom information of a head-mounted display device; acquire parameter information of a camera of the head-mounted display device; and determine a first pixel area of the first canvas in a rendering scene of the 3D desktop application according to the degree-of-freedom information, the parameter information of the camera, and the canvas information.

[0127] In an embodiment, the second creating module 420 is specifically configured to: clip a primitive of the first canvas within a frustum range according to the degree-of-freedom information, the parameter information of the camera, and the canvas information; convert a coordinate of the primitive of the first canvas after clipping to a screen coordinate system; and take vertex information of the primitive of the first canvas after conversion as the first pixel area of the first canvas in the rendering scene of the 3D desktop application.

[0128] In an embodiment, the canvas information relates to at least one of the following: position information, rotation information, and size information.

[0129] In an embodiment, the first creating module 410 is further configured to start a second virtual screen to run a second 3D application, and create a second canvas corresponding to the second virtual screen in the rendering scene of the 3D desktop application.

[0130] The second creating module 420 is further configured to create a second mask in the second canvas, and fill the application picture of the second 3D application in the second mask.

[0131] In an embodiment, the apparatus further includes a receiving module (not shown in the figure).

[0132] The receiving module is configured to receive an input of zooming in on the first mask.

[0133] The second creating module 420 is further configured to, in response to the input of zooming in, create a third mask in each third canvas corresponding to each third 3D application respectively in a case that the first mask is displayed full screen in a display area of a display device, and fill the application picture of the corresponding third 3D application in the third mask.

[0134] The third 3D application is another 3D application run by the head-mounted display device other than the first 3D application, and the another 3D application includes the 3D desktop application.

[0135] According to the embodiment of the present application, during the running of the 3D desktop application, if the first 3D application needs to be started, the first virtual screen is opened to run the first 3D application, a first canvas corresponding to the first virtual screen is created in the rendering scene of the 3D desktop application, a first mask is created in the first canvas, and the application screen of the first 3D application is filled in the first mask. In this way, during the running of the 3D desktop application, the first mask is added in the rendering scene of the 3D desktop application, and the display content of the first mask is the application screen of the first 3D application, so that the effect of opening another 3D application in the scene of the 3D desktop application is realized, and the multi-opening of 3D applications is realized.

[0136] <Device Embodiment>

[0137] FIG. 5 is a schematic diagram of the hardware structure of the head-mounted display device according to an embodiment. As shown in FIG. 5, the head-mounted display device 500 includes a processor 510 and a memory 520.

[0138] The memory 520 can be used to store executable computer instructions.

[0139] The processor 510 can be used to control the execution of the control method according to the embodiments of the method of the present application according to the executable computer instructions.

[0140] The head-mounted display device 500 can be the head-mounted display device 1000 as shown in FIG. 1.

[0141] In another embodiment, the head-mounted display device 500 can include the control device 400 described above.

[0142] In an embodiment, the modules of the control device 400 described above can be realized by the processor 510 running the computer instructions stored in the memory 520.

[0143] <Computer-readable storage medium>

[0144] The embodiments of the present application also provide a computer-readable storage medium having computer instructions stored thereon, and the computer instructions are run by a processor to execute the control method provided by the embodiments of the present application.

[0145] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, and the computer readable program instructions are used to cause a processor to implement various aspects of the present application.

[0146] Computer readable storage media can be tangible storage media that can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or

[0147] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0148] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0149] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0150] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0151] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0152] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0153] The above-mentioned embodiments of the application are only intended to illustrate the present application, not to limit the present application.

[0154] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art to make contributions can be embodied in the form of software products, the computer software product is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including several instructions to make a terminal device (may be a mobile phone, computer, server, television, or network equipment, etc.) executes the method of various embodiments of the present application.

[0155] The above-mentioned is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, all in the inventive concept of the present application, using the present application specification and the equivalent structure transformation of the drawing contents, or direct / indirectly applied in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A control method, characterized in that, The method includes: Start the first virtual screen to run the first 3D application, and create the first canvas corresponding to the first virtual screen in the rendering scene of the 3D desktop application; Create a first mask on the first canvas and fill the first mask with the application screen of the first 3D application.

2. The method according to claim 1, characterized in that, The step of creating a first mask on the first canvas and filling the first mask with the application screen of the first 3D application includes: Obtain the first pixel region of the first canvas in the rendering scene of the 3D desktop application; Create the first mask in the first pixel area, and fill the first mask with the application screen of the first 3D application.

3. The method according to claim 2, characterized in that, The step of obtaining the first pixel region of the first canvas in the rendering scene of the 3D desktop application includes: Obtain the canvas information of the first canvas; Obtain the degrees of freedom information of the head-mounted display device; Obtain parameter information of the camera of the head-mounted display device; Based on the degree of freedom information, the camera parameter information, and the canvas information, the first pixel region of the first canvas in the rendering scene of the 3D desktop application is determined.

4. The method according to claim 3, characterized in that, Determining the first pixel region of the first canvas in the rendering scene of the 3D desktop application based on the degree of freedom information, the camera parameter information, and the canvas information includes: Based on the degree of freedom information, the camera parameter information, and the canvas information, the primitives of the first canvas within the field of view are cropped; Transform the coordinates of the primitives that constitute the first canvas after cropping to the screen coordinate system; Based on the vertex information of the primitives of the first canvas after conversion, the first pixel region of the first canvas in the rendering scene of the 3D desktop application is determined.

5. The method according to claim 3, characterized in that, The canvas information includes at least one of the following: position information, rotation information, and size information.

6. The method according to claim 1, characterized in that, After creating a first mask on the first canvas and filling the first mask with the application screen of the first 3D application, the method further includes: Start the second virtual screen to run the second 3D application, and create the second canvas corresponding to the second virtual screen in the rendering scene of the 3D desktop application; Create a second mask on the second canvas, and fill the second mask with the application screen of the second 3D application.

7. The method according to claim 1, characterized in that, After creating a first mask on the first canvas and filling the first mask with the application screen of the first 3D application, the method further includes: Receive magnified input to the first mask; In response to the magnified input, when the first mask is fully displayed in the display area of ​​the display device, a corresponding third mask is created for each third 3D application on the third canvas, and the application screen of the corresponding third 3D application is filled into the third mask. The third 3D application is any 3D application other than the first 3D application that runs on the head-mounted display device, and the other 3D application includes the 3D desktop application.

8. A control device, characterized in that, The device includes: The first creation module is used to start the first virtual screen to run the first 3D application and create the first canvas corresponding to the first virtual screen in the rendering scene of the 3D desktop application. The second creation module is used to create a first mask on the first canvas and fill the application screen of the first 3D application into the first mask.

9. The apparatus according to claim 8, characterized in that, The second creation module is specifically used to: obtain the first pixel region of the first canvas in the rendering scene of the 3D desktop application; Create the first mask in the first pixel area, and fill the first mask with the application screen of the first 3D application.

10. The apparatus according to claim 9, characterized in that, The second creation module is specifically used for: obtaining the canvas information of the first canvas; Obtain the degrees of freedom information of the head-mounted display device; Obtain parameter information of the camera of the head-mounted display device; Based on the degree of freedom information, the camera parameter information, and the canvas information, the first pixel region of the first canvas in the rendering scene of the 3D desktop application is determined.

11. A head-mounted display device, characterized in that, The head-mounted display device includes: Memory is used to store executable computer instructions; A processor configured to execute the control method according to any one of claims 1-7, under the control of the executable computer instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, perform the control method according to any one of claims 1-7.

Images