Control method and apparatus, head-mounted display device, and medium

Abstract

Embodiments of the present invention disclose a control method and apparatus, a head-mounted display device, and a medium. The method comprises: separately displaying a corresponding 3D application image in at least one canvas created in a rendering scene of a 3D desktop application; detecting whether a first canvas among the at least one canvas is located at a central position of the rendering scene, the first canvas displaying an image of a first 3D application; and when the first canvas is not located at the central position of the rendering scene, on the basis of canvas information of the first canvas and first degree-of-freedom information of a head-mounted display device, determining second degree-of-freedom information, and allocating the second degree-of-freedom information to the first 3D application, thereby enabling the first canvas to display the image of the central position of the first 3D application.

Description

Control method and device, head-mounted display device, and medium

[0001] The present application claims priority to the Chinese patent application No. 202411036048.8, filed on July 30, 2024, and entitled "Control method and device, head-mounted display device, and medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of head-mounted display devices, and more particularly, to a control method, a control device, a head-mounted display device, and a computer-readable storage medium. BACKGROUND

[0003] In the user experience of augmented reality, application multi-opening in the AR Launcher is an important use scenario. Generally, when implementing 3D application multi-opening in the AR glasses, multiple windows can be opened in the rendering scene of the AR Launcher, and a picture of a 3D application can be displayed in each window. However, when any window of the multiple windows is not located at the center position of the rendering scene of the AR Launcher, the picture of the corresponding 3D application cannot be normally displayed in the any window. SUMMARY

[0004] An object of the present application is to provide a control method, a control device, a head-mounted display device, and a computer-readable storage medium.

[0005] According to a first aspect of the present application, a control method is provided, the method comprising:

[0006] displaying a picture of a corresponding 3D application in each of at least one canvas created in a rendering scene of a 3D desktop application;

[0007] detecting whether a first canvas of the at least one canvas is located at a center position of the rendering scene, wherein the first canvas displays a picture of a first 3D application;

[0008] in a case where the first canvas is not located at the center position of the rendering scene, determining second degree of freedom information according to canvas information of the first canvas and first degree of freedom information of a head-mounted display device and assigning the second degree of freedom information to the first 3D application, so that the first canvas displays a picture of a center position of the first 3D application.

[0009] Optionally, the canvas information of the first canvas comprises a first included angle between a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device,

[0010] determining the second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, comprises:

[0011] obtaining a first product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the first degree of freedom information as the second degree of freedom information.

[0012] Optionally, the canvas information of the first canvas is a first included angle between a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device,

[0013] after the second degree of freedom information is determined according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device and is assigned to the first 3D application without being located at the center position of the rendering scene, the method further comprises:

[0014] obtaining a second product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the third degree of freedom information of the control device as the fourth degree of freedom information and assigning the fourth degree of freedom information to the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is consistent with the direction of the second virtual identifier corresponding to the 3D desktop application.

[0015] Optionally, the method further comprises:

[0016] in a case where the fourth degree of freedom information is assigned to the first virtual identifier corresponding to the first 3D application, detecting whether a touch event sent by the control device is received;

[0017] in a case where the touch event sent by the control device is detected, obtaining a collision position of the second virtual identifier and the rendering scene;

[0018] in a case where the collision position is located at the first canvas, controlling the first 3D application to respond to the touch event through the first virtual identifier.

[0019] Optionally, the collision position is a set position,

[0020] in a case where the collision position is located at the first canvas, controlling the first 3D application to respond to the touch event through the first virtual identifier, comprises:

[0021] in a case where the collision position is located at the first canvas, rotating coordinate information of the collision position reversely by the first included angle with the center of a world coordinate system as an origin, and then controlling the first 3D application to respond to the touch event through the first virtual identifier.

[0022] Optionally, the method further comprises:

[0023] receiving an adjustment input implemented on the first canvas;

[0024] updating a position of the first canvas in the rendering scene in response to the adjustment input;

[0025] After updating the position of the first canvas in the rendering scene, the step of detecting whether the first canvas in the at least one canvas is located at the center position of the rendering scene is performed again.

[0026] According to a second aspect of the embodiments of the present disclosure, a control device is provided, which comprises:

[0027] a display module configured to display a picture of a corresponding 3D application in each of at least one canvas created in a rendering scene of a 3D desktop application;

[0028] a detection module configured to detect whether a first canvas in the at least one canvas is located at a center position of the rendering scene, wherein the first canvas displays a picture of a first 3D application;

[0029] a determination module configured to, in a case where the first canvas is not located at the center position of the rendering scene, determine second degree-of-freedom information according to canvas information of the first canvas and first degree-of-freedom information of a head-mounted display device and assign the first 3D application to the first canvas, so that the first canvas displays a picture of a center position of the first 3D application.

[0030] Optionally, the canvas information of the first canvas is a first included angle between a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, and the determination module is specifically configured to:

[0031] obtain a first product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the first degree-of-freedom information as the second degree-of-freedom information.

[0032] According to a third aspect of the embodiments of the present disclosure, a head-mounted display device is provided, which comprises:

[0033] a memory configured to store executable computer instructions;

[0034] a processor configured to execute the control method according to the first aspect above according to control of the executable computer instructions.

[0035] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to perform the control method according to the first aspect above.

[0036] One beneficial effect of the embodiments of the present disclosure is that, in the case of displaying the picture of the corresponding 3D application in each of the at least one canvas of the creative scene in the rendering scene of the 3D desktop application, if it is detected that the first canvas in the at least one canvas is not located at the center position of the rendering scene of the 3D desktop application, the second degree of freedom information is allocated to the first 3D application according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, so that the display picture of the first canvas is the picture of the center position of the corresponding first 3D application.

[0037] Other features of the present specification and its advantages will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0039] Fig. 1 is a schematic diagram of a hardware configuration of a head-mounted display device according to an embodiment of the present disclosure;

[0040] Fig. 2 is a schematic diagram of a control method according to an embodiment of the present disclosure;

[0041] Fig. 3 is a schematic diagram of a display area of a head-mounted display device according to an embodiment of the present disclosure;

[0042] Fig. 4 is a schematic diagram of a control device according to an embodiment of the present disclosure;

[0043] Fig. 5 is a schematic diagram of a head-mounted display device according to an embodiment of the present disclosure.

[0044] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0046] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, movement condition, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0047] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. For example, "A and / or B" includes A scheme, or B scheme, or A and B scheme. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of the ordinary skilled in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope required by the present application.

[0048] <Hardware configuration>

[0049] Fig. 1 is a block diagram of a hardware configuration of a head-mounted display device 1000 according to an embodiment of the present disclosure.

[0050] As shown in Fig. 1, the head-mounted display device 1000 can be smart glasses, which can be AR glasses, and of course can also be other devices, and the embodiments of the present disclosure do not limit this.

[0051] In one embodiment, as shown in Fig. 1, the head-mounted display device 1000 can include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like.

[0052] The processor 1100 can include, but is not limited to, a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, and the like. The communication device 1400 can perform wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display screen, an LED display screen, an OLED (Organic Light-Emitting Diode) display screen, and the like. The input device 1600 includes, for example, a touch screen, a keyboard, a handle, and the like. The head-mounted display device 1000 can output audio information through the speaker 1700, and can collect audio information through the microphone 1800.

[0053] Those skilled in the art should understand that although a plurality of devices of the head-mounted display device 1000 are shown in FIG. 1, the head-mounted display device 1000 of the embodiments of the present disclosure can only involve part of the devices, or can further include other devices, which are not limited herein.

[0054] In the present embodiment, the memory 1200 of the head-mounted display device 1000 is configured to store instructions for controlling the processor 1100 to operate to implement or support implementation of the control method according to any embodiment. The skilled person can design the instructions according to the solutions disclosed in the present specification. How the instructions control the processor to operate is well known in the art, and thus will not be described in detail herein.

[0055] In the above description, the skilled person can design the instructions according to the solutions provided in the present disclosure. How the instructions control the processor to operate is well known in the art, and thus will not be described in detail herein.

[0056] The head-mounted display device shown in FIG. 1 is merely illustrative and is by no means intended to limit the present disclosure, its application or use.

[0057] In the following, various embodiments and examples according to the present disclosure are described with reference to the accompanying drawings.

[0058] <Method Embodiment>

[0059] FIG. 2 shows a control method of one embodiment of the present disclosure, which can be implemented by a head-mounted display device, or can be implemented by a control device independent of the head-mounted display device and the head-mounted display device together, or can be implemented by a cloud server, a control device and a head-mounted display device together, wherein the head-mounted display device can be an AR glasses, which can be a split-type AR glasses or an integrated AR glasses, and the control device can be a handle, a mouse, a mobile phone, etc.

[0060] As shown in FIG. 2, the control method of the present embodiment can include the following steps S2100-S2300:

[0061] In step S2100, a picture of a corresponding 3D application is respectively displayed in at least one canvas created in a rendering scene of a 3D desktop application.

[0062] In the present embodiment, the desktop application of the head-mounted display device is referred to as an AR Launcher application, which is a 3D application, and the AR Launcher application is automatically started when the head-mounted display device is powered on.

[0063] In the running process of the AR Launcher application, if the head-mounted display device receives a starting instruction for starting a 3D application, the AR Launcher application can be started to run the 3D application on a virtual screen, and a canvas corresponding to the virtual screen is newly created in the rendering scene of the AR Launcher application. Generally, the created canvas can include a left canvas and a right canvas, the left canvas is located in the left part of the display area, and the right canvas is located in the right part of the display area. Moreover, a mask can be created for the canvas corresponding to the virtual screen, and the screen of the corresponding 3D application is filled in the mask to achieve the effect of starting other 3D applications in the AR Launcher application.

[0064] It should be noted that by adding a mask calculation module in the OpenXR Runtime rendering process, the mask calculation module can be used to calculate the pixel area occupied by the above-mentioned canvas in the rendering scene of the AR Launcher application according to the canvas information of the above-mentioned canvas, so as to create a corresponding mask based on the pixel area. Wherein, the pixel area includes a plurality of pixel points, and since the canvas usually includes a left canvas and a right canvas, the created mask also usually includes two masks, that is, the mask includes a mask corresponding to the left canvas and a mask corresponding to the right canvas.

[0065] It should be noted that the above-mentioned starting instruction can be a touch input to the icon of the 3D application to be started, or a ray event sent by a control device to the icon of the 3D application to be started. The control device can be a handle, a mouse, a mobile phone, etc. It can also be a gesture event of the user to the icon of the 3D application to be started.

[0066] In the embodiment, in the eye movement interaction mode, the gaze state of the wearer of the head-mounted display device is obtained, and an operation matched with the gaze state is performed. The control device generates a ray event to the icon of the 3D application to be started, and the ray event and the eye movement double interaction mode obtain the gaze position of the wearer of the head-mounted display device and the projection position of the ray event in the fusion image, and perform an operation matched with the gaze position and the projection position. Generally, eye movement interaction needs to perform eye movement tracking, and the head-mounted display device will capture three-dimensional information of the human eye such as three-dimensional coordinates of the human eye when performing eye movement tracking, and then start the 3D application according to the three-dimensional information of the human eye. That is, the 3D application can be started by using the three-dimensional information of the human eye, which can improve the starting accuracy of the 3D application.

[0067] It should be noted that in the eye tracking mode, the gaze position of the wearer on the fusion image and the gaze time period corresponding to the gaze position can be determined by the eye tracking algorithm. The gaze position and the gaze time period can be recorded as a gaze state. In addition, in the eye tracking mode, the control device performs different operations for different gaze states. For example, in the case where the gaze state indicates that the gaze position does not change and the gaze time period corresponding to the gaze position reaches a preset duration, it is determined that the wearer is in a state of gazing at the gaze position, and at this time, an operation matched with the state of gazing at the gaze position is performed. For example, the operation matched with the state of gazing at the gaze position can be a confirmation operation.

[0068] Of course, in the case where the distance between the position of the ray event on the fusion image and the gaze position of the wearer is less than a preset threshold, it can be determined that the position of the ray event on the fusion image coincides with the gaze position of the wearer, and at this time, an operation corresponding to the gaze position and / or the ray event at the projection position is performed. For example, in the case where the gaze position and / or the ray event at the projection position corresponds to the icon of the 3D application, the 3D application is opened.

[0069] Exemplarily, during the running of the AR Launcher application, the user opens the 3D application 1, the AR Launcher application opens the virtual screen 1 to run the 3D application 1, and a canvas 1 corresponding to the virtual screen 1 is newly created in the rendering scene of the AR Launcher application, and the canvas information of the canvas 1 is transmitted to the OpenXR Runtime rendering process. The OpenXR Runtime rendering process determines the corresponding pixel points occupied by the canvas 1 in the rendering scene of the AR Launcher application as the pixel points corresponding to the corresponding mask 1 according to the canvas information of the canvas 1. Then, referring to FIG. 3, based on the corresponding pixel points occupied by the canvas 1 in the rendering scene of the AR Launcher application, the mask 1 (including the mask 1 corresponding to the left canvas and the mask 1 corresponding to the right canvas) is created, and the screen of the 3D application 1 is filled into the mask 1, so that the effect of opening a scene "window" of the 3D application 1 in the scene of the AR Launcher application can be realized.

[0070] Then, during the running of the AR Launcher application and the 3D application 1, the user opens the 3D application 2, the AR Launcher application opens the virtual screen 2 to run the 3D application 2, and a new canvas 2 corresponding to the virtual screen 2 is created in the rendering scene of the AR Launcher application, and the canvas information of the canvas 2 is transmitted to the OpenXR Runtime rendering process. The OpenXR Runtime rendering process determines the corresponding pixel points occupied by the canvas 2 in the rendering scene of the AR Launcher application as the pixel points corresponding to the mask 2 according to the canvas information of the canvas 2. Then, referring to FIG. 3, based on the corresponding pixel points occupied by the canvas 2 in the rendering scene of the AR Launcher application, the mask 2 (involving the mask 2 corresponding to the left canvas and the mask 2 corresponding to the right canvas in the canvas 2) is created, and the screen of the 3D application 2 is filled into the mask 2, so that the scene "window" effect of the 3D application 2 is continued to be opened on the basis of the scene "window" effect of the 3D application 1 which has been opened in the scene of the AR Launcher application.

[0071] It should be noted that the user can continue to open 3D applications such as the 3D application 3 and the 3D application 4 to open the "windows" of multiple scenes.

[0072] After the above step S2100 is performed to display the screens of the corresponding 3D applications in the at least one canvas created in the rendering scene of the 3D desktop application, the following step is performed:

[0073] Step S2200: detecting whether a first canvas in the at least one canvas is located at a center position of the rendering scene.

[0074] The first canvas displays the screen of the first 3D application. The first canvas is any one of the at least one canvas.

[0075] In this embodiment, in the case where the screens of the corresponding 3D applications are displayed in the at least one canvas created in the rendering scene of the AR Launcher application of the head-mounted display device, the head-mounted display device can detect whether the first canvas in the at least one canvas is located at the center position of the rendering scene of the AR Launcher application. If the first canvas is located at the center position of the rendering scene of the AR Launcher application, it indicates that the display content of the first canvas is the screen of the first 3D application at the center position. If the first canvas is not located at the center position of the rendering scene of the AR Launcher application, the six-degree-of-freedom information (6DoF data) of the first 3D application needs to be recalculated through the step S2300.

[0076] After the above step S2200 is performed to detect whether the first canvas in the at least one canvas is located at the center position of the rendering scene, the following step is performed:

[0077] Step S2300, in the case that the first canvas is not located at the center position of the rendering scene, determining second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and assigning the first 3D application to make the first canvas display the picture of the center position of the first 3D application.

[0078] In the embodiment, in the case that the first canvas is not located at the center position of the rendering scene of the AR Launcher application, the head-mounted display device can determine the second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and assign the first 3D application to make the first canvas display the picture of the center position of the first 3D application.

[0079] The canvas information of the first canvas is the first included angle between the canvas center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device. It should be noted that since the rendering engine used by the head-mounted display device is the Unity engine, the Unity engine uses a left-handed coordinate system, and therefore the device coordinate system of the head-mounted display device is usually a left-handed coordinate system, and the vertical positive direction is the positive direction of the X axis.

[0080] The first degree of freedom information is the 6DoF data.

[0081] In an optional embodiment, the step S2300 of determining the second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device can further include: obtaining the first product between the inverse of the rotation quaternion of the first included angle and the rotation quaternion of the first degree of freedom information as the second degree of freedom information.

[0082] Taking the first canvas as canvas 1 for example, the head-mounted display device detects that the canvas 1 is not located at the center position of the rendering scene of the AR Launcher application. At this time, in order to make the 3D application 1 have effective display content in the rendering scene, the head-mounted display device can calculate the included angle 1 between the canvas center position of the canvas 1 and the positive direction of the Z axis in the rendering scene of the AR Launcher application, and when assigning the 6DoF data to the 3D application 1 in the OpenXR Runtime rendering process, the rotation quaternion of the 6DoF data is multiplied by the inverse of the rotation quaternion of the included angle 1 every frame, and is assigned to the 3D application 1. At this time, the display picture of the canvas 1 is the picture of the center position of the 3D application 1, and the 6DoF display effect is normal.

[0083] Taking the first canvas as canvas 2 as an example, the head-mounted display device detects that the canvas 2 is not located at the center position of the rendering scene of the AR Launcher application. At this time, in order to make the 3D application 2 have effective display content in the rendering scene, the included angle 2 between the canvas center position of the canvas 2 and the positive direction of the Z axis in the rendering scene of the AR Launcher application can be calculated, and when the 6DoF data of the 3D application 2 is allocated in the OpenXR Runtime rendering process, the rotation quaternion of the 6DoF data is multiplied by the inverse rotation quaternion of the included angle 2 every frame, and is allocated to the 3D application 2. At this time, the display picture of the canvas 2 is the picture of the center position of the 3D application 2, and the 6DoF display effect is normal.

[0084] According to the embodiments of the present disclosure, in the case that the pictures of the corresponding 3D applications are respectively displayed in the at least one canvas of the creative scene in the rendering scene of the 3D desktop application, if it is detected that the first canvas in the at least one canvas is not located at the center position of the rendering scene of the 3D desktop application, the second degree of freedom information can be determined according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and is allocated to the first 3D application, so that the display picture of the first canvas is the picture of the center position of the corresponding first 3D application.

[0085] In one embodiment, after the above step S2300, in the case that the first canvas is not located at the center position of the rendering scene, the second degree of freedom information is determined according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and is allocated to the first 3D application, the control method of the embodiments of the present disclosure further includes the following steps S3100-S3400:

[0086] Step S3100, obtaining a second product between the inverse rotation quaternion of the first included angle and the rotation quaternion of the third degree of freedom information of the control device as the fourth degree of freedom information and allocating it to the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is consistent with the direction of the second virtual identifier corresponding to the 3D desktop application.

[0087] Wherein, the control device, for example but not limited to, includes a handle, a mouse, a mobile phone, and the like, and the embodiments of the present disclosure do not limit the same.

[0088] Wherein, the third degree of freedom information is three degrees of freedom information (abbreviated as 3DoF data), which can be understood as the rotation degree of freedom of the control device. Correspondingly, the fourth degree of freedom information is also three degrees of freedom information.

[0089] Wherein, the first virtual identifier and the second virtual identifier can be a virtual ray, which can be a straight line or a curve, and the embodiments of the present disclosure do not limit the same.

[0090] In the embodiment, after the head-mounted display device determines the second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device and assigns the first 3D application, the head-mounted display device can multiply the rotation quaternion of the 3DoF data of the control device by the inverse of the first included angle, as the fourth degree of freedom information, and assign the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is consistent with the direction of the second virtual identifier corresponding to the 3D desktop application.

[0091] It should be noted that for each 3D application that has been started in the rendering scene of the AR Launcher application, there is a corresponding virtual identifier.

[0092] Step S3200, in the case of assigning the fourth degree of freedom information to the first virtual identifier corresponding to the first 3D application, detecting whether a touch event sent by the control device is received.

[0093] The touch event can be a click event.

[0094] In the embodiment, in the case of assigning the recalculated 3DoF data to the first virtual identifier corresponding to the first 3D application, the head-mounted display device can further detect whether a touch event sent by the control device is received.

[0095] Step S3300, in the case of detecting the touch event sent by the control device, obtaining the collision position of the second virtual identifier and the rendering scene.

[0096] In the embodiment, in the case of detecting the touch event sent by the control device, the head-mounted display device can further detect the collision position of the second virtual identifier and the rendering scene of the AR Launcher application through the second virtual identifier corresponding to the AR Launcher application.

[0097] Step S3400, in the case of the collision position being located in the first canvas, controlling the first 3D application to respond to the touch event through the first virtual identifier.

[0098] In the embodiment, in the case of detecting that the collision position is located in the first canvas, the first 3D application can be controlled to respond to the touch event through the first virtual identifier corresponding to the first 3D application.

[0099] Continuing the above example, the head-mounted display device can multiply the 3DoF data of the control device by the inverse of the rotation quaternion of the above-mentioned included angle 1 for the virtual ray 1 corresponding to the 3D application 1, and assign the virtual ray 1, so that the direction of the virtual ray 1 and the direction of the virtual ray 2 corresponding to the AR Launcher application remain consistent. In addition, the head-mounted display device can multiply the 3DoF data of the control device by the inverse of the rotation quaternion of the above-mentioned included angle 2 for the virtual ray 3 corresponding to the 3D application 2, and assign the virtual ray 3, so that the direction of the virtual ray 3 and the direction of the virtual ray 2 corresponding to the AR Launcher application remain consistent.

[0100] In the case where the head-mounted display device receives the touch event sent by the control device, the head-mounted display device detects the collision position of the virtual ray 2 with the rendering scene of the AR Launcher application, and if the collision position is located in the canvas 1, the head-mounted display device can control the 3D application 1 to respond to the touch event through the virtual ray 1. If the collision position is located in the canvas 2, the head-mounted display device can control the 3D application 2 to respond to the touch event through the virtual ray 3. Otherwise, the head-mounted display device can control the AR Launcher application to respond to the touch event through the virtual ray 2.

[0101] Through the embodiment, the head-mounted display device can ensure correct control of the control device by recalculating the 3DoF data of the control device.

[0102] In one embodiment, the collision position can be a set position, which can be understood as a position that needs to be combined with virtual and real objects, such as but not limited to a position for image recognition and a position for plane detection.

[0103] The above step S3400 can further include, in the case where the collision position is located in the first canvas, reversing rotating the coordinate information of the collision position by the first included angle with the center of the world coordinate system as the origin, and then controlling the first 3D application to respond to the touch event through the first virtual identifier.

[0104] In the embodiment, in the case where the collision position is located in the first canvas, the head-mounted display device can reverse rotate the coordinate information of the collision position by the first included angle with the center of the world coordinate system as the origin, and then control the first 3D application to respond to the touch event through the first virtual identifier, so that the virtual combination effect is not affected.

[0105] Through the embodiment, the head-mounted display device recalculates the coordinates of the touch position that needs to be combined with virtual and real objects, and thus correct virtual and real combination is achieved.

[0106] In one embodiment, the control method of the present disclosure further includes steps S4100-S4300.

[0107] Step S4100, receiving an adjustment input implemented on the first canvas.

[0108] Step S4200, updating the position of the first canvas in the rendering scene in response to the adjustment input.

[0109] Step S4300, after updating the position of the first canvas in the rendering scene, re-executing the step of detecting whether the first canvas in the at least one canvas is located at the center position of the rendering scene.

[0110] Through the present embodiment, after the user adjusts the position of the first canvas, the first included angle between the canvas center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device is automatically adjusted, at this time, the AR Launcher application transmits the adjusted first included angle to the OpenXR Runtime rendering process to re-calculate the 6DoF data corresponding to the first 3D application.

[0111] <Device Embodiment>

[0112] FIG. 4 is a schematic diagram of a control device according to one embodiment. Referring to FIG. 4, the control device 400 includes a display module 410, a detection module 420, and a determination module 430.

[0113] The display module 410 is configured to display the screens of the corresponding 3D applications in the at least one canvas created in the rendering scene of the 3D desktop application respectively.

[0114] The detection module 420 is configured to detect whether a first canvas in the at least one canvas is located at a center position of the rendering scene, wherein the first canvas displays a screen of a first 3D application.

[0115] The determination module 430 is configured to, in the case where the first canvas is not located at the center position of the rendering scene, determine second degree-of-freedom information according to canvas information of the first canvas and first degree-of-freedom information of a head-mounted display device and assign the second degree-of-freedom information to the first 3D application, so that the first canvas displays a screen of a center position of the first 3D application.

[0116] In one embodiment, the canvas information of the first canvas is a first included angle between a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, and the determination module 430 is specifically configured to obtain a first product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the first degree-of-freedom information as the second degree-of-freedom information.

[0117] In one embodiment, the canvas information of the first canvas is a first included angle of a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, and the apparatus 400 further comprises an obtaining module (not shown in the figure).

[0118] The obtaining module is configured to obtain a second product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the third degree-of-freedom information of the control device, as fourth degree-of-freedom information, and distribute the fourth degree-of-freedom information to a first virtual mark corresponding to the first 3D application, so that the first virtual mark is consistent with a direction of a second virtual mark corresponding to the 3D desktop application.

[0119] In one embodiment, the apparatus 400 further comprises a control module (not shown in the figure).

[0120] The detection module 420 is further configured to, in a case where the fourth degree-of-freedom information is distributed to the first virtual mark corresponding to the first 3D application, detect whether a touch event sent by the control device is received.

[0121] The obtaining module is further configured to, in a case where the touch event sent by the control device is detected, obtain a collision position of the second virtual mark and the rendering scene.

[0122] The control module is configured to, in a case where the collision position is located in the first canvas, control the first 3D application to respond to the touch event through the first virtual mark.

[0123] In one embodiment, the collision position is a set position, and the control module is specifically configured to, in a case where the collision position is located in the first canvas, inversely rotate coordinate information of the collision position by the first included angle with the center of a world coordinate system as an origin, and then control the first 3D application to respond to the touch event through the first virtual mark.

[0124] In one embodiment, the apparatus 400 further comprises a receiving module and an updating module (neither shown in the figure).

[0125] The receiving module is configured to receive an adjustment input implemented on the first canvas.

[0126] The updating module is configured to, in response to the adjustment input, update a position of the first canvas in the rendering scene.

[0127] The detection module 420 is configured to, after the position of the first canvas in the rendering scene is updated, detect whether the first canvas in the at least one canvas is located at a center position of the rendering scene.

[0128] According to the embodiment of the present disclosure, in the case that the at least one canvas of the creative scene is respectively displayed with the picture of the corresponding 3D application in the rendering scene of the 3D desktop application, if it is detected that the first canvas in the at least one canvas is not located at the center position of the rendering scene of the 3D desktop application, the second degree of freedom information is allocated to the first 3D application according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, so that the display picture of the first canvas is the picture of the center position of the corresponding first 3D application.

[0129] <Device Embodiment>

[0130] FIG. 5 is a schematic diagram of a hardware structure of a head-mounted display device according to an embodiment. As shown in FIG. 5, the head-mounted display device 500 includes a processor 510 and a memory 520.

[0131] The memory 520 can be used to store executable computer instructions.

[0132] The processor 510 can be used to control the execution of the control method according to the executable computer instructions.

[0133] The head-mounted display device 500 can be the head-mounted display device 1000 as shown in FIG. 1.

[0134] In another embodiment, the head-mounted display device 500 can include the control device 400 described above.

[0135] In one embodiment, each module of the control device 400 described above can be implemented by the processor 510 running the computer instructions stored in the memory 520.

[0136] <Computer readable storage medium>

[0137] The embodiment of the present disclosure also provides a computer readable storage medium having computer instructions stored thereon, wherein the computer instructions are run by a processor to execute the control method provided by the embodiment of the present disclosure.

[0138] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to cause a processor to implement various aspects of the present application.

[0139] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0140] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0141] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0142] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0143] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0144] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0145] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0146] The above-mentioned embodiments of the application are only intended to describe the application, not to limit the application.

[0147] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, television, or network equipment, etc.) executes the method of various embodiments of the present application.

[0148] The above-mentioned is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, all in the inventive concept of the present application, using the content of the present application specification and drawings, or directly / indirectly applied in other related technical field equivalent structure transformation is included in the patent protection scope of the present application.

Claims

1. A control method characterized by, The method comprises: displaying the pictures of the corresponding 3D applications respectively in at least one canvas created in a rendering scene of a 3D desktop application; detecting whether a first canvas in the at least one canvas is located at a central position of the rendering scene; wherein the first canvas displays a picture of a first 3D application; in the case that the first canvas is not located at the central position of the rendering scene, determining second degree of freedom information according to canvas information of the first canvas and first degree of freedom information of a head-mounted display device, and assigning the second degree of freedom information to the first 3D application, so that the first canvas displays a picture of a central position of the first 3D application.

2. The method of claim 1, wherein, the canvas information of the first canvas is a first included angle between a canvas central position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, the determining of the second degree of freedom information according to the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device comprises: obtaining a first product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the first degree of freedom information as the second degree of freedom information.

3. The method of claim 1, wherein, the canvas information of the first canvas is a first included angle between a canvas central position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, after the assigning of the second degree of freedom information to the first 3D application in the case that the first canvas is not located at the central position of the rendering scene, the method further comprises: obtaining a second product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of third degree of freedom information of a control device as fourth degree of freedom information and assigning the fourth degree of freedom information to a first virtual marker corresponding to the first 3D application, so that the first virtual marker is consistent with a direction of a second virtual marker corresponding to the 3D desktop application.

4. The method of claim 3, wherein, the method further comprises: in the case that the fourth degree of freedom information is assigned to the first virtual marker corresponding to the first 3D application, detecting whether a touch event sent by the control device is received; in the case that the touch event sent by the control device is detected, obtaining a collision position of the second virtual marker with the rendering scene; in the case that the collision position is located at the first canvas, controlling the first 3D application to respond to the touch event through the first virtual marker.

5. The method of claim 4, wherein, the collision position is a set position, the controlling of the first 3D application to respond to the touch event through the first virtual marker in the case that the collision position is located at the first canvas comprises: in the case that the collision position is located at the first canvas, inversely rotating coordinate information of the collision position by the first included angle with a center of a world coordinate system as an origin, and then controlling the first 3D application to respond to the touch event through the first virtual marker.

6. The method of claim 1, wherein, the method further comprises: receiving an adjustment input implemented on the first canvas; updating a position of the first canvas in the rendering scene in response to the adjustment input; After updating the position of the first canvas in the rendering scene, the step of detecting whether the first canvas in the at least one canvas is located at the center position of the rendering scene is performed again.

7. A control device characterized by comprising: The device comprises: a display module configured to display a picture of a corresponding 3D application in each of at least one canvas created in a rendering scene of a 3D desktop application; a detection module configured to detect whether a first canvas in the at least one canvas is located at a center position of the rendering scene, wherein the first canvas displays a picture of a first 3D application; a determination module configured to, in a case where the first canvas is not located at the center position of the rendering scene, determine second degree-of-freedom information according to canvas information of the first canvas and first degree-of-freedom information of a head-mounted display device and assign the second degree-of-freedom information to the first 3D application, so that the first canvas displays a picture of a center position of the first 3D application.

8. The apparatus of claim 7, wherein, The canvas information of the first canvas is a first included angle between a canvas center position of the first canvas and a vertical positive direction of a device coordinate system of the head-mounted display device, and the determination module is specifically configured to: obtain a first product between an inverse of a rotation quaternion of the first included angle and a rotation quaternion of the first degree-of-freedom information as the second degree-of-freedom information.

9. A head-mounted display device, comprising: The head-mounted display device comprises: a memory configured to store executable computer instructions; a processor configured to execute the control method according to any one of claims 1-6 according to control of the executable computer instructions.

10. A computer readable storage medium having computer instructions stored thereon, wherein the computer instructions are run by a processor to execute the control method according to any one of claims 1-6.

Images