Security boundary generation method, playing terminal and head-mounted display equipment

Abstract

The invention relates to the technical field of virtual reality, and provides a safety boundary generation method, a playing terminal and head-mounted display equipment. According to the method, a space coordinate system of a target space is used as a medium, and a second relative pose between a device coordinate system of a second head-mounted display device and the space coordinate system and a first relative pose between a device coordinate system of a first head-mounted display device and the space coordinate system are utilized. Therefore, conversion from the initial model of the second safety boundary under the device coordinate system of the second head-mounted display device to the target model of the first safety boundary under the device coordinate system of the first head-mounted display device is realized. Therefore, the first head-mounted display equipment can utilize the security boundary drawn by the second head-mounted display equipment in the target space to guarantee the security of immersive experience of a user, and the security boundary does not need to be independently drawn for each first head-mounted display equipment in the target space, so that the workload of drawing the security boundary is reduced; the drawing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology and provides a method for generating a safety boundary, a playback terminal, and a head-mounted display device. Background Art

[0002] In the streaming scenario, for multiple head-mounted display devices that are simultaneously in a larger physical space, the playback terminal returns the streaming images of the same 3D video to the corresponding head-mounted display devices based on the positions of each head-mounted display device in the physical space, thereby achieving highly interactive and immersive experience for multiple people.

[0003] When multiple users wearing head-mounted display devices immerse themselves in the same 3D video in a larger physical space, they cannot see the real environment of this physical space, and this physical space may contain some obstacles (such as pillars, pools, tables, etc.). Therefore, in order to avoid collisions between users, it is necessary to generate a safety boundary of the physical space for the head-mounted display device to ensure user safety.

[0004] Typically, safety boundaries are displayed based on the device's own coordinate system, and the coordinate systems of different head-mounted display devices are different. Therefore, current related technologies will draw a safety boundary for each head-mounted display device in a larger physical space. This not only increases the workload of drawing the safety boundary and reduces the efficiency of drawing the safety boundary, but also because the coordinate system may change every time the head-mounted display device is turned on or the screen is turned on after sleep, resulting in poor versatility of this drawing solution. Summary of the Invention

[0005] The embodiments of the present application provide a method for generating a safety boundary, a playback terminal, and a head-mounted display device, which are used to improve the efficiency and versatility of drawing the safety boundary.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a security boundary, which is applied to a playback terminal, comprising:

[0007] receiving a security boundary acquisition request sent by a first head-mounted display device in a target space, wherein the first head-mounted display device is any head-mounted display device in the target space;

[0008] In response to the safety boundary acquisition request, the general model is converted into the device coordinate system of the first head-mounted display device according to the first relative pose to generate a target model of the first safety boundary; wherein the general model is a 3D model in the spatial coordinate system of the target space, the general model is determined based on the initial model and the second relative pose, the initial model is a 3D model generated and uploaded by the second head-mounted display device in the target space according to the second safety boundary of the second head-mounted display device, the second relative pose is the relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system, and the first relative pose is the relative pose between the device coordinate system of the first head-mounted display device and the spatial coordinate system;

[0009] The target model of the first safety boundary is sent to the first head-mounted display device, so that the first head-mounted display device displays the target model.

[0010] The beneficial effects of the above technical solution are as follows: generally, the safety boundary is displayed based on the coordinate system of the device itself, and the coordinate systems of different devices are different. To enable the first head-mounted display device in the target space to also use the safety boundary drawn by the second head-mounted display device, the second head-mounted display device sends an initial model of the first safety boundary drawn in the target space based on its own device coordinate system to the playback terminal. In this way, the playback terminal can use the spatial coordinate system of the target space as an intermediary, combine the initial model of the first safety boundary with the second relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system, and obtain a general model of the safety boundary in the target space. Then, combining the first relative pose between the device coordinate system of the first head-mounted display device and the spatial coordinate system, the playback terminal obtains a target model of the second safety boundary in the device coordinate system of the first head-mounted display device and sends the target model to the first head-mounted display device, thereby achieving conversion of the 3D model of the safety boundary between the device coordinate system of the second head-mounted display device and the device coordinate system of the first head-mounted display device. In subsequent streaming scenarios, the first head-mounted display device can use the safety boundary drawn by the second head-mounted display device to ensure the safety of the user's immersive experience, eliminating the need to draw a safety boundary separately for each first head-mounted display device in the target space, thereby reducing the workload of drawing the safety boundary.

[0011] Optionally, before converting the universal model to the device coordinate system of the first head-mounted display device, the method further includes:

[0012] Receive an initial model of a second safety boundary drawn in the target space and the second relative pose sent by the second head-mounted display device, wherein the initial model is a 3D model of the second safety boundary in the device coordinate system of the second head-mounted display device, and the second relative pose is calculated based on the first pose of the second head-mounted display device in its own device coordinate system and the second pose in the space coordinate system.

[0013] The beneficial effects of the above technical solution are: the process of drawing the safety boundary is generally executed before streaming, and the real-time requirements are relatively low. Therefore, when the second head-mounted display device draws the first safety boundary in the target space based on its own device coordinate system, it can use its own computing resources to calculate the second relative posture between its own device coordinate system and the spatial coordinate system of the target space, and send it to the playback terminal, so that the playback terminal can subsequently obtain a general model of the second safety boundary through coordinate system conversion.

[0014] Optionally, before receiving the security boundary acquisition request sent by the first head-mounted display device in the target space, the method further includes:

[0015] receiving an initial model of a second safety boundary drawn in the target space and sent by the second head-mounted display device, where the initial model is a 3D model of the second safety boundary in a device coordinate system of the second head-mounted display device;

[0016] Receive the first pose in the device coordinate system of the second head-mounted display device sent by the second head-mounted display device, and combine it with the second pose of the second head-mounted display device in the spatial coordinate system determined by itself to obtain a second relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system.

[0017] The beneficial effect of the above technical solution is: due to the limitations of the volume and weight of the second head-mounted display device, the chip computing power of the second head-mounted display device is relatively low, while the computing power of the playback terminal is relatively strong. Therefore, the playback terminal calculates the second relative posture between the spatial coordinate system and the device coordinate system based on its positioning results of the second head-mounted display device in the spatial coordinate system and the positioning results of the second head-mounted display device in the device coordinate system, thereby improving the calculation efficiency of the second relative posture.

[0018] Optionally, after receiving the safety boundary acquisition request sent by the first head-mounted display device in the target space, and before converting the universal model into the device coordinate system of the first head-mounted display device according to the first relative pose, the method further includes:

[0019] According to the second relative posture, the initial model of the second safety boundary is converted into the spatial coordinate system to obtain a universal model of the second safety boundary.

[0020] The beneficial effect of the above technical solution is: after the playback terminal receives the safety boundary acquisition request sent by the first head-mounted display device, it first converts the initial model of the safety boundary into the spatial coordinate system according to the second relative posture to obtain a general model of the safety boundary, and then uses the spatial coordinate system as an intermediary, and then converts the general model of the safety boundary into the device coordinate system of the first head-mounted display device according to the first relative posture, thereby obtaining a target model of the first safety boundary suitable for the first head-mounted display device, reducing the workload of drawing the safety boundary for the first head-mounted display device.

[0021] Optionally, before receiving the security boundary acquisition request sent by the first head-mounted display device in the target space, the method further includes:

[0022] According to the second relative posture, the initial model of the second safety boundary is converted into the spatial coordinate system to obtain a universal model of the second safety boundary.

[0023] The beneficial effect of the above technical solution is that before receiving the safety boundary acquisition request sent by the first head-mounted display device, the playback terminal converts the initial model of the second safety boundary into the spatial coordinate system according to the second relative posture in advance to obtain the general model of the second safety boundary. In this way, after receiving the safety boundary acquisition request sent by the first head-mounted display device, the general model can be directly converted into the coordinate system of the first head-mounted display device according to the first relative posture, thereby reducing the conversion process of the 3D model of the safety boundary and improving the efficiency of sending the target model of the safety boundary.

[0024] Optionally, converting the initial model of the second safety margin into the spatial coordinate system according to the second relative posture to obtain a general model of the second safety margin includes:

[0025] Multiplying the initial posture angles of multiple boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture angle in the second relative posture to obtain intermediate posture angles of the multiple boundary points in the space coordinate system;

[0026] Multiplying the initial positions of multiple boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture corner point, and then adding the initial positions to the first relative position in the second relative posture to obtain the middle positions of the multiple boundary points in the spatial coordinate system;

[0027] A general model of the second safety boundary in the space coordinate system is obtained according to the intermediate posture angles and intermediate positions of the multiple boundary points in the space coordinate system.

[0028] The beneficial effect of the above technical solution is: in 6Dof positioning, the posture generally includes 3-dimensional position and 3-dimensional attitude angle. Therefore, when the initial model of the second safety boundary is converted between the device coordinate system and the spatial coordinate system of the second head-mounted display device, the conversion is divided into two parts: position and attitude angle. By converting the initial model to the spatial coordinate system, the subsequent safety boundary drawing work of the first head-mounted display device can be reduced, thereby improving the streaming efficiency.

[0029] Optionally, converting the universal model into a device coordinate system of the first head-mounted display device according to the first relative posture to generate a target model of the first safety boundary includes:

[0030] multiplying the intermediate pose angles of a plurality of boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative pose angle in the first relative pose to obtain target pose angles of the plurality of boundary points in the device coordinates of the first head-mounted display device;

[0031] Multiplying the middle positions of the plurality of boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative posture corner point, and then adding the middle positions to the second relative positions in the first relative posture to obtain target positions of the plurality of boundary points in the device coordinate system of the first head-mounted display device;

[0032] A target model of a first safety boundary in the device coordinate system of the first head-mounted display device is generated according to the target posture angles and target positions of the multiple boundary points in the device coordinate system of the first head-mounted display device.

[0033] The beneficial effect of the above technical solution is: in 6Dof positioning, the posture generally includes 3-dimensional position and 3-dimensional attitude angle. Therefore, when the general model of the second safety boundary is converted between the device coordinate system and the spatial coordinate system of the first head-mounted display device, the conversion is divided into two parts: position and attitude angle. By converting the general model to the spatial coordinate system, the drawing work of the first safety boundary of the first head-mounted display device can be reduced, and the streaming efficiency can be improved.

[0034] Optionally, before converting the universal model to the device coordinate system of the first head-mounted display device, the method further includes:

[0035] receiving the first relative posture sent by the first head-mounted display device, where the first relative posture is calculated based on the third posture of the first head-mounted display device in its own device coordinate system and the fourth posture in the space coordinate system;

[0036] The beneficial effect of the above technical solution is: the first head-mounted display device uses its own computing resources to calculate the first relative posture between its own device coordinate system and the spatial coordinate system of the target space, and sends it to the playback terminal so that the subsequent playback terminal can use it when performing coordinate system conversion between the spatial coordinate system and the device coordinate system of the first head-mounted display device for the 3D model of the safety boundary.

[0037] Optionally, before converting the universal model to the device coordinate system of the first head-mounted display device, the method further includes:

[0038] Receive the third posture in the device coordinate system of the first head-mounted display device sent by the first head-mounted display device, and combine it with the fourth posture of the first head-mounted display device in the spatial coordinate system determined by itself to obtain a first relative posture between the device coordinate system of the first head-mounted display device and the spatial coordinate system.

[0039] The beneficial effect of the above technical solution is that: the streaming device and the first head-mounted display device will both locate the first head-mounted display device, and the computing power of the playback terminal is higher than that of the first head-mounted display device. Therefore, the first head-mounted display device sends its third posture in the device coordinate system to the playback terminal, and the playback terminal combines the fourth posture of the first head-mounted display device in the spatial coordinate system determined by itself to calculate the first relative posture between the spatial coordinate system and the device coordinate system, thereby improving the calculation efficiency of the first relative posture.

[0040] In a second aspect, an embodiment of the present application provides a method for generating a safety boundary, which is applied to a second head-mounted display device, and the method includes:

[0041] Start the security boundary drawing application and establish a connection with the playback terminal;

[0042] receiving a second posture of the second head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combining the second head-mounted display device with the first posture in its own device coordinate system to obtain a second relative posture between the device coordinate system and the spatial coordinate system;

[0043] Turning on the perspective function, drawing a second safety boundary applicable to the target space in the device coordinate system, and generating an initial model of the second safety boundary;

[0044] The initial model of the second safety boundary and the second relative posture are sent to the playback terminal, and the second relative posture is used to convert the initial model between the device coordinate system and the space coordinate system to generate a general model of the second safety boundary.

[0045] The beneficial effect of the above technical solution is that before streaming, the second head-mounted display device draws a second safety boundary in the target space based on its own device coordinate system and generates an initial model of the second safety boundary. Since the safety boundary is displayed based on the device's own coordinate system, and the coordinate systems of different devices are different, in order to enable any first head-mounted display device in the target space to also use the safety boundary drawn by the second head-mounted display device, the second head-mounted display device obtains a second relative pose between its own device coordinate system and the spatial coordinate system of the target space based on its second pose in the spatial coordinate system of the target space sent by the playback terminal and the first pose in its own device coordinate system. In this way, the initial model of the second safety boundary is converted to the spatial coordinate system through the second relative pose, and a universal model of the second safety boundary in the target space is obtained for subsequent use by the first head-mounted display device during streaming. This eliminates the need to draw the first safety boundary separately for each first head-mounted display device in the target space, reducing the workload of drawing the safety boundary.

[0046] On the other hand, the safe boundary drawing process is performed by the safe boundary drawing application. Since the safe boundary drawing application is installed on the second head-mounted display device and is independent of the 3D application, it is suitable for the immersive experience of all 3D applications in the target space. Changes to the target space will not affect the drawing process, and it is more versatile.

[0047] In a third aspect, an embodiment of the present application provides a method for generating a safety boundary, which is applied to a first head-mounted display device, and the method includes:

[0048] Start the streaming application and establish a connection with the playback terminal;

[0049] receiving a fourth posture of the first head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combining the fourth posture of the first head-mounted display device in its own device coordinate system to obtain a first relative posture between the device coordinate system and the spatial coordinate system;

[0050] According to the first relative posture, the general model is converted to the device coordinate system of the first head-mounted display device to generate a target model of the first safety boundary in the device coordinate system of the first head-mounted display device; wherein the general model is a 3D model in the spatial coordinate system of the target space, and the general model is determined based on the initial model and the second relative posture, the initial model is a 3D model generated and uploaded by the second head-mounted display device in the target space based on the second safety boundary of the second head-mounted display device, and the second relative posture is the relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system.

[0051] The beneficial effects of the above technical solution are: for multiple first head-mounted display devices in the target space, since the device coordinate system of each first head-mounted display device is different, in order for any first head-mounted display device to use the second safety boundary drawn by the second head-mounted display device, the first head-mounted display device calculates the first relative pose between its own device coordinate system and the spatial coordinate system based on the third pose in its own device coordinate system and its fourth pose in the spatial coordinate system of the target space, and uses the first relative pose to convert the general model of the second safety boundary into its own device coordinate system to obtain the target model of the first safety boundary that can be used by itself, thereby eliminating the need to draw the first safety boundary separately for each first head-mounted display device, reducing the workload of drawing the safety boundary and improving the drawing efficiency. Moreover, since the first safety boundary is converted from the second safety boundary, it can change as the device coordinate system of the first head-mounted display device changes, and is more versatile.

[0052] In a fourth aspect, an embodiment of the present application provides a playback terminal, comprising a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0053] The communication interface is used to communicate with a head-mounted display device;

[0054] The memory stores a computer program, and the processor executes any one of the security boundary generation methods in the first aspect according to the computer program.

[0055] In a fifth aspect, an embodiment of the present application provides a head-mounted display device, comprising a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus;

[0056] The communication interface is used to communicate with the playback terminal;

[0057] The display screen is used to display 3D applications;

[0058] The memory stores a computer program, and the processor executes the security boundary generation method of any one of the second and third aspects according to the computer program.

[0059] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned security boundary positioning methods are implemented.

[0060] The technical effects brought about by any implementation method in the fourth to sixth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 A schematic diagram of a large virtual reality space provided in an embodiment of the present application;

[0063] Figure 2A-2C A diagram of the streaming scenario architecture provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a flow chart of a method for generating a security boundary provided in an embodiment of the present application;

[0065] Figure 4 A schematic flow chart of another method for generating a security boundary provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the conversion process of the initial model of the second security boundary to the general model provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the conversion process from a general model to a target model of the second security boundary provided in an embodiment of the present application;

[0068] Figure 7 A schematic flow chart of another method for generating a security boundary provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of the interaction process between a head-mounted display device and a playback terminal provided in an embodiment of the present application;

[0070] Figure 9A schematic diagram of another interaction process between a head-mounted display device and a playback terminal provided in an embodiment of the present application;

[0071] Figure 10 A structural diagram of a playback terminal provided in an embodiment of the present application;

[0072] Figure 11 This is a structural diagram of the head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0074] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is presented based on one or several exemplary examples, it should be understood that each aspect of the disclosure can independently constitute a complete technical solution.

[0075] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0076] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0077] The following is an explanation of the terms involved in the embodiments of the present application.

[0078] A large virtual reality space is a large physical space that uses virtual reality technology to create a highly interactive and immersive experience for multiple people. It usually has at least one of the following characteristics:

[0079] High-precision positioning: Using technologies such as optical, laser, or inertial sensors, or external positioning technology, to accurately locate and capture the user's movements within the physical space;

[0080] Multi-user experience: There is sufficient space to support the free movement and interaction of multiple users wearing head-mounted display devices, achieving a high level of social interaction and mutual collaboration;

[0081] Immersion and realism: head-mounted displays provide a realistic audio-visual experience through virtual reality and surround sound technology.

[0082] ●Safety boundary: Used to avoid collisions during highly interactive and immersive experiences with multiple people, protecting user safety.

[0083] like Figure 1 As shown, this is a schematic diagram of a large virtual reality space provided in an embodiment of the present application. The large virtual reality space accommodates multiple users, and the walls and the ground contain markers for positioning. When multiple users wear head-mounted display devices to experience 3D videos transmitted by the playback terminal, if there is a risk of collision, a safety boundary can be displayed, opening a door to the same virtual world for multiple users.

[0084] It should be noted that Figure 1 The large virtual reality space shown is only an example and is not intended to be limiting of the embodiments of the present application. It may be an indoor space or an outdoor space having the above-mentioned characteristics.

[0085] The following summarizes the design concept of the embodiments of the present application in combination with application scenarios.

[0086] Currently, when multiple head-mounted display devices are streaming experiences within a large VR space, the administrator of the VR space draws a safety boundary for each head-mounted display device to prevent user collisions. However, this approach not only increases the workload and reduces the efficiency of drawing the safety boundary, but also reduces the universality of this drawing solution because the coordinate system may change every time the head-mounted display device is powered on or turned on after sleep.

[0087] In view of this, an embodiment of the present application provides a method for generating a safety boundary, which uses a playback terminal in a streaming scenario to achieve unified management of multiple head-mounted display devices. Before streaming, a head-mounted display device (denoted as the second head-mounted display device) draws a safety boundary (denoted as the second safety boundary) in the target space, and generates a 3D model of the second safety boundary in its own device coordinate system (denoted as the initial model), and then sends it to the playback terminal; the playback terminal converts the second safety boundary to the spatial coordinate system according to the relative pose (denoted as the second relative pose) between the device coordinate system of the second head-mounted display device and the spatial coordinate system of the target space, obtains a general model of the second safety boundary, and converts the second safety boundary to the device coordinate system of the first head-mounted display device according to the relative pose (denoted as the first relative pose) between the device coordinate system and the spatial coordinate system of any head-mounted display device (denoted as the first head-mounted display device) in the target space, obtains a target model of the first safety boundary, and sends the target model to the first head-mounted display device. By using the spatial coordinate system of the target space as an intermediary, the conversion of the safety boundary between the device coordinate systems of the second head-mounted display device and the first head-mounted display device is achieved, so that the first head-mounted display device can use the safety boundary drawn by the second head-mounted display device in the target space, reducing the workload of drawing the safety boundary and improving the efficiency of drawing the safety boundary. In addition, the safety boundary is not affected by changes in the device coordinate system of the first head-mounted display device, and has stronger versatility.

[0088] The following describes a method for generating a security boundary provided by an exemplary embodiment of the present application with reference to the accompanying drawings.

[0089] See also Figure 2A , which is a streaming scenario architecture diagram within a target space provided in an embodiment of the present application, including a second head-mounted display device 100, multiple first head-mounted display devices 200, and a playback terminal 300. The second head-mounted display device 100, the first head-mounted display device 200, and the playback terminal 300 are in the same local area network, and the second head-mounted display device 100 and the first head-mounted display device 200 are located in the same target space.

[0090] The second head-mounted display device 100 is installed with a safety boundary drawing application, which is used to draw a safety boundary within the target space based on the device coordinate system of the second head-mounted display device 100 and generate a 3D model of the safety boundary and upload it to the playback terminal 300. The playback terminal 300 is installed with a positioning application, a central control application, a 3D application, and a first streaming application. The positioning application is used to locate the second head-mounted display device 100 and the first head-mounted display device 200 within the target space, and the positioning results are used to convert the 3D model of the safety boundary between different coordinate systems. The central control application is used to distribute the 3D model of the safety boundary drawn by the second head-mounted display device 100. The 3D application is used to transmit 3D video to the first head-mounted display device 200. The first streaming application is used to communicate with the second streaming application of the first head-mounted display device 200. The first head-mounted display device 200 is installed with a second streaming application, which uses the 3D model finally converted by the playback terminal 300 to protect user safety.

[0091] See also Figure 2B , which is another streaming scenario architecture diagram within a target space provided by an embodiment of the present application, including a second head-mounted display device 100, multiple first head-mounted display devices (respectively denoted as 201-202), and multiple playback terminals (respectively denoted as 301-303). The second head-mounted display device 100, the first head-mounted display devices 201-202, and the playback terminals 301-303 are in the same local area network, and the second head-mounted display device 100 and the first head-mounted display devices 201-202 are located in the same target space. The playback terminal 302 corresponds to the first head-mounted display device 201, and the playback terminal 303 corresponds to the first head-mounted display device 202.

[0092] The second head-mounted display device 100 is equipped with a safety boundary drawing application, which is used to draw a safety boundary within the target space based on the device coordinate system of the second head-mounted display device 100, and generate a 3D model of the safety boundary and upload it to the playback terminal 301. The playback terminal 301 is equipped with a positioning application and a central control application. The positioning application is used to locate the second head-mounted display device 100 and the first head-mounted display devices 201-202 within the target space. The positioning results are used to realize the conversion of the 3D model of the safety boundary between different coordinate systems. The central control application is used to distribute the 3D model of the safety boundary drawn by the second head-mounted display device 100. The playback terminal 302 is equipped with a 3D application and a first streaming application. The 3D application is used to transmit 3D video to the first head-mounted display device 201, and the first streaming application is used to communicate with the second streaming application of the first head-mounted display device 201. The playback terminal 303 is installed with a 3D application and a first streaming application. The 3D application is used to transmit 3D video to the first head-mounted display device 202, and the first streaming application is used to communicate with the second streaming application of the first head-mounted display device 202. The first head-mounted display devices 201-202 are each installed with a second streaming application that uses the 3D model finally converted by the playback terminal 300 to protect user safety.

[0093] See also Figure 2C , which is another streaming scenario architecture diagram within a target space provided in an embodiment of the present application, including a second head-mounted display device 100, multiple first head-mounted display devices (respectively denoted as 201-202), and multiple playback terminals (respectively denoted as 301-302). The second head-mounted display device 100, the first head-mounted display devices 201-202, and the multiple playback terminals 301-302 are in the same local area network, and the second head-mounted display device 100 and the first head-mounted display devices 201-202 are located in the same target space. The playback terminal 301 corresponds to the first head-mounted display device 201, and the playback terminal 203 corresponds to the first head-mounted display device 202.

[0094] The second head-mounted display device 100 is installed with a safety boundary drawing application, which is used to draw a safety boundary within the target space based on the device coordinate system of the second head-mounted display device 100, and generate a 3D model of the safety boundary and upload it to the playback terminal 301. The playback terminal 301 is installed with a positioning application, a central control application, a 3D application, and a first streaming application. The positioning application is used to locate the second head-mounted display device 100 and the first head-mounted display devices 201-202 within the target space. The positioning results are used to realize the conversion of the 3D model of the safety boundary between different coordinate systems. The central control application is used to distribute the 3D model of the safety boundary drawn by the second head-mounted display device 100. The 3D application is used to transmit 3D video to the first head-mounted display device 201. The first streaming application is used to communicate with the second streaming application of the first head-mounted display device 201. The playback terminal 302 is installed with a 3D application and a first streaming application. The 3D application is used to transmit 3D video to the first head-mounted display device 202, and the first streaming application is used to communicate with the second streaming application of the first head-mounted display device 202. The first head-mounted display devices 201-202 are each installed with a second streaming application that uses the 3D model finally converted by the playback terminal 300 to protect user safety.

[0095] It should be noted that Figure 2A-2C This is only an example of a streaming scenario architecture and is not a restrictive requirement for streaming scenarios in the embodiments of this application.

[0096] Combined with the above streaming scenario, see Figure 3 This is a flow chart of a method for generating a safety boundary provided in an embodiment of the present application. The process is performed by a second head-mounted display device and mainly includes the following steps:

[0097] S301: Start a security boundary drawing application and establish a connection with a playback terminal.

[0098] Typically, safety boundaries are drawn within the 3D application appropriate for the target space. This means the 3D application monitors the user's position in real time during display. When the user's movement distance meets certain conditions, a 3D graphic safety boundary is proactively displayed to prevent collisions. However, since the safety boundary is drawn within the 3D application, it needs to be redrawn when switching between applications, which is quite cumbersome.

[0099] To address this issue, a second head-mounted display device launches a dedicated safe boundary drawing application for the safe boundary drawing process and establishes a connection with the playback terminal. This safe boundary drawing application is installed on the second head-mounted display device and runs independently of the playback terminal. Therefore, it provides an immersive experience for all 3D applications within the target space. It also eliminates the need to redraw the safe boundary when switching between 3D applications within the target space, and changes to the target space do not affect the drawing process, making it more versatile.

[0100] S302: Receive a second posture of the second head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combine the first posture of the second head-mounted display device in its own device coordinate system to obtain a second relative posture between the device coordinate system and the spatial coordinate system.

[0101] Among them, the first pose and the second pose are poses at the same moment. Each pose includes position and attitude angle. Therefore, the second relative pose can be calculated into two parts: position and attitude angle. The specific acquisition process is as follows:

[0102] 1. Obtain a first relative attitude angle between a device coordinate system of the second head-mounted display device and a space coordinate system of the target space based on a first attitude angle in the first attitude and a second attitude angle in the second attitude.

[0103] Assume that the first pose pose_local0 is (pos local0 ,rot local0 ), the second pose pos world0 (pos world0 ,rot world0 ), where pos local0 Indicates the first position, rot local0 Indicates the first posture angle, pos world0 Indicates the second position, rot world0 Indicates the second attitude angle, because it is necessary to calculate the rot local0 to rot world0 The rotation difference, so first take rot local0 The inverse of , and then use rot world0 dot product rot local0 The inverse of rot, that is, first rotate in the opposite direction local0 Rotate again world0 Get the first relative attitude angle rot diff0 , the formula is:

[0104] rot diff0 =rot world0 *Inverse(rot local0 ) Formula 1

[0105] 2. According to the second posture angle, the second position in the second posture is restored to the original position before the posture angle conversion, and the first relative position between the device coordinate system and the space coordinate system is obtained by combining the first posture angle and the first position in the first posture.

[0106] Because of the presence of the attitude angle, the rotation state needs to be considered in the process of calculating the first relative position. In the normal geometric operation of rotation and displacement, rotation is performed first and then displacement. Therefore, reverse rotation and reverse displacement are required to restore to the original position pos tmp0 , the formula is:

[0107] POS tmp0 =Inverse(rot local0 )*(-pos local0 ) Formula 2

[0108] In calculation from pos local0 to POS world0 When the displacement difference is pos, the reverse operation cancels out the local0 The role of returning to the original position, the next thing to consider is pos world0 The role of rot is to execute it first. world0 Rotation, then pos tmp0 The displacement, then superimpose pos world0 The displacement of the first relative position pos is obtained. diff0 , the formula is:

[0109] POS diff0 =rot world0 *pos tmp0 +pos world0 Formula 3

[0110] 3. Obtain a second relative posture according to the first relative posture angle and the first relative position.

[0111] It should be noted that the target space has the characteristics of the aforementioned large virtual reality space.

[0112] In some embodiments, the process of obtaining the second posture of the second head-mounted display device in the spatial coordinate system of the target space is as follows: the second head-mounted display device captures a first environment image in the target space and sends the first environment image to the playback terminal. The playback terminal performs feature matching on the first environment image and the historical environment image in the map based on a pre-stored map of the target space, and uses the posture of the matched historical environment image as the second posture of the second head-mounted display device in the target space.

[0113] It should be noted that the map of the target space can be pre-built by the second head-mounted display device. The map includes historical environmental images of the target space and the corresponding position and posture of each historical environmental image. The spatial coordinate system of the target space is the coordinate system used when building the map. The map construction process is relatively mature and is not the focus of the embodiments of this application, so it will not be described in detail here.

[0114] S303: Turn on the perspective function, draw a second safety boundary applicable to the target space in the device coordinate system, and generate an initial model of the second safety boundary.

[0115] After turning on the perspective function, the administrator wearing the second head-mounted display device can see the real environment of the target space and draw a second safety boundary based on the real environment seen. During the drawing process, the second head-mounted display device will store multiple boundary points on the second safety boundary at regular intervals (such as thousands of boundary points). The coordinates of these boundary points are the coordinates of the device coordinate system of the second head-mounted display device. By giving these boundary points multiple height values, a 3D model of the second safety boundary is obtained and an .obj file is generated. At this time, the 3D model of the second safety boundary is located in the device coordinate system of the second head-mounted display device and is recorded as the initial model.

[0116] S304: Sending the initial model of the second safety boundary and the second relative posture to the playback terminal, where the second relative posture is used to convert the initial model between the device coordinate system and the space coordinate system to generate a universal model of the second safety boundary.

[0117] The second head-mounted display device generates an initial model of the second safety boundary and a second relative pose through a safety boundary drawing application, and transmits this to the playback terminal. In order for multiple users wearing the first head-mounted display device to experience the 3D application running on the playback terminal within the target space, the initial model can be converted from the device coordinate system of the second head-mounted display device to the spatial coordinate system using the second relative pose, thereby obtaining a universal model of the second safety boundary in the spatial coordinate system.

[0118] Before streaming, the second head-mounted display device draws a second safety boundary in the target space based on its own device coordinate system and generates an initial model of the second safety boundary. Since the safety boundary is displayed based on the device's own coordinate system, and the coordinate systems of different devices are different, in order to enable any first head-mounted display device in the target space to also use the safety boundary drawn by the second head-mounted display device, the second head-mounted display device obtains a second relative pose between its own device coordinate system and the spatial coordinate system of the target space based on its second pose in the spatial coordinate system of the target space sent by the playback terminal, combined with the first pose in its own device coordinate system. In this way, the initial model of the second safety boundary is converted to the spatial coordinate system through the second relative pose, and a universal model of the second safety boundary in the target space is obtained for subsequent use by the first head-mounted display device during streaming. This eliminates the need to draw the first safety boundary separately for each first head-mounted display device in the target space, reducing the workload of drawing the safety boundary. On the other hand, the safe boundary drawing process is performed by the safe boundary drawing application. Since the safe boundary drawing application is installed on the second head-mounted display device and is independent of the 3D application, it is suitable for the immersive experience of all 3D applications in the target space. Changes to the target space will not affect the drawing process, and it is more versatile.

[0119] In some embodiments, after the playback terminal receives the initial model and the second relative posture of the second safety boundary, when any user in the target space wears the first head-mounted display device to experience the 3D application running on the playback terminal, since the playback terminal and the first head-mounted display device are located in the same local area network, once the first head-mounted display device is turned on, the central control application on the playback terminal can send the initial model and the second relative posture of the second safety boundary to the first head-mounted display device for use.

[0120] See also Figure 4 , which is a flow chart of a method for generating a safety boundary according to an embodiment of the present application. The process is executed by any first head-mounted display device in the target space and mainly includes the following steps:

[0121] S401: Start the streaming application and establish a connection with the playback terminal.

[0122] A first streaming application is installed on the playback terminal, and a second streaming application is installed on the first head-mounted display device. During the streaming process, the first head-mounted display device and the playback terminal establish a connection by opening the streaming application.

[0123] S402: Receive the fourth posture of the first head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combine it with the third posture of the first head-mounted display device in its own device coordinate system to obtain a first relative posture between the device coordinate system and the spatial coordinate system.

[0124] Among them, the third and fourth poses are poses at the same moment. Each pose includes position and attitude angle. Therefore, the first relative pose can be calculated into two parts: position and attitude angle. The specific acquisition process is as follows:

[0125] 1. Obtain a second relative posture angle between a device coordinate system of the first head-mounted display device and a space coordinate system of the target space based on a third posture angle in the third posture and a fourth posture angle in the fourth posture.

[0126] Assume that the third pose pose_local1 is (pos local1 ,rot local1 ), the fourth pose pos world1 (pos world1 ,rot world1 ), where pos local1 Indicates the third position, rot local1 Indicates the third posture angle, pos world1 Indicates the fourth position, rot world1 Indicates the fourth attitude angle, because it is necessary to calculate the rot world1 to rot local1 The rotation difference, so first take rot world1 The inverse of , and then use rot local1 dot product rot world1 The inverse of rot, that is, first rotate in the opposite direction world1 Rotate again local1 Get the first relative attitude angle rot diff1 , the formula is:

[0127] rot diff1 =rot local1 *Inverse(rot world1 ) Formula 4

[0128] 2. According to the fourth posture angle, the fourth position in the fourth posture is restored to the original position before the posture angle conversion, and the second relative position between the device coordinate system and the space coordinate system is obtained by combining the third posture angle and the third position in the third posture.

[0129] Because of the presence of the attitude angle, the rotation state needs to be considered in the process of calculating the second relative position. In the normal geometric operation of rotation and displacement, rotation is performed first and then displacement. Therefore, reverse rotation and reverse displacement are required to restore to the original position pos tmp1 , the formula is:

[0130] POS tmp1 =Inverse(rot world1)*(-pos world1 ) Formula 5

[0131] In calculation from pos world1 to POS local1 When the displacement difference is pos, the reverse operation cancels out the local1 The role of returning to the original position, the next thing to consider is pos local1 The role of rot is to execute it first. local1 Rotation, then pos tmp1 The displacement, then superimpose pos local1 The displacement of the first relative position pos is obtained. diff1 , the formula is:

[0132] POS diff1 =rot local1 *pos tmp1 +pos local1 Formula 6

[0133] 3. Obtain a first relative posture according to the second relative posture angle and the second relative position.

[0134] Optionally, the embodiment of the present application does not impose any restrictive requirements on the calculation process of the first relative position. For example, the second relative position can be obtained by inverting the first relative position.

[0135] In some embodiments, the process of obtaining the fourth posture of the first head-mounted display device in the spatial coordinate system of the target space is as follows: the first head-mounted display device captures a second environment image in the target space and sends the second environment image to the playback terminal. The playback terminal performs feature matching on the second environment image and the historical environment image in the map based on a pre-stored map of the target space, and uses the posture of the matched historical environment image as the fourth posture of the first head-mounted display device in the target space.

[0136] S403: According to the first relative posture, the universal model is converted into a device coordinate system of the first head-mounted display device, and a target model of a first safety boundary is generated in the device coordinate system of the first head-mounted display device.

[0137] In some embodiments, after the first head-mounted display device starts streaming, the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device is converted to the spatial coordinate system according to the second relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system of the target space, thereby obtaining a universal model of the second safety boundary. Then, according to the first relative posture between its own device coordinate system and the spatial coordinate system of the target space, the universal model of the second safety boundary in the spatial coordinate system is converted to its own device coordinate system to obtain the target model of the first safety boundary.

[0138] In some embodiments, after receiving the second relative posture and the initial model of the second safety boundary sent by the playback terminal, the first head-mounted display device can convert the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device to the spatial coordinate system in advance based on the second relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system of the target space, thereby obtaining a universal model of the second safety boundary. In this way, after starting the streaming, the universal model of the second safety boundary in the spatial coordinate system is converted to the device coordinate system of the device according to the first relative posture between the device coordinate system of the device and the spatial coordinate system of the target space, thereby obtaining the target model of the first safety boundary. By converting to the universal model in advance before streaming, the conversion efficiency of the 3D model of the safety boundary is improved, thereby improving the streaming efficiency.

[0139] When the second relative posture is obtained, the conversion from the device coordinate system of the second head-mounted display device to the space coordinate system of the target space can be achieved.

[0140] Since each relative posture includes relative position and relative attitude angle, the process of converting the initial model to the general model can be divided into two parts: relative position and relative attitude angle. For the specific process, see Figure 5 , mainly includes the following steps:

[0141] S4031: Multiply the initial posture angles of multiple boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture angle in the second relative posture to obtain the intermediate posture angles of the multiple boundary points in the spatial coordinate system.

[0142] Specifically, the conversion formula of attitude angle is:

[0143] boundary_rot world = rot diff0 * rot point Formula 7

[0144] Among them, rot pointboundary_rot is the initial pose angle of the boundary point in the device coordinate system of the second head-mounted display device. world is the intermediate attitude angle of the boundary point in the spatial coordinate system of the target space.

[0145] S4032: Multiply the initial positions of multiple boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture corner point, and then add them to the first relative position in the second relative posture to obtain the middle positions of the multiple boundary points in the spatial coordinate system.

[0146] Specifically, the position conversion formula is:

[0147] boundary_pos world = rot diff0 *pos point +pos diff0 Formula 8

[0148] Among them, pos point boundary_rot is the initial position of the boundary point in the device coordinate system of the second head-mounted display device, world It is the middle position of the boundary point in the spatial coordinate system of the target space.

[0149] S4033: Obtain a general model of the second safety boundary in the spatial coordinate system based on the intermediate posture angles and intermediate positions of the plurality of boundary points in the spatial coordinate system.

[0150] After obtaining the intermediate attitude angles and intermediate positions of multiple boundary points in the spatial coordinate system, a general model of the second safety boundary in the spatial coordinate system can be obtained by assigning multiple height values to these boundary points.

[0151] Furthermore, when the first relative posture is obtained, the conversion of the space coordinate system of the target space to the device coordinate system of the second head-mounted display device can be realized. The process of converting the general model to the target model is as follows: Figure 6 As shown, it mainly includes the following steps:

[0152] S4034: Multiply the intermediate posture angles of multiple boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative posture angle in the first relative posture to obtain the target posture angles of the multiple boundary points in the device coordinates of the first head-mounted display device.

[0153] Specifically, the conversion formula of attitude angle is:

[0154] boundary_rot local = rot diff1* boundary_rot world Formula 9

[0155] Among them, boundary_rot local is the target pose angle of the boundary point in the device coordinate system of the first head-mounted display device.

[0156] S4035: Multiply the middle positions of the multiple boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative posture corner point, and then add them to the second relative position in the first relative posture to obtain the target positions of the multiple boundary points in the device coordinate system of the first head-mounted display device.

[0157] Specifically, the position conversion formula is:

[0158] boundary_pos local = rot diff1 *boundary_pos world +pos diff1 Formula 10

[0159] Among them, boundary_pos local is the target position of the boundary point in the device coordinate system of the first head-mounted display device.

[0160] S4036: Generate a target model of a first safety boundary in the device coordinate system of the first head-mounted display device according to the target posture angles and target positions of the plurality of boundary points in the device coordinate system of the first head-mounted display device.

[0161] Based on the target attitude angles and target positions of multiple boundary points in the device coordinate system of the first head-mounted display device, a first safety boundary in the device coordinate system of the first head-mounted display device can be obtained. By assigning multiple height values to these boundary points, a target model of the first safety boundary can be obtained.

[0162] In the streaming scenario, for multiple first head-mounted display devices in the target space, since the device coordinate system of each first head-mounted display device is different, in order for any first head-mounted display device to use the second safety boundary drawn by the second head-mounted display device, the first head-mounted display device calculates the first relative pose between its own device coordinate system and the spatial coordinate system based on the third pose in its own device coordinate system and its fourth pose in the spatial coordinate system of the target space, and uses the first relative pose to convert the universal model of the second safety boundary into its own device coordinate system to obtain the target model of the first safety boundary that can be used by itself. Therefore, there is no need to draw the first safety boundary separately for each first head-mounted display device, which reduces the workload of drawing the safety boundary and improves drawing efficiency. Moreover, since the first safety boundary is converted from the second safety boundary, it can change as the device coordinate system of the first head-mounted display device changes, and is more universal.

[0163] In some embodiments, considering that the processing performance of the head-mounted display device is affected by factors such as the device size and weight, the computing power of the playback terminal is usually higher than that of the head-mounted display device. Therefore, the calculation of relative posture and the conversion of safety boundaries can be performed by the playback terminal.

[0164] See also Figure 7 , is a flow chart of a method for generating a safety boundary provided in an embodiment of the present application. The process is executed by a head-mounted display device and mainly includes the following steps:

[0165] S701: Receive a security boundary acquisition request sent by a first head-mounted display device in a target space.

[0166] After receiving the initial model of the second security boundary sent by the second head-mounted display device, the playback terminal can manage the initial model process. Since either of the first head-mounted display devices and the playback terminal are located in the same local area network, the first head-mounted display device can send a security boundary acquisition request to the playback terminal in order to use the second security boundary drawn by the second head-mounted display device within the target space.

[0167] S702: In response to the safety boundary acquisition request, convert the general model into the device coordinate system of the first head-mounted display device according to the first relative posture to generate a target model of the first safety boundary.

[0168] After receiving the safety boundary acquisition request sent by the first head-mounted display device, the playback terminal converts the general model of the second safety boundary in the spatial coordinate system to the device coordinate system of the first head-mounted display device based on the first relative posture between the device coordinate system of the first head-mounted display device and the spatial coordinate system of the target space, and generates the target model of the first safety boundary in the device coordinate system of the first head-mounted display device.

[0169] S703: Send the target model to the first head-mounted display device, so that the first head-mounted display device displays the target model.

[0170] After the first head-mounted display device receives the target model, when experiencing the 3D application running on the playback terminal in the target space, when it is about to hit an obstacle, the target model of the first safety boundary can be displayed to avoid collision.

[0171] Typically, the safety boundary is displayed based on the device's own coordinate system, which can vary from device to device. To enable the first head-mounted display device in the target space to also use the safety boundary drawn by the second head-mounted display device, the second head-mounted display device sends an initial model of the first safety boundary drawn in the target space based on its own device coordinate system to the playback terminal. In this way, the playback terminal can use the spatial coordinate system of the target space as an intermediary, combine the initial model of the first safety boundary with the second relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system, and obtain a general model of the safety boundary in the target space. Then, combining the first relative pose between the device coordinate system of the first head-mounted display device and the spatial coordinate system, the playback terminal obtains a target model of the second safety boundary in the device coordinate system of the first head-mounted display device and sends the target model to the first head-mounted display device, thereby achieving conversion of the 3D model of the safety boundary between the device coordinate system of the second head-mounted display device and the device coordinate system of the first head-mounted display device. In subsequent streaming scenarios, the first head-mounted display device can use the safety boundary drawn by the second head-mounted display device to ensure the safety of the user's immersive experience, eliminating the need to draw a safety boundary separately for each first head-mounted display device in the target space, thereby reducing the workload of drawing the safety boundary.

[0172] In some embodiments, before converting the universal model to the device coordinate system of the first head-mounted display device, the playback terminal also receives an initial model of the second safety boundary drawn in the target space and a second relative pose from the second head-mounted display device, both of which are sent by the second head-mounted display device. The initial model is a 3D model of the second safety boundary in the device coordinate system of the second head-mounted display device. The initial model is combined with the second relative pose to obtain a universal model of the second safety boundary in the spatial coordinate system.

[0173] Optionally, the second relative posture can be determined by the second head mounted display device according to the first posture of the second head mounted display device in its own device coordinate system and the second posture of the second head mounted display device in the space coordinate system of the target space. Figure 3 The corresponding contents in the execution process of the second head-mounted display device will not be repeated here.

[0174] Typically, the safety boundary drawing process is generally performed before streaming, and has low real-time requirements. Therefore, when the second head-mounted display device draws the first safety boundary in the target space based on its own device coordinate system, it can use its own computing resources to calculate the second relative posture between its own device coordinate system and the spatial coordinate system of the target space, and send it to the playback terminal so that the playback terminal can subsequently obtain a general model of the second safety boundary through coordinate system conversion.

[0175] Optionally, the second relative posture can also be calculated by the playback terminal. In specific implementation, before receiving the safety boundary acquisition request sent by the first head-mounted display device in the target space, the playback terminal receives an initial model of the second safety boundary drawn in the target space and sent by the second head-mounted display device, and receives the first posture in the device coordinate system of the second head-mounted display device and sent by the second head-mounted display device, and obtains the second relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system in combination with the second posture of the second head-mounted display device in the spatial coordinate system determined by itself.

[0176] The specific calculation process of the second relative posture can be found in the above embodiment and will not be repeated here.

[0177] Due to the size and weight limitations of the second head-mounted display device, the chip computing power of the second head-mounted display device is relatively low, while the computing power of the playback terminal is relatively strong. Therefore, the playback terminal calculates the second relative posture between the spatial coordinate system and the device coordinate system based on its positioning results of the second head-mounted display device in the spatial coordinate system and the positioning results of the second head-mounted display device in the device coordinate system, thereby improving the calculation efficiency of the second relative posture.

[0178] In some embodiments, after receiving a safety boundary acquisition request sent by the first head-mounted display device in the target space, the playback terminal first converts the initial model of the second safety boundary into the spatial coordinate system according to the second relative posture to obtain a general model of the second safety boundary, and then converts the general model into the device coordinate system of the first head-mounted display device according to the first relative posture to obtain the target model of the first safety boundary.

[0179] After receiving the safety boundary acquisition request sent by the first head-mounted display device, the playback terminal first converts the initial model of the safety boundary into the spatial coordinate system according to the second relative posture to obtain a general model of the safety boundary, and then uses the spatial coordinate system as an intermediary to convert the general model of the safety boundary into the device coordinate system of the first head-mounted display device according to the first relative posture to obtain a target model of the first safety boundary suitable for the first head-mounted display device, thereby reducing the workload of drawing the safety boundary for the first head-mounted display device.

[0180] In some implementations, before receiving the safety boundary acquisition request sent by the first head-mounted display device in the target space, the playback terminal converts the initial model of the second safety boundary into the spatial coordinate system according to the second relative posture to obtain a general model of the second safety boundary. In this way, after receiving the safety boundary acquisition request, the general model of the second safety boundary in the spatial coordinate system can be directly converted into the device coordinate system of the first safety boundary according to the first relative posture to obtain the target model of the first safety boundary, thereby reducing the conversion process of the 3D model of the safety boundary and improving the efficiency of issuing the target model of the safety boundary.

[0181] The conversion from the general model to the target model and from the initial model to the general model can be found in Figure 6 and Figure 5 The process shown will not be repeated here.

[0182] In some embodiments, before converting the universal model to the device coordinate system of the first head-mounted display device, the playback terminal receives a first relative pose sent by the first head-mounted display device. The first head-mounted display device uses its own computing resources to calculate the first relative pose between its own device coordinate system and the spatial coordinate system of the target space, and sends the first relative pose to the playback terminal for subsequent use by the playback terminal in performing coordinate system conversion between the spatial coordinate system and the device coordinate system of the first head-mounted display device on the 3D model of the safety boundary.

[0183] In some embodiments, before converting the universal model to the device coordinate system of the first head-mounted display device, the playback terminal receives a third pose in the device coordinate system of the first head-mounted display device sent by the first head-mounted display device, and combines the third pose of the first head-mounted display device in the spatial coordinate system determined by the playback terminal to obtain a first relative pose between the device coordinate system of the first head-mounted display device and the spatial coordinate system. Since both the streaming terminal and the first head-mounted display device will locate the first head-mounted display device, and the computing power of the playback terminal is higher than that of the first head-mounted display device, the first head-mounted display device sends its third pose in the device coordinate system to the playback terminal, and the playback terminal combines the fourth pose of the first head-mounted display device in the spatial coordinate system determined by the playback terminal to calculate the first relative pose between the spatial coordinate system and the device coordinate system, thereby improving the calculation efficiency of the first relative pose.

[0184] The first relative posture is calculated based on the third posture of the first head-mounted display device in its own device coordinate system and the fourth posture in the space coordinate system. For the specific calculation process, see Figure 4 The corresponding content on the first head-mounted display device side will not be repeated here.

[0185] See also Figure 8 , which is a schematic diagram of the interaction process between the head-mounted display device and the playback terminal provided in an embodiment of the present application, mainly includes the following steps:

[0186] S1: The second head-mounted display device surrounds the target space and continuously captures multiple frames of environmental images.

[0187] S2: The second head-mounted display device uses SLAM technology to generate a map of the target space based on multiple frames of environmental images and sends it to the playback terminal for storage.

[0188] The above is the preprocessing process. The playback terminal realizes the positioning of the head-mounted display device in the target space based on the stored map.

[0189] S3: The second head-mounted display device starts a security boundary drawing application and establishes a connection with the positioning application of the playback terminal.

[0190] S4: The second head-mounted display device captures the first environment image and sends the environment image to the playback terminal.

[0191] S5: The positioning application of the playback terminal queries the map according to the first environment image to obtain a second posture of the second head-mounted display device in the target space.

[0192] S6: The positioning application of the playback terminal receives the first position sent by the second head-mounted display device.

[0193] S7: The positioning application of the playback terminal calculates a second relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system of the target space according to the first posture and the second posture.

[0194] S8: The second head-mounted display device turns on the perspective mode, draws the second safety boundary based on its own device coordinate system, generates an initial model of the second safety boundary, and sends it to the central control application of the playback terminal.

[0195] The initial model of the second security boundary may be sent to the central control application of the playback terminal in the form of an obj file.

[0196] The above is the process of drawing the security boundary before streaming.

[0197] S9: The playback terminal starts the first streaming application and establishes a connection with the second streaming application of the first head-mounted display device.

[0198] S10: The first head-mounted display device receives the second relative posture and the initial model of the second safety boundary sent by the central control application of the playback terminal.

[0199] S11: The second streaming application of the first head-mounted display device converts multiple boundary points included in the second safety boundary into a spatial coordinate system according to the second relative posture to obtain a general model of the second safety boundary.

[0200] S12: The second head-mounted display device collects a second environment image and sends the second environment image to a positioning application of the playback terminal.

[0201] S13: The positioning application of the playback terminal queries the map according to the second environment image to obtain a fourth posture of the first head-mounted display device in the target space.

[0202] S14: The positioning application of the playback terminal receives the third posture sent by the first head-mounted display device.

[0203] S15: The positioning application of the playback terminal calculates a first relative posture between the device coordinate system of the first head-mounted display device and the space coordinate system based on the third posture and the fourth posture, and sends the first relative posture to the first head-mounted display device.

[0204] S16: The second stream application of the first head-mounted display device converts the multiple boundary points included in the second safety boundary into the device coordinate system of the first head-mounted display device according to the first relative posture to obtain a target model of the first safety boundary.

[0205] S17: The playback terminal starts a 3D application, renders a streaming image of the 3D application according to the position of the first head-mounted display device in the target space, and sends the streaming image to the first head-mounted display device for display through the first streaming application.

[0206] S18: When the first head-mounted display device encounters a collision risk while displaying the streaming image, the target model of the first safety boundary is displayed.

[0207] The above is the process of using security boundaries during streaming.

[0208] See also Figure 9 , is a schematic diagram of the interaction process between another head-mounted display device and a playback terminal provided in an embodiment of the present application, which mainly includes the following steps:

[0209] S1: The second head-mounted display device surrounds the target space and continuously captures multiple frames of environmental images.

[0210] S2: The second head-mounted display device uses SLAM technology to generate a map of the target space based on multiple frames of environmental images and sends it to the playback terminal for storage.

[0211] The above is the preprocessing process. The playback terminal realizes the positioning of the head-mounted display device in the target space based on the stored map.

[0212] S3: The second head-mounted display device starts a security boundary drawing application and establishes a connection with the positioning application of the playback terminal.

[0213] S4: The second head-mounted display device captures the first environment image and sends the environment image to the playback terminal.

[0214] S5: The positioning application of the playback terminal queries the map according to the first environment image, obtains the second position of the second head-mounted display device in the target space, and sends it to the second terminal device.

[0215] S6: The second head-mounted display device calculates a first pose in its own device coordinate system, and calculates a second relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system of the target space in combination with the second pose.

[0216] S7: The second head-mounted display device turns on the perspective mode, draws the second safety boundary based on its own device coordinate system, generates an initial model of the second safety boundary, and sends it to the central control application of the playback terminal.

[0217] The initial model of the second security boundary may be sent to the central control application of the playback terminal in the form of an obj file.

[0218] The above is the process of drawing the security boundary before streaming.

[0219] S8: The central control application of the playback terminal sends the initial model of the second relative posture and the second safety boundary to the first head-mounted display device when the first head-mounted display device is turned on.

[0220] S9: The first head-mounted display device converts the multiple boundary points included in the second safety boundary into a spatial coordinate system according to the second relative posture to obtain a general model of the second safety boundary.

[0221] S10: The first head mounted display device starts the second streaming application and establishes a connection with the first streaming application of the playback terminal.

[0222] S11: The first head-mounted display device collects a second environment image and sends the image to a positioning application of the playback terminal.

[0223] S12: The positioning application of the playback terminal queries the map according to the second environment image, obtains the fourth posture of the first head-mounted display device in the target space, and sends it to the first head-mounted display device.

[0224] S13: The first head-mounted display device calculates a third posture in its own device coordinate system, and calculates a first relative posture between the device coordinate system of the first head-mounted display device and the spatial coordinate system of the target space in combination with the fourth posture.

[0225] S14: The first head-mounted display device converts the multiple boundary points included in the second safety boundary into the device coordinate system of the first head-mounted display device according to the first relative posture to obtain a target model of the first safety boundary.

[0226] S15: The playback terminal starts a 3D application, renders a streaming image of the 3D application according to the position of the first head-mounted display device in the target space, and sends the streaming image to the first head-mounted display device for display through the first streaming application.

[0227] S16: When the first head-mounted display device encounters a collision risk while displaying the streaming image, the target model of the first safety boundary is displayed.

[0228] The above is the process of using security boundaries during streaming.

[0229] Based on the same technical concept, an embodiment of the present application provides a playback terminal that can implement the steps of the above-mentioned security boundary generation method and achieve the same technical effect.

[0230] See also Figure 10 , the playback terminal includes a processor 1001, a memory 1002 and a communication interface 1003, and the communication interface 1003, the memory 1002 and the processor 1001 are connected via a bus 1004;

[0231] The communication interface 1003 is used to communicate with the head mounted display device;

[0232] The memory 1002 stores a computer program, and the processor 1001 executes the computer program according to the computer program. Figure 7 The method steps shown.

[0233] It should be noted that Figure 10 The playback terminal shown is only an example. Optionally, the playback terminal may also include conventional components such as a display screen, a pickup, a microphone, and a power supply.

[0234] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device that can implement the steps of the above-mentioned method for generating a safety boundary and achieve the same technical effect.

[0235] See also Figure 11 The head-mounted display device includes a processor 1101, a memory 1102, a display screen 1103, and a communication interface 1104. The communication interface 1104, the display screen 1103, the memory 1102, and the processor 1101 are connected via a bus 1105.

[0236] The communication interface 1104 is used to communicate with the playback terminal;

[0237] The camera 1103 is used to display images of the 3D application and the target space;

[0238] The memory 1102 stores a computer program, and the processor 1101 executes the computer program according to the computer program. Figure 3 or Figure 4 Steps of the method shown.

[0239] It should be noted that Figure 11 The head-mounted display device shown is only an example. Optionally, the head-mounted display device may also include conventional devices such as a camera, an IMU, a handle pickup, a microphone, and a power supply.

[0240] exist Figure 10 and Figure 11 In the embodiment, the memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories. The processor may include one or more central processing units (CPUs), GPUs or digital processing units, etc. The processor is configured to implement the steps of any of the above-mentioned security boundary generation methods when calling a computer program stored in the memory.

[0241] In the embodiment of the present application, the bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0242] For the convenience of description, the present invention can be divided into modules (or units) according to their functions and described separately. Of course, when implementing the present invention, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0243] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0244] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the steps of any one of the security boundary generation methods in the aforementioned embodiments can be completed.

[0245] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the security boundary generation methods in the aforementioned embodiments.

[0246] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0247] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0248] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0249] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0250] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for generating a safety boundary, characterized in that: Applied to a playback terminal, the method includes: receiving a security boundary acquisition request sent by a first head-mounted display device in a target space, wherein the first head-mounted display device is any head-mounted display device in the target space; In response to the safety boundary acquisition request, the general model is converted into the device coordinate system of the first head-mounted display device according to the first relative pose to generate a target model of the first safety boundary; wherein the general model is a 3D model in the spatial coordinate system of the target space, the general model is determined based on the initial model and the second relative pose, the initial model is a 3D model generated and uploaded by the second head-mounted display device in the target space according to the second safety boundary of the second head-mounted display device, the second relative pose is the relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system, and the first relative pose is the relative pose between the device coordinate system of the first head-mounted display device and the spatial coordinate system; The target model of the first safety boundary is sent to the first head-mounted display device, so that the first head-mounted display device displays the target model.

2. The method according to claim 1, wherein Before converting the universal model into the device coordinate system of the first head-mounted display device, the method further includes: receiving an initial model of a second safety boundary drawn in the target space and the second relative pose sent by the second head-mounted display device, wherein the initial model is a 3D model of the second safety boundary in the device coordinate system of the second head-mounted display device, and the second relative pose is calculated based on the first pose of the second head-mounted display device in its own device coordinate system and the second pose in the space coordinate system; or Receive an initial model of a second safety boundary drawn in the target space and sent by the second head-mounted display device, and receive a first pose in the device coordinate system of the second head-mounted display device and sent by the second head-mounted display device, and obtain a second relative pose between the device coordinate system of the second head-mounted display device and the spatial coordinate system in combination with the second pose of the second head-mounted display device in the spatial coordinate system determined by itself; wherein the initial model is a 3D model of the second safety boundary in the device coordinate system of the second head-mounted display device.

3. The method according to claim 1, wherein After receiving a safety boundary acquisition request sent by a first head-mounted display device in the target space, and before converting the general model into a device coordinate system of the first head-mounted display device according to the first relative pose, the method further includes: transforming the initial model of the second safety margin into the spatial coordinate system according to the second relative posture to obtain a general model of the second safety margin; or Before receiving the security boundary acquisition request sent by the first head-mounted display device in the target space, the method further includes: According to the second relative posture, the initial model of the second safety boundary is converted into the spatial coordinate system to obtain a universal model of the second safety boundary.

4. The method according to claim 3, wherein The converting the initial model of the second safety boundary into the spatial coordinate system according to the second relative posture to obtain a general model of the second safety boundary includes: Multiplying the initial posture angles of the plurality of boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture angle in the second relative posture to obtain intermediate posture angles of the plurality of boundary points in the space coordinate system; Multiplying the initial positions of multiple boundary points included in the initial model of the second safety boundary in the device coordinate system of the second head-mounted display device by the first relative posture corner point, and then adding the initial positions to the first relative position in the second relative posture to obtain the middle positions of the multiple boundary points in the spatial coordinate system; A general model of the second safety boundary in the space coordinate system is obtained according to the intermediate posture angles and intermediate positions of the multiple boundary points in the space coordinate system.

5. The method according to claim 1, wherein The converting the universal model into the device coordinate system of the first head-mounted display device according to the first relative posture to generate a target model of the first safety boundary includes: multiplying the intermediate pose angles of a plurality of boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative pose angle in the first relative pose to obtain target pose angles of the plurality of boundary points in the device coordinates of the first head-mounted display device; Multiplying the middle positions of the plurality of boundary points included in the general model of the second safety boundary in the spatial coordinate system by the second relative posture corner point, and then adding the middle positions to the second relative positions in the first relative posture to obtain target positions of the plurality of boundary points in the device coordinate system of the first head-mounted display device; A target model of a first safety boundary in the device coordinate system of the first head-mounted display device is generated according to the target posture angles and target positions of the multiple boundary points in the device coordinate system of the first head-mounted display device.

6. The method according to claim 1, wherein Before converting the universal model into the device coordinate system of the first head-mounted display device, the method further includes: receiving the first relative posture sent by the first head-mounted display device, where the first relative posture is calculated based on a third posture of the first head-mounted display device in its own device coordinate system and a fourth posture in the space coordinate system; or Receive the third posture in the device coordinate system of the first head-mounted display device sent by the first head-mounted display device, and combine it with the fourth posture of the first head-mounted display device in the spatial coordinate system determined by itself to obtain a first relative posture between the device coordinate system of the first head-mounted display device and the spatial coordinate system.

7. A method for generating a safety boundary, characterized in that: Applied to a second head-mounted display device, the method includes: Start the security boundary drawing application and establish a connection with the playback terminal; receiving a second posture of the second head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combining the second head-mounted display device with the first posture in its own device coordinate system to obtain a second relative posture between the device coordinate system and the spatial coordinate system; Turning on the perspective function, drawing a second safety boundary applicable to the target space in the device coordinate system, and generating an initial model of the second safety boundary; The initial model of the second safety boundary and the second relative posture are sent to the playback terminal, and the second relative posture is used to convert the initial model between the device coordinate system and the space coordinate system to generate a general model of the second safety boundary.

8. A method for generating a safety boundary, characterized in that: Applied to a first head-mounted display device, the method includes: Start the streaming application and establish a connection with the playback terminal; receiving a fourth posture of the first head-mounted display device in the spatial coordinate system of the target space sent by the playback terminal, and combining the fourth posture of the first head-mounted display device in its own device coordinate system to obtain a first relative posture between the device coordinate system and the spatial coordinate system; According to the first relative posture, the general model is converted to the device coordinate system of the first head-mounted display device to generate a target model of the first safety boundary in the device coordinate system of the first head-mounted display device; wherein the general model is a 3D model in the spatial coordinate system of the target space, and the general model is determined based on the initial model and the second relative posture, the initial model is a 3D model generated and uploaded by the second head-mounted display device in the target space based on the second safety boundary of the second head-mounted display device, and the second relative posture is the relative posture between the device coordinate system of the second head-mounted display device and the spatial coordinate system.

9. A playback terminal, characterized in that: comprising a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used to communicate with a head-mounted display device; The memory stores a computer program, and the processor executes the method according to any one of claims 1 to 6.

10. A head-mounted display device, characterized in that: It includes a processor, a memory, a display screen and a communication interface, wherein the communication interface, the display screen, the memory and the processor are connected via a bus; The communication interface is used to communicate with the playback terminal; The display screen is used to display images of 3D applications and target spaces; The memory stores a computer program, and the processor executes the method according to claim 7 or 8 according to the computer program.

Images