Systems and methods for pairing virtual objects with real-world environments

Abstract

The present invention relates to systems and methods for pairing a virtual object with a real world environment. Pairing a virtual object in a virtual reality environment involves generating a bounding box having a plurality of faces corresponding to a plurality of exterior surfaces of a subject virtual object. Generating a spatial mesh corresponding to the surfaces of the real world environment. Generating a magnetic pairing to initially align a first face of the bounding box with the first spatial mesh surface. Providing a shadow pairing between a second face of the bounding box and a second spatial mesh surface by projecting a virtual ray from a second face of the subject virtual object bounding box toward the second spatial mesh surface, determining a pairing point corresponding to an intersection of the virtual ray and the second spatial mesh surface, and displaying a pairing button in the virtual reality environment at the pairing point.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 803,367, filed on February 8, 2019, entitled “SYSTEM AND METHOD FOR PAIRING VIRTUAL OBJECTS WITH REAL-WORLD ENVIRONMENTS,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to graphics processing, and more particularly, to scaling / positioning of virtual objects in a mixed reality environment. Background Art

[0004] A mixed reality environment refers to a virtual reality (VR) environment that includes one or more aspects of the real world environment of a user of a virtual reality device. In a mixed reality environment, the use of a head mounted device (as a non-limiting example, Microsoft's HoloLens) enables the user's head to function as a camera. While this technology provides the user with the freedom to naturally observe virtual objects, such as assemblies or models, by walking around the virtual objects, certain interactions may pose challenges to the user when the user manipulates the virtual object assembly that the user is observing.

[0005] Challenges include but are not limited to:

[0006] (1) Environmental constraints: Accessing angles of a virtual object that are difficult for a person to reach in the real world. This is particularly true when establishing a primary pairing (anchor) between a virtual object and a real-world environment, as the pairing may constrain the virtual object such that the virtual object may be difficult to observe and / or manipulate, thereby requiring the user to occupy a difficult / uncomfortable position in the real-world (RW) environment to further interact with the virtual object. For example, Figure 1 As shown, the user 110 must lie on the floor 125 of the RW environment 100 with the head mounted device to observe the desired part of the virtual object 130 (here, the bottom surface of the virtual object 130) and interact with the desired part of the virtual object 130.

[0007] (2) Inefficient gesture interaction: Certain interactions between people and VR environments, such as gaze + air tap gestures, may be difficult for users to master and / or cause fatigue. Therefore, it is preferable to minimize the number of steps to manipulate the assembly.

[0008] (3) VR size changes, mesh changes, and source point differences of loaded assemblies can significantly affect the pairing or alignment efficiency with the real-world environment.

[0009] The process currently used to pair a virtual object with its real-world environment can be, for example:

[0010] (1) The user wearing the headset selects a face on the virtual assembly and then selects another real-world plane with which the user intends the virtual assembly to be paired. The user then confirms the pairing method through the user interface (UI) of the headset. If some locations in the virtual assembly are difficult for the user to reach, the user can use a ladder or lie down in the real world to reach those locations. If the user has difficulty aiming at a precise location due to unstable movement of the camera / person's head, the user can compensate by trying to stabilize the camera / head.

[0011] (2) The user manually measures the distance between the real-world object / surface and the virtual object by first selecting (clicking) the surface of the virtual object and then selecting the real-world surface, and then moving the virtual object to adjust the distance.

[0012] (3) The user then manually enters the offset distance via a virtual keyboard, which may be difficult to do when using a mixed reality headset.

[0013] Therefore, there is a need to address one or more of the issues mentioned above. Summary of the invention

[0014] Embodiments of the present invention provide a system and method for pairing a virtual object with a real-world environment. Briefly described, the present invention is directed to pairing virtual objects in a virtual reality environment. A bounding box having a plurality of faces corresponding to a plurality of exterior surfaces of a subject virtual object is generated. A spatial grid corresponding to a surface of the real-world environment is generated. Magnetic pairing initially aligns a first face of the bounding box with a first spatial grid surface. A shadow pairing is provided between a second face of the bounding box and a second spatial grid surface by projecting a virtual ray from a second face of the subject virtual object bounding box toward a second spatial grid surface, determining a pairing point corresponding to an intersection of the virtual ray and the second spatial grid surface, and displaying a pairing button at the pairing point in the virtual reality environment.

[0015] By examining the following drawings and detailed description, other systems, methods and features of the present invention will be or become apparent to those of ordinary skill in the art. It is intended that all such additional systems, methods and features be included in this specification, within the scope of the present invention and protected by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated in and constitute a part of this specification. The components in the drawings are not necessarily represented to scale, emphasis is placed on the contrary, and are placed in a manner to clearly illustrate the principles of the present invention. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0017] Figure 1 is an illustration of a user with a VR headset interacting with virtual objects in a real-world environment.

[0018] Figure 2A is a flow chart of a first exemplary embodiment of a method for pairing a virtual object with a real-world environment from a user's perspective.

[0019] Figure 2B It is shown Figure 2A A flowchart of the details of the Magnetic Pairing Mode box.

[0020] Figure 2C It is shown Figure 2A Flowchart of the details of the Shadow Control Box.

[0021] Figure 3 is an illustration of a mixed reality environment user interface.

[0022] Figure 4A A first virtual space map virtually presented to a user is shown.

[0023] Figure 4B A second virtual space map is shown that is virtually presented to the user.

[0024] Figure 5A A first user interaction in the user interface of an embodiment is shown.

[0025] Figure 5B A second user interaction in the user interface of an embodiment is shown.

[0026] Fig. 6A and Figure 6B Shown are virtual rays generated and other tools in the user interface of an embodiment.

[0027] Fig. 7A and Figure 7B The effect of a user selection of a pairing button in the user interface of an embodiment is shown.

[0028] Fig. 8A and Figure 8B Further user interactions in the user interface of an embodiment are shown.

[0029] 9A to 9D User interface objects of an embodiment are shown.

[0030] FIG. 10A to FIG. 10D Shown in the shadow pairing FIG. 2A to FIG. 2C Simplified examples of the elements mentioned in the method.

[0031] Fig.11is a block diagram of a first exemplary embodiment of a mixed reality system.

[0032] Fig. 12A is a screenshot of an example API call of Physics.RaycastAll inputting the ray origin position.

[0033] Fig. 12B is a screenshot of an example API call for Physics.RaycastAll inputting a ray direction.

[0034] Fig. 12C is a screenshot of an example API call of Physics.RaycastAll outputting a RaycastHit list (example points and normals of hit results).

[0035] Fig.12D is a screenshot of an exemplary impact point data (including normals and positions) structure.

[0036] Fig.13A is a screenshot showing an example of Vector3.ProjectOnPlane input data.

[0037] Fig. 13B is a screenshot showing an example of Vector3.ProjectOnPlane output data.

[0038] Fig.14 is a schematic diagram of a computer network support embodiment of the present invention.

[0039] Fig.15 yes Fig.14 Block diagram of a computer device / node. DETAILED DESCRIPTION

[0040] The terms used to explain the features applied to the embodiments disclosed herein are defined below and are meant to define the elements within the present disclosure only.

[0041] As used within this disclosure, "mixed reality" generally refers to the interaction between a virtual reality environment and a real world environment. Depending on the context, a mixed reality environment may refer to any of mixed reality, virtual reality, and augmented reality.

[0042] As used within this disclosure, a "mixed reality headset" generally refers to any head-mounted device that can detect a real physical environment and generate a representation of the physical environment data. As mentioned above, such a device may refer to one or more of mixed reality, virtual reality, and augmented reality. For example, different manufacturers of such devices may attempt to use different terms to distinguish their products. For example, Microsoft uses virtual reality (Dreamwalker) headsets and holographic glasses as mixed reality devices. Historically, the difference between such headsets lies in the ability to enable the user to see the real physical environment. If the user cannot see the physical environment, the headset refers to a virtual reality headset. If the user can see the physical environment, the headset is an augmented reality headset. However, recently, this distinction has been blurred because headsets that refer to virtual reality headsets have begun to include cameras, thereby enabling users to see the real physical environment on virtual reality devices. In this article, mixed reality / virtual reality / augmented reality headsets all refer to observation devices that detect the real physical world and generate representations of the real physical world.

[0043] As used herein, "collide", "collision" generally refers to an action related to a Unity engine class / component / script called a collider, which belongs to a physics engine component. A collider detects whether a first object mathematically intersects with a second another object. Collisions supported by Unity include boxes, balls, capsules, and grids (a grid means making a grid close to the shape of an object by using small triangles). MRTK attaches a collider component to an object (such as a cursor) and sets its center and size. In these embodiments, the collider component can be attached to a pairing button and set its center and size. The physics engine performs corresponding calculations and broadcasts / sends Unity events to MRTK. After MRTK receives an event from Unity, MRTK then excites an event of its own type and broadcasts / sends it to the script of the embodiment, thereby providing an alert that a cursor hits something with a collider ("collides" with something with a collider).

[0044] As used within this disclosure, "gravity mode" refers to a configuration that is set to align objects in a mixed reality environment. Gravity mode can be used to resolve some noise / errors between the real physical environment and the spatial mapping grid. For example, in a real-world environment, walls and floors provide an absolute horizontal / vertical reference, but the associated spatial mapping may not be absolutely horizontal / vertical. For example, noise / errors can cause deviations of points on the spatial mapping grid, resulting in incorrect normal vectors and incorrect corresponding positions of impact points. Since mixed reality users typically want to align models with walls and floors, gravity mode is a user option that helps minimize noise / errors in pairing. In an embodiment, the difference between gravity mode off / on is whether the impact normal data is used directly or pre-processed according to gravity mode.

[0045] As used herein, "frame" is a time reference used in graphics processing. For example, a frame refers to the state of a VR image rendered at a given point in time. Just as a video is composed of thousands of images played sequentially, VR can also be rendered by sequential images, but unlike video, VR rendering can be unconstrained by time. For each rendered image, the rendering system calculates and updates the data of the offsetMateIndicator and the rendered pixels on the rendered image to represent the offsetMateIndicator. The rendered image is refreshed at a certain frequency ("frame rate").

[0046] As used within this disclosure, "model," "assembly," and "virtual object" may be used interchangeably to refer to an object generated / manipulated by a VR headset.

[0047] Assembly: A document that pairs together parts, features, and other assemblies (subassemblies). Parts and subassemblies can exist in documents separate from an assembly. For example, in an assembly, a piston can be paired with other parts such as a connecting rod or a cylinder. This assembly can then be used as a subassembly in an assembly of an engine.

[0048] Body: A solid body includes topological data and geometric data. Topological data (e.g., faces, edges, and vertices) in a solid body has corresponding geometric data in the same solid body. Each vertex corresponds to a point. Each edge corresponds to a curve. Each face corresponds to a surface. An example multi-body component for a wheel may include an axle body, a rim body, and a plurality of spoke bodies connecting the axle body and the rim body.

[0049] Component: Any part or subassembly within an assembly.

[0050] Edge: The single outer boundary of a feature.

[0051] Entity: A discrete component, such as a face, edge, vertex, etc.

[0052] Face: A selectable area of ​​a model or surface (planar or otherwise) that has boundaries that help define the shape of the model or surface. For example, a rectangular solid has six faces and a cylindrical solid has three faces.

[0053] Feature: An independent shape that combines with other features to form a part or assembly.

[0054] Mate: A geometric relationship between entities (such as parts or bodies) in an assembly. Example mates include angle, coincident, concentric, distance, parallel, perpendicular, and tangent. A mate can also refer to a common reference point between a virtual object and the real world (RW) environment.

[0055] Reference Plane: A plane used as a reference for applying constraints.

[0056] Part: A single 3D object made up of features. A part can consist of multiple bodies. A part can become a component in an assembly. Examples of parts include bolts, pins, and plates.

[0057] Plane: Flat construction geometry.

[0058] Point: A single location in a 3D model.

[0059] As used within this disclosure, "ray" refers to the mathematical concept that indicates a semi-infinite portion of a line having a starting point ("origin") and a direction. A ray is therefore defined by its starting point coordinates and direction and is therefore similar to a vector. In the described embodiments, rays are visually represented by a ruler, and for clarity, depictions of rays and rulers are represented using the same element labels in the accompanying drawings. When describing a ray / ruler, "normal" refers to a unit vector having a direction that is orthogonal to a plane or parallel planes that include the ray's origin.

[0060] Subassembly: An assembly that is a part of a larger assembly. For example, the steering mechanism of a car is a subassembly of the car.

[0061] Surface: A 3D solid with edge boundaries or a zero-thickness plane.

[0062] Vertex: A point where two or more edges meet. Vertices can be selected by sketching, dimensioning, and many other CAD operations.

[0063] As used herein, "parent" and "child" refer to the logical relationship between objects such as objects in the Unity engine. As an example, in the real world, a tree may have branches and leaves, which can be considered as subcomponents of a tree. In a given parent / child relationship, in this case the parent can be a tree, and the leaves and branches can be children. If the user moves / rotates / scales the tree, all branches and leaves move / rotate / scale with the tree, so that in the world coordinates, the position vector change of the tree is also applicable to all its branches and leaves. Moving the tree one meter along the north direction also moves all branches and leaves one meter along the north direction. The local positions of the branches and leaves remain unchanged relative to the coordinates of the tree. The movement / rotation / scaling of the parent automatically performs the same operation on each child. For the embodiment described herein, the physicalized bounding box face is translated by one meter, and the virtual object is also moved by one meter. Typically, the movement (rotation / translation) of the child object is automatically handled by Unity according to the manipulation of the parent object. In Unity, any object can be assigned to another object as a child / parent. If the parent of an object is empty, the parent of the object is the scene.

[0064] Now, reference will be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0065] Embodiments of the present invention address environmental constraints that VR users may experience when pairing virtual objects with real-world surfaces. As described above, if a suitable positional relationship is not established between virtual objects in the virtual environment relative to the RW environment, it may be difficult for a user in the RW environment to interact with virtual objects in a mixed reality system. Embodiments described herein provide improved pairing of virtual objects with the RW environment in a mixed reality system, which at a high level include:

[0066] (1) Magnetic pairing that initially aligns the VO with at least one RW ambient surface;

[0067] (2) projecting a virtual ray from each of the plurality of VO surfaces toward a corresponding potential pairing surface in the RW environment, and indicating a pairing button at the location of each potential pairing point; and

[0068] (3) Combining offset pairing and measurement methods from projected virtual rays to present the user with a graphical indication of the distance and alignment between a set VO and its surrounding RW environment, thereby increasing the efficiency of, for example, complex floorplanning solutions.

[0069] Figure 2AIt is a flowchart of a first exemplary embodiment of a method 200 for pairing a virtual object with a real-world environment from a user's perspective. It should be noted that any process description or box in the flowchart should be understood to represent a module, segment, code portion or step including one or more instructions, which is used to implement a specific logical function in the process, and alternative implementations are included within the scope of the present invention, wherein the functions may not be performed in the order shown or discussed, including being performed substantially simultaneously or in reverse order according to the functions involved, as can be understood by a person with reasonable skills in the field of the present invention.

[0070] What follows below is a description of the functionality of a system that performs the method. The method 200 occurs in a mixed reality environment that supports the use of 3D models (eg, Microsoft Corporation's HoloLens).

[0071] Figure 3 Shown for FIG. 2A to FIG. 2C An example user interface for a mixed reality environment is described by the flowchart in . Figure 3 As shown, the user has the authority to command to manipulate the virtual model loaded into the mixed reality environment.

[0072] return Figure 2A , the user starts the pairing process, as shown in box 210. When the pairing process begins, the user selects the pairing button, at which time the virtual object enters the pairing mode and the method begins to execute the magnetic pairing mode, as shown in box 220. The details of the magnetic pairing mode are shown in Figure 2B In the magnetic pairing mode, the first surface 335a of the virtual object 335 is selected and paired with the first RW surface. For example, the bottom surface 335a of the virtual object 335 can be conveniently paired with the RW floor in the magnetic pairing mode.

[0073] In paired mode, the Holographic Glasses headset automatically performs spatial mapping to collect real-world environmental surface data and continuously generates and updates a virtual spatial mapping mesh representing the real-world surface. The virtual representation of this spatial mapping can be viewed on the Figure 4A and Figure 4B (and subsequent figures), blue triangles 410 represent surfaces of the real-world environment. It should be noted that in this document, the interaction between a virtual object and a real-world surface generally refers to the interaction between a virtual object and a virtual space mapping grid 410 representing the corresponding real-world surface.

[0074] Also in pairing mode, a three-dimensional bounding box 335 is displayed around the virtual object 330 (in green), as shown in FIG. Figure 3For simplicity, the virtual object 330 for this example can be represented as a six-sided rectangular box 335. Initially, the user selects only one bounding box face 335a from the six faces of the bounding box 335 to pair with the RW environment (magnetic), as shown in ( Figure 2B ) as shown in the frame 221. The selected bounding box face 335a is highlighted, as shown in FIG. Figure 4A shown.

[0075] After selecting a bounding box face 335a, the platform allows the user to move the virtual object 330 along the spatial mapping grid 410, such as Figure 5A The movement is performed using a ray whose starting position corresponds to the position of the user's head and whose direction corresponds to the user's gaze direction. The ray intersects the mapping grid 410 at the impact point on the mapping grid corresponding to the user's gaze direction, as shown in ( Figure 2B ) as shown in box 222. The system calculates the ray impact point on the spatial mapping grid 410 and moves the virtual object 330 relative to the spatial mapping grid 410 so that the center position of the selected bounding box face 335a is paired with the position of the ray impact point (equal to the position of the ray impact point). The system also rotates the virtual object 330 so that the negative normal vector of the selected bounding box face 335a is equal to the normal vector of the ray impact point. The user can perform a "tap" gesture on the virtual object 330 at any time, so that the virtual object 330 is placed in a certain position and rotated, as in box 223 and as shown in Figure 5B At this point, the initial alignment of the magnetic pattern is completed and the shadow projection pairing mode begins, as shown in block 230 .

[0076] After the initial magnetic pairing of the first surface of the virtual object 330, the virtual object 330 is positioned in a shadow projection mode first position (both translatable and rotatable) to begin the shadow projection mode, as shown in block 230. In the shadow projection mode, virtual rays are generated from the virtual object 330 to detect the real world surface and are graphically displayed to the user as rulers 640. In this example, six virtual rays 640 are projected along six different directions corresponding to the centers of the faces of the bounding box 335, as shown in FIG. Fig. 6A Alternatively, ray 640 may be projected from a predetermined pairing point of virtual object 330, or according to another user-defined criterion. Fig. 6A 645, shown as a circle with a paper clip, is generated on the spatial mapping grid 410 at the location where the ray 640 intersects the grid 410. The center location of the pairing button 645 is the intersection point ("hit point") of the ray 640 with the grid 410 for each bounding box face. The normal vector of the pairing button 645 is the normal vector of the ray 640 hit point for each face 335a of the bounding box 335.

[0077] If a pairing button is generated, a corresponding offset ruler 640 is also generated and displayed along the corresponding ray, thereby displaying the distance from the virtual object to the surface of the RW environment, such as ( Figure 2C ) box 242, which is displayed as hash lines evenly spaced along the ruler 640. The starting position of the ruler 640 is the center of the bounding box 335 for the virtual object 330. The ending position of the ruler 640 is the center of the corresponding pairing button 645. In this example embodiment, the unit distance that separates the hash lines on the ruler 640 is 0.1m, but different units can be selected or preset according to the size of the virtual object 330 and / or the size of the VR environment. When the user looks at a point on the ruler 640, a slider 650 appears on the ruler at the position where the user is looking (in Figure 6B shown in green).

[0078] If the user selects the pairing button 645 ( Fig. 7A ), the virtual object 330 moves so that the center position of the corresponding bounding box surface 335a is aligned with the clicked center position of the pairing button 645, and the virtual object 330 rotates (if necessary) so that the negative normal vector of the corresponding bounding box surface 335a is equal to the normal vector of the selected pairing button 645 ( Figure 7B The end result is that the virtual object 330 is paired with a defined and desired point on the spatial mapping grid 410, as shown in ( Figure 2C ) as shown in box 244.

[0079] If the user clicks the ruler slider 650 ( Fig. 8A ), as shown in block 246, the virtual object 330 moves so that the center position of the corresponding bounding box surface 335a is aligned with the clicked center position of the ruler slider, and the platform rotates the virtual object so that the negative normal vector of the corresponding bounding box surface 335a is equal to the normal vector of the corresponding pairing button 645. The result is that the virtual object 330 is paired with the center point of the space mapping grid at a certain offset distance ( Figure 8B ) and complete the pairing, as shown in box 250.

[0080] After pairing, as shown in block 260, the user may continue with other activities, but the pairing relationship will still be respected, as shown in block 270. For example, if the user chooses to change the scale of the model, the paired surfaces will still be paired, as shown in block 270. 9A to 9D shown.

[0081] The establishment of a bounding box for the model is described below to provide context for describing the interactions and calculations in magnetic pairing operations and shadow pairing operations that depend on the bounding box. FIG. 10A to FIG. 10D The results of magnetic pairing and shadow pairing are shown in FIG. 2A to FIG. 2CHere, the virtual object (Coke machine) has a six-sided bounding box 335, and the shadow pairing offset ruler 1040 is projected from the faces of the bounding box 335. The pairing button 645 indicates where the ruler (normal ray) ends on the RW surface. The shadow pairing offset slider 650 is in Fig. 10D 1040 is shown in FIG.

[0082] The exemplary system 1100 for implementing the exemplary method embodiments described herein can be a combination of off-the-shelf third-party hardware / software (1120, 1130, 1140, 1150) and a customized pairing module 1110 that interacts with the third-party components to perform the exemplary method embodiments, such as Fig.11 As shown. User 110 uses a head-mounted VR headset 1150, which is configured to scan a RW environment 1160. VR headset 1150 interacts with mixed reality toolkit 1130 and graphics rendering engine 1120 to process input from the real world environment and generate rendered eye images by VR headset 1150. Pairing module 1110 interacts with mixed reality toolkit 1130 and graphics rendering engine 1120 through API 1124 to provide the functionality of the above-described embodiments in a manner further described in detail below.

[0083] Table 1 is a non-limiting list of examples of preferred and alternative, off-the-shelf third-party hardware / software ( 1120 , 1130 , 1140 , 1150 ) that interact directly and / or indirectly with the pairing module 1110 .

[0084]

[0085]

[0086] Table 1: Third-party hardware / software for use with paired modules.

[0087] The implementation of the embodiment is described with reference to the preferred hardware / software in Table 1. A non-limiting list of examples of Unity APIs that interact with the pairing module 1110 include, among others, Camera, Physical.RaycastAll, Quaternion.LookRotation, Transform.position, Transform,position and Transform.localScale, GameObject Instantiate, GameObject.Destroy, Vector3.Distance, Vector3.ProjectOnPlane, Math.Abs ​​and Mathf.Round APIs. Similarly, the pairing module 1110 interacts with the Mixed Reality Toolkit (MRTK), which uses, among others, the Input Module API (for click and focus events), the Solver Surface Magnetic API (for solving magnetic pairing target position / rotation) and the Spatial Mapping Manager (for starting / stopping spatial mapping).

[0088] The third-party asset UnityGLTF is used to load GLTF (Graphics Language (GL) Transmission Format) files input by the user at runtime. UnityGLTF obtains data from the GLTF file and creates objects on the spot, and then sets the BoxCollider component (including center and size data) for each child object. After that, the system iterates all child objects (for example, using a loop) and calculates the maximum point and minimum point values ​​of the bounding box 335 of the virtual object 330, as shown in Table 2:

[0089]

[0090]

[0091] Table 2: Minimum and maximum points of the bounding box for the virtual object.

[0092] The maximum point and the minimum point of the bounding box 335 of the virtual object 330 have been calculated, and the center and size of the bounding box are determined, as shown in Table 3:

[0093]

[0094] Table 3: Determining the center and size of the bounding box.

[0095] Based on the bounding box 335 of the virtual object, six bounding box faces 335a may be determined, and the local scale, position, and local rotation of the bounding box 335 may be determined. Table 3 provides an example of determining the local scale, position, and local rotation of a bounding box face (here, the back face):

[0096]

[0097]

[0098] Table 4: Parameters for computing the bounding box.

[0099] In a similar manner, the parameters of all six bounding box faces are determined and shown in the order in Table 5:

[0100] index noodle 0 Front 1 back 2 left 3 Right 4 Top 5 Bottom

[0101] Table 5: Bounding box faces.

[0102] Table 5 includes three sets of pairs of parallel faces {0, 1}, {2, 3}, and {4, 5}. For each pair, each face has a bounding box face normal that is the opposite vector of the other face (mathematically, this means that the vector is the negative of the other vector). The sequential nature of Table 5 can be used to simplify calculations in the magnetic pairing method and the shadow pairing method.

[0103] As above about Figure 2A and Figure 2B As described, the magnetic pairing process is used for the initial alignment of a virtual object with the real world environment. Briefly, the user selects an initial bounding box for magnetic pairing. The method uses surface magnetic features from a third-party tool (Mixed Reality Toolkit) to automatically align the bounding box with the surface grid area that the user is currently looking at. The user taps the virtual object to confirm the initial placement of the virtual object within the RW environment.

[0104] For example, the magnetic pairing process may be implemented as follows: After the user 110 initiates magnetic pairing, an additional bounding box is created, and the scale and position of the bounding box are set equal to the selected bounding box, as shown in Table 6:

[0105]

[0106] Table 6: Create, scale, and create bounding box faces for magnetic mating

[0107] In particular, for rotations, the opposite side of the chosen bounding box face is used, as shown in Table 7:

[0108]

[0109] Table 7: Rotation of selected bounding box faces.

[0110] In order to easily move / rotate the virtual object, the parent of the virtual object is set to the materialized bounding box surface 335a, so that the position and rotation of the virtual object are automatically handled by Unity, as shown in Table 8:

[0111]

[0112] Table 8: Selection of virtual object parents

[0113] During magnetic pairing, the position and rotation of the materialized bounding box face are set equal to the surface magnetic pointer object, as shown in Table 9:

[0114]

[0115] Listing 9: Initializing the position and rotation of a virtual object

[0116] Compared to shadow pairing, the user may not use a click event to notify the pairing module 1110 to place a virtual object because the surface magnetism is intended to make the virtual object 330 follow the position of the cursor. As a result, the cursor will not collide with the virtual object 330. The user click event provided to the graphics rendering engine 1120 by the MRTK 1130 uses the cursor to collide with the collider so that the Unity engine can detect the collision and trigger the click event, so the pairing module 1110 uses the data provided by the cursor class and the GazeManager class to detect that the user clicks on something. The pairing module 1110 clears any previously stored paired bounding box faces 335a and stores the latest bounding box faces. Once the magnetic pairing faces are paired, the magnetic pairing mode is exited and the materialized bounding box faces are destroyed, as shown in Table 10:

[0117]

[0118] Table 10: Ending magnetic pairing mode.

[0119] As described above, in shadow pairing mode, the system calculates virtual rays originating from virtual object 330 and graphically presents the virtual rays to the user as ruler 640. The location where the virtual ray interacts with the ("illuminated") spatial pairing grid 410 is determined and presented to the user as a potential pairing location. The following describes the interaction between the pairing module 1110 and the Unity engine 1120 according to the first embodiment.

[0120] When the Unity engine 1120 generates the spatial mapping meshes, the mixed reality toolkit 1130 sets all the spatial mapping meshes in a "spatial mapping" rendering mode, thereby visually providing the user with a visual representation of the real physical environment, such as Fig. 8A and Figure 8B The blue triangle mesh in the figure represents the

[0121] The pairing module 1110 sets the ray origin to the center of the bounding box, sets the ray direction to the normal vector of the bounding box, sets the maximum distance to 100 meters, and sets the layer mask to "spatial mapping", as shown in the sample code in Table 11:

[0122]

[0123] Table 11: Mapping of ray hits

[0124] The Unity engine API Physics.RaycastAll() returns the list of hit results to the pairing module 1110 and extracts two data (RaycastHit) from the hit results: RaycastHit.normal and RaycastHit.point. The pairing module 1110 iterates the ray hit results to find the closest hit result, which is used as the hit result, as shown in Table 12:

[0125]

[0126] Table 12: Processing ray hit results

[0127] FIG. 12A to FIG. 12D shows an example of a virtual ray data structure provided to the Unity API. Fig. 12A An example showing the Physics.RaycastAll input ray origin location. Fig. 12B An example showing the Physics.RaycastAll input ray direction. Fig. 12C An example is shown where Physics.RaycastAll outputs a RaycastHit list (example points and normals of hit results). Fig.12D An example of the structure of the impact point data (including normal and position) is shown.

[0128] The virtual object can be rotated and / or translated to correspond to the normal vector of the impact point. This can be implemented by enabling or disabling gravity mode. The difference between gravity mode enabled and gravity mode disabled is about setting the rotation of the virtual object: whether gravity mode is enabled or disabled, the same thing is done to set the position of the virtual object.

[0129] The pairing module 1110 translates the virtual object 330 by adding vectors at the starting point (center position of the bounding box) and the ending point (impact point position to the current position of the model) of the virtual object, as shown in Table 13:

[0130]

[0131] Table 13: Translating virtual objects

[0132] When rotating the virtual object 330 relative to the pairing button 645, a specific face of the bounding box is considered. In the model class / script, the indices of the top and bottom faces are calculated and stored. The top face is defined as the bounding box face whose world position has the maximum value of the y-axis, and the bottom face is defined as the bounding box face whose world position has the minimum value of the y-axis, as shown in Table 14. If there is more than one face with the same maximum / minimum value of the y-axis, any of these faces can be selected.

[0133]

[0134] Table 14: Bounding box top / bottom

[0135] If the paired bounding box face is the top face, the vector (0, 1, 0) (the up vector in world coordinates) is used as the input to Quaternion.FromToRotation. The final rotation result is that (0, 1, 0) becomes the normal vector of the paired bounding box face. Therefore, the paired bounding box faces are aligned horizontally, as shown in Table 15:

[0136]

[0137] Table 15: Horizontal alignment of the top face of the bounding box

[0138] Similarly, if the paired bounding box face is the bottom face, the vector (0, -1, 0) (the up vector in world coordinates) is used as the input to Quaternion.FromToRotation. The final rotation result is that (0, -1, 0) becomes the normal vector of the paired bounding box face, as shown in Table 16. Therefore, the paired bounding box face is also aligned horizontally.

[0139]

[0140] Table 16: Horizontal alignment of the bottom face of the bounding box

[0141] If the paired bounding box face is neither the top face nor the bottom face, the pairing module 1110 uses Vector3.ProjectOnPlane to calculate the projection vector of the impact normal vector on the horizontal plane (the plane whose normal vector is (0, 1, 0), i.e., the upward vector in world coordinates), and then the returned projection vector is used as the input of Quaternion.FromToRotation. This ensures that the paired bounding box faces are vertically aligned, as shown in Table 17.

[0142]

[0143] Table 17: Vertical alignment of bounding box mating faces

[0144] Fig.13A is a screenshot showing an example of Vector3.ProjectOnPlane input data. Fig. 13B is a screenshot showing an example of Vector3.ProjectOnPlane output data.

[0145] If gravity mode is disabled, the pairing module uses the normal vector of the bounding box face as the first input to Quaternion.FromToRotation, the negative (opposite) impact point normal vector as the second input to Quaternion.FromToRotation, and receives the output value of the Quaternion as the return value. The returned Quaternion value is multiplied by the model's current rotation value (also called the Quaternion value) as shown in Table 18.

[0146]

[0147] Table 18: Virtual object rotation.

[0148] Finally, the current rotation value is set to the multiplied value, as shown in Table 19.

[0149]

[0150] Table 19: Virtual object rotation (continued).

[0151] As mentioned above, the pairing module generates a ruler that covers the bounding box rays to facilitate user manipulation of virtual objects within the RW environment.

[0152] Initially, in the MateModelManager script / class, the offsetMateIndicators list stores six offsetMateIndicator objects. Only one virtual object is paired during the pairing process, so each offsetMateIndicator corresponds to one bounding box face. The bounding box 335 is rectangular, so the bounding box 335 has six bounding box faces 335a, and the corresponding length of the offsetMateIndicators list is also 6. When starting or exiting the shadow pairing mode, the pairing module 1110 destroys any existing offsetMateIndicator, so the offsetMateIndicators list stores 6 empty values ​​at this time. The pairing module creates a unified preset that includes a ruler, a pairing button, and a slider in a default position and rotation. Based on the ray hit results as described above, the pairing module generates two different indicators: a hit indicator and no hit indicator. For a hit indicator, if the corresponding value in the offsetMateIndicators list is empty, the entity Unity engine API is used to instantiate the preset and store the object in the offsetMateIndicators list. The position is set to RaycastHit.point. Note that the rotation of offsetMateIndicator is not involved at this time. If the corresponding value in the offsetMateIndicators list is not empty, it means that offsetMateIndicator exists. Since the user can move / rotate / scale the model, the spatial grid is always updated at a certain frequency (for example, once every 3 seconds), and the position and rotation of offsetMateIndicator are also updated in each frame based on the collision result.

[0153] The ruler consists of two objects: a single line and a marker, which includes a distance marker (shown as a hash line) and the number of zzRuler third-party asset processing. The marker module sets the RulerLine start position to the center position of the bounding box surface and the end position to the ray impact position, as shown in Table 20:

[0154]

[0155] Table 20: Setting the ruler endpoints

[0156] The mate module 1110 does not need to set the position of the ruler 640 at this time because when the offsetMateIndicator preset is created, the local position of the ruler 640 is initialized to 0. Therefore, when the position of the offsetMateIndicator is set to the ray hit position, the ruler 640 is automatically placed in the correct position.

[0157] The scale of the ruler 640 is set to be equal to the distance between the center position of the bounding box surface 335a and the ray impact position, so that the pairing module 1110 builds a complete ruler from the center position of the bounding box surface and the ray impact position, as shown in Table 21:

[0158]

[0159] Table 21: Scaling scale.

[0160] When instantiating the offsetMateIndicator, the mate module 1110 sets the rotation to Quaternion.LookRotation (normal of the bounding box face). The mate module 1110 uses Quaternion.LookRotation to make the second portion of the ruler 640 orthogonal to the bounding box face 335a, as shown in Table 22:

[0161]

[0162] Table 22: Makes the ruler rotate.

[0163] The mate button 645 is initialized similarly. When the mate module 1110 creates the offsetMateIndicator preset, the mate module 1110 initializes the local position of the mate button 645 to 0. When the mate module 1110 sets the position of the offsetMateIndicator to the ray hit position as previously described, the ruler is automatically placed in the correct position.

[0164] The pairing module 1110 sets the rotation of the pairing button 645 according to Quaternion.LookRotation (ray impact normal). The pairing module 1110 uses Quaternion.LookRotation so that the pairing button 645 visually represents the impact normal, as shown in Table 23:

[0165]

[0166] Table 23: Pairing button rotation.

[0167] Corresponding to the scale slider 650 ( Fig. 8A) contains two objects: an interactive (clickable) green capsule and text that displays help information and distance information. The slider can be set to active or inactive based on two conditions. If the user is looking at the ruler 640, the slider 650 is set to active, otherwise the slider 650 is set to inactive. The position interval size (precision) of the slider 650 is determined by the distance from the ray impact position to the user's gaze position. For example, if the ruler position is displayed as an integer, the floating point representation on the ruler can be determined, such as 0.1m, 0.2m...3m, 3.1m...4m...

[0168] The ruler 640 has a collider component to detect collisions. At each frame, the pairing module 1110 sets the user's head position, head orientation, and a maximum distance of 10 meters as inputs to Physics.Raycast, and receives the output hitRulerInfo. If the output is valid, the pairing module 1110 checks whether the object has the tag "measureslider". If so, it indicates that the user is looking at the ruler 640. The pairing module 1110 further checks the accuracy. If the accuracy meets the above requirements, the pairing module 1110 sets the slider 650 to active, as shown in Table 24:

[0169]

[0170]

[0171] Table 24: Scale slider

[0172] After setting the slider to active, the mate module 1110 sets the position, rotation, and distance text to display the correct distance, as follows. As previously described, the mate module 1110 calculates the distance between the user's gaze impact position and the position of the offsetMateIndicator, and then calls Mathf.Round and Math.Abs ​​to calculate the circular distance, as shown in Table 25:

[0173]

[0174] Table 25: Set the slider to active

[0175] The pairing module 1110 sets the slider position according to the position of the offsetMateIndicator and the product of the vector calculated as the circular distance and the forward vector of the offsetMateIndicator, as shown in Table 26:

[0176]

[0177] Table 26: Set the slider position.

[0178] The pairing module 1110 sets the rotation of the slider to be equal to the rotation of the ruler. For a better user experience, if necessary, the angle between the direction of the user's head and the slider's forward vector is made greater than 90 degrees, which means that the user is on the back of the slider. The pairing module 1110 rotates the slider 180 degrees around itself so that the slider always faces the user, as shown in Table 27:

[0179]

[0180] Table 27: Sets the slider rotation.

[0181] An invalid collision result indicates that the ray does not intersect the spatial mapping layer. Here, the pairing module 1110 uses the entity Unity engine API to create a no collision indicator, which displays a dashed line and text, such as Fig. 10D shown.

[0182] By hitting the indicator, for each frame, if the offsetMateIndicators list at that index does not store any items, instantiate the offsetMateNoHitIndicatorPrefab, as shown in Table 28:

[0183]

[0184] Table 28: Materialization has no impact indicator.

[0185] If a previous impact indicator is stored, the previous impact indicator is destroyed and replaced with a newly created one without an impact indicator, as shown in Table 29:

[0186]

[0187] Table 29: Replacement of the pairing indicator with a no impact indicator.

[0188] Then, the pairing module 1110 sets the position without the impact indicator to the center of the bounding box, and sets the rotation to Quaternion.LookRotation (normal of the bounding box), as shown in Table 30:

[0189]

[0190] Table 30: Setting the rotation for the no impact indicator

[0191] When resizing a virtual object, the pairing module 1110 further maintains the alignment of the virtual object with the pairing points and resolves the problem that the resizing may collide with one or more pairing points. When loading a virtual object, the pairing module 1110 calculates and stores the center of the corresponding real-size bounding box. For all 6 bounding box faces, the pairing module 1110 calculates and stores the local center position, local scale, and rotation.

[0192] After the user is satisfied with the pairing result, the pairing mode is exited. When the user subsequently wishes to scale the virtual object, for example by clicking a scale button, if the virtual object is already represented at the actual size, scaling cannot be performed. If the virtual object is not at the actual size, the virtual object can be scaled. Here, the current world position of the paired bounding box surface is stored, as shown in Table 31:

[0193]

[0194] Table 31: Stores pre-scaled bounding box parameters.

[0195] The pairing module 1110 scales the virtual object to its actual size around the center of its bounding box, as shown in Table 32:

[0196]

[0197] Table 32: Scaling virtual objects.

[0198] At this point, if the bounding box 335 of the virtual object collides with one or more paired points, the next two steps are used to resolve the collision. The current world position of the paired bounding box faces is stored, as shown in Table 33:

[0199]

[0200] Table 33: Stores the bounding box position after scaling.

[0201] For each paired bounding box face, the pairing module 1110 calculates the position difference between before scaling around the center of the bounding box and after scaling around the center of the bounding box, and then applies all of these position differences to the virtual object. In this case, the virtual object is scaled to actual size and maintains paired alignment, as shown in Table 34:

[0202]

[0203] Table 34: Restore to actual size.

[0204] Fig.14 A computer network or similar digital processing environment is shown in which the present invention may be implemented.

[0205] The client computer / device 50 and the server computer 60 provide processing devices, storage devices, and input / output devices, etc., for running application programs. The client computer / device 50 may also be linked to other computing devices, including other client devices / processors 50 and server computers 60, via a communication network 70. The communication network 70 may be part of a remote access network, a global network (e.g., the Internet), a cloud computing server or service, a global computer group, a local area network or a wide area network, and a gateway that currently uses corresponding protocols (TCP / IP, Bluetooth, etc.) to communicate with each other. Other electronic device / computer network architectures are suitable.

[0206] Fig.15 yes Fig.14 1 is a diagram of the internal structure of a computer (e.g., a client processor / device 50 or a server computer 60) in a computer system of FIG. 1 . Each computer 50, 60 contains a system bus 79, where a bus is a set of hardware lines used to transfer data between components of a computer or processing system. The bus 79 is essentially a shared conduit that connects the different elements of the computer system (e.g., processors, disk storage, memory, input / output ports, network ports, etc.), enabling information to be transferred between the elements. Attached to the system bus 79 is an I / O device interface 82, which is used to connect various input and output devices (e.g., keyboard, mouse, display, printer, speakers, etc.) to the computer 50, 60. A network interface 86 allows the computer to connect to a computer connected to a network (e.g., a computer system). Fig.14 The memory 90 provides volatile storage for computer software instructions 92 and data 94 (e.g., code described in detail above) used to implement embodiments of the present invention. Disk storage 95 provides non-volatile storage for computer software instructions 92 and data 94 used to implement embodiments of the present invention. The central processor unit 84 is also attached to the system bus 79 and provides for the execution of computer instructions.

[0207] In one embodiment, the computer program 92 and data 94 are a computer program product (generally labeled 92) comprising a computer readable medium (e.g., a removable storage medium, such as one or more DVD-ROMs, CD-ROMs, floppy disks, magnetic tapes, etc.) that provides at least a portion of the software instructions for the system of the present invention. The computer program product 92 can be installed by any suitable software installation program, as is known in the art. In another embodiment, at least a portion of the software instructions can also be downloaded via a cable, communication and / or wireless connection. In other embodiments, the program of the present invention is a signal product 107 propagated by a computer program, which is embedded in a propagation medium and is in the form of a propagation signal (e.g., radio waves, infrared waves, laser waves, sound waves, or radio waves propagated on a global network such as the Internet or other networks). Such a medium or signal carrier provides at least a portion of the software instructions for the program / routine 92 of the present invention.

[0208] In alternative embodiments, the propagated signal is an analog carrier wave or a digital signal carried on a propagation medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal transmitted over a propagation medium over a period of time, such as instructions for a software application program sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of the computer program product 92 is a propagation medium that can be received and read by the computer system 50, such as receiving the propagation medium and identifying the propagated signal embedded in the propagation medium, as described above for the computer program propagation signal product.

[0209] Generally speaking, the term "carrier medium" or volatile carrier includes the aforementioned volatile signals, propagation signals, propagation media, storage media, etc.

[0210] In other embodiments, program product 92 may be implemented as so-called software as a service (SaaS) or other installation or communication support to end-users.

[0211] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments as encompassed by the appended claims.

[0212] Possible practical uses of the above embodiments include but are not limited to:

[0213] (1) Machine installation inspection

[0214] (2) Floor planning and layout

[0215] Possible benefits of the present invention include, but are not limited to:

[0216] (1) Make it easier and more comfortable for users to pair in a physical environment;

[0217] (2) minimize the number of clicks and the time spent performing pairing and distance adjustments; and

[0218] (3) Increase the accuracy of pairing virtual objects with real-world environments.

[0219] It will be apparent to those skilled in the art that various modifications and changes may be made to the structure of the present invention without departing from the scope or spirit of the present invention. In view of the foregoing description, it is intended that the present invention encompass modifications and changes of the present invention as long as these modifications and changes fall within the scope of the claims and their equivalents.

Claims

1. A computer-based method for pairing a virtual object in a virtual reality environment with a real-world environment near a user of a virtual reality device in a mixed reality environment, the method comprising the steps of: generating a bounding box including a plurality of faces corresponding to a plurality of exterior surfaces of the dominated virtual object; generating a spatial mesh corresponding to a plurality of surfaces of the real-world environment; providing a magnetic pairing that initially aligns a first face of a subject virtual object bounding box with a first spatial grid surface; as well as Providing a shadow pairing between a second face of the subject virtual object bounding box facing the second spatial grid surface, the step further comprising the steps of: projecting a virtual ray from a second face of the dominated virtual object bounding box toward the second spatial grid surface; determining a pairing point corresponding to an intersection point of the virtual ray and the second spatial grid surface; as well as displaying a pairing button at the pairing point in the virtual reality environment; The mixed reality environment is an environment for interaction between the virtual reality environment and the real world environment.

2. The method according to claim 1, further comprising the steps of: displaying a ruler along a virtual ray pairing button at the pairing point in the virtual reality environment, in, The ruler further includes a first endpoint on a second face of the dominated virtual object bounding box and a second endpoint at the paired point.

3. The method according to claim 2, wherein: The ruler further includes a scaled indicator unit of a distance between the first endpoint and the second endpoint. 4 . The method according to claim 2 , further comprising the step of displaying a slide handle on the ruler at a midpoint between the first endpoint and the second endpoint.

5. The method according to claim 4, wherein: When it is detected that the user's gaze is directed toward the ruler, the display of the slider on the ruler is adjusted. 6 . The method according to claim 5 , further comprising the step of: translating the position of the virtual object along a path parallel to the ruler according to the user's manipulation of the slider along the ruler.

7. The method according to claim 1, further comprising the steps of: Based on a user action indicating selection of the pairing point, the virtual object is translated and / or rotated according to the pairing point and the magnetic pairing.

8. The method according to claim 7, further comprising the steps of: The virtual object is scaled relative to the pairing point and the magnetic pairing, wherein: The scaling avoids the virtual object from intersecting the spatial grid.

9. The method according to claim 7, further comprising the step of aligning the virtual object with the surface of the real world environment according to the direction of the second face of the virtual object and the direction of the second spatial grid surface.

10. A system (1100) for pairing a virtual object in a virtual reality environment with a real world environment near a user of a virtual reality device in a mixed reality environment, the system comprising: A virtual reality head mounted instrument subsystem (1150), the virtual reality head mounted instrument subsystem comprising: A virtual reality headset (1152) comprising sensors for sensing real-world environment data; and Virtual reality headset software driver (1154); a computer-based graphics rendering engine (1120) configured to receive the real-world environment data and render governed virtual objects and a spatial mesh comprising a graphical representation of the real-world environment for the virtual reality head mounted instrument subsystem; and A pairing module (1110) configured to interact with the graphics rendering engine and the virtual reality head mounted instrument subsystem, the pairing module comprising a processor and a memory configured to store non-volatile instructions, the non-volatile instructions performing the following steps when executed by the processor: generating a bounding box including a plurality of faces corresponding to a plurality of exterior surfaces of the dominated virtual object; providing a magnetic pairing configured to initially align a first face of a subject virtual object bounding box with a first spatial grid surface; and Providing a shadow pairing between a second face of the subject virtual object bounding box facing the second spatial grid surface, the step further comprising the steps of: projecting a virtual ray from a second face of the dominated virtual object bounding box toward the second spatial grid surface; determining a paired point corresponding to an intersection of the virtual ray and the second spatial grid surface; and displaying a pairing button at the pairing point in the virtual reality environment; The mixed reality environment is an environment for interaction between the virtual reality environment and the real world environment.

11. The system of claim 10, further comprising a ruler module, the ruler module being configured to perform the following steps: providing a ruler along a virtual ray pairing button at the pairing point in the virtual reality environment, in, The ruler further includes a first endpoint on a second face of the dominated virtual object bounding box and a second endpoint at the paired point.

Images