A mixed reality visual guidance method for remote collaboration between trainees and experts

Abstract

The present invention proposes a mixed reality visual guidance method for remote collaboration between trainees and experts. Experts write visual instructions through a virtual reality (VR) interface. The instructions are displayed to trainees by overlaying them on the workspace through an augmented reality (AR) interface. First, when showing the visual instructions written by the expert to the trainee, the occlusion of the three-dimensional workspace should be taken into account. Second, the expert can write highly suggestive instructions by selecting, copying, pasting and repositioning the objects in the workspace to truly describe the target state of the workspace; third, multiple instructions can be connected in sequence and executed independently by the trainee without any additional guidance from the expert; for this purpose, it is necessary to obtain the changing workspace in real time and adjust the instructions to take into account the actual state of the workspace; supporting robust batch execution of instructions improves the efficiency of cooperation between trainees and experts and enables experts and trainees to work asynchronously.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics and mixed reality technology, and in particular to a mixed reality visual guidance method for remote collaboration between trainees and experts. Background Art

[0002] Mixed reality technology enables effective collaboration across vast geographical distances by connecting the worlds of collaborating parties. In one type of remote collaboration, a local trainee must manipulate physical objects in their workspace under the guidance of a remote expert. The trainee's workspace is captured using multi-view depth and color cameras that acquire the appearance and geometry of the workspace in real time. The acquired color and depth data is then sent to the remote site, where the expert can perform immersive visualization using a virtual reality helmet. A powerful way to provide guidance to trainees is through graphical annotations of the workspace, which is more effective and efficient than verbal instructions.

[0003] Previous work has also studied the improvement of collaboration efficiency brought by AR annotation in several collaborative scenarios. In one scenario, collaborators work together to complete a task. Many methods have studied hybrid methods based on several types of visual cues, such as gestures, virtual ray pointing, and painting. Studies have found that specific combinations of cues are beneficial for specific tasks in reducing task load and improving the social quality of collaboration, but further exploration is needed for visual instructions in remote collaboration scenarios.

[0004] In another collaborative scenario, which is also explored by the method of the present invention, one of the collaborators is a local trainee who must perform tasks under the guidance of a remote expert. In this case, the additional task of the collaborative system is to communicate the workspace to the remote expert and allow the expert to write annotations. Some studies capture the workspace with static images or flat videos. The instructor then annotates the workspace, but the workspace is more restricted and the expert cannot explore freely. Another study focuses on text annotations of videos, which helps collaborators use cohesive language and makes verbal communication more efficient, but cannot provide more complex graphical annotations. Some methods solve the ambiguity of 2D annotations by having experts draw annotations from two perspectives and triangulate 2D annotations into 3D space, but they also cannot provide more complex graphical annotations in 3D space. With the development of depth cameras, some studies have begun to rely on active workspace acquisition. By exploiting the geometry of the workspace, or relying on simplified flat proxy assumptions, hand-drawn 2D visual cues are projected into 3D space, which allows trainees to change perspectives freely. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a mixed reality visual guidance method for remote collaboration between trainees and experts, which can be effectively and efficiently used for remote visual guidance, alleviate occlusion in the three-dimensional workspace, provide richer visual instructions and achieve good communication efficiency between trainees and experts.

[0006] In order to solve the above problems, a mixed reality visual guidance method for remote collaboration between trainees and experts is proposed. The method can automatically detect and alleviate the incompleteness of translation instructions caused by occlusion of the workspace; allow experts to place point clouds or geometric model copies of objects to the target's posture, thereby depicting the expected state of rotation instructions with high fidelity; and support expert asynchronous writing / trainee independent execution of multi-step instruction sequences.

[0007] The technical solution adopted by the present invention mainly includes the following parts:

[0008] (1) Mixed reality system settings

[0009] The method of the present invention relies on a VR interface for experts to write instructions, and an AR interface for trainees to execute instructions. Trainees stand in front of their physical workspace and manipulate objects through visual instructions displayed by an optical perspective augmented reality head-mounted display (AR HMD). The geometry and color of the workspace and trainees are acquired in real time using 4x2 RGBD cameras. Point-based geometry and color models of the workspace and trainees are transmitted to the remote expert, who visualizes them using a virtual reality head-mounted display (VR HMD). The expert writes visual graphical instructions for the workspace, which are sent to the trainees to provide visual guidance for the trainees.

[0010] (2) Visual instructions for occlusion perception

[0011] For the instruction of translating an object, given a need to move from the starting position P s Translate to the target end position P e objects, simply connect them with straight arrows s and P e , when the arrow intersects with other objects in the workspace, it will lead to confusing visual indications without properly resolving the visibility between the arrow and the geometry in the workspace. One option is to Z-buffer the straight arrow with the geometry of the workspace, which can provide correct visual occlusion relationship cues when the arrow passes through the three-dimensional workspace, but this comes at the expense of partial occlusion of the arrow. The approach taken by the present invention is to use an occlusion-aware algorithm to calculate a curved arrow to avoid occluding objects and provide a good indication for translation without visibility inconsistencies.

[0012] (3) Visual instructions based on the actual target state

[0013] Instructions for rotating objects can benefit from richer visualization. In fact, visualizations that rely on arrows and coordinate system axes to express rotation operations are difficult to understand and difficult to execute accurately. Once the trainee moves the object from its original position and posture, the trainee must remember the coordinate axes in the initial state in order to correctly understand the target state. In addition, abstract rotation visual instructions based on arrows and coordinate systems are not only difficult to execute, but also difficult to write. The present invention proposes visualization instructions based on real target states, with the aim of supporting fast and accurate expert writing and trainee execution.

[0014] (4) Adaptive Sequence of Visual Instructions

[0015] In order to improve the efficiency of expert / trainee communication, let one expert provide guidance to multiple trainees at the same time, and allow expert / trainees to collaborate asynchronously, the present invention designs a program for writing and executing multiple instruction sequences. The expert writes multiple instructions and completes them at once in chronological order. These instructions are written in the color and depth point cloud model of the workspace obtained in advance. In other words, the workspace is not obtained in real time, which allows trainees to submit their workspaces asynchronously. Trainees execute a series of instructions each time. The instructions to be executed are updated based on the actual execution of earlier instructions. In this way, the accuracy of the instructions can be maintained despite minor changes in the execution of earlier instructions. This robustness is particularly important in the case of long instruction sequences, where errors can accumulate and make future instructions unusable if the actual state of the workspace is not adapted.

[0016] In the above part (2), the occlusion perception algorithm is specifically implemented as follows:

[0017] 21) The algorithm first starts by rendering the workspace depth buffer from the current trainee's perspective. The present invention uses point-based rendering to obtain the trainee's perspective depth buffer with a constant three-dimensional size, which is then used to resolve the translation instruction occlusion check. The visibility of the translation instruction is determined by the visibility of the sampling point, which is the point along the line segment that is checked for occlusion.

[0018] 22) When not all sample points are visible, each iteration of the algorithm raises the candidate curve higher in the workspace to alleviate occlusion. The step size starts at 1 cm and then increases exponentially by a factor of, for example, 20% in each iteration. The algorithm starts with a straight arrow and iterates out curves with increasing curvature. The curve is modeled as a quadratic Bezier curve with three control points. The first and last control points are the start and end points of the translation command. The middle control point starts as the midpoint of the translation command line segment and then increases with the step size.

[0019] 23) The algorithm tracks the visibility of the best candidate arrow by scoring it, which is equal to the number of points visible in the sample points, initialized to -1. The algorithm iterates if no arrow is found, and the vertical displacement step of the arrow does not exceed a maximum value. The present invention defines the maximum value as the length of the translation guide segment.

[0020] In the above section (3), the visualization instruction of the real target state is specifically implemented as follows:

[0021] 31) This instruction offers two options. The first two stages (A) and (B) are the same for both options. In stage A, the expert points out the object to be manipulated by creating a selection polygon one vertex at a time to define a loose bounding box of the object. The expert uses a virtual laser attached to a tracked handheld controller. The intersection between the virtual laser beam and the workspace depth buffer is calculated by sliding a point along the laser beam from near to far until the point is occluded. The expert selects polygon vertices one by one using the controller buttons.

[0022] 32) In stage B, the loose bounding polygon is refined to a subset of the workspace point cloud that belongs to the target object. Points belonging to the object are segmented from all points within the bounding polygon. This is done with a filter that retains only those bounding polygon points that lie above the flat workspace table and whose color and depth are similar enough to the mean of all points, i.e. within three standard deviations. Once the object point cloud has been determined, the instruction flow proceeds in one of two ways.

[0023] 33) If a geometric model of the object is not available, the expert uses a point cloud representation of the object to locate and orient the object. (C) If a geometric model of the object is available, the expert repositions the approximate point cloud representation of the object with the full and high-fidelity geometric model representation of the object. The expert locates and orients the object as needed by grasping and moving it using a handheld controller (E).

[0024] In the above section (4), the adaptive sequence of visual instructions is specifically implemented as follows:

[0025] The current instruction k is adapted to the actual state of the workspace in two steps, the actual position A of the object acted on by instruction k-1 is calculated, and the actual state of the workspace is adapted by adjusting instruction k to position A. The actual position of the acted object is calculated by subtracting the depth buffer of the workspace after instruction k from the depth buffer after instruction k-1. In order to robustly calculate the difference, the present invention uses an average of 30 depth frames, filters out the difference below the minimum threshold, and calculates the maximum connected area R of the non-zero depth difference. The center of the axis-aligned bounding box of the three-dimensional points in the difference area R is approximately the center of the actual position of the object. Once the actual position of the object is known, the starting point or end point of the arrow of instruction k is adjusted accordingly.

[0026] The advantages of the present invention compared with the prior art are:

[0027] (1) Considering the occlusions in the 3D workspace, when expert-written visual instructions are shown to trainees, it is more effective to automatically mitigate the occlusions than to require trainees to self-assume an unobstructed viewpoint.

[0028] (2) In addition to abstract visual instructions implemented through arrows, experts can also write highly suggestive instructions by realistically depicting the target state of the workspace; experts can directly rearrange the workspace by manipulating objects in the workspace to quickly write rich visual instructions that trainees can perform effectively and accurately. For example, if the visual instructions show the target pose of an object, the trainee's task becomes trivial because they only need to place the object to match the displayed pose without going through the trial and error of abstract visual instructions.

[0029] (3) Multiple instructions can be strung together into a sequence and executed by the trainee himself without any additional guidance from the expert. To do this, it is necessary to obtain the changing workspace in real time and adjust the instructions taking into account the actual intermediate state of the workspace. Supporting instruction sequences, rather than just single instructions, improves the efficiency of the trainee-expert collaboration and enables experts and trainees to work asynchronously. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall effect diagram of the method of the present invention;

[0031] Figure 2 It is a schematic diagram of the system of the present invention;

[0032] Figure 3 This is a diagram illustrating visual instructions for occlusion perception of the present invention;

[0033] Figure 4 It is a visual instruction flow chart based on the real target state;

[0034] Figure 5Illustration of the adaptive sequence of visual instructions. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0036] Figure 1 The trainee's AR interface is shown (two columns on the left): a comparison between the method of the present invention (top) and the traditional method (bottom) in providing visual instructions for object translation (left picture) and object rotation (right picture). The method of the present invention bends the translation arrow upward to reduce the occlusion of the work area (A), while the traditional method occludes the translation arrow, resulting in incompleteness (C). The method of the present invention accurately conveys the ideal posture of the object to be rotated through the rendering of the object's geometric model (B), rather than the visualization of the coordinate system axes of the traditional method (D). Automatic adjustment of instructions to support asynchronous collaboration (right column): a sequence of four instructions written by an expert (E) is adjusted in real time when the trainee executes asynchronously (F). To illustrate the problem, the brightness of the workspace is turned down. The starting point of instruction 1 is adjusted from c to reflect the actual position c' of the object to be translated; instruction 3 requires the user to stack the object at a onto the object at b, and after the object at b moves, the end point of instruction 3 is updated to b'.

[0037] like Figure 2 As shown, trainees manipulate objects in their workspace. The workspace is captured using an RGBD camera. Color and depth data of the workspace is transmitted to the remote expert. The workspace and trainees are displayed to the expert through a VR head mounted display. The expert can annotate the workspace and provide visual instructions to the trainee through the AR head mounted display.

[0038] (1) Mixed reality system settings

[0039] The hybrid system is set up as follows Figure 2 As shown. The method of the present invention relies on a VR interface for experts to write instructions, and an AR interface for trainees to execute instructions. Trainees stand in front of their physical workspace and manipulate objects through visual instructions displayed by an optical perspective augmented reality head-mounted display (AR HMD). The geometry and color of the workspace and trainees are acquired in real time through 4x2 RGBD cameras. The point-based geometry and color model of the workspace and trainees is transmitted to the remote expert, who visualizes it using a virtual reality head-mounted display (VR HMD). The expert writes visual graphical instructions for the workspace, which are sent to the trainees to provide visual guidance to the trainees.

[0040] (2) Visual instructions for occlusion perception

[0041] For the instruction of translating an object, given a need to move from the starting position Ps Translate to the target end position P e objects, simply connect them with straight arrows s and P e , which can lead to confusing visual indications when arrows intersect other objects in the workspace without properly resolving visibility between the arrow and the geometry in the workspace. One option is to Z-buffer the straight arrow with the geometry in the workspace, which can provide correct visual occlusion relationship cues when the arrow passes through the 3D workspace, but this comes at the expense of partial occlusion of the arrow. The approach taken by the present invention is to calculate a curved arrow to avoid occluding objects and provide a good indication for translation without visibility inconsistencies, see Figure 3 .

[0042] (3) Visual instructions based on real target state

[0043] Instructions for rotating objects can benefit from richer visualization. In fact, visualizations that rely on arrows and coordinate system axes to express rotation operations are difficult to understand and difficult to execute accurately. Once the trainee moves the object from its original position and posture, the trainee must remember the coordinate axes in the initial state in order to correctly understand the target state. In addition, abstract rotation visual instructions based on arrows and coordinate systems are not only difficult to execute, but also difficult to write. The present invention proposes visualization instructions based on real target states, with the aim of supporting fast and accurate expert writing and trainee execution.

[0044] (4) Adaptive Sequence of Visual Instructions

[0045] In order to improve the efficiency of expert / trainee communication, let one expert provide guidance to multiple trainees at the same time, and allow expert / trainees to collaborate asynchronously, the present invention designs a program for writing and executing multiple instruction sequences. The expert writes multiple instructions and completes them at once in chronological order. These instructions are written in the color and depth point cloud model of the workspace obtained in advance. In other words, the workspace is not obtained in real time, which allows trainees to submit their workspaces asynchronously. Trainees execute a series of instructions each time. The instructions to be executed are updated based on the actual execution of earlier instructions. In this way, the accuracy of the instructions can be maintained despite minor changes in the execution of earlier instructions. This robustness is particularly important in the case of long instruction sequences, where errors can accumulate and make future instructions unusable if the actual state of the workspace is not adapted.

[0046] Figure 3 A graphic representation of the algorithm selecting candidate arrows is given.

[0047] (1) The algorithm first starts to render the workspace depth buffer from the current trainee's perspective. The present invention uses point-based rendering to obtain the trainee's perspective depth buffer with a constant three-dimensional size, which is then used to resolve the translation instruction occlusion check. The visibility of the translation instruction is determined by the visibility of the sampling point, which is the point along the line segment that is checked for occlusion.

[0048] (2) When not all sample points are visible, each iteration of the algorithm raises the candidate curve within the workspace to alleviate occlusion. The step size starts at 1 cm and then increases exponentially, for example, by 20% in each iteration. The algorithm starts with a straight arrow and then iterates out curves with increasing curvature. The curve is modeled as a quadratic Bezier curve with three control points. The first and last control points are the start and end points of the translation command. The middle control point starts as the midpoint of the translation command line segment and then increases with the step size.

[0049] (3) The algorithm tracks the visibility of the best candidate arrow by scoring it, whose value is equal to the number of points visible in the sample points, initialized to -1. The algorithm iterates without finding an arrow, and arrows 1 and 2 are partially occluded and therefore not selected. At the same time, the vertical displacement step of the arrow does not exceed a maximum value. The present invention defines the maximum value as the length of the translation guide segment.

[0050] Arrow 3 is the first arrow that is curved enough to eliminate occlusion in the working area. The algorithm returns arrow 3 (although arrow 4 is also unoccluded, the algorithm does not consider arrow 4).

[0051] Figure 4 The visual instruction algorithm flow based on the real target state is given.

[0052] (1) The instruction offers two options, when the geometric model of the object is not available (A, B, C), and when the geometric model of the object is available (A, B, D, E). The first two stages (A) and (B) are the same for both options. In stage A, the expert indicates the object to be manipulated by creating a selection polygon one vertex at a time to define a loose bounding box of the object. The expert uses a virtual laser attached to a tracked handheld controller. The intersection between the virtual laser beam and the workspace depth buffer is calculated by sliding a point along the laser beam from near to far until the point is occluded. The expert selects polygon vertices one by one using the controller buttons.

[0053] (2) In stage B, the loose boundary polygon is refined to a subset of the workspace point cloud that belongs to the target object. Points belonging to the object are segmented from all points within the boundary polygon. This is done using a filter that retains only those boundary polygon points that lie above the planar workspace table and whose color and depth are sufficiently similar to the mean of all points, i.e., within three standard deviations. Once the object point cloud has been determined, the instruction flow proceeds in one of two ways.

[0054] (3) If the geometric model of the object is not available, the expert uses the point cloud representation of the object to locate and orient the object. If the geometric model of the object is available, the expert repositions the approximate point cloud representation of the object with the complete and high-fidelity geometric model representation of the object. The expert locates and orients the object as needed by grasping and moving the object using a handheld controller (E).

[0055] Figure 5 The need for adaptability of instructions to support accurate instruction sequences is illustrated. The left column (A, B, C) shows a situation where the starting position of the second instruction depends on the ending position of the first instruction. The first instruction specifies moving the object to an intermediate position, and then the second instruction specifies moving it to a final position (A). If the trainee moves the object to a slightly different position, the starting point of the arrow of the second instruction is incorrect (B). Using the method of the present invention, the second instruction is adjusted to correctly start from the center of the actual intermediate position of the object (C). The right column (D, E, F) shows a situation where the ending position of the second instruction depends on the ending position of the first instruction. The two instructions here (1 and 2 in D) require the trainee to move the first block to a new position and then stack another block on top of the first block. Because the position where the user moved the first block is slightly different from the position specified by the first instruction, the ending point of the second instruction is inconsistent with the actual center of the object (E). Using the method of the present invention (F), the ending point of the arrow of the second instruction is adjusted to track the center of the object moved by the trainee. In this way, the second instruction correctly points to the ideal stacking of the second block on top of the first block.

[0056] The present invention adapts the current instruction k to the actual state of the workspace in two steps, calculates the actual position A of the object acted on by instruction k-1, and adapts the actual state of the workspace by adjusting instruction k to position A. The actual position of the acted object is calculated by subtracting the depth buffer of the workspace after instruction k from the depth buffer after instruction k-1. In order to robustly calculate the difference, the present invention averages 30 depth frames, filters out the difference below the minimum threshold, and calculates the maximum connected area R of the non-zero depth difference. The center of the axis-aligned bounding box of the three-dimensional points in the difference area R is approximately the center of the actual position of the object. Once the actual position of the object is known, the starting point or end point of the arrow of instruction k is adjusted accordingly.

[0057] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A mixed reality visual guidance method for remote collaboration between trainees and experts, characterized in that The steps include: S1, visual instructions for occlusion perception; For instructions to translate an object, given a position that needs to be moved from the starting position Pan to target end position objects, connected by straight arrows and ,When the arrow intersects with other objects in the workspace, it will lead to confusing visual indication, an occlusion-aware algorithm is used to calculate a curved arrow to avoid the occluded objects; S2, visual instructions based on the real target state; Instructions for rotating objects, based on visualization of the real target state; S3, adaptive sequence of visual instructions, specific implementation: Adapt the current instruction k to the actual state of the workspace in two steps: calculate the actual position A of the object affected by instruction k−1, and adapt the instruction k to the actual state of the workspace by adjusting it to position A; calculate the actual position of the object affected by the instruction k by subtracting the depth buffer of the workspace after instruction k from the depth buffer after instruction k−1; Among them, in step S1, The occlusion perception algorithm is specifically implemented as follows: S11, the algorithm first starts to render the workspace depth buffer from the current trainee's perspective; using point-based rendering, the depth buffer of the trainee's perspective is obtained with a constant three-dimensional size, and then it is used to solve the translation instruction occlusion check; the visibility of the translation instruction is determined by the visibility of the sampling point, which is the point along the line segment that is checked for occlusion; S12, when not all sampling points are visible, each iteration of the algorithm raises the candidate curve within the working area to alleviate occlusion; S13, the algorithm tracks the visibility of the best candidate arrow by scoring it, whose value is equal to the number of points visible among the sampled points, initialized to -1.

2. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 1, characterized in that: Step S12 further comprises: The step size starts at 1 cm and then increases exponentially. The algorithm starts with a straight arrow and then iterates out a curve with increasing curvature. The curve is modeled as a quadratic Bezier curve with three control points. The first and last control points are the start and end points of the translation command. The middle control point starts as the midpoint of the translation command line segment and then increases with the step size.

3. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 2, characterized in that: Step S13 further comprises: The algorithm iterates if no arrow is found, and the vertical displacement step of the arrow does not exceed a maximum value, which is defined as the length of the translation guide segment.

4. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 1, characterized in that: In the step S2, The visualization instructions of the real target state are specifically implemented as follows: S21, this instruction offers two choices, the first two stages are the same for both choices; in the first stage, the expert indicates the object to be manipulated by creating a selection polygon one vertex at a time to define a loose bounding box of the object; the expert uses a virtual laser attached to a tracked handheld controller; the intersection between the virtual laser beam and the workspace depth buffer is calculated by sliding a point along the laser beam from near to far until the point is occluded; the expert selects polygon vertices one by one using the controller buttons; S22, in the second stage, is refined from loose boundary polygons to a subset of the workspace point cloud belonging to the target object; Points belonging to the object are segmented from all points inside the bounding polygon; this is done by a filter that retains only the bounding polygon points that lie above the flat workspace table and whose color and depth are sufficiently similar to the mean of all points, i.e. within three standard deviations; S23, if the geometric model of the object is not available, the expert uses a point cloud representation of the object to locate and orient the object; if the geometric model of the object is available, the expert repositions the approximate point cloud representation of the object with the complete and high-fidelity geometric model representation of the object; the expert locates and orients the object as needed by grabbing and moving the object using a handheld controller.

5. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 1, characterized in that: The step S3 further comprises: Let an expert provide guidance to multiple trainees at the same time and allow expert / trainee collaboration asynchronously. The expert writes multiple instructions, which are completed at once in chronological order; these instructions are written in a pre-acquired color and depth point cloud model of the workspace; the workspace is not acquired in real time, which enables trainees to submit their workspaces asynchronously.

6. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 1, characterized in that: The step S3 further comprises: In order to robustly calculate the difference between the depth buffer after instruction k-1 and the depth buffer of the working area after instruction k, take an average of 30 depth frames, filter out the difference below the minimum threshold, and calculate the maximum connected area R of non-zero depth difference; the center of the axis-aligned bounding box of the three-dimensional points within the maximum connected area R is approximately the center of the actual position of the object.

7. The mixed reality visual guidance method for remote collaboration between trainees and experts according to claim 6, characterized in that: After determining the center of the object's actual position: According to the actual position of the object, adjust the starting or ending point of the arrow of instruction k accordingly.

Images