Content provision supporting method, program, and system

Abstract

To provide a content provision method, a program, and a system that are utilized in a class with multiple viewers, using a head-mounted display (HMD) that supports providing of a content using a wide-range image captured from a predetermined viewpoint.SOLUTION: A method includes: determining relative positional relationships between avatars of a plurality of participants using an arrangement pattern table which determines arrangement of avatars corresponding to participants, centering an avatar of a first participant in a screen configuration of a HMD of the first participant; and arranging an avatar of a second participant other than the first participant while maintaining the relative positional relationship between the avatars; and centering the avatar of the second participant in a screen configuration of a HMD of the second participant, and offsetting avatars of participants other than the second participant while maintaining the relative positional relationship between the avatars.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention mainly relates to a content providing method, a content providing program, and a content providing system using a head mounted display. [Background technology]

[0002] In recent years, the use of Virtual Reality (VR) technology, which allows users to experience a virtual space created inside a computer as if it were real, has become widespread. A similar technology is Augmented Reality (AR), which adds various information to the realistic scenery you see in front of you.

[0003] In addition, there is mixed reality (called "mixed reality" in Japanese, hereafter abbreviated as "MR" or "mixed reality") technology, which combines VR and AR. For example, VR would be putting on a head mounted display (hereafter referred to as "HMD") and entering a computer-generated aquarium, AR would be actually seeing an explanation of the fish kept at the aquarium on the HMD screen, and MR would be seeing 3D computer-generated images of the fish on display at the aquarium on the HMD screen.

[0004] Amid such technological developments, in recent years, as the spread of COVID-19 has made it difficult to provide face-to-face education, attention has been drawn to the benefits of non-face-to-face education using digital technologies such as VR, as mentioned above.

[0005] For example, as the birthrate declines and the population ages, the lack of successors and human resources is becoming more pronounced in various fields, but experiential learning systems that utilize HMDs are beginning to be used to impart work know-how from skilled workers. In particular, by using panoramic images (hereafter referred to as "360-degree images") that match the participants' line of sight and head movements, it is possible to reproduce the sense of realism felt when working, which is expected to further enhance the learning effect.

[0006] For example, in medical training such as surgery, many students are unable to participate in real time, and there was a demand to film the surgery and use it for later learning. However, even if the surgery was filmed, there was also the issue that students could not get a sufficient learning effect by simply viewing the video later. More specifically, videos of surgical procedures contain only the minimum amount of audio and do not include any explanation of what is being done, for what purpose, or what techniques or tools are being used, so even if students watch the videos, they are not able to gain a sufficient learning effect.

[0007] In addition, there were issues such as the difficulty of handling videos of surgeries and other procedures from the standpoint of protecting patients' personal information and privacy, the ethical necessity for doctors to be cautious about using videos and images from surgeries and other procedures due to the nature of their involvement in human lives, and security management, making it difficult for medical institutions to outsource video editing. Furthermore, even if the medical institution that performed the surgery were to add explanations to the numerous surgical videos or edit the videos later, this would require an enormous amount of time and effort, so it would be desirable to be able to use the filmed videos as they are for subsequent learning.

[0008] If the filmed video were to be used as is, it would be possible to observe from the perspective of the doctor performing the surgery, but if there are multiple participants, including a doctor acting as an instructor and students acting as participants, then in order for the participants to understand the video without feeling uncomfortable, it would be necessary to devise a way to position the avatar corresponding to the doctor acting as the instructor and the avatars of the students acting as participants, as well as to devise a way to overlay markers, pointers, etc. displayed by the doctor acting as the instructor.

[0009] Furthermore, in fields where various types of specialized equipment are operated, such as in the construction industry when operating large construction machinery or working at heights, there are inherent dangers. From the perspective of preventing workplace accidents, it is therefore important to learn in advance the work content from the worker's perspective and acquire the operational know-how necessary to prevent danger. Or, when operating complex equipment such as manufacturing equipment or various control panels, it is necessary to impart operating procedures and expert know-how, as well as to provide safety training.

[0010] Even in such fields, it is expected that safety knowledge and know-how can be effectively learned by recording the work situation and then using an HMD to review the work content using the recorded video. In this case, the video is shot from the worker's perspective and can be observed from the worker's perspective, but in order for the participants to understand the video without feeling uncomfortable, it is necessary to devise a way to position the presenter's avatar and the participants' avatars, as well as to devise a way to overlay the presenter's markers, pointers, etc.

[0011] VR technology is a system that allows users to virtually experience a virtual space created by a computer as if it were reality. In VR technology, a virtual three-dimensional space is reproduced on a computer, and the view is constructed from the viewpoint of an avatar existing in the three-dimensional space. Image data of the three-dimensional space is generally displayed on an HMD, which is like a pair of goggles worn by the user. For this reason, for example, when using VR technology with two or more people, a virtual space is constructed inside the computer and views generated from the perspective of avatars in different positions are displayed.

[0012] For example, Patent Document 1, "Virtual reality image providing device and program," discloses a technology that constructs a virtual three-dimensional space on a computer and then displays a view constructed from the viewpoint of each avatar placed in the space. According to this technology, when an avatar faces a direction in which another avatar is present, the avatar can be displayed as if it is present based on the positional relationship in which the real participants are present.

[0013] In addition, when using 360-degree images in fields such as education, a technique is known that allows users to clarify parts of a 360-degree image by pointing out objects in the image with a virtual pointer or virtual marker, such as a virtual laser pointer. As a technology relating to a virtual laser pointer, for example, there is a technology disclosed in Patent Document 2.

[0014] Patent document 2, "Input support method, input support program, and input support device," discloses technology that detects the movement of a finger wearing a wearable device, detects an axis indicating the posture of the finger based on the detected finger movement, and displays a virtual laser pointer linked to the detected axis and the trajectory of the axis on a head-mounted display.

[0015] According to the technology of Patent Document 2, the laser light of a virtual laser pointer can be displayed on a 360-degree image in accordance with the movement of a finger. [Prior art documents] [Patent documents]

[0016] [Patent Document 1] Re-tabled publication No. 2018-198777 [Patent Document 2] JP 2016-118929 A Summary of the Invention [Problem to be solved by the invention]

[0017] However, the technology described in Patent Document 1 requires constructing a virtual three-dimensional space on a computer and then generating a 360-degree image seen from the viewpoint of each avatar, so it is not a technology that can be applied when using 360-degree images that have already been taken from a specific viewpoint.

[0018] More specifically, if the surrounding rectangular area is a three-dimensional space constructed on a computer, as shown in Figure 6, when A, B, and C are participating, A's virtual reality space from A's viewpoint, B's virtual reality space from B's viewpoint, and C's virtual reality space from C's viewpoint are further constructed, and then a view screen from A's viewpoint, a view screen from B's viewpoint, and a view screen from C's viewpoint are generated, so this cannot be applied to applications where multiple people participate using a 360-degree image from a single viewpoint.

[0019] To make it easier to understand, let us use Figure 7 to illustrate that the virtual reality space of A from the viewpoint of A, the virtual reality space of B from the viewpoint of B, and the virtual reality space of C from the viewpoint of C each generate separate 360-degree images to display the views. For this reason, it is clear that the technology of Patent Document 1 cannot be applied to cases where a 360-degree image captured from a single predetermined viewpoint is used.

[0020] Furthermore, while the technology of Patent Document 2 describes the display of laser light when a virtual laser pointer is operated with a finger, it merely describes the display method as seen from the viewpoint of an operator operating the virtual laser pointer, and does not describe how, when multiple participants participate, the laser light of a virtual laser pointer emitted by an operator is displayed from the viewpoint of the other participants.

[0021] In particular, when using an image taken from a single viewpoint, other participants also need to use the same image taken from the same viewpoint to display the laser light emitted from the operator's virtual laser pointer, so the challenge was how to process it so that the display would not look unnatural.

[0022] Therefore, in the present invention, rather than constructing a three-dimensional space on a computer and then displaying a view screen from the viewpoint of each avatar as in typical VR technology, an image captured from a single viewpoint is used, and the object is to display a natural view by positioning each avatar at the center while maintaining the positional relationship between the avatar and the centrally positioned avatar in relation to the other avatars.

[0023] Furthermore, in the present invention, while using an image captured from a specific viewpoint, when multiple avatars are arranged, when the host's avatar operates a virtual laser pointer to emit laser light, in the view display of another participant, the host in a specific relative positional relationship as seen from the participant positioned at the center is positioned at an offset position as seen from the participant, and the laser light appears to be emitted from the host at that offset position, thereby making it possible to display a natural view. [Means for solving the problem]

[0024] In order to achieve the above object, the first invention provides: A method for supporting the provision of content using 180-degree to 360-degree images captured from a predetermined viewpoint in a system including a head mounted display and a hand controller, a means for detecting tilt of at least three axes, XYZ, in the head mounted display, a means for changing a display range of an image according to the tilt, and a means for detecting position information in the head mounted display as necessary, comprising: a placement pattern table for determining how to place participants, and a placement management table for managing the placement of the participants determined based on the placement pattern table; a placement information acquisition step of acquiring placement information (defined placement information or actual placement information) of two or more participants including at least participant I and participant J from the placement management table; an avatar generating step of generating a plurality of avatars corresponding to the plurality of participants; an avatar relative positional relationship determination step of determining a relative positional relationship of a plurality of avatars corresponding to the plurality of participants in the captured 180-degree or 360-degree image based on the acquired arrangement information of the plurality of participants; In the screen configuration of the head mounted display of the participant I among the plurality of participants, The avatar of participant I is placed in the center, an avatar arrangement determination step of arranging avatars of participants other than participant I, including participant J, while maintaining the relative positional relationship between the avatars of the plurality of participants determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of the participant J among the plurality of participants, The avatar of participant J is placed in the center, an avatar offset position determination step of offsetting and arranging avatars of participants other than participant J, including participant I, with respect to the arrangement of the avatar of participant J, while maintaining the relative positional relationship between the avatars of the multiple participants determined in the avatar relative positional relationship determination step; In the screen display of the head mounted display of the participant I, an avatar superimposition display step of superimposing and displaying avatars of participants other than the participant I on the 180-degree or 360-degree image captured from the predetermined viewpoint based on the avatar arrangement determined by the avatar arrangement determination step, with the avatar of the participant I at the center; In the screen display of the head mounted display of the participant J, an avatar offset superimposition display step of superimposing and displaying the avatars of the other participants at offset positions with the avatar of the participant J at the center on the 180-degree to 360-degree image captured from the predetermined viewpoint based on the avatar position determined by the avatar offset position determination step; The content provision support method is characterized by comprising:

[0025] This means that when a virtual three-dimensional space is constructed on a computer and multiple avatars are placed in the three-dimensional space, and it is not possible to generate a view from the perspective of each avatar, for example, when using a pre-recorded 360-degree video (where the view of each participant is the same) and multiple participants' avatars are placed, each participant will be able to watch without feeling uncomfortable with the others. More specifically, for example, if a pre-recorded 180-degree or 360-degree video of a surgery is available, the video is shot from a specific viewpoint, such as the viewpoint of a camera attached to the surgeon's head or the viewpoint of a camera attached to the operating table, and therefore it is inherently not possible to generate a new view by switching the viewpoint. Also, as explained in Figure 5 below, if a view that is shifted from the center is generated, there is the inconvenience of the space appearing distorted, and therefore it is usually not possible to position and use multiple avatars. However, according to the present invention, while multiple avatars of multiple participants are arranged in the video (information processing is performed by superimposing the avatars), the view from the viewpoint of each avatar displays a 180-degree or 360-degree video shot from the same viewpoint (the avatar is arranged so that it is in the center), and the position of the other avatars is offset while maintaining their relative positional relationship with the avatar viewing the view, which has the effect of enabling viewing without any sense of discomfort in the relative positions of the participants relative to each other.

[0026] The second invention is a gaze information acquisition step of acquiring information on the face direction or gaze direction of any one of the participants, the information corresponding to the inclination by a means for detecting the inclination of the XYZ three axes of the head mounted display of the any one of the participants; In the screen display of the head mounted display of the arbitrary one participant, a display range changing step of changing a display range of the 180-degree or 360-degree image taken from the predetermined viewpoint based on information on the face direction or gaze direction of the arbitrary one of the participants acquired by the gaze information acquiring step; The content provision support method according to the first aspect of the present invention is characterized by comprising:

[0027] This allows a participant to view an image of the desired display range within a 180-degree or 360-degree image captured from a specified viewpoint when the participant moves the head-mounted display up, down, left or right, thereby visualizing the relative positions of the avatars of each participant in the first invention.

[0028] The third invention is If there are at least two participants, I and J, In the screen configuration of the head mounted display of the participant I, The avatar of participant I is placed in the center, an avatar arrangement determination step of arranging the avatar of the other participant J while maintaining the relative positional relationship between the avatar of the participant I and the avatar of the other participant J determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of the participant J, The avatar of participant J is placed in the center, an avatar offset position determination step of offsetting the avatar of the other participant I from the position of the avatar of the participant J while maintaining the relative positional relationship between the avatar of the participant J and the avatar of the other participant I determined in the avatar relative positional relationship determination step; In the screen display of the head mounted display of the participant J, an offset superimposition display step of superimposing and displaying the avatar of the other participant I at an offset position while maintaining a relative positional relationship between the avatar of the participant J and the avatar of the other participant I; In the screen display of the head mounted display of the participant J, When the display range of the 180-degree or 360-degree image taken from the predetermined viewpoint is changed according to the acquired line-of-sight information of the participant J, if the avatar of the other participant I is present within the visual field range of the avatar of the participant J, The avatar of the other participant I is displayed at a predetermined offset position with the avatar of the participant J at the center; The content provision support method according to the second aspect of the present invention is characterized in that:

[0029] The third invention is a process carried out in cooperation with the first and second inventions. The first invention emphasizes the feature that, when other participants are present, the participant himself / herself is positioned at the center, while the other participants are offset while maintaining their relative positions relative to the participant. The second invention is based on the first invention and emphasizes the dynamic feature that, when the participant moves his / her line of sight and the view is reconstructed and displayed, if the avatar of another participant appears in the field of view, the participant himself / herself is positioned at the center, while the avatar of the other participant is superimposed and displayed at an offset position while maintaining their relative positional relationship. Meanwhile, the third invention emphasizes the feature that, when participant J faces in the direction of participant I's avatar on the screen display, participant I can actually be seen to be displayed at an offset position.

[0030] The fourth invention is If the participant has at least three cards (I, J, and K), In the screen configuration of the head mounted display of the participant I, The avatar of participant I is placed in the center, an avatar arrangement determination step of arranging the avatars of the other participants (J, K, . . . ) while maintaining the relative positional relationship between the avatar of the participant I and the avatars of the other participants (J, K, . . . ) determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of the participant K, The avatar of the participant K is placed at the center of an avatar offset position determination step of determining, while maintaining the relative positional relationship between the avatar of the participant K and the avatars of the other participants (I, J, . . . ) determined in the avatar relative positional relationship determination step, the avatars of the other participants (I, J, . . . ) are offset from the position of the avatar of the participant K; In the screen display of the head mounted display of the participant K, an offset display superimposition step of superimposing and displaying avatars of the other participants (I, J, . . . ) at offset positions while maintaining a relative positional relationship between the participant K and the other participants (I, J, . . . ); In the screen display of the head mounted display of the participant K, When the display range of the 180-degree or 360-degree image taken from the predetermined viewpoint is changed according to the acquired line-of-sight information of the participant K, if the avatar of the other participants (I, J, ...) is present within the visual field range of the avatar of the participant K, The avatars of the other participants (I, J, ...) are displayed at predetermined offset positions with the avatar of the participant K at the center; The content provision support method according to the second aspect of the present invention is characterized in that:

[0031] While the third invention is "the processing when there are two or more participants, I and J," the fourth invention describes "the processing when there are three or more participants, I, J, and K," and aims to clarify the invention by describing it from multiple perspectives using a so-called numerical sequence description.

[0032] The fifth invention is When there are two or more participants, including any one participant and another participant, In a screen configuration of a head mounted display of any one of the plurality of participants, The avatar of the arbitrary one of the participants is placed at the center, an avatar arrangement determination step of arranging the avatar of the other participant while maintaining the relative positional relationship between the avatar of the arbitrary one participant and the avatar of the other participant determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of one of the other participants, An avatar of one of the other participants is placed at the center, an avatar offset position determination step of, while maintaining the relative positional relationship between the participants determined in the avatar relative positional relationship determination step, offsetting and arranging an avatar other than the avatar of one of the other participants with respect to the arrangement of the avatar of the one of the other participants; In the screen display of the head mounted display of any one of the participants, an avatar superimposition display step of superimposing and displaying the avatar of the other participant while maintaining a relative positional relationship between the avatar of the arbitrary one participant and the avatar of the other participant; In the screen display of the head mounted display of one of the other participants, When the display range of the 180-degree or 360-degree image taken from the predetermined viewpoint is changed according to the line of sight information of one of the other participants, if an avatar other than the one of the other participants is present within the field of view of the avatar of the one of the other participants, The avatars of the other participants other than the one participant are displayed at a predetermined offset position with the avatar of the one participant among the other participants as the center; The content provision support method according to claim 2 is characterized in that:

[0033] While the third invention describes "the processing when there are two or more participants, at least I and J," and the fourth invention describes "the processing when there are three or more participants, at least I, J, and K," the fifth invention describes the number of participants as multiple and in a more general manner, such as any one participant and another participant, rather than using specific symbols such as participants I, J, and K.

[0034] The sixth invention is When at least one participant I among the plurality of participants is provided with a hand controller and operates the hand controller, the hand controller is provided with a tilt detection means for detecting tilts of at least three axes, X, Y and Z; a virtual laser pointer generating step of generating a virtual laser pointer corresponding to the hand controller in association with an avatar of participant I who operates the hand controller; In the screen display of participant I who operates the hand controller, a light emission coordinate acquisition step of acquiring light emission coordinates (RAZER-ORG-StartPoint) of a point where the generated virtual laser pointer emits laser light; In the screen display of participant I who operates the hand controller, a laser light superimposition display step of superimposing and displaying an optical axis of a laser light emitted from the light emission coordinates of the generated virtual laser pointer on a 180-degree or 360-degree image captured from the predetermined viewpoint; detecting tilts of three axes, XYZ, in response to an operation of the hand controller and tilting the generated virtual laser pointer; changing an optical axis of a laser light emitted from the virtual laser pointer in accordance with a tilt of the hand controller; In the screen display of participant I who operates the hand controller, a target coordinate acquisition step of acquiring a target coordinate (RAZER-ORG-TargetPoint) of a target point of an optical axis of the laser light emitted from the virtual laser pointer; The content provision support method according to the first aspect of the present invention is characterized by comprising:

[0035] The sixth invention corresponds to the operation of a virtual laser pointer when at least one participant I among a plurality of participants is equipped with a hand controller and operates it, and when the laser light emitted from the virtual laser pointer is irradiated onto a specified object, the emission coordinates of the point where the laser light is emitted in a 180-degree or 360-degree image and the arrival coordinates of the point where the optical axis of the laser light reaches are grasped in preparation for the next stage of coordinate conversion processing.

[0036] The seventh invention is When there are at least two participants, I and J, and participant I has a hand controller and operates a virtual laser pointer corresponding to the hand controller, the process is as follows: In the screen configuration of the head mounted display of the participant I, The avatar of participant I and the virtual laser pointer are placed in the center, an avatar arrangement determination step of arranging the avatar of the other participant J while maintaining the relative positional relationship between the avatar of the participant I and the avatar of the other participant J determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of the participant J, The avatar of participant J is placed in the center, an avatar & virtual laser pointer offset arrangement determination step of offsetting and arranging the avatar of the other participant I and the virtual laser pointer of the participant I while maintaining the relative positional relationship between the participant J and the participant I determined in the avatar relative positional relationship determination step; In the screen display of the head mounted display of the participant I, acquiring a light emission coordinate (RAZER-ORG-StartPoint) of a light emission point and a target coordinate (RAZER-ORG-TargetPoint) of a target point of a light axis of a laser light emitted from a virtual laser pointer of the avatar of the participant I; Based on the acquired light emission coordinates (RAZER-ORG-StartPoint) and the relative positional relationship between the participant J and the participant I, In the screen display of the head mounted display of the participant J, A step of calculating the emission coordinates (RAZER-OFFSET-StartPoint) of the emission point of the laser light of the virtual laser pointer of the avatar of participant I arranged with the offset; In the screen display of the head mounted display of the participant J, generating an optical axis of the laser light based on the calculated offset emission coordinate (RAZER-OFFSET-StartPoint) and a destination coordinate (RAZER-ORG-TargetPoint) of a destination point of the optical axis of the laser light; In the screen display of the head mounted display of the participant J, superimposing and displaying the optical axis of the generated laser light on the 180-degree or 360-degree image captured from the predetermined viewpoint so that the laser light is emitted from the calculated offset light emission coordinates (RAZER-OFFSET-StartPoint) toward the same coordinates as the arrival coordinates (RAZER-ORG-TargetPoint) on the screen display of the head mounted display of the participant I; The content provision support method according to the sixth aspect of the present invention is characterized in that:

[0037] In a seventh aspect of the present invention, when there are at least two participants, I and J, and when participant I has a hand controller and operates a virtual laser pointer corresponding to the hand controller, On the screen display of participant I's head mounted display, the emission coordinates of the emission point and the arrival coordinates of the arrival point of the optical axis of the laser light emitted from the virtual laser pointer of participant I's avatar are acquired, On the screen display of participant J's head mounted display, participant J is positioned at the center, and participant I is offset taking into consideration the relative positional relationship between participant J and participant (I). The emission point of the laser light of the virtual laser pointer operated by participant I is also relocated to a similarly offset position. On the other hand, the arrival point of the optical axis of the laser light is determined using the same arrival coordinates as the coordinates on the screen display of participant I, and the optical axis of the laser light connecting the offset emission coordinates and the original arrival coordinates is generated and displayed superimposed on the 180-degree or 360-degree image, thereby enabling participant J to view the image without any discomfort.

[0038] The eighth invention is In a case where there are at least three participants, I, J, and K, and participant I has a hand controller and operates a virtual laser pointer corresponding to the hand controller, In the screen configuration of the head mounted display of the participant I, The avatar of participant I is placed in the center, an avatar arrangement determination step of arranging the avatar of the other participant J while maintaining the relative positional relationship between the avatar of the participant I and the avatar of the other participant J determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of the participant K, The avatar of the participant K is placed in the center, an avatar & virtual laser pointer offset arrangement determination step of offsetting and arranging the avatars of the other participants (I, J, . . . ) and the virtual laser pointer of the participant I while maintaining the relative positional relationship between the participant K and the other participants (I, J, . . . ) determined in the avatar relative positional relationship determination step; In the screen display of the head mounted display of the participant I, A step of acquiring a target coordinate (RAZER-ORG-TargetPoint) of a target point of an optical axis of a laser light emitted from a virtual laser pointer of the avatar of the participant I; In the screen display of the head mounted display of the participant K, A step of acquiring the emission coordinates (RAZER-OFFSET-StartPoint) of the emission point of the laser light of the virtual laser pointer of the avatar of participant I arranged with the offset; In the screen display of the head mounted display of the participant K, and generating an optical axis of the laser light based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of the avatar of participant I positioned with an offset and the arrival coordinates (RAZER-ORG-TargetPoint) of the arrival point of the optical axis of the laser light, In the screen display of the head mounted display of the participant K, superimposing the optical axis of the laser light on the 180-degree or 360-degree image captured from the predetermined viewpoint so that the laser light is emitted from the emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of the avatar of participant I, which is positioned at the offset, toward the same coordinates as the arrival coordinates (RAZER-ORG-TargetPoint) acquired on the screen display of participant I's head mounted display; The content provision support method according to the sixth aspect of the present invention is characterized in that:

[0039] The eighth invention is In a case where there are at least three participants, I, J, and K, and participant I has a hand controller and operates a virtual laser pointer corresponding to the hand controller, On the screen display of participant I's head mounted display, the emission coordinates of the emission point and the arrival coordinates of the arrival point of the optical axis of the laser light emitted from the virtual laser pointer of participant I's avatar are acquired, On the screen display of participant K's head mounted display, participant K is positioned at the center, and participant I is offset taking into account the relative positional relationship between participant K and participant I. The emission point of the laser light of the virtual laser pointer operated by participant I is also relocated to a similarly offset position, and the arrival point of the optical axis of the laser light is determined using the same arrival coordinates as the original coordinates on the screen display of participant I, and the optical axis of the laser light connecting the offset emission coordinates and the original arrival coordinates is generated and displayed superimposed on the 180-degree or 360-degree image, thereby enabling participant K to view the image without any discomfort. In short, the eighth invention basically describes the processing content for an example in which participant K participates in addition to the two or more participants I and J in the seventh invention, and aims to clarify the invention by describing it from multiple perspectives using a so-called numerical sequence description.

[0040] The ninth invention is When there are two or more participants, including any one participant and another participant, In the screen configuration of the head mounted display of any one of the participants, Placing an avatar of the arbitrary one of the participants at the center of a 180-degree or 360-degree image taken from the predetermined viewpoint; an avatar arrangement determination step of arranging the avatar of the other participant while maintaining the relative positional relationship between the avatar of the arbitrary one participant and the avatar of the other participant determined in the avatar relative positional relationship determination step; In the screen configuration of the head mounted display of one of the other participants, an avatar & virtual laser pointer offset placement determination step of placing an avatar of one of the other participants at the center of the 180-degree or 360-degree image captured from the predetermined viewpoint, while maintaining the relative positional relationship between the other participants and the one of the participants equipped with a hand controller determined in the avatar relative positional relationship determination step, and placing an offset of the avatar of the one of the other participants and a virtual laser pointer of the one of the other participants; In the screen display of the head mounted display of any one of the participants, acquiring a target coordinate (RAZER-ORG-TargetPoint) of a target point of an optical axis of a laser light emitted from a virtual laser pointer of an avatar of the arbitrary one of the participants; In the screen display of the head mounted display of one of the other participants, A step of acquiring a luminous coordinate (RAZER-OFFSET-StartPoint) of a luminous point of a laser beam of a virtual laser pointer of an avatar of any one of the participants arranged with an offset; In the screen display of the head mounted display of one of the other participants, generating an optical axis of the laser light based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of the avatar of any one of the participants arranged with an offset and the arrival coordinates (RAZER-ORG-TargetPoint) of the arrival point of the optical axis of the laser light; In the screen display of the head mounted display of one of the other participants, superimposing an optical axis of a laser beam on a 180-degree or 360-degree image captured from the predetermined viewpoint so that a laser beam is emitted from the emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of the avatar of any one of the participants positioned with the offset toward the same coordinates as the arrival coordinates (RAZER-ORG-TargetPoint) acquired on the screen display of the head mounted display of the any one of the participants; The content provision support method according to the sixth aspect of the present invention is characterized in that:

[0041] While the seventh invention describes "the processing when there are two or more participants, at least I and J," and the eighth invention describes "the processing when there are three or more participants, at least I, J, and K," the ninth invention describes the number of participants in a more general manner, such as any one participant and another participant, without using specific symbols such as participants I, J, or K, as the seventh invention describes "the processing when there are two or more participants, at least I, J, and K,".

[0042] The tenth invention is In the case where one of the participants is the presenter (host) and the other participants are students (guests), In order to arrange the avatars of the participants in positions suitable for the presenter to explain to the participants as students by using the 180-degree or 360-degree image taken from the predetermined viewpoint without relying on the actual positional relationship of the participants when the number of participants (guests) as the students increases, a placement pattern table for determining how to place the participants, and a placement management table for managing the placement of the participants who have joined based on the placement pattern table; In the arrangement pattern table, How participants will be arranged according to the order in which they join will be defined, distinguishing between participants as explainers (hosts) and participants as students (guests), and The avatars of other participants are not placed in the direction in which the object to be explained exists in front of each participant. and updating an arrangement management table for managing the arrangement of the participants by referring to the arrangement pattern table according to the order in which the participants joined; To enable smooth explanations even when multiple participants (guests) participate as students by preventing the object to be explained from being hidden by the participants; The content provision support method according to any one of the first to ninth aspects of the present invention is characterized in that:

[0043] The tenth invention is an innovation regarding the placement of participants in order to promote the effect of providing surgical videos, videos explaining how to operate equipment, and the like, with the assumption that the object being explained is located in the front direction (for example, at 11 to 1 o'clock) when viewed from the center of the 180-degree or 360-degree image (see the ☆ marks in Figures 13 and 26), and thus providing the content for explanatory or educational purposes. The tenth invention provides an arrangement pattern table that defines how participants are arranged according to the order in which they joined, distinguishing between participants as presenters (hosts) and participants as students (guests). The arrangement pattern is such that, in the view display of each participant, the avatar of that participant is arranged at the center of the 180-degree or 360-degree image, and the avatars of other participants are not arranged directly in front of the participant, thereby preventing the avatars of other participants from overlapping with the object (and laser light) being explained, making it possible to give a smooth explanation.

[0044] In addition, the eleventh invention is a system invention corresponding to the first invention (method invention) in which each step of the first invention (method invention) is replaced with technical means. Similarly, the twelfth to twentieth inventions are system inventions corresponding to the second to tenth inventions (method inventions). A twenty-first aspect of the present invention is a program for causing a computer to execute the content delivery support method according to any one of the first to tenth aspects of the present invention. [Brief description of the drawings]

[0045] [Figure 1] FIG. 1 is a diagram showing an example of a 180 degree to 360 degree image. [Diagram 2] FIG. 1 is a diagram showing an outline of the principle of generating a 180-degree or 360-degree image. [Diagram 3] FIG. 1 is a diagram serving as a premise for explaining a stereoscopic image of 180 degrees or 360 degrees, and is a diagram for explaining binocular parallax. [Figure 4]FIG. 1 is a diagram showing an outline of a method for generating a stereoscopic image of 180 degrees to 360 degrees. [Diagram 5] FIG. 1 is a diagram showing the characteristics of 180-degree to 360-degree images. [Figure 6] FIG. 1 is a diagram showing an example of a conventional technology, which shows a technology for constructing a virtual three-dimensional space on a computer and then displaying a view composed of the viewpoints of each avatar placed in the space. [Figure 7] FIG. 1 is a diagram showing an example of a conventional technology, which shows a technology for constructing a virtual three-dimensional space on a computer and then displaying a view composed of the viewpoints of each avatar placed in the space. [Figure 8] This figure shows the basic concept of the present invention. When using images taken from a specific viewpoint, the avatars of multiple participants are positioned at the center since the same image is used, and although the images in each view are the same, by devising a way to handle the relative positions of participants with other participants, participants have the feeling that they are in different positions, which is the basic principle that allows for a realistic training session. [Figure 9] FIG. 1 is a diagram showing an example of a conventional technique for displaying a virtual laser pointer. [Figure 10] FIG. 1 is a diagram showing an example of a display of laser light from a virtual laser pointer in the prior art. [Figure 11] FIG. 2 is a diagram showing the basic concept of displaying the laser light of a virtual laser pointer in the present invention. [Figure 12] This figure shows the basic concept of the offset placement processing of the avatar of the present invention when using a 180-degree or 360-degree image taken from a specified viewpoint, and the coordinate conversion processing related to the display of the optical axis of the laser light of a virtual laser pointer. [Figure 13] FIG. 11 is a diagram showing an example of an arrangement of participants when the number of participants is increasing, which is effective for using a 360-degree image captured from a specific viewpoint as educational content in the present invention. [Figure 14]FIG. 1 shows an example of a system configuration diagram of the present invention, which is an example of a case where processing is basically performed within the HMD without requiring a server or external device. [Figure 15] FIG. 1 is a diagram showing an example of a system configuration diagram of the present invention, which is an example of a case in which a server (including a cloud) or an external device is used. [Figure 16] FIG. 1 is a diagram showing an example of a functional block diagram of the present invention, which shows an example of a case where processing is basically performed within the HMD without requiring a server or external device. [Figure 17] FIG. 1 is a diagram showing an example of a functional block diagram of the present invention, which is an example of a case in which a server (including a cloud) or an external device is used. [Figure 18] FIG. 11 illustrates an example of a user management table. [Figure 19] FIG. 13 is a diagram showing an example of an avatar management table for managing images of avatar appearances. [Figure 20] This is a table for managing images such as 180-degree or 360-degree images. [Figure 21] FIG. 13 is a diagram showing an example of a ROOM management table for managing the start date and time, end date and time, participants, images used, etc. of a generated ROOM. [Figure 22] FIG. 11 is a diagram showing an example of a placement pattern table in which placement patterns of participants are registered when the number of participants is increasing, which is effective for utilizing a 360-degree image captured from a specified viewpoint in the present invention. [Figure 23] FIG. 13 is a diagram showing an example of an arrangement management table for managing the arrangement of participants when the number of participants increases in the present invention. [Figure 24] FIG. 13 is a diagram showing an example of the relationship between the positions of real participants and their placement on a 180-degree or 360-degree image. [Diagram 25] This figure shows the screen display for each participant as the number of participants increases in the present invention, and shows an example of a case where a guest (participant J) joins in a 3-column B pattern after the host (participant I) joins. [Figure 26]This figure shows the screen display for each participant as the number of participants increases in the present invention, and shows an example in which a host (participant I) joins, then a guest (participant J) joins in a 3-column B pattern, and then another guest (participant K) joins. [Figure 27] FIG. 27 is a diagram showing an example of a screen display for a guest (participant L) when the guest further participates in addition to the event shown in FIG. 26. [Figure 28] In the present invention, when a guest (participant J) participates in addition to a host (participant I), this figure shows an example of the screen display of the guest (participant J) when the host (participant I) operates a virtual laser pointer to emit laser light. [Figure 29] In the present invention, when a guest (participant J) and participant K also participate in addition to a host (participant I), this figure shows an example of the screen display of the guest (participant J) and guest (participant K) when the host (participant I) operates a virtual laser pointer to emit laser light. [Diagram 30] FIG. 30 is a diagram showing an example of a screen display for a guest (participant L) when the guest further participates in addition to the case shown in FIG. 29. [Diagram 31] FIG. 4 is a flowchart showing an example of each processing step of an image registration process according to the present invention. [Diagram 32] FIG. 2 is a flowchart showing an example of an outline of the overall processing flow according to the present invention. [Diagram 33] FIG. 11 is a flowchart of the present invention showing an example of each processing step for controlling how the participants are displayed on the host's screen and how the host is displayed on the guest's screen when a guest joins after a host has joined, taking into consideration the relative positions of the host and guest while using 180-degree or 360-degree images taken from a specified viewpoint. [Diagram 34]FIG. 1 is a flowchart of the present invention, showing an example of each processing step for controlling how the laser light of the virtual laser pointer operated by the host is displayed on the host's screen display and how the laser light of the virtual laser pointer operated by the host is displayed on the guest's screen display, taking into consideration the positional relationship between the host and guest, when the host operates a virtual laser pointer, using a 180-degree or 360-degree image captured from a specified viewpoint.

[0046] <Terminology explanation> In the present invention, VR is an abbreviation for Virtual Reality (called "virtual reality" in Japanese), and is also called "virtual reality." In this invention, AR is an abbreviation of Augmented Reality, also known as "Augmented Reality." AR is a technology that adds various information to the realistic scenery you see in front of you. In this invention, MR is an abbreviation for mixed reality (called "mixed reality" in Japanese), and is also called "mixed reality."

[0047] ◇In this invention, a 180-degree or 360-degree image refers to an image obtained by projecting (mapping) a flat image onto a spherical surface, the flat image being captured in a 180-degree or 360-degree direction using a wide-angle lens, a fisheye lens, multiple cameras, or a rotating camera (hereinafter abbreviated as "wide-angle lenses, etc."). Generally, images in the range of 180 degrees to 360 degrees are handled, so they will be referred to as "180-degree to 360-degree images," but images in a range narrower than 180 degrees, such as 120-150 degrees, can also be handled, and will be referred to as so-called "wide-range images." It also includes a "stereoscopic image" which is an image that appears three-dimensional by displaying different images to the left and right eyes.

[0048] ◇180-degree or 360-degree images taken from a specified viewpoint include images that have already been taken, as well as images being taken in real time. In the present invention, an image includes not only a still image but also a moving image. ◇In this invention, content refers to images of surgery, images of the operation of a work machine at a construction site, images of the operation of a machine tool, images of a factory tour, images guiding the course of a travel tour, etc.

[0049] In this invention, HMD is an abbreviation of Head Mounted Display, a general term for VR (virtual reality) display devices. It is a head-mounted display, a goggle-shaped display that is worn on the head so that it covers both eyes. The HMD can be a 3DoF type equipped with a three-axis acceleration sensor capable of detecting rotations around the XYZ axes, or a 6DoF type equipped with a three-axis gyro sensor capable of detecting movements in the XYZ directions. Alternatively, a 9DoF type equipped with a geomagnetic sensor may be used as needed. In addition to the typical goggle-type devices mentioned above, HMDs also include types that use an attachment to set a smartphone on the head, or glasses-type devices. The HMD may also be equipped with a speaker and earphones for outputting audio. An HMD is sometimes called a VR headset. ◇In this invention, the hand controller may be an independent unit with a built-in three-axis sensor, a wearable device worn on a finger, or a device that captures fingertip movement using an image sensor (CCD), or even a head-mounted display that captures the line of sight.

[0050] In the present invention, binocular parallax refers to the difference in the "position" or "viewing direction" of the images seen by the right eye and the left eye due to the human eyes being spaced apart by a distance of several centimeters. More specifically, it refers to the slight difference that occurs between the left and right retinal images when viewing an object with depth; in the human brain, this difference is fused to produce a single image perceived by both eyes, which is one of the factors that allows us to perceive depth and three-dimensionality. In the present invention, a stereoscopic image refers to a 180-degree to 360-degree image that appears three-dimensional and is generated using images captured by utilizing binocular parallax. ◇ In this invention, ROOM refers to a virtual room (space) set up by a participant (acting as an instructor) who participates as a host, and is a room (space) for explaining images selected by the host participant to the student participants. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] Examples of the present invention will now be described. Note that the screen displays and system configurations described are merely examples and can be applied to other screen displays and system configurations.

[0052] 1. Images used in the content provision support system of the present invention 1-1. 180-degree or 360-degree images First, the 180-degree or 360-degree image used in the content delivery support system of the present invention will be described with reference to FIGS. FIG. 1 is a diagram showing an example of a 180 degree to 360 degree image. In Figure 1, an image with a field of view of about 180 degrees is displayed from the front to both sides. Therefore, as you move away from the front, you can see that the image becomes significantly curved and distorted. Here, a 180-degree or 360-degree image refers to an image obtained by projecting (mapping) a flat image captured in a 180-degree or 360-degree direction using a wide-angle lens, fisheye lens, multiple cameras, or a rotating camera (hereinafter abbreviated as "wide-angle lenses, etc.") onto a spherical surface.

[0053] The 180-degree to 360-degree image includes a 360-degree panoramic video or a panoramic image, but does not necessarily have to include the entire 4π space of 360 degrees. It is sufficient for the image to cover a relatively wide range of space, and it may be an image that can be treated as a part of a 360-degree global image. It may also be an image with a viewing angle of 180 degrees (180-degree image), in which case it is expected that the resolution of the image can be improved by allocating the number of effective pixels to the image with a viewing angle of 180 degrees. Furthermore, for example, an image that includes a relatively wide range (wide-range image), such as an image of 270 degrees between 180 degrees and 360 degrees, or an image of 150 degrees that is less than 180 degrees, may be used.

[0054] 1-2. Principles of generating 180-degree or 360-degree images 180-degree and 360-degree images are also used in VR technology, which generates and creates virtual three-dimensional spaces on a computer, but they can also be generated by photographing images from the real world. A known method for generating 180-degree or 360-degree images is to project a portion of an image captured by a wide-angle lens or the like onto a predetermined plane and display it as a converted planar image. FIG. 2 is a diagram showing an outline of the principle of generating a 180-degree or 360-degree image.

[0055] FIG. 2 shows an image of generating a 180-degree or 360-degree image by projecting a flat image onto a virtual sphere. Images captured by a camera are flat images, even if they are taken with a wide-angle or fisheye lens, and a 180-degree or 360-degree image is generated by projecting (mapping) this onto a virtual sphere. Known projection methods include equirectangular projection, but the method is not limited to this.

[0056] As the 180-degree or 360-degree image, a stereoscopic image can be used in addition to a planar image. A stereoscopic image of 180 degrees or 360 degrees will be described with reference to FIG. 3 and FIG. FIG. 3 illustrates binocular disparity, which is the basic principle of stereoscopic vision. Binocular disparity refers to the difference in the "position" or "viewing direction" of the image seen by the right and left eyes due to the human eye being several centimeters apart. By displaying images with binocular disparity on the left and right display screens of the HMD, it is possible to perceive the depth and three-dimensionality of objects.

[0057] 1-3. Principles for generating stereoscopic images of 180 degrees or 360 degrees FIG. 4 illustrates the principle of generating a stereoscopic image of a wide range image such as a 180-degree or 360-degree image. According to Figure 4, even if two cameras are used to capture images with binocular parallax, the viewpoint will be different from when the image was captured except when facing directly ahead, and it is difficult to create different binocular parallax for all visual fields. Therefore, a method has been devised to create the correct parallax for all angles by arranging many cameras on a circle and synthesizing each image using a special method.

[0058] For the sake of simplicity, a stereoscopic image of a 360-degree image will be described as follows. A stereoscopic 360-degree image is a pair of 360-degree images with horizontal parallax between the eyes (binocular parallax). To generate a real-life stereoscopic 360-degree image from images taken from multiple viewpoints with a 360-degree camera, partial regions corresponding to the left-eye image and right-eye image are extracted from the collection of multi-view images, and then the left-eye image and right-eye image are stitched together over 360 degrees. By displaying the generated left-eye 360-degree image and right-eye 360-degree image on the left and right display screens of the head-mounted display, a 360-degree image that appears three-dimensional can be provided.

[0059] 1-4. Issues when using 180-degree or 360-degree images Next, a problem that arises when using 180-degree or 360-degree images that have been captured and generated from a specific viewpoint as described above will be described. Since the present invention is intended for educational and training applications such as providing explanations using 180-degree or 360-degree images taken from a specified viewpoint, it is by its nature not possible to construct a virtual three-dimensional space on a computer and freely generate views from the viewpoints of multiple participants' avatars. Furthermore, if an attempt is made to generate a view from the viewpoint of another participant using a 180-degree or 360-degree image taken from a specific viewpoint, problems will arise as shown in FIG. 5, which will be explained below.

[0060] FIG. 5 is a diagram showing the characteristics of 180-degree to 360-degree images. As mentioned above, a 180-degree or 360-degree image refers to an image obtained by projecting (mapping) a planar image captured in a 180-degree or 360-degree direction using a wide-angle lens or the like onto a virtual spherical surface. Here, in particular, when considering an image that is expanded onto a virtual sphere based on an image taken from a specific viewpoint, as shown in the lower diagram of Figure 5, a problem arises in that in such an image, unless it is viewed from the center of the image taken, the space becomes distorted. In short, the challenge is that if the presenter's avatar is placed at the center and the avatars of the other participants (students) are placed off-center, and a view is generated from the students' perspective, the space will be distorted.

[0061] For this reason, there was the inconvenience that it was not suitable for applications such as arranging avatars of multiple participants and providing explanations using 180-degree or 360-degree images expanded onto a virtual sphere based on images taken from a specified viewpoint. On the other hand, if, as mentioned above, a virtual reality space is generated on a computer, avatars of multiple participants are placed in it, and a view from the viewpoint of each avatar is generated and displayed, the problem of spatial distortion does not arise (see Figures 6 and 7), but it requires a huge amount of time and cost to generate the content.

[0062] Therefore, in this invention, a 180-degree or 360-degree image is used, which is inexpensive and is expanded onto a virtual sphere based on an image that is being taken or has already been taken from a specified viewpoint, while providing a view that is undistorted even when multiple participant avatars are arranged. In addition, by devising ways to ensure consistency in the relative positions of the participants, to arrange the avatars with consideration for the object being explained, and to display the laser light when the presenter operates a virtual laser pointer to point to an object and explain, it is possible to have a large number of people view the 180-degree or 360-degree image and use it for training purposes.

[0063] 2. Basic Concept of the Content Delivery Support System of the Present Invention 2-1. Basic concept of the present invention regarding the placement of avatars of each participant when using 180-degree or 360-degree images taken from a specific viewpoint Next, the basic concept of avatar arrangement according to the present invention will be described with reference to FIG. 8 in comparison with the examples of the prior art shown in FIG. 6 and FIG. 6 and 7 are diagrams showing an example of a conventional technology, which shows a technology for constructing a virtual three-dimensional space on a computer and then displaying a view composed of the viewpoints of each avatar placed in the space.

[0064] According to Figure 6, a virtual reality space is constructed on a computer, and the virtual reality space seen by participants A, B, and C is generated from different perspectives for each participant and mapped onto a separate space (virtual sphere). In addition, Figure 7 is basically the same as Figure 6, but a separate figure is shown for each participant, showing the virtual reality space (virtual sphere) from viewpoints A, B, and C, respectively, and it can be seen that these are images from different viewpoints.

[0065] As described above, conventional techniques can generate a virtual reality space on a computer, place avatars of multiple participants, and generate and display a view from the viewpoint (space) of each avatar. However, in the present invention, the same image taken from a specific viewpoint (all participants are in the same space) is used, so if the participants are positioned off-center to generate views from different viewpoints for each participant, as in the conventional technology, distortion such as that shown in Figure 5 will occur.

[0066] Therefore, while the same image taken from a specified viewpoint is used for each participant, the avatar of each participant is positioned at the center of the 180-degree or 360-degree image when displaying the view of that participant's avatar, and the other participants are offset to a specified position while maintaining their relative positional relationship with the other participants, making it possible to use the same image taken from a specified viewpoint for multiple viewing and for use in a training session.

[0067] In short, rather than generating and displaying a view from the viewpoint of each participant's avatar as in conventional CG technology, the same image is used, but the positional relationship between each viewer and the other participants is maintained while offsetting the other participants, allowing for a natural viewing experience.

[0068] FIG. 8 shows the basic concept of the present invention. When using images captured from a specific viewpoint, the avatars of multiple participants are positioned at the center since the same image is used, and although the images in each view are the same, by devising ways to handle the relative positions of the participants, participants have the sensation that they are in different positions, thereby enabling a training session with a sense of realism.

[0069] According to FIG. 8, from the viewpoint of participant A, the avatar of participant A is positioned at the center, and the avatars of other participants B and C are positioned with a predetermined relative positional relationship defined. Furthermore, from the perspective of participant B, it can be seen that the avatar of participant B is positioned at the center, and the avatars of the other participants, A and C, are positioned offset from the avatar of participant B while maintaining the specified relative positional relationship defined above. Furthermore, from the perspective of participant C, it can be seen that the avatar of participant C is positioned at the center, and the avatars of the other participants, A and B, are positioned offset from the avatar of participant C while maintaining the specified relative positional relationship defined above.

[0070] In this way, in the present invention, since images (the same image) on the same virtual sphere are used, when displaying a certain participant on the screen, the avatar of that participant is placed at the center, and the avatars of the other participants are placed offset (while maintaining their relative positional relationships). In other words, this means that even the same participant's position changes depending on who is watching the screen (in this respect it differs from technology that creates a virtual three-dimensional space on a computer and positions avatars in it).

[0071] In short, by performing a process in which the viewer is always placed at the center of the screen display for that viewer and other participants are offset, it is possible to use the same image captured from a specified viewpoint, but by maintaining the positional relationship between that viewer and the other participants and offsetting the other participants for each viewer, allowing for a natural viewing experience.

[0072] 2-2. Basic concept of displaying the optical axis of the virtual laser pointer of the present invention when using a 180-degree or 360-degree image taken from a specific viewpoint Next, the basic concept of displaying the optical axis of the laser light of the virtual laser pointer of the present invention will be described with reference to FIGS. FIG. 10 is a diagram showing an example of a display of a laser beam from a virtual laser pointer in the prior art. Figure 10 shows an example in which participant A, who is playing the role of instructor, operates a virtual laser pointer to emit laser light, and the avatars of participant A, who is playing the role of instructor, and participants B and C, who are students, are lined up in a horizontal row.

[0073] As shown in the upper part of FIG. 10, participant A, who is playing the role of the instructor, will irradiate the laser light, so it is necessary to draw the image so that it is irradiated from the center of the image from participant A's viewpoint. On the other hand, in the present invention, the same image taken from a specific viewpoint is used for each participant, so from the viewpoint of student participant B, the optical axis of the laser light that has already been drawn is drawn as if it is irradiated from the center, resulting in an unnatural image for student participant B. Similarly, from the perspective of student Participant C, if no measures are taken, the same previously captured 360-degree image will be used, and the optical axis of the laser light that has already been drawn will be drawn as if it is shining from the center, resulting in an unnatural image for Student Participant C.

[0074] Therefore, in this invention, in addition to the above-mentioned ingenuity in the positioning of the avatars, the coordinates of the destination point of the laser light emitted by the participant operating the virtual laser pointer are used as is for the optical axis of the emitted laser light, and while maintaining the point of irradiation on the object being explained, the coordinates of the emission point are devised, so that even when the same image captured from a specified viewpoint is used, the irradiation of the laser light from the virtual laser pointer of the avatar of the participant operating the virtual laser pointer appears natural, making it possible for a large number of people to watch and use it like a training session.

[0075] FIG. 11 is a diagram showing the basic concept of displaying the laser light of the virtual laser pointer in the present invention. Here, the following explanation will be given on the assumption that participant A, who is playing the role of instructor, is operating the virtual laser pointer. As shown in the upper part of FIG. 11, participant A, who is playing the role of the instructor, irradiates the laser light, so that from participant A's viewpoint, the image is depicted as being irradiated from the center.

[0076] On the other hand, as shown in the middle part of FIG. 11, in the present invention, each participant uses the same image taken from a given viewpoint. From the viewpoint of participant B, the avatar of participant B is placed at the center, and the avatar of participant A is placed offset (while maintaining a relative positional relationship with the avatar of participant B), The virtual laser pointer operated by participant A is also offset in a similar manner, and the coordinates of the point where the laser light reaches are used as is in order to maintain the point where it is shone on the object being explained, but the coordinates of the point where the laser light is emitted are converted to offset coordinates (in the same way that participant A's avatar was offset), and by converting the coordinates of the optical axis of the laser light, it becomes possible for participant B to see as if the laser light is being shone from the direction of participant A's avatar, who is acting as the instructor.

[0077] Furthermore, as shown in the lower part of Figure 11, from the viewpoint of participant C, the avatar of participant A is offset (while maintaining a relative positional relationship with respect to the avatar of participant C), and the virtual laser pointer operated by participant A is also offset in a similar manner. While the coordinates of the arrival point of the laser light are used as is in order to maintain the point at which it is shone on the object being explained, the coordinates of the emission point of the laser light are converted to offset coordinates (in the same way as participant A's avatar is offset), and by converting the coordinates of the optical axis of the laser light, it becomes possible for participant C to see as if the laser light is being shone from the direction of the avatar of participant A, who is acting as the instructor.

[0078] 2-3. Basic concept of the offset placement process of the avatar of the present invention when using 180-degree or 360-degree images taken from a specific viewpoint, and the coordinate conversion process for displaying the optical axis of the laser beam of the virtual laser pointer In the above Chapter 2.1, the basic concept of the present invention regarding the arrangement of the avatars of each participant was explained. In that explanation, it was explained that since 180-degree or 360-degree images taken from the same viewpoint are used for all participants, in the view display of a certain viewer, the avatar of that viewer is arranged at the center to prevent spatial distortion (see FIG. 5), and the avatars of other participants are arranged at offset positions from that viewer while maintaining the relative positional relationship between them, enabling a natural viewing experience.

[0079] In addition, in the above mentioned Chapter 2.2, when the avatar of a participant acting as the instructor operates a virtual laser pointer to emit a laser beam, when the view of the other participants is displayed, the avatar of the other participants is placed at the center of the image (as a process explained in Chapter 2.1), so it was explained how the optical axis of the laser beam should be transformed to allow the viewer to view the video without feeling unnatural. In Section 2.3 of this chapter, the offset arrangement of the avatar in Section 2.1 and the specific contents of the coordinate conversion process of the optical axis of the laser light of the virtual laser pointer in Section 2.2 will be described below with reference to FIG.

[0080] FIG. 12 is a diagram showing the basic concept of the offset positioning process of the avatar of the present invention when using a 180-degree or 360-degree image taken from a specified viewpoint, and the coordinate conversion process for displaying the optical axis of the laser light of a virtual laser pointer. The following explanation will be given on the premise that when there are two participants, participant I who is an instructor and participant J who is a student, participant J is positioned to the right of participant I.

[0081] <Step 1> First, in step 1, from the viewpoint of participant I, participant J is placed to the right of participant I, and their respective coordinates are participant I (x1, y1, z1) and participant J (x2, y2, z2). Moreover, participant I, who plays the role of lecturer, operates a virtual laser pointer to emit laser light, points to an object (not shown) that is the subject of the explanation, and then gives an explanation.

[0082] In this case, it is natural that the coordinates of the original emission point of the laser light will be a certain distance away from the coordinates of participant I by the length of the hand, so in reality it is desirable to process it differently from the coordinates of participant I. However, for ease of explanation, the following explanation will assume that the coordinates are the same as the coordinates of participant I (x1, y1, z1). Here, the "emission coordinates", which are the coordinates of the point where this laser light is emitted, are the original emission coordinates determined by the operation of participant I who emitted the laser light, and will be called (RAZER-ORG-StartPoint) to give the meaning of the "original (org)" coordinates before the coordinate transformation is performed to offset and position participant I (when the view is later constructed from the viewpoint of participant J).

[0083] Similarly, the "arrival coordinates," which are the coordinates of the point where the laser light reaches, will be called (RAZER-ORG-TargetPoint) to give the sense that they are the original arrival coordinates determined by the operation of participant I who shone the laser light, and thus the "original (org)" coordinates. From the above, the laser light emitted by Participant I is composed of a line (optical axis) connecting the emission coordinates (RAZER-ORG-StartPoint) of (x1, y1, z1) to the arrival coordinates (RAZER-ORG-TargetPoint) of (x3, y3, z3).

[0084] <Step 2> Step 1 is explained from the perspective of participant I, whereas step 2 is explained below from the perspective of participant J. In step 2, the following process is performed as preprocessing before performing coordinate conversion processing of the laser light of the virtual laser pointer. First, in order to allow other participants to use a 180-degree or 360-degree image taken from a specific viewpoint, participant J's avatar is placed at the center (x1, y1, z1) instead of participant I's avatar.

[0085] On the other hand, the avatar of participant I is placed at a position (x4, y4, z4) offset from the avatar of participant J while maintaining the relative positional relationship between the avatar of participant I and the avatar of participant J, allowing participant J to watch without feeling uncomfortable. In this case, the coordinate conversion process for participant I is as follows:

[0086] When maintaining the relative positional relationship, (x4-x1)=(x1-x2) holds, so x4=x1+(x1-x2). Similarly, y4=y1+(y1-y2) z4 = y1 + (z1 - z2) Based on the coordinates (x1, y1, z1) of the original participant and the relative positional relationship between the participants, the coordinates (x4, y4, z4) of participant I after offset can be calculated.

[0087] <Step 3> In step 3, even when a 180-degree or 360-degree image (the same image as that of participant I) taken from a specific viewpoint is used, when participant I (his avatar) acting as the instructor operates a virtual laser pointer to shine laser light, we will explain how to display the view screen as seen from participant J's viewpoint so that participant J can view the content without feeling uncomfortable. When providing educational or explanatory content, it is envisioned that Participant I (or his avatar) acting as the instructor will use a virtual laser pointer to shine laser light onto objects in an image being explained, such as the treatment area in an image of surgery, or an operation panel in an image of equipment being operated.

[0088] Here, when the viewer is switched from participant I to participant J, it is desirable to use the arrival point (RAZER-ORG-TargetPoint) of the laser light irradiated on the object to be explained as it is without conversion. On the other hand, if the point of emission of the laser light emitted by participant I (or his / her avatar) remains in the same place as on the screen viewed by participant I, i.e., the emission coordinates (RAZER-ORG-StartPoint) near the center of the image, it will appear unnatural to viewer J, as if the light is being emitted from participant I himself, who is not actually holding the virtual laser pointer. Therefore, in the same way that the avatar of participant I is offset from the avatar of participant J on the viewing screen of participant J, it is desirable to perform processing to offset the emission coordinates of the laser light as well.

[0089] Here, the "emission coordinates", which are the coordinates of the point where the laser light is emitted, are the coordinates that are obtained by converting and offsetting the original emission coordinates (RAZER-ORG-StartPoint) operated by participant I who emitted the laser light, and then repositioning them. Therefore, to give the impression that they are "offset (OFFSET)" coordinates, we will call them (RAZER-OFFSET-StartPoint).

[0090] In this case, just as participant I's coordinates are converted from the original coordinates (x1, y1, z1) to the offset coordinates (x4, y4, z4), the optical axis of the laser light can be displayed as if it were emitted from the offset coordinates (RAZER-OFFSET-StartPoint), which are (x4, y4, z4), allowing participant J to view the image without feeling uncomfortable.

[0091] Therefore, on Participant J's viewing screen, the laser light emitted by Participant I was converted into a line (optical axis) connecting the post-offset emission coordinates (RAZER-OFFSET-StartPoint), which were (x4, y4, z4), to the arrival coordinates (RAZER-ORG-TargetPoint), which were (x3, y3, z3), and displayed. The same applies even if participant I is not visible on participant J's screen display; on participant J's screen display, it will appear as if the laser light from a virtual laser pointer is being shone from the direction in which participant I is located (see Figures 28 and 29).

[0092] 2-4. When providing 180-degree or 360-degree images taken from a specific viewpoint as educational content, what is the basic concept of the present invention regarding how to arrange participants as the number of participants increases? Next, with reference to FIG. 13, we will explain how participants should be arranged according to their order of participation in order to use 180-degree or 360-degree images taken from a specific viewpoint as educational content. FIG. 13 is a diagram showing an example of an arrangement of participants when the number of participants increases, which is effective for utilizing a 360-degree image captured from a predetermined viewpoint in the present invention. The following explanation will be given on the assumption that all the illustrations are shown from the viewpoint of participant A's avatar.

[0093] First, if participant A, who is playing the role of instructor, joins first, and A is positioned at the center, A's avatar will be placed at the center of the image from A's perspective (see Figure 13 (a)). Next, when student participant B wants to join, the question arises as to where to place participant B's avatar. The reason for this is that in this invention, when using a 180-degree or 360-degree image taken from a certain viewpoint to provide as educational content, when positioning avatars of the instructor and students to explain about an object in the image (such as the surgical site on a patient or the operation panel of a device), care must be taken to ensure that the avatars do not obstruct the object in the image.

[0094] For example, in the case of surgery videos, they are often shot from diagonally above the operating table, assuming the viewpoint of the surgeon performing the surgery. In such cases, the area being operated on will be slightly forward of the center of the image. With this in mind, for example, if we assume that the object to be explained (marked with a star) is located near the center of a 180-degree or 360-degree image, If, in the real-world positional relationship between participant A and participant B, participant B is located just before participant A at the 12 o'clock direction in relation to participant A in (a) of Figure 13, if participant B's avatar is positioned to reflect this real-world positional relationship, participant B's avatar will be placed at the position indicated by the dotted line, as shown in (b) of Figure 13, and the object will be hidden by participant B's avatar. In addition, when the 180-degree or 360-degree image is a surgical video, the area marked with a star (the surgical area to be explained) exists on a virtual sphere, and therefore exists at the point where it intersects with the virtual sphere when the gaze is directed slightly downward.

[0095] Therefore, in the present invention, the participants are positioned by using a 180-degree or 360-degree image taken from a certain viewpoint, without necessarily being bound by the actual positional relationships of the participants, and reconstructing the participants' position into one suitable for providing it as educational content. To explain this using Figure 13 (b), when participant B joins, it would be effective to place participant B's avatar to the right of participant A's avatar, at the 3 o'clock position as seen from participant A, avoiding the position of the object (marked with a star). Furthermore, if participant C then joins, it would be effective to place participant C's avatar to the left of participant A's avatar, at the 9 o'clock position from participant A's avatar, avoiding the position of the object (marked with a star) (see Figure 13 (c)).

[0096] Furthermore, if participant D then joins, it would be effective to place participant D's avatar to the left of participant A's avatar, at the 5 o'clock position from participant A's avatar, avoiding the position of the object (marked with a star) (see Figure 13 (c)). Although the explanation is omitted, the same applies to the cases where participant E and participant F also participate. As will be described later, in order to reconfigure the participant placement to suit the purpose of providing such educational or training content, various patterns of placement can be created using a placement pattern table (Figure 22) and a placement management table (Figure 23), etc.

[0097] 3. System configuration of the content provision support system of the present invention Next, the system configuration of the content delivery support system 600 of the present invention will be described with reference to FIGS. FIG. 14 is a diagram showing an example of a system configuration diagram of the present invention, which is an example of a case where processing is carried out within the HMD, basically without requiring a server or external device. FIG. 15 is a diagram showing an example of a system configuration diagram of the present invention, which is an example of a case where a server (including a cloud) or an external device is used. The content provision support system of the present invention can be realized by storing 180-degree or 360-degree image content in a memory unit within the HMD (100, 101, 102, ...) device and using a control means within the HMD (100, 101, 102, ...). Figure 14 shows the system configuration in this case.

[0098] FIG. 14 shows that it is possible to configure a content provision support system by utilizing the communication function between HMDs to synchronize the images (in the case of videos) viewed by each participant, and to transmit each other's facial orientation and movement, as well as the operation status of a virtual laser pointer using the hand controller 200. The communication function can be wireless LAN using Wi-Fi or other communication standards, as well as wireless communication using wireless communication standards such as WiFi that are not limited to LAN. The HMD may also include a microphone for inputting voice, and a speaker and earphones for outputting voice.

[0099] FIG. 15 also shows that it is possible to configure a content provision support system by storing 180-degree or 360-degree image content in server 300 and transmitting the content to each HMD via network 500 as necessary, or by centrally managing and transmitting the orientation and movement of each participant's HMD (100, 101, 102, ...) and the operation status of the virtual laser pointer using the hand controller 200.

[0100] Here, the server 300 includes not only a server device, but also a cloud or an external device. The terminal device 400 is also used to register images, register users, or set the environment on the server 300 . As described above, the system configuration of the content delivery support system 600 of the present invention can take various forms.

[0101] 4. Functional block diagram of the content provision support system of the present invention 4-1. System configuration overview Next, the functional blocks of the content delivery support system 600 of the present invention will be described with reference to FIGS. FIG. 16 is a diagram showing an example of a functional block diagram of the present invention, which shows an example in which processing is basically performed within the HMD without requiring a server or external device. FIG. 17 is a diagram showing an example of a functional block diagram of the present invention, and shows an example in which a server (including cloud) or an external device is used.

[0102] In Figures 16 and 17, various programs are stored in the memory unit of the HMD (100, 101, 102, ...) or server 300, and the programs are loaded into temporary storage means such as RAM (random access memory), and a central processing unit (not shown) executes each step of the program to configure each means of the control unit.

[0103] When the system configuration of FIG. 16 is adopted, content of 180-degree or 360-degree images is stored in the memory unit within the HMD (100, 101, 102, ...), and various processes are performed by the control unit within the HMD (100, 101, 102, ...). On the other hand, when the system configuration of FIG. 17 is adopted, the storage unit in the server 300 stores content of 180-degree or 360-degree images, and the control unit in the server 300 realizes various processes.

[0104] The differences between Figures 16 and 17 are that the memory unit and control unit are located in different places, and when there is no server, the HMD performs each control and generates the display screen, and even controls synchronization of the videos so that videos can be played at the same time between HMDs by communicating with each other, whereas when a server is present, processing is performed on the server side, the generated videos are transferred to each HMD, and the images are displayed on the HMD side. However, since the basic processing of the control unit and memory unit is the same, Figures 16 and 17 will be explained together below.

[0105] 4-2. Processing by Avatar-related Control Means In Figures 16 and 17, when participant J joins after participant I, the avatar-related control means first generates a view for participant I by placing participant I at the center, and then, in participant J's view, places participant J at the center and displays participant I offset, thereby providing control that allows for a natural viewing experience even when all participants are using wide-range images taken from the same viewpoint.

[0106] For example, the relative position information generating means refers to the arrangement pattern table in the memory unit to grasp the relative positional relationship of the avatars of the participants who joined in a specified order, and the avatar arrangement determining means determines the final arrangement of the participants' avatars based on the relative positional relationship and records it in the arrangement management table in the memory unit.

[0107] In addition, when displaying the view of another participant, the position of the other participant is offset using an avatar offset position determination means (coordinate transformation) to make the other participant appear to be present while maintaining the relative positional relationship with the other participants according to the direction of the gaze acquired by the gaze information acquisition means, thereby creating a sense of realism for the viewer.

[0108] 4-3. Processing by Virtual Laser Pointer Related Control Means In Figures 16 and 17, the virtual laser pointer-related control means detects the direction of the hand controller operated by the participant and adjusts the direction of the laser light of the virtual laser pointer, and also performs processing to offset and position the emission coordinates of the laser light of the virtual laser pointer when displaying the view of other viewers who are not operating the virtual laser pointer.

[0109] For example, an emission coordinate acquisition means acquires the emission coordinates of the virtual laser pointer, an arrival coordinate acquisition means acquires the arrival coordinates of the virtual laser pointer, a laser light generation means generates laser light based on the acquired emission coordinates (RAZER-ORG-StartPoint) and arrival coordinates (RAZER-ORG-TargetPoint), and a laser light superimposition means draws (superimposes) the laser light on the content image. In addition, when displaying the view of other participants (audiences) who are not operating the virtual laser pointer, the light emission coordinate conversion means performs coordinate conversion processing according to the offset amount of the coordinates of the participant who is placed offset from the original light emission coordinates, and calculates the light emission coordinates after the offset (RAZER-OFFSET-StartPoint).

[0110] 4-4. Processing by Screen Display Related Control Means 16 and 17, the screen display related control means changes the display range of the 180-degree or 360-degree image based on information from various sensors of the HMD, and the display screen generating means generates and displays a display screen for a specified range. If another avatar is present within the display range, the avatar is superimposed on the 180-degree or 360-degree image based on the positioning information obtained by performing the offset process described above. Furthermore, if the laser light of a virtual laser pointer operated by another participant is present within the display range, an avatar is superimposed on the 180-degree or 360-degree image based on the information positioned by performing the offset processing described above.

[0111] 4-5. Memory section In Figures 16 and 17, the memory unit includes avatar data including information such as the shape and color of the avatar, a placement pattern table and a placement management table regarding the placement of the participants' avatars, a relative position memory unit that records the relative positions of the participants' avatars, and a content memory unit that records 180-degree or 360-degree images of the surgery or equipment operation.

[0112] 4-6. Communications Department and Others In Figures 16 and 17, the HMD or server is provided with a communication unit, and the communication function can utilize wireless LAN using Wi-Fi or other communication standards, as well as wireless communication using wireless communication standards such as WiFi that are not limited to LAN. Although not shown, in Figs. 16 and 17, the HMD may be provided with an audio input unit (microphone) for inputting audio and an audio output unit (speaker or earphones) for inputting audio.

[0113] 5. About the various tables 5-1.User management table FIG. 18 is a diagram showing an example of a user management table, which is a table stored in the storage unit. In this table, for each participant (user), the HMD device ID, the user name, the user ID, the type of avatar selected by the user, and the like are recorded. Input of a user name and selection of an avatar may be performed by operating a physical switch on the HMD after displaying a virtual keyboard, tablet, or various selection screens on the HMD, or input and selection may be performed by reading finger movements using a camera on the HMD, or input and selection may be performed separately via a tablet or smartphone.

[0114] 5-2.Avatar Management Table FIG. 19 is a diagram showing an example of an avatar management table that is stored in the storage unit and that manages images of the appearances of avatars. In this table, the character ID, the image data of the avatar image, the storage destination information, and the like are recorded.

[0115] 5-3.Image Management Table FIG. 20 is a table stored in the storage unit, which is a table for managing images such as 180-degree to 360-degree images. This table records images of surgeries, operating heavy machinery, and operating control panels such as switchboards. Images are selected appropriately and read out, and by overlaying avatars and laser pointers, a sense of realism is created, making the viewer feel as if they are actually participating in the event.

[0116] 5-4.ROOM Management Table FIG. 21 is a diagram showing an example of a ROOM management table stored in the storage unit, which manages the start date and time, end date and time, participants, images used, etc. of a ROOM. ROOM refers to a virtual room (space) set up by the participant (acting as the instructor) who participates as the host, and is a room (space) where the host participant can explain images selected by him or her to the student participants. A ROOM is opened each time and closed once the image explanation is finished. This table records the start time and end time of the ROOM, the ROOM ID, the ID of the image used, the user ID of the participating host (instructor), the user ID of the participating guests (students), and, if necessary, the storage location of the lecture data, which is the lecture content of the ROOM.

[0117] 5-5. Placement pattern table FIG. 22 is a diagram showing an example of a placement pattern table stored in a storage unit, which is effective for utilizing a 360-degree image captured from a specified viewpoint, and which registers placement patterns of participants when the number of participants increases. In this table, information such as the placement ID, placement type, placement pattern, etc. is recorded. The placement pattern is defined according to the order of participation: 1 is the host participant (instructor) who opens the ROOM and joins first, 2 is the second participant (student) to join, etc. As the arrangement pattern, in addition to the default standard pattern, an arrangement pattern for when automatic adjustment is ON may be prepared, in which automatic adjustment is performed so that the avatar of each participant is not placed in front of other participants.

[0118] As an example, when the placement ID is H001, a placement that reflects the actual positional relationships is assumed, and when the placement ID is H031, the host (1) is in the center, the second participant is to the left of the host, the third participant is to the right of the host, the fourth participant is at the left end of the second row, etc. Or, as an example, for the arrangement ID H061, a U-shaped arrangement pattern of katakana characters is recorded as the character arrangement pattern with the instructor role number 1 as the base. It can be seen that in all placement patterns, care is taken to ensure that the avatars of other participants are not placed in close proximity to the front of the avatar of the participant in the front row. Here, if there are participants in the second row or later and the object to be explained is obscured by the avatar of a participant in the previous row, the participants may move so that the object to be explained is visible after each participant's avatar has been positioned. Alternatively, a placement pattern may be prepared in which the avatars of other participants are not placed in front of each participant (see the rightmost column in FIG. 22), and when automatic adjustment is ON, this placement pattern may be used. In particular, in the case of the layout pattern in the rightmost column of Figure 22, in all layout patterns, including the participants in the second row, no participants are placed in the area that corresponds to the front direction (from 11 o'clock to 1 o'clock) when viewed from the center of the screen, and it can be seen that consideration has been given to ensure that the object being explained is not hidden by the participants.

[0119] In addition, when the positioning ID is H081 (Face-to-Face A), the avatars of other participants are positioned in front of the participant. However, for example, in the case of a surgery video, the surgical site to be explained is located at a point where the gaze is directed slightly downward (slightly forward of the center of the image), so even if the avatars of other participants are positioned in front of the participant, it is assumed that the participant's avatars will not block the object to be explained.

[0120] 5-6. Placement management table FIG. 23 is a diagram showing an example of an arrangement management table stored in the storage unit, which manages the arrangement of participants when the number of participants increases. For example, if the ROOM ID is ROOM0456, the host will be the first to join and launch the ROOM, guest 1 will be the second to join, guest 2 will be the third to join, guest 3 will be the fourth to join, and so on. In addition, the host participant has selected the 3-row A type of arrangement in accordance with the image explained in the ROOM. In this case, the second participant (guest 1) is arranged to the right of the host, the third participant (guest 2) to the left of the host, the fourth participant (guest 3) at the far left of the row behind the host, and so on.

[0121] 6. Relationship between real participants and viewing space 6-1. About Figure 24(1) Fig. 24 is a table stored in the storage unit, showing an example of the relationship between the positions of real participants and their placement on a 180-degree or 360-degree image. Fig. 24 illustrates an example in which participant I is the host who plays the role of lecturer, and participants J and K are guest students. Of these, Figure 24(1) shows an example of a case where participants are present in the same classroom and their positions are changed to a pattern that is easier to explain, instead of their actual positions. In this case, if the real-world layout were used as is for the avatar layout, and the object to be explained (such as the surgical site or operation panel) is located near the center of the image, participant J's avatar would be covered by the object. Therefore, we envision a use in which the layout is not based on the real-world layout, but instead is based on one of the layout patterns provided by the system.

[0122] 6-2. About Figure 24(2) FIG. 24(2) shows an example of a case where participants are arranged to reflect their actual positions when they are present in the same classroom. When the number of participants is small, the host participant who acts as the lecturer can appropriately guide and rearrange the participants' positions so that they do not overlap the objects being explained, in line with the lecture using images, so this system is intended for such use.

[0123] 6-3. About Figure 24(3) Fig. 24(3) shows an example of a case where participants are located in remote locations and the actual positional relationship of the participants cannot be used, so the arrangement pattern is changed to one that is easier to explain. The contents of the arrangement process are the same as those in Fig. 24(1), so the explanation is omitted.

[0124] 7. Screen display (focusing on how the avatar appears) Next, the screen display of the content delivery support system of the present invention will be described with reference to Figs. 25, 26 and 27 regarding the offset processing of the avatar, focusing on how the avatar appears. 25, 26, and 27, an example will be described in which participants are arranged in an arrangement pattern of 3 columns B (see arrangement ID=H032 in FIG. 22 for the arrangement pattern).

[0125] 7-1. About Figure 25 (1) Prerequisites and Overview The upper part of Figure 25 is a diagram explaining the processing contents related to the placement of participants, and illustrates the processing related to the placement of each participant (including offset placement of avatars) when the 3 column B pattern (see placement ID = H032 in the placement pattern table of Figure 22) is selected for participant placement, and after the host (Participant I) who acts as the instructor joins, a guest (Participant J) who will be a student joins.

[0126] Before explaining the screen display, we will use the upper diagram in Figure 25 to explain the prerequisite process of calculating the position coordinates of the guest avatar based on the relative positions of the participants (avatar positioning determination process), and the coordinate calculation (avatar offset positioning determination) for offsetting and positioning the host (participant I) avatar based on the relative positions of the participants when the viewer watching the screen is changed from the host participant I to the guest participant J and that viewer (participant J) is placed at the center.

[0127] First, when participant I, who is the instructor and host, selects the arrangement pattern of 3 columns B, the system loads the arrangement pattern of the participants from the arrangement ID=H032 in the arrangement pattern table (see FIG. 22) and updates the arrangement management table (see FIG. 23) (steps 1 and 2). In this case, participant I is placed in the center, with participant J's avatar on his right side. First, in displaying participant I on the screen, based on the relative positions of the participants described above, a process is performed in which the avatar of participant I is superimposed on the 180-degree or 360-degree image so that participant I is at the center and participant J's avatar is to his right (step 3 and Figure 25 (a), (b), and (c)).

[0128] On the other hand, when displaying participant J on the screen, based on the relative positions of the participants described above, a process is performed in which the avatar of participant J is superimposed on the 180-degree or 360-degree image so that participant I's avatar is located to the left of participant J's avatar, with participant J at the center instead of participant I (step 4 and Figure 25 ii), iii), iv).

[0129] (2) Participant I’s screen display The middle rows of Figure 25, (a), (b), and (c), show an example of the screen display of Participant I when a guest (Participant J) who will be a student joins after the host (Participant I) who is acting as the instructor joins. First, in (b), the screen display of the host participant I is shown when the participant I is facing forward. In this case, a cylindrical object is visible on the left side in front of the participant I, and a rectangular object is visible on the right side.

[0130] In contrast, a) shows the screen display when participant I faces left, and c) shows the screen display when participant I faces right. When participant I turns to the left, it becomes clear that participant J's avatar is not in his field of vision. On the other hand, when participant I turns to the right, as shown in the participant arrangement shown in the upper part of Figure 25, participant J's avatar is positioned to the right of participant I's avatar, so that participant J's avatar is superimposed on the 180 degree or 360 degree image.

[0131] (3) Participant J’s screen display The lower row of FIG. 25, ii), e), and f) show examples of the screen display of participant J. FIG. 5 shows the screen display of guest participant J when he / she is facing forward. In this case, a cylindrical object is visible on the left side in front of him / her, and a rectangular parallelepiped object is visible on the right side. It should be noted that this screen display uses a 180-degree or 360-degree image taken from a specific viewpoint, so it can be seen that the screen display is the same as that of the host participant I.

[0132] Also, in the lower part of FIG. 25, ii) shows the screen display when participant J turns to the left, and f) shows the screen display when participant J turns to the right. As shown in ii), when participant J turns to the left, the avatar of participant I is superimposed on the 180-degree or 360-degree image in accordance with the arrangement of the participants shown in the upper part of Figure 25, reflecting the fact that the avatar of host participant I is offset to the left of the avatar of guest participant J, who is the viewer, while maintaining the relative positional relationship between the avatar of host participant I and the avatar of guest participant J.

[0133] Also, as shown in (e), when participant J turns to the right, the avatar of participant I is offset to the left of participant J due to the offset processing described above, and no participant's avatar is positioned to the right, reflecting the fact that no other avatars are in view.

[0134] As described above, since 180-degree or 360-degree images taken from a specified viewpoint are used, the avatar of Participant J is placed at the center instead of the avatar of Participant I, and the relative positional relationship between the avatars of Participant J and Participant I is maintained, and the avatar of Participant I is offset from the avatar of Participant J. By doing so, although the students are not actually participating in the surgery or operation of the equipment in the images used, it is possible to provide content that gives the students a sense of realism as if they were present at the scene and receiving explanations about the scene in the images.

[0135] 7-2. About Figure 26 FIG. 26 is a diagram for explaining the process contents regarding the arrangement of participants when a guest (participant K) further participates in the situation of FIG. (1) Prerequisites and Overview The basic processing is similar to steps 1 to 4 in the upper part of Figure 25, but before explaining the screen display, we will use the upper diagram of Figure 26 to explain the prerequisite processing of calculating the position coordinates of the guest avatar based on the relative positions of the participants (avatar position determination processing), and the coordinate calculation (avatar offset position determination) for offsetting and positioning the avatars of the host (participant I) and other participants based on the relative positions of the participants when the viewer watching the screen is changed from the host participant I to guest participant J or participant K and that viewer (participant J or participant K) is placed at the center.

[0136] First, when participant I, who is the instructor and host, selects the arrangement pattern of column 3B, the system loads the arrangement pattern of the participants from arrangement ID=H032 in the arrangement pattern table (see FIG. 22) and updates the arrangement management table (see FIG. 23) (steps 1 and 2). In this case, participant I is placed in the center, with participant J's avatar on the right and participant K's avatar on the left. Then, in the screen display of participant I, based on the relative positions of the participants described above, the avatar is superimposed on the 180-degree or 360-degree image so that participant I is at the center, participant J's avatar is to the right, and participant K's avatar is to the left (step 3 and Figure 26 (a), (b), and (c)).

[0137] On the other hand, when displaying participant J on the screen, based on the relative positions of the participants described above, a process is performed in which the avatar of participant J is superimposed on the 180-degree or 360-degree image so that participant J is at the center instead of participant I, with participant I's avatar to the left of participant J's avatar and participant K's avatar to the left of participant I (step 4 and Figure 26 ii), iii), iv).

[0138] Furthermore, in displaying participant K on the screen, based on the relative positions of the participants described above, instead of participant I through participant J, a process is performed in which the avatar of participant K is superimposed on the 180-degree or 360-degree image so that participant I's avatar is to the right of participant K's avatar and participant K's avatar is to the right of participant I's avatar (step 5 and Figure 26 (t), (h), (i)).

[0139] (2) Participant I’s screen display The second row from the top in Figure 26, (a), (b), and (c), show an example of the screen display of Participant I when the host (Participant I) acting as the instructor joins, and then guests (Participant J and Participant K) who will be students join. In Figure 26(b), the participant is facing forward and is therefore out of sight of both participants J and K, similar to Figure 25(a) for participant I in the middle row. In FIG. 26A, according to the placement result of step 3, participant K is placed to the left of participant I, so participant K is within view. In FIG. 26C), according to the placement result of step 3, participant J is placed to the right of participant I, so participant J is within view.

[0140] (3) Participant J’s screen display The third row from the top in FIG. 26, ii), e), and f) show an example of the screen display of participant J. First, as shown in (E) in the third row from the top of Fig. 26, the screen display of guest participant J is shown when participant J is facing forward. In this case, a cylindrical object is visible on the left side in front of the participant J, and a rectangular parallelepiped object is visible on the right side. It should be noted that this screen display uses a 180-degree or 360-degree image taken from a specific viewpoint, so it can be seen that the screen display is the same as that of the host participant I.

[0141] On the other hand, as shown in ii), when participant J faces to the left, the avatars of participant K and participant I are superimposed on the 180-degree or 360-degree image in accordance with the arrangement of the participants shown in the upper part of Figure 26, reflecting the fact that the avatars of host participant I and another guest participant K are offset to the left of the avatar of viewer participant J, while maintaining the relative positional relationship between the avatars of host participant I, guest participant J, and participant K.

[0142] Also, as shown in (e), when participant J turns to the right, the avatars of participants I and K are offset to the left of participant J due to the offset processing described above, and no participant's avatars are positioned to the right, reflecting the fact that no other avatars are in view.

[0143] (3) Screen display of participant K The fourth row from the top in FIG. 26, g), h), and i), show an example of the screen display of participant K. First, as shown in (h) in the third row from the top of Fig. 26, the screen display of guest participant K is shown when participant K is facing forward. In this case, a cylindrical object is visible on the left side in front of participant K, and a rectangular parallelepiped object is visible on the right side. It should be noted that this screen display uses a 180-degree or 360-degree image taken from a specific viewpoint, so it can be seen that the screen display is the same as that of the host participant I.

[0144] On the other hand, as shown in (i), when participant K turns to the right, the avatars of participant I and participant J are superimposed on the 180-degree or 360-degree image in accordance with the arrangement of the participants shown in the upper part of Figure 26, reflecting the fact that the avatars of host participant I and other guest participant J are offset to the right of the avatar of viewer participant K, while maintaining the relative positional relationship between the avatars of host participant I, guest participant J, and participant K.

[0145] Also, as shown in (g), when participant K turns to the left, the avatars of participants I and J are offset to the right of participant K due to the offset processing described above, and no participant's avatars are positioned to the left, reflecting the fact that no other avatars are in view.

[0146] 7-3. About Figure 27 FIG. 27 is a diagram showing an example of a screen display for a guest (participant L) when the guest (participant L) further participates in the event shown in FIG. In Figure 27, since participant L is a participant in the second row, an example is shown in which a placement pattern is loaded from the column with auto adjustment ON for placement ID = H032 in the placement pattern table (Figure 22) so that the object to be explained is less likely to be hidden by the avatars of the participants in the previous row. Here, in the case of the placement pattern in the rightmost column in Figure 22, consideration is given to ensuring that the avatars of other participants are not placed in front of the participants in the second column, so that the object being explained is usually not obscured by the avatars of the participants in the previous column (Step 6 in Figure 27). However, even with this arrangement, it is possible that the object to be explained may be obscured by the avatars of the participants in the previous row and become difficult to see. For example, as shown in Fig. 27B), an example of a screen display in such a case is a case in which the object to be explained (cylinder, cube, etc.) is obscured by the avatars of participant K and participant I and becomes difficult to see.

[0147] As a solution to such a case, for example, as shown in (e), by displaying only the head and face of the avatars of participant K and participant I, it is possible to make the object being explained easier to see by only displaying a part of the avatar that overlaps with the object being explained. Alternatively, as shown in step 7, it is effective to use the arrangement in the arrangement pattern table as a base, but further adjust the arrangement automatically or by having the participant move, so that there is more space in front of participant L. In this case, as shown in h), it is possible to make the object being explained easier to see.

[0148] 7-4. Summary of avatar offset processing and appearance As described above, according to the content provision support system of the present invention, when a host participant acting as an instructor and a guest participant acting as a student join a ROOM and receive an explanation from the instructor participant, the participants can view the same live-action video from the center, while maintaining their relative positional relationships with each other, and offset the avatars of the other participants acting as instructors, etc. This allows you to understand your own relative positions in relation to other participants and superimpose the avatars of other participants at an offset position, making it possible to give explanations and lectures that feel as if a three-dimensional space were being constructed on a computer and a view screen was generated and displayed from each participant's viewpoint.

[0149] At this time, attributes such as the name of the participant may be displayed near the avatar of the participant, and the individual may be distinguished by the color of the avatar. In addition to still images, videos can be used. In the case of videos, the host participant acting as the instructor can control playback etc. and synchronize with the video of the student guest participants, allowing them to view images at the same time.

[0150] 8. Screen display (focusing on how the laser light appears) Next, the screen display of the content delivery support system of the present invention will be described with reference to Figs. 28, 29 and 30 regarding the offset processing of the laser light of the virtual laser pointer, focusing on how the laser light appears. 28, 29, and 30, an example will be described in which participants are arranged in an arrangement pattern of 3 columns B (see arrangement ID=H032 in FIG. 22 for the arrangement pattern).

[0151] 8-1. About Figure 28 (1) Prerequisites and Overview The upper part of Figure 28 is a diagram explaining the positioning of participants and the processing contents related to the laser light of the virtual laser pointer. In addition to the processing related to the positioning of each participant (including offset positioning of avatars) when a guest (participant J) who will be a student joins after a host (participant I) who is acting as the instructor joins, the figure illustrates the processing related to the laser light of the virtual laser pointer operated by the host (participant I) who is acting as the instructor (including offset processing of the laser light).

[0152] The process of arranging the participants has been explained in Figure 25 and is therefore omitted here, but after the arrangement of the participants has been decided, on the screen display of participant I, the laser light of the virtual laser pointer operated by the host (participant I) acting as the lecturer is operated by the participant himself, so it is emitted from the light emission coordinates near participant I as the starting point and is displayed superimposed on the 180-degree or 360-degree image as a light axis extending to the arrival coordinates of the object being explained (step 3, and Figure 28 (a) and (b)).

[0153] On the other hand, in the screen display of Participant J, who is a student, the avatar of Participant I is offset and positioned, and the optical axis of the laser light is also offset and then superimposed on the 180-degree or 360-degree image (step 4, and Figure 28-2) and (e)).

[0154] More specifically, as explained in Figure 12 above, the location where the laser light reaches the object being explained needs to be the same, so for the coordinates where the laser light reaches, the coordinates where the laser light reached by the host participant I (RAZER-ORG-TargetPoint) are used as is. On the other hand, for the coordinates where the laser light is emitted, the original emission coordinates (RAZER-ORG-StartPoint) are offset to match the position where participant I repositions the object offset from the original position, and the optical axis of the laser light is converted using the offset emission coordinates (RAZER-OFFSET-StartPoint) that have been converted from the original emission coordinates (RAZER-ORG-StartPoint), and the laser light is superimposed on the 180-degree or 360-degree image (step 4, Figure 28-ii), e).

[0155] (2) Participant I’s screen display The middle rows of Figure 28, (a), (b), and (c), show an example of the screen display of Participant I when a guest (Participant J) who will be a student joins after the host (Participant I) who is acting as the instructor joins. First, (b) shows the screen display of the host participant I when he / she is facing forward. In this case, participant I is operating a virtual laser pointer, so the laser light is emitted from near the center and is shining on a cylindrical object. The system acquires the emission coordinates (RAZER-ORG-StartPoint) and arrival coordinates (RAZER-ORG-TargetPoint) of the laser light in this state.

[0156] In contrast, a) shows the screen display when participant I faces left, and c) shows the screen display when participant I faces right. When participant I faces left, the virtual laser pointer that participant I is operating with his / her right hand or other hand will be located slightly to the right of the screen, and the optical axis of the laser light will be superimposed on the 180-degree or 360-degree image so that the laser light is emitted from the right side of the screen. On the other hand, when participant I looked to the right, the laser light was superimposed on the screen at a position just outside his field of vision.

[0157] (3) Participant J’s screen display The lower row of FIG. 28, ii), e), and g) show examples of the screen display of participant J. FIG. 5E) shows the screen display of guest participant J when participant J is facing forward. In this case, participant I is offset to the left of participant J, and the position of the virtual laser pointer and laser light operated by participant I is also offset, so that the laser light is superimposed on the 180-degree or 360-degree image, as if it were being irradiated from the left of participant J.

[0158] Also, in the lower part of FIG. 28, ii) shows the screen display when participant J turns to the left, and f) shows the screen display when participant J turns to the right. As shown in ii), when participant J turns to the left, the relative positional relationship between the avatar of host participant I and the avatar of guest participant J is maintained, while the avatar of host participant I and the virtual laser pointer operated by participant I are offset to the left of the avatar of guest participant J, who is the viewer, to reflect this, a virtual laser pointer is placed near participant I, and an overlay is displayed on the 180-degree or 360-degree image so that the optical axis of the offset-processed laser light is irradiated onto the object.

[0159] Furthermore, as shown in (f), when participant J turns to the right, the avatar of participant I is offset to the left of participant J due to the offset process described above, and the laser light emitted from the virtual laser pointer does not enter his or her field of vision, so it can be seen that the laser light is not displayed.

[0160] 8-2. About Figure 29 Figure 29 is a diagram explaining the arrangement of participants when a guest (participant K) joins in the situation of Figure 28, and the processing contents related to the laser light of the virtual laser pointer. It illustrates the processing related to the laser light of the virtual laser pointer operated by the host (participant I) acting as the instructor (including offset processing of the laser light).

[0161] (1) Prerequisites and Overview The basic process is the same as steps 1 to 4 in the upper part of FIG. 28, except that participant K additionally participates. In step 3, regarding the positioning of Participant I's avatar on the screen display, Participant I is positioned at the center of the 180-degree or 360-degree image, with Participant J's avatar positioned to the right and Participant K's avatar positioned to the left, and laser light is shown being emitted from a virtual laser pointer (near the center) operated by the avatar of Participant I, who is the instructor and host.

[0162] In step 4, regarding the positioning of participant J's avatar on the screen display and the processing of the laser light (offset processing), participant J is placed at the center of the 180-degree or 360-degree image, and then participants I and K are offset while maintaining their relative positional relationship.In addition, regarding the emission coordinates of the virtual laser pointer, the offset emission coordinates (RAZER-OFFSET-StartPoint) after coordinate conversion are calculated from the original emission coordinates (RAZER-ORG-StartPoint), and the optical axis of the laser light is positioned from the offset emission coordinates (RAZER-OFFSET-StartPoint) to the same coordinates as the original destination coordinates (RAZER-ORG-TargetPoint).

[0163] In addition, in step 5, regarding the positioning of participant K's avatar on the screen display and the processing of the laser light (offset processing), participant K is positioned at the center of the 180-degree or 360-degree image, and then participants I and J are offset while maintaining their relative positional relationship.The coordinate-converted, offset light emission coordinates (RAZER-OFFSET-StartPoint) are calculated, and the optical axis of the laser light is positioned from the offset light emission coordinates (RAZER-OFFSET-StartPoint) to the same coordinates as the original arrival coordinates (RAZER-ORG-TargetPoint).

[0164] (2) Participant I’s screen display Figure 29(b) shows the screen display when participant I is facing forward, and a laser beam is emitted from the virtual laser pointer operated by participant I (see step 3), so the optical axis of the laser beam is superimposed on the 180-degree or 360-degree image from near the center. Figure 29(i) shows the display when participant I faces left, and if participant I is holding and operating the virtual laser pointer in his / her right hand, the optical axis of the laser light is superimposed and displayed slightly to the right. Figure 29(c) shows the display when participant I looks to the right, with the laser light superimposed on the very edge of his field of vision.

[0165] (3) Participant J’s screen display Figure 29(e) shows the screen display when participant J is facing forward, and since participant I is positioned offset, the emission coordinates of the laser light emitted by participant I's avatar are also offset (see step 4), so that the laser light is emitted from the direction in which the offset avatar of participant I is located, i.e., from the left of participant J, relative to the 180-degree or 360-degree image, and the optical axis of the laser light is superimposed.

[0166] Figure 29(b) shows that when participant J turns to the left, participant I, who is positioned offset, comes into view, and a laser beam is emitted from a virtual laser pointer operated by participant I's avatar, with the optical axis of the laser beam being superimposed. Also, in Figure 29(f), when participant J turns to the right, there are no other participants present, so neither the other participants nor the optical axis of the laser light are in his or her field of vision.

[0167] (4) Screen display of participant K Figure 29(h) shows the screen display when participant K is facing forward, and by offsetting the emission coordinates of the laser light from participant I and the virtual laser pointer (see step 5), the laser light is emitted from the right, reflecting the fact that the emission coordinates of the laser light emitted by participant I are located to the right of participant K, and the optical axis of the laser light is superimposed. Figure 29(t) shows the screen display when participant K is facing to the left, with the laser light being irradiated from further to the right than in (h), and the optical axis of the laser light being superimposed. Figure 29(i) shows that when participant K is facing to the right, participant I, who is positioned offset, comes into view, and a laser beam is emitted from an offset virtual laser pointer, with the optical axis of the laser beam superimposed.

[0168] 8-3. About Figure 30 FIG. 30 is a diagram showing an example of a screen display for a guest (participant L) when the guest (participant L) further participates in the event shown in FIG. In Figure 30, since participant L is a participant in the second row, an example is shown in which a placement pattern is loaded from the column with auto adjustment ON for placement ID = H032 in the placement pattern table (Figure 22) so that the object to be explained is less likely to be hidden by the avatars of the participants in the previous row. Here, in the case of the layout pattern of the rightmost column in Figure 22, consideration is given to ensuring that the avatars of other participants are not positioned directly in front of the participants in the second row, so that the object being explained and the optical axis or destination of the laser light are not usually hidden by the avatars of the participants in the previous row (Step 6 in Figure 30). However, even with such an arrangement, it is possible that the object to be explained and the optical axis or destination of the laser light may be obscured by the avatars of the participants in the previous row and become difficult to see. For example, as an example of a screen display in such a case, as shown in Fig. 30B), the object to be explained (cylinder, cube, etc.) may be obscured by the avatars of participant K and participant I and become difficult to see.

[0169] Therefore, for example, as shown in (e), by displaying only the heads and faces of the avatars of participant K and participant I, or by displaying only a part of the avatar that overlaps with the object being explained or the optical axis or destination of the laser light, it is possible to make the destination of the laser light irradiated on the object being explained more visible. Alternatively, as shown in step 7, it is effective to use the arrangement in the arrangement pattern table as a base, but further adjust the arrangement automatically or by having the participant move, so that there is more space in front of participant L. In this case, as shown in h), it is possible to make the arrival point of the laser light irradiated on the object being explained easier to see.

[0170] 8-4. Summary of laser light offset processing and appearance As described above, according to the content provision support system of the present invention, when a participant acting as an instructor uses a laser pointer within the ROOM space to point to and explain an object being explained, the participants can offset and position the avatars of other participants acting as instructors, etc., while watching the same live-action video from the center, and can reposition the avatars by performing coordinate conversion and offset processing of the laser light. This allows the instructor to grasp the relative positions of himself / herself and the participants acting as the instructor, and superimposes the optical axis of the laser light as if it were coming from near the avatar of the participant acting as the instructor, making it possible to give explanations and lectures that feel as if a three-dimensional space were being constructed on a computer and a view screen was generated and displayed from the viewpoint of each participant.

[0171] 9. Processing flow of the present invention 9-1. Image registration process Next, an example of each processing step of the image registration processing will be described with reference to the flowchart in FIG. First, an image (video or still image) being shot or already shot is obtained (step S1-1), and processing such as lens distortion correction is performed on the image (step S1-2). As a stitching process, overlapping areas of multiple images are processed (step S1-3) and image synthesis processing such as smoothing out seams is performed (step S1-4).

[0172] The synthesized image is projected (texture mapping) onto a virtual sphere to generate a 180-degree or 360-degree image (step S1-5). If necessary, a stereoscopic image of 180 degrees or 360 degrees may be generated (see Figs. 3 and 4). The generated images are appropriately classified into categories, given IDs (step S1-6), and registered in the storage unit (step S1-7).

[0173] 9-2. Overall processing flow Next, an example of the overall processing flow will be described with reference to the flowchart in FIG. In FIG. 32, the processing is described separately as host-side processing and guest-side processing, but these processing may be performed on the host-side HMD terminal and the guest-side HMD terminal, or a separate server (including cloud) or external device (hereinafter abbreviated as "server side") may be provided to receive the operation contents of the host-side HMD terminal and the guest-side HMD terminal, and the results of processing on the server side may be displayed on the host-side terminal and the guest-side HMD terminal. An example of processing steps in the case where processing is performed on the server side is shown by dotted lines on the server side, but is not limited to this.

[0174] First, on the host side, the HMD is started, the application is started, and after starting as the host, the participant logs into the system by selecting or entering their name (steps S2-H1 to H4). After the avatar selection screen is displayed, the participant selects their preferred avatar (step S2-H5), and then selects the image they wish to explain (step S2-H6). Thereafter, the system starts up a ROOM, which is a room (space) for explaining the selected image to the students, starts playing the image (step S2-H7), and waits for the participation of guests who are participants of the students (step S2-H8).

[0175] On the other hand, the guest also starts the HMD, starts the application, starts it as a host, and logs into the system by entering the participant's name (steps S2-G1 to G4), and after the avatar selection screen is displayed, the guest selects a preferred avatar (step S2-G5), and selects a host to act as the lecturer (step S2-G6). Note that instead of selecting a host, the guest may select a ROOM in which to explain the desired image. When the guest selects a host or a ROOM, a participation request is issued to the relevant host or ROOM (step S2-G7).

[0176] On the other hand, on the host side, the participation request from the guest is received, and if permitted, the guest's participation is accepted (step S2-H8, step S2-G7). After the participation of the guest is accepted, a synchronization process is carried out to reproduce the same image on the host side and the image on the guest side (step S2-H9, step S2-G8).

[0177] The synchronization process may be performed, for example, when an image is stored on the host terminal and the same image is transferred to the guest terminal, or when the same image is stored on the guest terminal and the synchronization process is performed so that an image at the same time as the image on the host terminal is displayed. Alternatively, if images are stored on the server side, the image synchronization process between the host and guest may be performed on the server side.

[0178] Once the synchronization process is complete, the host and guest can view the same image while changing the orientation of the HMD, allowing them to appropriately change the viewing direction within the 180-degree or 360-degree image (step S2-H9, step S2-G8). The above is an outline of the overall processing of the content delivery support system of the present invention. The process flow will be described in more detail below.

[0179] 9-3. Participant avatar placement processing (including offset processing) Next, the process of arranging participant avatars (including offset processing) will be described with reference to FIG. FIG. 33 is a diagram showing an example of each processing step for controlling how the participants are displayed on the host's screen and how the host is displayed on the guest's screen when a guest joins after a host has joined, taking into consideration the relative positions of the host and guest while using 180-degree or 360-degree images taken from a specified viewpoint.

[0180] Although FIG. 33 describes the host-side processing and the guest-side processing, the processing may be executed on the HMD terminal of the host or guest, or may be executed on the server side with only operation information and screen information being exchanged with the HMD terminal of the host or guest. An example of processing steps in the case where processing is performed on the server side is shown by dotted lines on the server side, but is not limited to this.

[0181] (1) Host-side processing First, the host selects an image (see step S2-H6 in FIG. 32), then loads the image (step S3-H1), starts ROOM and starts playing the image (step S3-H2), and in parallel accepts a participation request from the guest (step S3-H3), and in the case of a video, performs synchronization processing with the image on the guest side (step S3-H4). Then, the host selects a placement pattern registered in the placement pattern table (see FIG. 22), and then determines the relative positional relationship of the avatars based on that placement pattern, in accordance with the actual placement or a placement pattern prepared in advance for the training (step S3-H5) (...avatar relative positional relationship determination process).

[0182] In addition, the position information of the actual participants may be obtained and used as a reference position, and if the participants move, etc., the deviation from the determined relative positional relationship may be adjusted, and rearrangement may be performed taking into account the movement of the actual participants. After the relative positional relationship is determined, the placement of the hosts and guests is managed using a placement management table (see FIG. 23). It is assumed that the movement of the participant's avatar may be required, for example, when an object to be explained is hidden by a participant's avatar and cannot be seen, and the participant needs to move it.

[0183] Since a 180-degree or 360-degree image taken from a specified viewpoint is used, the host's avatar is positioned at the center of the host's screen display (step S3-H6), and based on the relative positional relationship, the position coordinates of the guest's avatar are calculated and the position of the guest's avatar is determined (step S3-H7) (...avatar position determination process). When the placement of the avatars is determined, the host updates screen configuration information regarding the placement of other avatars as seen from the host, which is information that is the basis for the screen display of the participants of the host (step S3-H8).

[0184] In addition, gaze information regarding the direction of the participant's gaze is obtained from the sensor information of each participant's HMD (step S3-H9), and a view screen of the range visible from the host participant's avatar is generated from a 180-degree or 360-degree image based on the obtained gaze information and specified field of view information (step S3-H10). Then, based on the screen configuration information, the host's avatar is drawn in the center (superimposed on the 180-degree or 360-degree image), and if the guest's avatar is within the viewing angle, the guest's avatar is drawn (steps S3-H11, H12).

[0185] In addition, when the viewer's avatar is facing forward, no body parts may be displayed since they are not visible, or when the viewer is looking down and parts of their body are visible, or when their hand is raised and their hand is visible, parts of their body may be displayed.

[0186] Thereafter, the process repeats (steps S3-H13, H14) where the relative positions of the avatars are updated based on updates to the line of sight information and updates to the position information due to real-world movement, etc., the placement of the avatars is updated, and a view screen with a specified viewing angle range in a 180-degree or 360-degree image is generated in response to changes in the line of sight, and the avatars are superimposed on the screen. When the explanation of the selected 180-degree or 360-degree image has been completed, the host ends the ROOM (Y in step S3-H15).

[0187] (2) Guest side processing It is assumed that there are multiple guests, such as 1 to n guests, but the processing content is basically the same for all guests, so the following will explain the participation of any one guest. On the guest side, after selecting the host or ROOM (steps S2-G6, G7 in FIG. 33), when the participation request is accepted (step S3-G1), an image is loaded (step S3-G2) and synchronization processing with the host's image is performed (step S3-G3).

[0188] Thereafter, the host determines the relative positions of the avatars based on the placement pattern selected (step S3-G4). Since a 180-degree or 360-degree image (the same image as the host) taken from a specified viewpoint is used, if the host and guest were positioned away from the host, which is placed at the center, based on the relative positions of the host and guest, the space would appear distorted (see Figure 5), so a process is performed to center the guest's avatar (instead of the host's avatar) on the guest's screen display as well (step S3-G5).

[0189] Accordingly, in order to make the position of the host's avatar look natural from the guest's viewpoint, the host's avatar is offset from the guest's avatar, which is placed at the center. More specifically, based on the relative positional relationship, coordinate calculation is performed to offset the host's avatar (step S3-G6) (...avatar offset placement determination process). Once the placement of the avatars has been determined, screen configuration information regarding the placement of other avatars as seen from the guest, which is the information on which the screen display of the guest participant is based, is updated (step S3-G7).

[0190] In addition, gaze information regarding the direction of the participant's gaze is obtained from the sensor information of each participant's HMD (step S3-G8), and a view screen of the range visible from the guest participant's avatar is generated from a 180-degree or 360-degree image based on the obtained gaze information and specified field of view information (step S3-G9). Then, based on the screen configuration information, the guest's avatar is drawn in the center (superimposed on the 180 degree or 360 degree image), and if the host's avatar is present within the viewing angle, the host's avatar is drawn (steps S3-G10, G11).

[0191] Thereafter, the process repeats (steps S3-G12, G13) of updating the relative positions of the avatars and updating the positions of the avatars based on updates to the gaze information and updates to the position information due to real-world movement, etc., generating a view screen with a specified viewing angle range in a 180-degree or 360-degree image as the gaze direction changes, and superimposing and displaying the avatars.

[0192] 9-4. Processing of laser light from virtual laser pointer (including offset processing) Next, the processing (including offset processing) of the laser light emitted from the virtual laser pointer will be described with reference to FIG. Figure 34 is a diagram showing an example of each processing step for controlling how the laser light of the virtual laser pointer operated by the host is displayed on the host's screen display and how the laser light of the virtual laser pointer operated by the host is displayed on the guest's screen display, taking into account the positional relationship between the host and guest, while using a 180-degree or 360-degree image captured from a specified viewpoint when the host operates a virtual laser pointer.

[0193] Here, in Figure 34, the host-side processing and the guest-side processing are described, but they may also be executed on the HMD terminal of the host or guest, or they may be executed on the server side and only operation information and screen information may be exchanged with the HMD terminal of the host or guest. Note that, hypothetically, an example of processing steps in the case where processing is performed on the server side is shown by dotted lines on the server side, but the present invention is not limited to this.

[0194] In addition, in FIG. 34, for the sake of simplicity, the processing steps for line of sight changes are omitted. In addition, the series of processes (steps S4-H1 to H5 and steps S4-G1 to G4) for the avatar placement process, including image loading process, ROOM startup process, participation acceptance process, image synchronization process, and avatar placement determination process, and the avatar offset placement determination process are the same as those in Figure 33, so the explanation will be omitted.

[0195] (1) Host-side processing When the avatar placement process and the avatar offset placement decision process are completed, the host places the host's avatar in the center and also places the virtual laser pointer near the center. The placement and orientation of the virtual laser pointer are appropriately corrected according to the position and orientation of the hand controller held by the participant of the instructor's host (step S4-H6).

[0196] Next, the location coordinates of the guest's avatar are calculated based on the relative positions of the participant's avatars (step S4-H7) (avatar location determination process). Then, the host updates screen configuration information regarding the arrangement of the avatars of all the other participants as seen from the host, which is information that is the basis for the screen display of the participants of the host (step S4-H8).

[0197] In the laser light processing, first, the emission coordinates (RAZER-ORG-StartPoint) of the emission point of the laser light emitted from the virtual laser pointer operated by the host participant are obtained (step S4-H9). Then, the inclination of the hand controller (direction of the optical axis) is detected (step S4-H10), and the target coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser light reaches on the virtual sphere of the 180-degree or 360-degree image are obtained (step S4-H11).

[0198] The optical axis of the laser light is generated based on the acquired emission coordinates (RAZER-ORG-StartPoint) and arrival coordinates (RAZER-ORG-TargetPoint) (step S4-H12), and the optical axis of the laser light is drawn (superimposed) on the 180-degree or 360-degree image near the center as if it were being emitted from a virtual laser pointer operated by the host (step S4-H13). Thereafter, in accordance with changes in the inclination of the hand controller and changes in the position information of the participant's avatar, the direction of the virtual laser pointer and the position information of the participant are updated to continue the avatar positioning and laser light display processing (steps S4-H14, H15).

[0199] (2) Guest side processing It is assumed that there are multiple guests, such as 1 to n guests, but the processing content is basically the same for all guests, so the following will explain the participation of any one guest. When the avatar arrangement process and the avatar offset arrangement determination process are completed, the guest side arranges the guest's avatar in the center of the 180-degree or 360-degree image in place of the host's avatar (step S4-G5).

[0200] On the other hand, for the avatars of the host and other guests, coordinate calculation is performed to offset and position the avatars of the host and other guests based on the relative positional relationship (avatar offset position determination process) (step S4-G6). Then, the screen configuration information, which is the information on which the screen display of the participant of the guest is based and which relates to the arrangement of the avatars of all the other participants as seen from the guest, is updated (step S4-G7).

[0201] Next, the emission coordinates (RAZER-OFFSET-StartPoint) of the emission point of the laser light of the virtual laser pointer operated by the host's avatar, which is placed with an offset, are calculated (step S4-G8). This is because, since a 180-degree or 360-degree image (the same image as the host) taken from a specified viewpoint is used, the guest is positioned at the center of the image to prevent distortion (see Figure 5). Therefore, if the optical axis of the laser light emitted by the host from near the center of the image were used as is, it would appear to the guest that the light is being emitted from close to the guest (see Figure 10). Therefore, in order to eliminate any sense of incongruity during the training, in the same way that the host is offset, the virtual laser pointer is also offset, and offset processing is also performed on the laser light (see Figures 11 and 12).

[0202] Next, the target coordinates (RAZER-ORG-TargetPoint) of the target point of the optical axis of the laser light are obtained (step S4-G9). In order to prevent the point indicated on the object to be explained from changing, it is desirable to use the coordinates of the target point of the laser light of the virtual laser pointer operated by the avatar of the host participant (coordinates on the virtual sphere of the 180-degree or 360-degree image) as they are.

[0203] Based on the above, the optical axis of the laser light is generated based on the emission coordinates (RAZER-ORG-StartPoint) of the laser light of the host's avatar's virtual laser pointer, which is positioned offset, and the arrival coordinates (RAZER-ORG-TargetPoint) that retain the original arrival coordinates of the laser light emitted by the host (step S4-G10). Then, by drawing (superimposing) the optical axis of the generated laser light on a 180-degree or 360-degree image, when viewed from the guest, the laser light appears to be emanating from the direction of the host's avatar, which is positioned offset (step S4-G10).

[0204] 10. Summary As described above, the content provision support system of the present invention can solve problems that arise when a large number of participants simultaneously watch the same live-action video shot from a specific viewpoint using VR goggles such as an HMD, rather than constructing a three-dimensional space on a computer (such as the problem of distortion occurring from anywhere other than the center in Figure 5, the problem of being unable to generate a virtual sphere as in Figures 6 and 7, unlike CG, and the problem of the optical axis of laser light in Figures 9 and 10).

[0205] In other words, according to the content provision support system of the present invention, when a participant acting as an instructor uses a laser pointer within the ROOM space to point to and explain an object being explained, the avatars of other participants acting as instructors, etc. can be offset and positioned while the participants watch the same live-action video from the center, and the laser light can be converted into coordinates and offset and repositioned.

[0206] This allows the instructor to grasp the relative positions of himself / herself and the participants acting as the instructor, and superimposes the optical axis of the laser light as if it were coming from near the avatar of the participant acting as the instructor, making it possible to give explanations and lectures that feel as if a three-dimensional space were being constructed on a computer and a view screen was generated and displayed from the viewpoint of each participant. Therefore, according to the present invention, it is possible for multiple participants to watch a video simultaneously with a sense of realism without incurring the enormous cost and labor required to construct a three-dimensional space on a computer, by using video that is currently being filmed or has already been filmed, without incurring the cost and labor. [Industrial Applicability]

[0207] The content provision support system of the present invention can be applied to 180-degree or 360-degree images that have already been captured, as well as to 180-degree or 360-degree images that are currently being captured. For example, at a shooting site, in addition to the members currently working, participants in remote locations can also participate, allowing explanations to a large number of participants, or instructions and guidance to be given to members currently working. In addition to surgical videos, there are a variety of possible applications for the technology, such as operating heavy machinery that requires safety training, operating control panels at power plants, imparting know-how in manufacturing factories, and imparting aquaculture know-how at fish farms. [Explanation of symbols]

[0208] 100 HMD 200 Hand Controller 300 Server (including cloud) or external device 400 terminals 410 Input section 500 Network or communication means 600 Content provision support system

Claims

1. A method for supporting content provision when multiple participants view a wide-angle image taken from a predetermined viewpoint, One or more computers, A placement information acquisition step to acquire placement information of the aforementioned multiple participants, Based on the arrangement information, the avatar relative position relationship determination step determines the relative positional relationship between multiple avatars corresponding to the multiple participants in the wide-area image, For each of the aforementioned participants, an avatar corresponding to that participant is placed at the center of the wide-area image, and while maintaining the relative positional relationship determined in the avatar relative positional relationship determination step, an avatar corresponding to a participant other than that participant is placed at an offset position relative to the avatar corresponding to that participant; For each of the aforementioned multiple participants, a display image generation step is performed to generate a display image in which avatars corresponding to participants other than the participant in question are superimposed on the wide-area image, based on the placement determined for that participant in the avatar placement determination step. A method for supporting content delivery, which involves executing this process.

2. A method for supporting the provision of content using 180-degree to 360-degree images taken from a predetermined viewpoint in a system comprising a head-mounted display and a hand controller, wherein the head-mounted display is provided with means for detecting tilt in at least the XYZ 3 axes, means for changing the display range of an image according to the tilt, and, if necessary, means for detecting position information in the head-mounted display, wherein The system includes a placement pattern table for determining how to arrange participants, and a placement management table for managing the placement of participants determined based on the placement pattern table. A placement information acquisition step involves obtaining placement information for two or more participants, including at least participant I and participant J, from the placement management table. An avatar generation step, which generates multiple avatars corresponding to the multiple participants mentioned above, Based on the acquired placement information of the multiple participants, the relative positional relationship of the multiple avatars corresponding to the multiple participants in the captured 180-degree or 360-degree images is determined in the avatar relative positional relationship determination step. Among the aforementioned multiple participants, in the screen configuration of participant I's head-mounted display, The avatar of participant I is placed in the center, An avatar placement determination step is performed to maintain the relative positional relationships between the avatars of the multiple participants determined in the avatar relative positional relationship determination step, and then to place the avatars of participants other than participant I, including participant J. Among the aforementioned multiple participants, in the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, An avatar offset placement determination step is performed, in which, while maintaining the relative positional relationships between the avatars of the multiple participants determined in the avatar relative positional relationship determination step, the avatars of participants other than participant J, including participant I, are placed offset from the placement of participant J's avatar. In the screen display of participant I's head-mounted display, Based on the avatar placement determined in the avatar placement determination step, the avatars of participants other than participant I are superimposed on a 180-degree or 360-degree image taken from the predetermined viewpoint, with participant I's avatar at the center. In the screen display of participant J's head-mounted display, Based on the avatar placement determined in the avatar offset placement determination step, the avatar offset superimposition display step involves superimposing the avatars of other participants at offset positions onto a 180-degree or 360-degree image taken from a predetermined viewpoint, with the avatar of participant J at the center. A content provision support method characterized by comprising the following features.

3. A gaze information acquisition step, which involves obtaining information on the orientation of the participant's face or gaze direction corresponding to the tilt, by means of detecting the tilt of the XYZ 3-axis of the head-mounted display of any one participant as described above, In the screen display of any one participant's head-mounted display, A display range changing step, which changes the display range of a 180-degree to 360-degree image taken from a predetermined viewpoint, based on the information of the direction of the face or gaze of any one participant obtained in the gaze information acquisition step, The content provision support method according to claim 2, characterized by comprising the above.

4. If there are at least two participants, I and J, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination step is performed to maintain the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, and then to place the avatar of other participant J. In the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, An avatar offset placement determination step is performed, in which, while maintaining the relative positional relationship between participant J's avatar and the avatar of other participant I determined in the avatar relative positional relationship determination step, the avatar of other participant I is placed offset from the placement of participant J's avatar. In the screen display of participant J's head-mounted display, The system includes an offset superimposition display step, in which the relative positional relationship between participant J's avatar and the avatar of another participant I is maintained, and the avatar of the other participant I is superimposed at an offset position. In the screen display of participant J's head-mounted display, When the display range of the 180-degree to 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of participant J obtained, if the avatar of another participant I is within the field of view of participant J's avatar, The avatar of participant J is displayed at a predetermined offset position, with the avatar of the other participant I at the center. The content provision support method according to claim 3, characterized by the above.

5. If there are at least three participants with I, J, and K, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination step is performed to maintain the relative positional relationship between participant I's avatar and the avatars of other participants (J, K, ...) as determined in the avatar relative positional relationship determination step, and to position the avatars of the other participants (J, K, ...). In the screen configuration of participant K's head-mounted display, The avatar of participant K is placed in the center, An avatar offset placement determination step is performed, in which, while maintaining the relative positional relationship between participant K's avatar and the avatars of other participants (I, J, ...) determined in the avatar relative positional relationship determination step, the avatars of the other participants (I, J, ...) are placed offset from the placement of participant K's avatar. In the screen display of participant K's head-mounted display, The system includes an offset display superimposition step, which maintains the relative positional relationship between participant K and other participants (I, J, ...) and superimposes the avatars of the other participants (I, J, ...) at an offset position. In the screen display of participant K's head-mounted display, When the display range of the 180-degree to 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of participant K acquired, if the avatars of the other participants (I, J, ...) are within the field of view of participant K's avatar, The avatar of participant K is displayed at a predetermined offset position, with the avatars of the other participants (I, J, ...) positioned around it. The content provision support method according to claim 3, characterized by the above.

6. When there are two or more participants, including any one participant and other participants, In the screen configuration of the head-mounted display of any one of the aforementioned participants, The avatar of any one of the aforementioned participants is placed in the center, An avatar placement determination step is performed to place the avatar of the other participant while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant, as determined in the avatar relative positional relationship determination step. In the screen configuration of the head-mounted display of one of the other participants, The avatar of one of the other participants is placed in the center, An avatar offset placement determination step is performed, in which, while maintaining the relative positional relationships between participants determined in the avatar relative positional relationship determination step, the avatars of all other participants except the avatar of one of the other participants are placed offset from the placement of the avatar of the other participant. In the screen display of any one participant's head-mounted display, The system includes an avatar superimposition display step in which the avatar of one participant is superimposed on the avatar of another participant while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant. In the screen display of the head-mounted display of one of the other participants, When the display range of the 180-degree or 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of one of the other participants obtained, if there is an avatar other than that of the one of the other participants within the field of view of the avatar of that one of the other participants, The avatar of one of the other participants is displayed at a predetermined offset position, with the avatars of all other participants displayed at a predetermined offset position. The content provision support method according to claim 3, characterized by the above.

7. When at least one participant I among the aforementioned multiple participants operates with a hand controller, the hand controller is equipped with tilt detection means that detects tilt in at least the XYZ 3 axes, A virtual laser pointer generation step, which generates a virtual laser pointer corresponding to the hand controller, in association with the avatar of participant I operating the hand controller, In the screen display of participant I who operates the hand controller, The process involves obtaining the emission coordinates (RAZER-ORG-StartPoint) of the point where the laser beam of the generated virtual laser pointer is emitted, as part of the emission coordinate acquisition step. In the screen display of participant I who operates the hand controller, A laser beam superimposition display step, in which the optical axis of the laser beam emitted from the emission coordinates of the generated virtual laser pointer is superimposed and displayed on a 180-degree to 360-degree image taken from the predetermined viewpoint, The steps include detecting the tilt of the XYZ 3 axes in response to the operation of the hand controller and tilting the generated virtual laser pointer, The steps include changing the optical axis of the laser beam emitted from the virtual laser pointer according to the tilt of the hand controller, In the screen display of participant I who operates the hand controller, The process involves obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser beam emitted from the virtual laser pointer reaches, and The content provision support method according to claim 2, characterized by comprising the above.

8. When there are at least two participants, I and J, and participant I is equipped with a hand controller and operates a virtual laser pointer corresponding to that hand controller, the process is as follows: In the screen configuration of the head-mounted display of participant I, The avatar of participant I and the virtual laser pointer are placed in the center, An avatar placement determination step is performed to maintain the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, and then to place the avatar of other participant J. In the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, The Avatar & Virtual Laser Pointer Offset Placement Determination Step involves maintaining the relative positional relationship between participant J and participant I determined in the Avatar Relative Positional Relationship Determination Step, and then offsetting the avatar of the other participant I and participant I's virtual laser pointer. In the screen display of the head-mounted display of participant I, The steps include obtaining the emission coordinates (RAZER-ORG-StartPoint) of the emission point and the arrival coordinates (RAZER-ORG-TargetPoint) of the destination point of the laser beam emitted from the virtual laser pointer of participant I's avatar, Based on the acquired light emission coordinates (RAZER-ORG-StartPoint) and the relative positional relationship between participant J and participant I, In the screen display of participant J's head-mounted display, The steps include calculating the emission coordinates (RAZER-OFFSET-StartPoint) of the laser beam emission point of the virtual laser pointer of participant I's avatar, which is positioned offset as described above, In the screen display of participant J's head-mounted display, The process includes the step of generating the optical axis of the laser beam based on the calculated offset emission coordinate (RAZER-OFFSET-StartPoint) and the destination coordinate (RAZER-ORG-TargetPoint) of the destination point of the optical axis of the laser beam, In the screen display of participant J's head-mounted display, From the calculated offset emission coordinate (RAZER-OFFSET-StartPoint), the laser beam is emitted towards the same coordinate as the target coordinate (RAZER-ORG-TargetPoint) displayed on the screen of the participant I's head-mounted display, by superimposing the optical axis of the generated laser beam onto a 180-degree to 360-degree image taken from the predetermined viewpoint. The content provision support method according to claim 7, characterized by the above.

9. If there are at least three participants, I, J, and K, and participant I is equipped with a hand controller and operates a virtual laser pointer corresponding to that hand controller, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination step is performed to maintain the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, and then to place the avatar of other participant J. In the screen configuration of participant K's head-mounted display, The avatar of participant K is placed in the center, The Avatar & Virtual Laser Pointer Offset Placement Determination Step involves maintaining the relative positional relationship between participant K and other participants (I, J, ...) determined in the Avatar Relative Positional Relationship Determination Step, and then offsetting the avatars of the other participants (I, J, ...) and the virtual laser pointer of participant I. In the screen display of participant I's head-mounted display, The steps include obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point reached by the optical axis of the laser beam emitted from the virtual laser pointer of participant I's avatar, In the screen display of participant K's head-mounted display, The steps include obtaining the emission coordinates (RAZER-OFFSET-StartPoint) of the point where the laser beam of the virtual laser pointer of participant I's avatar, which is positioned offset as described above, In the screen display of participant K's head-mounted display, The process includes the step of generating the optical axis of the laser beam based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of participant I's avatar, which is positioned at an offset, and the destination coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser beam reaches. In the screen display of participant K's head-mounted display, The optical axis of the laser beam is superimposed onto a 180-degree to 360-degree image taken from a predetermined viewpoint, so that the laser beam is emitted from the emission coordinate (RAZER-OFFSET-StartPoint) of the virtual laser pointer of participant I's avatar, which is positioned offset, toward the same coordinate (RAZER-ORG-TargetPoint) obtained on the screen display of participant I's head-mounted display. The content provision support method according to claim 7, characterized by the above.

10. When there are two or more participants, including any one participant and other participants, In the screen configuration of any one participant's head-mounted display, The avatar of any one participant is placed at the center of a 180-degree to 360-degree image taken from the predetermined viewpoint, An avatar placement determination step is performed to place the avatar of the other participant while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant, as determined in the avatar relative positional relationship determination step. In the screen configuration of the head-mounted display of one of the other participants, An avatar & virtual laser pointer offset placement determination step is performed, in which the avatar of one of the other participants is placed at the center of a 180-degree to 360-degree image taken from the predetermined viewpoint, and while maintaining the relative positional relationship between the other participants and any one participant equipped with a hand controller as determined in the avatar relative positional relationship determination step, the avatar of the any one participant and the virtual laser pointer of that any one participant are offset and placed; In the screen display of any one participant's head-mounted display, The steps include obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point reached by the optical axis of the laser beam emitted from the virtual laser pointer of any one participant's avatar, In the screen display of the head-mounted display of one of the other participants, The steps include obtaining the emission coordinates (RAZER-OFFSET-StartPoint) of the point where the laser beam of the virtual laser pointer of any participant's avatar positioned with the aforementioned offset is emitted, In the screen display of the head-mounted display of one of the other participants, The process includes the step of generating the optical axis of a laser beam based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of a virtual laser pointer of any participant's avatar positioned at an offset, and the destination coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser beam reaches. In the screen display of the head-mounted display of one of the other participants, The optical axis of the laser beam is superimposed onto a 180-degree to 360-degree image taken from a predetermined viewpoint, so that the laser beam is emitted from the emission coordinate (RAZER-OFFSET-StartPoint) of the virtual laser pointer of any one participant's avatar, which is positioned offset, toward the same coordinate (RAZER-ORG-TargetPoint) obtained on the screen display of the head-mounted display of any one participant. The content provision support method according to claim 7, characterized by the above.

11. When one participant acts as the presenter (host) and the other participants participate as students (guests), As the number of participants (guests) increases, the presenter will use 180-degree or 360-degree images taken from a predetermined viewpoint, without relying on the participants' actual spatial relationships, to position the participants' avatars in a suitable location for the presenter to explain to the participants. The system includes a placement pattern table for determining how to arrange the aforementioned participants, and a placement management table for managing the arrangement of participants based on the placement pattern table. In the aforementioned arrangement pattern table, In order of participation, the participants will be assigned to a specific group, distinguishing between those acting as presenters (hosts) and those acting as students (guests). The system is configured so that other participants' avatars are not positioned in the direction where the object being explained is located in front of each participant. The system includes the step of updating a placement management table for managing the placement of participants by referring to the placement pattern table according to the order in which the participants joined. Even when multiple participants (guests) are present as students, the system ensures that the object being explained is not obscured by the participants, thereby enabling the explanation to proceed smoothly. A method for supporting content provision according to any one of claims 2 to 10, characterized by the above.

12. A system comprising a head-mounted display and / or a hand controller, wherein the head-mounted display includes means for detecting tilt in at least the XYZ 3 axes, means for changing the display range of a captured 180-degree to 360-degree image according to the tilt, and means for detecting position information in the hand controller and / or head-mounted display as needed, wherein the system supports the provision of content using 180-degree to 360-degree images captured from a predetermined viewpoint, The system includes a placement pattern table for determining how to arrange participants, and a placement management table for managing the placement of participants determined based on the placement pattern table. A means for obtaining placement information of two or more participants, including at least participant I and participant J, from the placement management table, An avatar generation means for generating multiple avatars corresponding to the multiple participants mentioned above, Based on the acquired placement information of the multiple participants, an avatar relative position relationship determination means determines the relative position relationship of multiple avatars corresponding to the multiple participants in 180-degree to 360-degree images taken from the predetermined viewpoint. Among the aforementioned multiple participants, in the screen configuration of participant I's head-mounted display, The avatar of participant I is placed in the center, An avatar placement determination means that, while maintaining the relative positional relationships between the avatars of the multiple participants determined in the avatar relative positional relationship determination step, places the avatars of participants other than participant I, including participant J. Among the aforementioned multiple participants, in the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, An avatar offset placement determination means that, while maintaining the relative positional relationships between the avatars of the multiple participants determined in the avatar relative positional relationship determination step, places the avatars of participants other than participant J, including participant I, offset from the placement of participant J's avatar; In the screen display of participant I's head-mounted display, Based on the avatar placement determined in the avatar placement determination step, the avatars of participants other than participant I are superimposed on a 180-degree to 360-degree image taken from a predetermined viewpoint, with participant I's avatar at the center. In the screen display of participant J's head-mounted display, Based on the avatar placement determined in the avatar offset placement determination step, an avatar offset superimposition display means superimposes the avatars of other participants at an offset position, with the participant J's avatar at the center, onto a 180-degree to 360-degree image taken from the predetermined viewpoint. A content provision support system characterized by having the following features.

13. A means for detecting the tilt of any one participant's head-mounted display along the XYZ 3 axes, and acquiring information on the orientation of the participant's face or gaze corresponding to that tilt, is provided. In the screen display of any one participant's head-mounted display, A display range changing means changes the display range of a 180-degree to 360-degree image taken from a predetermined viewpoint based on the information of the direction of the face or gaze of any one participant obtained in the gaze information acquisition step, The content provision support system according to claim 12, characterized by comprising the above.

14. If there are at least two participants, I and J, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination means that places the avatar of participant J while maintaining the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, In the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, An avatar offset placement determination means that maintains the relative positional relationship between participant J's avatar and the avatar of other participant I, as determined in the avatar relative positional relationship determination step, and then places the avatar of other participant I offset from the placement of participant J's avatar. In the screen display of participant J's head-mounted display, The system includes an offset superimposition display means that maintains the relative positional relationship between participant J's avatar and the avatar of another participant I, and then superimposes the avatar of the other participant I at an offset position. In the screen display of participant J's head-mounted display, When the display range of the 180-degree to 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of participant J obtained, if the avatar of another participant I is within the field of view of participant J's avatar, The avatar of participant J is displayed at a predetermined offset position, with the avatar of the other participant I at the center. The content provision support system according to claim 13, characterized by the above.

15. If there are at least three participants with I, J, and K, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination means that, while maintaining the relative positional relationship between participant I's avatar and the avatars of other participants (J, K, ...) determined in the avatar relative positional relationship determination step, places the avatars of the other participants (J, K, ...). In the screen configuration of participant K's head-mounted display, The avatar of participant K is placed in the center, An avatar offset placement determination means that maintains the relative positional relationship between participant K's avatar and the avatars of other participants (I, J, ...) determined in the avatar relative positional relationship determination step, and then places the avatars of the other participants (I, J, ...) offset from the placement of participant K's avatar. In the screen display of participant K's head-mounted display, The system includes an offset display superimposition means that, while maintaining the relative positional relationship between participant K and other participants (I, J, ...), superimposes the avatars of the other participants (I, J, ...) at an offset position. In the screen display of participant K's head-mounted display, When the display range of the 180-degree to 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of participant K acquired, if the avatars of the other participants (I, J, ...) are within the field of view of participant K's avatar, The avatar of participant K is displayed at a predetermined offset position, with the avatars of the other participants (I, J, ...) positioned around it. The content provision support system according to claim 13, characterized by the above.

16. When there are two or more participants, including any one participant and other participants, In the screen configuration of the head-mounted display of any one of the aforementioned participants, The avatar of any one of the aforementioned participants is placed in the center, An avatar placement determination means that places the avatar of another participant while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant, as determined in the avatar relative positional relationship determination step, In the screen configuration of the head-mounted display of one of the other participants, The avatar of one of the other participants is placed in the center, An avatar offset placement determination means that, while maintaining the relative positional relationships between participants determined in the avatar relative positional relationship determination step, places the avatars of all other participants, except for the avatar of one of the other participants, offset from the placement of the avatar of the other participant; In the screen display of any one participant's head-mounted display, The system includes an avatar superimposition display means that superimposes the avatar of one participant onto another participant's avatar while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant. In the screen display of the head-mounted display of one of the other participants, When the display range of the 180-degree or 360-degree image taken from the predetermined viewpoint is changed according to the gaze information of one of the other participants obtained, if there is an avatar other than that of the one of the other participants within the field of view of the avatar of that one of the other participants, The avatar of one of the other participants is displayed at a predetermined offset position, with the avatars of the other participants displayed at the center. The content provision support system according to claim 13, characterized by the above.

17. When at least one participant I among the aforementioned multiple participants operates with a hand controller, the hand controller is equipped with tilt detection means for detecting tilt in at least the XYZ 3 axes, A virtual laser pointer generation means generates a virtual laser pointer corresponding to the hand controller, in association with the avatar of participant I operating the hand controller, In the screen display of participant I who operates the hand controller, A means for acquiring emission coordinates (RAZER-ORG-StartPoint) of the emission point of the laser light of the generated virtual laser pointer, In the screen display of participant I who operates the hand controller, A laser beam superimposition display means superimposes the optical axis of the laser beam emitted from the emission coordinates of the generated virtual laser pointer onto a 180-degree to 360-degree image taken from the predetermined viewpoint. A means for detecting the tilt of the XYZ 3 axes in response to the operation of the hand controller and tilting the generated virtual laser pointer, A means for changing the optical axis of the laser beam emitted from the virtual laser pointer in accordance with the tilt of the hand controller, In the screen display of participant I who operates the hand controller, A means for acquiring destination coordinates (RAZER-ORG-TargetPoint) of the destination point of the optical axis of the laser beam emitted from the virtual laser pointer, The content provision support system according to claim 12, characterized by comprising the above.

18. When there are at least two participants, I and J, and participant I is equipped with a hand controller and operates a virtual laser pointer corresponding to that hand controller, the process is as follows: In the screen configuration of the head-mounted display of participant I, The avatar of participant I and the virtual laser pointer are placed in the center, An avatar placement determination means that places the avatar of participant J while maintaining the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, In the screen configuration of participant J's head-mounted display, The avatar of participant J is placed in the center, An avatar and virtual laser pointer offset placement determination means that, while maintaining the relative positional relationship between participant J and other participant I determined in the avatar relative positional relationship determination step, offsets and places the avatar of the other participant I and participant I's virtual laser pointer. In the screen display of the head-mounted display of participant I, A means for obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point reached by the optical axis of the laser beam emitted from the virtual laser pointer of participant I's avatar, Based on the acquired light emission coordinates (RAZER-ORG-StartPoint) and the relative positional relationship between participant J and participant I, In the screen display of participant J's head-mounted display, A means for calculating the emission coordinates (RAZER-OFFSET-StartPoint) of the laser beam emission point of the virtual laser pointer of participant I's avatar, which is positioned offset as described above, In the screen display of participant J's head-mounted display, The system includes means for generating the optical axis of a laser beam based on the calculated offset emission coordinate (RAZER-OFFSET-StartPoint) and the destination coordinate (RAZER-ORG-TargetPoint) of the destination point of the optical axis of the laser beam, In the screen display of participant J's head-mounted display, From the calculated offset emission coordinate (RAZER-OFFSET-StartPoint), the laser beam is emitted towards the same coordinate as the target coordinate (RAZER-ORG-TargetPoint) displayed on the screen of the participant I's head-mounted display, by superimposing the optical axis of the generated laser beam onto a 180-degree to 360-degree image taken from the predetermined viewpoint. The content provision support system according to claim 17, characterized by the following:

19. If there are at least three participants, I, J, and K, and participant I is equipped with a hand controller and operates a virtual laser pointer corresponding to that hand controller, In the screen configuration of the head-mounted display of participant I, The avatar of participant I is placed in the center, An avatar placement determination means that places the avatar of participant J while maintaining the relative positional relationship between participant I's avatar and the avatar of other participant J, as determined in the avatar relative positional relationship determination step, In the screen configuration of participant K's head-mounted display, The avatar of participant K is placed in the center, An avatar & virtual laser pointer offset placement determination means that maintains the relative positional relationship between participant K and other participants (I, J, ...) determined in the avatar relative positional relationship determination step, and then offsets and places the avatars of the other participants (I, J, ...) and the virtual laser pointer of participant I. In the screen display of participant I's head-mounted display, A means for obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point reached by the optical axis of the laser beam emitted from the virtual laser pointer of participant I's avatar, In the screen display of participant K's head-mounted display, A means for obtaining the emission coordinates (RAZER-OFFSET-StartPoint) of the laser light emission point of the virtual laser pointer of participant I's avatar, which is positioned offset as described above, In the screen display of participant K's head-mounted display, The system includes means for generating the optical axis of a laser beam based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of the virtual laser pointer of participant I's avatar, which is positioned at an offset, and the destination coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser beam reaches. In the screen display of participant K's head-mounted display, The optical axis of the laser beam is superimposed onto a 180-degree to 360-degree image taken from a predetermined viewpoint, so that the laser beam is emitted from the emission coordinate (RAZER-OFFSET-StartPoint) of the virtual laser pointer of participant I's avatar, which is positioned offset, toward the same coordinate (RAZER-ORG-TargetPoint) obtained on the screen display of participant I's head-mounted display. The content provision support system according to claim 17, characterized by the following:

20. When there are two or more participants, including any one participant and other participants, In the screen configuration of any one participant's head-mounted display, The avatar of any one of the aforementioned participants is placed in the center, An avatar placement determination means that places the avatar of another participant while maintaining the relative positional relationship between the avatar of any one participant and the avatar of another participant, as determined in the avatar relative positional relationship determination step, In the screen configuration of the head-mounted display of one of the other participants, The avatar of one of the other participants is placed in the center, An avatar and virtual laser pointer offset placement determination means that, while maintaining the relative positional relationship between the other participants and any one participant equipped with a hand controller as determined in the avatar relative positional relationship determination step, places the avatar of the any one participant and the virtual laser pointer of the said any one participant with an offset, In the screen display of any one participant's head-mounted display, Means for obtaining the destination coordinates (RAZER-ORG-TargetPoint) of the point reached by the optical axis of the laser beam emitted from the virtual laser pointer of any one participant's avatar, In the screen display of the head-mounted display of one of the other participants, Means for obtaining the emission coordinates (RAZER-OFFSET-StartPoint) of the point where the laser beam of the virtual laser pointer of any participant's avatar, which is positioned offset as described above, In the screen display of the head-mounted display of one of the other participants, The system includes means for generating the optical axis of a laser beam based on the acquired emission coordinates (RAZER-OFFSET-StartPoint) of a virtual laser pointer of any participant's avatar positioned at an offset, and the destination coordinates (RAZER-ORG-TargetPoint) of the point where the optical axis of the laser beam reaches. In the screen display of the head-mounted display of one of the other participants, The optical axis of the laser beam is superimposed onto a 180-degree to 360-degree image taken from a predetermined viewpoint, such that the laser beam is emitted from the emission coordinate (RAZER-OFFSET-StartPoint) of the virtual laser pointer of any participant's avatar, which is positioned offset, toward the same coordinate (RAZER-ORG-TargetPoint) obtained on the screen display of the participant's head-mounted display. The content provision support system according to claim 17, characterized by the following:

21. When one participant acts as the presenter (host) and the other participants participate as students (guests), As the number of participants (guests) increases, the presenter will use 180-degree or 360-degree images taken from a predetermined viewpoint, without relying on the participants' actual spatial relationships, to position the participants' avatars in a suitable location for the presenter to explain to the participants. The system includes a placement pattern table for determining how to arrange the aforementioned participants, and a placement management table for managing the arrangement of participants based on the placement pattern table. In the aforementioned arrangement pattern table, In order of participation, the participants will be assigned to a specific group, distinguishing between those acting as presenters (hosts) and those acting as students (guests). The system is configured so that other participants' avatars are not positioned in the direction where the object being explained is located in front of each participant. The system includes means for updating a placement management table for managing the placement of participants by referring to the placement pattern table in accordance with the order in which the participants joined, Even when multiple participants (guests) are present as students, the system ensures that the object being explained is not obscured by the participants, thereby enabling the explanation to proceed smoothly. A content provision support system according to any one of claims 12 to 20, characterized by the above.

22. A program for causing a computer to execute the content provision support method described in any one of claims 2 to 11.

Images