Display control method and display control program

Abstract

To reduce the unnaturalness of body guidance in virtual space. [Solution] The display control device 201 refers to the operation definition file and determines whether the display avatar corresponding to the position and orientation of the beginner avatar is a Mix. If the display avatar is not a Mix, the display control device 201 displays the beginner avatar in the virtual space displayed on the HMD 202. If the display avatar is a Mix, the display control device 201 refers to the expert avatar recording result, generates a fused avatar, and displays the generated fused avatar in the virtual space displayed on the HMD 202.

Description

Technical Field

[0001] The present invention relates to a display control method and a display control program.

Background Art

[0002] In recent years, by wearing a head-mounted display (HMD), virtual reality (VR) that allows immersion in a virtual space has been used in many contents, enabling an experience in a three-dimensional digital space different from the real world. In addition, the introduction of an education and training system that can perform training aimed at acquiring the physical skills of experts using a virtual space is progressing.

[0003] As a prior art, it measures the exemplary operation data of others, measures the operation data of the user, detects the start point and end point of reproduction of an exemplary operation video specified by the user based on the operation data of the user, and changes the exemplary operation data of others so that the reproduction speed of the exemplary operation video becomes the speed specified by the user, and creates exemplary operation video data that is three-dimensional video data of the human body based on the exemplary operation data after the reproduction speed adjustment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art, when guiding the user's body by an avatar displayed in a virtual space, there is a problem that the movement of the avatar becomes unnatural and the user's sense of self-operation may be lost. For example, in training aimed at acquiring the physical skills of an expert, if the user's sense of self-operation is lost, the learning effect of the physical skills may decrease and the fixation of the skills may be inhibited.

[0006] In one aspect, the present invention aims to reduce the unnaturalness that occurs when performing physical guidance in a virtual space. [Means for solving the problem]

[0007] One embodiment provides a display control method for displaying a virtual part image representing a part of a user in a virtual space at a position corresponding to that part of the user, and the method involves referring to a storage unit that stores information that identifies a location in the virtual space where the position of the virtual part image is displayed is corrected according to a pre-recorded position corresponding to a part of a skilled person in the virtual space, and determining whether or not to correct the position of the virtual part image based on the user's position in the virtual space. [Effects of the Invention]

[0008] According to one aspect of the present invention, it has the effect of reducing the unnaturalness that occurs when performing body guidance in a virtual space. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is an explanatory diagram showing one embodiment of the display control method according to the embodiment. [Figure 2] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of the display control device 201. [Figure 4] Figure 4 is an explanatory diagram showing a specific example of an avatar. [Figure 5] Figure 5 is an explanatory diagram showing an example of an object hierarchy. [Figure 6] Figure 6 is an explanatory diagram showing an example of how an avatar is displayed. [Figure 7] Figure 7 is an explanatory diagram showing an example of the contents of the expert avatar record DB220. [Figure 8]FIG. 8 is an explanatory diagram showing an example of a virtual space. [Figure 9] FIG. 9 is a block diagram showing a functional configuration example of the display control device 201. [Figure 10] FIG. 10 is an explanatory diagram showing a first position management example of the virtual space. [Figure 11] FIG. 11 is an explanatory diagram showing a second position management example of the virtual space. [Figure 12] FIG. 12 is an explanatory diagram showing a specific example of an operation definition file. [Figure 13] FIG. 13 is an explanatory diagram showing an interpolation example of the position and orientation of an object of a fusion avatar. [Figure 14] FIG. 14 is a flowchart showing an example of a procedure for recording an expert avatar by the display control device 201. [Figure 15] FIG. 15 is a flowchart (Part 1) showing an example of a display control procedure of the display control device 201. [Figure 16] FIG. 16 is a flowchart (Part 2) showing an example of a display control procedure of the display control device 201.

BEST MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, embodiments of a display control method and a display control program according to the present invention will be described in detail with reference to the drawings.

[0011] (Embodiment) FIG. 1 is an explanatory diagram showing an embodiment of a display control method according to an embodiment. In FIG. 1, an information processing device 101 is a computer that controls to display a virtual part image representing a part of a user in a virtual space at a position corresponding to the part. The virtual space is a virtual space constructed on a computer.

[0012] The user's part is a part of the body, such as the head, chest, shoulder, upper arm, forearm, hand, etc. The virtual part image is an image that virtually represents the user's part, and is displayed, for example, as the part (object) of the user's avatar. The avatar is a character that is displayed as the user's alter ego in the virtual space. The avatar may represent the user's whole body or only a part of the user's body.

[0013] By wearing a display device such as an HMD (Head-Mounted Display), the user can experience a virtual space different from the real space by using VR technology. For example, there is an education and training system that uses a virtual space with fewer restrictions compared to the real space. The education and training system is a computer system for training beginners and others to acquire the physical skills of experts.

[0014] In the education and training system, in order to enable beginners to acquire the physical skills of experts, it is conceivable to guide the user's body in the virtual space based on the movements of the experts. However, conventionally, emphasis has been placed on constructing an environment in the virtual space that is difficult to realize in the real space, and the user's actions have been guided by guide displays such as characters on the image and voices.

[0015] Here, in the training of physical skills, the sense of self-operation that the subject of the movement is oneself is important, and the higher the sense of self-operation, the higher the skill fixation is recognized. Therefore, if physical guidance is constantly provided in accordance with the movements of the experts during training, it may become unnatural for the user and the sense of self-operation may be lost.

[0016] For example, during training, in the virtual space, the user may wait at a certain place or move to another place in order to perform work. If physical guidance using the movements of the experts is provided even when not performing such work, the user's sense of self-operation may be lost, the learning effect of physical skills may decrease, and skill fixation may be inhibited.

[0017] Therefore, this embodiment describes a display control method that can reduce unnaturalness when performing body guidance in a virtual space. Here, an example of processing by the information processing device 101 will be described. Here, an example will be given in which a virtual body part image representing the body part of the user 102 is displayed at a position corresponding to the body part of the user 102 in the virtual space 104 displayed on the HMD 103 worn by the user 102.

[0018] The information processing device 101 refers to the storage unit 110 and determines whether or not to correct the position where the virtual body part image representing the user 102's body part is displayed, based on the user 102's position in the virtual space 104. The storage unit 110 stores information that identifies the correction application location. The correction application location is the location where the correction of the position where the virtual body part image is displayed is performed. The correction of the position where the virtual body part image is displayed is performed according to the position corresponding to the body part of the expert in the virtual space 104, which has been recorded in advance. An expert is a skilled person, for example, a person who has acquired skills, experience, know-how, etc.

[0019] In the example shown in Figure 1, the virtual space 104 is divided into a grid when viewed from directly above, and is divided into regions 1 to 28. Furthermore, the virtual space 104 is assumed to be a virtual space that reproduces the environment in which work X,Y is performed. User 102 wears an HMD 103 and performs training aimed at acquiring the physical skills of an expert.

[0020] In the training, in the virtual space 104, user 102 first waits in area 7, then moves from area 7 to area 11 as indicated by arrow 105, and performs task X in area 11. Furthermore, after completing task X, moves from area 11 to area 26 as indicated by arrow 106, and performs task Y in area 26. After completing task Y, moves from area 26 to area 28 as indicated by arrow 107, and waits there.

[0021] The memory unit 110 stores information that identifies the regions 11 and 26 where the correction is applied. The memory unit 110 also has in advance the positions corresponding to the body parts of the skilled worker when the skilled worker performs work X in region 11 within the virtual space 104. The memory unit 110 also has in advance the positions corresponding to the body parts of the skilled worker when the skilled worker performs work Y in region 26 within the virtual space 104.

[0022] First, when user 102 is waiting in area 7, user 102's position is "area 7," which is not a correction application location. Therefore, the information processing device 101 refers to the storage unit 110 and determines that no correction is performed on the position where the virtual part image is displayed. In this case, the information processing device 101 displays a virtual part image representing the part of user 102 at the position corresponding to that part in the virtual space 104.

[0023] As a result, while user 102 is waiting in the virtual space 104, user 102's own movements are represented as an avatar (virtual body part image).

[0024] Furthermore, until user 102 moves from area 7 to area 11, user 102's position is "area 14 ⇒ area 13 ⇒ area 12," none of which are locations where correction is applied. For this reason, the information processing device 101 refers to the storage unit 110 and determines that no correction is performed on the position where the virtual part image is displayed. In this case, the information processing device 101 displays a virtual part image representing the part of user 102 at the position corresponding to that part of user 102 in the virtual space 104.

[0025] As a result, while user 102 is moving in the virtual space 104, user 102's own movements are represented as an avatar.

[0026] Next, when user 102 moves to area 11, user 102's position becomes "area 11," which is the correction application location. Therefore, the information processing device 101 refers to the storage unit 110 and determines that it should correct the position where the virtual body part image is displayed. In this case, the information processing device 101 corrects the position where the virtual body part image representing user 102 is displayed according to the position corresponding to the body part of the expert that has been recorded in advance.

[0027] The position corresponding to the skilled worker's body part here is the position corresponding to the skilled worker's body part when the skilled worker performed task X in area 11 within the virtual space 104, and is recorded in advance. Specifically, for example, the information processing device 101 corrects the position for displaying the virtual body part image by integrating the position corresponding to the skilled worker's body part and the position corresponding to the user 102's body part in area 11 within the virtual space 104. Then, the information processing device 101 displays the virtual body part image representing the user 102's body part at the corrected position in the virtual space 104.

[0028] As a result, the information processing device 101 can use the movements of an expert to guide the user 102 when they perform task X. At this time, instead of simply displaying the expert's movements when task X is performed, the device can represent a fusion of the user 102's own movements and the expert's movements as an avatar, thereby guiding the user 102's body without them noticing.

[0029] After the completion of task X, until user 102 moves from area 11 to area 26, user 102's position is "area 10 ⇒ area 9 ⇒ area 8 ⇒ area 15 ⇒ area 22 ⇒ area 23 ⇒ area 24 ⇒ area 25," none of which are locations where correction is applied. Therefore, the information processing device 101 refers to the storage unit 110 and determines that no correction is performed on the position where the virtual part image is displayed. In this case, the information processing device 101 displays a virtual part image representing the part of user 102 at the position corresponding to that part of user 102 in the virtual space 104.

[0030] As a result, while user 102 is moving in the virtual space 104, user 102's own movements are represented as an avatar.

[0031] Next, when user 102 moves to area 26, user 102's position becomes "area 26," which is the correction application location. Therefore, the information processing device 101 refers to the storage unit 110 and determines that it should correct the position where the virtual body part image is displayed. In this case, the information processing device 101 corrects the position where the virtual body part image representing user 102 is displayed, according to the position corresponding to the body part of the expert that has been recorded in advance.

[0032] The position corresponding to the skilled worker's body part here is the position corresponding to the skilled worker's body part when the skilled worker performed task Y in area 26 within the virtual space 104, and is recorded in advance. Specifically, for example, the information processing device 101 corrects the position for displaying the virtual body part image by integrating the position corresponding to the skilled worker's body part and the position corresponding to the user 102's body part in area 26 within the virtual space 104. Then, the information processing device 101 displays the virtual body part image representing the user 102's body part at the corrected position in the virtual space 104.

[0033] As a result, the information processing device 101 can use the movements of an expert to guide the user 102 when they perform task Y. At this time, instead of simply displaying the movements of the expert when they perform task Y, the device can represent an avatar that is a fusion of the user 102's own movements and the expert's movements, thereby guiding the user 102's body without them noticing.

[0034] After the completion of task Y, until user 102 moves to area 28, user 102's position is "area 27 ⇒ area 28," neither of which is a location where correction is applied. Therefore, the information processing device 101 refers to the storage unit 110 and determines that no correction is performed on the position where the virtual part image is displayed. In this case, the information processing device 101 displays a virtual part image representing the part of user 102 at the position corresponding to that part of user 102 in the virtual space 104.

[0035] As a result, while user 102 is moving in the virtual space 104, user 102's own movements are represented as an avatar.

[0036] Thus, the information processing device 101 can reduce the unnaturalness of guiding the user 102's body in the virtual space 104. For example, when the user 102 is waiting or moving in the virtual space 104, the information processing device 101 represents the user 102's own movements as an avatar, thus eliminating unnecessary body guidance and preventing the user 102 from feeling uncomfortable.

[0037] Furthermore, once user 102 moves to the location where they will perform task X,Y in the virtual space 104, the information processing device 101 can guide their body using pre-recorded movements of an expert. This allows user 102 to learn the movements of an expert while maintaining a sense of self-control, by being guided only for predetermined actions (task X,Y), for example, to an unnoticed degree.

[0038] (Example of system configuration for information processing system 200) Next, an example of the system configuration of the information processing system 200, including the information processing device 101 shown in Figure 1, will be described. Here, the example will be described using the case where the information processing device 101 shown in Figure 1 is applied to the display control device 201 within the information processing system 200. The information processing system 200 can be applied, for example, to an educational training system for beginners to acquire the physical skills of experts.

[0039] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. In Figure 2, the information processing system 200 includes a display control device 201 and a plurality of HMDs 202. In the information processing system 200, the display control device 201 and the HMDs 202 are connected via a wired or wireless network 210. The network 210 is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0040] Here, the display control device 201 is a computer that has a skilled user avatar record DB (Database) 220 and controls the display of avatars on the HMD 202. The display control device 201 is, for example, a server. The display control device 201 may also be a PC (Personal Computer), a tablet PC, etc.

[0041] The contents of the expert avatar record DB220 will be described later using Figure 7.

[0042] HMD202 is a computer used by users of the information processing system 200. Users include, for example, beginners and experts. Beginners are those who undergo training to acquire the physical skills of experts. Experts are those who are proficient in the task being trained. HMD202 is a display device worn on the head, and can take the form of goggles, helmets, or glasses.

[0043] By wearing the HMD202 so that it covers both eyes, the user can experience a virtual space through the images displayed on the HMD202's screen. In the virtual space, the user can, for example, control an avatar displayed as their alter ego using the HMD202's input devices. The user may also be able to control the avatar using motion sensors.

[0044] The HMD202 may be connected to the display control device 201 via a PC or tablet PC. Furthermore, while the display control device 201 and HMD202 are provided as separate components in this description, this is not the only option. For example, the display control device 201 may be implemented using the HMD202. Also, while the HMD202 is used as an example of user-used equipment in this explanation, it is not limited to this. For example, a PC with a large display or a projector may be used instead of the HMD202.

[0045] (Example hardware configuration of the display control device 201) Next, we will describe an example of the hardware configuration of the display control device 201.

[0046] Figure 3 is a block diagram showing an example of the hardware configuration of the display control device 201. In Figure 3, the display control device 201 includes a CPU (Central Processing Unit) 301, memory 302, disk drive 303, disk 304, communication interface 305, portable recording medium interface 306, and portable recording medium 307. Each component is connected by a bus 300.

[0047] Here, the CPU 301 is responsible for the overall control of the display control device 201. The CPU 301 may have multiple cores. The memory 302 includes, for example, ROM (Read Only Memory) and RAM (Random Access Memory). The program stored in the memory 302 is loaded into the CPU 301, causing the CPU 301 to execute the coded process.

[0048] The disk drive 303 controls the reading and writing of data to the disk 304 according to the control of the CPU 301. The disk 304 stores the data written under the control of the disk drive 303. The disk 304 is, for example, a magnetic disk, an optical disk, etc.

[0049] The communication interface 305 is connected to the network 210 via a communication line, and through the network 210, it is connected to an external computer (for example, the HMD202 shown in Figure 2). The communication interface 305 manages the interface between the network 210 and the inside of the device, and controls the input and output of data from the external computer. The communication interface 305 is, for example, a modem or a LAN adapter.

[0050] The portable recording medium interface 306 controls the reading and writing of data to the portable recording medium 307 according to the control of the CPU 301. The portable recording medium 307 stores the data written under the control of the portable recording medium interface 306. The portable recording medium 307 is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0051] In addition to the components described above, the display control device 201 may also have, for example, an input device, a display, etc. Furthermore, the display control device 201 does not necessarily have to have, for example, the portable recording medium I / F 306 and the portable recording medium 307 among the components described above.

[0052] Furthermore, the HMD202 shown in Figure 2 can also be realized with the same hardware configuration as the display control device 201. However, in addition to the components described above, the HMD202 includes, for example, an input device, a display, various sensors, a camera, a speaker, a microphone, etc. The various sensors include, for example, an accelerometer, a gyroscope, and a geomagnetic sensor that detect the orientation and rotation direction of the body.

[0053] (Specific examples of avatars) Next, we will explain a specific example of an avatar using Figure 4.

[0054] Figure 4 is an explanatory diagram illustrating a specific example of an avatar. In Figure 4, avatar av is a character displayed in the virtual space as a representation of the user using HMD202. Avatar av is formed by a combination of objects (for example, objects 401-404) that represent parts of the user.

[0055] For example, object 401 represents the user's right shoulder. Object 402 represents the user's right upper arm. Object 403 represents the user's right forearm. Object 404 represents the user's right hand. The avatar av is managed by a hierarchical structure of objects, such as shown in Figure 5.

[0056] Figure 5 is an explanatory diagram showing an example of an object hierarchy. In Figure 5, the hierarchy 500 shows the hierarchical relationships between the objects that make up the avatar av shown in Figure 4. However, in the example in Figure 5, only a portion of the hierarchy for the avatar av is shown.

[0057] For example, hierarchical structure 500 indicates that directly below the object representing the upper chest are objects representing the left shoulder (Left_Shoulder), neck, and right shoulder (Right_Shoulder). Furthermore, hierarchical structure 500 indicates that directly below the object representing the left shoulder is an object representing the left upper arm (Left_UpperArm), directly below the object representing the left upper arm is an object representing the left forearm (Left_LowerArm), and directly below the object representing the left forearm is an object representing the left hand (Left_Hand). Additionally, hierarchical structure 500 indicates that directly below the object representing the right shoulder is an object representing the right upper arm (Right_UpperArm), directly below the object representing the right upper arm is an object representing the right forearm (Right_LowerArm), and directly below the object representing the right forearm is an object representing the right hand (Right_Hand).

[0058] In a virtual space, each object is managed by, for example, its coordinates (position) and rotation (orientation, orientation). For instance, the display of each object is controlled by specifying its relative coordinates and rotation from its parent object. For example, the parent object of the object representing the right hand (Right_Hand) is the object representing the right forearm (Right_LowerArm).

[0059] In the virtual space, only some objects of the avatar (parts of the user) may be displayed. For example, instead of displaying the entire avatar, only objects 401-404, representing the lower part of the body from the right shoulder down, may be displayed.

[0060] (Example of avatar display) Next, we will explain an example of avatar AV display using Figure 6. Here, we will explain using the example of a HMD202 user acting as a crane signaler and training in the task of giving signals (operational instructions) to the crane operator. A crane is a material handling machine that lifts heavy objects and moves them horizontally or vertically.

[0061] Figure 6 is an explanatory diagram showing an example of avatar display. In Figure 6, VR screen 600 is an example of an image representing the virtual space 601 that is displayed when giving operation instructions to the crane 602. VR screen 600 includes the left screen 610 and the right screen 620. The left screen 610 is an image of the avatar av viewed from the front (camera footage of the avatar av).

[0062] The right screen 620 shows the view from the user's (avatar av) perspective. The right screen 620 displays object 403 representing the user's (avatar av) right forearm and object 404 representing the right hand. The images on the left screen 610 and the right screen 620 move in sync.

[0063] By controlling an avatar (av) in the virtual space 601, users can perform tasks such as giving signals (operational instructions) to a crane operator as if they were doing so themselves. Here, training is being conducted on the task of giving the signal to raise the jib by bringing the right hand in front of the face, extending it to the upper left, and then bringing it to the upper right. The left hand is used to adjust the speed by continuously swinging it from side to side.

[0064] In addition, in VR screen 600, the left screen 610 may be hidden and the right screen 620 may be displayed in full screen.

[0065] (Memory contents of the expert avatar record DB220) Next, using Figure 7, the contents of the expert avatar record DB 220 of the display control device 201 will be explained. The expert avatar record DB 220 is implemented by storage devices such as memory 302 and disk 304.

[0066] Figure 7 is an explanatory diagram showing an example of the contents stored in the Expert Avatar Record DB220. In Figure 7, the Expert Avatar Record DB220 has fields for work ID, work name, operation definition file, and expert avatar record result. By setting information in each field, expert avatar information (for example, expert avatar information 700-1, 700-2) is stored as a record.

[0067] Here, the task ID is an identifier that uniquely identifies the task for which training is conducted using the virtual space. The task name is the name of the task. The operation definition file contains information used to switch display avatars. The display avatar is the avatar av displayed in the virtual space. A specific example of the operation definition file will be described later using Figure 12.

[0068] The expert avatar recording results are motion data representing the time-dependent changes in the position and orientation of the expert's body parts when they perform a task in a virtual space. The position and orientation of the expert's body parts correspond to the position (coordinates) and orientation (rotation, posture) of the object representing the expert's body part.

[0069] For example, expert avatar information 700-1 shows the work ID "W1", the work name "Line Work 1", the operation definition file F1, and the expert avatar record result R1. In the following explanation, work with work ID "#" may be written as "Work #".

[0070] (An example of a virtual space) Next, using Figure 8, we will explain an example of a virtual space that reproduces the environment in which the training task is performed. Here, we will use task W1 of task name "Line Work 1" as an example of the training task.

[0071] Figure 8 is an explanatory diagram illustrating an example of a virtual space. In Figure 8, the virtual space 800 is a three-dimensional digital space in which the environment in which work W1 is performed is reproduced. Work W1 is a type of line work performed in a manufacturing plant, etc., and is carried out according to the following steps (i) to (iv).

[0072] (i) Receive the assembled product from the tray that has come down the machine 801. (ii) Move to shelf 802, take the parts from the designated shelf and attach them to the assembly. (iii) Move to desk 803 and pick up the designated part. (iv) Move to machine 801, place the assembly and designated parts on the tray, and send out the tray. Then return to position (i) and wait for the next tray.

[0073] For example, by wearing the HMD202 shown in Figure 2, the user can become an avatar av and experience the task W1 (procedures (i) to (iv)) in the virtual space 800.

[0074] (Example of functional configuration of the display control device 201) Next, we will describe an example of the functional configuration of the display control device 201.

[0075] Figure 9 is a block diagram showing an example of the functional configuration of the display control device 201. In Figure 9, the display control device 201 includes a reception unit 901, a display control unit 902, a recording unit 903, a determination unit 904, a correction unit 905, and a storage unit 910. The reception unit 901 to the correction unit 905 function as the control unit 900, and specifically, they realize their functions by having the CPU 301 execute a program stored in a storage device such as the memory 302, disk 304, or portable recording medium 307 shown in Figure 3, or by using the communication I / F 305. The processing results of each functional unit are stored in a storage device such as the memory 302 or disk 304. The storage unit 910 is realized by a storage device such as the memory 302 or disk 304. Specifically, for example, the storage unit 910 stores the expert avatar record DB 220 shown in Figure 7.

[0076] Reception unit 901 receives requests for recording expert avatars. Here, an expert avatar recording request is a request to record the movements of the avatar corresponding to the expert when the expert performs a specified task in the virtual space. The specified task is the task to be trained. The training is a training program in which beginners and others aim to acquire the physical skills of experts.

[0077] A skilled user avatar recording request includes, for example, a user ID, a task ID, and a task name. The user ID included in the skilled user avatar recording request is an identifier that uniquely identifies a skilled user. The task ID and task name included in the skilled user avatar recording request are the task ID and task name of a given task.

[0078] Specifically, for example, the reception unit 901 receives a request to record a skilled user avatar by receiving it from the HMD202 (see Figure 2) used by a skilled user. Alternatively, the reception unit 901 may also receive a request to record a skilled user avatar by receiving an input from a user (skilled user) using an input device (not shown) of the device.

[0079] When the display control unit 902 receives a request to record a skilled worker avatar, it displays the skilled worker avatar in the virtual space. Here, the virtual space is a three-dimensional digital space in which the environment in which a predetermined task is performed is reproduced. The virtual space is generated, for example, for each predetermined task. Information for displaying the virtual space is stored in a storage device such as memory 302 or disk 304, associated with, for example, the task ID of the predetermined task.

[0080] The expert avatar is an avatar AV that directly reflects the movements of an expert user. The expert avatar is represented by a combination of objects (virtual body part images) that represent the body parts of the expert user. The design of the avatar AV can be freely configured (see, for example, Figure 4).

[0081] The display destination for the expert avatar is, for example, the HMD202 that received the expert avatar recording request. Specifically, for example, the display control unit 902 displays the expert avatar (avatar av) in the virtual space displayed on the HMD202, which operates in accordance with the movements of the user (expert) wearing the HMD202. The virtual space is the virtual space corresponding to the work ID included in the received expert avatar recording request. Note that the information for displaying the virtual space may be stored in the HMD202, for example.

[0082] In the following explanation, a user who is an expert may sometimes be simply referred to as an "expert."

[0083] The recording unit 903 records the positions corresponding to the body parts of the expert in the virtual space. Here, the positions corresponding to the body parts of the expert are the positions of the virtual body part images representing the body parts of the expert in the virtual space. The positions corresponding to the body parts of the expert may also be expressed as relative positions to virtual body part images representing other body parts of the expert in the virtual space. The virtual body part images correspond to the objects that make up the avatar AV.

[0084] Specifically, for example, the recording unit 903 records motion data representing the time change in the position of a part of the skilled worker when the skilled worker performs a predetermined task in the virtual space. The predetermined task is identified, for example, by a task ID included in the skilled worker avatar recording request.

[0085] Furthermore, the motion data may represent not only the position corresponding to the skilled person's body part, but also the time change in orientation corresponding to the skilled person's body part. The orientation corresponding to the skilled person's body part is the orientation of the virtual body part image representing the skilled person's body part in the virtual space. The orientation corresponding to the skilled person's body part may also be expressed by its relative orientation to the virtual body part images representing other parts of the skilled person in the virtual space.

[0086] Motion data may, for example, represent the time-dependent changes in the position and orientation of each object of a skilled avatar in a virtual space. The orientation of each object corresponds to information that identifies the rotation and posture of each object in the virtual space. Motion data may also, for example, be video data that can reproduce the movements of a skilled avatar in a virtual space.

[0087] As an example, let's assume the task ID included in the expert avatar recording request is "W1". In this case, the recording unit 903 records motion data representing the time change in the position and orientation of objects corresponding to the expert's body parts when the expert wearing the HMD202 performs task W1 in the virtual space 800 (see Figure 8) displayed on the HMD202.

[0088] The recorded operation data is stored in the expert avatar record DB220 shown in Figure 7, associated with the work ID (e.g., W1) included in the expert avatar record request, for example, as the expert avatar record result (e.g., R1).

[0089] Furthermore, the reception unit 901 receives training commencement requests. Here, a training commencement request is a request to start training aimed at acquiring the physical skills of an expert for a specified task, using a virtual space. A training commencement request includes, for example, a user ID, a task ID, and a task name.

[0090] The user ID included in the training start request is an identifier that uniquely identifies the user who is the trainer. The trainer is, for example, a beginner. The task ID and task name included in the training start request are the task ID and task name of the task to be trained (the specified task).

[0091] Specifically, for example, the reception unit 901 receives a training start request from the HMD202 used by a beginner user. Alternatively, the reception unit 901 may also receive a training start request through user (beginner) operation input using an input device (not shown) of the device.

[0092] When the display control unit 902 receives a training start request, it displays a beginner avatar or a fused avatar in the virtual space. The beginner avatar is an avatar AV that directly reflects the movements of a beginner user. The beginner avatar is represented by a combination of objects (virtual body part images) that represent the body parts of the beginner user.

[0093] A fused avatar is an avatar created by fusing a beginner avatar and a skilled avatar. The fused avatar is generated by correcting the position of the virtual body part images representing the beginner's body parts according to the positions corresponding to the skilled avatar's body parts in the virtual space. The positions corresponding to the skilled avatar's body parts in the virtual space correspond to the positions of the skilled avatar's objects in the virtual space. The virtual body part images representing the beginner's body parts correspond to the objects of the beginner avatar. The display destination for the beginner avatar or the fused avatar is, for example, the HMD202 that received the training start request.

[0094] In the following explanation, a user who is a beginner may sometimes be simply referred to as "beginner."

[0095] Specifically, for example, first, in response to a training start request, the display control unit 902 displays a beginner avatar (avatar av) in the virtual space displayed on the HMD202, which operates according to the movements of the beginner wearing the HMD202. The virtual space is the virtual space corresponding to the work ID included in the training start request. Displaying the beginner avatar corresponds, for example, to the start of training.

[0096] The determination unit 904 refers to the storage unit 910 and determines whether or not to correct the position of the virtual body part image representing the beginner's body part, based on the beginner's position in the virtual space. The storage unit 910 stores information (for example, an operation definition file) that identifies the location in the virtual space where the position of the virtual body part image representing the beginner's body part is corrected.

[0097] The position of the virtual body part images is corrected according to the positions corresponding to the body parts of the expert in the pre-recorded virtual space. The position of the beginner in the virtual space corresponds to the position of the beginner avatar in the virtual space. The position of the beginner avatar may be determined by the position of any object of the beginner avatar.

[0098] For example, the position of a beginner avatar may be determined by the position of an object representing the beginner avatar's head. In this case, the beginner's position in the virtual space will be the position of the object representing the beginner avatar's head. Alternatively, the position of a beginner avatar may be determined by the position of an object representing the beginner avatar's left foot. In this case, the beginner's position in the virtual space will be the position of the object representing the beginner avatar's left foot.

[0099] In the following explanation, the position of a beginner in the virtual space may be referred to as the "beginner avatar's position."

[0100] Specifically, for example, the determination unit 904 refers to the expert avatar record DB220 and obtains an action definition file corresponding to the work ID included in the training start request. The action definition file contains information that identifies the location in the virtual space where the position of displaying the beginner avatar object (a virtual body part image representing the beginner's body parts) is corrected.

[0101] Furthermore, the determination unit 904 acquires the position of the beginner avatar in the virtual space (beginner's position). The beginner avatar's position is acquired, for example, at predetermined time intervals during training. The predetermined time intervals can be set arbitrarily. The determination unit 904 then refers to the acquired motion definition file and determines whether the acquired position of the beginner avatar is a location where correction should be applied.

[0102] The correction application location is the location where the position of the virtual body part image representing the beginner's body part is corrected. Here, if the beginner avatar's position is the correction application location, the determination unit 904 determines that the position of the virtual body part image should be corrected. On the other hand, if the beginner avatar's position is not the correction application location, the determination unit 904 determines that the position of the virtual body part image should not be corrected.

[0103] The method for managing the position of beginner avatars in the virtual space will be described later using Figures 10 and 11. Furthermore, a concrete example of an operation definition file will be described later using Figure 12.

[0104] Furthermore, the determination unit 904 may determine whether or not to correct the position of the virtual body part image based not only on the position of the beginner avatar in the virtual space, but also on the orientation of the beginner avatar in the virtual space. In this case, the storage unit 910 stores information (for example, an operation definition file) that identifies the location and orientation in the virtual space for correcting the position of the virtual body part image representing the beginner's body parts.

[0105] The orientation of a beginner avatar may be determined by the orientation of one of the objects representing the beginner avatar. For example, the orientation of a beginner avatar may be determined by the orientation of the object representing the beginner avatar's head. In this case, the orientation of the beginner avatar in the virtual space would be the orientation of the object representing the beginner avatar's head.

[0106] Specifically, for example, the determination unit 904 acquires the position and orientation of the beginner avatar in the virtual space. The position and orientation of the beginner avatar are acquired, for example, at predetermined time intervals during training. The determination unit 904 then refers to the acquired motion definition file and determines from the acquired position and orientation of the beginner avatar whether or not it is a location where correction should be applied.

[0107] Here, if the beginner avatar's position is a correction application location, the determination unit 904 determines that it will correct the position where the virtual body part image is displayed. On the other hand, if the beginner avatar's position is not a correction application location, the determination unit 904 determines that it will not correct the position where the virtual body part image is displayed.

[0108] Furthermore, an example of whether or not to correct the position of the virtual body part image representing the body part for beginners will be described later using Figures 10 to 12.

[0109] If the correction unit 905 determines that it is necessary to correct the position where a virtual body part image representing a beginner's body part is displayed, it corrects the position where the virtual body part image is displayed according to the position corresponding to the expert's body part in the virtual space, which has been recorded in advance. For example, the correction unit 905 corrects the position where the virtual body part image is displayed by integrating the position corresponding to the expert's body part and the position corresponding to the beginner's body part at the correction application location in the virtual space. The position corresponding to the beginner's body part is the real-time position.

[0110] Furthermore, when the correction unit 905 corrects the position in which the virtual body part image is displayed, it may adjust the position of the expert avatar to match the position of the beginner avatar. For example, the correction unit 905 may adjust the position of the object representing the left foot of the expert avatar to match the position of the object representing the left foot of the beginner avatar, and then correct the position in which the virtual body part image is displayed.

[0111] The body parts to be corrected can be set arbitrarily. For example, all body parts of a beginner may be set as the target of correction. Also, depending on the training content, there may be cases where only certain body parts should be guided based on the movements of an expert. In this case, only those body parts that need to be guided may be set as the target of correction. These body parts may include, for example, "both arms," ​​"right arm," "left arm," "upper body," and "lower body."

[0112] Furthermore, the body parts to be corrected may be set in association with a predetermined task. For example, information identifying the body parts to be corrected in association with the task ID of a predetermined task may be stored in a storage device such as memory 302 or disk 304. In this case, the correction unit 905 may identify the body parts to be corrected that correspond to the task ID (predetermined task) included in the training start request. The correction unit 905 may then correct the position in which the virtual body part image is displayed for the identified body parts to be corrected among the body parts of a beginner, according to the position corresponding to the body parts of an expert in a pre-recorded virtual space.

[0113] Furthermore, the correction unit 905 does not simply correct the position corresponding to the expert's body part to the position corresponding to the beginner's body part. Instead, for example, it interpolates the position of the fused avatar's object from the position corresponding to the expert's body part and the position corresponding to the beginner's body part. The position of the fused avatar's object corresponds to the corrected position corresponding to the beginner's body part. However, the position of the object representing a body part other than the one being corrected in the fused avatar becomes the position of the object representing the same body part in the beginner avatar.

[0114] Specifically, for example, the correction unit 905 corrects the position where the virtual body part image is displayed by referring to the recorded motion data and integrating the positions corresponding to the skilled user's body parts and the beginner's body parts at regular time intervals. The regular time interval can be set arbitrarily; for example, if the motion data is video data and its frame rate is around several tens of fps, it may be a time interval of 1 to several frames. The time interval at which the position and orientation of the beginner avatar are acquired is, for example, the same as or approximately the same as this regular time interval.

[0115] To explain in more detail, for example, the correction unit 905 refers to the expert avatar record DB220 and obtains the expert avatar record result corresponding to the work ID included in the training start request. Next, the correction unit 905 seeks the playback position of the expert avatar record result until the position of the expert (expert avatar) becomes the position of the beginner.

[0116] The beginner's position is the real-time position of the beginner's avatar, and corresponds to the position where, for example, correction of the display position of the beginner's avatar object is determined. The correction unit 905 may seek the playback position of the expert avatar recording result until the position and orientation of the expert (expert avatar) become the position and orientation of the beginner (beginner avatar).

[0117] Then, the correction unit 905 refers to the expert avatar recording results from the sought playback position and interpolates the position of the fused avatar's objects from the positions of the expert avatar's objects and the beginner avatar's objects at regular time intervals. During this process, the expert avatar is hidden.

[0118] This allows the correction unit 905 to interpolate the position of the object of the fused avatar from the positions of the expert avatar's object and the beginner avatar's object at the same timing. As a result, the fused avatar is generated.

[0119] Furthermore, the correction unit 905 may correct the orientation of the virtual body part image displayed according to the orientation corresponding to the expert's body part in the virtual space, which has been recorded in advance. The correction of the orientation of the virtual body part image is performed together with the correction of the position where the virtual body part image is displayed. For example, the correction unit 905 may correct the orientation of the virtual body part image representing the beginner's body part by integrating the orientation corresponding to the expert's body part and the orientation corresponding to the user's body part at the correction application location in the virtual space. The orientation corresponding to the beginner's body part is the real-time orientation.

[0120] In this case, the correction unit 905 does not simply correct the position corresponding to the expert's body part to match the orientation of the beginner's body part. Instead, for example, it interpolates the orientation of the fused avatar's objects from the orientation corresponding to the expert's body part and the orientation corresponding to the beginner's body part. The orientation of the fused avatar's objects corresponds to the corrected orientation corresponding to the beginner's body part. However, the orientation of objects representing body parts other than those being corrected within the fused avatar becomes the orientation of objects representing the same body parts of the beginner avatar.

[0121] Specifically, for example, the correction unit 905 refers to the recorded motion data and, at regular time intervals, corrects the orientation in which the beginner avatar's object is displayed by integrating the orientations corresponding to the skilled user's body parts and the orientations corresponding to the beginner user's body parts.

[0122] To explain in more detail, for example, the correction unit 905 refers to the expert avatar recording results from the sought playback position and interpolates the orientation of the fused avatar's objects from the orientation of the expert avatar's objects and the orientation of the beginner avatar's objects at regular time intervals.

[0123] As a result, the correction unit 905 can interpolate the position and orientation of the objects of the fused avatar from the position and orientation of the objects of the expert avatar and the beginner avatar at the same timing. As a result, the fused avatar is generated.

[0124] An example of interpolation of the position and orientation of objects in a fused avatar will be explained using Figure 13.

[0125] The display control unit 902 displays a virtual body part image representing the beginner's body part at a corrected position in the virtual space. The virtual body part image displayed at the corrected position corresponds, for example, to an object of the fused avatar. Alternatively, the display control unit 902 may display a virtual body part image representing the beginner's body part at a corrected position in the virtual space with a corrected orientation. The virtual body part image displayed at a corrected position with a corrected orientation corresponds, for example, to an object of the fused avatar. Specifically, for example, the display control unit 902 displays the generated fused avatar in the virtual space displayed on the HMD202. In this case, the display control unit 902 hides the beginner's avatar.

[0126] Furthermore, if the display control unit 902 determines that no correction is needed for the position where the virtual body part image representing the beginner's body part is displayed, it displays the virtual body part image representing the beginner's body part at the position corresponding to the beginner's body part in the virtual space. Specifically, for example, the display control unit 902 displays the beginner avatar in the virtual space displayed on the HMD202.

[0127] (Example of location management in a virtual space) Next, using Figures 10 and 11, we will explain an example of managing the position of a beginner avatar in a virtual space. Here, we will use the virtual space 800 shown in Figure 8 as an example. First, using Figure 10, we will explain the first example of position management in the virtual space.

[0128] Figure 10 is an explanatory diagram showing a first example of position management in a virtual space. In Figure 10, the virtual space 800 is a three-dimensional digital space in which the environment for performing the task W1 described in Figure 8 is reproduced. Here, the virtual space 800 is divided into a grid when viewed from directly above, and is divided into multiple regions. Each region is identified by a combination of row numbers (A, B, C, D, E, F, G, H, I) and column numbers (α, β, γ, Δ). For example, the location for performing procedure (ii) is region Bβ. Also, the location for performing procedure (iii) is region Hα.

[0129] Next, we will explain a second example of location management in a virtual space using Figure 11.

[0130] Figure 11 is an explanatory diagram illustrating a second example of location management in a virtual space. In Figure 11, regions 1101 and 1102 are set within the virtual space 800. Each region 1101 and 1102 is managed by its relative distance (direction, distance) from a reference point P0 set within the virtual space 800. For example, the location where procedure (ii) is performed is region 1101. The location where procedure (iii) is performed is region 1102.

[0131] (Example of an operation definition file) Next, we will explain a concrete example of an operation definition file using Figure 12. Here, we will use the operation definition file F1 for task W1 as an example. We will also explain the case where the position of the beginner avatar in virtual space 800 is managed using the method shown in Figure 10 (first position management example).

[0132] Figure 12 is an explanatory diagram showing a specific example of an operation definition file. In Figure 12, operation definition file F1 is used to switch the avatar av (display avatar) displayed when performing task W1 in virtual space 800 (see Figure 8). Operation definition file F1 contains operation definition information 1201 to 1203.

[0133] Each of the operation definition information items 1201 to 1203 indicates the beginner avatar position, the beginner avatar orientation, and the avatar for display. The beginner avatar position indicates the position of the beginner avatar in virtual space 800. For example, the beginner avatar position "Bβ" corresponds to region Bβ (see Figure 8) in virtual space 800. Also, the beginner avatar position "*" indicates no specification. Multiple regions may be specified as the beginner avatar position. For example, if regions Bβ and Cβ in virtual space 800 are specified, the beginner avatar position will be "Bβ+Cβ".

[0134] The beginner avatar orientation indicates the orientation of the beginner avatar in virtual space 800. Based on the beginner avatar's position, the beginner avatar orientation indicates the direction the beginner avatar (e.g., face) is facing. For example, the beginner avatar orientation "Bα" in the motion definition information 1201 indicates that the beginner avatar is facing from region Bβ to region Bα (see Figure 9) within virtual space 800. Note that "*" indicates no orientation is specified.

[0135] The display avatar indicates the type of display avatar. Mix indicates a fused avatar. Beginner indicates a beginner avatar. For example, the operation definition information 1201 indicates the display avatar "Mix" when the position and orientation of the beginner avatar in virtual space 800 are beginner avatar position "Bβ" and beginner avatar orientation "Bα".

[0136] For example, the determination unit 904 refers to the operation definition file F1 shown in Figure 12 and determines whether or not to correct the position of the virtual body part image representing the beginner's body part, based on the position and orientation of the beginner's avatar in the virtual space 800.

[0137] Here, the position of the beginner avatar in the virtual space 800 is defined as "Dβ," and the orientation of the beginner avatar is defined as "Cβ." In this case, the determination unit 904 refers to the operation definition file F1 to identify the operation definition information 1203 corresponding to the beginner avatar's position "Dβ" and orientation "Cβ." Then, the determination unit 904 refers to the identified operation definition information 1203 to identify the display avatar "Beginner."

[0138] The display avatar "Beginner" indicates that the avatar av displayed in virtual space 800 is a beginner avatar. In other words, the display avatar "Beginner" indicates that the position of the beginner avatar is not a correction application area, and therefore no correction will be applied to the position where the virtual body part image representing the beginner's body part is displayed.

[0139] Furthermore, the position of the beginner avatar in the virtual space 800 is defined as "Bβ," and the orientation of the beginner avatar is defined as "Bα." In this case, the determination unit 904 refers to the operation definition file F1 to identify the operation definition information 1201 corresponding to the beginner avatar's position "Bβ" and orientation "Bα." Then, the determination unit 904 refers to the identified operation definition information 1201 to identify the display avatar "Mix."

[0140] The display avatar "Mix" indicates that the avatar av displayed in virtual space 800 is a fused avatar. In other words, the display avatar "Mix" indicates that the position of the beginner avatar is the correction application location, and that the position of the virtual body part image representing the beginner's body part will be corrected.

[0141] As a result, the display control device 201 can determine that work has begun at shelf 802 (start of procedure (ii)) when the beginner avatar enters area Bβ in the virtual space 800 and faces towards shelf 802 (see Figure 8), and can start displaying the fused avatar. Furthermore, when work at shelf 802 (procedure (ii)) is completed and, for example, the beginner avatar moves to area Cβ in the virtual space 800, the display avatar can be switched from the fused avatar to the beginner avatar.

[0142] Furthermore, the position of the beginner avatar in the virtual space 800 is defined as "Hα," and the orientation of the beginner avatar is defined as "Iα." In this case, the determination unit 904 refers to the operation definition file F1 to identify the operation definition information 1202 corresponding to the position "Hα" and orientation "Iα" of the beginner avatar. Then, the determination unit 904 refers to the identified operation definition information 1202 to identify the display avatar "Mix."

[0143] As a result, the display control device 201 can determine that work has begun at desk 803 (start of procedure (iii)) when the beginner avatar enters area Hα in the virtual space 800 and faces towards desk 803 (see Figure 8), and can start displaying the fused avatar. Furthermore, when work at desk 803 (procedure (iii)) is completed and, for example, the beginner avatar moves to area Hβ in the virtual space 800, the display avatar can be switched from the fused avatar to the beginner avatar.

[0144] Furthermore, when the determination unit 904 determines whether or not to correct the position of the virtual body part image based solely on the position of the beginner avatar in the virtual space 800, it does not consider the orientation of the beginner avatar in the operation definition file F1. For example, suppose the position of the beginner avatar in the virtual space 800 is "Bβ". In this case, the determination unit 904 refers to the operation definition file F1 to identify the operation definition information 1201 corresponding to the beginner avatar's position "Bβ". Then, the determination unit 904 refers to the identified operation definition information 1201 to identify the display avatar "Mix".

[0145] Furthermore, when the position of the beginner avatar within the virtual space 800 is managed using the method shown in Figure 11 (second position management example), the determination unit 904 determines whether the beginner avatar's position is included in regions 1101 and 1102. The beginner avatar's position is represented by the relative distance (direction, distance) from a reference point P0 set within the virtual space 800.

[0146] The determination unit 904 then determines that if the beginner avatar's position is within areas 1101 and 1102, the beginner avatar's position is a correction application location, and therefore the position for displaying the virtual body part image representing the beginner's body parts will be corrected. If the beginner avatar's position is not within areas 1101 and 1102, the determination unit 904 determines that the beginner avatar's position is not a correction application location, and therefore the position for displaying the virtual body part image representing the beginner's body parts will not be corrected.

[0147] (Example of interpolation of object position and orientation in a fused avatar) Next, using Figure 13, we will explain an example of interpolation of the position and orientation of objects in a fused avatar.

[0148] Figure 13 is an explanatory diagram illustrating an example of interpolation of the position and orientation of objects in a fused avatar. In Figure 13, objects 1301-1305 are objects of the beginner avatar. Object 1301 represents the upper chest of the beginner avatar. Object 1302 represents the shoulder of the beginner avatar. Object 1303 represents the upper arm of the beginner avatar. Object 1304 represents the forearm of the beginner avatar. Object 1305 represents the hand of the beginner avatar.

[0149] Furthermore, objects 1311-1314 are objects for the expert avatar. Object 1311 represents the expert avatar's shoulder. Object 1312 represents the expert avatar's upper arm. Object 1313 represents the expert avatar's forearm. Object 1314 represents the expert avatar's hand.

[0150] Furthermore, objects 1321-1324 are objects of the fused avatar. Object 1321 represents the shoulder of the fused avatar. Object 1322 represents the upper arm of the fused avatar. Object 1323 represents the forearm of the fused avatar. Object 1324 represents the hand of the fused avatar.

[0151] Objects 1302-1305, 1311-1314, and 1321-1324 represent either the left or right shoulder, upper arm, forearm, or hand. Furthermore, the areas to be corrected are defined here as the shoulder, upper arm, forearm, and hand.

[0152] The position of each object (for example, objects 1301-1305) is managed, for example, by its relative coordinates from its parent object (see, for example, Figure 5). The orientation of each object is managed, for example, by its relative angle to its parent object.

[0153] Here, taking an object representing the shoulder (for example, object 1302 of the beginner avatar) as an example, its position is managed by relative coordinates from the parent object representing the upper chest (for example, object 1301 of the beginner avatar), and its orientation is managed by the relative angle with respect to the parent object.

[0154] In this case, the correction unit 905 interpolates the position of object 1321 of the fused avatar from the position of object 1311 of the expert avatar and the position of object 1302 of the beginner avatar. Specifically, for example, the correction unit 905 can interpolate the position of object 1321 of the fused avatar using the following formula (1). Here, α represents the fusion rate. The fusion rate can be arbitrarily set in the range of 0 to 1, and is set to a value of approximately 0.5, for example.

[0155] The position of object 1321 of the fused avatar = α × the position of object 1302 of the beginner avatar + (1-α) × the position of object 1311 of the expert avatar ... (1)

[0156] Furthermore, the correction unit 905 interpolates the orientation of object 1321 of the fused avatar from the orientation of object 1311 of the expert avatar and the orientation of object 1302 of the beginner avatar. Specifically, for example, the correction unit 905 can interpolate the orientation of object 1321 of the fused avatar using the following equation (2).

[0157] The orientation of object 1321 of the fused avatar = α × the orientation of object 1302 of the beginner avatar + (1-α) × the orientation of object 1311 of the expert avatar ... (2)

[0158] The correction unit 905 can interpolate the position and orientation of other objects 1322 to 1324 of the fused avatar, in the same way as object 1321.

[0159] Furthermore, the fusion rate α may be changed dynamically. For example, the correction unit 905 may increase the value of the fusion rate α included in the above equations (1) and (2) if the error between the position of the beginner avatar's object and the position of the expert avatar's object for the same part exceeds a threshold.

[0160] Specifically, for example, the correction unit 905 calculates the vector distance between the beginner avatar object 1305 and the expert avatar object 1314. The vector distance is obtained, for example, by calculating the absolute coordinates of each object 1305 and 1314 in the virtual space 800 and determining the distance between those coordinates. Then, if the calculated vector distance becomes greater than or equal to the threshold Th1, the correction unit 905 may change the value of the fusion rate α from "0.5" to "0.9". The threshold Th1 can be set arbitrarily.

[0161] The correction unit 905 may also calculate the difference in orientation between the beginner avatar object 1305 and the expert avatar object 1314. The orientation of each object 1305 and 1314 in this case may be, for example, the orientation relative to the shoulders of each avatar. The correction unit 905 may then change the value of the fusion rate α from, for example, "0.5" to "0.9" if the calculated difference in orientation is greater than or equal to the threshold Th2. The threshold Th2 can be set arbitrarily. The correction unit 905 may also change the value of the fusion rate α from, for example, "0.5" to "0.9" if the calculated distance between vectors is greater than or equal to the threshold Th1 and the calculated difference in orientation is greater than or equal to the threshold Th2.

[0162] This allows the display control device 201 to avoid causing discomfort to the user (beginner) by keeping the beginner avatar's state relatively unchanged when there is a large discrepancy in the state (position and orientation) between the beginner avatar and the experienced avatar.

[0163] Next, we will explain a more specific interpolation example for the position and orientation of the fused avatar's objects. Here, we will assume that the position (coordinates) of the fused avatar's objects is managed by Vector3(x,y,z), and the orientation (rotation) of the fused avatar's objects is managed by Quaternion(x,y,z,w).

[0164] Regarding Quaternions, for example, see "Edo, "Notes on Quaternions (Rotation)", [online], June 25, 2018, Hatena Blog, [Retrieved October 10, 2024], Internet<URL:https: / / edom18.hateblo.jp / entry / 2018 / 06 / 25 / 084023> You can refer to "[...]." Also, regarding Quaternions, see "Unity Documentation, Script Reference, Quaternion, [online], [searched November 18, 2024], Internet."<URL:https: / / docs.unity3d.com / ja / 560 / ScriptReference / Quaternion.html> You can refer to "[...]."

[0165] Furthermore, the interpolation of the object's position (coordinates) in the fused avatar will be performed using Lerp.

[0166] Regarding Lerp, for example, see "Yuna Kaihatsu, "How to use Vector3.Lerp", [online], April 15, 2018, Qiita, [Searched October 10, 2024], Internet<URL:https: / / qiita.com / aimy-07 / items / ad0d99191da21c0adbc3> You can refer to "[...]."

[0167] Furthermore, Slerp will be used to interpolate the orientation (rotation) of the fused avatar objects.

[0168] Regarding Slerp, for example, see "Edo, "Notes on Quaternions (Rotation)", [online], June 25, 2018, Hatena Blog, [Retrieved October 10, 2024], Internet<URL:https: / / edom18.hateblo.jp / entry / 2018 / 06 / 25 / 084023#クォータニオンのSlerp> You can refer to the following for information on Slerp: "Unity Documentation, Script Reference, Quaternion.Slerp, [online], [searched November 18, 2024], Internet<URL:https: / / docs.unity3d.com / ja / 560 / ScriptReference / Quaternion.Slerp.html> You can refer to the following. Also, for Vector3, see, for example, "Unity Documentation, Script Reference, Vector3, [online], [Searched November 18, 2024], Internet<URL:https: / / docs.unity3d.com / ja / 560 / ScriptReference / Vector3.html> You can refer to the following. Also, for Vector3.Slerp, see, for example, "Unity Documentation, Script Reference, Vector3.Slerp, [online], [Searched November 18, 2024], Internet<URL:https: / / docs.unity3d.com / ja / 560 / ScriptReference / Vector3.Slerp.html> You can refer to "[...]."

[0169] In this case, the correction unit 905 can interpolate the position (coordinates) of the fused avatar's objects using, for example, the following equation (3). Here, α represents the fusion rate. A represents the beginner avatar's objects. B represents the expert avatar's objects. C represents the fused avatar's objects.

[0170] Relative coordinate of C = Vector3.Lerp(relative coordinate of A, relative coordinate of B, α) ...(3)

[0171] Furthermore, the correction unit 905 can interpolate the orientation (rotation) of the fused avatar object using, for example, the following equation (4).

[0172] Relative rotation of C = Quaternion.Slerp(Relative rotation of A, Relative rotation of B, α) ... (4)

[0173] Furthermore, the positions (coordinates) of the fused avatar other than the objects can be expressed using, for example, equation (5) below. Also, the orientation (rotation) of the fused avatar other than the objects can be expressed using, for example, equation (6) below.

[0174] Relative coordinates of C = Relative coordinates of A ... (5) Relative rotation of C = Relative rotation of A ... (6)

[0175] (Various processing procedures of the display control device 201) Next, we will explain the various processing procedures of the display control device 201. First, we will explain the expert avatar recording processing procedure of the display control device 201 using Figure 14.

[0176] Figure 14 is a flowchart showing an example of the expert avatar recording process procedure of the display control device 201. In the flowchart of Figure 14, first, the display control device 201 determines whether or not it has received an expert avatar recording request from the HMD 202 (step S1401). Here, the display control device 201 waits to receive the expert avatar recording request (step S1401: No).

[0177] When the display control device 201 receives a request to record a skilled worker avatar (step S1401: Yes), it displays a skilled worker avatar (avatar av) in the virtual space displayed on the HMD 202, which operates in accordance with the movements of the user (skilled worker) wearing the HMD 202 (step S1402). The virtual space displayed on the HMD 202 is the virtual space corresponding to the work ID included in the skilled worker avatar recording request.

[0178] Then, the display control device 201 starts recording the operation data for the skilled avatar (step S1403). Next, the display control device 201 determines whether or not the recording of the operation data for the skilled avatar has finished (step S1404). The recording of operation data ends, for example, in response to a termination instruction from the HMD 202.

[0179] Here, the display control device 201 waits for the recording of the operation data to finish (step S1404: No). Then, when the recording of the operation data is finished (step S1404: Yes), the display control device 201 registers the recorded operation data as the expert avatar recording result in the expert avatar recording DB 220 (step S1405), and finishes the series of processes according to this flowchart. The expert avatar recording result is registered in association with the work ID included in the expert avatar recording request.

[0180] This allows the display control device 201 to pre-record the movements of the expert's avatar when the expert performs a predetermined task in a virtual space, prior to training for that task.

[0181] Next, the display control processing procedure of the display control device 201 will be explained using Figures 15 and 16.

[0182] Figures 15 and 16 are flowcharts illustrating an example of the display control processing procedure of the display control device 201. In the flowchart of Figure 15, first, the display control device 201 determines whether or not it has received a training start request from the HMD 202 (step S1501). Here, the display control device 201 waits to receive the training start request (step S1501: No).

[0183] If the display control device 201 receives a training start request (step S1501: Yes), it refers to the expert avatar record DB 220 and obtains the operation definition file corresponding to the work ID included in the training start request (step S1502). Next, the display control device 201 refers to the expert avatar record DB 220 and obtains the expert avatar record result corresponding to the work ID included in the training start request (step S1503).

[0184] Then, the display control device 201 displays a beginner avatar (avatar av) in the virtual space displayed on the HMD202, which operates in accordance with the movements of the user (beginner) wearing the HMD202 (step S1504). The virtual space displayed on the HMD202 is the virtual space corresponding to the work ID included in the training start request.

[0185] Next, the display control device 201 starts acquiring the position and orientation of the beginner avatar in the virtual space (step S1505). The acquisition of the beginner avatar's position and orientation is performed at predetermined time intervals during training. Then, the display control device 201 identifies the most recent position and orientation of the beginner avatar from among the acquired positions and orientations (step S1506).

[0186] Next, the display control device 201 refers to the acquired operation definition file to identify the display avatar corresponding to the position and orientation of the most recently identified beginner avatar (step S1507), and then proceeds to step S1601 shown in Figure 16.

[0187] In the flowchart of Figure 16, first, the display control device 201 determines whether the identified display avatar is Mix or not (step S1601). If the display avatar is not Mix (step S1601: No), the display control device 201 displays a beginner avatar in the virtual space displayed on the HMD 202 (step S1602) and proceeds to step S1607.

[0188] On the other hand, if the display avatar is Mix (step S1601: Yes), the display control device 201 seeks the playback position of the expert avatar recording result until the position of the expert avatar becomes the position of the beginner avatar (step S1603). In this case, the display control device 201 may seek the playback position of the expert avatar recording result until the position and orientation of the expert avatar become the position and orientation of the beginner avatar.

[0189] Next, the display control device 201 generates a fused avatar by referring to the expert avatar recording results from the sought playback position (step S1604). Specifically, for example, the display control device 201 generates a fused avatar by playing the expert avatar invisibly from the sought playback position and interpolating the object positions of the fused avatar from the object positions of the expert avatar and the object positions of the beginner avatar.

[0190] Then, the display control device 201 displays the generated fused avatar in the virtual space displayed on the HMD 202 (step S1605). At this time, the display control device 201 hides the beginner avatar. The processes in steps S1604 and S1605 are repeatedly executed at regular time intervals.

[0191] Next, the display control device 201 determines whether or not the beginner avatar (hidden) has moved in the virtual space displayed on the HMD 202 (step S1606). For example, the display control device 201 may determine that the beginner avatar (hidden) has moved if the area corresponding to the position of the beginner avatar acquired immediately before (see Figure 10) is different from the area corresponding to the position of the beginner avatar acquired immediately before that.

[0192] Here, the display control device 201 waits for the beginner avatar (hidden) to move (step S1606: No). Then, if the beginner avatar (hidden) moves (step S1606: Yes), the display control device 201 determines whether the training has ended or not (step S1607). The training ends, for example, in response to an end command from the HMD 202.

[0193] If the training is not yet complete (step S1607: No), the display control device 201 returns to step S1506 as shown in Figure 15. On the other hand, if the training is complete (step S1607: Yes), the display control device 201 terminates the series of processes according to this flowchart.

[0194] As a result, when training a predetermined task, the display control device 201 can use pre-recorded movements of an expert to provide subtle physical guidance to the user (beginner) without them noticing.

[0195] As described above, according to the display control device 201 of this embodiment, when displaying a virtual body part image representing a body part of a user (beginner) in a virtual space, the storage unit 910 can be used to determine whether or not to correct the position in which the virtual body part image is displayed based on the position of the user (beginner avatar) in the virtual space. The virtual space is, for example, a virtual space (e.g., virtual space 800) that is displayed on the HMD 202 worn by the user and reproduces an environment in which a predetermined task (e.g., task W1) is performed. The predetermined task is a task that is the target of training aimed at acquiring the physical skills of an expert. The storage unit 910 stores information (e.g., operation definition file F1) that identifies the correction application location in the virtual space. The correction application location is a place in which the position in which the virtual body part image is displayed is corrected according to the position corresponding to the expert's body part in the virtual space that has been recorded in advance.

[0196] This allows the display control device 201 to reduce unnaturalness when guiding the user's body in the virtual space. For example, a location where a predetermined task is performed in the virtual space can be registered as a location for correcting the position of displaying the virtual body part image. In this case, the display control device 201 can refrain from correcting the position of displaying the virtual body part image when the user is waiting or moving in the virtual space. Furthermore, once the user moves to a location where a predetermined task is performed in the virtual space, the display control device 201 can correct the position of displaying the virtual body part image using the movements of a pre-recorded expert.

[0197] Furthermore, according to the display control device 201, if it determines that it is necessary to correct the position in which the virtual body part image is displayed, it can correct the position in which the virtual body part image is displayed according to the position corresponding to the body part of the expert in the virtual space that has been recorded in advance. Then, according to the display control device 201, the virtual body part image can be displayed at the corrected position in the virtual space.

[0198] As a result, once the user moves to a location in the virtual space where they will perform a predetermined task, the display control device 201 can guide the user's body using pre-recorded movements of an expert. Therefore, the user can learn the movements of an expert while maintaining a sense of self-control, as they are only guided through predetermined actions while still feeling that they are moving on their own.

[0199] Furthermore, the display control device 201 can correct the position where the virtual body part image is displayed by integrating the positions corresponding to the skilled person's body parts and the positions corresponding to the user's body parts at the correction application location in the virtual space. Specifically, for example, the display control device 201 records motion data (e.g., skilled person avatar recording result R1) that represents the time change of the positions corresponding to the skilled person's body parts when the skilled person performs a predetermined task in the virtual space (e.g., virtual space 800). Then, the display control device 201 refers to the recorded motion data and corrects the position where the virtual body part image is displayed by integrating the positions corresponding to the skilled person's body parts and the positions corresponding to the user's body parts at regular time intervals.

[0200] As a result, the display control device 201 can perform body guidance in a way that is less noticeable to the user compared to using the movements of a skilled user's avatar directly for body guidance.

[0201] Furthermore, according to the display control device 201, if it is determined that no correction is made to the position in which the virtual body part image is displayed, the virtual body part image can be displayed at the position corresponding to the user's body part in the virtual space.

[0202] As a result, the display control device 201 can represent the user's own movements as an avatar AV while the user is waiting or moving in the virtual space, thus preventing the user from feeling any discomfort.

[0203] Furthermore, the display control device 201 can refer to the storage unit 910 to determine whether or not to correct the position in which the virtual body part image is displayed based on the user's position and orientation in the virtual space. The storage unit 910 stores information that identifies the location and orientation in the virtual space in which the position in which the virtual body part image is displayed is corrected according to the position corresponding to the body part of the expert.

[0204] This allows the display control device 201 to accurately determine the timing for initiating user body guidance by considering not only the user's position but also their orientation in the virtual space. For example, in the virtual space, a user may move to a certain position and then turn in a specific direction before performing a task. In such cases, the display control device 201 can initiate body guidance not immediately after the user moves to the position, but after the user has turned in a specific direction, making the movement of the fused avatar more natural.

[0205] Furthermore, the display control device 201 can correct the orientation in which the virtual body part image is displayed by integrating the orientation corresponding to the skilled worker's body part and the orientation corresponding to the user's body part at the correction application location in the virtual space. The display control device 201 can then display the virtual body part image in the corrected position and orientation in the virtual space. Specifically, for example, the display control device 201 records motion data representing the time change of the position and orientation corresponding to the skilled worker's body part when the skilled worker performs a predetermined task in the virtual space. The display control device 201 then refers to the recorded motion data and, at regular time intervals, corrects the position in which the virtual body part image is displayed by integrating the position corresponding to the skilled worker's body part and the position corresponding to the user's body part, and also corrects the orientation in which the virtual body part image is displayed by integrating the orientation corresponding to the skilled worker's body part and the orientation corresponding to the user's body part.

[0206] As a result, the display control device 201 can perform more precise body guidance by taking into account not only the position but also the orientation of the skilled person's body parts when using the skilled person's movements to guide the user's body.

[0207] Furthermore, the display control device 201 can identify the part of the user's body that is subject to correction for a predetermined task, and correct the position in which the virtual body image is displayed for the identified part to be corrected, according to the position corresponding to the part of the skilled worker.

[0208] As a result, the display control device 201 can position and orient parts that do not affect the acquisition of physical skills (parts other than those targeted for correction) to correspond to the parts of a beginner, making the movements of the fused avatar more natural.

[0209] Based on these considerations, the display control device 201 according to this embodiment allows for the user (beginner) to be guided through specific movements—specifically, movements that should be learned to acquire the physical skills of an expert—without the user noticing. This enables the user to acquire the movements of an expert while maintaining a high degree of self-control, thereby improving the learning effect of physical skills and promoting skill retention. For example, the display control device 201 can enable effective training using a virtual space even for tasks where it is difficult to construct an implementation environment in the real world, such as factory work (e.g., line work, inspection work, etc.) or construction site work (e.g., crane operation instruction work, work at height, etc.).

[0210] The display control method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This display control program is recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, DVD, or USB memory, and is executed when read from the recording medium by the computer. This display control program may also be distributed via a network such as the Internet.

[0211] Furthermore, the information processing device 101 (display control device 201) described in this embodiment can also be implemented using application-specific ICs such as standard cells and structured ASICs (Application Specific Integrated Circuits), or PLDs (Programmable Logic Devices) such as FPGAs.

[0212] With regard to the embodiments described above, the following additional information is disclosed.

[0213] (Note 1) A display control method for displaying a virtual part image representing a part of a user at a position corresponding to that part of the user in a virtual space, The system refers to a storage unit that stores information identifying a location within the virtual space, which corrects the position of the virtual body part image according to the position of the user in the virtual space, based on the user's position in the virtual space, and determines whether or not to correct the position of the virtual body part image based on the user's position in the virtual space. A display control method characterized by having a computer perform the processing.

[0214] (Note 2) If it is determined that the above correction should be performed, the position in which the virtual body part image is displayed will be corrected according to the position corresponding to the body part of the skilled worker. The virtual part image is displayed at the corrected position in the virtual space. The display control method according to Appendix 1, characterized in that the computer performs the processing.

[0215] (Appendix 3) The correction process described above is: The display control method according to Appendix 2, characterized in that the position for displaying the virtual body part image is corrected by integrating the position corresponding to the body part of the expert and the position corresponding to the body part of the user in the location within the virtual space.

[0216] (Note 4) The virtual space is a virtual space in which the environment for performing a predetermined task is reproduced. In the virtual space, motion data representing the time change of the position corresponding to the part of the skilled person when the skilled person performs the predetermined task is recorded. The computer performs the processing, The aforementioned correction process is: The display control method according to Appendix 3, characterized in that, by referring to the recorded operation data, the position in which the virtual body part image is displayed is corrected by integrating the position corresponding to the skilled person's body part and the position corresponding to the user's body part at regular time intervals.

[0217] (Note 5) If it is determined that the above correction will not be performed, the virtual body part image will be displayed at the position corresponding to the user's body part in the virtual space. A display control method according to any one of the appendices 2 to 4, characterized in that the processing is performed by the computer.

[0218] (Note 6) The storage unit stores information that identifies the location and orientation within the virtual space for correcting the position in which the virtual part image is displayed according to the position corresponding to the part of the skilled person, The process for making the determination is as follows: By referring to the storage unit, it is determined whether or not to correct the position where the virtual part image is displayed based on the user's position and orientation in the virtual space. A display control method as described in any one of the appendices 1 to 5, characterized by the above.

[0219] (Note 7) The process for performing the above correction is: By integrating the position corresponding to the skilled person's body part and the position corresponding to the user's body part at the location in the virtual space, the position for displaying the virtual body part image is corrected. By integrating the orientation corresponding to the skilled person's body part and the orientation corresponding to the user's body part at the location in the virtual space, the orientation in which the virtual body part image is displayed is corrected. The aforementioned display process is, In the virtual space, the virtual part image is displayed at the corrected position with the corrected orientation. The display control method described in Appendix 2, characterized by the features described herein.

[0220] (Note 8) The virtual space is a virtual space in which the environment for performing a predetermined task is reproduced. In the virtual space, motion data representing the time change in the position and orientation of the parts of the skilled person when the skilled person performs the predetermined task is recorded. The computer performs the processing, The aforementioned correction process is: By referring to the recorded motion data, the position in which the virtual body part image is displayed is corrected by integrating the position corresponding to the skilled person's body part and the position corresponding to the user's body part at regular time intervals, and the orientation in which the virtual body part image is displayed is corrected by integrating the orientation corresponding to the skilled person's body part and the orientation corresponding to the user's body part. The display control method described in Appendix 7, characterized by the features described herein.

[0221] (Note 9) The virtual space is a virtual space in which the environment for performing a predetermined task is reproduced. The aforementioned correction process is: Identify the part to be corrected corresponding to the predetermined operation, The display control method according to Appendix 2, characterized in that, with respect to the identified part of the user's body that is subject to correction, the position in which the virtual part image is displayed is corrected according to the position corresponding to the part of the expert.

[0222] (Note 10) The display control method according to any one of Notes 1 to 9, characterized in that the virtual space is displayed on an HMD (Head Mounted Display) worn by the user.

[0223] (Note 11) A display control method for displaying a virtual part image representing a part of a user at a position corresponding to that part of the user in a virtual space, The system refers to a storage unit that stores information identifying a location within the virtual space, which corrects the position of the virtual body part image according to the position of the user in the virtual space, based on the user's position in the virtual space, and determines whether or not to correct the position of the virtual body part image based on the user's position in the virtual space. A display control program characterized by having a computer perform the processing. [Explanation of symbols]

[0224] 101 Information Processing Device 102 users 103,202 HMD 104,601,800 virtual space 105, 106, 107 Arrows 110,910 storage section 200 Information Processing Systems 201 Display Control Device 210 Network 220 Expert Avatar Record Database 300 bus 301 CPU 302 memory 303 Disk Drive 304 disks 305 Communication I / F 306 Portable recording medium interface 307 Portable recording media 401, 402, 403, 404, 1301, 1302, 1303, 1304, 1305, 1311, 1312, 1313, 1314, 1321, 1322, 1323, 1324 objects 500 hierarchical structure 600 VR screens 602 Crane 610 Left screen 620 Right screen 801 Machinery 802 shelves 803 desk 900 Control Unit 901 Reception Department 902 Display Control Unit 903 Records Department 904 Judgment section 905 Correction Unit 1101,1102 area 1201,1202,1203 Operation definition information F1 Operation Definition File R1 Expert Avatar Record Results W1 work

Claims

1. A display control method for displaying a virtual body part image representing a body part of a user at a position corresponding to that body part in a virtual space, The system refers to a storage unit that stores information identifying a location within the virtual space, which corrects the position of the virtual body part image according to the position of the user in the virtual space, based on the user's position in the virtual space, and determines whether or not to correct the position of the virtual body part image based on the user's position in the virtual space. A display control method characterized by having a computer perform the processing.

2. If it is determined that the above correction should be performed, the position in which the virtual part image is displayed is corrected according to the position corresponding to the part of the skilled person. The virtual part image is displayed at the corrected position in the virtual space. The display control method according to claim 1, characterized in that the processing is performed by the computer.

3. The aforementioned correction process is: The display control method according to claim 2, characterized in that the position for displaying the virtual body part image is corrected by integrating the position corresponding to the body part of the expert and the position corresponding to the body part of the user at the location in the virtual space.

4. The aforementioned virtual space is a virtual space in which the environment for performing a predetermined task is reproduced. In the virtual space, motion data representing the time change of the position corresponding to the part of the skilled person when the skilled person performs the predetermined task is recorded. The computer performs the processing, The aforementioned correction process is: The display control method according to claim 3, characterized in that, by referring to the recorded operation data, the position for displaying the virtual body part image is corrected by integrating the position corresponding to the skilled person's body part and the position corresponding to the user's body part at regular time intervals.

5. If it is determined that the above correction will not be performed, the virtual body part image will be displayed at the position corresponding to the user's body part in the virtual space. The display control method according to claim 2, characterized in that the processing is performed by the computer.

6. The storage unit stores information that identifies the location and orientation within the virtual space for correcting the position in which the virtual part image is displayed according to the position corresponding to the part of the skilled person. The process for making the determination is as follows: By referring to the storage unit, it is determined whether or not to correct the position where the virtual part image is displayed based on the user's position and orientation in the virtual space. The display control method according to feature 1.

7. The display control method according to any one of claims 1 to 6, characterized in that the virtual space is displayed on an HMD (Head Mounted Display) worn by the user.

8. A display control method for displaying a virtual body part image representing a body part of a user at a position corresponding to that body part in a virtual space, The system refers to a storage unit that stores information identifying a location within the virtual space, which corrects the position of the virtual body part image according to the position of the user in the virtual space, based on the user's position in the virtual space, and determines whether or not to correct the position of the virtual body part image based on the user's position in the virtual space. A display control program characterized by having a computer perform the processing.

Images