Work assistance method, work assistance device, and program

Abstract

The present invention provides a work assistance method for assisting work carried out, in a virtual space (S) in which an object (O) is disposed, on the object (O) by a plurality of users including a target user. The work assistance method includes: acquiring first information including at least one of audio information, input information, and schedule information (S11); acquiring second information indicating an operation on the object (O) by the target user (S12); determining, on the basis of the first information, whether or not to reflect the operation by the target user on other users among the plurality of users (S13); generating, on the basis of the second information and the result of determining whether or not to reflect the operation by the target user on the other users, a video image (P) to be viewed by each respective other user (S24); and outputting the generated image (P) to a terminal of the other users (S25).

Description

[Technical Field]

[0001] The present disclosure relates to a work assistance method, a work assistance device, and a program. [Background technology]

[0002] In recent years, in the field of virtual reality and other technologies, a technique has been considered in which, when a user moves their body in the real world, the movement is reflected in an avatar or the like in a virtual space. With such a technique, a user can wear a device such as a head-mounted display and view the virtual space, while moving their body in the real world to touch objects present in the virtual space, thereby enjoying a highly realistic virtual space experience. For example, Patent Document 1 discloses a device capable of realizing various inputs (operations) to objects in a virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6535641 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned technology, multiple users who are in different locations and far from each other may share the same virtual space and work on objects in the shared virtual space. In this case, if an operation performed by one user on an object is uniformly reflected in the images of the other users, the other users may feel uncomfortable because the object has changed unintentionally.

[0005] Therefore, the present disclosure provides a task assistance method, a task assistance device, and a program that can appropriately reflect an operation performed by one user on an object in a virtual space to another user. [Means for solving the problem]

[0006] A work assistance method according to one embodiment of the present disclosure is a work assistance method that assists a plurality of users, including a target user, in work on one or more objects in a virtual space in which the one or more objects are arranged, and acquires first information including at least one of audio information based on utterances of one or more of the plurality of users, input information based on input from the one or more of the plurality of users, and schedule information based on a plan related to the work, acquires second information indicating operations of the target user on the one or more objects, determines based on the first information whether or not to reflect the operations by the target user on other users among the plurality of users, generates an image for each of the other users to view based on the result of the determination whether or not to reflect the operations by the target user on the other users and the second information, and outputs the generated image to a terminal of the other users.

[0007] A work assistance device according to one embodiment of the present disclosure is a work assistance device that assists a plurality of users, including a target user, in working on one or more objects in a virtual space in which the one or more objects are arranged, and includes: a first acquisition unit that acquires first information including at least one of audio information based on utterances of one or more of the plurality of users, input information indicating input from the one or more of the plurality of users, and schedule information indicating a plan for the work; a second acquisition unit that acquires second information indicating an operation on the one or more objects by the target user, a determination unit that determines, based on the first information, whether or not to reflect the operation of the target user on other users among the plurality of users; a generation unit that generates, for each of the other users, an image to be viewed by the other users, based on the result of the determination of whether or not to reflect the operation by the target user on the other users and the second information; and an output unit that outputs the generated image to terminals of the other users.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described task assistance method. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, it is possible to realize a work assistance method or the like that can appropriately reflect an operation performed by one user on an object in a virtual space to another user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a work support system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a flowchart showing the operation of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of details of step S13 shown in FIG. [Figure 5] FIG. 5 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination in step S25 shown in FIG. 4 is made. [Figure 6] FIG. 6 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination in step S27 shown in FIG. 4 is made. [Figure 7] FIG. 7 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination in step S28 shown in FIG. 4 is made. [Figure 8] FIG. 8 is a diagram illustrating schedule information according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A work assistance method according to one embodiment of the present disclosure is a work assistance method for assisting a plurality of users, including a target user, in work on one or more objects in a virtual space in which the one or more objects are arranged, the work assistance method comprising: acquiring first information including at least one of audio information based on utterances of one or more of the plurality of users, input information based on input from the one or more of the plurality of users, and schedule information based on a plan related to the work; acquiring second information indicating operations of the target user on the one or more objects; determining, based on the first information, whether or not to reflect the operations by the target user on other users among the plurality of users; generating, for each of the other users, an image to be viewed by that user based on the result of the determination of whether or not to reflect the operations by the target user on the other users and the second information; and outputting the generated image to a terminal of the other users.

[0012] As a result, based on the first information, it is determined whether or not the target user's operations on one or more objects should be reflected in the images of other users. In other words, the target user's operations are not uniformly reflected in the images of other users. Furthermore, since the first information includes any of voice information, input information, and schedule information, the determination can be made depending on the target user. Therefore, operations performed by the target user (a certain user) on objects in the virtual space can be appropriately reflected in other users.

[0013] Furthermore, for example, the first information may include at least the voice information, and the content of the utterances of the one or more users based on the voice information may be analyzed, and the determination may be made based on the analysis result of the content of the utterances.

[0014] This makes it possible to determine whether or not to reflect the target user's operation on the images of other users based on the content of the user's speech in the virtual space. For example, if the content of the speech indicates that the target user's operation should be reflected on other users, the operation can be reflected on the images of other users. Therefore, the operation performed by the target user on an object in the virtual space can be appropriately reflected on other users according to the content of the speech.

[0015] Furthermore, for example, the determination may involve determining, based on the first information, whether each time interval is in a collaborative work mode in which the multiple users work together, or in an individual work mode in which the multiple users work individually, and determining that the operation by the target user in a time interval determined to be the collaborative work mode will be reflected to the other users, and determining that the operation by the target user in a time interval determined to be the individual work mode will not be reflected to the other users.

[0016] This allows a determination to be made as to whether or not to reflect the target user's operations on the images of other users depending on the current operation mode, thereby allowing operations performed by the target user on objects in the virtual space to be appropriately reflected on other users depending on the operation mode.

[0017] Furthermore, for example, when it is determined that the collaborative mode is in effect, it may further determine whether the target user will give a presentation, and if the target user will give the presentation, it may determine that the operation by the target user will be reflected to the other users, and if the target user will not give the presentation, it may determine that the operation by the target user will not be reflected to the other users.

[0018] This allows a determination as to whether or not to reflect the target user's operations on the images of other users based on whether or not the target user is a presenter, and therefore allows operations performed by the target user on objects in the virtual space to be appropriately reflected on other users depending on whether or not the target user is a presenter.

[0019] Furthermore, for example, the first information may include at least the input information, and the input information may include information indicating whether the target user is a user who will give the presentation.

[0020] This makes it possible to easily determine whether or not the target user is a presenter simply by acquiring the input information.

[0021] Also, for example, when generating the image, if it is determined that the operation by the target user should be reflected on the other user, the operation by the target user may be reflected on the image of the other user, and if it is determined that the operation by the target user should not be reflected on the other user, the operation by the target user may not need to be reflected on the image of the other user.

[0022] As a result, only when it is determined that the operation of the target user is to be reflected to other users, the operation of the target user can be shared with other users.

[0023] Also, for example, when generating the image, if it is determined that the operation by the target user should be reflected on the other users, the operation by the target user may be reflected on the image of one or more specific users among the other users, but may not be reflected on the image of the one or more specific users among the other users.

[0024] This allows the target user's operations to be reflected only in the images of one or more specific users, rather than all other users. Therefore, the operations performed by the target user on an object in the virtual space can be reflected only in the more appropriate users among the other users. Furthermore, the amount of communication between the user's terminal and the information processing device can be reduced compared to when operations are reflected in the images of all users included in the other users.

[0025] Also, for example, the one or more specific users may be predetermined for each of the plurality of users.

[0026] This allows the operation of the target user to be reflected in the image of a predetermined user, so that the operation performed by the target user on an object in the virtual space can be reflected only in the image of the more appropriate user.

[0027] Furthermore, for example, the one or more specific users may be determined by an input from the target user during a period in which it is determined that the operation by the target user is to be reflected on the other users.

[0028] This allows the target user's operations to be reflected in the images of users selected by the target user. In other words, the target user's operations can be reflected in the images of one or more specific users based on the target user's intention. Therefore, the operations performed by the target user on objects in the virtual space can be reflected only in the most appropriate users.

[0029] Furthermore, for example, the one or more specific users may be determined based on at least one of information indicating the positions of the other users in the virtual space and information indicating the attributes of the other users.

[0030] This allows the system to determine the user to whom the target user's operation is reflected based on at least one of the positional relationships of each user in the virtual space and the attributes of other users. In other words, the system can determine the user to whom the target user's operation is reflected depending on the situation in the virtual space. Therefore, the operation performed by the target user on an object in the virtual space can be reflected only to the most appropriate user.

[0031] Furthermore, for example, the first information may include at least the schedule information, and the schedule information may include information indicating a time period in which the collaborative work mode is active and a time period in which the individual work mode is active.

[0032] This makes it possible to easily determine the current work mode simply by acquiring the schedule information.

[0033] Furthermore, for example, the operation on the one or more objects may include at least one of moving, rotating, enlarging, and reducing the one or more objects.

[0034] This allows at least one of the operations of movement, rotation, enlargement, and reduction performed by the target user on one or more objects in the virtual space to be reflected in the images of the other users.

[0035] A work assistance device according to an embodiment of the present disclosure is a work assistance device that assists a plurality of users, including a target user, in working on one or more objects in a virtual space in which the one or more objects are arranged, the work assistance device including: a first acquisition unit that acquires first information including at least one of voice information based on utterances of one or more users among the plurality of users, input information indicating input from the one or more users among the plurality of users, and schedule information indicating a plan for the work; a second acquisition unit that acquires second information indicating operations on the one or more objects by the target user; a determination unit that determines, based on the first information, whether to reflect the operation of the target user on other users among the plurality of users; a generation unit that generates, for each of the other users, an image to be viewed by the target user based on a result of the determination of whether to reflect the operation of the target user on the other users and the second information; and an output unit that outputs the generated image to a terminal of the other users. A program according to an embodiment of the present disclosure is a program for causing a computer to execute the work assistance method described above.

[0036] As a result, the same effects as those of the above-mentioned work assistance method can be achieved.

[0037] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0038] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0039] Note that the embodiments described below are all comprehensive or specific examples. The numerical values, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. For example, numerical values ​​are not expressions that express only strict meanings, but expressions that include a substantially equivalent range, for example, a difference of a few percent. Furthermore, among the components in the following embodiments, components that are not described in independent claims are described as optional components.

[0040] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales and the like do not necessarily match in each figure. Furthermore, in each figure, substantially the same configurations are assigned the same reference numerals, and duplicate explanations are omitted or simplified.

[0041] Furthermore, in this specification, numerical values ​​and numerical ranges are not expressions that express only the strict meaning, but also expressions that include a substantially equivalent range, for example, a difference of about a few percent (e.g., about 5%).

[0042] (Embodiment) The work support system according to this embodiment will be described below with reference to FIGS.

[0043] [1. Work support system configuration] First, the configuration of a work support system according to this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing a schematic configuration of a work support system 1 according to this embodiment.

[0044] As shown in Fig. 1, the work assistance system 1 includes a head-mounted display 10 incorporating an information processing device 20. Note that, although Fig. 1 illustrates only the head-mounted display 10 worn by user U1, the information processing device 20 is also incorporated into the head-mounted displays 10 worn by each of users U2 to U4.

[0045] 1 also shows an example in which four users (users U1 to U4) are present in the virtual space S. In the following, the head-mounted display 10 worn by user U1 will be described, but the other users U2 to U4 may also wear similar head-mounted displays 10.

[0046] The head-mounted display 10 is, for example, a glasses-type head-mounted display, has an information processing device 20 built in, and displays an image P acquired from the information processing device 20 to the user U1. In the example of FIG. 1, the head-mounted display 10 displays an image P showing avatars representing users U2 to U4 and an object O in a virtual space S to the user U1. The object O is a virtual object that exists in the virtual space S. In this embodiment, the object O is a car, and the work support system 1 is used, for example, in a design review meeting for the car. Note that the object O is not limited to a car, and may be any object in the virtual space S. Furthermore, the use of the work support system 1 is not particularly limited, and the work support system 1 may be used for any purpose other than a design review meeting.

[0047] The head-mounted display 10 may be realized as a device that executes a built-in program as the head-mounted display 10 alone, without relying on an external processing device such as a server (e.g., a cloud server) or an image processing device, i.e., a so-called standalone device, or may be realized as a device that is connected to an external processing device via a network, executes applications, and transmits and receives data.

[0048] The head mounted display 10 may be a transmissive type or a non-transmissive type. The head mounted display 10 is an example of a terminal.

[0049] Each of the multiple users U1 to U4 (hereinafter also referred to as user U1, etc.) can operate an object O in the virtual space S. The method by which user U1, etc. operates object O is not particularly limited. For example, user U1 may hold a controller (not shown) in his / her hand and operate object O by moving the controller, etc. Alternatively, user U1, etc. may operate object O by voice. In this case, the task support system 1 has a sound collection device (for example, a microphone), etc. Alternatively, user U1, etc. may operate object O by gestures, etc. In this case, the task support system 1 has a camera, etc. The controller, sound collection device, camera, etc. are communicably connected to the information processing device 20. The sound collection device and camera may be built into the head-mounted display 10.

[0050] The number of objects O present in the virtual space S is not particularly limited, and may be one or more.

[0051] The information processing device 20 is a device for assisting a plurality of users, including a target user, in working on an object in a virtual space S in which the object O is arranged. The information processing device 20 performs processing for generating an image P to be displayed on the head-mounted display 10, for example. For example, when the information processing device 20 acquires an operation on the object O by the user U1, the information processing device 20 generates an image P corresponding to the operation if a predetermined condition is satisfied, and outputs the image P to the other users U2 to U4. The information processing device 20 is an example of a work assistance device. Note that the target user may be, for example, one of the users U1, etc., who performed an operation on the object O. The following describes a case in which the target user is the user U1.

[0052] In such a task support system 1, when a user U1 performs an operation on an object O, there are cases where the operation performed by the user U1 on the object O should be reflected to other users (for example, at least one of users U2 to U4) and cases where it should not be reflected. The information processing device 20 according to the present embodiment performs processing to appropriately reflect the operation performed by the user U1 on the object O on other users.

[0053] The operation is an operation that changes the appearance of the object O, and in this embodiment, may include, for example, an operation for at least one of moving, rotating, enlarging, and reducing the object O within the virtual space S. The operation may also include, for example, an operation for changing the design of the object O. The operation may also be, for example, an operation for changing at least one of the color, shape, and texture of the object O. The operation may also be, for example, an operation for deleting or hiding the object O from the virtual space S, or an operation for making another object O appear in the virtual space S.

[0054] Note that "reflecting" refers to a process of applying a similar change in appearance to the object O viewed by other users as the change in appearance of the object O due to the target user's operation. For example, "reflecting" refers to making the change in appearance of the object O after the target user's operation, i.e., the object O viewed by the target user, the same as the object O viewed by other users. "Reflecting" refers to a process of sharing with other users the change in appearance of the object O before and after the target user's operation. For example, if the target user performs an operation to double the size of the object O, "reflecting" this includes doubling the size of the object O viewed by other users. Note that "reflecting" does not include matching the viewpoint (camera position) of other users with the viewpoint (camera position) of the target user. For example, "reflecting" the above-mentioned operation to enlarge does not include making the image of the object O of other users the same as the image viewed from the target user's camera position (e.g., switching to that image).

[0055] Furthermore, "reflecting" does not include making the same change to the viewpoint of other users as the change in the viewpoint of the target user. For example, if the target user shifts their viewpoint by 90 degrees when viewed from above (for example, when they change their viewpoint from looking at object O from the front to looking at it from the side), "reflecting" does not include shifting the viewpoint from which other users view object O by 90 degrees when viewed from above. Even if the target user changes the viewpoint from which they view object O, the viewpoint from which other users view object O does not change.

[0056] In this way, reflecting is a process of sharing with other users only the operations on object O (e.g., zooming in, etc.) performed by the target user and the operations on the avatar (e.g., moving the viewpoint, etc.).

[0057] Next, the configuration of the information processing device 20 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the information processing device 20 according to this embodiment.

[0058] As shown in FIG. 2, the information processing device 20 includes a first acquisition unit 21, a second acquisition unit 22, a determination unit 23, a generation unit 24, and an output unit 25. The information processing device 20 is a computer including a processor (microprocessor), a user interface, a communication interface, a memory, and the like. The user interface includes input / output devices such as a display, a keyboard, and a touch panel. The memory is a ROM, a RAM, or the like, and can store a control program (computer program) executed by the processor. The first acquisition unit 21, the second acquisition unit 22, the determination unit 23, the generation unit 24, and the output unit 25 are realized by the processor operating in accordance with the control program. The information processing device 20 may include one memory or multiple memories.

[0059] The first acquisition unit 21 acquires first information including at least one of audio information based on speech by a user U1, etc., input information based on input from a user U1, etc., and schedule information indicating a schedule indicating plans for work on an object O.

[0060] The first acquisition unit 21 acquires, for example, audio information based on the speech of one or more users, such as user U1. The first acquisition unit 21 is configured to include, for example, a sound collection device, and can directly acquire audio information based on the speech of each user, such as user U1, when the users, such as user U1, are within range of the speech of the users, such as user U1, in the same room. The first acquisition unit 21 may also acquire audio information indicating the speech from the sound collection device corresponding to each user, such as user U1.

[0061] Furthermore, the first acquisition unit 21 acquires input information based on input from one or more users, for example, user U1, etc. The first acquisition unit 21 is configured to include an acquisition device (for example, a communication circuit) that acquires input information received from user U1, etc. by an input device, for example, a mouse, a touch panel, a keyboard, etc., and when user U1, etc. are in the same room, etc., the first acquisition unit 21 can acquire input information from the input device corresponding to each user U1, etc.

[0062] The input information includes information indicating whether or not the operation of the target user on the object O is to be reflected in the video P of the other user. The input information may include, for example, information selected by the target user as to whether or not the operation of the target user is to be reflected in the video P of the other user. The input information may also include information indicating the current presenter. Information indicating the current presenter is an example of information indicating whether the target user is a presenter. The input information may also include information indicating the current work mode (for example, an individual work mode or a collaborative work mode, which will be described later).

[0063] Furthermore, the first acquisition unit 21 may be configured to include, for example, a communication circuit, and may be capable of communicating with at least one of the sound collection device and the input device.

[0064] The second acquisition unit 22 acquires second information indicating operations performed by the user U1 or the like on the object O. The second acquisition unit 22 acquires the second information from a controller, a sound collection device, a camera, or the like corresponding to each of the users U1 or the like. The second acquisition unit 22 is configured to include, for example, a communication circuit, and is capable of communicating with at least one of the controller, the sound collection device, and the camera. Alternatively, the second acquisition unit 22 may have a built-in controller, a sound collection device, a camera, or the like, and acquire the second information directly.

[0065] The determination unit 23 determines whether or not an operation on an object O by a target user (e.g., user U1) among users U1, etc. should be reflected in the object O in the video P of another user (e.g., at least one of users U2 to U4), based on the first information acquired by the first acquisition unit 21. The determination unit 23 may make this determination periodically, or may make this determination each time an operation on the object O of the target user is detected. Note that reflecting the operation on the object O of the target user in the object O in the video P of the other user is also referred to simply as reflecting on the other user, or reflecting on the video P of the other user.

[0066] The generation unit 24 generates the video P to be viewed by the users U1, etc., based on the determination result of the determination unit 23 and the second information. The generation unit 24 generates, for example, a video P corresponding to each user, such as user U1. When generating a video P to be viewed by user U2, for example, the generation unit 24 generates a video P in which the avatars of users U1, U3, and U4 are displayed and which shows the object O as seen from the viewpoint of user U2 in FIG. 1. In this way, each of the users U1, etc., views the video P in which the object O is viewed from a viewpoint corresponding to the position of their own avatar, for example.

[0067] The generation unit 24 may use an image including an object O stored in advance in the head mounted display 10 to generate an image P to be visually recognized by the user U1 or the like.

[0068] Furthermore, as will be described in detail later, when the determination unit 23 determines that the operation of the target user on the object O should be reflected on the other users, the generation unit 24 reflects the operation of the target user in the video P of the other users, and when the determination unit 23 determines that the operation of the target user should not be reflected on the other users, the generation unit 24 does not reflect the operation of the target user in the video P of the other users. For example, when the determination unit 23 determines that the operation of the target user on the object O should be reflected on the other users, the generation unit 24 generates, as the video P of the other users, a video P in which the operation of the target user is reflected. Furthermore, when the determination unit 23 determines that the operation of the target user should not be reflected on the other users, the generation unit 24 generates, as the video P of the other users, a video P in which the operation of the target user is not reflected.

[0069] The output unit 25 outputs the video P generated by the generation unit 24 to the head-mounted display 10 of the user U1 or the like. The output unit 25 is configured to include, for example, a communication circuit, and is capable of communicating with the head-mounted display 10.

[0070] [2. Operation of the work support system] Next, the operation of the task support system 1 configured as described above will be described with reference to Figs. 3 to 8. Fig. 3 is a flowchart showing the operation of the information processing device 20 according to this embodiment. The flowchart shown in Fig. 3 shows the operation when a user U1 or the like enters the virtual space S. The operation shown in Fig. 3 is performed, for example, by each of the information processing devices 20 included in the head-mounted displays 10 of the user U1 or the like. The information processing devices 20 of the head-mounted displays 10 of the user U1 or the like may perform the operation shown in Fig. 3 independently of each other, or may perform the operation shown in Fig. 3 in cooperation with each other.

[0071] As shown in FIG. 3, the first acquisition unit 21 acquires at least one of voice information of the user U1 or the like, input information from the user U1 or the like, and schedule information (S11). The first acquisition unit 21 acquires, for example, voice information based on the speech of the user U1 or the like who is present in the virtual space S. The voice information may include speech from at least one of the users U1 or the like. The first acquisition unit 21 also acquires, for example, input information. The input information may include input from at least one of the users U1 or the like. The first acquisition unit 21 also acquires schedule information from, for example, a management device (not shown) that manages a schedule for a design review meeting or the like using the virtual space S, or from the user U1 or the like. The schedule information is, for example, information in which a time period (time interval) is associated with information indicating whether an operation on an object O by a target user is to be reflected on other users. The schedule information may be, for example, information shown in FIG. 8, which will be described later. The schedule information may be stored in a storage unit (not shown) included in the head mounted display 10, and the first acquisition unit 21 may read out the schedule information from the storage unit.

[0072] The first acquisition unit 21 outputs the acquired first information to the determination unit 23.

[0073] Next, the second acquisition unit 22 acquires second information indicating operations on one or more objects O (S12). The second acquisition unit 22 acquires the second information for each user U1, etc. The second acquisition unit 22 outputs the acquired second information to the generation unit 24. Note that, below, an example will be described in which the second information includes information indicating operations on one or more objects O by the target user.

[0074] Next, the determination unit 23 determines, based on the first information, whether or not the operation by the target user on the object O in the video P viewed by the target user should be reflected on the object O in the video P viewed by other users (S13). Details of the determination method will be described later, but in step S13, it is determined whether or not to reflect the operation on other users, and if so, whether to reflect the operation on all of the other users or only on some of the other users.

[0075] Next, if the determination unit 23 determines that the operation should be reflected on the object O that the other users are looking at (Yes in S13), the generation unit 24 generates image data (video P) that reflects the operation on one or more objects O (S14). The generation unit 24 generates image data for, for example, each of the other users or for some of the other users by reflecting the operation on the object O of the target user.

[0076] For example, if the target user is user U1 and the other users are users U2 to U4, and user U1's operation is to rotate object O by a predetermined angle, generation unit 24 rotates object O in video P for each of users U2 to U4 by the predetermined angle. Furthermore, generation unit 24 generates image data for each of users U2 to U4 according to that user. Generation unit 24 outputs the generated image data to output unit 25.

[0077] Next, the output unit 25 outputs the image data (video P) generated by the generation unit 24 to the head-mounted displays 10 of the other users (for example, users U2 to U4) (S15). This allows the target user and the other users to share the change in the appearance of the object O.

[0078] Furthermore, if the determination unit 23 determines that the operations on one or more objects O of the target user are not to be reflected in the objects O that other users are looking at (No in S13), the generation unit 24 does not reflect the operations on one or more objects O of the target user in the video P of other users. The case where the determination unit 23 determines that the operations on one or more objects O of the target user are not to be reflected in the video P that the target user is looking at.

[0079] The operation shown in FIG. 3 is repeatedly executed at predetermined time intervals, for example.

[0080] Next, the processing of step S13 will be described with reference to Figures 4 to 8. Figure 4 is a flowchart showing an example of the details of step S13 shown in Figure 3. Step S13 is processing that is performed when a user U1 or the like has entered the virtual space S, for example, when all members who will hold a conference in the virtual space S have gathered.

[0081] 4, first, the determination unit 23 determines whether or not the current mode is the individual work mode based on the first information (S21). The individual work mode is a mode in which users U1 and the like are present in the virtual space S and each user performs work individually.

[0082] For example, when the first information includes at least schedule information (see FIG. 8 described later) including a time period during which the device operates in the personal work mode and a time period during which the device operates in the collaborative work mode, the determination unit 23 may determine that the device is in the personal work mode when the current time is within a time period during which the device operates in the personal work mode.

[0083] Furthermore, for example, when at least voice information is included in the first information, the determination unit 23 may analyze the content of the utterance of the user U1, etc. based on the voice information, and make the determination of step S21 based on the analysis result of the content of the utterance. The analysis of the content of the utterance may, for example, be detecting a predetermined keyword from the voice information. The keyword is a word for identifying whether the current mode is individual work mode or collaborative work mode. For example, when the determination unit 23 detects keywords such as "work individually," "consider individually," "do not reflect," or "break," the determination unit 23 determines that the current mode is individual work mode.

[0084] Furthermore, for example, when the determination unit 23 receives an input from any user indicating that the current work mode is the personal work mode, the determination unit 23 may determine that the current work mode is the personal work mode.

[0085] If the individual work mode is selected (Yes in S21), the determination unit 23 determines that the operation of each user is not reflected on the object O that other users are viewing (S22). This corresponds to determining No in step S13. If the result in step S21 is Yes, it can also be said that the publicness of the operation of each user on the object O is low (for example, lower than a predetermined reference value). Low publicness may mean, for example, that there is no publicness.

[0086] Note that even after the determination in step S22 is made, the information processing device 20 may continue to acquire the first information of the user U1 and the like.

[0087] Furthermore, if the first information indicates that the mode is not the individual work mode (No in S21), the determination unit 23 further determines whether the mode is the collaborative work mode (S23). The collaborative work mode is a mode in which the user U1 or the like is present in the virtual space S and works collaboratively on one or more objects O. For example, if the first information includes schedule information and the current time is within a time zone in which the mode is operational in the collaborative work mode, the determination unit 23 may determine that the mode is the collaborative work mode.

[0088] Furthermore, for example, when voice information is included in the first information, the determination unit 23 may analyze the content of the utterance of the user U1 or the like based on the voice information, and make the determination of step S23 based on the analysis result of the content of the utterance. The analysis of the content of the utterance may be, for example, detecting a predetermined keyword from the voice information. The keyword is a word for identifying whether or not the current mode is the collaborative work mode. For example, when the determination unit 23 detects keywords such as "start of meeting," "reflect," or "end of break," it determines that the current mode is the collaborative work mode.

[0089] Furthermore, the determination unit 23 may determine that the current work mode is the collaborative work mode when, for example, an input indicating that the current work mode is the collaborative work mode is acquired from any of the users.

[0090] If the mode is collaborative work mode (Yes in S23), the determination unit 23 proceeds to step S24, and if the mode is not collaborative work mode (No in S23), the determination unit 23 ends the process. Note that if the result in step S23 is Yes, it can also be said that the publicness of the operations of each user on the object O is high (for example, above a predetermined reference value). High publicness may mean, for example, that there is publicness. It can also be said that steps S21 and S23 determine whether or not there is publicness of the operations.

[0091] If the mode is the collaborative work mode, the determination unit 23 further determines whether the mode is the presentation mode (S24). The presentation mode is one mode included in the collaborative work mode, and is a mode in which one or more users give a presentation to other users during the collaborative work mode.

[0092] For example, when the first information includes schedule information including a time period during which the device operates in presentation mode, the determination unit 23 may determine that the device is in presentation mode when the current time is within the time period during which the device operates in presentation mode. In this case, the schedule information may include information for identifying the user (presenter) who will give the presentation.

[0093] Furthermore, for example, when voice information is included in the first information, the determination unit 23 may analyze the content of the utterance of the user U1 or the like based on the voice information, and make the determination of step S24 based on the analysis result of the content of the utterance. The analysis of the content of the utterance may be, for example, detecting a predetermined keyword from the voice information. The keyword is a word for determining whether or not the current mode is presentation mode. For example, when the determination unit 23 detects words such as "Mr. / Ms. XX will explain" or "I will explain," the determination unit 23 determines that the current mode is presentation mode.

[0094] Furthermore, the determination unit 23 may determine that the mode is the presentation mode when, for example, an input indicating that the mode is the presentation mode is acquired from any of the users.

[0095] If the mode is the presentation mode (Yes in S24), the determination unit 23 determines that only the operation of the user giving the presentation (presenter) is reflected in the object O that other users (for example, all other users) are looking at (S25).

[0096] If the mode is not the presentation mode (No in S24), the determination unit 23 determines whether a specific user is registered (S26). The specific user is a user among other users who reflects the operation of the target user. The specific user may be registered in advance for each user U1, etc., and stored in a memory (not shown) of the information processing device 20, or may be acquired from a user (e.g., the target user) when it is determined No in step S24.

[0097] If a specific user is registered (Yes in S26), the determination unit 23 determines that the operation of the user (target user) is reflected on the object O that the specific user corresponding to the user is looking at (S27). If the result is Yes in step S26, the operation of the target user on the object O is reflected on the video P of only some of the users other than the target user. If the result is No in S26, the determination unit 23 determines that the operation of each user is reflected on the object O that other users are looking at (S28). If the result is No in step S26, the operation of the target user on the object O is reflected uniformly on the video P of all the users other than the target user.

[0098] The determinations in steps S25, S27, and S28 correspond to a "Yes" determination in step S13.

[0099] As described above, when the judgment unit 23 judges that the collaborative mode is active, it further judges whether the target user will give a presentation or not, and if the target user will give a presentation, it judges that the target user's operations on one or more objects O will be reflected to other users, and if the target user will not give a presentation, it judges that the target user's operations on one or more objects O will not be reflected to other users.

[0100] The determinations in steps S21, S23, and S24 may be performed for each time interval based on the first information, for example. The time interval may be a time period included in schedule information or the like, or may be a preset time interval (for example, 5 minutes, 10 minutes, etc.). In steps S21 and S23, the determination unit 23 determines whether the mode is collaborative work mode or individual work mode, and determines that operations by the target user on one or more objects O in a time interval determined to be collaborative work mode should be reflected to other users, and determines that operations by the target user on one or more objects O in a time interval determined to be individual work mode should not be reflected to other users. Note that steps S21 and S23 may be performed by a single determination process.

[0101] 4, three modes, namely, the individual work mode, the collaborative work mode, and the presentation mode, are determined, but the number of modes is not limited to this, and the number of modes may be two, or may be four or more. When the number of modes is two, the two modes may be two of the individual work mode, the collaborative work mode, and the presentation mode.

[0102] Next, the video P generated when the determination is made in steps S25, S27, and S28 will be described with reference to Figs. 5 to 7. Fig. 5 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination is made in step S25 shown in Fig. 4. Figs. 5 to 7 show a case where six users, including the target user and first to fifth users, have entered the virtual space S. The first to fifth users are examples of other users.

[0103] As shown in FIG. 5, if the target user is a presenter, the operations of the target user on one or more objects O are reflected in the video P of each of the first to fifth users, whereas if the target user is not a presenter, the operations of the target user on one or more objects O are not reflected (not reflected) in the video P of each of the first to fifth users. In this way, only the operations of the presenter are reflected to other users, so that other users can view the video P according to the explanation of the presenter. Furthermore, since the operations of people who are not presenters are not reflected to other users, it is possible to prevent the video P that does not match the explanation of the presenter from being shared with other users. Note that the presenter is not limited to one person, and there may be two or more presenters.

[0104] FIG. 6 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination is made in step S27 shown in FIG. 4. FIG. 6 shows an example in which the first user and the second user are specific users, and the third user to the fifth user are not specific users. Note that the "operation of a user whose operation is reflected" shown in FIGS. 6 and 7 refers to the operation of a user whose determination in step S24 is No. Also, the "operation of a user whose operation is not reflected" shown in FIGS. 6 and 7 refers to the operation of a user whose determination in step S21 is Yes. In the case of the "operation of a user whose operation is not reflected," that is, when the target user is not a user whose operation is reflected, the operation of the target user on one or more objects O is not reflected (not reflected) in the video P of each of the first user to the fifth user.

[0105] As shown in FIG. 6, if the target user is a user to whom operations are reflected, the operations of the target user on one or more objects O are reflected in the images P of only the first and second users among the first to fifth users, and the operations of the target user on one or more objects O are not reflected (unreflected) in the images P of the third to fifth users. In this case, in step S14, the generation unit 24 generates image data in which the operations of the target user on one or more objects O are reflected only in a specific user among the other users. Note that the specific user does not include all of the other users.

[0106] In this way, the target user's operations on one or more objects O are reflected only in the specific user, so that the target user can share the video P only with desired users. The first user and the second user are examples of the one or more specific users.

[0107] The specific user may be determined in advance for each of the users U1 and the like, and may be stored in the memory of the information processing device 20.

[0108] The specific user may be determined based on an input from the target user during a period in which it is determined that the operation of the target user is to be reflected to other users. For example, the specific user may be acquired and determined based on an input from the target user during collaborative work mode.

[0109] The specific user may be automatically determined based on at least one of information indicating the positions of other users in the virtual space S and information indicating the attributes of the other users. The information indicating the positions of other users in the virtual space S may include, for example, information indicating the relative positional relationship between the target user or a predetermined object in the virtual space S, such as a table, and the other users in the virtual space S. For example, the information indicating the positions of the other users in the virtual space S may include information indicating whether the other users are within a predetermined distance from the target user or the predetermined object. The determination unit 23 may, for example, determine other users who are within a predetermined distance from the target user or the predetermined object as specific users. The information indicating the attributes of the other users may include, for example, information indicating at least one of the user's department, position, gender, age, role in a meeting, etc. The determination unit 23 may, for example, determine other users who match a list of user attributes reflecting the target user's operation as specific users corresponding to the target user based on the list. The information on user attributes may be acquired from each user, for example, when the user enters the virtual space S.

[0110] FIG. 7 is a diagram showing, for each user, whether or not the operation of the target user is reflected when the determination in step S28 shown in FIG. 4 is made.

[0111] As shown in FIG. 7, if the target user is a user whose operations are reflected, the operations of the target user on one or more objects O are reflected in the video P of each of the first to fifth users. For example, when any user, such as user U1, performs an operation on one or more objects O, the operation is reflected in each of the other users. In this way, the operations of the target user on one or more objects O are reflected in all of the other users, so that the target user can share the video P with each of the users in the virtual space S.

[0112] FIG. 8 is a diagram for explaining schedule information according to the present embodiment.

[0113] As shown in FIG. 8, the schedule information is, for example, information in which time is associated with each mode. The schedule information can also be said to include information indicating time periods in which the collaborative work mode is active and time periods in which the individual work mode is active. The schedule information also includes information regarding time periods in which the presentation mode is active and the presenter during the collaborative work mode. For example, within the collaborative work mode starting at 10:00, a presentation mode is performed in which person C is the presenter. Person C is an example of a target user.

[0114] In this case, for example, in the collaborative work mode starting at 10 o'clock, the operations of the target user are reflected to other users according to the determination in step S27 or S28 shown in Fig. 4, and when it is time for C to become the presenter, the determination in step S25 is made and the operations of only C are reflected to other users. In other words, when the work mode is switched from the collaborative work mode to the presentation mode in the collaborative work mode, the user (for example, the target user) who can reflect operations on one or more objects O to other users is switched.

[0115] In this way, the users who can reflect their operations on other users may change depending on, for example, the mode at that time, etc. For example, the schedule information shown in Fig. 8 is acquired in step S11 shown in Fig. 3.

[0116] (Other embodiments) While the work assistance method according to one or more aspects has been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and forms constructed by combining components of different embodiments may also be included in the present disclosure.

[0117] For example, in each of the above embodiments, there is no particular limitation on the communication method between the head-mounted display 10 and the information processing device 20. For example, the head-mounted display 10 and the information processing device 20 communicate with each other via wireless communication, but wired communication may also be used. Furthermore, there is no particular limitation on the communication standard used for wireless communication or wired communication, and any communication standard may be used.

[0118] Furthermore, in the above embodiment, an example has been described in which the object O is an automobile, but the object O may be, for example, a vehicle other than an automobile, such as a train, a display device, a lighting device, a home appliance such as a smartphone, an air vehicle such as a drone, clothing, furniture, a whiteboard, a sticky note, or food. An operation on the object O may be an operation for realizing a function of the object O. For example, if the object O is a display device, the operation on the object O may be an operation for displaying a video P on the display device. For example, if the object O is a sticky note, the operation on the object O may be an operation for writing text on the sticky note. An operation on the object O may be an operation for changing the appearance of at least a part of the virtual space S.

[0119] In the above embodiment, the determination unit 23 determines a work mode such as a personal work mode in step S13. However, the determination unit 23 is not limited to determining a work mode. For example, the determination unit 23 may make the determination in step S13 from the first information. For example, if the voice information includes information indicating a specific user, the determination unit 23 may make the determination in step S27 directly based on the voice information.

[0120] Furthermore, when generating an image P that reflects an operation of a target user on one or more objects O, the generation unit 24 in the above embodiment may superimpose information indicating the target user on the image P. That is, the generation unit 24 may display which of the users U1, etc., the operation of which is reflected in the image P. Furthermore, when the determination unit 23 determines the current working mode, the generation unit 24 in the above embodiment may superimpose information indicating the current working mode on the image P to be generated.

[0121] Furthermore, the information processing device 20 corresponding to the target user in the above embodiment may be configured to be able to communicate with the information processing device 20 corresponding to another user. The information processing device 20 corresponding to the target user may output information acquired in at least one of steps S11 and S12 to the information processing device 20 corresponding to the other user.

[0122] Furthermore, the object O in the above embodiment is, for example, a three-dimensional object, but may also be a two-dimensional object.

[0123] Furthermore, although the example in which the target user in the above embodiment is one user out of a plurality of users has been described, the target user may be two or more users out of the plurality of users.

[0124] Furthermore, the image P in the above embodiment is, for example, a moving image, but may also be a still image. Furthermore, the image P may be, for example, a color image or a black and white image.

[0125] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0126] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0127] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0128] Furthermore, the information processing device 20 according to the above embodiment may be realized as a single device or multiple devices. When the information processing device 20 is realized by multiple devices, the components of the information processing device 20 may be distributed among the multiple devices in any manner. At least one mechanical component of the functional components of the information processing device 20 may be realized by, for example, a cloud server. The information processing device 20 in this specification also includes a configuration in which the functions of the information processing device 20 are realized by a head-mounted display 10 and a cloud server. In this case, each head-mounted display 10 of a user U1 or the like is communicatively connected to the cloud server. For example, a component requiring a large amount of processing, such as the generation unit 24, may be realized by a cloud server or the like. When the information processing device 20 is realized by multiple devices, the communication method between the multiple devices is not particularly limited and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0129] Furthermore, if the information processing device 20 according to the above embodiment has a configuration capable of acquiring position information possessed by the head-mounted display 10 (for example, has a GPS (Global Positioning System) sensor), the information processing device 20 may generate an image P according to the position of a user U1, etc.

[0130] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, an integrated circuit. These components may be integrated individually on a single chip, or some or all of them may be integrated on a single chip. While LSI is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; dedicated circuits or general-purpose processors may also be used. Field-programmable gate arrays (FPGAs), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the reconfiguration of the connections or settings of circuit cells within an LSI, may also be used. Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or other derivative technologies, that technology may naturally be used to integrate the components.

[0131] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system comprising a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0132] Another aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the task assistance method shown in FIG. 3 or FIG.

[0133] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes. [Industrial Applicability]

[0134] The present disclosure is useful for a server device or the like that supports work in a virtual space by multiple users. [Explanation of symbols]

[0135] 1. Work support system 10 Head-mounted display (terminal) 20 Information processing device (work support device) 21 First acquisition part 22 Second acquisition part 23 Judgment section 24 Generation part 25 Output section O Object P video S Virtual Space U1, U2, U3, U4 users

Claims

1. 1. A work assistance method executed by a work assistance device that assists a plurality of users, including a target user, in work on one or more objects in a virtual space in which the one or more objects are arranged, the method comprising: acquiring first information including at least one of voice information based on speech of one or more users among the plurality of users, input information based on input from the one or more users among the plurality of users, and schedule information based on a plan related to the work; After acquiring the first information, acquire second information indicating an operation of the target user on the one or more objects; determining whether or not to reflect the operation by the target user on other users among the plurality of users based on the first information; generating an image of each of the other users that is visually recognized by the target user based on a result of a determination as to whether or not the operation by the target user should be reflected on the other users and the second information; outputting the generated video to the terminal of the other user; In the determination, determining, for each time interval based on the first information, whether the mode is a collaborative work mode in which the plurality of users work together or an individual work mode in which the plurality of users work individually; Determine that the operation by the target user in the time period determined to be the collaborative work mode is to be reflected to the other users, and determine that the operation by the target user in the time period determined to be the individual work mode is not to be reflected to the other users. Work support method.

2. the first information includes at least the audio information; Analyzing the speech content of the one or more users based on the voice information, and making the determination based on the analysis result of the speech content. The work support method according to claim 1 .

3. If the collaborative work mode is determined, it is further determined whether the target user will give a presentation; When the target user gives the presentation, it is determined that the operation by the target user is reflected to the other users, and when the target user does not give the presentation, it is determined that the operation by the target user is not reflected to the other users. The work support method according to claim 1 or 2.

4. the first information includes at least the input information, The input information includes information indicating whether the target user is a user who will give the presentation. The work support method according to claim 3 .

5. In generating the image, if it is determined that the operation by the target user is to be reflected on the other user, the operation by the target user is reflected on the image of the other user, and if it is determined that the operation by the target user is not to be reflected on the other user, the operation by the target user is not reflected on the image of the other user. The work support method according to any one of claims 1 to 4.

6. In generating the image, if it is determined that the operation by the target user is to be reflected on the other users, the operation by the target user is reflected on the image of one or more specific users among the other users, and is not reflected on the image of the one or more specific users among the other users. The work support method according to any one of claims 1 to 4.

7. The one or more specific users are predetermined for each of the plurality of users. The work support method according to claim 6.

8. The one or more specific users are determined by an input from the target user during a period in which it is determined that the operation by the target user is reflected on the other users. The work support method according to claim 6.

9. The one or more specific users are determined based on at least one of information indicating the positions of the other users in the virtual space and information indicating the attributes of the other users. The work support method according to claim 6.

10. the first information includes at least the schedule information; The schedule information includes information indicating a time period during which the collaborative work mode is active and a time period during which the individual work mode is active. The work support method according to any one of claims 1 to 3.

11. The manipulation of the one or more objects includes at least one of moving, rotating, enlarging, and reducing the one or more objects. The work support method according to any one of claims 1 to 10.

12. A work assistance device that assists a plurality of users, including a target user, in work on one or more objects in a virtual space in which the one or more objects are arranged, the work assistance device comprising: a first acquisition unit that acquires first information including at least one of voice information based on speech of one or more users among the plurality of users, input information indicating input from the one or more users among the plurality of users, and schedule information indicating a plan related to the work; a second acquisition unit that acquires, after acquiring the first information, second information indicating an operation performed by the target user on the one or more objects; a determination unit that determines whether or not the operation of the target user is to be reflected on other users among the plurality of users based on the first information; a generation unit that generates an image for each of the other users to be viewed by the other users based on a result of a determination as to whether or not the operation by the target user should be reflected on the other users and the second information; an output unit that outputs the generated video to a terminal of the other user; The determination unit determining, for each time interval based on the first information, whether the mode is a collaborative work mode in which the plurality of users work together or an individual work mode in which the plurality of users work individually; The operation by the target user in the time period determined to be the collaborative work mode is reflected to the other users, and the previous operation in the time period determined to be the individual work mode is reflected to the other users. It is determined that the operation by the target user is not reflected to the other users. Work support equipment.

13. A program for causing a computer to execute the work support method according to any one of claims 1 to 11.

Images