Information processing device, information processing method, and program

The information processing device addresses the issue of unpredictable virtual object appearance in XR by dynamically adjusting the drawing range based on distance and orientation, ensuring smooth observation and reducing surprise.

JP2024158242APending Publication Date: 2024-11-08CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023073271
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing XR technologies struggle with the unpredictable appearance of virtual objects that move unrelated to the user's intention, causing surprise or interruption in the experience due to lack of appropriate observation methods when virtual objects and users are close.

Method used

An information processing device that dynamically adjusts the drawing range of virtual objects based on the distance, position, and orientation relative to the user, ensuring appropriate observation by limiting the displayed range when objects are close to prevent interruption.

Benefits of technology

Enhances the user's ability to observe virtual objects effectively by reducing surprise and maintaining a seamless XR experience by adjusting the displayed range of virtual objects based on proximity and orientation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158242000001_ABST
    Figure 2024158242000001_ABST
Patent Text Reader

Abstract

To provide a technology for allowing a user to more appropriately observe a virtual object in an XR experience.SOLUTION: An information processing device for generating an image of a space with a virtual body arranged, the image being displayed on a display device to be attached to a user's head, includes: first acquisition means for acquiring a position at which a first virtual object whose position and posture dynamically change is arranged; second acquisition means for acquiring a position of the user in the space; designation means for designating a first range being a range to be drawn in the space in the first virtual object on the basis of the user's position in the space and the position of the first virtual object; and generation means for generating an image of the space in which the first range in the first virtual object is drawn.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a participant's observation of a virtual object in an XR experience. [Background technology]

[0002] In recent years, the use of XR (Extensible Reality) has become a trend, combining virtual and real worlds in real time to create new experiences. In XR technology, a user experiences a mixed reality space while viewing an image using a head mounted display (HMD).

[0003] In XR experiences, in order to realize an experience in which the user feels as if they are with other users in remote locations, virtual objects (avatars) that move following the physical movements of other users in remote locations may be displayed. Because the user cannot predict the movements of other users, an avatar may suddenly appear in the HMD (in front of the user's eyes) worn by the user. If such a virtual object that moves unrelated to the user's will appears from an unexpected direction during an XR experience, the user may be surprised by its appearance and the experience may be disrupted.

[0004] Therefore, Patent Documents 1 and 2 disclose a technique for increasing the transparency of a virtual object and a technique for hiding the virtual object when it is determined that the positions of the virtual object and the display device worn by the user are close to each other based on the position and orientation of the virtual object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-2390 [Patent Document 2] Patent Publication No. 2021-135776 Summary of the Invention [Problem to be solved by the invention]

[0006] With the above techniques, when the virtual object and the user are close to each other, the observation may be made through a transparent virtual object or the virtual object may be hidden, which may not be ideal for observation.

[0007] Therefore, an object of the present invention is to provide a technique that enables a user to more appropriately observe a virtual object during an XR experience. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: An information processing device that generates an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, a first acquisition means for acquiring a position for placing a first virtual object whose position and orientation change dynamically; A second acquisition means for acquiring a position of the user in the space; a designation means for designating a first range, which is a range of the first virtual object to be rendered in the space, based on a position of the user and a position of the first virtual object in the space; a generating means for generating an image of the space in which the first range of the first virtual object is rendered; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: An information processing device that generates an image of a space in which an avatar whose position and posture change dynamically can be arranged, the image being displayed on a display device worn on a user's head, the information processing device comprising: a first generating means for generating a first virtual object indicating a position at which the avatar gazes in the space; a second generating means for generating an image of the space in which the first virtual object is rendered; The information processing device is characterized by having:

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: 1. An information processing method for generating an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, the method comprising: a first acquisition step of acquiring a position for placing a first virtual object whose position and orientation change dynamically; a second acquisition step of acquiring a position of the user in the space; a designation step of designating a first range, which is a range of the first virtual object to be rendered in the space, based on a position of the user and a position of the first virtual object in the space; a generating step of generating an image of the space in which the first range of the first virtual object is rendered; The information processing method is characterized by having the following features.

[0011] One aspect of the present invention is a method for producing a composition comprising the steps of: 1. An information processing method for generating an image to be displayed on a display device worn on a user's head, the image being an image of a space in which an avatar whose position and posture change dynamically can be placed, comprising: a first generation step of generating a first virtual object indicating a position at which the avatar gazes in the space; a second generation step of generating an image of the space in which the first virtual object is rendered; The information processing method is characterized by having the following features. Effect of the Invention

[0012] According to the present invention, a user can more appropriately observe a virtual object during an XR experience. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an XR experience according to a first embodiment. [Diagram 2] 1 is a configuration diagram of a system according to a first embodiment. [Diagram 3] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 4]11 is a flowchart of a process for changing a drawing range according to the first embodiment. [Diagram 5] 5A to 5C are diagrams illustrating a mixed reality space when the distance is large according to the first embodiment. [Figure 6] 5A to 5C are diagrams illustrating a mixed reality space when the distance is small according to the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating a mixed reality space according to the second embodiment. [Figure 8] FIG. 11 is a configuration diagram of a system according to a second embodiment. [Figure 9] 13 is a flowchart of a process for changing a drawing range according to the second embodiment. [Figure 10] FIG. 11 is a configuration diagram of a system according to a third embodiment. [Figure 11] 13 is a flowchart of a virtual object rendering process according to the third embodiment. [Figure 12] FIG. 11 is a diagram showing a virtual object according to the third embodiment. [Figure 13] FIG. 11 is a configuration diagram of a system according to a fourth embodiment. [Figure 14] 13 is a flowchart of a drawing process of a drawing range according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0015] There are cases where a user wants to closely observe a virtual object that moves unrelated to the user's intention. For example, it is conceivable that training in assembly work is conducted using a device of a virtual object. In training in assembly work, for example, the user observes the instructor's body movements and the way tools are used while the instructor is working. In this case, if training is conducted using an XR experience, since the device and avatar are virtual objects, the user can observe the work in a place where the device and person (person) are not supposed to exist. In this way, by realizing an "experience that would not be possible in reality," the learning efficiency of the training is improved.

[0016] <Embodiment 1> In the first embodiment, a user (hereinafter referred to as an "observing user") observes an avatar. In this manner, an example is shown in which the observing user trains for assembly work. The avatar moves in accordance with (follows) the movement of a user (hereinafter referred to as a "remote user") who is located at a distance from the observing user. FIGS. 1A and 1B show the state of the XR experience according to the first embodiment.

[0017] 1A shows the state of the mixed reality space at the site LA. A remote user 100 wears an HMD 101 on his head and performs assembly work while looking at a virtual car object 102. At this time, a plurality of reflective markers 103 are attached to the whole body of the remote user 100 and the HMD 101. A motion sensor 104 captures the plurality of reflective markers 103 to obtain the position and orientation (position and orientation) of the whole body of the remote user 100.

[0018] In the first embodiment, the motion sensor 104 is described as an optical motion sensor, but the present invention is not limited thereto. Instead of the method using the motion sensor 104, any method capable of acquiring the position and orientation of the whole body (each part) of the remote user 100 may be used, for example, skeletal estimation based on image analysis or a magnetic sensor. Furthermore, the position and orientation of the whole body of the remote user 100 may be acquired based on pressure fluctuations acquired from a suit equipped with a pressure sensor. After the position and orientation of a part of the hands and head are acquired, the position and orientation of the whole body (each part) of the remote user 100 may be acquired from the acquired information by prediction using machine learning or the like.

[0019] 1B shows the state of the XR experience at base LB. The observing user 110 wears the HMD 210 and observes the car virtual object 102 and the avatar 112. The avatar 112 moves according to the position and orientation (position and orientation of the remote user 100) acquired by the motion sensor 104 shown in FIG. 1A. This allows the observing user 110 to feel as if he or she is together with the remote user 100 in the mixed reality space.

[0020] 2 is a block diagram showing an example of the configuration of a system 1 for performing an XR experience by an observing user 110. The system 1 includes an information processing device 200, an HMD 210, an information processing device 220, and a motion sensor 221.

[0021] In the system 1, the information processing device 200 is connected to the HMD 210 and the information processing device 220. Furthermore, the information processing device 220 is connected to a motion sensor 221. The connection between each of the components may be realized by either a wired or wireless connection. Furthermore, the connection between each of the components may be realized by a combination of a wired and wireless connection.

[0022] First, the information processing device 200 will be described. The information processing device 200 is, for example, a computer. The information processing device 200 includes an image acquisition unit 201, a position and orientation measurement unit 202, a static The image forming apparatus includes a data storage unit 203 , a dynamic data storage unit 204 , a position and orientation acquisition unit 205 , a designation unit 206 , a virtual image generation unit 207 , and a synthesis unit 208 .

[0023] The image acquisition unit 201 acquires an image (hereinafter referred to as a “real image”) of a real space captured by an imaging unit 211 of the HMD 210. The image acquisition unit 201 outputs the acquired real image to the position and orientation measurement unit 202 and the synthesis unit 208.

[0024] The position and orientation measuring unit 202 performs image processing using the real image acquired by the image acquiring unit 201 to extract characteristic information such as points and lines in the real image. The position and orientation measuring unit 202 measures the position and orientation of the HMD 210 (observing user) in real space based on the characteristic information. At this time, the position and orientation measuring unit 202 acquires the position and orientation of the HMD 210 (observing user) in mixed reality space (virtual space) from the position and orientation of the HMD 210 (observing user) in real space. The position and orientation measuring unit 202 outputs information on the position and orientation of the HMD 210 in mixed reality space (virtual space) to the designation unit 206 and the virtual image generating unit 207. In the first embodiment, the measurement of the position and orientation using image processing will be described, but other methods may be used. The position and orientation may be measured using infrared light. The position and orientation may be measured using an ultrasonic sensor, a magnetic sensor, or a depth sensor.

[0025] The static data storage unit 203 stores static data. The static data includes data related to a stationary virtual object existing in a virtual space (shape information, structure information, color information, position and orientation information), data related to a light source that irradiates the virtual space, etc. The static data storage unit 203 outputs the static data to the virtual image generation unit 207.

[0026] The dynamic data storage unit 204 stores dynamic data. The dynamic data is data (shape information, structure information, and color information) of a virtual object whose position and orientation changes dynamically during the XR experience (hereinafter, referred to as a "dynamic virtual object"). The dynamic virtual object is, for example, an avatar of a remote user. The dynamic data storage unit 204 outputs the dynamic data to the designation unit 206.

[0027] The position and orientation acquisition unit 205 acquires the position and orientation of a dynamic virtual object when the dynamic virtual object is placed in a mixed reality space (virtual space). The position and orientation acquisition unit 205 acquires the position and orientation of the dynamic virtual object in the mixed reality space from, for example, the position and orientation of the whole body of the remote user acquired from the data transmission unit 222. The position and orientation acquisition unit 205 outputs information on the position and orientation of the dynamic virtual object to the designation unit 206.

[0028] The designation unit 206 designates a range of the dynamic virtual object to be drawn (hereinafter referred to as a "drawing range") based on the position and orientation of the HMD 210 and the position and orientation of the dynamic virtual object. For example, the designation unit 206 designates the drawing range depending on whether or not the distance between the observing user (HMD 210) and the dynamic virtual object in the mixed reality space (virtual space) is equal to or greater than a threshold. The designation unit 206 outputs data related to the drawing range to the virtual image generation unit 207. The drawing range is, for example, a range corresponding to a body part of the avatar. The drawing range is designated from, for example, the upper body, the lower body, the fingers, the hands, and the feet.

[0029] The virtual image generating unit 207 generates a virtual space using the static virtual object acquired from the static data holding unit 203 and the drawing range specified by the designation unit 206. The virtual image generating unit 207 does not generate a range (part) other than the drawing range of the dynamic virtual object in the virtual space. Furthermore, the virtual image generating unit 207 generates an image (hereinafter referred to as a "virtual image") that represents the virtual space seen from the position and orientation measured by the position and orientation measuring unit 202. The virtual image generating unit 207 outputs the generated virtual image to the synthesis unit 208. Note that the technology for generating an image that represents the virtual space seen from a predetermined position is a known technology, and therefore a detailed description of this technology will be omitted.

[0030] The synthesis unit 208 superimposes (synthesizes) the virtual image generated by the virtual image generation unit 207 onto the real image acquired by the image acquisition unit 201. In this way, the synthesis unit 208 generates an image of a mixed reality space in which the real space and the virtual space are merged (hereinafter referred to as a "mixed reality image"). The synthesis unit 208 outputs the generated mixed reality image to the display unit 212 of the HMD 210. Note that the synthesis unit 208 may not generate a mixed reality image, and instead the virtual image may be output to the display unit 212.

[0031] In the first embodiment, a case will be described in which the MR experience is realized by displaying a mixed reality image on the HMD 210, but a VR experience may also be realized. For example, a virtual image generated by the virtual image generating unit 207 may be output to the display unit 212 of the HMD 210, thereby realizing a VR experience for the observing user.

[0032] 3 is a diagram showing a hardware configuration of the information processing device 200. The information processing device 200 includes a CPU 301, a storage device 302, a ROM 303, a RAM 304, an input I / F 305, and an output I / F 306.

[0033] The CPU 301 performs overall control of each device via the bus 300. The CPU 301 reads and executes programs and the like stored in the ROM 303 and the RAM 304. The storage device 302 is a storage unit that stores various programs and data.

[0034] The ROM 303 stores an operating system (OS), device drivers, and a boot program. The RAM 304 temporarily stores programs and data loaded from the storage device 302 and the ROM 303. The RAM 304 also has a work area for the CPU 301 to appropriately execute each process.

[0035] The input I / F 305 acquires an input signal in a format that can be processed by the information processing device 200 from an external device. The output I / F 306 outputs an output signal in a format that can be processed by an external device including the HMD 210 to the external device.

[0036] Next, the HMD 210 will be described. The HMD 210 is a display device including a liquid crystal screen and the like. The HMD 210 is provided with two liquid crystal screens, one for the right eye and one for the left eye. The two liquid crystal screens are attached so as to be positioned in front of the right eye and the left eye of an observing user wearing the HMD 210 on his / her head. In addition, images having a parallax with respect to each other are displayed in a stereo format on the left and right liquid crystal screens.

[0037] In the first embodiment, the HMD 210 is a video see-through type HMD that displays a mixed reality image. However, the HMD 210 may be an optical see-through type HMD. In this case, the HMD 210 displays a virtual image superimposed on a display medium that allows observation through the real space. This allows the observing user to feel (perceive) a mixed reality space in which the real space and the virtual space are merged.

[0038] The HMD 210 has an imaging unit 211 and a display unit 212. The imaging unit 211 captures an image of a real space located in front of the HMD 210 to acquire a real image. The imaging unit 211 outputs the real image to the information processing device 200. The display unit 212 displays a mixed reality image generated by the information processing device 200.

[0039] Finally, the information processing device 220 will be described. The information processing device 220 is a device owned by a remote user. The information processing device 220 is connected to a motion sensor 221.

[0040] The data transmission unit 222 converts the positions and orientations of the reflective markers acquired by the motion sensor 221 into the positions and orientations of each part of the remote user (positions and orientations in the world coordinate system). The data transmission unit 222 outputs the information on the positions and orientations of each part of the remote user to the position and orientation acquisition unit 205.

[0041] The motion sensor 221 recognizes a plurality of reflective markers attached to the body of the remote user, calculates the position and orientation of each reflective marker, and outputs information on the position and orientation of each reflective marker to the data transmission unit 222.

[0042] In the first embodiment, the information processing device 220 only has a role of transmitting information on the position and orientation of the whole body of the remote user. However, the information processing device 220 may display a mixed reality image similarly to the information processing device 200, in addition to transmitting information on the position and orientation of the whole body of the remote user. For example, the information processing device 220 may be connected to an HMD (not shown) and have the same configuration as each of the configurations of the information processing device 200.

[0043] FIG. 4 is an example of a flowchart showing "processing for changing the rendering range based on the distance between an avatar, which is a dynamic virtual object, and an observing user."

[0044] In step S400, the position and orientation measuring unit 202 measures the position and orientation of the HMD 210 in the mixed reality space (virtual space) based on the real image acquired from the image acquiring unit 201. Note that since the observing user wears the HMD 210 on his / her head, the position and orientation of the HMD 210 is the position and orientation of the observing user's head.

[0045] In step S401, the position and orientation acquisition unit 205 acquires the position and orientation of each part of the remote user from the data transmission unit 222 of the information processing device 220. Furthermore, the position and orientation acquisition unit 205 determines the position and orientation of the avatar based on the acquired position and orientation of each part.

[0046] In the first embodiment, the position and orientation acquisition unit 205 determines the position and orientation of the avatar in the mixed reality space (virtual space) based on the position and orientation of the HMD 101 worn by the remote user. However, the position and orientation acquisition unit 205 may determine the position and orientation of the avatar based on the position and orientation of any part of the remote user, or may determine the position and orientation of the avatar from the median value of the acquired positions and orientations of each part.

[0047] In step S402, the designation unit 206 calculates the distance between the HMD 210 (=observing user) and the avatar in the mixed reality space (virtual space) based on the position (position and orientation) of the HMD 210 and the position (position and orientation) of the avatar. If it is determined that the distance between the HMD 210 and the avatar is equal to or greater than the threshold, the process proceeds to step S403. If it is determined that the distance between the HMD 210 and the avatar is smaller than the threshold, the process proceeds to step S404.

[0048] In step S403, the designation unit 206 designates the entire body (all parts of the avatar) as the drawing range.

[0049] In step S404, designation unit 206 designates the hands as the drawing range. In this case, only the hands of the avatar are drawn in the virtual space.

[0050] In step S405, the virtual image generation unit 207 acquires dynamic data of the drawing range specified by the designation unit 206 from the dynamic data storage unit 204. Furthermore, the virtual image generation unit 207 generates a virtual space based on the static data stored in the static data storage unit 203. Then, the virtual image generation unit 207 places the drawing range of the avatar in the virtual space based on the position and orientation of the drawing range (the position and orientation acquired by the position and orientation acquisition unit 205). After that, the virtual image generation unit 207 calculates the virtual space as viewed from the "position and orientation measured by the position and orientation measurement unit 202". A virtual image that represents a virtual space is generated.

[0051] In step S406, the synthesizer 208 superimposes the virtual image on the real image to generate a mixed reality image. The synthesizer 208 outputs the mixed reality image to the display unit 212 of the HMD 210.

[0052] In step S407, if the condition for ending the process is satisfied (for example, an end instruction is input by the observing user), the synthesis unit 208 ends the process of this flowchart. On the other hand, if the condition for ending the process is not satisfied, the process returns to step S400.

[0053] An example of a process for changing a rendering range based on a distance between an observing user and an avatar will be described with reference to Figures 5A to 6B. In Figures 5A to 6B, the same components as those in Figures 1A and 1B are given the same reference numbers, and therefore descriptions of those components will be omitted.

[0054] 5A and 5B show the distance X A is a diagram illustrating a mixed reality space when is equal to or greater than a threshold. FIG. 5A is a diagram showing an overhead view of the mixed reality space. The diagram shows a position 500 of an avatar of a remote user performing work on a virtual object 102, and a position 501 of an HMD 210 worn by an observing user 110. The observing user 110 observes the avatar of the remote user 100 from a distant position. FIG. 5B is a mixed reality image displayed on the HMD 210 worn by the observing user 110. The avatar 112 of the remote user performing work on the virtual object 102 is drawn entirely.

[0055] 6A and 6B show the distance X C 13 is a diagram illustrating a mixed reality space when is smaller than a threshold value.

[0056] Fig. 6A is a bird's-eye view of the mixed reality space. Fig. 6A illustrates a position 500 of an avatar of a remote user performing work on a virtual object 102, and a position 501 of an HMD 210 worn by an observing user 110. The observing user 110 observes the avatar of the remote user 100 from a close distance. Fig. 6B illustrates a mixed reality image displayed on the HMD 210 worn by the observing user 110. Only the hands of an avatar 600 of a remote user performing work on a virtual object 102 are drawn.

[0057] According to the first embodiment, it is possible to change the rendering range based on the distance between the observing user and the avatar. When the observing user is close to the avatar (dynamic virtual object), the rendering range of the avatar is limited. This allows the observing user to observe the avatar while reducing the possibility of impeding the XR experience.

[0058] (Variation 1) In the first embodiment, the designation unit 206 sets one threshold value, and when the distance between the observing user and the avatar is smaller than the threshold value, the designation unit 206 designates the avatar's hands as the drawing range (the range of the avatar to be drawn). Also, when the distance between the observing user and the avatar is equal to or greater than the threshold value, the designation unit 206 designates the avatar's entire body as the drawing range. However, the designation unit 206 may set multiple threshold values ​​and change the drawing range depending on the magnitude relationship between the multiple threshold values ​​and the distance.

[0059] In this way, when the observing user is far from the avatar, the HMD 210 can display the entire avatar. On the other hand, as the distance between the observing user and the avatar decreases, the drawing range can be limited to, for example, the upper body, arms, hands, and fingers (or the lower body, feet, and toes). Furthermore, if the distance between the observing user and the avatar is smaller than a threshold, the hands may be specified as the drawing range, and if the distance between the observing user and the avatar is equal to or greater than the threshold, the feet may be specified as the drawing range. In this way, A completely different part may be specified as the drawing range depending on the distance between the observing user and the avatar. Also, the drawing range may be specified such that the ratio of the drawing range to the entire avatar decreases as the distance between the observing user and the avatar decreases.

[0060] (Variation 2) In the first embodiment, an example has been shown in which the dynamic virtual object is an avatar that moves based on the position and posture of a remote user. The dynamic virtual object may be any virtual object that moves regardless of the intention of an observing user.

[0061] For example, the information processing device 200 may have an animation recording unit that records an animation by linking and recording a "position and orientation", a "part that changes the position and orientation", and a "time". The dynamic virtual object may be a virtual object whose part moves according to the recorded time and position and orientation. In this case, the position and orientation information recorded in the animation recording unit is output to the position and orientation acquisition unit 205 at the time recorded in association with the position and orientation.

[0062] Furthermore, when there is a specific virtual object that is continuous with a dynamic virtual object, the dynamic virtual object and the specific virtual object may be treated as one dynamic virtual object. For example, when an avatar is holding a virtual object such as a tool, the avatar and the tool may be treated as a dynamic virtual object. In such a case, the tool may be specified as the drawing range of the dynamic virtual object.

[0063] (Variation 3) In the first embodiment, the designation unit 206 uniquely determines the drawing range in advance based on a threshold value. However, the designation unit 206 may arbitrarily determine the drawing range based on an input from the observing user.

[0064] <Embodiment 2> In the first embodiment, an example is described in which the drawing range (the range of the avatar that is drawn) is changed based only on the distance between the observing user and the avatar (dynamic virtual object). In the second embodiment, an example is described in which the drawing range is changed based on the position of the avatar in addition to the distance between the observing user and the avatar.

[0065] Fig. 7 is an overhead view of a mixed reality space in embodiment 2. In Fig. 7, the same components as those in Fig. 5 are given the same reference numerals, and description of those components will be omitted.

[0066] The mixed reality space (virtual space) is divided into area 700 and area 701. When the observing user and the avatar become closer in each area, the range (part) of the avatar corresponding to each area is specified as the drawing range. In this way, the drawing range is changed in each area, allowing the observing user to observe a more appropriate part of the avatar. For example, in area 700, work is performed using the hands, so the hands are specified as the drawing range, and in area 701, work is performed using the feet, so the feet are specified as the drawing range. In this way, the range corresponding to the work is specified and can be observed by the observing user.

[0067] Fig. 8 is a block diagram showing an example of the configuration of a system 1 according to embodiment 2. In Fig. 8, the same components as those in Fig. 2 are given the same reference numerals, and description of those components will be omitted. The information processing device 200 has an area dividing unit 800 and a designation unit 801.

[0068] The area division unit 800 divides the mixed reality space (virtual space) into a plurality of areas. The area division unit 800 then associates the range (part) of an avatar (dynamic virtual object) with each area. The area division unit 800 outputs information on the correspondence between the area and the range of the avatar to the designation unit 801. .

[0069] Any method may be used for dividing the regions, as long as the observing user can associate the range of the avatar to be observed with each region. Each region may be arbitrarily determined based on the input of the observing user. Furthermore, each region may be determined using information recorded in an animation recording unit (not shown). For example, the region dividing unit 800 calculates the amount of change in the position and orientation of each part of the remote user for each specific range in the mixed reality space, and associates parts with large amounts of change with the range. The region dividing unit 800 may treat a set of specific ranges associated with the same part as one region.

[0070] The designation unit 801 determines whether the distance between the observing user and the avatar in the mixed reality space (virtual space) is equal to or greater than a threshold, based on the position and posture of the observing user and the position and posture of the avatar. If the distance between the observing user and the avatar is equal to or greater than the threshold, the designation unit 801 designates a range corresponding to the position of the avatar (such as the whole body) as the rendering range. If the distance between the observing user and the avatar is smaller than the threshold, the designation unit 801 determines in which of the regions divided by the region division unit 800 the avatar is located, and designates the rendering range based on the determination result. The dynamic data holding unit 204 outputs data relating to the rendering range from among the data relating to the virtual object held by the dynamic data holding unit 204 to the virtual image generation unit 207.

[0071] The flowchart in Fig. 9 shows a process of changing the drawing range based on the area in which the avatar is located in addition to the distance between the observing user and the avatar. In Fig. 9, the same processes as those in Fig. 4 are given the same reference numbers and their explanations are omitted. When it is determined in step S402 that the distance between the observing user and the avatar is less than the threshold, the process in step S900 starts.

[0072] In step S900, the designation unit 801 determines whether or not an avatar is located in the region 700 divided by the region division unit 800, based on the position and orientation of the avatar acquired from the position and orientation acquisition unit 205. If it is determined that the avatar is located in the region 700, the process proceeds to step S901. If it is determined that the avatar is not located in the region 700, the process proceeds to step S902.

[0073] In step S901, the designation unit 801 designates the hand corresponding to the area 700 as the drawing range.

[0074] In step S902, the designation unit 801 determines whether or not an avatar is located in the region 701 divided by the region division unit 800, based on the position and orientation of the avatar acquired from the position and orientation acquisition unit 205. If it is determined that an avatar is located in the region 701, the process proceeds to step S903. If no avatar is present in the region 701, the process proceeds to step S904.

[0075] In step S903, the designation unit 801 designates the foot corresponding to the area 701 as the drawing range.

[0076] In step S903, the designation unit 801 designates that the avatar is not to be drawn.

[0077] According to the second embodiment, when the distance between the avatar, which is a dynamic virtual object, and the observing user is short, the rendering range is limited based on the area in which the avatar is located. This makes it possible to observe the avatar more appropriately while reducing the possibility of impeding the XR experience.

[0078] <Embodiment 3> In the first and second embodiments, when the avatar, which is a dynamic virtual object, is close to the observing user, the information processing device 200 renders only a part of the avatar in a mixed reality space (virtual space). In the third embodiment, rendering is performed based on information possessed by the avatar. In the third embodiment, the information processing device 200 renders the avatar in a mixed reality space (virtual space) based on line-of-sight information of the avatar, with the intersection of the line of sight and the virtual object as the gaze point.

[0079] Fig. 10 is a block diagram showing an example of the configuration of system 1 according to embodiment 3. The same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted. Note that the position and orientation acquisition unit 205 can acquire the viewpoint position (the position at which the avatar is looking) and line of sight direction of the avatar (remote user) based on the position and orientation of the entire body of the avatar (for example, the direction of the head).

[0080] The gaze point generation unit 1000 judges whether the distance between the observing user and the avatar is equal to or greater than a threshold based on the position and posture of the observing user and the position and posture of the avatar. If it is judged that the distance between the observing user and the avatar is smaller than the threshold, the gaze point generation unit 1000 acquires the viewpoint position and gaze direction of the avatar from the position and posture acquisition unit 205. Furthermore, the gaze point generation unit 1000 judges whether the gaze and a virtual object (a virtual object held in the dynamic data holding unit 204) intersect in the virtual space based on the viewpoint position and gaze direction. If it is judged that the gaze and the virtual object intersect, the gaze point generation unit 1000 generates a virtual object showing the intersection (the point where the gaze and the virtual object intersect) as a gaze point, and outputs it to the virtual image generation unit 1001.

[0081] The virtual image generation unit 1001 uses the static virtual objects acquired from the static data storage unit 203 and a virtual object indicating the gaze point to configure a virtual space in which an avatar can be placed (places the static virtual objects and the virtual object indicating the gaze point in the virtual space). Furthermore, the virtual image generation unit 1001 generates a virtual image that expresses the virtual space as viewed from the position and orientation measured by the position and orientation measurement unit 202. The virtual image generation unit 1001 outputs the generated virtual image to the synthesis unit 208.

[0082] The flowchart in Fig. 11 shows a process of drawing a virtual object indicating the gaze point of an avatar. In Fig. 11, the same processes as those in Fig. 4 are given the same reference numbers, and the description of those processes will be omitted. If it is determined in step S402 that the distance between the avatar and the observing user is less than the threshold, the process of step S1100 starts. Note that, when the process of step S401 ends, the process of step S1100 may start without performing the process of step S402 (regardless of the distance between the avatar and the observing user).

[0083] In step S1100, the gaze point generation unit 1000 acquires the viewpoint position and gaze direction of the avatar from the position and orientation acquisition unit 205. Furthermore, the gaze point generation unit 1000 acquires data related to a virtual object (a virtual object held by the static data holding unit 203) and determines whether or not the gaze intersects with the virtual object. If it is determined that the gaze intersects with the virtual object, the process proceeds to step S1101. If it is determined that the gaze does not intersect with the virtual object, the process proceeds to step S1102.

[0084] In step S1101, the gaze point generating unit 1000 determines an intersection point where the line of sight and a virtual object intersect as a gaze point. Then, the gaze point generating unit 1000 generates a virtual object indicating the gaze point.

[0085] FIG. 12 shows an example of a virtual object indicating the gaze point. The gaze point generating unit 1000 generates a circular virtual object 1201 indicating the gaze point at the intersection of the line of sight 1200 of the avatar and the virtual object 102 of a car. In the third embodiment, the virtual object indicating the gaze point is a circular virtual object, but it may be a virtual object of another shape. The virtual object indicating the gaze point may be a virtual object shaped like a crosshair, or a virtual object simulating a view frustum corresponding to the line of sight and the gaze point (view frustum shape The virtual object may be a virtual object.

[0086] In step S1102, if a virtual object indicating the gaze point has been generated, the virtual image generation unit 1001 configures a virtual space based on the static virtual object and the virtual object indicating the gaze point. If a virtual object indicating the gaze point has not been generated, the virtual image generation unit 1001 configures a virtual space based on the static virtual object. Then, the virtual image generation unit 1001 generates a virtual image representing the virtual space viewed from the position and orientation measured by the position and orientation measurement unit 202. Note that the virtual image generation unit 1001 does not need to generate an avatar of the remote user in the virtual space. This allows the observing user to grasp the position where the remote user is looking without the field of view being obstructed by the avatar of the remote user.

[0087] In the third embodiment, an example has been described in which a virtual object representing the gaze point is drawn when the observing user is close to the avatar (dynamic virtual object) based on the gaze information held by the avatar. However, in addition to drawing a part of the avatar, a virtual object representing the gaze point may also be drawn using the first and second embodiments.

[0088] According to the third embodiment, when the observing user is close to the avatar, the information processing device 200 renders a virtual object indicating the gaze point of the avatar. This makes it possible to observe the avatar more appropriately while reducing the possibility that the XR experience will be hindered.

[0089] <Embodiment 4> In the fourth embodiment, the information processing device 200 measures the positions and orientations of not only parts of a dynamic virtual object (avatar) but also parts of an observing user. When a part of the observing user overlaps with a part of the avatar in the mixed reality space, the information processing device 200 does not designate the overlapping part of the avatar as a drawing range. For example, when the right hand of the observing user overlaps with the right hand of the avatar, the information processing device 200 does not draw the right hand of the avatar. This makes it possible to prevent a situation in which, when an observing user who is a participant imitates the movement of an avatar of a remote user who is an instructor in XR training, it becomes difficult to know whether the observing user has imitated the movement due to the overlapping of the observing user and the avatar.

[0090] Fig. 13 is a block diagram showing an example of the configuration of a system 1 according to embodiment 4. In Fig. 13, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0091] The motion sensor 1300 recognizes reflective markers attached to the body of the observing user and acquires the position and orientation of each reflective marker. The motion sensor 1300 outputs the acquisition results of the position and orientation of each reflective marker to a position and orientation measurement unit 1301. A method of acquiring the position and orientation of the entire body using an optical sensor will be described below, but any of the methods described in FIG. 1A may be used.

[0092] The position and orientation measuring unit 1301 measures the position and orientation of the HMD 210 in the same manner as the position and orientation measuring unit 202. Furthermore, the position and orientation measuring unit 1301 converts the positions and orientations of the reflective markers acquired from the motion sensor 1300 into the positions and orientations of each part of the observing user in a mixed reality space (virtual space), and outputs information on the converted positions and orientations to the designation unit 1302.

[0093] The designation unit 1302 designates a drawing range based on the position and orientation of each body part of the observing user and the position and orientation of each body part of the avatar. First, the designation unit 1302 determines whether the distance between the observing user and the avatar in the mixed reality space (virtual space) is equal to or greater than a threshold, and determines a candidate for the drawing range. Furthermore, the designation unit 1302 designates the drawing range based on whether the body part of the observing user in the mixed reality space (virtual space) interferes with (overlaps with) the candidate for the avatar's drawing range. The dynamic data storage unit 204 stores data related to the virtual object that it stores. That is, data regarding the drawing range is output to the virtual image generating unit 207 .

[0094] The flowchart in Fig. 14 shows the process of designating the drawing range. In Fig. 14, the same processes as those in Fig. 4 are given the same reference numbers, and the description of those processes will be omitted. When it is determined in step S402 that the distance between the observing user and the avatar is less than the threshold, the process of step S1400 starts.

[0095] In step S1400, the designation unit 1302 determines the hand as a candidate for the avatar's drawing range. The designation unit 1302 further determines whether the observing user's hand and the avatar's hand interfere with each other. The interference determination can be realized, for example, by the following procedures (1) and (2).

[0096] (1) First, the designation unit 1302 generates a three-dimensional polygon model (3D model) that imitates the shape of the hand of the observing user in the mixed reality space. To generate the three-dimensional polygon model, a method of extracting the area of ​​the hand by image processing and measuring the position information by estimating the depth by stereo matching or SLAM (Simultaneous Localization and Mapping) can be used. Furthermore, the position and orientation of the hand including the fingers may be measured using Leap Motion by Leap Motion, and the area of ​​the hand may be detected from a built-in stereo camera to generate a three-dimensional polygon model that imitates the shape of the hand. The position and orientation of the hand and fingers may be estimated using a depth sensor such as Microsoft's Kinect. Furthermore, a method of generating a three-dimensional polygon model in which the orientation of the hand and fingers is estimated by repeatedly calculating from an initial position based on a depth image of the shape of the hand obtained by a depth sensor and optimizing the cost may be used.

[0097] (2) The designation unit 1302 extracts all triangular polygons constituting the three-dimensional polygon model, and extracts all triangular polygons constituting the virtual object of the avatar's hand. The designation unit 1302 determines whether or not the triangular polygons constituting the three-dimensional polygon model intersect with the triangular polygons constituting the virtual object of the avatar's hand. The designation unit 1302 performs this determination for all combinations of triangular polygons, and if it is determined that at least one intersects, it determines that the hand of the observing user and the hand of the avatar interfere with each other. If it is determined that the hand of the observing user and the hand of the avatar interfere with each other, the hand is not specified in the drawing range, and the process proceeds to step S405. If it is determined that the hand of the observing user and the hand of the avatar do not interfere with each other, the process proceeds to step S404. Note that if it is determined that the hand of the observing user and the hand of the avatar interfere with each other, the range of the hand of the avatar that does not interfere with the hand of the observing user may be specified as the drawing range.

[0098] In the fourth embodiment, the designation unit 1302 performs interference determination for one part, the hand. However, the designation unit 1302 may perform interference determination individually for multiple parts, such as dividing the hand into a right hand and a left hand. When the right hand is interfering but the left hand is not interfering, the designation unit 1302 may designate only the left hand as the drawing range.

[0099] In the fourth embodiment, an example was shown in which the hand is not included in the drawing range when two identical parts, the observing user's hand and the avatar's hand, interfere with each other. However, when two different parts interfere with each other, the hand may be excluded from the drawing range. For example, when the avatar's head is a candidate for the drawing range, if the observing user's hand interferes with the avatar's head, the avatar's head may be excluded from the parts to be drawn.

[0100] (Variation 4) In the third embodiment, the information processing device 200 determines whether or not a part of the observing user interferes with a part of the avatar (dynamic virtual object) based on the three-dimensional shape data and the position and orientation, but the present invention is not limited to this.

[0101] When it is determined based on the real image and the virtual image that a part of the observing user and a part of the avatar overlap, the synthesis unit 208 may superimpose the virtual image excluding the part of the avatar on the real image. Specifically, for example, the position and orientation measurement unit 1301 detects the hand of the observing user by using an image recognition technique on the real image captured by the imaging unit 211 or the like. Furthermore, the designation unit 1302 determines whether the hand of the observing user in the real image and the hand of the avatar in the virtual image overlap, and when the two hands overlap, excludes the hand of the avatar from the virtual image. Then, the synthesis unit 208 superimposes the real image and the virtual image to generate a mixed reality image. When the two hands do not overlap, the synthesis unit 208 superimposes the virtual image as it is on the real image to generate a mixed reality image.

[0102] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0103] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0104] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0105] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0106] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An information processing device that generates an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, a first acquisition means for acquiring a position for placing a first virtual object whose position and orientation change dynamically; A second acquisition means for acquiring a position of the user in the space; Based on the position of the user and the position of the first virtual object in the space, A designation means for designating a first range, which is a range of a virtual object to be rendered in the space; a generating means for generating an image of the space in which the first range of the first virtual object is rendered; 13. An information processing device comprising: (Configuration 2) the generating means does not render a portion of the first virtual object other than the first range, 2. The information processing device according to configuration 1. (Configuration 3) the designation means designates the first range based on whether a distance in the space between the user and the first virtual object is greater than a threshold value. 3. The information processing device according to configuration 1 or 2. (Configuration 4) the designation means designates the first range based on a magnitude relationship between a plurality of thresholds and the distance. 4. The information processing device according to configuration 3. (Configuration 5) The first virtual object is an avatar that moves in accordance with the body movement of a specific person different from the user. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) the designation means designates any one of an upper body, a lower body, fingers, hands, and feet of the avatar as the first area; 6. The information processing device according to any one of configurations 5 to 6 above. (Configuration 7) The space is a virtual space that is merged with a real space, or a mixed reality space in which the real space and the virtual space are merged, the designation means designates the first range based on whether or not the user and the first virtual object overlap when the real space and the virtual space are merged. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) the designation means, when the same parts of the user and the first virtual object overlap, designates the first range so as not to include the parts of the first virtual object that overlap with the user. 8. The information processing device according to configuration 7. (Configuration 9) the designation means determines whether or not the user and the first virtual object overlap based on a three-dimensional model of the user and a three-dimensional model of the first virtual object. 9. The information processing device according to configuration 7 or 8. (Configuration 10) The image capturing device further includes an image capturing unit for capturing a real image of the user in the real space, the designation means determines whether or not the user and the first virtual object overlap, based on an image of the virtual space in which the first virtual object is placed and the real image; 9. The information processing device according to configuration 7 or 8. (Configuration 11) a range of one of the first virtual objects is associated with each of a plurality of regions in the space; the designation means designates the first range based on a range of the first virtual object corresponding to an area in which the first virtual object is to be placed. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) An information processing device that generates an image of a space in which an avatar whose position and posture change dynamically can be arranged, the image being displayed on a display device worn on a user's head, the information processing device comprising: a first generating means for generating a first virtual object indicating a position at which the avatar gazes in the space; a second generating means for generating an image of the space in which the first virtual object is rendered; 13. An information processing device comprising: (Configuration 13) The space includes a second virtual object. when a line of sight of the avatar intersects with the second virtual object, the first generation means generates the first virtual object by setting a point of intersection between the line of sight of the avatar and the second virtual object as a position at which the avatar is gazing. 13. The information processing device according to configuration 12. (Configuration 14) The first virtual object is a virtual object having a shape of a viewing frustum according to the direction of the line of sight of the avatar and the position at which the avatar is gazing. 14. The information processing device according to configuration 12 or 13. (Configuration 15) The avatar is a virtual object that moves according to the position and posture of a specific person different from the user. 15. The information processing device according to any one of configurations 12 to 14. (Method 1) 1. An information processing method for generating an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, the method comprising: a first acquisition step of acquiring a position for placing a first virtual object whose position and orientation change dynamically; a second acquisition step of acquiring a position of the user in the space; a designation step of designating a first range, which is a range of the first virtual object to be rendered in the space, based on a position of the user and a position of the first virtual object in the space; a generating step of generating an image of the space in which the first range of the first virtual object is rendered; 13. An information processing method comprising: (Method 2) 1. An information processing method for generating an image to be displayed on a display device worn on a user's head, the image being an image of a space in which an avatar whose position and posture change dynamically can be placed, comprising: a first generation step of generating a first virtual object indicating a position at which the avatar gazes in the space; a second generation step of generating an image of the space in which the first virtual object is rendered; An information processing method comprising the steps of: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0107] 200: information processing device, 210: HMD (head mounted display), 202: position and orientation measurement unit, 205: position and orientation acquisition unit, 206: Designation unit, 207: Virtual image generation unit

Claims

1. An information processing device that generates an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, a first acquisition means for acquiring a position for placing a first virtual object whose position and orientation change dynamically; A second acquisition means for acquiring a position of the user in the space; a designation means for designating a first range, which is a range of the first virtual object to be rendered in the space, based on a position of the user and a position of the first virtual object in the space; a generating means for generating an image of the space in which the first range of the first virtual object is rendered; 13. An information processing device comprising:

2. the generating means does not render a portion of the first virtual object outside the first range, 2. The information processing apparatus according to claim 1,

3. the designation means designates the first range based on whether a distance in the space between the user and the first virtual object is greater than a threshold value.

2. The information processing apparatus according to claim 1,

4. the designation means designates the first range based on a magnitude relationship between a plurality of thresholds and the distance.

4. The information processing apparatus according to claim 3.

5. The first virtual object is an avatar that moves in accordance with the body movement of a specific person different from the user.

2. The information processing apparatus according to claim 1,

6. the designation means designates any one of an upper body, a lower body, fingers, hands, and feet of the avatar as the first area; 6. The information processing apparatus according to claim 5,

7. The space is a virtual space that is merged with a real space, or a mixed reality space in which the real space and the virtual space are merged, the designation means designates the first range based on whether or not the user and the first virtual object overlap when the real space and the virtual space are merged.

2. The information processing apparatus according to claim 1,

8. the designation means, when a same part of the user and the first virtual object overlap, designates the first range so as not to include the part of the first virtual object that overlaps with the user.

8. The information processing apparatus according to claim 7,

9. the designation means determines whether or not the user and the first virtual object overlap on the basis of a three-dimensional model of the user and a three-dimensional model of the first virtual object.

8. The information processing apparatus according to claim 7,

10. The image capturing device further includes an image capturing unit for capturing a real image of the user in the real space, The designation means is configured to designate an image of the virtual space in which the first virtual object is placed and the real image. determining whether the user and the first virtual object overlap based on the 8. The information processing apparatus according to claim 7,

11. a range of one of the first virtual objects is associated with each of a plurality of regions in the space; the designation means designates the first range based on a range of the first virtual object corresponding to an area in which the first virtual object is to be placed.

2. The information processing apparatus according to claim 1,

12. An information processing device that generates an image of a space in which an avatar whose position and posture change dynamically can be arranged, the image being displayed on a display device worn on a user's head, the information processing device comprising: a first generating means for generating a first virtual object indicating a position at which the avatar gazes in the space; a second generating means for generating an image of the space in which the first virtual object is rendered; 13. An information processing device comprising:

13. The space includes a second virtual object. when a line of sight of the avatar intersects with the second virtual object, the first generation means generates the first virtual object by setting an intersection point between the line of sight of the avatar and the second virtual object as a position at which the avatar is gazing.

13. The information processing apparatus according to claim 12.

14. the first virtual object is a virtual object having a shape of a viewing frustum corresponding to a line of sight direction of the avatar and a position at which the avatar is gazing, 13. The information processing apparatus according to claim 12.

15. The avatar is a virtual object that moves according to the position and posture of a specific person different from the user.

13. The information processing apparatus according to claim 12.

16. 1. An information processing method for generating an image of a space in which a virtual object is arranged, the image being displayed on a display device worn on a user's head, the method comprising: a first acquisition step of acquiring a position for placing a first virtual object whose position and orientation change dynamically; a second acquisition step of acquiring a position of the user in the space; a designation step of designating a first range, which is a range of the first virtual object to be rendered in the space, based on a position of the user and a position of the first virtual object in the space; a generating step of generating an image of the space in which the first range of the first virtual object is rendered; 13. An information processing method comprising:

17. 1. An information processing method for generating an image to be displayed on a display device worn on a user's head, the image being an image of a space in which an avatar whose position and posture change dynamically can be placed, comprising: a first generation step of generating a first virtual object indicating a position at which the avatar gazes in the space; a second generation step of generating an image of the space in which the first virtual object is rendered; 13. An information processing method comprising:

18. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • Program, method, and information processing apparatus

    JP2021002390A

  • Information processor, information processing method, and program

    JP2021135776A