Information processing device, information processing method, and program

Abstract

We provide information processing technology that can improve usability. [Solution] An information processing device 100 capable of displaying cross-reality content that fuses the real physical space and virtual space on a display, comprising: an acquisition unit 130 that acquires position-related information of the information processing device 100; an execution unit 140 that performs raycasting on a spatial mesh based on the position-related information; a determination unit 150 that determines whether or not a hit has occurred on the spatial mesh by raycasting; a first generation unit 161 that generates a prediction model based on the direction of the normal vector of the hit spatial mesh; and a display control unit 170 that displays the prediction model on the display.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In the field of games, games utilizing technologies (so-called XR (extended reality)) that create new experiences by integrating the real physical space (real world) and the virtual space (virtual world) have been continuously announced.

[0003] For example, it is possible to enjoy communication as if in the real world, such as taking a walk as an avatar in the virtual world or having voice chats or text chats between users.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when a user installs an object model (for example, a ladder model) desired by the user in the virtual world, it is difficult to generate, display, etc. an object model with an appropriate shape at a desired position of the spatial mesh (that is, a mesh generated based on the shape scanned from the real world), and improvement of user usability has been demanded.

[0006] Therefore, one of the objectives of the present invention is to provide an information processing technology capable of improving user usability compared to the prior art.

Means for Solving the Problems

[0007] An information processing device according to one aspect of the present invention is an information processing device capable of displaying cross-reality content that fuses a real physical space and a virtual space on a display, and is characterized by comprising: an acquisition unit that acquires position-related information of the information processing device; an execution unit that performs raycasting on a spatial mesh based on the position-related information; a determination unit that determines whether or not a hit has occurred on the spatial mesh by raycasting; a first generation unit that generates a prediction model based on the direction of the normal vector of the hit spatial mesh; and a display control unit that displays the prediction model on the display. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve usability compared to conventional methods. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram showing how to place and display an object model desired by the user at any location on a spatial mesh. [Figure 2] This figure shows an example configuration of the content display system according to this embodiment. [Figure 3] This figure shows an example of the main hardware configuration of the HMD and server device. [Figure 4] This figure shows an example of the main functional block configuration of a server device. [Figure 5] This figure shows an example of the main functional block configuration of an HMD. [Figure 6] This is a diagram illustrating a spatial mesh. [Figure 7] This is an explanatory diagram illustrating the normal vector of a spatial mesh. [Figure 8] This is an explanatory diagram illustrating a predictive model. [Figure 9] This flowchart illustrates the overview of processing performed by the HMD. [Figure 10] This flowchart illustrates the process of displaying a predictive model using an HMD (Head-Mounted Display). [Figure 11]This flowchart illustrates the process of displaying object models using an HMD (Head-Mounted Display). [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described with reference to the attached drawings. In each drawing, components denoted by the same reference numerals have the same or similar configurations.

[0011] A. Embodiment <Overview of the technology related to this disclosure> The technology disclosed herein enables a user wearing an XR-based HMD (Head Mounted Display) to place and display a desired object model (in this case, a ladder model) Mo at any location on a spatial mesh in a game application (content) for an HMD (Head Mounted Display) (see Figure 1).

[0012] <System Configuration> Figure 2 shows an example configuration of the content display system 1000 according to this embodiment. The content display system 1000 includes an HMD 100 and a server device 200.

[0013] The HMD100 and the server device 200 are interconnected via a communication network N. The communication network N refers to a communication path capable of data communication, and includes not only dedicated lines (dedicated cables) for direct connection, LANs (Local Area Networks) using Ethernet®, etc., but also communication networks such as telephone networks, cable networks, and the Internet. Furthermore, the communication method is either wired or wireless.

[0014] The HMD (information processing device, computer) 100 is, for example, a terminal for XR that combines AR (Augmented Reality) and VR (Virtual Reality), and is used by being worn on the user's head. In the present embodiment, an HMD 100 for XR is assumed, but it is also applicable to an HMD 100 for AR or an HMD 100 for VR. Further, the HMD 100 is not limited to being wearable on the user's head, and includes any terminal capable of realizing AR, VR, etc., such as, for example, glasses-type smart glasses.

[0015] The HMD 100 includes various sensors, buttons, speakers, microphones, batteries, memories, etc., and can be connected to the outside either wired or wirelessly. Software for displaying game applications (contents) on the display is stored in the HMD 100. The contents can be acquired from a server device 200 or the like via the communication network N.

[0016] Further, the HMD 100 may have a function of detecting the user's line of sight and pointing in the line-of-sight direction, and a function of detecting the movement of the user's finger (for example, a movement such as pinching the index finger and the thumb) and clicking. By using these functions, the user can place and display a desired object model (for example, a ladder model, etc.) included in the contents at an arbitrary location on the spatial mesh.

[0017] The server device 200 controls the contents to be displayed on the HMD 100 in accordance with instructions transmitted from the HMD 100, a user terminal (not shown), etc. As an example, when the server device 200 receives a selection instruction for the contents desired by the user from the HMD 100, it extracts the corresponding contents from the database and transmits them to the HMD 100. The server device 200 may be composed of one or more servers, or may be configured using a cloud server.

[0018] <Hardware Configuration> Figure 3 shows an example of the main hardware configuration of the HMD100 and server device 200.

[0019] The HMD100 and server device 200 include a processor 1, memory 2, storage device 3, a communication device 4 for wired or wireless communication, an input device 5, an output device 6, and the like.

[0020] Processor 1 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and it comprehensively controls the entire device.

[0021] Memory 2 consists of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and / or RAM (Random Access Memory).

[0022] The storage device 3 consists of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and / or an eMMC (embedded Multi Media Card).

[0023] Communication device 4 is a device that communicates via a wired and / or wireless network, such as a network card or a communication module.

[0024] The input device 5 is, for example, an operation button, a touch panel, and / or a microphone, and the output device 6 is, for example, a display and / or a speaker.

[0025] In addition to this hardware, the HMD100 is equipped with a Camera CA. The Camera CA is located, for example, at the edge of the HMD100 frame and captures images of the real world (e.g., spatially reconstructed photographs or videos) that are within the field of view of the user wearing the HMD100. The HMD100 also uses the Camera CA to perform raycasting. Raycasting is a function that emits a ray of light from a certain point (e.g., the user's viewpoint or the camera) and obtains information about the object that is hit by the ray.

[0026] <Functional Block Configuration> (Server device 200) Figure 4 shows an example of the main functional block configuration of the server device 200. The server device 200 includes a storage unit 210, a control unit 220, and a communication unit 230. The storage unit 210 may be implemented by a storage device 3, and the control unit 220 may be implemented by the processor 1 executing a program stored in the storage device 3. The communication unit 230 may be implemented by a communication device 4, or by the processor 1 executing a program stored in the storage device 3 in addition to the communication device 4.

[0027] The memory unit 210 stores various control programs and various types of data. The memory unit 210 stores programs that provide content in response to instructions from the user using the HMD 100.

[0028] The storage unit 210 also includes a user database DB1, a content database DB2, and the like. User database DB1 stores basic information such as user ID, name, address, telephone number, and email address for each user who receives content (in this case, various game applications). In addition to each user's basic information, user database DB1 may also store content history information (date and time of use, number of times, content, etc.).

[0029] The content database DB2 stores information for each piece of content, including the content ID, content genre, and main information that makes up the content. There are no particular restrictions on the content genre, but if it is a game, it may be a role-playing game (RPG), action game, adventure game, simulation game, etc.

[0030] The control unit 220 comprehensively controls each part of the server device 200. The control unit 220 also controls content provision (transmission), etc., according to instructions transmitted from each user's HMD 100. The communication unit 230 exchanges various content-related instructions with the HMD 100 and other devices.

[0031] (HMD100) Figure 5 shows an example of the main functional block configuration of the HMD100. The HMD100 includes a storage unit 110, a control unit 120, an acquisition unit 130, an execution unit 140, a determination unit 150, a generation unit 160, a display control unit 170, a reception unit 180, and a communication unit 190. The storage unit 110 may be implemented by a storage device 3, and the control unit 120, acquisition unit 130, execution unit 140, determination unit 150, generation unit 160, display control unit 170, and reception unit 180 may be implemented by the processor 1 executing a program stored in the storage device 3. Furthermore, the communication unit 190 may be implemented by a communication device 4, or by the processor 1 executing a program stored in the storage device 3 in addition to the communication device 4.

[0032] The memory unit 110 stores various control programs and various data. Among the control programs is the content control application AP1 for displaying content on the display. The HMD 100 executes the content control application AP1 according to the user's instructions, enabling the user wearing the HMD to place and display a desired object model (here, a ladder model is assumed) at any location on the spatial mesh (see Figure 1).

[0033] The control unit 120 comprehensively controls each part of the HMD 100. The acquisition unit 130 acquires position-related information such as the position and orientation of the user's viewpoint (or camera) while wearing the HMD 100.

[0034] The execution unit 140 performs raycasting and other operations on the spatial mesh based on the position-related information acquired by the acquisition unit 130. Figure 6 is an example of a spatial mesh. The execution unit 140 emits a ray from a specific point (for example, the user's viewpoint or camera) onto the spatial mesh as shown in Figure 6 and acquires information on the objects hit by the ray.

[0035] The determination unit 150 determines, by raycasting, whether or not a hit has occurred on the spatial mesh (in other words, whether or not a hit has occurred on any of the meshes that make up the spatial mesh), etc.

[0036] The generation unit 160 comprises a first generation unit 161 and a second generation unit 162. The first generation unit 161 checks whether the direction of the normal vector of the hit spatial mesh is within a range perpendicular to the upward vector (X:0, Y:0, Z:1) in the 3D virtual space (virtual world) (see, for example, Figure 7). The vertical range can be set and changed by, for example, the operator of the game application, and is set to 30 degrees to 150 degrees. Of course, the setting of the vertical range is arbitrary, and it may be set to 90 degrees ± 30 degrees, for example.

[0037] The first generation unit 161 then generates a prediction model (in this case, a fixed ladder model before extension; see Figure 8) Mp if the direction of the normal vector is within a range perpendicular to the upward vector in the virtual world.

[0038] After the user selects and confirms the prediction model Mp, the second generation unit 162 generates a portion of the object model Mo corresponding to the prediction model (in this case, the upper part of the ladder model) at the position where the prediction model is displayed. Subsequently, if predetermined conditions are met, the second generation unit 162 extends the portion of the object model to generate the extended object model Mo (see Figure 1). The detailed operation of the second generation unit 162 will be described in detail later, so it is omitted here.

[0039] The display control unit 170 is responsible for controlling the display of the display, and controls the display of the prediction model Mp and the object model Mo corresponding to the prediction model.

[0040] The reception unit 180 receives various instructions regarding the predictive model Mp, such as instructions from the user to select and confirm the predictive model Mp. The communications unit 190 exchanges various types of information with the HMD 100.

[0041] The following describes the process from when a user wearing the HMD100 places and displays a desired object model (here, a ladder model) at any location on the spatial mesh, with reference to the diagrams.

[0042] Figures 9 to 11 are flowcharts showing the main operations of the HMD100. As a prerequisite, we assume that the content control application AP1 and other necessary software are installed on the HMD100 and that the device is ready to display content.

[0043] <Overview of the process (see Figure 9)> While running content (such as a game application), the user operates the HMD100 to instruct it to display a desired prediction model (in this case, a ladder model). Upon receiving the instruction to display the prediction model, the HMD100 executes the process of displaying the prediction model (step S1; see Figure 8).

[0044] The HMD100 determines whether the user has performed a selection and confirmation operation (e.g., touch operation) of the displayed prediction model (step S2). If the HMD100 does not perform a selection and confirmation operation of the prediction model after a predetermined time has elapsed (step S2; NO), it skips step S3 and terminates.

[0045] On the other hand, when the HMD100 detects that the user has selected and confirmed a prediction model through touch operation or other means (step S2; YES), it executes the display process of the object model corresponding to the prediction model (in this case, the extended ladder model) (step S3), and then terminates the process.

[0046] <Display process of the prediction model (see Figure 10)> The acquisition unit 130 of the HMD100 acquires position-related information such as the position and orientation of the user's viewpoint (or camera) while wearing the HMD100 (step S101).

[0047] The execution unit 140 performs a raycast on the spatial mesh based on the position-related information acquired by the acquisition unit 130 (step S102).

[0048] The determination unit 150 determines whether or not a hit has occurred on the spatial mesh by raycasting (step S103). The first generation unit 161 receives the determination result from the determination unit 150 and, if it confirms that no hit has occurred on the spatial mesh (step S103; NO), returns to step S101.

[0049] On the other hand, when the first generation unit 161 confirms that it has hit a spatial mesh (step S103; YES), it checks whether the direction of the normal vector of the hit spatial mesh is within a range perpendicular to the upward vector (X:0, Y:0, Z:1) in the 3D virtual space (virtual world) (step 104; see Figure 7).

[0050] If the first generation unit 161 determines that the direction of the normal vector of the hit spatial mesh is outside the range perpendicular to the upward vector in the virtual world (step 104; NO), it returns to step S101.

[0051] On the other hand, if the first generation unit 161 determines that the direction of the normal vector of the hit spatial mesh is within a range perpendicular to the upward vector in the virtual world (step S104; YES), it generates a prediction model (here, a fixed ladder model before stretching; see Figure 8) Mp (step S105).

[0052] The display control unit 170 displays the prediction model Mp generated by the first generation unit 161 on the display (step S106) and terminates the process.

[0053] <Display process for object models (see Figure 11)> After the user has selected and confirmed the prediction model Mp, the second generation unit 162 of the HMD100 generates a portion of the object model Mo corresponding to the prediction model (in this case, the upper part of the ladder model) at the position where the prediction model is displayed (step S201).

[0054] The execution unit 140 performs a downward raycast starting from a part of the generated object model Mo (step S202). The determination unit 150 determines whether or not the downward raycast hit a spatial mesh (step S203).

[0055] The second generation unit 162 receives the determination result from the determination unit 150 and, if it confirms that it has not hit the spatial mesh (step S203; NO), it skips step S204 and terminates the process.

[0056] Meanwhile, when the second generation unit 162 confirms that it has hit a spatial mesh (step S203; YES), it measures the distance from a part of the object model Mo to the hit spatial mesh (e.g., the floor mesh), calculates the number of steps of the object mesh (i.e., ladder mesh) required to extend to the floor mesh, and generates the extended object model Mo (step S204; see Figure 1).

[0057] The display control unit 170 displays the stretched object model Mo generated by the second generation unit 162 on the display (step S205), and then terminates the process.

[0058] As described above, according to this embodiment, a user wearing an HMD can place and display a desired object model (here, a ladder model) Mo at any location on the spatial mesh, thereby improving usability.

[0059] B. Variations In the embodiment described above, a ladder model was used as an example of an object model, but this is not the only example. For example, a bridge model connecting specific spatial meshes may be generated. In the embodiment described above, a game application was used as an example of content information, but of course, the present invention is not limited to this. For example, the present invention can be applied to various applications, including 3D architecture applications.

[0060] Furthermore, while the above-described embodiment assumes the use of the HMD100 and describes the case where content is displayed in a virtual three-dimensional space (virtual world) on the display, this is not the only applicable case. For example, it can also be applied when a general-purpose computer (information processing device) displays some kind of content in two dimensions on a display.

[0061] Computers can be any device equipped with a display, such as laptops, personal computers, smartphones, tablet devices, mobile phones, and wearable devices. Even in such cases, the user operating the device can place and display the desired object model (here, a ladder model) Mo at any location on the spatial mesh in two-dimensional space, thereby improving usability.

[0062] The embodiments described above are provided to facilitate understanding of the present invention and are not intended to limit its interpretation. The flowcharts, sequences, elements of the embodiments, and their arrangement, materials, conditions, shapes, and sizes described in the embodiments are not limited to those exemplified and can be modified as appropriate. [Explanation of Symbols]

[0063] 1000...Content display system, 100...HMD, 110...Storage unit, AP1...Content control application, 120...Control unit, 130...Acquisition unit, 140...Execution unit, 150...Decision unit, 160...Generation unit, 161...First generation unit, 162...Second generation unit, 170...Display control unit, 180...Reception unit, 190...Communication unit, 200...Server device, 210...Storage unit, DB1...User database, DB2...Content database, 220...Control unit, 230...Communication unit, Mp...Prediction model, Mo...Object model

Claims

1. An information processing device capable of displaying cross-reality content that fuses the real physical space and virtual space on a display, An acquisition unit that acquires location-related information of the information processing device, An execution unit that performs raycasting on a spatial mesh based on the aforementioned position-related information, The aforementioned raycast includes a determination unit that determines whether or not a hit occurred on the spatial mesh, A first generation unit generates a prediction model based on the direction of the normal vector of the hit spatial mesh, A display control unit that displays the prediction model on the display and An information processing device equipped with the following.

2. The information processing apparatus according to claim 1, wherein the first generation unit generates the prediction model when the direction of the normal vector of the hit spatial mesh is within a vertical range set with respect to the upward vector in the virtual space.

3. A reception unit that receives instructions for selecting and confirming the aforementioned prediction model, The information processing apparatus according to claim 1 or 2, further comprising: a second generation unit that, after receiving an instruction to select and confirm the prediction model, generates a part of an object model corresponding to the prediction model at the position where the prediction model is displayed.

4. The execution unit performs a raycast on a downward spatial mesh, starting from a portion of the object model generated by the second generation unit. The determination unit determines whether or not the downward raycast hits the spatial mesh. The information processing apparatus according to claim 3, wherein the second generation unit, when the downward raycast hits the spatial mesh, generates the object model extended to the hit spatial mesh based on the distance from a part of the object model to the hit spatial mesh.

5. The information processing apparatus according to claim 4, wherein the display control unit displays the stretched object model generated by the second generation unit on the display.

6. A control method for an information processing device capable of displaying cross-reality content that fuses the real physical space and virtual space on a display, A step of acquiring location-related information of the information processing device, An execution step of performing a raycast on a spatial mesh based on the aforementioned position-related information, The aforementioned raycasting includes a determination step of determining whether or not it hits a spatial mesh, A first generation step generates a predictive model based on the direction of the normal vector of the hit spatial mesh, A display control step of displaying the prediction model on the display, A control method including

7. A computer capable of displaying cross-reality content that fuses the real physical space and virtual space on a display, An acquisition unit that acquires location-related information of the aforementioned computer, An execution unit that performs raycasting on a spatial mesh based on the aforementioned position-related information, The aforementioned raycast includes a determination unit that determines whether or not a hit occurred on the spatial mesh, A first generation unit generates a prediction model based on the direction of the normal vector of the hit spatial mesh, A program for causing the predictive model to function as a display control unit that displays the predictive model on the display.

Images