Information processing device and information processing program

Abstract

To provide an information processing device and program for preventing a first virtual object image from being overlooked by a user when the first virtual object image is hidden by a second virtual object image in a two-dimensional image obtained by projection converting a three-dimensional space in which a plurality of virtual objects are arranged, as compared with the case where the two-dimensional image is displayed directly as is.SOLUTION: In the information processing device, an image processing unit 26 projects nodes as a plurality of virtual objects arranged in a virtual three-dimensional space to a virtual screen, on the basis of virtual viewpoints set in the virtual three-dimensional space and a visual line direction, and generates a two-dimensional image that includes a plurality of node images corresponding respectively to the nodes. When a first node image is hidden by a second node image in the two-dimensional image, a two-dimensional image where at least one of the first and second node images is highlighted, or a two-dimensional image where the second node image is transparently displayed is displayed on a display 12.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an information processing device and an information processing program. [Background technology]

[0002] Conventionally, there is a known device that places multiple virtual objects, each associated with predetermined information, in a virtual three-dimensional space, sets a virtual viewpoint and line of sight in the virtual three-dimensional space, and displays a two-dimensional image obtained by projecting the virtual three-dimensional space from the virtual viewpoint to the line of sight on a display.

[0003] Patent Document 1 discloses an information processing device that arranges multiple icons, each representing electronic content, as multiple virtual objects in a virtual three-dimensional space, displays a projected image obtained by projecting the virtual three-dimensional space onto a field of view, and moves the icon in the virtual space closer to the user when a cursor indicating the user's viewpoint overlaps with an icon image in the projected image. Patent Document 2 discloses a software analysis support system that arranges multiple display elements, each corresponding to software components, as multiple virtual objects in a virtual three-dimensional space, and displays an image projected from the virtual three-dimensional space or from the user's viewpoint set outside of the virtual space. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2017-117008 [Patent Document 2] Japanese Patent Publication No. 2020-184259 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Depending on the position or direction of the virtual viewpoint set in the virtual 3D space, in a 2D image obtained by projecting a virtual 3D space containing multiple virtual objects, multiple virtual object images corresponding to multiple virtual objects may overlap, obscuring some virtual object images. For example, in a 2D image, the image of the first virtual object corresponding to the first virtual object may be obscured by the image of the second virtual object corresponding to the second virtual object. In this case, the user may overlook the first virtual object image because they cannot see it. If the first virtual object image is overlooked, the user will miss the information associated with the first virtual object.

[0006] The object of the present invention is to reduce the likelihood of a user overlooking a first virtual object image when the first virtual object image is hidden by a second virtual object image in a two-dimensional image obtained by projecting a virtual three-dimensional space in which multiple virtual objects are arranged, compared to displaying the two-dimensional image as is. [Means for solving the problem]

[0007] The invention according to claim 1 comprises a processor, the processor arranges a plurality of virtual objects, each associated with predetermined information, in a virtual three-dimensional space, sets a virtual viewpoint and a line of sight in the virtual three-dimensional space, projects the plurality of virtual objects onto a virtual screen based on the virtual viewpoint and the line of sight, generates a two-dimensional image including a plurality of virtual object images corresponding to the plurality of virtual objects, and in the two-dimensional image, if the first virtual object image is obscured by the second virtual object image, the first virtual object image It remains hidden The second virtual object The statue The highlighted 2D image The statue This is an information processing device characterized by displaying information on a display. 。 request request 2The invention according to claim 1 is characterized in that the processor determines whether or not the first virtual object image is hidden by the second virtual object image within a target area, which is an area within a predetermined viewing angle with respect to the line-of-sight direction, in the two-dimensional image. Claim 3 The invention according to claim 1 is characterized in that the processor generates a two-dimensional image including an attribute image indicating an attribute of information associated with a virtual object corresponding to the virtual object image, which is arranged accompanying the virtual object image, and when the attribute image accompanying the first virtual object image is hidden by the second virtual object image or the attribute image accompanying the second virtual object image in the two-dimensional image, causes the display to display the two-dimensional image in which at least one of the first virtual object image, the attribute image accompanying the first virtual object image, the second virtual object image, or the attribute image accompanying the second virtual object image is highlighted, or the two-dimensional image in which the second virtual object image is transparently displayed. 。 request Claim 4 The invention according to claim 1 is characterized in that a computer arranges a plurality of virtual objects, each associated with predetermined information, in a virtual three-dimensional space, sets a virtual viewpoint and a line-of-sight direction in the virtual three-dimensional space, projects the plurality of virtual objects onto a virtual screen based on the virtual viewpoint and the line-of-sight direction to generate a two-dimensional image including a plurality of virtual object images corresponding to the plurality of virtual objects, and when the first virtual object image is hidden by the second virtual object image in the two-dimensional image, causes the display to display It remains hidden the first virtual object image The statue or the second virtual object The statue highlighted two-dimensional image

Advantages of the Invention

[0008] Claim 1 or 4According to the invention related to the above, in a two-dimensional image obtained by performing a projection transformation on a virtual three-dimensional space in which a plurality of virtual objects are arranged, when a first virtual object image is hidden by a second virtual object image, it is possible to suppress the user from overlooking the first virtual object image as compared with the case of displaying the two-dimensional image as it is. Claim 2 According to the invention related to the above, it is possible to reduce the processing amount of the determination process as to whether or not virtual object images overlap each other and the highlighting process when they overlap, as compared with the case of determining whether or not virtual object images overlap each other for the entire field of view of the user. Claim 3 According to the invention related to the above, it is possible to suppress the user from overlooking an attribute image corresponding to a first virtual object image as compared with the case of displaying the two-dimensional image as it is.

Brief Description of the Drawings

[0009] [Figure 1] It is a schematic configuration diagram of an information processing apparatus according to the present embodiment. [Figure 2] It is a diagram showing a network graph arranged in a virtual three-dimensional space. <o [Figure 3] It is a diagram showing a state where a node is projected onto a virtual screen. [Figure 4] It is a first diagram showing a highlighted node. [Figure 5] It is a second diagram showing a highlighted node. [Figure 6] It is a third diagram showing a highlighted node. [Figure 7] It is a fourth diagram showing a highlighted node. [Figure 8] It is a diagram showing an example in which a node image hiding other node images is transparently displayed. [Figure 9] It is a first diagram showing a state where a node image is moved. [Figure 10] It is a diagram showing a state where a node image is enlarged. [Figure 11]The second figure shows how the node image is moved. [Figure 12] This diagram shows how a node moves in a virtual 3D space in response to the movement of its image. [Figure 13] This figure shows the node image and the direction of node movement when the height of the virtual viewpoint is above a threshold height. [Figure 14] This figure shows the node image and node movement direction when the height of the virtual viewpoint is below the threshold height. [Figure 15] This is a first flowchart showing the processing flow of the information processing device according to this embodiment. [Figure 16] This is a second flowchart showing the processing flow of the information processing device according to this embodiment. [Modes for carrying out the invention]

[0010] Figure 1 is a schematic diagram of the information processing device 10 according to this embodiment. The information processing device 10 is a device that generates a 2D image showing how multiple virtual objects, which are virtually arranged in a virtual 3D space, appear from a virtual viewpoint set in the virtual 3D space, and displays it on the display 12.

[0011] The information processing device 10 may be a VR (Virtual Reality) device that displays a 2D image showing multiple virtual objects placed in a virtual 3D space along with a background of the virtual 3D space. Alternatively, the information processing device 10 may be an AR (Augmented Reality) device that displays a 2D image showing multiple virtual objects placed in a virtual 3D space along with a background of the real 3D space. Furthermore, the information processing device 10 may be an MR (Mixed Reality) or SR (Substitutional Reality) device that realizes a combination of VR and AR. VR, AR, MR, SR, etc., are collectively referred to as XR (Extended Reality). In other words, the information processing device 10 may be an XR device.

[0012] The information processing device 10 is envisioned as a device that is worn or held by the user. For example, the information processing device 10 could be an HMD (Head Mounted Display), smart glasses, or a tablet terminal. Of course, the information processing device 10 is not limited to these devices.

[0013] The display 12 is composed of, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) display. A two-dimensional image formed by the processor 20, described later, is displayed on the display 12. The display 12 may also be a transparent display.

[0014] The acceleration sensor 14 is a sensor that detects the position and orientation of the information processing device 10. Specifically, the acceleration sensor 14 performs a calibration process when the information processing device 10 is in a predetermined position and in a predetermined orientation, and detects the displacement of the information processing device 10 from the calibration position in three mutually orthogonal axis directions defined in real three-dimensional space, and the rotation angle from the calibration orientation with each of the three axes as the central axis.

[0015] The input interface 16 consists of buttons, a touch panel, and the like. The input interface 16 is used when a user inputs instructions to the information processing device 10.

[0016] The memory 18 is composed of, for example, an eMMC (embedded Multi Media Card), ROM (Read Only Memory), or RAM (Random Access Memory). The memory 18 stores information processing programs for operating each part of the information processing device 10. The information processing programs can also be stored on a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory or a CD-ROM. The information processing device 10 can read and execute information processing programs from such storage media.

[0017] The processor 20 refers to a processor in a broad sense and consists of at least one of a general-purpose processor (e.g., a CPU (Central Processing Unit)) and a dedicated processing unit (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a programmable logic device). The processor 20 may not consist of a single processing unit, but rather of multiple processing units located in physically separate locations working together. As shown in Figure 1, the processor 20 performs its functions as an object control unit 22, a viewpoint setting unit 24, and an image processing unit 26 according to the information processing program stored in the memory 18.

[0018] The object control unit 22 performs control over a plurality of virtual objects virtually arranged in a three-dimensional space. In this embodiment, the object control unit 22 arranges a network graph in the virtual three-dimensional space.

[0019] Figure 2 shows X V Axis, Y V Axis, and Z V This figure shows a network graph 42 virtually arranged in a virtual three-dimensional space 40 defined by axes. The network graph 42 consists of a plurality of nodes 44, each associated with predetermined information, and edges 46 that indicate the relationship between two nodes 44 (more precisely, the relationship between the information associated with those two nodes 44). In this embodiment, the nodes 44 are spherical objects having a predetermined size (radius). The edges 46 are linear objects connecting the nodes 44.

[0020] The information associated with node 44 can be various types of information. For example, it could be a single component used in a product, or a single module that makes up a program, but it is not limited to these. The information associated with each node 44 is stored in memory 18 beforehand. The positions of multiple nodes 44 in the virtual three-dimensional space 40 are determined according to the information associated with each node 44. For example, multiple nodes 44 with similarly meaningful information are placed close to each other. Also, the edges 46 may be directed edges that have a direction. According to the network graph 42, the user can easily grasp the relationships between multiple pieces of information.

[0021] Multiple nodes 44 included in the network graph 42 correspond to multiple virtual objects. However, the virtual objects that the object control unit 22 places in the virtual 3D space 40 are not limited to nodes 44.

[0022] The object control unit 22 places the node 44 as a virtual object in the virtual 3D space 40, and also performs the process of moving the node 44 in the virtual 3D space 40. This will be described later.

[0023] The viewpoint setting unit 24 sets a virtual viewpoint 50 and a line of sight direction 52 in the virtual three-dimensional space 40 where the nodes 44 (i.e., the network graph 42) are located. In this embodiment, the viewpoint setting unit 24 sets the position of the virtual viewpoint 50 based on the position of the information processing device 10 in the real three-dimensional space, and sets the line of sight direction 52 based on the orientation of the information processing device 10 in the real three-dimensional space.

[0024] Specifically, when the acceleration sensor 14 performs calibration, a virtual viewpoint 50 is set at a predetermined position in the virtual 3D space 40. From there, the position of the virtual viewpoint 50 in the virtual 3D space 40 is changed according to the displacement of the information processing device 10 from the calibration position detected by the acceleration sensor 14 (displacement in the real 3D space). Also, when the acceleration sensor 14 performs calibration, a predetermined direction in the virtual 3D space 40 is set as the line of sight direction 52. From there, the line of sight direction 52 in the virtual 3D space 40 is changed according to the rotation angle of the information processing device 10 from the calibration orientation detected by the acceleration sensor 14 (rotation angle in the real 3D space). The field of view from the virtual viewpoint 50 has a predetermined field of view angle (for example, 180°), and the line of sight direction 52 means the direction that is the center of that field of view. In other words, the field of view from the virtual viewpoint 50 has a field of view angle centered on the line of sight direction 52.

[0025] Furthermore, the viewpoint setting unit 24 may also set an upward vector indicating the upward direction of the field of view from the virtual viewpoint 50. The upward vector can also be set based on the orientation of the information processing device 10 in the real 3D space. Note that the upward vector is set to the vertically upward side (Z) of the virtual 3D space 40. V It may be fixed to the positive side of the axis.

[0026] The image processing unit 26 generates a two-dimensional image showing how multiple nodes 44 arranged in the virtual three-dimensional space 40 appear when viewed from the virtual viewpoint 50 in the direction of the line of sight 52, based on the virtual viewpoint 50 and line of sight direction 52 set by the viewpoint setting unit 24.

[0027] Figure 3 is a conceptual diagram showing the processing of the image processing unit 26. First, the image processing unit 26 defines a virtual screen 54 in the virtual three-dimensional space 40. The virtual screen 54 is a plane perpendicular to the line of sight direction 52 and may be sized to cover the field of view from the virtual viewpoint 50. The image processing unit 26 projects multiple nodes 44 onto the virtual screen 54 to generate a two-dimensional image containing multiple node images 56 corresponding to the multiple nodes 44. The node images 56 are basically opaque images.

[0028] In this embodiment, the image processing unit 26 generates a two-dimensional image by perspective projection. That is, projection lines are drawn from the virtual viewpoint 50 to each point of the node 44, and the collection of the intersections with the virtual screen 54 is defined as the node image 56 corresponding to the node 44. The closer the node 44 is to the virtual viewpoint 50 in the virtual three-dimensional space 40, the larger the corresponding node image 56 becomes.

[0029] Note that the horizontal direction of the two-dimensional image (i.e., the virtual screen 54) is referred to as the X I axis, and the vertical direction of the two-dimensional image is referred to as the Y I axis. The Y I axis direction is parallel to the extension direction of the upward vector. When the upward vector is on the upper side in the vertical direction of the virtual three-dimensional space 40, the X I axis points in the horizontal direction of the virtual three-dimensional space 40, and the Y I axis points in the vertical direction of the virtual three-dimensional space 40.

[0030] When the image processing unit 26 generates a two-dimensional image, it obtains the center coordinates of each node image 56 in the X I Y I coordinate system, and the size of each node image 56. In this embodiment, since the node 44 is spherical, the node image 56 is circular, so the image processing unit 26 obtains the radius of the node image 56 as the size of the node image 56.

[0031] In the example of FIG. 3, the state in which the node 44 is projected is shown, but the edge 46 is also projected onto the virtual screen 54, and the edge image corresponding to the edge 46 may be included in the two-dimensional image. Further, when the information processing device 10 is a VR device, the image processing unit 26 may project the background (objects other than the node 44) of the virtual three-dimensional space 40 onto the virtual screen 54 to generate a two-dimensional image.

[0032] Furthermore, the image processing unit 26 may generate an attribute image that shows the attributes of the information associated with the node 44, and place the attribute image adjacent to the node image 56 corresponding to the node 44. For example, in the example in Figure 3, a label image 58 showing the name of the node 44 is placed near the node image 56 corresponding to the node 44 as an attribute image. Note that the attributes shown by the attribute image are not limited to the name of the node 44, but may be various types of information related to the information associated with the node 44.

[0033] In the example in Figure 3, node images 56a corresponding to node 44a, node image 56b corresponding to node 44b, node image 56c corresponding to node 44c, and node image 56d corresponding to node 44d are formed. Now, let's focus on nodes 44c and 44d. Nodes 44c and 44d are located in different positions in the virtual 3D space 40, but from the virtual viewpoint 50, nodes 44c and 44d are in the same direction. Therefore, when projected onto the virtual screen 54, node images 56c and 56d overlap each other. Specifically, the node image 56c corresponding to node 44c overlaps the node image 56d corresponding to node 44d, which is located further from the virtual viewpoint 50 in the virtual 3D space 40. In other words, in the 2D image, the node image 56d, as the first virtual object image, is hidden by the node image 56c, as the second virtual object image. This could cause a user viewing the 2D image to overlook node image 56d (i.e., node 44d). In this embodiment, the process described below prevents a user from overlooking node image 56 (i.e., node 44 corresponding to node image 56) that has been obscured by other node images 56 in the 2D image.

[0034] The image processing unit 26 displays the generated two-dimensional image on the display 12. If the information processing device 10 is a VR device, the image processing unit 26 displays a two-dimensional image on the display 12 that includes the background and node images 56 of the virtual three-dimensional space 40. If the information processing device 10 is an AR device, MR device, or SR device, the image processing unit 26 displays the two-dimensional image it generated on the display 12 by superimposing it onto an image of the real space captured by a camera (not shown) of the information processing device 10. Alternatively, if the display 12 is a transparent display, the image processing unit 26 may display the generated two-dimensional image on the display 12, so that the user sees an image in which the real world seen through the display 12 and the two-dimensional image generated by the image processing unit 26 are superimposed.

[0035] The image processing unit 26 determines whether the node images 56 overlap each other in the generated two-dimensional image. In other words, the image processing unit 26 determines whether the first node image 56 is hidden by the second node image 56 in the two-dimensional image. In this embodiment, the image processing unit 26 determines that the two node images 56 overlap each other if the distance between the center of the first node image 56 and the center of the second node image 56 in the two-dimensional image is within a threshold distance. The threshold distance may be a predetermined fixed value, or it may be determined based on the radius of the node image 56 with the larger size (radius) among the two node images 56 that are the subject of the overlap determination process. Note that there may be other methods for determining whether the node images 56 overlap each other. Furthermore, although this embodiment describes the case where two node images 56 overlap each other as an example, the same process can be performed when three or more node images 56 overlap each other.

[0036] If there are no overlapping node images 56 in the generated 2D image, the image processing unit 26 displays the 2D image as is on the display 12. If there are overlapping node images 56 in the 2D image, the image processing unit 26 displays a 2D image on the display 12 in which at least one of the two overlapping node images 56 is highlighted.

[0037] The following explanation describes an example of highlighting, referring to Figures 4 to 6. In the following explanation, we will use the case where node image 56d is hidden by node image 56c as an example. In Figures 4 to 6, the figures shown to the left of the arrows show node images 56c and 56d before highlighting, and the figures shown to the right of the arrows show node images 56c and 56d after highlighting.

[0038] Figure 4 shows an example where node image 56c is highlighted by changing its color. Figure 5 shows an example where node image 56c is highlighted by displaying a highlight image 60 surrounding it. In this way, by highlighting node image 56c which is hiding node image 56d, the user can understand that there are other node images 56 hidden behind the highlighted node image 56, even though they cannot directly see node image 56d. The user, having understood that there are hidden node images 56, can change the position and orientation of the information processing device 10 to change the virtual viewpoint 50 and line of sight direction 52 in the virtual 3D space 40, thereby confirming the hidden node image 56. In other words, the user is less likely to overlook node image 56d, i.e., node 44d.

[0039] Figure 6 shows an example where node image 56d is highlighted by enlarging its size. By enlarging node image 56d, which was hidden by node image 56c, a portion of node image 56d becomes visible beyond node image 56c, allowing the user to see node image 56d.

[0040] Furthermore, the image processing unit 26 may determine whether the first label image 58 associated with the first node image 56 is hidden by the second node image 56 or the second label image 58 associated with the second node image 56 in the generated two-dimensional image. If the position of the label image 58 relative to the node image 56 and the size of the label image 58 are known, the image processing unit 26 can determine whether the label image 58 is hidden by other node images 56 or other label images 58. If the first label image 58 associated with the first node image 56 is hidden by the second node image 56 or the second label image 58, the image processing unit 26 may display a two-dimensional image on the display 12 in which at least one of the first node image 56, the first label image 58, the second node image 56, or the second label image 58 is highlighted.

[0041] Figure 7 shows an example where the label image 58d associated with the node image 56d is hidden by the node image 56c (the figure to the left of the arrow), and the node image 56c is highlighted by changing its color (the figure to the right of the arrow). As described above, in this case, the label 58c, the node image 56d, or the label image 58d may be highlighted instead of or in addition to the node image 56c. For example, the label 58c or the node image 56d may be highlighted by changing its color, or by enlarging the label image 58d.

[0042] Furthermore, the image processing unit 26 may also display a two-dimensional image on the display 12 in which the second node image 56 is transparent when the first node image 56 is hidden by the second node image 56. Figure 8 shows the case where node image 56d is hidden by node image 56c (the figure to the left of the arrow), and the node image 56c is displayed transparently (the figure to the right of the arrow). By displaying node image 56c, which is hiding node image 56d, transparently, the user can see node image 56d. It is advisable to either not display the frame of node image 56c transparently, or not set the transparency to 100%, so that the user can also see the transparently displayed node image 56c.

[0043] If the image processing unit 26 were to determine whether or not the node image 56 or label image 58 is hidden in the entire generated 2D image and then perform highlighting processing, the processing load for both the determination process and the highlighting process would become large. A large processing load for both the determination process and the highlighting process could lead to a problem where the processing time until highlighting is extended. On the other hand, users tend to mainly look at the area around the line of sight direction 52 (i.e., the center of the field of view) in a 2D image. Therefore, the image processing unit 26 may set the area within a predetermined field of view angle based on the line of sight direction 52 as the target area for the determination process in the generated 2D image, and determine whether or not the node image 56 or label image 58 is hidden only within that target area.

[0044] As described above, when a node image 56 or label image 58 is hidden by another node image 56 or label image 58, highlighting can be performed to allow the user to recognize the hidden node image 56 or label image 58. In addition, when a node image 56 or label image 58 is hidden by another node image 56 or label image 58, the node 44 corresponding to the hidden node image 56 or label image 58 can be temporarily moved in the virtual 3D space 40 to allow the user to recognize the hidden node image 56 or label image 58. The following describes an embodiment of moving the node 44 in the virtual 3D space 40.

[0045] Similar to the embodiment described above, the image processing unit 26 determines whether the node images 56 in the generated two-dimensional image overlap each other. If there are overlapping node images 56, the image processing unit 26 first moves at least one of the two overlapping node images 56 on the two-dimensional image so that they no longer overlap each other.

[0046] Figure 9 shows the case where node image 56d is obscured by node image 56c (the figure to the left of the arrow), and node image 56d is X IThis figure shows the movement in the axial direction (the figure to the right of the arrow). Since it is sufficient to obtain a state in which node image 56c and node image 56d do not overlap with each other, the moving node image 56 may be either node image 56c or node image 56d. The preferred direction of movement for node image 56c or 56d will be described later, but the direction of movement for node image 56c or 56d may be any direction as long as they do not overlap.

[0047] The image processing unit 26 moves the node image 56 by a threshold displacement amount or more. The threshold displacement amount may be a predetermined fixed value, but it may also be determined based on the radius of the node image 56 with the larger size (radius) of the two overlapping node images 56.

[0048] The image processing unit 26 may, instead of moving at least one of the overlapping node images 56, enlarge the size of the node image 56, or enlarge it and change its overlapping order. This is equivalent to moving the corresponding node 44 in the line of sight direction 52 in the virtual three-dimensional space 40. Figure 10 shows the case where node image 56d is hidden by node image 56c (figure to the left of the arrow), and how node image 56d is enlarged and its overlapping order with node image 56c is changed (figure to the right of the arrow).

[0049] Furthermore, if the first label image 58 associated with the first node image 56 is hidden by the second node image 56 or the second label image 58, the image processing unit 26 may move at least one of the first node image 56 or the second node image 56 on the two-dimensional image so that the first node image 56, the first label image 58, the second node image 56, and the second label image 58 can be seen by the user. Figure 11 shows the case where the label image 58d associated with the node image 56d is hidden by the node image 56c (figure to the left of the arrow), and the node image 56c has been moved (figure to the right of the arrow).

[0050] After moving or enlarging the node image 56 on the 2D image, the object control unit 22 moves the node 44 corresponding to the node image 56 in the virtual 3D space 40 in accordance with the movement of the node image 56. In this embodiment, the object control unit 22 performs inverse perspective projection on the moved node image 56 to determine the position (coordinates) of the node 44 corresponding to the moved node image 56 in the virtual 3D space 40, and moves the node 44 to that position.

[0051] Figure 12 shows how node 44 moves in the virtual 3D space 40 in response to the movement of node image 56. Specifically, node image 56d is hidden by node image 56c, and when node image 56d is moved in the 2D image, the corresponding node 44d is moved. This process means that at least one of the two nodes 44 corresponding to the two node images 56 is moved in the virtual 3D space 40 so that the user can see both of the two overlapping node images 56 from the user's perspective (i.e., looking in the line of sight direction 52 from the virtual viewpoint 50).

[0052] As described above, the position of each node 44 in the virtual 3D space 40 is determined according to the information associated with each node 44. Therefore, if node 44d moves in the virtual 3D space 40, it may be necessary to move other nodes 44 as well. In that case, the object control unit 22 moves the other node 44 in the virtual 3D space 40. The image processing unit 26 then projects the moved other node 44 onto the virtual screen 54 and regenerates a 2D image including the node image 56 corresponding to the moved other node 44.

[0053] As described above, when nodes 44 corresponding to overlapping node images 56 are moved in the virtual 3D space 40, other nodes 44 may also move as a result. Therefore, in the regenerated 2D image, node images 56 may overlap due to the movement of other nodes. The image processing unit 26 determines whether or not the node images 56 overlap in the regenerated 2D image. If there are no overlapping node images 56 in the regenerated 2D image, the object control unit 22 determines the position of each node 44 in the virtual 3D space 40. If there are overlapping node images 56 in the regenerated 2D image, the image processing unit 26 moves at least one of the two overlapping node images 56 again in the 2D image so that they do not overlap.

[0054] The object control unit 22 and the image processing unit 26 repeat the above process until the position of each node 44 in the virtual three-dimensional space 40 is determined.

[0055] After the position of each node 44 is determined in the virtual 3D space 40, the image processing unit 26 generates a 2D image containing multiple node images 56 corresponding to the multiple moved nodes 44 and displays it on the display 12.

[0056] Thus, if node image 56d is hidden by node image 56c, the user can see node image 56d in the 2D image generated by the image processing unit 26 by moving node 44c corresponding to node image 56c or node 44d corresponding to node image 56d in the virtual 3D space 40. This reduces the likelihood of the user overlooking node image 56d, i.e., node 44d.

[0057] The following describes the preferred movement direction of the node image 56 on the 2D image, that is, the preferred movement direction of the node 44 in the virtual 3D space 40, with reference to Figures 13 and 14.

[0058] If node images 56 overlap in a 2D image, the image processing unit 26 first obtains the height h of the virtual viewpoint 50 in the virtual 3D space 40. Then, it determines whether the obtained height h is greater than or equal to a predetermined threshold height. Here, the threshold height is set to a height at which it can be determined that the user is standing in the real 3D space. In other words, determining that the height h of the virtual viewpoint 50 is greater than or equal to the threshold height means that it is highly likely that the user is standing, and determining that the height h of the virtual viewpoint 50 is less than the threshold height means that it is highly likely that the user is sitting.

[0059] If the height h of the virtual viewpoint 50 is greater than or equal to the threshold height, the image processing unit 26 determines, based on the upward vector set by the viewpoint setting unit 24, that the corresponding node 44 in the virtual 3D space 40 is in the vertical direction (Z V Move at least one of the two overlapping node images 56 on the 2D image so that it moves in the axial direction.

[0060] Referring to Figure 13, consider the case where node image 56f, corresponding to node 44f, is hidden by node image 56e, corresponding to node 44e. Here, the direction of the upward vector is Z. V Assuming they coincide in the positive axis direction, the image processing unit 26 determines that at least one of the node images 56e and 56f (node ​​image 56f in the example of Figure 13) is Y I Move in the axial direction. In the example in Figure 13, the image processing unit 26 moves the node image 56f to the Y I Although it is being moved in the negative direction of the axis, node image 56f is Y I You may also move it in the positive direction of the axis.

[0061] In response to the movement of the node image 56f on the 2D image, the object control unit 22 moves the node 44f corresponding to the node image 56 in the virtual 3D space 40 in the vertical direction (Z V Move it in the axial direction.

[0062] If the height h of the virtual viewpoint 50 is greater than or equal to a threshold height, that is, if the user is likely to be standing, the user can easily move the information processing device 10 left or right by walking, that is, move the virtual viewpoint 50 horizontally (X) in the virtual 3D space 40. V Y V It can be said that it is easy to move (in the plane direction). Therefore, if node 44f is moved horizontally in the virtual 3D space 40, the virtual viewpoint 50 will also move horizontally, and if the moved node 44f, node 44e, and the moved virtual viewpoint 50 are aligned in a straight line, then from the perspective of the moved virtual viewpoint 50, the node image 56f corresponding to the moved node 44f will be hidden by the node image 56e. In this embodiment, in order to avoid such a situation, if the height h of the virtual viewpoint 50 is greater than or equal to the threshold height, at least one of node 44e or node 44f is moved vertically in the virtual 3D space 40.

[0063] On the other hand, if the height h of the virtual viewpoint 50 is less than the threshold height, the image processing unit 26 determines that the corresponding node 44 in the virtual 3D space 40 is in the horizontal direction (X) based on the upward vector set by the viewpoint setting unit 24. V Y V Move or enlarge at least one of two overlapping node images 56 on the 2D image so that it moves in the plane direction of the surface.

[0064] Referring to Figure 14, consider the case where node image 56f, corresponding to node 44f, is hidden by node image 56e, corresponding to node 44e. Here, the direction of the upward vector is Z. V Assuming they coincide with the positive axis direction, the image processing unit 26 processes at least one of the node images 56e and 56f (node ​​image 56f in the example of Figure 14) on the 2D image using X I Move in the axial direction. In the example in Figure 14, the image processing unit 26 moves the node image 56f to the X I Although it is being moved in the negative direction of the axis, node image 56f is X I You may also move it in the positive direction of the axis.

[0065] In response to the movement of the node image 56f on the 2D image, the object control unit 22 moves the node 44f corresponding to the node image 56 in the virtual 3D space 40 in the horizontal direction (X V Y V Move it (in the direction of the surface).

[0066] If the height h of the virtual viewpoint 50 is below a threshold, that is, if the user is likely to be sitting, the user is likely to move the information processing device 10 vertically, such as by standing up, that is, in the virtual 3D space 40, the virtual viewpoint 50 is likely to move vertically (Z V It can be said that it is easy to move it in the axial direction. Therefore, if node 44f is moved vertically in the virtual 3D space 40, the virtual viewpoint 50 will also move vertically, and if the moved node 44f, node 44e, and the moved virtual viewpoint 50 are aligned in a straight line, then from the perspective of the moved virtual viewpoint 50, the node image 56f corresponding to the moved node 44f will be hidden by the node image 56e. In this embodiment, in order to avoid such a situation, if the height h of the virtual viewpoint 50 is less than the threshold height, at least one of node 44e or node 44f is moved horizontally in the virtual 3D space 40.

[0067] Similar to the highlighting process example described above, if the image processing unit 26 determines whether the node image 56 or label image 58 is hidden in the entire generated 2D image, and the object control unit 22 performs the movement process for the corresponding node 44, the processing load for the determination process and the movement process becomes large. A large processing load for the determination process and the movement process can lead to the problem of a longer processing time until the 2D image is displayed. Therefore, the image processing unit 26 may set the area within a predetermined field of view angle based on the line of sight direction 52 as the target area for the determination process, and determine whether the node image 56 or label image 58 is hidden only within that target area.

[0068] Furthermore, moving the node 44 in the virtual 3D space 40 will change the shape of the network graph 42. However, there may be users who do not want the shape of the network graph 42 to be changed. In such cases, the object control unit 22 and the image processing unit 26 may, in response to instructions from the user (in other words, waiting for instructions from the user), execute the process of moving the node image 56 on the 2D image, the process of moving the node 44 in the virtual 3D space 40, and the process of displaying the 2D image containing the multiple node images 56 corresponding to the multiple moved nodes 44 on the display 12. The user can input these instructions to the information processing device 10 using the input interface 16.

[0069] The overview of the information processing device 10 according to this embodiment is as described above. The processing flow of the information processing device 10 according to this embodiment will now be explained according to the flowcharts shown in Figures 15 and 16.

[0070] First, referring to Figure 15, we will explain the processing flow when highlighting at least one of the overlapping node images 56.

[0071] In step S10, the object control unit 22 places the network graph 42, which includes the node 44 as a virtual object, into the virtual three-dimensional space 40.

[0072] In step S12, the viewpoint setting unit 24 sets a virtual viewpoint 50 and a line of sight direction 52 in the virtual three-dimensional space 40 based on the position and orientation of the information processing device 10.

[0073] In step S14, the image processing unit 26 projects the multiple nodes 44 onto the virtual screen 54 in perspective based on the virtual viewpoint 50 and line of sight direction 52 set in step S12, and generates a two-dimensional image including multiple node images 56 corresponding to the multiple nodes 44.

[0074] In step S16, the image processing unit 26 determines whether or not there are overlapping node images 56 in the two-dimensional image generated in step S14. If there are overlapping node images 56, the process proceeds to step S18; otherwise, the process proceeds to step S20.

[0075] In step S18, the image processing unit 26 highlights at least one of the two overlapping node images 56 in the two-dimensional image generated in step S14.

[0076] In step S20, the image processing unit 26 displays the two-dimensional image generated in step S14, or the two-dimensional image in which the node image 56 was highlighted in step S18, on the display 12.

[0077] Next, referring to Figure 16, we will explain the processing flow when moving at least one node 44 corresponding to the overlapping node images 56 in the virtual 3D space 40. In the example in Figure 16, the upward vector set by the viewpoint setting unit 24 is in the vertical direction (Z) of the virtual 3D space 40. V Assume it coincides with the positive axis direction.

[0078] Since the processes from steps S30 to S36 are the same as those from steps S10 to S16 in the flowchart of Figure 15, redundant explanations will be omitted.

[0079] If, in the two-dimensional image generated in step S34, the two node images 56 overlap each other (yes in step S36), then in step S38, the image processing unit 26 obtains the height h of the virtual viewpoint 50 in the virtual three-dimensional space 40 and determines whether the height h is greater than or equal to a threshold height.

[0080] If the height h is greater than or equal to the threshold height, the process proceeds to step S40, in which step S40 the image processing unit 26 determines that at least one of the two overlapping node images 56 on the 2D image is Y IMove in the axial direction. On the other hand, if the height h is less than the threshold height, proceed to step S42, in step S42, the image processing unit 26 moves at least one of the two overlapping node images 56 on the 2D image to X I Move it along the axis.

[0081] In step S44, the object control unit 22 moves the node 44 corresponding to the node image 56 in the virtual three-dimensional space 40 in accordance with the movement of the node image 56 in step S40 or S42. If the node image 56 is moved in step S40, then in step S44, the node 44 corresponding to the node image 56 is moved in the vertical direction of the virtual three-dimensional space 40. If the node image 56 is moved in step S42, then in step S44, the node 44 corresponding to the node image 56 is moved in the horizontal direction of the virtual three-dimensional space 40.

[0082] After step S44, the process returns to step S34, and in step S34 again, a two-dimensional image is regenerated that includes the node image 56 corresponding to the node 44 moved in step S44.

[0083] In step S36, the image processing unit 26 determines whether two node images 56 overlap each other in the 2D image regenerated in step S34. If there are overlapping node images 56, the processing in steps S38 to S44 is executed again. That is, the image processing unit 26 and the object control unit 22 repeat the processing in steps S38 to S44 until there are no more overlapping node images 56 in the 2D image.

[0084] If there are no more overlapping node images 56 in the 2D image, the process proceeds to step S46. In step S46, the image processing unit 26 displays the 2D image generated in step S34 on the display 12.

[0085] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications are possible without departing from the spirit of the invention.

[0086] For example, each function of the processor 20 of the information processing device 10 may be implemented in a device separate from the device worn or held by the user (e.g., a server). In this case, the server or other device receives information indicating the position and orientation of the device from the device worn or held by the user to set a virtual viewpoint and line of sight. It also transmits a two-dimensional image generated by the image processing unit 26 of the device to the device, causing it to be displayed on the device's display. [Explanation of symbols]

[0087] 10 Information processing unit, 12 Display, 14 Accelerometer, 16 Input interface, 18 Memory, 20 Processor, 22 Object control unit, 24 Viewpoint setting unit, 26 Image processing unit, 40 Virtual 3D space, 42 Network graph, 44 Node, 46 Edge, 50 Virtual viewpoint, 52 Line of sight direction, 54 Virtual screen, 56 Node image, 58 Label image, 60 Highlight image.

Claims

1. Equipped with a processor, The aforementioned processor, By placing multiple virtual objects, each associated with predetermined information, in a virtual three-dimensional space, A virtual viewpoint and line of sight are set in the aforementioned virtual three-dimensional space. Based on the virtual viewpoint and the line of sight, the plurality of virtual objects are projected onto a virtual screen to generate a two-dimensional image containing the plurality of virtual object images corresponding to the plurality of virtual objects. In the aforementioned two-dimensional image, if the first virtual object image is hidden by the second virtual object image, the two-dimensional image in which the second virtual object image is highlighted while the first virtual object image remains hidden is displayed on the display. An information processing device characterized by the following:

2. The aforementioned processor, In the aforementioned two-dimensional image, within a target region which is a region within a predetermined field of view angle based on the line of sight direction, it is determined whether or not the first virtual object image is hidden by the second virtual object image. The information processing apparatus according to feature 1.

3. The aforementioned processor, A two-dimensional image is generated that includes an attribute image, which is placed in conjunction with the virtual object image and shows the attributes of the information associated with the virtual object corresponding to the virtual object image. In the two-dimensional image, if the attribute image associated with the first virtual object image is obscured by the second virtual object image or the attribute image associated with the second virtual object image, the two-dimensional image in which at least one of the first virtual object image, the attribute image associated with the first virtual object image, the second virtual object image, or the attribute image associated with the second virtual object image is highlighted, or the two-dimensional image in which the second virtual object image is transparently displayed, is displayed on the display. The information processing apparatus according to feature 1.

4. On the computer, Multiple virtual objects, each associated with specific information, are placed in a virtual three-dimensional space. A virtual viewpoint and line of sight are set in the aforementioned virtual three-dimensional space. Based on the virtual viewpoint and the line of sight, the plurality of virtual objects are projected onto a virtual screen to generate a two-dimensional image containing the plurality of virtual object images corresponding to the plurality of virtual objects. In the aforementioned two-dimensional image, if the first virtual object image is hidden by the second virtual object image, the two-dimensional image in which the second virtual object image is highlighted while the first virtual object image remains hidden is displayed on the display. An information processing program characterized by the following features.

Images