Information prompt system, method, computer readable medium and computer program product

Abstract

The present invention provides an information prompting system, an information prompting method and a computer-readable medium. The information prompting system includes: a display device, which provides a virtual three-dimensional work space for a user; an input device, which allows the user to operate an object in the three-dimensional work space; and a processor, which executes a program to display the three-dimensional work space on the display device, display a document in the three-dimensional work space, and display a representative object representing the document in the three-dimensional work space in association with the document, and when an operation of the representative object in the three-dimensional work space performed by the input device is detected, the processor performs a process of operating multiple pages constituting the document represented by the representative object.

Description

Technical Field

[0001] The present disclosure relates to an information prompting system, an information prompting method and a computer-readable medium. Background Art

[0002] A physical workspace requires a place to place a desk, etc., and once it is determined, its size cannot be easily changed. Also, because data cannot be synchronized between analog media and digital media, reconversion frequently occurs. On the other hand, a virtual workspace can remove these restrictions and provide an environment that allows work to be performed without interrupting thinking by increasing the freedom of display or operation.

[0003] Japanese Patent Laid-Open No. 2000-194475 describes a technology that allows even users who are not very skilled in machine operation to easily manage or browse documents. In the generation information input process, as information required to generate a three-dimensional graphic model of a book, the number of pages M of the book, the size of the page A4, and the file index of the page image attached to each page are input. The file index has two indexes of high resolution and low resolution. In the three-dimensional model production process, the three-dimensional graphic of the book is produced based on the generation information obtained through the input process. In the texture mapping process, based on the saved information of the image corresponding to each page of the three-dimensional model of the book obtained through the input process, the page image in the storage device is attached to each page model through texture mapping. As the page image to be saved, two resolution images of low resolution and high resolution are prepared for the same page.

[0004] Japanese Patent Laid-Open No. 2004-246712 describes a technology that can improve the overview of objects by providing a three-dimensional overview of the hierarchical structure of objects with deep hierarchies, and can analyze or view the relationship and growth process of a group of objects. On a display unit, multiple images as thumbnails of files and videos that constitute an object are displayed in the shape of a polyhedron. When there are a large number of images, the images can be stacked at the bottom (or top), and the user can confirm the content and structure of the object based on the images displayed as thumbnails in a three-dimensional graphical user interface (GUI). The user can operate a cursor in the display area through an input unit, and when the cursor is aligned with an image, the title of the image is displayed in text.

[0005] Japanese Patent Publication No. 2013-175161 describes a method for panoramically visualizing documents or their document components by using a method that reflects the logical relationship between documents / components. The panoramic visualization document navigation system includes a navigation engine and a request interface. The navigation engine is configured to receive the layout of the document components of a panoramic visualization document collection of document components, and the document components respectively have associated metadata, and the associated metadata provides information related to each document component. The navigation engine is configured to adjust the visual representation of the layout based on and in accordance with the request. Summary of the invention

[0006] In addition, when reading a large number of documents, the pages are often arranged or the documents are overlapped for comparison or reference. In this operation, a work space such as a table is required to place the documents. However, by using a virtual three-dimensional work space, the work space can be ensured or the freedom of operation is improved, thereby improving the readability of the documents.

[0007] However, if a plurality of pages constituting a document are arranged over a wide range due to a high degree of freedom, it will take a lot of time to rearrange the pages according to certain rules, which will reduce work efficiency.

[0008] The present disclosure aims to provide a technology that can perform sorting operations on a plurality of pages constituting a document as needed in a virtual three-dimensional work space generated by a computer.

[0009] According to a first scheme of the present disclosure, there is provided an information prompting system, comprising: a display device, providing a virtual three-dimensional working space for a user; an input device, allowing the user to operate an object in the three-dimensional working space; and a processor, wherein the processor executes a program to display the three-dimensional working space on the display device, display a document in the three-dimensional working space, and display a representative object representing the document in the three-dimensional working space in association with the document, and when an operation of the representative object in the three-dimensional working space performed through the input device is detected, the processor performs processing to operate multiple pages constituting the document represented by the representative object.

[0010] According to the second aspect of the present disclosure, when a movement operation of the representative object is detected by the input device, the processor performs a process of moving the representative object while maintaining relative positions of a plurality of pages constituting the document represented by the representative object.

[0011] According to a third aspect of the present disclosure, upon detecting a display / non-display switching operation of the representative object performed through the input device, the processor performs a process of uniformly switching display / non-display of a plurality of pages constituting the document represented by the representative object.

[0012] According to a fourth aspect of the present disclosure, when a specific operation of the representative object performed through the input device is detected, the processor performs a process of collectively aligning a plurality of pages constituting the document represented by the representative object.

[0013] According to a fifth aspect of the present disclosure, the representative object presents a hierarchical structure including at least a higher-level representative object and a lower-level representative object lower than the higher-level representative object.

[0014] According to a sixth aspect of the present disclosure, the representative object includes a partial representative object temporarily representing only a part of the plurality of pages constituting the document.

[0015] According to the seventh aspect of the present disclosure, when a specific operation of the representative object performed through the input device is detected, the processor performs processing to move only the representative object without moving a plurality of pages constituting the document represented by the representative object.

[0016] According to an eighth aspect of the present disclosure, when an operation of the representative object performed through the input device is detected, the processor performs a process of changing the display form of a plurality of pages constituting a document represented by the representative object whose operation is detected.

[0017] According to a ninth aspect of the present disclosure, the processor performs processing for displaying a plurality of pages constituting a document represented by the representative object whose operation is detected in a manner that enables identification of the relationship with the representative object.

[0018] According to the tenth scheme of the present disclosure, the processor displays a first representative object representing a first document and a second representative object representing a second document as the representative objects, and when an operation is detected through the input device to associate a specific page constituting the first document with the second representative object, the processor changes the representative target of the specific page from the first representative object to the second representative object.

[0019] According to the eleventh scheme of the present disclosure, there is provided a computer-readable medium storing a program for causing a computer to execute processing, wherein the processing includes the following steps: displaying a three-dimensional work space on a display device; displaying a document in the three-dimensional work space; associating a representative object representing the document with the document and displaying it in the three-dimensional work space; and upon detecting an operation of the representative object in the three-dimensional work space performed through an input device, performing processing for operating multiple pages constituting the document represented by the representative object.

[0020] According to the twelfth scheme of the present disclosure, there is provided an information prompt method, comprising the following steps: displaying a three-dimensional work space on a display device; displaying a document in the three-dimensional work space; associating a representative object representing the document with the document and displaying it in the three-dimensional work space; and upon detecting an operation of the representative object in the three-dimensional work space performed through an input device, performing processing on multiple pages constituting the document represented by the representative object.

[0021] (Effect)

[0022] According to the first, eleventh or twelfth aspect, in the virtual three-dimensional work space, a plurality of pages constituting a document can be sorted as needed.

[0023] According to the second aspect, the plurality of pages constituting the document can be moved while maintaining their relative positions.

[0024] According to the third aspect, further, a plurality of pages constituting a document can be collectively switched between display and non-display.

[0025] According to the fourth aspect, further, a plurality of pages constituting a document can be aligned collectively.

[0026] According to the fifth aspect, a hierarchical structure can be established between representative objects.

[0027] According to the sixth aspect, further, only a part of the plurality of pages constituting the document can be temporarily represented.

[0028] According to the seventh aspect, further, it is possible to move only the representative object without moving the plurality of pages constituting the document.

[0029] According to the eighth aspect, the display form of the plurality of pages constituting the document can be changed.

[0030] According to the ninth aspect, it is further possible to recognize the relationship between a plurality of pages constituting a document and the representative object.

[0031] According to the tenth aspect, the representative object can be further changed. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural block diagram of an implementation method.

[0033] Figure 2 It is an explanatory diagram of installing a head mounted display according to an embodiment.

[0034] Figure 3 It is an explanatory diagram showing an example of the xR space according to the embodiment.

[0035] Figure 4 It is an explanatory diagram of a file system according to an embodiment.

[0036] Figure 5 It is a diagram for explaining display of objects in the xR space according to the embodiment.

[0037] Figure 6 This is a diagram for explaining the operation of a representative object of the embodiment.

[0038] Figure 7 It is a diagram for explaining a coordinate system of a representative object of the embodiment and objects belonging to the representative object.

[0039] Figure 8 This is a table explanatory diagram showing a representative object of an embodiment and a data structure of objects belonging to the representative object.

[0040] Fig. 9 It is a process flow chart of an implementation method.

[0041] Fig.10 is another process flow diagram of an embodiment.

[0042] Fig.11A This is a diagram for explaining the moving operation of a representative object according to an embodiment (part 1).

[0043] Fig. 11B This is a diagram for explaining the moving operation of a representative object according to the embodiment (part 2).

[0044] Fig. 12A This is a diagram (part 1) illustrating the movement operation of only a representative object according to the embodiment.

[0045] Fig. 12B This is a diagram for explaining the movement operation of only a representative object according to the embodiment (part 2).

[0046] Fig.13A This is a diagram for explaining display / non-display operation of a representative object according to an embodiment (part 1).

[0047] Fig. 13B This is a diagram for explaining display / non-display operation of a representative object according to the embodiment (part 2).

[0048] Fig.14A This is a diagram for explaining alignment operation of representative objects according to an embodiment (part 1).

[0049] Fig. 14B This is a diagram for explaining the alignment operation of representative objects according to the embodiment (part 2).

[0050] Fig.15A This is a diagram (part 1) illustrating the structure and operation of a representative object of the embodiment.

[0051] Fig. 15B This is a structural and operational explanation diagram (part 2) of a representative object of the implementation scheme.

[0052] Fig.16A This is a diagram for explaining a partial selection operation of a representative object according to an embodiment (part 1).

[0053] Fig. 16B This is a diagram for explaining a partial selection operation of a representative object according to an embodiment (part 2).

[0054] Fig.17A This is a diagram for explaining the change operation of a representative object according to the embodiment (part 1).

[0055] Fig. 17B This is a diagram for explaining the change operation of a representative object of the implementation mode (part 2). DETAILED DESCRIPTION

[0056] Hereinafter, embodiments of the present disclosure will be described based on the drawings.

[0057] <Structure>

[0058] Figure 1 The information presentation system in this embodiment includes a display unit 10 , an input unit 12 , a position / posture detection unit 14 , an xR space calculation unit 16 , a contact determination unit 18 , a display image calculation unit 20 , a calculation unit 22 , and a file system 24 .

[0059] The display unit 10 presents a virtual three-dimensional space for viewing or operating a document to the user. The display unit 10 can present parallax to the user, thereby enabling the user to recognize the space, such as depth, in three dimensions. Examples of the display unit 10 that can present parallax include, but are not limited to, a binocular head-mounted display. The position or posture of the display unit 10 in the real space is detected by the position / posture detection unit 14 and reflected on the display object in the virtual three-dimensional space.

[0060] The input unit 12 is a device for the user to operate an object displayed in a virtual three-dimensional space. The position and posture of the input unit 12 in the real space are detected by the position / posture detection unit 14 and reflected on the input unit object in the virtual three-dimensional space. The input unit 12 may be a physically existing device such as a controller for a virtual three-dimensional space, or the user's hand detected by a sensor may be used as the input unit 12. Moreover, the input unit 12 may be single or multiple, and the detection signal of the position and posture of the input unit 12 is sent from the position / posture detection unit 14 to the operation unit 22 as an operation signal for the object. In addition, for the timing of sending the operation signal, the physical button operation provided in the input unit 12 may be used as a trigger, or the recognition result of the user's hand gesture detected by the sensor may be used as a trigger.

[0061] The position / posture detection unit 14 detects the position or posture of the display unit 10 and the input unit 12 in the real space. The position / posture detection unit 14 may be built in the display unit 10 or may be provided outside the display unit 10 independently of the display unit 10 .

[0062] The xR space calculation unit 16 reflects the position or posture of the display unit 10 and the input unit 12 detected by the position / posture detection unit 14 to the virtual three-dimensional space to calculate the position and posture of all objects in the virtual three-dimensional space. All objects in the virtual three-dimensional space refer to display unit objects or input unit objects, representative objects of documents, or objects belonging to the representative objects. The display unit object calculates the position and posture in the virtual three-dimensional space based on the information of the real space detected by the position / posture detection unit 14. Moreover, the input unit object calculates the position and posture in the virtual three-dimensional space based on the information of the real space detected by the position / posture detection unit 14. In addition, here, the so-called xR refers to the general term for technologies used to construct virtual three-dimensional space, specifically, virtual reality (VR), augmented reality (AR), and mixed reality (MR) are all included. Hereinafter, the virtual three-dimensional space is referred to as xR space.

[0063] The contact determination unit 18 determines whether objects in the xR space are in contact with each other.

[0064] The display image calculation unit 20 calculates an image to be displayed based on the position and posture of the display unit object in the xR space and outputs the calculated image to the display unit 10 .

[0065] The calculation unit 22 performs various calculations. The calculation unit 22 may be built in the display unit 10.

[0066] The file system 24 is a system that manages the input / output of documents that the user browses or operates in the xR space. The documents are managed by the file system 24 as electronic data. The file system 24 can exist as a server computer independently of the operation unit 22, etc., or it can be connected using a communication network. A document generally contains multiple pages, each of which has its own attributes. However, a document may also contain only one page. When a document is output to the xR space, a representative object is created. Moreover, the pages that constitute the document are converted into separate objects and belong to the representative object.

[0067] The representative object and the object have position and posture information in the xR space. Moreover, the object belonging to the representative object has relative position and posture information relative to the coordinate system of the representative object. The relative position and posture information can be stored in advance or calculated each time. These position and posture information are stored in the file system 24.

[0068] Specifically, the xR space calculation unit 16, the contact determination unit 18, the display image calculation unit 20, and the operation unit 22 can be implemented using the processor 23 of the computer. The processor 23 reads out the processing program stored in the program memory such as the read-only memory (ROM) and executes it, thereby functioning as the xR space calculation unit 16, the contact determination unit 18, the display image calculation unit 20, and the operation unit 22. Here, the so-called processor 23 refers to a processor in a broad sense, including a general-purpose processor (central processing unit (CPU), etc.) or a dedicated processor (graphics processing unit (GPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA), programmable logic device, etc.). Moreover, the action of the processor 23 can also be completed not only by one processor, but also by a plurality of processors located in physically separated positions. If the main processing of the processor 23 is listed, it is as follows.

[0069] (1) Calculate the xR space and display it on the display unit 10

[0070] (2) Arrange and display multiple objects in the document in the xR space

[0071] (3) Create representative objects representing multiple objects in the document and display them in the xR space

[0072] (4) Detecting the operation of an object in the xR space according to the user's operation on the input unit 12, and operating the object according to the operation content

[0073] (5) Detecting the operation of the representative object in the xR space according to the operation of the input unit 12 by the user, and uniformly operating all objects belonging to the representative object according to the operation content

[0074] Moreover, as described above, the display unit 10, the input unit 12, and the position / posture detection unit 14 can be realized, for example, by using a head-mounted display and an xR controller. The head-mounted display is a display worn on the head, and can be a virtual image projection method that forms a virtual image by using a half-mirror or the like so that the image can be observed, or a retinal projection method that uses the lens of the eye to directly form an image on the retina. The head-mounted display can also be a three-degree-of-freedom method that detects the tilt of the three axes of XYZ, but it is ideal to detect the position information in the three-dimensional space in addition to the tilt. The head-mounted display can convert the user's movement into the xR space by tracking an infrared light emitting diode (IRLED). The xR controller is linked to the head-mounted display, and by operating the xR controller, the user can make the object displayed in the xR space, that is, the virtual hand (virtual hand), move like his own hand. Of course, the xR controller is not necessary, and the user's hand can also be detected by a sensor built into the head-mounted display.

[0075] Figure 2 The head mounted display 30 is used. The user wears the head mounted display 30 on the head and observes the xR space displayed on the display unit 10. The head mounted display 30 has the display unit 10 and the position / posture detection unit 14 built therein. The input unit 12 detects the movement of the user's hand 32.

[0076] Figure 3 An example of an xR space viewed by a user wearing the head mounted display 30 is shown. The xR space 34 is a three-dimensional space having depth, and various objects are displayed at specific positions and in specific postures in the xR space. Figure 3In the xR space 34, an object 40 and an object 42 of the document are displayed. In addition, in the xR space 34, an object 36 of a virtual hand (virtual hand) is displayed, and an object 38 as a marker is displayed at the front end thereof. When the user moves his hand 32 in the real space, the object 36 of the virtual hand in the xR space moves in the xR space in conjunction with the movement of the actual hand. The user can move the object 36 in the xR space 34 just like his own hand. When the user changes the direction of his head in the real space, the xR space 34 also changes in conjunction with this, and the type, position, and posture of the displayed object change.

[0077] The objects 40, 42, etc. are displayed, for example, in the same size as the real objects. Furthermore, the user can grasp the objects 40, 42, etc. by operating the object 36 of the virtual hand and move the hand 32 in the real space, thereby being able to move the grasped objects 40, 42, etc. in the xR space 34. The user can also place the objects 40, 42 moving in the xR space 34 at a desired position in the xR space 34. That is, the user can place the objects 40, 42 at any position in the xR space 34, unlike in the real space, without having to worry about physical obstacles.

[0078] In this way, if objects constituting a plurality of pages of a document are widely arranged in the xR space 34 due to the high degree of freedom of operation of the objects 40 and 42, etc., it will take a lot of time to rearrange the pages according to certain rules, resulting in a decrease in work efficiency. Figure 3 In the case where there are not only object 40 and object 42 but also a large number of objects in addition to them, when moving several of these objects to the desired positions in the xR space 34, it will be time-consuming and labor-intensive to use the virtual hand object 36 to grasp and move the objects one by one. It is ideal to be able to operate the multiple objects in a collective and unified manner.

[0079] Therefore, in this embodiment, a representative object representing a plurality of objects is used.

[0080] Figure 4 Indicates documents 44 and 46 stored in the file system 24. Document 44 is a document A and includes a plurality of pages. Document 46 is another document B different from document A and also includes a plurality of pages. A document generally includes a plurality of pages, each of which has its own attributes, but may include only one page as described above. When a document is output to the xR space 34, a representative object is created, and the pages constituting the document are converted into independent objects belonging to each of the representative objects.

[0081] The representative object and the object each have position and posture information in the xR space 34. The object belonging to the representative object has relative position and posture information with respect to the coordinate system possessed by the representative object. That is, the object belonging to the representative object can be said to be an object subordinate to the representative object.

[0082] Figure 5 It shows an example of displaying document 44 and document 46 in xR space 34 respectively. Document 44 is displayed as objects of multiple pages A-1, page A-2, page A-3, page A-4, and page A-5 constituting document 44 in xR space 34 represented by XYZ three axes. Hereinafter, they are referred to as object A-1, object A-2, ..., and object A-5. And, representative object A representing these objects A-1, object A-2, ..., and object A-5 is arranged near object A-1, object A-2, ..., and object A-5. Here, the so-called "near" refers to the vicinity in the xR space, and refers to a close distance to the extent that it can be seen that the representative object A has some relationship with objects A-1 to A-5. Figure 5 In the diagram, objects A-1 to A-5 are arranged so as to face the representative object A. In other words, the representative object A is arranged so as to be located at the center of the space surrounded by objects A-1 to A-5. This arrangement is an example of the association between the representative object and the objects belonging to the representative object.

[0083] Moreover, the document 46 is displayed as objects of multiple pages B-1, page B-2, page B-3, page B-4, and page B-5 constituting the document 46 in the xR space 34 represented by the XYZ three axes. Hereinafter, they are referred to as object B-1, object B-2, ..., and object B-5. Furthermore, representative objects B representing these objects B-1, object B-2, ..., and object B-5 are arranged near objects B-1, object B-2, ..., and object B-5. The "nearby" here also refers to the vicinity in the xR space, and refers to a close distance to the extent that it can be seen that the representative object B has some association with objects B-1 to B-5.

[0084] Figure 5In the figure, representative object A and representative object B are respectively displayed as cubes, but are not limited to this. However, it is ideal that representative object A and representative object B are different in at least one of the shapes, patterns, colors, and texts of the objects subordinate to the representative objects, and the user can easily identify them as representative objects. For example, representative object A and representative object B are set as blue cubes, etc. Alternatively, representative object A and representative object B can also be set in the shape of a clip that simulates reality. Thus, the user can easily understand that representative object A and representative object B have the function of organizing multiple objects based on their shapes. Furthermore, the document name can also be displayed near representative object A and representative object B. Display the name "Document A" near representative object A, etc.

[0085] The representative object A and the representative object B may be a single object or a combination of a plurality of objects. For example, a cube is surrounded by spheres.

[0086] The operation of the representative object A and the representative object B can also be performed by inputting the object of the input unit 12, for example Figure 3 The trigger is when the object 36 of the virtual hand in the embodiment touches the representative object A or the representative object B. Alternatively, a specific operation signal from the input unit 12 may be used as a trigger when the object 36 is in contact or overlapped.

[0087] The operation contents for the representative objects A and B may be predetermined or selected based on the operation signal of the input unit 12. Furthermore, the representative objects A and B may be selected based on the positions where the representative objects A and B are touched by the object 36. Furthermore, how the representative objects A and B move may be calculated by physical simulation, and the selection may be made based on the calculation results.

[0088] In addition, representative objects may have a hierarchical structure. For example, representative object B may belong to representative object A, representative object A may function as a higher-level representative object, and representative object B may function as a lower-level representative object.

[0089] Figure 6 Indicates a case where the representative objects have a hierarchical structure. Objects A-1 to A-5 belong to the representative object A, and another representative object B belongs to the representative object A. Objects B-1 to B-3 belong to the representative object B.

[0090] At this time, the operation performed on representative object A is also performed on objects A-1 to A-5 and representative object B belonging to representative object A. As a result, objects B-1 to B-3 belonging to representative object B are similarly operated according to the operation performed on representative object B.

[0091] On the other hand, the operation performed on the representative object B is also performed on the objects B-1 to B-3, and has no influence on the objects A-1 to A-5.

[0092] Next, the data structure representing the object will be described in more detail.

[0093] Figure 7 A coordinate system representing a representative object A and an object A-1 belonging to the representative object is shown.

[0094] The xR space has a three-dimensional coordinate system of XYZ axes. All objects including the representative object have their own coordinate system, which has position and posture data (absolute coordinates) relative to the coordinate system of the xR space. The position data is maintained as a three-dimensional vector such as (x, y, z). The posture data is maintained as Euler angles such as (α, β, γ).

[0095] Furthermore, when the object has a shape, the object has the shape information.

[0096] Furthermore, the object has attribute information, such as the information of the representative object, the name representing the object, the order of objects belonging to the same representative object, etc.

[0097] The object belonging to the representative object has data (relative coordinates) of the position and posture of its own coordinate system relative to the coordinate system of the representative object. The data may be calculated and stored in advance, or the result of each calculation may be stored as necessary.

[0098] Figure 8 Specifically represents the data structure of representative object A and object A-1. For representative object A, there are position data relative to the coordinate system of the xR space, Euler angles as posture data, name as attribute information, name of the representative object to which it belongs, and order. In addition, since representative object A itself is a representative object and does not belong to other representative objects, the name of the representative object to which it belongs does not exist (invalid (null) data).

[0099] <Processing Flowchart>

[0100] Fig. 91 is a flowchart of the process of this embodiment. This is a process in which the object 36 of the input unit 12 contacts the object of the document as a trigger for the operation.

[0101] First, the position and posture of the display unit 10 and the input unit 12 are detected by the position and posture detection unit 14 ( S101 ). The position and posture detection unit 14 outputs the detected position data and posture data to the xR space calculation unit 16 .

[0102] The xR space calculation unit 16 calculates the xR space (S102) and outputs it to the display image calculation unit 20. The display image calculation unit 20 calculates and creates a display image to be displayed on the display unit 10, and outputs it to the display unit 10. For example, a display image Figure 3 As shown, the display image includes the document object 40, the object 42 and the virtual hand object 36 as the object of the input unit 12. When a document includes a plurality of pages, the objects of each page are included in the display image, and representative objects representing the plurality of pages are included in the display image.

[0103] Next, the contact determination unit 18 determines whether the object 36 of the input unit 12 has contacted the object of the document in the display image ( S103 , S104 ).

[0104] If the object has been touched (Yes in S104 ), the contact determination unit 18 then determines whether the object touched by the object 36 of the input unit 12 is a representative object ( S105 ).

[0105] If the representative object is touched (Yes in S105), the operation unit 22 calculates the position and posture of the object in order to execute the operation indicated by the user on the touched representative object and all objects belonging to the representative object, and outputs it to the display image calculation unit 20 (S106). The display image calculation unit 20 calculates and creates the display image after the operation based on the calculation result from the operation unit 22, and outputs it to the display unit 10 (S108). The operation indicated by the user includes the movement of the object, the display / non-display of the object, the alignment, the explicit indication of the object belonging to the representative object, the partial selection, the change of the representative object, etc. The specific content of the operation will be further described later.

[0106] On the other hand, if the representative object is not touched (No in S105), the calculation unit 22 calculates the position and posture of the object in order to execute the operation instructed by the user on the touched object, and outputs it to the display image calculation unit 20 (S107). The display image calculation unit 20 calculates and creates a display image after the operation based on the calculation result from the calculation unit 22, and outputs it to the display unit 10 (S108). The processing of S101 to S108 is repeatedly executed in a predetermined control cycle.

[0107] Fig.10 This is another processing flow chart of the present embodiment. This is a process that uses an operation signal from the input unit 12 as a trigger.

[0108] First, an operation signal from the input unit 12 is acquired (S201). Then, when the operation signal from the input unit 12 is acquired, the operation signal is executed. Fig. 9 The same processing as S101 to S108 in the above is performed, and the object is operated according to the user operation and output to the display unit 10 (S202 to S209).

[0109] Next, specific operations using the representative object will be described.

[0110] <Move>

[0111] Fig.11A and Fig. 11B A display example of the xR space when an operation to move the representative object is performed is shown. Fig.11A This is an example of the display before moving. Fig. 11B This is an example of the display after moving.

[0112] When the user moves the representative object A, the objects A-1 to A-5 belonging to the representative object A move while maintaining their relative positions and postures. For example, when the input unit 12 is an xR controller having buttons, the user moves the objects of the input unit 12, specifically, Figure 3 The object 36 in the image is in contact with or overlaps with the representative object A, and the button is pressed, thereby starting the movement of the representative object A and the objects A-1 to A-5 belonging to the representative object A. While the button is pressed, the movement continues, and the movement is performed according to the position and posture of the input unit 12. When the button is released, the movement ends, and the representative object A and the objects A-1 to A-5 belonging to it are arranged in the xR space at the current position and posture.

[0113] When the user's hand 32 is sensed and used as the input unit 12, a gesture may be used instead of a button pressing operation.

[0114] <Move only the representative object>

[0115] Fig. 12A and Fig. 12B A display example of the xR space when an operation of moving only the representative object is performed is shown. Fig. 12A This is an example of the display before moving. Fig. 12B This is an example of the display after moving.

[0116] This mode is a mode in which only the representative object A is moved in order to readjust the position and posture of the representative object A according to the positions of the objects A-1 to A-5 belonging to the representative object A. This mode can be applied to situations where the configuration of the page is biased, etc. In the case of this mode, the positions and postures of the objects A-1 to A-5 belonging to the representative object A are maintained, and only the position and posture of the representative object A are changed. Of course, only the position of the representative object A or only the posture of the representative object A can also be changed. The switching of modes can be performed, for example, by the user pressing a button or by a user gesture.

[0117] Specifically, when the input unit 12 is a controller having buttons, the user controls the object of the input unit 12, that is, controls the Figure 3 The button is pressed while the object 36 in the image is in contact with or overlaps with the representative object A, thereby starting the movement of the representative object A. While the button is pressed, the movement continues, and the movement is performed according to the position and posture of the input unit 12. When the button is released, the movement ends, and the representative object A is arranged in the xR space at the current position and posture.

[0118] <Switching between display and non-display>

[0119] Fig.13A and Fig. 13B An example of display in the xR space when a display / non-display operation is performed on a representative object is shown. Fig.13A For an example of display status, Fig. 13B This is an example of a display in a non-display state.

[0120] When the display / non-display switching operation is performed on the representative object A, the display / non-display switching is performed on all objects belonging to the representative object A, that is, objects A-1 to A-5.

[0121] Ideally, the representative object A notifies the user whether it is in a display state or a non-display state by changing its shape or color.

[0122] For example, when the input unit 12 is a controller having buttons, the user controls the object of the input unit 12, that is, controls the Figure 3 By pressing the button while the object 36 is in contact with or overlapped with the representative object A, the display / non-display state of the objects A-1 to A-5 belonging to the representative object A can be switched.

[0123] After the display / non-display switching mode is switched in advance by a button operation or the like, the user may bring the object of the input unit 12 into contact with the representative object A in the xR space to switch the display / non-display.

[0124] <Alignment>

[0125] Fig.14A and Fig. 14B An example of display of the xR space when the alignment operation is performed on the representative object is shown. Fig.14A This is an example of the display before alignment. Fig. 14B This is an example of the display after alignment.

[0126] When the representative object A and the representative object B are aligned, the objects A-1 to A-5 and the objects B-1 to B-5 are aligned according to the attributes of the objects.

[0127] For example, when the input unit 12 is a controller having buttons, the user controls the object of the input unit 12, that is, controls the Figure 3 The button is pressed while the object 36 is in contact with or overlapped with the representative object A, thereby moving the objects A-1 to A-5 belonging to the representative object A to the aligned state. Figure 3 The button is pressed in a state where the object 36 is in contact with or overlapped with the representative object B, thereby moving the objects B-1 to B-5 belonging to the representative object B to be aligned.

[0128] After the alignment instruction mode is switched in advance by button operation, the user may bring the object 36 of the input unit 12 into contact with the representative object A in the xR space to align them. Alternatively, an icon object for indicating alignment may be arranged near the representative object A, and the object 36 of the input unit 12 may be brought into contact with the representative object A to align them. The type of alignment may be switched according to the type of button pressed, or may be switched in sequence according to the number of times the button is pressed. Fig. 14B hereinafter indicates that the types of alignment are different between the representative object A and the representative object B.

[0129] <Explanation of structure>

[0130] Fig.15A and Fig. 15B An example of display of the xR space when an operation of explicitly indicating the object to which the representative object belongs is performed is shown. Fig.14A This is an example of the display before operation. Fig. 15B This is an example of the display after the operation.

[0131] When an operation of explicitly indicating the objects to which the representative objects A and B belong is performed, the objects belonging to the representative objects A and B are explicitly indicated.

[0132] For example, when the input unit 12 is a controller having buttons, the user presses the button while the object 36 of the input unit 12 contacts or overlaps the representative object A in the xR space, thereby explicitly indicating the objects belonging to the representative object A. The explicit display form is arbitrary, for example, the outlines of the objects belonging to the same representative object may be changed to the same color, or may be displayed using arrows or the like. Fig. 15B In the figure, objects belonging to representative object A are changed to the same color (indicated by the same hatching in the figure), and objects belonging to representative object B are displayed as arrows starting from representative object B and ending at the objects they belong to. In short, objects belonging to the representative object that is the operation target are displayed with greater emphasis than other objects.

[0133] <Partial selection>

[0134] Fig.16A and Fig. 16B A display example of the xR space when a partial selection operation is performed on the representative object is shown. Fig.16A This is an example of the display before operation. Fig. 16B This is an example of the display after a partial selection operation.

[0135] The information prompting system of the embodiment basically operates all objects belonging to the representative object by operating the representative object, but may also have a partial selection mode that operates only specific objects among the objects belonging to the representative object. The partial selection mode further has a selection mode and an operation mode, and the selection mode or the operation mode can be switched according to the button used, or can be switched in sequence by button operation.

[0136] For example, in the state switched to the selection mode, the user can select the desired object by touching the object 36 of the input unit 12. Specifically, if the object A-2, the object A-3, and the object A-5 are touched, the three objects are selected. It is desirable that the selected object is prompted by changing the outline color so that the user can recognize it.

[0137] In the state of partial selection, any object in the selected object group has the same function as the representative object. Therefore, when the object A-2 is moved, the object A-2 is used as a substitute for the representative object, and the same movement is performed on the objects A-3 and A-5.

[0138] <Changes in representative objects>

[0139] Fig.17A and Fig. 17B A display example of the xR space when a change operation of the representative object is performed is shown. Fig.17AThis is an example of the display before the change. Fig. 17B This is an example of the display after the change.

[0140] An object belonging to a certain representative object can be changed to one belonging to another representative object.

[0141] For example, the object can be changed by moving it and making it contact with another representative object different from the representative object to which it belongs. By moving object B-1 belonging to representative object B and making it contact with representative object A, object B-1 changes from an object belonging to representative object B to an object belonging to representative object A.

[0142] As described above, in this embodiment, when objects constituting multiple pages of a document are displayed in the xR space, representative objects representing these objects are displayed, and through the operation of the representative objects, unified operations can be performed on multiple objects belonging to the representative objects, thereby saving the time required to operate each object.

[0143] <Modification>

[0144] When a document includes multiple pages, objects on each page and representative objects representing the multiple objects are displayed in the xR space. However, when a document includes one page, a representative object may be created and displayed near the object on the page. Fig.17A and Fig. 17B As shown, this is particularly effective when changing an object belonging to a certain representative object so that it belongs to another representative object.

Claims

1. An information prompting system, comprising: A display device provides a virtual three-dimensional working space for the user; An input device for a user to operate a virtual hand object in the three-dimensional working space; as well as processor, The processor executes the program, thereby causing the three-dimensional work space to be displayed on the display device, causing the document to be displayed in the three-dimensional working space, A representative object representing the document and a plurality of pages constituting the document is associated with the document and displayed in the three-dimensional workspace, the representative object being produced in response to the document being output to the three-dimensional workspace, each of the plurality of pages having position and posture information relative to a coordinate system possessed by the representative object, determining whether the virtual hand object contacts the representative object in the three-dimensional work space, When the virtual hand object contacts the representative object, in the case of detecting an operation of the representative object in the three-dimensional working space performed by the virtual hand object, processing is performed on the representative object and the multiple pages belonging to the representative object. When the virtual hand object does not contact the representative object, when an operation of the contacted object in the three-dimensional working space is detected through the virtual hand object, processing is performed only on the contacted object.

2. The information prompting system according to claim 1, wherein When a movement operation of the representative object is detected by the input device, the processor performs a process of moving the representative object while maintaining relative positions of a plurality of pages constituting the document represented by the representative object.

3. The information prompting system according to claim 1, wherein When detecting a display / non-display switching operation of the representative object performed through the input device, the processor performs a process of collectively switching display / non-display of a plurality of pages constituting the document represented by the representative object.

4. The information prompting system according to claim 1, wherein When a specific operation of the representative object is detected through the input device, the processor performs a process of collectively aligning a plurality of pages constituting the document represented by the representative object.

5. The information prompting system according to any one of claims 1 to 4, wherein The representative object presents a hierarchical structure including at least a superordinate representative object and subordinate representative objects subordinate to the superordinate representative object.

6. The information prompting system according to any one of claims 1 to 4, wherein The representative objects include partial representative objects temporarily representing only a part of the pages among the plurality of pages constituting the document.

7. The information prompting system according to claim 1, wherein When a specific operation of the representative object performed through the input device is detected, the processor performs processing to move only the representative object without moving a plurality of pages constituting the document represented by the representative object.

8. The information prompting system according to claim 1, wherein When an operation of the representative object is detected through the input device, the processor performs a process of changing a display form of a plurality of pages constituting a document represented by the representative object whose operation is detected.

9. The information prompting system according to claim 8, wherein The processor performs processing for displaying a plurality of pages constituting a document represented by the representative object whose operation is detected so that a relationship with the representative object can be identified.

10. The information prompting system according to claim 1, wherein The processor causes a first representative object representing a first document and a second representative object representing a second document to be displayed as the representative objects, When an operation of associating a specific page constituting the first document with the second representative object is detected through the input device, a process of changing the representative target of the specific page from the first representative object to the second representative object is performed.

11. A computer-readable medium storing a program for causing a computer to execute a process, The process comprises the following steps: Displaying a three-dimensional work space on a display device; detecting user operations through an input device to operate a virtual hand object in the three-dimensional working space; Displaying a document in the three-dimensional workspace; Displaying a representative object representing the document and a plurality of pages constituting the document in the three-dimensional workspace in association with the document, the representative object being produced in response to the document being output to the three-dimensional workspace, each of the plurality of pages having position and posture information relative to a coordinate system of the representative object; determining whether the virtual hand object contacts the representative object in the three-dimensional work space; When the virtual hand object contacts the representative object, in the case where an operation of the representative object in the three-dimensional work space is detected through the input device, processing is performed to operate the representative object and the plurality of pages belonging to the representative object; as well as When the virtual hand object does not contact the representative object, when an operation of the contacted object in the three-dimensional working space is detected through the virtual hand object, processing is performed only on the contacted object.

12. An information prompting method, comprising the following steps: Displaying a three-dimensional work space on a display device; detecting user operations through an input device to operate a virtual hand object in the three-dimensional working space; Displaying a document in the three-dimensional workspace; Displaying a representative object representing the document and a plurality of pages constituting the document in the three-dimensional workspace in association with the document, the representative object being produced in response to the document being output to the three-dimensional workspace, each of the plurality of pages having position and posture information relative to a coordinate system of the representative object; determining whether the virtual hand object contacts the representative object in the three-dimensional work space; When the virtual hand object contacts the representative object, in the case where an operation of the representative object in the three-dimensional work space is detected through the input device, processing is performed to operate the representative object and the plurality of pages belonging to the representative object; as well as When the virtual hand object does not contact the representative object, when an operation of the contacted object in the three-dimensional working space is detected through the virtual hand object, processing is performed only on the contacted object.

13. A computer program product causing a computer to perform a process, The process comprises the following steps: Displaying a three-dimensional work space on a display device; detecting user operations through an input device to operate a virtual hand object in the three-dimensional working space; Displaying a document in the three-dimensional workspace; Displaying a representative object representing the document and a plurality of pages constituting the document in the three-dimensional workspace in association with the document, the representative object being produced in response to the document being output to the three-dimensional workspace, each of the plurality of pages having position and posture information relative to a coordinate system of the representative object; determining whether the virtual hand object contacts the representative object in the three-dimensional work space; When the virtual hand object contacts the representative object, in the case where an operation of the representative object in the three-dimensional work space is detected through the input device, processing is performed to operate the representative object and the plurality of pages belonging to the representative object; as well as When the virtual hand object does not contact the representative object, when an operation of the contacted object in the three-dimensional working space is detected through the virtual hand object, processing is performed only on the contacted object.

Images