Display device, recording medium, information processing device, system, and method

Abstract

The present application provides a display device, recording medium, information processing device, system and method, even if a plurality of objects are arranged close to each other in a virtual space, a user can easily select a desired object. An information processing device characterized by comprising a processor that performs processing of displaying a plurality of objects and an operation object that operates the objects according to input information from a user in a virtual space; extracting an object group from the plurality of objects; determining one object from the object group according to an operation in the virtual space of the operation object; and displaying the determined one object so as to be identifiable with respect to other objects other than the one object.

Description

Technical Field

[0001] This invention relates to an information processing device, a display device, an information processing system, a recording medium, and an information processing method. Background Technology

[0002] Virtual Reality (VR), a spatial augmentation technology, has long existed, which uses computers to create virtual spaces and allows users to experience actions within those spaces. Furthermore, spatial augmentation technologies also include Augmented Reality (AR), which overlays virtual spaces onto real-world spaces on displays, and Mixed Reality (MR), which combines real and virtual spaces, allowing real and virtual spaces to interact in real time.

[0003] Patent Document 1 describes a system in which, during a fixed-point operation, the peak speed of the indicator movement is correlated with the distance to the target at that point in time. The distance to the target is calculated from the peak speed, and the target position is predicted from the indicator position and direction. Thus, during a fixed-point operation, the system can predict the position of the object to which the user wants to move the indicator before the operation ends, and then perform operations such as starting the application represented by the icon in that location, thereby constructing a system that can shorten working time.

[0004] Furthermore, Patent Document 2 describes a method where, on a screen displaying computer graphics and an indicator, an operation menu is displayed near the indicator's display position based on user-defined operation input. The desired item is then commanded through this operation menu, thereby executing the operation command. Here, the operation menu is ring-shaped, with multiple item areas arranged around the indicator's display position, forming a cavity between the multiple item areas and the indicator's display position.

[0005] Furthermore, Patent Document 3 describes an information processing apparatus comprising: a first detection unit for detecting the position and posture of a user's head wearing a head-mounted display device; a first identification unit for identifying an input device for performing a task within the user's field of vision; a second identification unit for identifying an operating medium for the user to operate the input device within the user's field of vision; a second detection unit for detecting a first distance from the user's viewpoint to the input device based on the position and posture of the user's head; a third detection unit for detecting a second distance from the user's viewpoint to the operating medium based on the position and posture of the user's head; and a determination unit for determining the task to be performed based on the first and second distances.

[0006] Patent Document 1: Japanese Patent Publication No. 2005-352738

[0007] Patent Document 2: Japanese Patent Application Publication No. 2005-107963

[0008] Patent Document 3: Japanese Patent Application Publication No. 2017-059062 Summary of the Invention

[0009] When multiple objects are displayed in a virtual space, and a user wants to select one, if the objects are close together in the virtual space, the selected object may differ from the user's desired object. Therefore, to select the desired object, sometimes it is necessary to perform subtle operations such as accurately specifying the object's position.

[0010] The purpose of this invention is to provide an information processing device, etc., in which a user can easily select the desired object even when multiple objects are configured close to each other in a virtual space.

[0011] The invention described in Scheme 1 is an information processing device, characterized in that it includes a processor, which performs the following processing: displaying a plurality of objects in a virtual space and an operation object that operates the objects according to input information from a user; extracting an object group from the plurality of objects; determining an object from the object group according to the operation of the operation object in the virtual space; and displaying the determined object in a way that is recognizable relative to other objects.

[0012] The invention described in Scheme 2 is the information processing device described in Scheme 1, characterized in that the processor extracts objects existing within the range of user visual recognition as the object group.

[0013] The invention described in Scheme 3 is the information processing device described in Scheme 1, characterized in that the processor extracts the object group from the object according to the position of the operation object.

[0014] The invention described in Scheme 4 is the information processing device described in Scheme 1, characterized in that when a user selects an object from the object group, the processor sets a guidance area to guide the movement of the operation object in the virtual space.

[0015] The invention described in Scheme 5 is the information processing device described in Scheme 4, characterized in that the processor, as the guiding area, also displays an object that guides the movement of the operation object in the virtual space, namely the guiding object.

[0016] The invention described in Scheme 6 is the information processing device described in Scheme 5, characterized in that the guiding object is displayed as at least one of a line and a surface.

[0017] The invention described in Scheme 7 is the information processing device described in Scheme 5, characterized in that the guiding object is divided into a number of objects constituting the object group, each interval of the division corresponds to any one of the objects, and the processor determines the object corresponding to the interval closest to the operation object in the interval as the object, and displays it as recognizable relative to the other objects.

[0018] The invention described in Scheme 8 is the information processing device described in Scheme 1, characterized in that the processor changes the display mode of the determined object according to the relationship between the position of the object group in the virtual space and the position of the user's head.

[0019] The invention described in Scheme 9 is the information processing device described in Scheme 8, characterized in that the processor changes the display mode of the determined object according to the relationship between the position of the object group in the virtual space and the user's field of vision.

[0020] The invention described in Scheme 10 is the information processing device described in Scheme 8, characterized in that the change of the display mode is to the direction when moving and displaying the determined position of the object.

[0021] The invention described in Scheme 11 is the information processing device described in Scheme 8, characterized in that the processor changes and displays the determined posture of the object according to the posture of the objects constituting the object group in the virtual space.

[0022] The invention described in Scheme 12 is a display device that displays a plurality of objects in a virtual space and an operation object that operates the objects according to input information from the user. Based on the operation of the operation object in the virtual space, an object determined from the object group extracted from the plurality of objects is displayed in a way that is recognizable relative to other objects.

[0023] The invention described in Scheme 13 is the display device described in Scheme 12, characterized in that when the user selects an object from the object group, an object guiding the operation object to move in the virtual space, namely a guide object, is also displayed.

[0024] The invention described in Scheme 14 is the display device described in Scheme 12, characterized in that the display mode of the object is changed and displayed according to the relationship between the position of the object group in the virtual space and the position of the user's head.

[0025] The invention described in Scheme 15 is an information processing system, characterized in that it comprises: a display device for displaying a virtual space; and an information processing device for generating display information of the virtual space displayed by the display device, the information processing device comprising a processor, the processor performing the following processing: displaying a plurality of objects in the virtual space and operation objects for operating the objects according to input information from a user; extracting an object group from the plurality of objects; determining an object from the object group according to the operation of the operation object in the virtual space; and displaying the determined object in a way that is recognizable relative to other objects besides the determined object.

[0026] The invention described in Scheme 16 is a recording medium that records a program that enables a computer to perform the following functions: displaying a plurality of objects in a virtual space and an operation object that operates the objects according to input information from the user; extracting an object group from the plurality of objects; determining an object from the object group based on the operation of the operation object in the virtual space; and displaying the determined object in a way that is recognizable relative to other objects.

[0027] The invention described in Scheme 17 is an information processing method, which includes the following steps: displaying a plurality of objects and an operation object that operates the objects according to input information from the user in a virtual space; extracting an object group from the plurality of objects; determining an object from the object group according to the operation of the operation object in the virtual space; and displaying the determined object in a way that is recognizable relative to other objects.

[0028] Invention Effects

[0029] According to the first aspect of the present invention, an information processing device is provided that allows a user to easily select a desired object even when multiple objects are configured close to each other in a virtual space.

[0030] According to the second and third embodiments of the present invention, it is possible to extract a more suitable set of objects for determining an object.

[0031] According to the fourth aspect of the invention, the operation of the input device becomes easier when the user selects an object.

[0032] According to the fifth aspect of the present invention, the user can visually grasp the guidance area as a guided object.

[0033] According to the sixth aspect of the present invention, it is possible to display a guide object that is easily identifiable by the user.

[0034] According to the seventh aspect of the invention, the identification of an object becomes easier.

[0035] According to embodiments 8 to 11 of the present invention, it becomes easier for the user to identify an object.

[0036] According to the 12th aspect of the present invention, a display device is provided that allows a user to easily select a desired object even when multiple objects are close to each other in a virtual space.

[0037] According to the 13th aspect of the present invention, the user can more easily select an object.

[0038] According to the 14th aspect of the invention, it becomes easier for the user to identify an object.

[0039] According to the 15th aspect of the present invention, an information processing system is provided that allows a user to easily select the desired object even when multiple objects are configured close to each other in a virtual space.

[0040] According to the 16th aspect of the present invention, a computer enables a user to easily select the desired object even when multiple objects are configured close to each other in a virtual space.

[0041] According to the 17th aspect of the present invention, an information processing method is provided that allows a user to easily select the desired object even when multiple objects are configured close to each other in a virtual space. Attached Figure Description

[0042] The embodiments of the present invention will be described in detail with reference to the following figures.

[0043] Figure 1 This is a diagram illustrating an example of the structure of the information processing system in this embodiment;

[0044] Figure 2 middle, Figure 2 (a) to (c) are diagrams illustrating examples of when it becomes difficult to select a document object;

[0045] Figure 3 This is a block diagram illustrating an example of the functional structure of the information processing device and the display device in this embodiment;

[0046] Figure 4 It is a flowchart that describes the actions of an information processing system;

[0047] Figure 5 This is a flowchart explaining the process by which the decision unit determines the position and orientation of the input device, etc., based on the marked position;

[0048] Figure 6 middle, Figure 6 (a) to (b) indicate that in Figure 4 The graph of the first object group extracted in step 108;

[0049] Figure 7 It means in Figure 4 The graph of the second object group extracted in step 109;

[0050] Figure 8 middle, Figure 8 (a) to (e) are diagrams illustrating the method by which a determining unit determines a document object based on the movement of the operation object, moves it sequentially relative to other document objects, and displays it.

[0051] Figure 9 middle, Figure 9 (a) to (b) are diagrams representing other ways of guiding objects;

[0052] Figure 10 middle, Figure 10 (a) to (c) are diagrams showing how the movement direction of the determined document object is changed;

[0053] Figure 11 middle, Figure 11 (a) to (b) are graphs representing the distance a document object moves;

[0054] Figure 12 middle, Figure 12 (a) to (b) are diagrams showing how the pose changes when a document object is moved.

[0055] Symbol Explanation

[0056] 1-Information processing system, 10-Information processing device, 20-Display device, 20a-Display screen, 30-Input device, 40-Camera, 110-Information acquisition unit, 120-Decision unit, 130-Storage unit, 140-Confirmation unit, 150-Production unit. Detailed Implementation

[0057] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0058] <Overall Description of the Information Processing System>

[0059] Figure 1 This is a diagram illustrating an example of the structure of the information processing system 1 in this embodiment.

[0060] like Figure 1 As shown, the information processing system 1 of this embodiment includes an information processing device 10 for generating display information; a display device 20 for displaying images on a display screen 20a based on the display information; an input device 30 for inputting information; and a camera 40 for capturing images of the user's head, the display device 20, and the input device 30.

[0061] The information processing device 10 generates display information for a virtual reality space (VR space) displayed on the display screen 20a of the display device 20. Details will be described later, but various objects are displayed on the display screen 20a of the display device 20. These objects include, for example, a user head object representing a user's head, a monitor object representing the display screen 20a, an input device object representing the input device 30, a document object representing a document, and a guide object for moving the input device object. Here, "document" refers to an electronic file that digitizes text and image information. As long as the electronic file can be processed in the information processing system 1, there are no particular restrictions on its format or data structure. The objects displayed on the display screen 20a do not need to actually exist in real space. In this case, the user head object representing a user's head, the monitor object representing the display screen 20a, and the input device object representing the input device 30 exist in real space, but the document object representing a document does not need to exist in real space. Furthermore, the guide object for moving the input device object does not exist in real space.

[0062] Furthermore, the information processing device 10 calculates the position and posture of the user's head within the VR space based on information about the position and posture of the user's head detected in real space, and displays the user's head object. Similarly, the information processing device 10 calculates the position and posture of the display screen 20a within the VR space based on information about the position and posture of the display device 20 detected in real space, and displays the display object. Moreover, the information processing device 10 calculates the position and posture of the input device 30 within the VR space based on information about the position and posture of the input device 30 detected in real space, and displays the input device object. Additionally, here, "real space" refers to space in the real world, and "virtual reality space (VR space)" refers to a three-dimensional space representing a world constructed on a computer that simulates the real world.

[0063] Furthermore, the document objects can be positioned and positioned in any VR space, and can also reflect the results of the user's past operations, with the information processing device 10 calculating the position and posture.

[0064] The information processing device 10 is, for example, a so-called general-purpose personal computer (PC). Moreover, under the management of an OS (Operating System), the information processing device 10 performs information processing in this embodiment by causing various application software to operate.

[0065] The information processing device 10 includes a processing unit, namely a CPU (Central Processing Unit), a storage unit, namely main memory, and storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive). Here, the CPU is an example of a processor that executes various software programs, such as an operating system (OS) and application software. Furthermore, the main memory is a storage area for storing various software programs and the data used during their execution, while the storage area is a storage area for storing input data to and output data from various software programs.

[0066] Furthermore, the information processing device 10 has a communication interface for communicating with the outside world.

[0067] Display device 20 displays the aforementioned VR space. Display device 20 is, for example, a dual-lens head-mounted display (HMD). That is, the user wears the head-mounted display and views images displayed on a screen 20a disposed inside the head-mounted display and placed in front of the user's eyes. The screen 20a is, for example, composed of a liquid crystal display (LCD) or an organic electroluminescence display (OELD) that has the function of displaying images.

[0068] In this embodiment, the display device 20 displays a VR space for viewing and manipulating documents on a display screen 20a. Furthermore, the display screen 20a can indicate parallax, allowing the user to perceive the depth and other aspects of the VR space in three dimensions.

[0069] The input device 30 operates on objects existing in the VR space. There are no particular restrictions on the input device 30 as long as it can input information. Furthermore, the input device 30 can be a physically existing object, or it can be a user's hand or finger. When the input device 30 is a physically existing object, it can be, for example, a mouse-type, pen-type, or keyboard-type input device. When the input device 30 is a mouse-type input device capable of detecting three-dimensional position, cursor movement and button selection are performed by moving the device within space. The figure shows a user using a mouse-type input device as the input device 30. Furthermore, when the input device 30 is a pen-type input device, the user can hold it and operate it like a pen to write. It is pen-shaped and can input text and graphics. The pen-type input device can be a stylus, or a ballpoint pen, pencil, or fountain pen that can actually take notes. Moreover, the keyboard-type input device has multiple keys for information input by pressing them. Therefore, the keyboard-type input device can be an actual keyboard.

[0070] When the input device 30 is a user's hand or finger, such as Microsoft Corporation's Kinect (registered trademark), it takes a picture using a camera 40 equipped with an RGB camera and a depth sensor. Furthermore, the information processing device 10 can obtain the three-dimensional coordinates of the user's hand or finger by analyzing the movements of the user's hand or finger.

[0071] Furthermore, there can be one or multiple input devices 30. The input devices 30 can be configured to operate not only objects existing in the VR space by sending operation signals, but also to determine the start and end of a mode. The operation signals can be sent via physical buttons or by recognizing hand or finger gestures.

[0072] The camera 40, for example, has an optical system that converges incident light and an image sensor that detects the light converged through the optical system.

[0073] An optical system is composed of a single lens or a combination of multiple lenses. Optical systems eliminate various aberrations through lens combinations and coatings applied to the lens surfaces. Image sensors are constructed by arranging imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor).

[0074] Camera 40 captures images of the user's head, display device 20, and input device 30. Camera 40 may also include, for example, an RGB camera and a depth sensor, thereby sensing the captured object and performing three-dimensional measurements. The information processing device 10 can then calculate the position and posture of the user's head, display device 20, and input device 30. There are no particular limitations on this camera 40; commercially available cameras can be used. For example, Leap Motion (registered trademark) from Leap Motion Corporation, RealSense (registered trademark) from Intel Corporation, and Kinect from Microsoft Corporation can be used.

[0075] Furthermore, markers can be placed on the user's head, display device 20, and input device 30, and their respective positions and postures can be determined based on photographic images of these markers. The "marker" can be any identifier. More specifically, the "marker" is, for example, a marker with LEDs or other light-emitting devices arranged in a defined three-dimensional shape. It can also be a one-dimensional barcode or a two-dimensional barcode, or a marker printed on a flat surface. The installation position of the marker is preset and held by the information processing device 10. The marker has a unique ID number that can distinguish between the user's head, display device 20, and input device 30.

[0076] The information processing device 10 and the display device 20 are connected, for example, via DVI (Digital Visual Interface). Alternatively, instead of DVI, they can be connected via HDMI (High-Definition Multimedia Interface) or DisplayPort.

[0077] Furthermore, the information processing device 10 and the input device 30 are connected, for example, via USB (Universal Serial Bus). Alternatively, instead of USB, they can be connected via IEEE 1394 or RS-232C. However, they are not limited to these; wireless connections such as wireless LAN (Local Area Network) and Bluetooth (registered trademark) can also be used.

[0078] Furthermore, in the illustrated example, the information processing device 10 and the camera 40 are connected via a wired connection, such as via USB, IEEE 1394, or RS-232C. Thus, image information from images captured by the camera 40 is transmitted to the information processing device 10 via a wired connection. However, this is not a limitation; a wireless connection as described above is also possible.

[0079] In this information processing system 1, the user observes the display screen 20a while using the input device 30 to manipulate document objects within the VR space, thereby processing the document. Document processing includes, for example, document creation, document selection, document viewing, document correction, and document deletion.

[0080] However, when multiple document objects are located close to each other in VR space through overlapping or other means, document selection can sometimes become difficult.

[0081] Figure 2 Figures (a) through (c) are examples of situations where selecting a document object becomes difficult.

[0082] in, Figure 2 (a) shows the state before the document object is selected.

[0083] In the example diagram, document objects Od1 to Od5 are displayed on screen 20a. The user's hand is displayed as input device object On. Furthermore, an operation object Os is displayed within input device object On. The operation object Os represents the part of the input device object On where the object is manipulated. This operation object Os occupies a certain area in VR space. In this case, the operation object Os can also be considered an indicator. Therefore, for example, when selecting document object Od5, the user aligns the operation object Os with document object Od5 and performs a pre-set operation using input device 30. When input device 30 is a mouse-type input device, this operation is, for example, a click. And when input device 30 is the user's hand or finger, this operation is a pre-set gesture.

[0084] One example of displaying multiple objects in the VR space is document objects Od1 to Od5. Furthermore, an operation object Os is displayed in the VR space to manipulate these objects based on user input. "Input information" refers to the information input into the information processing device 10 as a user command when the aforementioned clicks, gestures, or other operations are performed.

[0085] Moreover, users from Figure 2 The state of (a), such as Figure 2 As shown in (b), the input device object On is brought close to the document object Od5, and the document object Od5 is selected. However, at this time, as Figure 2 As shown in (c), sometimes a nearby document object Od4 is selected. That is, even if the input device 30 is used to select one of a plurality of document objects Od1 to Od5, due to the slight positional deviation of the operation object Os in VR space, other nearby document objects may sometimes be selected. Furthermore, since the pages constituting the document are located close to each other in the same document, the same situation may occur when selecting a page.

[0086] Therefore, in this embodiment, the information processing system 1 is configured as follows to suppress this problem.

[0087] <Description of Information Processing Device 10 and Display Device 20>

[0088] Figure 3 This is a block diagram illustrating a functional structure example of the information processing device 10 and the display device 20 in this embodiment. Additionally, in Figure 3 The diagram shows the parts of the various functions of the information processing device 10 and the display device 20 that are relevant to this embodiment.

[0089] like Figure 3As shown, the information processing apparatus 10 of this embodiment includes: an information acquisition unit 110, which acquires three-dimensional measurement results and image information from a camera 40; a decision unit 120, which determines the position and posture of the user's head, display device 20, and input device 30 in real space; a storage unit 130, which stores information about the object, such as the size and shape of the user's head, display device 20, and input device 30; a determination unit 140, which determines a document object; and a production unit 150, which produces display information displayed on the display screen 20a of the display device 20.

[0090] The information acquisition unit 110 acquires the three-dimensional measurement results and image information of the camera 40 that captures the user's head, the display device 20 and the input device 30.

[0091] In this case, such as Figure 1 As shown, camera 40 captures images of the user's head, display device 20, and input device 30 in the real space. Furthermore, the results of these three-dimensional measurements and the information from the captured images are transmitted from camera 40 to information processing device 10 and acquired by information acquisition unit 110. In this case, the image information is the image information of a moving image, but it could also be, for example, the image information of a plurality of still images captured at preset short time intervals, such as every 40 milliseconds.

[0092] Furthermore, the information acquisition unit 110 acquires input information from the input device 30.

[0093] The decision unit 120 determines the position and posture of the user's head, display device 20, and input device 30. At this time, the decision unit 120 identifies the user's head, display device 20, and input device 30 from the three-dimensional measurement results and determines their position and posture. Furthermore, when markers are used, their position and posture are determined based on the image information of the markers.

[0094] The storage unit 130 retains the user's head, the ID numbers of the display device 20 and the input device 30, as well as their size and shape in real space. Regarding size and shape, the storage unit 130 can, for example, retain the shape as a 3D model represented in three-dimensional coordinates or a formula using those coordinates. Furthermore, when a marker is used, the storage unit 130 retains the marker's mounting position.

[0095] Furthermore, the storage unit 130 stores information about the object displayed in the VR space. This information includes, for example, the object's size and shape, which can be represented using three-dimensional coordinates or a formula using those coordinates. The information may also include attribute information indicating the object's characteristics. This characteristic information could be information about the surface texture, weight, hardness, etc. Additionally, when the object is a document object (Od), information about the document content is stored. That is, the text and image information contained in the document are stored.

[0096] The determining unit 140 determines a document object Od based on the operations performed on the operation object Os within the VR space. Details will be described later, but the determination is made by distinguishing a document object Od from other document objects Od based on the position of the operation object Os within the VR space.

[0097] The production unit 150 configures the 3D models of the user's head, display device 20, and input device 30 as user head object, display object, and input device object On in the VR space, and creates display information. At this time, in the VR space, it is preferable, for example, to configure each object in the same position and posture as in real space. Furthermore, it is preferable, for example, to configure each object in the same size and shape as in real space. However, this is not a limitation, and the size and shape can be changed. Also, regarding color, it is not necessarily required that the VR space be identical to the real space.

[0098] Furthermore, the production unit 150 configures document objects (Od) and other objects in the VR space and creates display information. At this time, the content of the document is displayed on the display surface of the document object Od. Details will be described later, but when a user wants to select a document object Od, the production unit 150 creates display information that can be recognized by displaying a document object Od relative to other document objects Od.

[0099] And, as Figure 3 As shown, the display device 20 includes a display information acquisition unit 210 for acquiring display information and an image display unit 220 for displaying images based on the display information.

[0100] The display information acquisition unit 210 acquires the image information produced by the production unit 150. Furthermore, the image display unit 220 displays an image based on the image information produced by the production unit 150. The image display unit 220 is, for example, the display screen 20a described above.

[0101] Therefore, the image displayed by the display device 20 is the aforementioned VR space, and the image display unit 220 displays the aforementioned object in the VR space.

[0102] <Description of the operation of Information Processing System 1>

[0103] Next, the operation of information processing system 1 will be explained.

[0104] Figure 4 This is a flowchart illustrating the actions of information processing system 1.

[0105] First, camera 40 captures images of the user's head, display device 20, and input device 30 (step 101). Then, the 3D measurement results and the captured image information are transmitted to information processing device 10.

[0106] The three-dimensional measurement results and image information are acquired by the information acquisition unit 110 of the information processing device 10 (step 102).

[0107] Next, the decision unit 120 determines the position and posture of the user's head, display device 20 and input device 30 in the real space based on the three-dimensional measurement results and image information acquired by the information acquisition unit 110 (step 103).

[0108] Figure 5 This is a flowchart explaining the process by which the decision unit 120 determines the position and orientation of the input device 30, etc., based on the position of the marker. Figure 5 Yes Figure 4 The process of step 103 is illustrated in more detail with a diagram.

[0109] Here, firstly, the markers are extracted from the image (step 201). At this point, if there are multiple markers, the image of each marker is extracted.

[0110] Then, obtain the ID number from the extracted tag (step 202).

[0111] Furthermore, the position and orientation of the input device 30, etc., are determined based on the size and deformation state of the markers in the captured image (step 203).

[0112] Then, update the position and orientation information of the input device 30, etc., corresponding to the ID number (step 204).

[0113] return Figure 4 The production unit 150 produces display information displayed in the VR space based on the position and posture determined by the decision unit 120 and the information of the object stored in the storage unit 130 (step 104).

[0114] Furthermore, the production unit 150 sends the produced display information to the display device 20, and the display information acquisition unit 210 of the display device 20 acquires it (step 105).

[0115] Furthermore, the image display unit 220 displays an image on the display screen 20a based on the display information (step 106). Thus, various objects such as the user head object, the display object, the input device object On, and the document object Od are displayed in the VR space.

[0116] Next, the determining unit 140 determines, based on the user's operation, whether a document selection mode has been entered (step 107). The document selection mode is used when selecting one document object Od from a plurality of nearby document objects Od. This mode can be entered by the user performing a pre-set operation. This operation can be, for example, a button operation or a gesture operation. Whether the document selection mode has been entered is preferably explicitly indicated to the user. For example, by changing the color and shape of the input device object On and the operation object Os, or by displaying text and icons indicating that the mode has been entered near these objects. The document selection mode ends when the user selects a document. Alternatively, it can be set to end when the user performs a pre-set operation to end the mode. This operation can be, for example, a button operation or a gesture operation.

[0117] As a result, if it is not in the near document selection mode ("No" in step 107), return to step 101.

[0118] Conversely, when in the near-document selection mode ("Yes" in step 107), the determining unit 140 extracts object groups from a plurality of document objects Od in the VR space. An "object group" is a collection of objects, here referred to as the first object group and the second object group.

[0119] In this case, firstly, the determining unit 140 extracts the first object group G1 from the plurality of document objects Od in the VR space (step 108).

[0120] Furthermore, the determining unit 140 extracts the second object group G2 from the first object group G1 based on the position of the operation object Os (step 109).

[0121] Figure 6 (a) to (b) indicate that in Figure 4 The graph of the first object group G1 extracted in step 108.

[0122] The determining unit 140 extracts document objects Od that exist within the user's visual recognition range R1 as the first object group G1. The determining unit 140 calculates the user's visual recognition range R1 based on the position and posture of the user's head T. Furthermore, the determining unit 140 selects the document objects Od contained within this range R1 as the first object group G1.

[0123] The range R1 can be set to all areas displayed through the display screen 20a. Figure 6(a) indicates the case where all areas displayed by the display screen 20a are set as the range R1 for user visual recognition. Furthermore, as document objects Od within this range R1, the case where document objects Od1 to Od12 are extracted as the first object group G1 is shown.

[0124] Furthermore, the range R1 can also be set to a portion of the area displayed on the display screen 20a. Figure 6 (b) indicates the case where a portion of the area displayed on display screen 20a is set as the range R1 for user visual recognition. Furthermore, it shows the case where document objects Od5 to Od8 within this range R1 are extracted as the first object group G1.

[0125] Figure 7 It means in Figure 4 The graph of the second object group G2 extracted in step 109.

[0126] Here, the determining unit 140 is shown from Figure 6 The state of (a) selects the second object group G2. The determination unit 140 extracts the second object group G2 from the document object Od according to the position of the operation object Os.

[0127] Here, we illustrate the case where document objects Od5 to Od8 from the first object group G1 (i.e., document objects Od1 to Od12) are selected as the second object group G2. That is, in this case, document objects Od5 to Od8 located closer to the operation object Os are selected as the second object group G2. In this case, the proximity is determined based on the Euclidean distance in VR space. Specifically, in this case, document object Od with the smaller Euclidean distance from the operation object Os in VR space is selected as the second object group G2.

[0128] return Figure 4 Next, the determining unit 140 determines a document object Od from the second object group G2 based on the operation within the VR space of the operation object Os (step 110). Specifically, the determining unit 140 determines the nearest document object Od from the second object group G2 based on the movement of the operation object Os.

[0129] Furthermore, the production unit 150 displays the determined document object Od in a way that is recognizable relative to other document objects Od besides that document object Od (step 111). Specifically, the production unit 150 creates display information in which other document objects Od besides that document object Od sequentially move and display the document object Od determined by the determination unit 140.

[0130] Figure 8(a) to (e) are diagrams illustrating the method by which the determining unit 140 determines a document object Od based on the movement of the operation object Os, and moves and displays it sequentially relative to other document objects Od.

[0131] in, Figure 8 (a) is a diagram representing the state of a document object Od before the movement is determined. If the second object group G2 is extracted, the production unit 150 displays a guide object Oa near the second object group G2, which is the guide object Oa that guides the movement of the operation object Os in VR space. This guide object Oa is an example of a guide area that guides the movement of the operation object Os in VR space when the user selects a document object Od from the second object group G2. The "guide area" is the area displayed in VR space to guide the movement of the operation object Os when the user selects a document object Od.

[0132] The guiding object Oa is displayed as at least one of a line and a surface. Figure 8 Example (a) shows an example of using straight lines to display the guiding object Oa. This guiding object Oa consists of a dotted arrow Oa1 and a dividing line Oa2. The guiding object Oa is divided by the dividing line Oa2 into the number of document objects Od that make up the second object group G2, with each interval corresponding to any one of these document objects Od. In this case, the document objects Od that make up the second object group G2 are document objects Od5 to Od8. Therefore, the guiding object Oa is divided into four intervals. Figure 8 In (a), the four intervals are illustrated as intervals (1) to (4). Moreover, intervals (1) to (4) correspond to document objects Od5 to Od8, respectively. In addition, (1) to (4) can be displayed as part of the guide object Oa, but it is not necessary to display them. In addition, when they are displayed, the display of (1) to (4) can be understood as part of the guide object Oa.

[0133] Here, the user operates the input device 30 by moving the operation object Os along the guide object Oa. Figure 8 In (a), the trajectory K is used to illustrate the movement of the manipulated object Os.

[0134] Furthermore, as the operation object Os moves, the determining unit 140 determines a document object Od, and the creating unit 150 creates, as shown in the figure. Figure 8 The information shown in (b) to (e) is displayed on display screen 20a.

[0135] Specifically, the determining unit 140 determines the document object Od corresponding to the interval closest to the operation object Os within the set interval as a document object Od. Furthermore, the creating unit 150 moves this document object Od sequentially relative to the other document objects Od.

[0136] For example, when the operation object Os is located Figure 8 When the position shown in (b) is reached, the interval of the guide object Oa closest to the operation object Os is interval (1). At this time, the determination unit 140 determines the document object Od5 corresponding to interval (1) as a document object Od. Moreover, the production unit 150 produces display information that moves and displays the document object Od5 relative to the other document objects Od6 to Od8. In this case, the document object Od5 moves upward relative to the other document objects Od6 to Od8.

[0137] Furthermore, when the operation object Os is located Figure 8 When the position shown in (c) is reached, the interval of the guide object Oa closest to the operation object Os is interval (2). At this time, the determination unit 140 determines the document object Od6 corresponding to interval (2) as a document object Od. Moreover, the production unit 150 produces display information that moves the document object Od6 upward relative to the other document objects Od5, Od7, and Od8. In addition, at this time, the document object Od5 returns to its original position. Figure 8 The original position is shown in (a). That is, the document object Od5 is temporarily moved as the operation object Os moves, and returns to the original position and is displayed in the interval outside the interval (1).

[0138] Similarly, when the operation object Os is located Figure 8 When the positions shown in (d) to (e) are reached, the intervals of the guide object Oa closest to the operation object Os are intervals (3) and (4), respectively. At this time, the determination unit 140 determines the document objects Od7 and Od8 corresponding to intervals (3) and (4) as a single document object Od. Furthermore, the production unit 150 produces display information that moves document objects Od7 and Od8 upwards relative to other document objects Od. Additionally, at this time, the document objects Od other than the moved document object Od return to the previous position. Figure 8 The original position is shown in (a).

[0139] Furthermore, the order in which document objects (Od) are moved is determined based on their positions in VR space. Also, when multiple document objects (Od) have the same position and pose in VR space, the order is determined based on attributes such as their page number.

[0140] Figure 9 (a) to (b) are diagrams representing other ways of guiding object Oa.

[0141] in, Figure 9Figure (a) shows the case where a curve is used and displayed as the guiding object Oa. This guiding object Oa consists of curve Oa3 and dividing line Oa2. Furthermore, with... Figure 8 Similarly, in case (a), the guiding object Oa is divided into four intervals, and each interval (1) to (4) corresponds to the document objects Od5 to Od8 respectively.

[0142] and, Figure 9 (b) is a diagram showing the use of a face as a guide object Oa. This guide object Oa consists of face Oa4 and dividing line Oa2. Furthermore, with... Figure 8 Similarly, in case (a), the guiding object Oa is divided into four intervals. Figure 9 In (b), the four intervals are illustrated as intervals (1) to (4). Furthermore, with... Figure 8 Similarly, in case (a), the guiding object Oa is divided into four intervals, and each interval (1) to (4) corresponds to the document objects Od5 to Od8 respectively.

[0143] Return again Figure 4 Next, the determining unit 140 determines whether the user has selected the moved document object Od (step 112). The selection of document object Od can be performed by the user using the input device 30 to perform a preset operation. When the input device 30 is a mouse-type input device, this operation is, for example, a click. Furthermore, when the input device 30 is the user's hand or finger, this operation is a preset gesture. That is, as described above, if this operation is performed when any of the document objects Od5 to Od8 has been moved, the determining unit 140 determines that the user has selected the moved document object Od.

[0144] As a result, when document object Od is selected ("Yes" in step 112), the determination unit 140 ends the approach document selection mode (step 113). Furthermore, pre-defined processing is performed on the selected document object Od (step 114). This processing includes, for example, enlarging the display of the selected document object Od and displaying the editing screen.

[0145] Conversely, when document object Od is not selected ("No" in step 112), the determining unit 140 determines whether the end of the near-document selection mode has been selected (step 115). The end of the near-document selection mode can be performed by the user using the input device 30 to perform a preset operation. Furthermore, when the input device 30 is the user's hand or finger, it can be performed by a preset gesture.

[0146] As a result, when the end of the proximity document selection mode is selected ("Yes" in step 115), the proximity document selection mode ends and a series of processes conclude.

[0147] Conversely, if the end of the document selection mode is not selected ("No" in step 115), return to step 110.

[0148] Furthermore, in the example above, as the manipulated object Os moves, document objects Od5 to Od8 are temporarily moved and displayed, but this is not a limitation. For example, the designated document object Od can be highlighted, or its color and shape can be temporarily changed. Additionally, actions such as rotating the document object Od can also be performed.

[0149] Furthermore, the production unit 150 can also be configured to change the display mode of a determined document object Od. Specifically, changing the display mode could be, for example, changing the direction in which the determined document object Od is moved and displayed.

[0150] Figure 10 (a) to (c) are diagrams showing how the movement direction of the determined document object Od is changed.

[0151] in, Figure 10 Figures (a) to (b) illustrate how the movement direction of a document object Od is changed based on the relationship between the position of the second object group G2 in VR space and the position of the user's head T.

[0152] Figure 10 Figure (a) shows the case where, in VR space, the center position of the second object group G2 in the vertical direction is lower than the position of the user's head T. Furthermore, it shows the direction of movement of the identified document object Od, i.e., document object Od6. In this case, as shown in the figure, the user can more easily identify document object Od6 when it moves upwards.

[0153] and, Figure 10 Figure (b) shows the case where the center position of the second object group G2 in the vertical direction within the VR space is located higher than the position of the user's head T. Furthermore, it shows the direction of movement of the identified document object Od, i.e., document object Od6. In this case, as shown in the figure, the user can more easily identify document object Od6 when it moves downwards.

[0154] and, Figure 10 (c) illustrates a case where other document objects Od, namely document objects Od21 to Od22, exist around document objects Od5 to Od8. In this case, the identified document object Od, namely document object Od6, preferably moves in a direction that avoids document objects Od21 to Od22. In the example shown, document object Od6 moves in a direction other than the direction in which document objects Od21 to Od22 exist.

[0155] Furthermore, the distance that the document object Od moves is preferably a distance that is easily identifiable by the user.

[0156] That is, if the distance is too small, the user will find it difficult to recognize the movement of the document object Od. Conversely, if the distance is too large, the document object Od will move out of the user's field of vision, again making it difficult for the user to recognize the movement of the document object Od.

[0157] Figure 11 (a) to (b) are graphs representing the distance traveled by the document object Od.

[0158] in, Figure 11 (a) is a diagram showing methods to prevent the document object Od from moving too far.

[0159] Here, in VR space, a cuboid region R2 is defined, containing document objects Od5 to Od8 belonging to the second object group G2. Furthermore, a case is shown where document object Od6 moves upwards beyond the boundaries of region R2, with a distance that does not overlap with region R2. In this case, document object Od6 does not overlap with other document objects Od5, Od7, and Od8, making it easy for the user to recognize the movement of document object Od6.

[0160] and, Figure 11 (b) is a diagram showing methods to prevent the document object Od from moving too far.

[0161] Here, we illustrate the case where a document object Od6 is moved in VR space towards the user's head T. The distance moved is set to directly in front of the user's head T. In this case, the document object Od6 moves towards the user, thus preventing it from moving out of the user's field of vision. In this scenario, the user can easily recognize the movement of the document object Od6.

[0162] In this case, it can also be said that the production unit 150 changes the display mode of a document object Od based on the relationship between the position of the second object group G2 in VR space and the user's field of vision.

[0163] Furthermore, the pose of the document object Od when it moves can be changed in a way that is easier for users to recognize.

[0164] Figure 12 (a) to (b) are diagrams showing how the pose of the document object Od changes when it moves.

[0165] Here, Figure 12(a) is a diagram showing the case where the document object Od moves without changing its orientation. In this case, the document object Od7 is moving, but the display face of the moved document object Od will not face the user, making it difficult for the user to recognize the display face.

[0166] on the contrary, Figure 12 (b) is a diagram illustrating the case where the document object Od changes its pose when it moves. In this case, the display face of the moved document object Od7 changes its pose so that it faces the user, making it easier for the user to identify. Furthermore, the change in pose is temporary; when the document object Od7 returns to its original position, the pose is restored to its original pose.

[0167] In this case, it can also be said that the production unit 150 changes and displays the determined pose of a document object Od based on the pose of the document object Od constituting the second object group G2 in VR space.

[0168] According to the information processing system 1 described above, even if multiple document objects Od are configured close to each other in VR space, the user can easily select the desired document object Od.

[0169] Furthermore, the display device 20 that displays the display information generated by the information processing device 10 described above can be understood as an example of a display device that displays a plurality of document objects Od and an operation object Os that operates the document objects Od according to input information from the user in VR space. Based on the operation of the operation object Os in VR space, a document object Od determined from the object group extracted from the plurality of document objects Od is displayed in a way that is recognizable relative to other document objects Od besides the document object Od.

[0170] In this case, when the user selects a document object Od from the object group, the display device 20 also displays the object that guides the operation object Os to move in the VR space, namely the guide object Oa.

[0171] Furthermore, in the display device 20, the display mode of a document object Od can be changed and displayed according to the relationship between the position of the object group in VR space and the position of the user's head T.

[0172] In addition, in the above-described method, after extracting the first object group G1, the second object group G2 is also extracted, but it is also possible to extract the second object group G2 without extracting the first object group G1.

[0173] Furthermore, the above explanation describes the case where the document object Od is selected as the object, but it is not limited to this. As long as the object is displayed in VR space, it can be applied without any special restrictions.

[0174] Furthermore, in the above-described method, the guiding object Oa is set by lines and surfaces, but it can also be set by space or set as a three-dimensional object.

[0175] Furthermore, in the manner described above, the virtual space is displayed through the display device 20, that is, a display using VR (Virtual Reality) is performed, but it is not limited to this, AR (Augmented Reality) or MR (Mixed Reality) can also be used.

[0176] <Program Description>

[0177] Here, the processing performed by the information processing device 10 in this embodiment described above is prepared, for example, by a program such as application software.

[0178] Therefore, the processing performed by the information processing device 10 in this embodiment can be understood as a program for enabling the computer to perform the following functions: displaying a plurality of document objects Od and an operation object Os that operates the document objects Od according to input information from the user in the VR space; extracting an object group from the plurality of document objects Od; determining a document object Od from the object group according to the operation of the operation object Os in the VR space; and displaying the determined document object Od in a way that it can be recognized relative to other document objects Od besides the document object Od.

[0179] In addition, the program for implementing this embodiment can be provided by the communication unit, or it can be stored in a recording medium such as a CD-ROM and then provided.

[0180] The above description of this embodiment is not limited to the scope described in the above embodiments. As clearly understood from the claims, various modifications or improvements to the above embodiments also fall within the scope of this invention.

[0181] The embodiments of the present invention described above are provided for illustrative purposes. Furthermore, these embodiments do not encompass the entirety of the invention, nor do they limit the invention to the disclosed methods. It will be apparent to those skilled in the art that various modifications and variations will be readily understood. These embodiments were chosen and described to most readily explain the principles and applications of the invention. Thus, those skilled in the art can understand the invention through various modifications that are assumed to be optimized for specific uses of various embodiments. The scope of the invention is defined by the foregoing claims and their equivalents.

Claims

1. An information processing device, characterized in that, Equipped with a processor The processor performs the following processing: Displaying multiple objects in a virtual space and operating objects that manipulate the objects based on input information from the user; Extract object groups from the plurality of objects; Based on the operation within the virtual space of the operation object, determine an object from the object group; Display the identified object in a way that makes it recognizable relative to other objects besides the identified object; and Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

2. The information processing device according to claim 1, characterized in that, The processor extracts objects that exist within the range of the user's visual recognition as the object group.

3. The information processing device according to claim 1, characterized in that, The processor extracts the object group from the object based on the location of the object being operated on.

4. The information processing apparatus according to claim 1, characterized in that, When a user selects an object from the object group, the processor sets a guidance area to guide the movement of the object in the virtual space.

5. The information processing apparatus according to claim 4, characterized in that, The processor, as the guiding area, also displays the object that guides the movement of the operation object in the virtual space, namely the guiding object.

6. The information processing apparatus according to claim 5, characterized in that, The guide object is displayed as at least one of a line and a surface.

7. The information processing apparatus according to claim 5, characterized in that, The guiding object is divided into the number of objects constituting the object group, and each interval of the division corresponds to any one of the objects. The processor identifies the object corresponding to the interval closest to the operation object within the interval as the object and displays it as recognizable relative to the other objects.

8. The information processing apparatus according to claim 1, characterized in that, The processor changes the display mode of a determined object based on the relationship between the position of the object group in the virtual space and the user's field of vision.

9. The information processing apparatus according to claim 1, characterized in that, The processor changes and displays the determined pose of an object based on the pose of the objects constituting the object group in the virtual space.

10. A display device, wherein, Displaying multiple objects in a virtual space and operating objects that manipulate these objects based on user input. Based on the operations performed within the virtual space of the operation object, an object determined from the object group extracted from the plurality of objects will be displayed in a way that is recognizable relative to other objects besides that object; and Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

11. The display device according to claim 10, characterized in that, When a user selects an object from the object group, a guide object is also displayed to guide the movement of the object in the virtual space.

12. An information processing system, characterized in that, have: Display device, displaying virtual space; and An information processing device that generates display information for a virtual space displayed via the display device. The information processing device includes a processor. The processor performs the following processing: Displaying multiple objects in a virtual space and operating objects that manipulate the objects based on input information from the user; Extract object groups from the plurality of objects; Based on the operation within the virtual space of the operation object, determine an object from the object group; Display the identified object in a way that makes it recognizable relative to other objects besides the identified object; and Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

13. A recording medium having a program that enables a computer to perform the following functions: Displaying multiple objects in a virtual space and operating objects that manipulate the objects based on input information from the user; Extract object groups from the plurality of objects; Based on the operation within the virtual space of the operation object, determine an object from the object group; Display the identified object in a way that makes it recognizable relative to other objects besides the identified object; and Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

14. An information processing method, comprising the following steps: Displaying multiple objects in a virtual space and operating objects that manipulate the objects based on input information from the user; Extract object groups from the plurality of objects; Based on the operation within the virtual space of the operation object, determine an object from the object group; Display the identified object in a way that is recognizable relative to other objects besides the identified object; Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

15. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, performs the following processing: Displaying multiple objects in a virtual space and operating objects that manipulate the objects based on input information from the user; Extract object groups from the plurality of objects; Based on the operation within the virtual space of the operation object, determine an object from the object group; Display the identified object in a way that makes it recognizable relative to other objects besides the identified object; and Based on the relationship between the position of the object group in the virtual space and the position of the user's head, the display mode of the determined object is changed to the direction in which the determined object is moved and displayed at its position.

Images