Information processing device, information processing method, and program

Abstract

An information processing device according to one embodiment of the present technology comprises an acquisition unit and a display control unit. The acquisition unit acquires operation-body height information with respect to a display surface of a display device. The display control unit controls, on the basis of the height information, the display on the display device of a structural object representing the structure of a display target which is in accordance with a display level corresponding to the height information.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program used for image display.

[0002] Patent Document 1 describes a touch panel input device provided on a display, which makes it possible to improve operability of the touch panel by setting the sensitivity of the touch panel for each application program being executed.

[0003] JP 2013-196564 A

[0004] For example, as disclosed in Patent Literature 1, a display equipped with a touch device allows direct manipulation of objects such as displayed images. Technology that enables manipulation of objects on a display is expected to be applied in various fields such as design, education, medicine, and amusement, and there is a demand for technology that can improve the user experience when manipulating objects.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that can improve the user experience when operating an object.

[0006] In order to achieve the above object, an information processing device according to an embodiment of the present technology includes an acquisition unit and a display control unit. The acquisition unit acquires height information of an operating object relative to a display surface of a display device. The display control unit controls, based on the height information, display of a structure object on the display device that represents a structure to be displayed according to a display level corresponding to the height information.

[0007] In this information processing device, height information of the operating body relative to the display surface of the display device is used to control the display of a structural object representing the structure of the display target according to the display level corresponding to the height information. This makes it possible to change the display content of the structural object by changing the height of the operating body, for example, and improve the user experience when operating the object.

[0008] The structure object may include a plurality of image objects representing a structure of the display target according to the display level. In this case, the display control unit may determine a display object to be displayed on the display device from the plurality of image objects based on the height information.

[0009] At least some of the plurality of graphical objects may be 3D objects.

[0010] The structural object may be an object in which the plurality of image objects are layered, and in this case, the display level may be a hierarchical level of the structural object.

[0011] The display control unit may determine a layer among the layers of the structure object that corresponds to the height information as a display layer, and set the image object set in the display layer as the display object.

[0012] The display control unit may determine the display layer by threshold determination that compares the height information with one or more layer thresholds associated with the layer of the structure object.

[0013] The display control unit may display a first UI indicating the display hierarchy.

[0014] The display control unit may change at least one of a size and a color of the first UI depending on the depth of the display hierarchy.

[0015] The acquisition unit may acquire spatial position information including the height information of the operating object and in-plane position information of the operating object in an in-plane direction perpendicular to the height direction indicated by the height information, and detect a gesture operation by the operating object based on the spatial position information. In this case, the display control unit may control display of the display object on the display device in accordance with the gesture operation.

[0016] The gesture operation may include a flick operation performed by the operating object while the operating object is not in contact with the display surface. The display control unit may rotate the display object in response to the flick operation.

[0017] The display control unit may display a second UI indicating an acceptable operation that can be accepted as the gesture operation performed by the operating object on the display object.

[0018] The structure object may be an object in which the plurality of image objects are arranged in a layered structure, in which case the acceptable operation may be set for each layer of the structure object.

[0019] The display control unit may display the second UI by superimposing it on a portion of the display object where the acceptable operation can be performed.

[0020] The display control unit may change at least one of a size and a color of the second UI in accordance with the height information.

[0021] The display device may be a touch device equipped with a touch sensor. In this case, the acquisition unit may acquire the height information based on a detection result of the operation object by the touch sensor.

[0022] The display device may be an autostereoscopic display.

[0023] According to an embodiment of the present technology, there is provided an information processing method executed by a computer system, the information processing method including acquiring height information of an operating object relative to a display surface of a display device, and controlling, based on the height information, display of a structural object on the display device that represents a structure of a display target according to a display level corresponding to the height information.

[0024] A program according to an embodiment of the present technology causes a computer system to execute the following steps: acquiring height information of an operating body relative to a display surface of a display device, and controlling, based on the height information, display on the display device of a structure object representing a structure of a display target according to a display level corresponding to the height information.

[0025] 1 is a schematic diagram showing a configuration example of an image display device according to an embodiment of the present technology; FIG. 2 is a block diagram showing a configuration example of an image display device; FIG. 3 is a schematic diagram for explaining a structure object; FIG. 4 is a flowchart showing an example of an operation of the image display device; FIG. 5 is a schematic diagram showing a display example of a structure object according to height information; FIG. 6 is a schematic diagram showing a display example of a structure object according to a gesture operation; FIG. 7 is a schematic diagram showing a display example of a UI; FIG. 8 is a schematic diagram showing another display example of an operation UI.

[0026] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0027] Fig. 1 is a schematic diagram showing a configuration example of an image display device according to an embodiment of the present technology. The image display device 100 is a device equipped with a touch display 10 and displays an image on a display surface 11 of the touch display 10. In the example shown in Fig. 1, the image display device 100 is configured as a mobile terminal such as a smartphone. Note that the image display device 100 may also be configured as a stationary device. In this embodiment, the touch display 10 is an example of a display device.

[0028] In this embodiment, the touch display 10 is configured using a stereoscopic display 13 (described later) and is a display capable of displaying a stereoscopic image (3D display) using naked-eye stereoscopic vision. Therefore, a user viewing the display surface 11 of the touch display 10 can perceive the object (structural object 1) to be displayed as a stereoscopic image.

[0029] In the example shown in FIG. 1 , a structural object 1 (human body model) representing the structure of a human body is schematically illustrated as an example of an object to be displayed three-dimensionally by the touch display 10. In this case, the user can visually recognize the shape and arrangement of each part of the human body model as a three-dimensional image. Of course, the content and type of the structural object 1 are not limited. The configuration of the structural object 1 will be described in detail later.

[0030] The touch display 10 is a touch device equipped with a touch sensor 12 (described later) and accepts touch operations by the user on the display surface 11. For example, the user can perform various input operations on the image displayed on the display surface 11 by touching the touch display 10 with an operating object 5 (the user's own finger 6 in FIG. 1 ) for operating the touch display 10. Note that the operating object 5 may be anything other than the user's finger 6 as long as it is responsive to the touch display 10 (touch sensor 12), and a touch pen, for example, may also be used.

[0031] Furthermore, the image display device 100 is configured to detect the height position and planar position of the operating tool 5 even when the operating tool 5 is not in contact with the touch display 10, and to enable input operations using the operating tool 5. This point will be described in detail later.

[0032] 2 is a block diagram showing an example of the configuration of an image display device 100. As shown in FIG. 2, the image display device 100 includes the touch display 10 having the touch sensor 12 and the 3D display 13, a viewpoint detection camera 15, a storage unit 16, a communication unit 17, and a controller 20.

[0033] The touch sensor 12 is a sensor that detects contact of the operating object 5 with the display surface 11 on which an image is displayed in the touch display 10 (three-dimensional display 13). A capacitance type sensor arranged along the display surface 11 is used as the touch sensor 12. A projection type sensor such as a self-capacitance type or a mutual capacitance type is typically used as the capacitance type touch sensor 12, but a surface type sensor or the like may also be used.

[0034] The capacitive touch sensor 12 detects the position of the operating object 5 relative to the display surface 11 by detecting a change in capacitance that occurs when the operating object 5 approaches a plurality of transparent electrodes formed along the display surface 11, for example. In the capacitive touch sensor 12, the capacitance changes even when the operating object 5 is not in contact with the display surface 11, for example. Utilizing this, the position of the operating object 5 in the space directly above the display surface 11 can be detected from the detection result of the touch sensor 12.

[0035] Hereinafter, the direction perpendicular to the display surface 11 may be referred to as the Z direction, the left-right direction (short direction) of the display surface 11 as the X direction, and the up-down direction (longitudinal direction) of the display surface 11 as the Y direction. The detection result by the touch sensor 12 becomes data capable of representing the X coordinate, Y coordinate, and Z coordinate of the operating object 5 as viewed from the display surface 11.

[0036] The 3D display 13 is a naked-eye stereoscopic display that allows the user to perceive a 3D image with the naked eyes. The 3D display 13 displays a right-eye image and a left-eye image for the user's right eye and left eye, respectively, detected by the viewpoint detection camera 15. As the 3D display 13, for example, a display that displays a right-eye image and a left-eye image separately using a lenticular lens or a parallax barrier is used. The specific configuration of the 3D display 13 is not limited thereto.

[0037] The viewpoint detection camera 15 is, for example, a camera disposed in front of the image display device 100 where the display surface 11 is provided, and captures an image of the face of a user looking at the display surface 11. The captured image captured by the viewpoint detection camera 15 is output to the controller 20. As the viewpoint detection camera 15, for example, a digital camera equipped with an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is used. The specific configuration of the viewpoint detection camera 15 is not limited, and for example, a multi-lens camera such as a stereo camera may be used. Furthermore, an infrared camera that captures infrared images by irradiating infrared light, a Time of Flight (ToF) camera that functions as a distance measurement sensor, or the like may be used as the viewpoint detection camera 15.

[0038] The storage unit 16 is a non-volatile storage device, and may be, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The storage unit 16 stores a control program according to this embodiment, content data, etc. The control program is a program that controls the overall operation of the image display device 100. The content data is content data including a structure object 1, which will be described later, etc. The content data records information such as the three-dimensional shape of the structure object 1 and the surface color. In this embodiment, the storage unit 16 corresponds to a computer-readable recording medium on which a program is recorded. The control program corresponds to a program recorded on the recording medium.

[0039] The communication unit 17 is a communication module that enables the image display device 100 to communicate with an external device. A Wi-Fi (registered trademark) module, a Bluetooth (registered trademark) module, or the like is used as the communication unit 17. The content data may be read from an external device via the communication unit 17.

[0040] The controller 20 is an information processing device that controls the overall operation of the image display device 100. The controller 20 has hardware components necessary for a computer, such as a CPU and memory (RAM, ROM), etc. The CPU loads a control program stored in the storage unit 16 into the RAM and executes it, thereby performing various processes.

[0041] The controller 20 may be a programmable logic device (PLD) such as a field programmable gate array (FPGA), or another device such as an application specific integrated circuit (ASIC). Alternatively, the controller 20 may be a graphics processing unit (GPU).

[0042] In this embodiment, the CPU of the controller 20 executes a program (control program) according to this embodiment, thereby realizing functional blocks including a viewpoint detection unit 21, an operation detection unit 22, and a display control unit 23. These functional blocks then execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0043] The viewpoint detection unit 21 detects the position of the viewpoint of the user observing the display surface 11 based on the captured image captured by the viewpoint detection camera 15. For example, facial recognition of the user looking at the display surface 11 is performed, and three-dimensional coordinates of the user's viewpoint position as seen from the image display device 100 are calculated. The method for detecting the viewpoint is not limited, and for example, viewpoint detection using pattern matching or the like, or viewpoint estimation processing using machine learning or the like may be performed.

[0044] The operation detection unit 22 acquires the position of the operating object 5 operated by the user, and detects various operations performed by the user using the operating object 5. In this embodiment, the operation detection unit 22 corresponds to an acquisition unit.

[0045] Specifically, the operation detection unit 22 acquires height information of the operating object 5 relative to the display surface 11 of the touch display 10. The height information is, for example, the distance in the Z direction between the operating object 5 and the display surface 11 (the height level of the operating object 5). By using the height information, it becomes possible to set an arbitrary threshold value in the height direction as seen from the display surface 11, for example, and to perform display control for each threshold value.

[0046] The operation detection unit 22 also acquires in-plane position information of the operating body 5 in an in-plane direction perpendicular to the height direction represented by the height information. The in-plane position information is, for example, the in-plane position of the operating body 5 in an XY plane perpendicular to the Z direction, which is the height direction. By using the in-plane position information, it becomes possible to detect, for example, movement of the operating body 5 in the horizontal direction (XY plane direction) relative to the display surface 11. This makes it possible to detect gesture operations, which will be described later, and to perform display control corresponding to the gesture operations.

[0047] In this way, the operation detection unit 22 acquires spatial position information including height information of the operating body 5 and in-plane position information of the operating body 5. The spatial position information is the three-dimensional position (X coordinate, Y coordinate, Z coordinate) of the operating body 5 relative to the display surface 11.

[0048] In this embodiment, the touch sensor 12 is used as a sensor for detecting the position of the operating object 5. That is, the operation detection unit 22 acquires spatial position information (height information and in-plane position information) of the operating object 5 based on the detection result of the operating object 5 by the touch sensor 12.

[0049] For example, the amount of change in capacitance detected by the transparent electrodes provided in the touch sensor 12 increases as the operating object 5 approaches the display surface 11. Therefore, height information about the operating object 5 can be calculated by referring to the capacitance value, etc. Furthermore, among the multiple transparent electrodes, the electrodes closer to the operating object 5 experience a larger change in capacitance. Therefore, in-plane position information about the operating object 5 can be calculated by comparing the change in capacitance at each transparent electrode. By using the detection results of the touch sensor 12 in this manner, it is possible to calculate spatial position information about the operating object 5 even when the operating object 5 is not in contact with the display surface 11. Note that the method for calculating the height information and in-plane position information (spatial position information) of the operating object 5 using the touch sensor 12 is not limited, and other methods may be used.

[0050] The operation detection unit 22 also detects a gesture operation performed by the operating object 5 based on the spatial position information. A gesture operation is, for example, an input operation performed by a user by moving the operating object 5. The operation detection unit 22 detects, for example, a gesture operation performed in the air when the operating object 5 is not in contact with the display surface 11. The operation detection unit 22 can also detect a gesture operation performed when the operating object 5 is in contact with the display surface 11.

[0051] The display control unit 23 is a functional block that controls image display on the touch display 10. As shown in FIG.

[0052] The display data generation unit 24 generates display data to be displayed on the touch display 10. The display data is data that is the source of the right-eye image and the left-eye image generated by the 3D display processing unit 25. The display data includes, for example, data on the structure object 1 and other UIs (User Interfaces), etc.

[0053] The structure object 1 is an object that represents the structure of the display target according to the display level. Here, the display level is, for example, a level that specifies the display content of the structure object 1. From another perspective, the structure object 1 can also be said to be an object whose display content changes according to the display level.

[0054] For example, an object modeled as a 3D CG display target may include multiple parts. These parts can be displayed by dividing them into related groups (e.g., skin, skeleton, etc.). In this case, the parameters specifying each group are the display levels.

[0055] In this embodiment, the display level is associated with height information of the operating body 5. For example, height information (height of the operating body 5 from the display surface 11) is appropriately converted into the display level. This makes it possible to change the display level by changing the height of the operating body 5, and to operate the display content of the structure object 1 displayed on the touch display 10.

[0056] The display data generation unit 24 reads the height information detected by the above-mentioned operation detection unit 22, and generates display data so that the structure object 1 is displayed at a display level converted from the height information. In this way, the display data generation unit 24 controls the display on the touch display 10 of the structure object 1 representing the structure to be displayed according to the display level corresponding to the height information, based on the height information of the operating body 5.

[0057] The 3D display processing unit 25 performs image processing for stereoscopically displaying the display data based on the display data generated by the display data generation unit 24. Specifically, the 3D display processing unit 25 generates an image for the right eye and an image for the left eye using the display data as original data in accordance with the user's viewpoint position detected by the viewpoint detection unit 21. The 3D display processing unit 25 also assigns the image for the right eye to pixels that display an image for the user's right eye, and the image for the left eye to pixels that display an image for the left eye, and synthesizes the output image to be output to the stereoscopic display 13.

[0058] When the stereoscopic display 13 displays the output image, a mechanism (such as a lenticular lens) that separates pixel light for the right eye and left eye causes the image for the right eye to be displayed to the user's right eye, and the image for the left eye to be displayed to the user's left eye. As a result, the user looking at the display surface 11 can three-dimensionally perceive the structural object 1 and the like included in the display data.

[0059] [Structure Object] Fig. 3 is a schematic diagram for explaining a structure object. Below, the configuration of the structure object 1 displayed on the touch display 10 will be specifically described with reference to Fig. 3. Here, a structure object 1 modeled after the structure of the human body will be used as an example. In this case, the display target is the human body, and the structure object 1 is an object modeled after the structure of each part of the human body.

[0060] The structure object 1 includes a plurality of image objects 2 representing the structure of the display target according to the display level. The plurality of image objects 2 are, for example, objects that display the structure of the display target divided into groups, and each image object 2 is configured as an object modeling the parts that make up each group. Each image object 2 (group) is specified by the display level.

[0061] 3, as image objects 2 representing the structure of the human body, an image object 2a representing the external appearance of the human body, an image object 2b representing the internal organs, and an image object 2c representing the skeleton are schematically illustrated in this order from the top of the figure. For example, the image object 2a is a group of external parts (limbs, torso, head, face, etc.), the image object 2b is a group of internal organ parts (heart, lungs, intestines, etc.), and the image object 2c is a group of bony parts (skull, spine, ribs, etc.).

[0062] The multiple image objects 2 are 3D objects. That is, each image object 2 is configured as a model (3DCG, etc.) having a three-dimensional shape. This makes it possible to realistically present the structure of the display target. Note that the image objects 2 do not necessarily have to be 3D objects, and may include, for example, 2D objects (planar images). Here, image objects 2a to 2c representing the appearance, internal organs, and skeleton of the human body are each configured as 3D objects.

[0063] In this way, when a structure object 1 consisting of a plurality of image objects 2 is used, the display data generation unit 24 determines a display object to be displayed on the touch display 10 from the plurality of image objects 2 based on height information. Specifically, the height information detected by the operation detection unit 22 is converted into a display level. The image object 2 specified by that display level is set as the display object. Therefore, in the example shown in FIG. 3, a plurality of image objects 2a to 2c are displayed by switching between them depending on the height information.

[0064] Furthermore, the structure object 1 stores a plurality of image objects 2 in a predetermined order, layer by layer. That is, the structure object 1 is an object in which a plurality of image objects 2 are layered. In this case, the display level is the hierarchical level of the structure object 1, and is a parameter that specifies the layer of the structure object 1. In the layered structure object 1, the image objects 2 are arranged, for example, so that they form a structure from the outside to the inside of the display object, or from top to bottom. This enables intuitive display operations for the structure object 1 linked to height information.

[0065] In the example shown in Figure 3, the layer including image object 2a is the first layer, the layer including image object 2b is the second layer, and the layer including image object 2c is the third layer. This order is, for example, such that the structures are arranged from the outside of the human body to the inside, from the first layer to the third layer. As a result, the deeper the layer (the larger the layer number), the deeper the structure object 1 represents the structure of the human body. Note that the order of the layers is not limited, and for example, the numbers of the layers may be set in the opposite order to that shown in Figure 3.

[0066] For example, the display object may be an industrial product such as an automobile or electronic device, or a structure such as a house or building. For example, if the display object is a car, a structure object 1 is configured including image objects 2 representing the car's body, interior, drivetrain, electrical system, etc. If the display object is a building, a structure object 1 is configured including image objects 2 representing the building's exterior, framework, three-dimensional structure and planar structure of each floor, etc.

[0067] Furthermore, the structure object 1 does not have to include a 3D object. For example, two-dimensional graphic data (two-dimensional CAD data, vector image data, etc.) with layers may be used as the structure object 1. In this case, the image data of each layer becomes the image object 2, and the display level becomes a parameter that specifies the layers in order. In addition, the configuration of the structure object 1 with a hierarchical structure is not limited.

[0068] [Display control of structure object according to height information] Fig. 4 is a flowchart showing an example of the operation of the image display device. Fig. 5 is a schematic diagram showing an example of display of structure object according to height information. Below, with reference to Figs. 4 and 5, display control of structure object 1 performed using height information of the operating body 5 will be described.

[0069] 4 is a loop process that is repeatedly executed during operation of, for example, an application that displays the structural object 1 on the image display device 100. First, the operation detection unit 22 acquires height information relative to the display surface 11 of the touch display 10 (step 101). For example, the detection result of the touch sensor 12 is read, and the distance (height information) to the operating object 5 is calculated based on the capacitance value. Note that the height information does not necessarily need to be the actual distance, and may be a relative parameter that can represent the magnitude of the distance to the operating object 5.

[0070] Next, based on the detected height information, the display data generation unit 24 executes a hierarchical level determination process (step 102). Here, the hierarchical level of the structure object 1 that corresponds to the height information is determined as the display hierarchical level, and the image object 2 set in the display hierarchical level is set as the display object. This is a process of selecting the image object 2 (display object) to be displayed on the touch display 10 using the height information. The hierarchical level determination process will be described later with reference to FIG. 5.

[0071] Once the display object is set, an object display process is executed to display the display object on the touch display 10 (step 103). In this process, display data including the display object is first generated by the display data generation unit 24. Information such as a UI to be displayed together with the display object is also generated as display data.

[0072] Once the display data is generated, the 3D display processing unit 25 generates an output image to be output to the touch display 10 (stereoscopic display 13). For example, a rendering process of the display object is performed based on the user's viewpoint (right eye and left eye), and an image for the right eye and an image for the left eye are generated. These images are synthesized according to the user's viewpoint and the characteristics of the display, and an output image is generated. The generated output image is output to the touch display 10. As a result, a stereoscopic image of the display object is displayed.

[0073] [Layer Determination Process] In the tier determination process executed in step 102, the displayed tier is determined by threshold determination, which compares the height information with one or more tier thresholds associated with the tiers of the structure object 1. Here, the tier threshold is a threshold for associating the height information with the tier of the structure object 1. The threshold determination will be described below with reference to FIG. 5 .

[0074] 5, the hierarchy of the structure object 1 is illustrated to correspond to the height level Zo relative to the display surface 11 of the touch display 10. The upward arrow in the figure represents the height direction (Z direction) as seen from the display surface 11, and the rightward arrow in the figure represents the horizontal direction (X direction or Y direction) parallel to the display surface 11.

[0075] As described above, this embodiment uses a structure object 1 having first to third hierarchical levels. In this case, as shown in Fig. 5, a threshold value separating the first and second hierarchical levels (first hierarchical level threshold Z1) and a threshold value separating the second and third hierarchical levels (second hierarchical level threshold Z2) are set.

[0076] 5, when the height level Zo of the operating object 5 is higher than the first hierarchical level threshold Z1 (Z1<Zo), the first hierarchical level is determined to be the display hierarchical level, and an image object 2a representing the external appearance of a human body set in the first hierarchical level is set as the display object 3. Also, when the height level of the operating object 5 is equal to or less than the first hierarchical level threshold Z1 and greater than the second hierarchical level threshold Z2 (Z2<Zo≦Z1), the second hierarchical level is determined to be the display hierarchical level, and an image object 2b representing the internal organs of a human body set in the second hierarchical level is set as the display object 3. Also, when the height level of the operating object 5 is equal to or less than the second hierarchical level threshold Z2 (Zo≦Z2), the third hierarchical level is determined to be the display hierarchical level, and an image object 2c representing the skeleton of a human body set in the third hierarchical level is set as the display object 3.

[0077] In this manner, in this embodiment, a threshold value for dividing the layers of the structure object 1 is set in the height direction (Z direction) of the touch display 10. As a result, for example, when the user moves the operating body 5 from above the display surface 11 so as to approach the display surface 11, the image objects 2a, 2b, and 2c (the external appearance, internal organs, and skeleton of the human body) are displayed in this order on the touch display 10. Conversely, when the operating body 5 is moved away from the display surface 11, the structures of the human body are displayed in the reverse order to when the operating body 5 was moved closer.

[0078] That is, if one wishes to see a deeper structure (internal structure) of the human body, one can simply move the operating body 5 closer to the display surface 11, and if one wishes to see a shallower structure (external structure) of the human body, one can simply move the operating body 5 away from the display surface 11. By performing such display control, it becomes possible to intuitively display, for example, the layer of the structure object 1 that the user wants to see, thereby improving the user experience in displaying and manipulating objects.

[0079] Furthermore, in this embodiment, by using the 3D display 13, the user can perceive the display object 3 as if it were a real object. This allows the user to have an operation experience in which the deeper structure of the displayed object is displayed, with the feeling that the user is approaching the surface or interior of the real object. This makes it possible to provide a superior user experience that cannot be obtained by operations performed by touching the display surface 11.

[0080] 6 is a schematic diagram showing an example of display of a structural object in response to a gesture operation. In this embodiment, the display of the display object 3 on the touch display 10 is controlled in response to a gesture operation performed by a user using the operating object 5. That is, the display object 3 selected in response to the height information of the operating object 5 is displayed in response to the gesture operation. This enables not only switching of the display object 3 but also various display operations.

[0081] In the example shown in Fig. 6, the height level of the operating object 5 is greater than the first layer threshold. In this case, the first layer becomes the display layer, and the image object 2a becomes the display object 3. Note that although the operating object 5 is not in contact with the display surface 11, as described with reference to Fig. 2 etc., the operation detection unit 22 can acquire planar position information in addition to height information of the operating object 5. This allows a gesture operation in the air by the operating object 5 to be detected. Of course, gesture operations are also possible when other layers (the second layer and the third layer) are selected.

[0082] The gesture operations detected by the operation detection unit 22 include a flick operation by the operating object 5 performed in a state where the operating object 5 is not in contact with the display surface. Fig. 6 schematically illustrates an example of a flick operation. The flick operation is, for example, an operation of moving the operating object 5 (here, the user's fingertip) along the XY plane in a flicking motion. The flick operation is detected based on, for example, the amount of change in the in-plane position and the speed of the operating object 5.

[0083] As shown in Fig. 6 , when a flick operation is performed, for example, a process of rotating the display object 3 in accordance with the flick operation is executed. For example, when a flick operation is performed in the left-right direction of the touch display 10, the display object 3 is displayed so as to rotate around an axis in the up-down direction. Note that the rotation axis of the display object 3 may be fixed in advance or may be set according to the direction of the flick operation. For example, a rotation axis may be set so as to be perpendicular to the direction of the flick operation, and the display object 3 may be rotated around the rotation axis.

[0084] Furthermore, for example, a swipe operation that translates the operating object 5 in the horizontal direction may be detected. A swipe operation is an operation that is slower than, for example, a flick operation. The swipe operation performs a display process that translates the display object 3, for example. Alternatively, a process of displaying another structural object 1 may be performed. Furthermore, for example, if the operating object 5 is a user's finger, a pinch-out operation or a pinch-in operation may be detected. For example, if a pinch-out operation is detected, the display object 3 is enlarged, and if a pinch-in operation is detected, the display object 3 is reduced. Besides, the type of gesture operation detected by the operation detection unit 22 is not limited.

[0085] It is also conceivable that performing a gesture operation may cause the height level of the operating body 5 to exceed the hierarchical threshold. In such a case, when the display object 3 is switched, the object cannot be stably viewed. For this reason, for example, when the above-described gesture operation is detected or when the height level of the operating body 5 does not exceed the threshold for a certain period of time, the switching process of the display object 3 according to the height information may be temporarily stopped. This makes it possible to stably display the display object 3 even when a gesture operation is performed.

[0086] [Hierarchical UI and Operation UI] Fig. 7 is a schematic diagram showing a display example of a UI. Fig. 7 shows a schematic diagram of an example of a UI (hierarchical UI 31 and operation UI 32) displayed together with a display object 3. Here, each UI is displayed in a UI display area 30 set in the upper right corner of the display surface 11 in the figure.

[0087] In this embodiment, a hierarchy UI 31 indicating the display hierarchy is displayed on the touch display 10. The hierarchy UI 31 is a UI for informing the user of the hierarchy of the currently displayed structure object 1. In this embodiment, the hierarchy UI 31 corresponds to the first UI.

[0088] The hierarchy UI 31 shown in FIG. 7 is designed with an L-shaped hierarchy mark to indicate which hierarchy level the current hierarchy level is, counting from the bottom. That is, the hierarchy UI 31 can display the number of hierarchy levels. For example, if the first hierarchy level is the display hierarchy level, three hierarchy marks are displayed on the hierarchy UI 31. If the second hierarchy level and the third hierarchy level are the display hierarchy levels, two hierarchy marks and one hierarchy mark are displayed, respectively. Of course, the design of the hierarchy UI 31 is not limited to the example shown in FIG. 7, and any design including numbers, etc., can be used.

[0089] Furthermore, the size and color of the hierarchical UI 31 change in conjunction with the depth direction. That is, at least one of the size and color of the hierarchical UI 31 is changed according to the depth of the display hierarchical layer (height information). In the example shown in FIG. 7 , the deeper the display hierarchical layer (the larger the hierarchical layer number), the smaller the size of the hierarchical UI 31 is set. Also, for example, processing may be performed such that the shading of the hierarchical UI 31 becomes darker (or lighter) as the display hierarchical layer becomes deeper. Of course, the color itself may also be changed for each hierarchical layer. In this way, by changing the size and color of the hierarchical UI 31, the user can intuitively grasp the depth of the current hierarchical layer, etc.

[0090] In this embodiment, an operation UI 32 indicating acceptable operations that can be accepted as gesture operations on the display object 3 by the operating body 5 is displayed on the touch display 10. In other words, the operation UI 32 can be said to be a UI indicating gesture operations that are permitted at the current position of the operating body 5. In this embodiment, the operation UI 32 corresponds to the second UI.

[0091] 7 illustrates an example of an operation UI 32 indicating a rotation operation (flick operation) as an acceptable operation. Here, three arrows arranged along a circle indicate that a rotation operation is possible. Alternatively, if the rotation axis is predetermined, a UI indicating the rotation axis by the rotation direction of a sphere or the like may be used. If the rotation axis can be freely set, a UI indicating this may be used.

[0092] Furthermore, the size and color of the operation UI 32 change in conjunction with the depth direction, as in the case of the hierarchical UI. That is, at least one of the size and color of the operation UI 32 is changed according to the depth of the displayed hierarchy (height information). In the example shown in FIG. 7 , the size of the operation UI 32 is set smaller as the displayed hierarchy level (the higher the hierarchy number) increases. In addition, the shading, color, etc. of the operation UI 32 may be changed. In this way, by changing the display of the operation UI 32, it is possible to inform the user of the current hierarchy level, etc.

[0093] There are no limitations on the types of operation UI 32 or acceptable operations. For example, the operation UI 32 may display an operation such as a zoom-in operation (pinch-out operation / pinch-in operation) or a movement operation (swipe operation) as described with reference to FIG. 6 .

[0094] Note that acceptable operations may be set for each layer of the structure object 1. Therefore, for example, different gesture operations may be accepted for each layer of the structure object 1, or two or more gesture operations may be accepted for one layer. For example, it is possible to set the first layer to allow rotation operations, the second layer to allow rotation operations and movement operations, and the third layer to allow rotation operations and zoom operations.

[0095] Furthermore, the operation UI 32 may display prohibited operations instead of acceptable operations. For example, in a layer where a rotation operation is prohibited, the operation UI 32 representing the rotation operation may be grayed out, and in a layer where a rotation operation is acceptable, the operation UI 32 representing the rotation operation may be clearly displayed in black. This makes it possible to easily inform the user of the gesture operations that are permitted and prohibited in each layer.

[0096] 8 is a schematic diagram showing another display example of the operation UI. In the example shown in FIG. 8, the operation UI 32 is displayed on the display object 3, not on the UI display area 30. Specifically, the operation UI 32 is displayed superimposed on a portion of the display object 3 where an acceptable operation can be performed. Here, the operation UI 32 is superimposed on a portion of the image object 2b in the second layer where a rotation operation can be performed (such as a portion that is the center of the rotation operation). This makes it possible to inform the user that the display object 3 can be rotated around the portion where the operation UI 32 is superimposed.

[0097] Furthermore, for example, each part included in the display object 3 may be individually operable, such as for rotation. In this case, the operation UI 32 is superimposed on the part that can be operated, such as for rotation. This allows, for example, the user to selectively perform display operations such as rotation, movement, enlargement, reduction, etc., on the part that the user wants to check in detail. This improves usability.

[0098] As described above, in the image display device 100 according to this embodiment, the display of the structural object 1 representing the structure of the display target according to the display level corresponding to the height information is controlled using height information of the operating body 5 relative to the display surface 11 of the touch display 10. This makes it possible to change the display content of the structural object 1 by changing the height of the operating body 5, for example, and improve the user experience when operating the object.

[0099] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0100] In the above embodiment, a configuration has been described in which height information and in-plane position information (spatial position information) of the operating object are acquired using a touch sensor. The method of acquiring the height information and in-plane position information of the operating object is not limited, and a sensor other than a touch sensor may be used. For example, a front camera or the like may be used to detect the height information of the operating object. In this case, for example, a ToF camera or a stereo camera or the like may be used.

[0101] Furthermore, the height information may be acquired using a proximity sensor provided in a smartphone or the like. The proximity sensor is a sensor for avoiding malfunctions that may occur when the ear touches the touch display during a call, and is configured, for example, by combining an infrared LED and an infrared detector that detects the reflected light. The use of a proximity sensor makes it possible to easily detect the height information. In addition, a distance measurement sensor such as a radar sensor may also be used. The position of the operating object may also be detected by appropriately combining a touch sensor, a camera sensor, a proximity sensor, a radar sensor, etc.

[0102] In the above embodiment, a configuration using a stereoscopic display has been described, but the present technology can also be applied to a case where a display that displays a normal two-dimensional image is used. For example, when a 3D object or the like is displayed on a two-dimensional surface, the display of the 3D object is controlled according to the height of the operating body relative to the display surface of the display. In this case, too, by changing the height of the operating body, it is possible to manipulate the display content of the 3D object, thereby improving the user experience.

[0103] In the above embodiment, a configuration has been described in which a structure object including a plurality of image objects is switched and displayed according to height information. For example, a structure obtained by cutting a 3D object may be displayed according to the height information. In this case, the depth of the cross-section corresponds to the height information. Furthermore, the structure object is not limited to data (3DCG) that models the object to be displayed, but may also be an object that displays measurement results such as a CT scan. In this case, by setting the depth of the cross-section according to the height information, it becomes possible to easily check measurement results at a desired depth.

[0104] In the above, the information processing method according to the present technology is executed by a controller provided in the image display device. However, the present technology may be implemented by linking the controller with another computer that can communicate with the controller via a network or the like, thereby executing the information processing method and program according to the present technology, thereby implementing an information processing device according to the present technology.

[0105] In other words, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0106] The information processing method and program execution according to the present technology by a computer system include both cases where, for example, acquisition of height information of an operating object and display control of a structural object are executed by a single computer, and cases where each process is executed by a different computer. Furthermore, execution of each process by a specific computer includes having another computer execute part or all of the process and acquiring the results.

[0107] In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

[0108] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0109] In the present disclosure, the terms "same," "equal," "orthogonal," etc. are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, they also include states that fall within a predetermined range (e.g., a range of ±10%) based on "completely the same," "completely equal," "completely orthogonal," etc.

[0110] The present technology may also have the following configurations. (1) An information processing device comprising: an acquisition unit that acquires height information of an operating body relative to a display surface of a display device; and a display control unit that controls, based on the height information, display of a structural object on the display device that represents a structure of the display target according to a display level corresponding to the height information. (2) The information processing device described in (1), wherein the structural object includes a plurality of image objects that represent a structure of the display target according to the display level, and the display control unit determines a display object to be displayed on the display device from the plurality of image objects based on the height information. (3) The information processing device described in (2), wherein at least some of the plurality of image objects are 3D objects. (4) The information processing device described in (2) or (3), wherein the structural object is an object in which the plurality of image objects are hierarchically arranged, and the display level is a hierarchical level of the structural object. (5) The information processing device described in (4), wherein the display control unit determines a layer among the layers of the structural object that corresponds to the height information as a display layer, and sets the image object set in the display layer as the display object. (6) The information processing device according to (5), wherein the display control unit determines the display layer by threshold determination that compares the height information with one or more layer thresholds associated with the layer of the structure object. (7) The information processing device according to (5) or (6), wherein the display control unit displays a first UI indicating the display layer. (8) The information processing device according to (7), wherein the display control unit changes at least one of a size and a color of the first UI according to the depth of the display layer.(9) The information processing device according to any one of (2) to (8), wherein the acquisition unit acquires spatial position information including the height information of the operating body and in-plane position information of the operating body in an in-plane direction orthogonal to a height direction represented by the height information, and detects a gesture operation by the operating body based on the spatial position information, and the display control unit controls display of the display object on the display device in accordance with the gesture operation. (10) The information processing device according to (9), wherein the gesture operation includes a flick operation by the operating body performed in a state where the operating body is not in contact with the display surface, and the display control unit rotates the display object in accordance with the flick operation. (11) The information processing device according to (9) or (10), wherein the display control unit displays a second UI indicating an acceptable operation that can be accepted as the gesture operation by the operating body on the display object. (12) The information processing device according to (11), wherein the structure object is an object in which the plurality of image objects are hierarchically arranged, and the acceptable operation is set for each layer of the structure object. (13) The information processing device according to (11) or (12), wherein the display control unit displays the second UI superimposed on a portion of the display object where the acceptable operation can be performed. (14) The information processing device according to any one of (11) to (13), wherein the display control unit changes at least one of a size and a color of the second UI according to the height information. (15) The information processing device according to any one of (1) to (14), wherein the display device is a touch device equipped with a touch sensor, and the acquisition unit acquires the height information based on a detection result of the operating object by the touch sensor. (16) The information processing device according to any one of (1) to (15), wherein the display device is a naked-eye stereoscopic display.(17) An information processing method executed by a computer system, in which height information of an operating body relative to a display surface of a display device is acquired, and based on the height information, display of a structural object on the display device that represents the structure of a display target in accordance with a display level corresponding to the height information is controlled. (18) A program that causes a computer system to execute the steps of: acquiring height information of an operating body relative to a display surface of a display device; and controlling, based on the height information, display of a structural object on the display device that represents the structure of a display target in accordance with a display level corresponding to the height information.

[0111] REFERENCE SIGNS LIST 1 structural object 2, 2a to 2c image objects 3 display object 5 operating body 10 touch display 11 display surface 16 storage unit 20 controller 21 viewpoint detection unit 22 operation detection unit 23 display control unit 31 hierarchical UI 32 operation UI 100 image display device

Claims

1. An information processing device comprising: an acquisition unit that acquires height information of an operating body relative to a display surface of a display device; and a display control unit that controls the display of a structural object on the display device that represents the structure of a display target according to a display level corresponding to the height information, based on the height information.

2. An information processing device according to claim 1, wherein the structural object includes a plurality of image objects representing the structure of the display target according to the display level, and the display control unit determines a display object to be displayed on the display device from the plurality of image objects based on the height information.

3. An information processing device according to claim 2, wherein at least some of the plurality of image objects are 3D objects.

4. An information processing device according to claim 2, wherein the structure object is an object in which the plurality of image objects are hierarchically arranged, and the display level is a hierarchical level of the structure object.

5. An information processing device according to claim 4, wherein the display control unit determines a layer among the layers of the structural object that corresponds to the height information as a display layer, and sets the image object set in the display layer as the display object.

6. An information processing device according to claim 5, wherein the display control unit determines the display layer by threshold determination that compares the height information with one or more layer thresholds associated with the layer of the structure object.

7. An information processing device according to claim 5, wherein the display control unit displays a first UI indicating the display hierarchy.

8. An information processing device according to claim 7, wherein the display control unit changes at least one of the size and color of the first UI according to the depth of the display hierarchy.

9. An information processing device according to claim 2, wherein the acquisition unit acquires spatial position information including the height information of the operating body and in-plane position information of the operating body in an in-plane direction perpendicular to the height direction represented by the height information, detects a gesture operation by the operating body based on the spatial position information, and the display control unit controls the display of the display object on the display device in accordance with the gesture operation.

10. An information processing device according to claim 9, wherein the gesture operation includes a flick operation by the operating body performed in a state where the operating body is not in contact with the display surface, and the display control unit rotates the display object in response to the flick operation.

11. An information processing device according to claim 9, wherein the display control unit displays a second UI indicating acceptable operations that can be accepted as the gesture operation on the display object by the operating object.

12. An information processing device according to claim 11, wherein the structure object is an object in which the plurality of image objects are hierarchically arranged, and the acceptable operations are set for each hierarchical level of the structure object.

13. An information processing device according to claim 11, wherein the display control unit displays the second UI superimposed on a portion of the display object where the acceptable operation can be performed.

14. An information processing device according to claim 11, wherein the display control unit changes at least one of the size and color of the second UI in accordance with the height information.

15. An information processing device according to claim 1, wherein the display device is a touch device equipped with a touch sensor, and the acquisition unit acquires the height information based on the detection result of the operating object by the touch sensor.

16. An information processing device according to claim 1, wherein the display device is a naked-eye stereoscopic display.

17. An information processing method executed by a computer system, which acquires height information of an operating body relative to a display surface of a display device, and controls the display of a structural object on the display device, based on the height information, which represents the structure of a display target according to a display level corresponding to the height information.

18. A program that causes a computer system to execute the steps of: acquiring height information of an operating body relative to a display surface of a display device; and controlling, based on the height information, the display of a structural object on the display device that represents the structure of a display target in accordance with a display level corresponding to the height information.

Images