Information processing device and information processing method

Abstract

This information processing device comprises: an acquisition unit which acquires line of sight information indicating a gazing point of a user and object information indicating a plurality of objects; and a selection unit which selects one among the plurality of objects on the basis of location relationships between the gazing point of the user and the plurality of objects.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, with the development of sensing technology, methods for inputting information to and controlling devices have become more diverse. For example, a technology relating to an operation for selecting an object such as an icon using a user's line of sight is known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 189254 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when selecting a target based on the user's line of sight, as in the conventional technology, there is a problem that it is difficult to select a target appropriately due to the user's line of sight being unstable, etc. Therefore, it is desired to flexibly select a target according to the user's line of sight.

[0005] Therefore, the present disclosure proposes an information processing device and an information processing method that are capable of flexibly selecting an object according to the user's line of sight. [Means for solving the problem]

[0006] In order to solve the above problem, one form of information processing device according to the present disclosure includes an acquisition unit that acquires gaze information indicating a user's gaze point and object information indicating a plurality of objects, and a selection unit that selects one of the plurality of objects based on the positional relationship between the user's gaze point and the plurality of objects. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating a procedure of a selection process according to an embodiment of the present disclosure. [Figure 5] 10 is a flowchart showing a procedure for determining an area by selecting multiple points. [Figure 6] 10 is a flowchart illustrating a procedure for selecting an area according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating an example of area selection processing. [Figure 8] FIG. 10 is a diagram illustrating an example of a selection process by specifying a range. [Figure 9] FIG. 10 is a diagram illustrating an example of a process for selecting consecutive partial regions. [Figure 10] FIG. 10 is a diagram illustrating an example of changing the selection range. [Figure 11] FIG. 10 is a diagram illustrating an example of editing by changing points. [Figure 12] FIG. 10 is a diagram illustrating an example of editing by adding a point. [Figure 13] FIG. 10 is a diagram illustrating an example of selection using multiple layers. [Figure 14] FIG. 10 is a diagram illustrating an example of selection using multiple layers. [Figure 15] FIG. 10 is a diagram illustrating an example of selection using multiple layers. [Figure 16] FIG. 10 is a diagram illustrating a configuration example of an information processing system according to a modified example of the present disclosure. [Figure 17] FIG. 10 is a diagram illustrating a configuration example of an information processing device according to a modified example of the present disclosure. [Figure 18] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the information processing device and information processing method according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The present disclosure will be described in the following order: 1. Embodiment 1-1. Overview of information processing according to an embodiment of the present disclosure 1-1-1. Background and effects 1-2. Configuration of the information processing device according to the embodiment 1-3. Information processing procedure according to the embodiment 1-4. Processing example 1-4-1. Area selection example 1-4-2. Example of selection by range specification 1-4-3. Example of continuous selection of partial areas 1-4-4. Example of changing the selection range 1-4-5. Editing example 1-4-5-1. Changing the selected point 1-4-5-2. Adding points 1-4-6.Multiple layer example 1-4-6-1. Selection example 1 (medical images) 1-4-6-2. Selection example 2 (map) 1-4-6-3. Selection example 3 (items) 2. Other embodiments 2-1. Variations 2-2.Other configuration examples 2-3.Other 3. Effects of this disclosure 4. Hardware Configuration

[0010] [1. Embodiment] [1-1. Overview of Information Processing According to an Embodiment of the Present Disclosure] 1 and 2 are diagrams illustrating an example of information processing according to an embodiment of the present disclosure. Specifically, Fig. 1 is a diagram illustrating an example of processing (selection processing) for selecting points on the boundary of an object based on the boundary (edge) of the object and the gaze point of a user U1. Fig. 2 also illustrates an overview of processing related to changing the object selected based on a change in the gaze point of the user U1 and determining (generating) an area by selecting multiple points.

[0011] Information processing according to an embodiment of the present disclosure is realized by an information processing device 100 shown in Fig. 3. The information processing device 100 shown in Fig. 3 is an example of an information processing device that performs processes such as selection processing and determining (generating) an area based on a plurality of selected points. The information processing device 100 is an information processing device that executes information processing according to an embodiment. For example, the information processing device 100 is a terminal device used by a user.

[0012] The information processing device 100 is a computer that executes a process of selecting one of a plurality of objects based on the positional relationship between the user's gaze point and the plurality of objects. The information processing device 100 also selects one of a plurality of regions based on the positional relationship between the user's gaze point and the plurality of regions.

[0013] FIG. 1 illustrates an example in which an information processing device 100, which is a terminal device used by a user, performs a selection process, etc. In this case, the information processing device 100 is a device used by a user. The information processing device 100 accepts input from the user. For example, the information processing device 100 accepts input from the user's gaze, voice input from the user's speech, and input from the user's operation. The information processing device 100 displays information according to the user's input. The information processing device 100 may be any device capable of implementing the processes described in the embodiments. For example, the information processing device 100 may be a smartphone, a smart speaker, a television, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. The information processing device 100 may also be a wearable device worn by a user. For example, the information processing device 100 may be various devices such as a head-mounted display, a headset-integrated VR (Virtual Reality) goggles, a smartphone-mounted VR goggles, or a wearable device integrated with glasses or earphones.

[0014] The information processing device 100 may also have software modules for speech signal processing, speech recognition, speech semantic analysis, dialogue control, and the like. The information processing device 100 may have a speech recognition function. For example, the information processing device 100 may have a natural language understanding (NLU) or automatic speech recognition (ASR) function. For example, the information processing device 100 may infer information about a user's intent or an entity (target) from input information obtained by the user's speech. The information processing device 100 functions as a speech recognition server having the functions of natural language understanding and automatic speech recognition.

[0015] The device that performs the selection process described below is not limited to a terminal device used by a user, and may be any device. For example, the information processing device that performs the selection process may be separate from the terminal device used by the user. The system configuration when the selection process is performed on the server side will be described later.

[0016] Next, the details of the examples shown in Figures 1 and 2 will be described. Note that in the examples of Figures 1 and 2, a case where a selection process is performed using a surgical image (surgical video) as the image (content) to be processed is described as an example, but the image (content) to be processed is not limited to a surgical image and may be various types of image (content). For example, it may be various images (content) such as an image (content) of a map or an image (content) of a product, and examples of these will be described later.

[0017] 1, the information processing device 100 displays a surgical image (surgical video) such as image DP10. Note that the information processing device 100 may capture a surgical image such as image DP10 using an image sensor (camera) of the sensor unit 16, or may acquire (receive) a surgical image such as image DP10 captured by another sensor device (for example, an endoscopic camera) from another device.

[0018] Then, the information processing device 100 performs edge detection on the surgical images such as the image DP10 (step S11). The detection unit 152 performs boundary detection in the image DP10 etc. by appropriately using edge detection technology. The detection unit 152 performs boundary detection using algorithms such as segmentation detection and depth map, not limited to edge detection. Since the information processing device 100 performs boundary detection using known techniques such as those described above, detailed description thereof will be omitted.

[0019] In the example of FIG. 1, the information processing device 100 detects boundaries ED (corresponding to, for example, the thick line portions in the image DP11) as shown in the image DP11 by edge detection. That is, the information processing device 100 automatically detects edges (boundaries) from the image. As a result, the information processing device 100 divides the image DP11 into a plurality of objects such as objects OB1, OB2, OB3, OB4, etc. by the detected boundaries ED. For example, in the image DP11, the object OB1 is a pair of forceps, which is a surgical instrument, and the objects OB2 to OB4, etc. represent the organs of a patient.

[0020] The information processing device 100 selects one of the plurality of objects OB1 to OB4, etc., based on the positional relationship between the point of gaze of the user U1, who is the doctor (for example, the surgeon) performing the surgery, and the plurality of objects OB1 to OB4, etc. (step S11). For example, the information processing device 100 selects a point on the boundary ED that separates the plurality of objects OB1 to OB4, etc., that is closest to the point of gaze of the user U1. Note that, although a case where the user's point of gaze is displayed will be described as an example, the user's point of gaze does not have to be displayed.

[0021] As a result, the information processing device 100 selects one object that is the closest object to the gaze point of the user U1. In the example of FIG. 1, the information processing device 100 selects point PT10, which is the point on the boundary ED that is closest to the gaze point SP11 of the user U1, as shown in image DP12. That is, the information processing device 100 selects object OB1, which is a pair of forceps. In this way, the information processing device 100 can flexibly select an object according to the user's line of sight.

[0022] Next, we will use Figure 2 to explain how the selection of a point on the boundary ED is changed by changing the position of the gaze point of user U1. Note that the following will explain, as an example, a case where the location (point) that user U1 wants to select is not the boundary of object OB1, but the boundary of object OB2. In Figure 2, in order to explain how the selected point (also called the "selected point") is changed by changing the line of sight of user U1, the surgical image portion is omitted from images DP13 and DP14, and only the boundary ED indicating the edge detected from the surgical image is illustrated, but it is assumed that images DP13 and DP14 also include surgical images.

[0023] As shown in image DP13, the information processing device 100 changes the position of the selection point in response to a change in the line of sight of the user U1 (step S12). For example, the user U1 may change the line of sight (point of gaze) by moving only his or her eyes, or may change the line of sight (point of gaze) by changing the position or posture of his or her head. For example, the user U1 may change the line of sight (point of gaze) by continuously adjusting the position of his or her head. In the example of FIG. 2, the user U1 wants to select object OB2, so he or she changes the line of sight (point of gaze) in a direction closer to object OB2.

[0024] In response to the change in the position of the gaze point of user U1, the information processing device 100 selects a point on the boundary ED that is closest to the gaze point of user U1 (step S13). User U1 determines the position of the point by adjusting it so that the desired boundary ED is selected. In the example of FIG. 2, the information processing device 100 selects point PT11, which is the point on the boundary ED that is closest to the gaze point SP12 of user U1 after the position change, as shown in image DP14. In this way, the information processing device 100 selects object OB2, which is an object that is a part of the organ that user U1 wanted to select.

[0025] Then, after confirming that point PT11 on the boundary ED corresponding to the object OB2 that the user U1 wanted to select has been selected, the user U1 performs an operation (also referred to as a "confirmation operation") to confirm (determine) the point PT11 as the selected point. For example, as shown in image DP14, the user U1 performs a confirmation operation to set the point PT11 as the confirmed selected point (also referred to as a "confirmed selected point") by uttering an utterance such as "select this point" or "select here" while the point PT11 is displayed. In this case, the information processing device 100 confirms (determines) the point PT11 as the selected point (confirmed selected point) in response to detecting the utterance of user U1 such as "select this point" or "select here." As a result, the point PT11 confirmed as the selected point continues to be displayed.

[0026] The information processing device 100 selects multiple points on the boundary ED in response to the movement of the gaze point of the user U1 (step S14). The information processing device 100 confirms (determines) multiple confirmed selected points in response to the operation of the user U1 as described above. In the example of FIG. 2, the information processing device 100 selects eight points in order from the smallest number of the symbol "PT*" (* is an arbitrary number) attached to the points, that is, points PT11, PT12, PT13, PT14, PT15, PT16, PT17, and PT18. For example, as shown in image DP15, the information processing device 100 confirms (determines) eight confirmed selected points, points PT11 to PT18. Note that point PT11, which is the first confirmed point (also referred to as the "start point") among the confirmed selected points, and point PT18, which is the last confirmed point (also referred to as the "end point") among the confirmed selected points, are located at overlapping positions (for example, the same position), and therefore seven points are illustrated in image DP15.

[0027] Here, if the selected start point and end point overlap, the information processing device 100 generates an area by connecting the selected multiple points (confirmed selected points). For example, the information processing device 100 connects the points with lines in the order in which they were set as confirmed selected points, and if the start point and end point overlap (for example, they match), generates an area formed by the lines connecting the confirmed selected points. In the example of FIG. 2, as shown in image DP15, the information processing device 100 connects multiple points PT11 to PT18 with lines in the order in which they were selected, thereby generating an area AR1 whose boundary is a line LN connecting the multiple points PT11 to PT18. In this way, the information processing device 100 can select (generate) a desired area by selecting points with the user's line of sight.

[0028] 2, for the sake of explanation, the case where the line LN connecting the multiple points PT11 to PT18 is curved is shown as an example, but the line LN connecting the multiple points PT11 to PT18 is not limited to a curve, and may be any type of line such as a straight line or a dashed line. For example, the information processing device 100 may accept a user's designation regarding the shape of the line connecting the selected points. For example, the information processing device 100 may allow the user to designate the shape of the line connecting the selected points by voice.

[0029] In this case, when the user U1 utters "select an area with a straight line," the information processing device 100 may make the line LN a straight line. Also, when the user U1 utters "select an area with a curve," the information processing device 100 may make the line LN a curve as in the case of FIG. 2. In this case, the information processing device 100 may make the line LN a curve that follows the boundary ED. Also, the information processing device 100 may make the line LN a curve that connects multiple points along the path of movement of the user U1's gaze. Also, when the user U1 utters "select an area with a wavy line," the information processing device 100 may make the line LN a wavy line.

[0030] Note that the above is merely an example, and any format can be adopted for the format of the line LN. Furthermore, if the user U1 does not want to select an edge (a point on the boundary ED), the information processing device 100 may allow the user U1 to select a point by free pointing by uttering "select freely." For example, when selecting a point by free pointing, the information processing device 100 may set the position of the gaze point of the user U1 as the selected point.

[0031] [1-1-1. Background and effects] In the past, for example, when setting an area using hands-free operation, it was possible to specify multiple points using voice commands, gaze detection, and head direction detection, but there were issues with drawing the intended curve or setting a complex shape, and there was also the problem of it being difficult to precisely specify the desired location due to errors in gaze or head direction detection.

[0032] Therefore, when specifying a position using the gaze or head direction, the information processing device 100 extracts boundary information from the field of view image using edges, segmentation, depth maps, etc., and allows the user to select a desired target, such as a point on a boundary (edge), based on the extracted boundary information. This allows the information processing device 100 to easily set an object or area in the field of view image in a shape closer to the intended shape than before when selecting an object or area in the field of view image in a hands-free manner. In this way, the information processing device 100 can easily set a desired shape by using feature points in the field of view image when selecting an area in a hands-free manner, even if the accuracy of the gaze and head direction is insufficient.

[0033] [1-2. Configuration of the information processing device according to the embodiment] Next, a configuration of an information processing device 100, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 3 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment of the present disclosure. For example, the information processing device 100 shown in Fig. 3 is an example of an information processing device. The information processing device 100 is a computer that realizes functions as an information processing device, which will be described later.

[0034] 3, the information processing device 100 includes a communication unit 11, an input unit 12, a display unit 13, a storage unit 14, a control unit 15, and a sensor unit 16. In the example of Fig. 3, the information processing device 100 includes the input unit 12 (e.g., a keyboard, a mouse, etc.) that receives various operations from an administrator of the information processing device 100, and the display unit 13 (e.g., a liquid crystal display, etc.) that displays various information.

[0035] The communication unit 11 is realized by, for example, a NIC (Network Interface Card), a communication circuit, etc. The communication unit 11 is connected to a communication network N (a network such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices, etc. via the communication network N.

[0036] The input unit 12 receives various operations input from the user. The input unit 12 accepts input from the user. The input unit 12 accepts input by the user's speech (voice). For example, the input unit 12 accepts user input by voice via a sound sensor such as a microphone included in the sensor unit 16. The input unit 12 accepts various operations by the user's speech. Furthermore, for example, the input unit 12 may accept operations by user gestures using a motion sensor that detects user gestures or the like. In this case, the sensor unit 16 has a motion sensor. Furthermore, the input unit 12 may accept various operations from the user via a keyboard, mouse, or touch panel provided in the information processing device 100.

[0037] The display unit 13 displays various types of information. The display unit 13 is a display device (display unit) such as a display, and displays various types of information. The display unit 13 displays information on the detection results by the detection unit 152. The display unit 13 displays information selected by the selection unit 153. The display unit 13 displays information generated by the generation unit 154.

[0038] Furthermore, the information processing device 100 may have a functional configuration for outputting information, not limited to the display unit 13. The information processing device 100 may have a function for outputting information as sound. For example, the information processing device 100 may have an audio output unit such as a speaker that outputs sound.

[0039] The storage unit 14 may be, for example, a RAM (Random Access Memory) or a flash memory. The storage unit 14 is realized by a semiconductor memory element such as a memory (RAM) or a storage device such as a hard disk or an optical disk. The storage unit 14 stores various information required for processing. For example, the storage unit 14 stores various information such as information required for displaying information such as an image, and information required for selection processing. The storage unit 14 stores information on edge detection results. For example, the storage unit 14 stores information to be displayed based on the detection results, such as a boundary ED.

[0040] For example, the storage unit 14 stores images that are the subject of edge detection, point selection, region selection, region generation, etc. For example, the storage unit 14 stores various images such as surgical images (surgical videos). For example, the storage unit 14 stores information regarding the positions of selected points such as selected points and confirmed selected points. For example, the storage unit 14 stores information regarding the arrangement of lines connecting multiple selected points. Furthermore, for example, the storage unit 14 stores information regarding the arrangement of regions such as selected regions and generated regions. Note that the above is merely an example, and the storage unit 14 stores various information necessary for displaying (presenting) information to the user.

[0041] Furthermore, for example, the storage unit 14 stores information on a voice recognition application (program) that realizes a voice recognition function. For example, the information processing device 100 can execute voice recognition by activating a voice recognition application (also simply referred to as "voice recognition"). The storage unit 14 stores various information used for voice recognition. The storage unit 14 stores information on dictionaries (voice recognition dictionaries) used in the voice recognition dictionary. The storage unit 14 stores information on multiple voice recognition dictionaries.

[0042] The control unit 15 is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 using a RAM (Random Access Memory) or the like as a work area. The control unit 15 is a controller, and may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0043] 3, control unit 15 has an acquisition unit 151, a detection unit 152, a selection unit 153, a generation unit 154, and a transmission unit 155, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 15 is not limited to the configuration shown in FIG. 3, and may be any other configuration that performs the information processing described below.

[0044] The acquisition unit 151 acquires various types of information. The acquisition unit 151 acquires various types of information from an external information processing device. The acquisition unit 151 acquires various types of information from the storage unit 14. The acquisition unit 151 acquires information accepted by the input unit 12.

[0045] The acquisition unit 151 acquires various pieces of information detected by the detection unit 152. The acquisition unit 151 acquires various pieces of information selected by the selection unit 153. The acquisition unit 151 acquires various pieces of information generated by the generation unit 154. The acquisition unit 151 acquires sensor information detected by the sensor unit 16.

[0046] The acquisition unit 151 acquires gaze information indicating a user's gaze point and object information indicating a plurality of objects. The acquisition unit 151 acquires object information indicating boundaries of a plurality of objects. The acquisition unit 151 acquires object information indicating boundaries of a plurality of objects detected from an image. The acquisition unit 151 acquires object information indicating boundaries of a plurality of objects detected from each of a plurality of images.

[0047] The acquisition unit 151 acquires gaze information indicating a user's gaze point and area information indicating multiple areas based on the arrangement of an object. The acquisition unit 151 acquires area information indicating multiple areas based on the boundaries of an object. The acquisition unit 151 acquires designation information indicating designation by the user for multiple partial areas. The acquisition unit 151 acquires designation information indicating designation by the user's gaze for multiple partial areas. The acquisition unit 151 acquires designation information indicating designation of a range within one area by the user's gaze.

[0048] The detection unit 152 performs detection processing. The detection unit 152 detects various types of information. The detection unit 152 detects various types of information based on information acquired from an external information processing device. The detection unit 152 detects various types of information based on information stored in the storage unit 14.

[0049] The detection unit 152 detects various types of information from an image by appropriately using techniques such as image analysis. The detection unit 152 detects boundaries in an image by appropriately using techniques such as image analysis. The detection unit 152 detects boundaries by appropriately using edge detection techniques. Note that the detection unit 152 is not limited to edge detection and may detect boundaries using algorithms such as segmentation detection and depth map.

[0050] The selection unit 153 performs a selection process. The selection unit 153 makes various selections. The selection unit 153 selects various pieces of information based on information stored in the storage unit 14. The selection unit 153 selects various pieces of information based on information acquired by the acquisition unit 151. The selection unit 153 selects various pieces of information based on information detected by the detection unit 152.

[0051] The selection unit 153 makes various determinations. The selection unit 153 makes various determinations based on information stored in the storage unit 14. The selection unit 153 makes various determinations based on information acquired by the acquisition unit 151. The selection unit 153 makes various determinations based on information detected by the detection unit 152.

[0052] The selection unit 153 selects one object from the plurality of objects based on the positional relationship between the user's gaze point and the plurality of objects. The selection unit 153 selects the one object that is the nearest object from the user's gaze point.

[0053] The selection unit 153 selects a point on the boundary of the one object that is closest to the user's point of gaze, and selects a plurality of points on the boundary of the one object in accordance with the movement of the user's point of gaze.

[0054] The selection unit 153 selects one of the plurality of regions based on the positional relationship between the user's point of gaze and the plurality of regions. The selection unit 153 selects one of the plurality of regions, which is the region where the user's point of gaze is located.

[0055] The selection unit 153 selects one area corresponding to an object within whose boundary the user's gaze point is located, and selects the one area as an area to be processed for editing.

[0056] The selection unit 153 selects a partial region corresponding to a user's specification from among the plurality of partial regions. The selection unit 153 selects a partial region corresponding to a user's specification based on the user's line of sight. The selection unit 153 selects a partial region located within a range from among the plurality of partial regions.

[0057] The generation unit 154 performs a generation process to generate various types of information. The generation unit 154 generates various types of information based on information stored in the storage unit 14. The generation unit 154 generates various types of information based on information acquired by the acquisition unit 151. The generation unit 154 generates various types of information based on information selected by the selection unit 153. The generation unit 154 generates an image indicating the edge detection results.

[0058] The generation unit 154 generates an area whose boundary is the line connecting the multiple points by connecting the multiple points with a line in the selected order. The generation unit 154 changes the area by changing the position of a point among the multiple points. The generation unit 154 changes the area by adding a new point to the multiple points.

[0059] The generation unit 154 generates a mesh that divides one region into a plurality of partial regions. The generation unit 154 generates a mesh that includes a plurality of partial regions that divide one region into rectangles.

[0060] The generation unit 154 may generate various types of information to be displayed on the display unit 13. The generation unit 154 may generate various types of information, such as character information and image information such as graphs, to be displayed on the display unit 13. In this case, the generation unit 154 generates information (images) related to the screen by appropriately using various conventional techniques related to images. The generation unit 154 generates images by appropriately using various conventional techniques related to GUIs. For example, the generation unit 154 may generate images using CSS, JavaScript (registered trademark), HTML, or any language capable of describing information processing such as the above-mentioned information display and operation reception.

[0061] The transmitting unit 155 transmits various types of information. The transmitting unit 155 provides various types of information. The transmitting unit 155 provides various types of information to an external information processing device. The transmitting unit 155 transmits various types of information to an external information processing device. The transmitting unit 155 transmits information stored in the memory unit 14. The transmitting unit 155 transmits information acquired by the acquiring unit 151. The transmitting unit 155 transmits the detection result by the detecting unit 152. The transmitting unit 155 transmits information selected by the selecting unit 153. The transmitting unit 155 transmits information generated by the generating unit 154.

[0062] Note that each of the processes performed by the control unit 15 described above may be realized by, for example, JavaScript (registered trademark). Furthermore, when the information processing and other processes performed by the control unit 15 described above are performed by a predetermined application, each unit of the control unit 15 may be realized by, for example, the predetermined application. For example, the information processing and other processes performed by the control unit 15 may be realized by control information received from an external information processing device. For example, when the display process described above is performed by a predetermined application (e.g., an information output application), the control unit 15 may have, for example, an application control unit that controls the predetermined application or a dedicated application.

[0063] The sensor unit 16 has various sensors for detecting various objects. In the example of Fig. 3, the sensor unit 16 has a position and orientation sensor 161 and a gaze sensor 162. The sensor unit 16 also has an audio sensor such as a microphone for detecting audio.

[0064] The position and orientation sensor 161 detects the position and orientation of the user using a camera, a gyro sensor, or the like. For example, the position and orientation sensor 161 detects the position and orientation of the user's head. For example, the position and orientation sensor 161 detects changes in the position and orientation of the user's head using an image of the user's head captured by a camera. Note that the position and orientation sensor 161 may have any configuration as long as it can detect the state of the user's head required for processing.

[0065] The gaze sensor 162 detects the gaze of the user. The gaze sensor 162 detects the gaze direction of the user using eye tracking technology based on the detection results of a camera, an optical sensor, a motion sensor (none of which are shown) mounted on the terminal device 10, for example. The gaze sensor 162 determines a gaze area on the screen where the user is gazing based on the detected gaze direction. The gaze sensor 162 transmits gaze information including the determined gaze area to the information processing device 100. Note that the above is merely an example, and the gaze sensor 162 may have any configuration as long as it can detect the gaze information of the user required for processing.

[0066] The sensor unit 16 has various sensors other than the position and orientation sensor 161 and the gaze sensor 162. The sensor unit 16 has a sensor (biological signal sensor) for detecting a biosignal of the user. For example, the terminal device 10 may have sensors for detecting various biosignals, such as a heartbeat sensor for detecting the heartbeat of the user, an electroencephalogram sensor for detecting the electroencephalogram of the user, a pulse sensor (pulse wave sensor) for detecting the pulse of the user, a respiration sensor (exhalation sensor) for detecting the respiration of the user, and a sweat sensor for detecting the sweat of the user.

[0067] The sensor unit 16 also has a camera (image sensor) that detects images. For example, the sensor unit 16 has an image sensor for capturing an image of the user. The sensor unit 16 may have multiple cameras (image sensors). For example, the sensor unit 16 may have a first image sensor for capturing a surgical image and a second image sensor for capturing an image of the user. For example, the sensor unit 16 may have a motion sensor.

[0068] For example, the sensor unit 16 has an acceleration sensor. For example, the sensor unit 16 has a sensor that detects acceleration information of the terminal device 10 when a predetermined operation is performed by the user. The sensor unit 16 has a sensor (position sensor) that detects the position of the terminal device 10. For example, the sensor unit 16 may have a GPS (Global Positioning System) sensor. When acquiring the location information of the terminal device 10 as sensor information, the sensor unit 16 may acquire the location information of the base station with which the terminal device 10 is communicating or the location information of the terminal device 10 estimated using WiFi (registered trademark) (Wireless Fidelity) radio waves.

[0069] Furthermore, the sensor unit 16 is not limited to the above, and may include various other sensors. For example, the sensor unit 16 may include a sensor that detects information outside the terminal device 10.

[0070] The sensors in the sensor unit 16 that detect the above-mentioned various pieces of information may be a common sensor, or may be realized by different sensors.

[0071] [1-3. Information Processing Procedure According to the Embodiment] Next, the procedure of information processing according to the embodiment will be described with reference to FIGS.

[0072] First, the flow of the selection process performed by the information processing device according to the embodiment of the present disclosure will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the procedure of the selection process according to the embodiment of the present disclosure.

[0073] 4, the information processing device 100 acquires gaze information indicating a user's gaze point and object information indicating a plurality of objects (step S101). The information processing device 100 selects one of the plurality of objects based on the positional relationship between the user's gaze point and the plurality of objects (step S102).

[0074] Next, a process flow of determining an area by selecting multiple points performed by an information processing device according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the procedure for determining an area by selecting multiple points.

[0075] 5, the information processing device 100 executes edge detection processing (step S201). For example, the information processing device 100 executes edge detection processing on a captured surgical image.

[0076] Then, the information processing device 100 selects a point on the edge that is closest to a pointer (point) related to the user's line of sight (step S202). For example, the information processing device 100 selects a point on the edge that is closest to the user's point of gaze. For example, the information processing device 100 may set the point of gaze or a position fine-tuned with the head based on the point of gaze position as the pointer position.

[0077] Then, the information processing device 100 determines whether or not the position has been determined (step S203). For example, the information processing device 100 determines whether or not the user has decided to use the position selected in step S202. If the position has not been determined (step S203: No), the information processing device 100 returns to step S202 and repeats the process.

[0078] On the other hand, when the position is determined (step S203: Yes), the information processing device 100 determines whether the position matches the start position (step S204). For example, when the user determines that the selected position will be used, the information processing device 100 determines whether the position matches the start position, i.e., the position initially selected in the selection process. When the determined position does not match the start position (step S204: No), the information processing device 100 returns to step S202 and repeats the process.

[0079] On the other hand, if the determined position matches the start position (step S204: Yes), the information processing device 100 confirms the area (step S205). For example, if the determined position matches the start position, the information processing device 100 confirms the area surrounded by the determined points as the selected area.

[0080] In the above example, a case has been described in which the information processing device 100 selects (generates) an area based on a plurality of points selected based on the user's gaze point, but the information processing device 100 may select an area based on the user's gaze point. This point will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an area selection procedure according to an embodiment of the present disclosure. For example, FIG. 6 is a diagram illustrating the flow of area selection processing performed by the information processing device according to an embodiment of the present disclosure.

[0081] 6, the information processing device 100 acquires gaze information indicating a user's gaze point and region information indicating a plurality of regions based on the arrangement of objects (step S301). The information processing device 100 selects one of the plurality of regions based on the positional relationship between the user's gaze point and the plurality of regions (step S302).

[0082] [1-4. Processing example] From here, examples of various processes will be described with reference to Figures 7 to 15. Note that explanations of points similar to those in Figure 1 will be omitted where appropriate.

[0083] [1-4-1. Area selection example] First, the area selection corresponding to the processing flow of Fig. 6 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the area selection processing.

[0084] 7, the information processing device 100 detects a boundary ED (corresponding to, for example, the thick line portion in the image DP11) as shown in the image DP11 by edge detection. This is the same as the image DP11 in FIG. 1, so a description thereof will be omitted.

[0085] The information processing device 100 selects one of the multiple regions based on the positional relationship between the gaze point of the user U1 and the multiple regions (step S21). For example, the information processing device 100 selects, as one region (also referred to as a "selected region"), the region where the gaze point of the user U1 is located, from the multiple regions formed by the boundary ED that separates the multiple objects OB1 to OB4, etc.

[0086] For example, the information processing device 100 selects, as the selected region, the region where the gaze point of the user U1 is located from among a plurality of regions, such as regions corresponding to each of a plurality of objects OB1 to OB4. In the example of FIG. 7, as shown in image DP21, the gaze point SP21 of the user U1 is located in the region corresponding to object OB4, and therefore the information processing device 100 selects, as the selected region, the region corresponding to object OB4, which is part of an organ. That is, the information processing device 100 selects, as the selected region, the region formed by the boundary ED, which is the region corresponding to object OB4, which is part of an organ. In this way, the information processing device 100 can flexibly select a target according to the user's line of sight.

[0087] Then, upon confirming that the area corresponding to object OB4 has been selected, user U1 performs an operation (confirmation operation) to confirm (decide) the area as the selected area. For example, as shown in image DP21, user U1 performs a confirmation operation to set the area corresponding to object OB4 as the confirmed selected area (also referred to as the "confirmed selected area") by uttering UT1 such as "select here" while the gaze point SP21 is located in the area corresponding to object OB4. In this case, the information processing device 100 confirms (decides) the area corresponding to object OB4 as the selected area (confirmed selected area) in response to detection of user U1's utterance UT1. In this way, user U1 speaks while looking at the candidate area to be meshed. As a result, the information processing device 100 selects the area corresponding to object OB4 as the area to be processed for editing.

[0088] Then, the information processing device 100 generates a mesh that divides the selected region into a plurality of partial regions (step S22). In the example of FIG. 7, the information processing device 100 generates a mesh MS that divides the region corresponding to the object OB4 into a plurality of partial regions SA. For example, the information processing device 100 generates a mesh MS that divides the region corresponding to the object OB4 into a plurality of partial regions SA that are separated by rectangles. Then, as shown in image DP22, the information processing device 100 displays the generated mesh MS superimposed on the region corresponding to the object OB4. Note that the generation of the mesh and the superimposed display are performed using known mesh-related technology, and therefore detailed description thereof will be omitted.

[0089] As described above, the information processing device 100 meshes an area where the gaze point is located if it is a closed space. This allows the information processing device 100 to select any area and perform desired editing, such as filling it with a mesh MS. Note that the mesh MS shown on the screen DP22 in Fig. 7 is merely an example, and the size or style of the mesh MS may be changeable to any format.

[0090] For example, the information processing device 100 may allow the user to specify the size or style of the mesh MS by voice. In this case, if the user U1 utters "coarse mesh," the information processing device 100 changes the style of the mesh MS to a coarse format. If the user U1 utters "fine mesh," the information processing device 100 changes the style of the mesh MS to a fine format. If the user U1 utters "triangular mesh," the information processing device 100 changes the shape of the partial area SA of the mesh MS from a rectangle (quadrilateral) to a triangle. Also, for example, the information processing device 100 may generate a mesh to conform to the three-dimensional shape of the object using a method such as depth measurement or machine learning.

[0091] The information processing device 100 may use both the area selection process described in Figures 6 and 7 and the point selection process (also called "point selection") shown in Figure 1, or may switch between the area selection process mode and the point selection process mode.

[0092] For example, when operating with the above-mentioned free pointing, the information processing device 100 may operate in the point selection mode described in Fig. 1, and when operating with the above-mentioned free pointing, it may operate in the area selection mode described in Fig. 6 and Fig. 7. Note that the above is merely an example, and the information processing device 100 may operate by appropriately switching between the area selection processing mode and the point selection processing mode using various information, for example, designation by the user, as long as it is possible to realize the desired operation.

[0093] [1-4-2. Example of selection by range specification] The information processing device 100 may specify an area by selecting two points: a voice and a line of sight or a head direction. In this regard, selection using the area selected in FIG. 7 will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of selection processing by range specification. Note that the same points as in FIG. 7 will not be described as appropriate.

[0094] Image DP23 in Fig. 8 is image DP22 in Fig. 7 with user U1's gaze point SP22 displayed. After confirming that the point at the position of gaze point SP22 has been selected, user U1 performs an operation (confirmation operation) to confirm (decide) point PT20 at the position of gaze point SP22 as the first point. For example, user U1 performs a confirmation operation to set point PT20 at the position of gaze point SP22 as the confirmed first point by uttering UT11 such as "From here." In this case, the information processing device 100 confirms (decides) point PT20 at the position of gaze point SP22 as the first point in response to detection of user U1's utterance UT11.

[0095] Thereafter, user U1 changes his / her line of sight to select another point. Image DP24 in FIG. 8 shows the position of user U1's gaze point SP23 after user U1's line of sight has been changed. After confirming that the point at the position of gaze point SP23 has been selected, user U1 performs an operation (confirmation operation) to confirm (decide) the point at the position of gaze point SP23 as the second point. For example, user U1 performs the confirmation operation to confirm (decide) the point at the position of gaze point SP23 as the confirmed second point by uttering UT12 such as "Up to here." In this case, the information processing device 100 confirms (decides) the point at the position of gaze point SP23 as the second point in response to detection of user U1's utterance UT12.

[0096] After determining the first and second points, the information processing device 100 selects only a portion corresponding to the mesh MS within a rectangular region having a diagonal line connecting the first and second points (step S23). In the example of Fig. 8, as shown in image DP24, the information processing device 100 selects only a portion of the plurality of partial regions SA within the mesh MS that overlaps with a rectangular region (dotted rectangle) having a diagonal line connecting the first point PT20 and the position of the second point, the point of interest SP23. Note that the example of Fig. 8 shows an example in which only a portion of the plurality of partial regions SA within the mesh MS that completely overlaps with a rectangular region (dotted rectangle) having a diagonal line connecting the first point PT20 and the position of the second point, the point of interest SP23. In addition, the information processing device 100 may select one of the multiple partial areas SA within the mesh MS that overlaps at least partially with a rectangular area (dotted rectangle) whose diagonal is a line connecting the first point, point PT20, and the second point, the point at the position of the gaze point SP23.

[0097] As described above, the information processing device 100 selects only a partial area AR3 of the mesh MS within a rectangular area whose diagonal is a line connecting the point PT20 and the position of the gaze point SP23. As a result, the information processing device 100 selects only a portion of the multiple partial areas SA of the mesh MS that overlaps with the rectangular area whose diagonal is a line connecting any two points.

[0098] [1-4-3. Example of continuous selection of partial areas] The selection of partial regions is not limited to the process shown in FIG. 8, and various other processes may be used. For example, the information processing device 100 may continuously select a trajectory in the direction of the gaze point or head. This will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the process of continuously selecting partial regions. Note that, although the following describes a case where a region is selected in accordance with a change in the position of the gaze point, a region may also be selected in accordance with a change in the position or posture of the head.

[0099] As shown in image DP31, the information processing device 100 selects an area in response to a change in the position of the gaze point of user U1. For example, user U1 may change the position of their gaze (gazing point) by moving only their eyes, or may change the position or posture of their head to change the position of their gaze (gazing point). For example, user U1 may change their gaze (gazing point) by continuously adjusting the position of their head. As a result, the information processing device 100 selects area AR11 as shown in image DP31.

[0100] Then, the information processing device 100 selects an area in accordance with the change in the position of the point of gaze of the user U1 (step S31). In the example of Fig. 9, the information processing device 100 selects an area AR12 as shown in an image DP32. That is, the information processing device 100 expands the selected area from the area AR11 to the area AR12.

[0101] Furthermore, the information processing device 100 selects an area in accordance with a change in the position of the point of gaze of the user U1 (step S32). In the example of Fig. 9, the information processing device 100 selects an area AR13 as shown in an image DP33. That is, the information processing device 100 expands the selected area from the area AR12 to the area AR13.

[0102] As described above, the information processing device 100 continuously expands and adjusts the area in response to changes in the position of the user's point of gaze. For example, when using line of sight, the information processing device 100 may not draw for a certain period of time if the point of gaze temporarily moves outside the mesh. Furthermore, the information processing device 100 may be able to draw any shape by changing the mesh size.

[0103] [1-4-4. Example of changing the selection range] The selection range may be changed as appropriate. This will be explained with reference to Fig. 10. Fig. 10 is a diagram showing an example of changing the selection range.

[0104] For example, the information processing device 100 may adjust the brush thickness according to the tilt of the head of the user U1. The information processing device 100 may increase the selection width (brush size, etc.) for selecting an area as the tilt of the head of the user U1 increases.

[0105] 10 shows a state in which the selection width is small. In this case, the information processing device 100 selects a region corresponding to the small-sized fixation point SP31. For example, the information processing device 100 selects a partial region SA through which the small-sized fixation point SP31 has passed, such as region AR14, from among the multiple partial regions SA of the mesh MS.

[0106] 10 shows a state in which the selection width is large. In this case, the information processing device 100 selects a region corresponding to the large-sized fixation point SP32. For example, the information processing device 100 selects a partial region SA, such as region AR15, that is small among the multiple partial regions SA of the mesh MS and through which the fixation point SP32 has passed.

[0107] The information processing device 100 may reduce the size of the selection width depending on the tilt of the head of the user U1. In this way, the information processing device 100 increases or decreases the selection width depending on the tilt of the head of the user U1. This allows the information processing device 100 to arbitrarily set the selection width and select an area.

[0108] Furthermore, as described above, there are cases where the subject moves or the viewpoint of the camera moves during the selection operation. In such cases, to prevent an unintended area from being filled in (selected), the information processing device 100 may keep the mesh fixed on the screen so that the filling position (selection position) does not shift on the screen. For example, the information processing device 100 may use an existing method for performing a matching process of feature points between video frames and tracking them as a method for realizing such processing.

[0109] [1-4-5. Editing example] Furthermore, the area generated by selecting a plurality of points may be edited as appropriate, which will be described with reference to FIGS.

[0110] [1-4-5-1. Changing the selected point] First, editing of an area by adjusting the position of a selected point will be described as an example of changing a point using Fig. 11. Fig. 11 is a diagram showing an example of editing by changing a point. Note that the change of a point is not limited to adjusting (changing) the position of the selected point, and may also be deletion of the position of the selected point, etc.

[0111] 11 shows an example in which seven points PT21 to PT27 have been confirmed (determined) as confirmed selected points. For example, the information processing device 100 selects seven points in the order of points PT21, PT22, PT23, PT24, PT25, PT26, and PT27, and generates an area AR21 by connecting them in that order with a line LN. That is, the information processing device 100 generates an area AR21 with point PT21 as the start point, point PT27 as the end point, and a line LN connecting the multiple points PT21 to PT27 as the boundary. For example, the multiple points PT21 to PT27 are points specified when the area was set.

[0112] Then, the information processing device 100 selects a point to be changed from among the multiple points PT21 to PT27 according to the line of sight of the user U1. For example, the information processing device 100 selects a point that overlaps the position of the point of gaze of the user U1 as the point to be changed. In FIG. 11, the information processing device 100 selects point PT23 corresponding to the position of the point of gaze of the user U1 as the point to be changed. In this way, the information processing device 100 selects an object by line of sight.

[0113] For example, the user U1 selects the point PT23 as the point to be changed by uttering an utterance UT21 such as "select here" while the gaze point is overlapped with the point PT23. The information processing device 100 selects the point PT23 as the point to be changed in response to the utterance UT21 of the user U1.

[0114] For example, if the user U1 wishes to move the position of point PT23, the user U1 changes the position of the point of gaze. The information processing device 100 moves the position of point PT23 in accordance with the change in the position of the point of gaze of the user U1. Note that the user U1 may change the position of the line of sight (point of gaze) by moving only their eyes, or may change the position or posture of their head to change the position of their line of sight (point of gaze). In this way, the information processing device 100 may continuously adjust the position (of the point) in the direction of their head.

[0115] Furthermore, if the user U1 wishes to delete point PT23, the user U1 selects point PT23 as the point to be changed and utters an utterance such as "delete this." When the information processing device 100 detects an utterance such as "delete this" from the user U1 with point PT23 selected as the point to be changed, the information processing device 100 deletes point PT23. In this case, the information processing device 100 excludes point PT23 from the multiple points PT21 to PT27. Then, the information processing device 100 updates the area AR21 to a shape in which six points are connected by a line LN in the order of points PT21, PT22, PT24, PT25, PT26, and PT27.

[0116] [1-4-5-2. Adding points] Next, editing of an area by adding a new point will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of editing by adding a point. Specifically, Fig. 12 is a diagram showing an example of editing by adding a new point. Note that a description of the same points as in Fig. 11 will be omitted.

[0117] User U1 selects, with his / her line of sight, a position where he / she wants to add a new point. For example, user U1 selects, with his / her line of sight, a position where he / she wants to add a new point on line LN, which is a boundary that forms area AR21. In the example of FIG. 12, user U1 desires to add a new point between points PT24 and PT25. Note that, if the gaze point of user U1 does not overlap with line L, the information processing device 100 may select a position (point) on line LN that is closest to the gaze point of user U1.

[0118] In this case, the information processing device 100 adds a new point NP1 on the line LN according to the line of sight of the user U1. In FIG. 12, the information processing device 100 adds the point NP1 at the position of the gaze point of the user U1. In this case, the information processing device 100 generates an area AR21 whose boundary is a line LN that connects multiple points PT21, PT22, PT23, PT24, NP1, PT25, PT26, and PT27 in this order. The user U1 may change the position of the newly added point NP1 by performing an operation such as that shown in FIG. 11, thereby changing the shape of the area AR21.

[0119] [1-4-6. Example of multiple layers] Furthermore, the information processing device 100 may perform selection processing using not only one layer (image) but also multiple layers (images), as will be described with reference to FIGS.

[0120] [1-4-6-1. Selection example 1 (medical images)] First, a selection process using multiple layers of selected points in a case where a medical image is the target will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of selection using multiple layers.

[0121] Image DP51 in FIG. 13 corresponds to layer A and is an image (layer) containing information about boundary ED detected by edge detection, as in FIG. 1. Image DP52 in FIG. 13 corresponds to layer B and is an image (layer) containing information about an object (organ) hidden behind a portion of another object (organ). In the example of FIG. 13, image DP52 contains boundary ED2 indicating the edge of an object hidden behind a portion of another object (organ). For example, image DP52 may be 3D (three-dimensional) information generated from CT (Computed Tomography) scan data or the like.

[0122] The information processing device 100 generates information in which information on layer A and information on layer B are superimposed, as shown in image DP53 (step S41). The information processing device 100 may display image DP53 showing information on layer A and information on layer B. Then, the information processing device 100 may accept a selection by user U1, with both information on layer A (e.g., boundary ED) and information on layer B (e.g., boundary ED2) as selection targets.

[0123] For example, user U1 may make an utterance UT31 such as "I will use layer A and layer B," thereby making both information on layer A (e.g., boundary ED) and information on layer B (e.g., boundary ED2) available for selection. In this case, the information processing device 100, in response to detection of user U1's utterance UT31, accepts user U1's selection with both information on layer A (e.g., boundary ED) and information on layer B (e.g., boundary ED2) available for selection. For example, the information processing device 100 may take an OR (logical sum) of layer A and layer B as edge information (boundary information). Then, the information processing device 100 may display edge information (boundary information) that is the logical sum of layer A and layer B.

[0124] Furthermore, the information processing device 100 may switch the information of the layer to be selected in response to a specification by the user U1. For example, when the user U1 makes an utterance specifying a single layer switch, such as "only layer B," the information processing device 100 may set only the information of one layer as the selection target for the user U1. The information processing device 100 may specify a layer in the depth direction by moving the user U1's head forward or backward. In this case, when the user U1 moves his / her head forward, the information processing device 100 may select the layer at the back (e.g., layer B) as the selection target, and when the user U1 moves his / her head forward, the information processing device 100 may select the layer at the front (e.g., layer A) as the selection target. In this way, the information processing device 100 may switch the layer to be selected in response to a change in the position or posture of the user U1's head.

[0125] When the information processing device 100 detects the user U1's utterance of "only layer B," it accepts the user U1's selection with both information on layer B (e.g., boundary ED2) as the selection target. In this case, the information processing device 100 may display only the information on layer B (i.e., image DP52), or may display both information on layer A (e.g., boundary ED) and information on layer B (e.g., boundary ED2) (i.e., image DP53). In this way, the user U1 selects the desired target by selecting points or areas while specifying the selection target from multiple layers.

[0126] 13, the information processing device 100 selects a plurality of points including points on the boundary ED2 of an object hidden behind a part of another object (organ) by the user U1 making a selection using information on the boundary ED2 of layer B. As a result, the information processing device 100 generates an area AR31 whose boundary is a line LN that connects the plurality of points selected by the user U1 using the plurality of layers, as shown in image DP54 (step S42).

[0127] 13, the information processing device 100 can specify an area by overlapping layers of information in the depth direction. That is, the information processing device 100 can select an area including the deep part by using multiple pieces of layer information.

[0128] The information processing device 100 may use, as one layer, edge information (boundary information) detected by each edge detection algorithm, such as an algorithm for edge detection, segmentation detection, depth map, etc. For example, the information processing device 100 may use the edge information (boundary information) detected by edge detection as a first layer (e.g., layer A) and the edge information (boundary information) detected by segmentation detection as a second layer.

[0129] [1-4-6-2. Selection example 2 (map)] The above-described selection by switching between multiple layers may be applied to various information other than surgical images. This will be briefly described with reference to Figures 14 and 15. Note that the same points as those in Figure 13 will not be described again.

[0130] First, a selection process using multiple layers for a selected point when a map is used will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of selection using multiple layers.

[0131] 14 shows an image corresponding to layer A, which is an image (layer) including building information among the map information. For example, image DP61 includes information about the edges (boundaries) of buildings among the map information.

[0132] 14 indicates an image corresponding to layer B, which is an image (layer) including road information among the map information. For example, image DP62 includes road information among the map information.

[0133] The information processing device 100 generates information in which information on layer A and information on layer B are superimposed, as shown in image DP63 (step S51). The information processing device 100 may display image DP63 showing information on layer A and information on layer B. Then, the information processing device 100 may accept a selection by user U1, with both information on layer A (for example, edges of buildings) and information on layer B (for example, roads) as selection targets.

[0134] In the example of Fig. 14, the information processing device 100 selects a building area by user U1 making a selection using information on building boundaries on layer A, and selects a road by user U1 making a selection using information on roads on layer B. As a result, as shown in image DP64, the information processing device 100 generates an area AR31 formed by a line LN1 connecting points selected using information on layer A, and a line LN2 connecting points selected using information on layer B (step S52). Note that the line LN2 may be a line connecting multiple selected points in the order of selection, or may be a road (line) corresponding to the selected points.

[0135] 14, the information processing device 100 can make a selection from a plurality of layers of map information, each of which has a different target type. Specifically, the information processing device 100 can surround (select) a building using information on layer A and draw (select) a route using information on layer B. That is, the information processing device 100 can make a selection from a map using information on a plurality of layers.

[0136] [1-4-6-3. Selection example 3 (items)] Next, a selection process using multiple layers of selected points when an object is the target will be described with reference to FIG. 15. FIG. 15 is a diagram showing an example of selection using multiple layers. Specifically, FIG. 15 is a diagram showing an example of selection using multiple layers for an automobile. For example, FIG. 15 shows an example of selecting an intended location in a 3D modeling application. Note that the object is not limited to an automobile, and may be a structure (real estate) such as a house, or various other objects such as clothing, etc.

[0137] Image DP71 in FIG. 15 corresponds to layer A and is an image (layer) containing information about the car body on a part-by-part basis (part information). Image DP72 in FIG. 15 corresponds to layer B and is an image (layer) containing 3D mesh information about the car body. Note that images DP71, DP72, and DP75 in FIG. 15 illustrate only the portion necessary for explanation, i.e., the front part of the car body, but images DP71, DP72, and DP75 contain information about the entire car body.

[0138] As shown in images DP73 and DP74, the information processing device 100 switches between displaying information on layer A and information on layer B (step S61). Then, the information processing device 100 may accept a selection by user U1, using the displayed layer information as a selection target, between information on layer A (e.g., edges of parts of the car body) and information on layer B (e.g., a 3D mesh on the car body). In this way, in the example of Fig. 15, the information processing device 100 allows user U1 to select a point or an area while switching between information on layer A and information on layer B.

[0139] For example, the information processing device 100 selects points PT31 and PT32 by allowing the user U1 to make selections using information for each part of the car body on layer A. The user U1 can easily select the boundaries between the parts of the car body on layer A.

[0140] Furthermore, for example, the information processing device 100 selects points PT33 and PT34 by having the user U1 make a selection using information on the 3D mesh of the car body on layer B. The user U1 can select a detailed position of the car body on layer B. As a result, the information processing device 100 generates an area using the points selected using the information on layer A and the information on layer B (step S62). In the example of FIG. 15, the information processing device 100 generates an area AR41 formed by a line LN11 connecting the points (points PT31 to PT34, etc.) selected using the information on layer A and the information on layer B, as shown in image DP75. Furthermore, the information processing device 100 generates an area AR42 formed by a line LN12 connecting the points selected using the information on layer A and the information on layer B, as shown in image DP75.

[0141] 15, the information processing device 100 can perform selection across multiple layers of different types, even if the layers are information about the same car body. Specifically, the information processing device 100 allows the user U1 to select the boundary between parts using information on layer A, and allows the user U1 to select a detailed position of the car body using information on layer B. In other words, the information processing device 100 can perform selection across an object such as a car body using information on multiple layers. This allows the information processing device 100 to select a complex curved surface portion.

[0142] [2. Other embodiments] The processing according to each of the above-described embodiments may be implemented in various different forms (modifications) other than the above-described embodiments. For example, the system configuration is not limited to the above-described examples, and may be in various forms. This point will be described below. Note that, in the following, explanations of points similar to those of the information processing device 100 according to the embodiment will be omitted as appropriate.

[0143] [2-1. Modifications] For example, in the above example, an example was shown in which the information processing device 100, which is a terminal device used by a user, performs part-of-speech estimation, but the information processing device that performs part-of-speech estimation may be separate from the terminal device used by the user. This point will be explained using Figures 16 and 17. Figure 16 is a diagram showing an example configuration of an information processing system according to a modified example of the present disclosure. Figure 17 is a diagram showing an example configuration of an information processing device according to a modified example of the present disclosure.

[0144] As shown in FIG. 16, the information processing system 1 includes a display device 20, a sensor device 50, and an information processing device 100A. The display device 20, the sensor device 50, and the information processing device 100A are communicably connected via a communication network N, either wired or wirelessly. The information processing system 1 shown in FIG. 16 may include a plurality of display devices 20, a plurality of sensor devices 50, and a plurality of information processing devices 100A. In this case, the information processing device 100A communicates with the display device 20 and the sensor device 50 via the communication network N, provides information to the display device 20, and acquires various sensor information detected by the sensor device 50. The information processing device 100A also acquires user input and images to be processed via the sensor device 50.

[0145] The sensor device 50 has a sensor similar to the sensor unit 16 of the information processing device 100 described above. The sensor device 50 has sensors that detect various sensors. For example, the sensor device 50 has a position and orientation sensor 161 and a gaze sensor 162. Furthermore, for example, the sensor device 50 has an audio sensor that detects audio, such as a microphone. Note that if each sensor is configured individually, the information processing system 1 has multiple sensor devices 50. In this case, for example, the information processing system 1 has multiple sensor devices 50, such as a sensor device 50 that detects the position and orientation of a user, a sensor device 50 that detects the gaze of a user, and a sensor device 50 that detects the user's speech.

[0146] The display device 20 has a display unit (for example, a liquid crystal display) for displaying various types of information. The display device 20 displays information received from the information processing device 100A. The display device 20 displays images received from the information processing device 100A. The display device 20 displays selection results received from the information processing device 100A.

[0147] The information processing device 100 is a computer that transmits various types of information to the display device 20. The information processing device 100 is a server device used to provide services to users. The information processing device 100A realizes information processing similar to that of the information processing device 100, except that the information processing device 100A provides information to the display device 20 and acquires information from the terminal device 10. The information processing device 100A is a server that provides services to the display device 20 using various types of sensor information detected by the sensor device 50. For example, the information processing device 100A executes a selection process based on the information detected by the sensor device 50, and transmits the execution result to the display device 20.

[0148] 17, the information processing device 100A has a communication unit 11, a storage unit 14, and a control unit 15A. The communication unit 11 is connected to a communication network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from a display device 20 and a sensor device 50 via the communication network N. In this case, the information processing device 100A does not need to have a function to display information like the information processing device 100. Note that the information processing device 100A may have an input unit (for example, a keyboard, a mouse, etc.) and a display unit (for example, a liquid crystal display, etc.) used by an administrator of the information processing device 100A, etc.

[0149] The control unit 15A is realized, for example, by a CPU, an MPU, or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100A using a RAM, etc. as a work area. The control unit 15A may also be realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0150] 17, control unit 15A has an acquisition unit 151A, a detection unit 152, a selection unit 153, a generation unit 154, and a transmission unit 155A, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 15A is not limited to the configuration shown in FIG. 17, and may be any other configuration as long as it performs the information processing described below.

[0151] The acquisition unit 151A acquires various types of information in the same manner as the acquisition unit 151. The acquisition unit 151A acquires various types of information from the terminal device 10. The acquisition unit 151A acquires various types of sensor information from the sensor device 50. The acquisition unit 151A acquires various types of information from the storage unit 14.

[0152] The transmission unit 155A provides various types of information in the same manner as the transmission unit 155. The transmission unit 155A provides various types of information to the display device 20. The transmission unit 155A transmits various types of information to the display device 20. The transmission unit 155A provides the information generated by the generation unit 154 to the display device 20. The transmission unit 155A provides the detection result by the detection unit 152 to the display device 20. The transmission unit 155A transmits information to be displayed on the display device 20 to the display device 20. The transmission unit 155A transmits the detection result by the detection unit 152 and the image generated by the generation unit 154 to the display device 20.

[0153] [2-2. Other configuration examples] Furthermore, the processes according to the above-described embodiments and modifications may be implemented in various different forms (modifications) other than the above-described embodiments and modifications. For example, the device configuration of the information processing system 1 described above is merely an example, and any device configuration may be used as long as the server-side information processing device 100A can realize, for example, a selection process similar to that of the information processing device 100.

[0154] For example, the information processing system 1 may include a terminal device in which the display device 20 and the sensor device 50 are integrated. In this case, the terminal device is a computer (client terminal) used by a user, and may be realized by, for example, a notebook PC, a desktop PC, a smartphone, a tablet terminal, a mobile phone, a PDA, or the like.

[0155] In this case, the terminal device may be any terminal device that can transmit information required by the information processing device 100A to the information processing device 100A and display information provided by the information processing device 100A. For example, the terminal device may be various devices such as a head-mounted display, VR goggles integrated with a headset, VR goggles fitted to a smartphone, or a wearable device integrated with glasses, earphones, etc. Note that the above is just an example, and the information processing system 1 may be realized by various configurations.

[0156] [2-3.Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0157] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0158] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0159] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0160] [3. Effects of this disclosure] As described above, an information processing device according to the present disclosure (information processing devices 100 and 100A in the embodiments) includes an acquisition unit (acquisition units 151 and 151A in the embodiments) and a selection unit (selection unit 153 in the embodiments). The acquisition unit acquires gaze information indicating a user's gaze point and object information indicating multiple objects. The selection unit selects one object from the multiple objects based on the positional relationship between the user's gaze point and the multiple objects.

[0161] As a result, the information processing device according to the present disclosure can appropriately select an object according to the user's line of sight by selecting one object from among a plurality of objects based on the positional relationship between the user's gaze point and the plurality of objects. Thus, the information processing device can flexibly select an object according to the user's line of sight.

[0162] The selection unit selects one object that is the nearest object from the user's gaze point. This allows the information processing device to flexibly select an object according to the user's gaze even when the user's gaze is unstable, since it selects the nearest object from the user's gaze point.

[0163] The acquisition unit acquires object information indicating boundaries of the plurality of objects. The selection unit selects a point on the boundary of one of the objects that is closest to the user's gaze point. This allows the information processing device to flexibly select a target according to the user's line of sight by selecting the boundary of the one of the objects that is closest to the user's gaze point.

[0164] The selection unit selects a plurality of points on the boundary of the object in accordance with the movement of the user's gaze point, thereby enabling the information processing device to flexibly select an object in accordance with the user's line of sight by selecting a plurality of points on the boundary of the object.

[0165] The information processing device also includes a generation unit (generation unit 154 in this embodiment). The generation unit generates an area whose boundary is the line connecting the multiple points by connecting the multiple points in the selected order. This allows the information processing device to generate an area by connecting the selected multiple points, thereby enabling flexible area selection (generation) according to the user's line of sight.

[0166] The generation unit also changes the region by changing the position of a point among the multiple points, which allows the information processing device to change the position of the point after selection, thereby enabling the selected region to be flexibly changed according to the user's line of sight.

[0167] The generation unit also changes the area by adding a new point to the multiple points, which allows the information processing device to add a new point after the area is generated, thereby enabling the area selected according to the user's line of sight to be flexibly changed.

[0168] The acquisition unit also acquires object information indicating boundaries of the plurality of objects detected from the image. This allows the information processing device to appropriately select an object according to the user's line of sight using the information on the boundaries of the plurality of objects detected from the image. Therefore, the information processing device can flexibly select an object according to the user's line of sight.

[0169] The acquisition unit also acquires object information indicating boundaries of the plurality of objects detected from each of the plurality of images. This allows the information processing device to appropriately select an object according to the user's line of sight using the information on the boundaries of the plurality of objects detected from the plurality of images (layers). Therefore, the information processing device can flexibly select an object according to the user's line of sight.

[0170] The acquisition unit acquires gaze information indicating a user's gaze point and area information indicating a plurality of areas based on the arrangement of objects. The selection unit selects one of the plurality of areas based on the positional relationship between the user's gaze point and the plurality of areas. By selecting an area based on the positional relationship between the user's gaze point and the plurality of areas, the information processing device can flexibly select an object according to the user's gaze even when the user's gaze is unstable.

[0171] The selection unit selects one of the plurality of regions, which is the region where the user's gaze point is located. This allows the information processing device to select the object closest to the user's gaze point, so that even if the user's gaze is unstable, the information processing device can flexibly select an object according to the user's gaze.

[0172] The acquisition unit acquires area information indicating a plurality of areas based on the boundaries of the object. The selection unit selects one area corresponding to the object whose boundary is located at the user's point of gaze. This allows the information processing device to flexibly select an object according to the user's line of sight by selecting the area of ​​the object where the user's point of gaze is located.

[0173] The selection unit selects one area as an area to be processed for editing. This allows the information processing device to appropriately select an area to be edited, thereby enabling flexible target selection according to the user's line of sight.

[0174] The generating unit also generates a mesh that divides one region into a plurality of partial regions, which allows the information processing device to easily select partial regions within the region, and thus allows the selected region to be flexibly changed according to the user's line of sight.

[0175] The generating unit also generates a mesh including a plurality of partial regions that divide one region into rectangles, thereby enabling the information processing device to easily select partial regions within the region, and thus flexibly change the selected region depending on the user's line of sight.

[0176] The acquisition unit acquires designation information indicating user designations for the plurality of partial regions. The selection unit selects, from the plurality of partial regions, a partial region corresponding to the user designation. This allows the information processing device to flexibly select an object according to the user's line of sight by selecting, from the plurality of partial regions within the region, a partial region corresponding to the user designation.

[0177] The acquisition unit acquires designation information indicating designation of the plurality of partial regions by the user's line of sight. The selection unit selects a partial region corresponding to the user's designation based on the user's line of sight. This allows the information processing device to flexibly select targets according to the user's line of sight by having the user select a partial region based on the user's line of sight.

[0178] The acquisition unit acquires designation information indicating a range within one region designated by the user's gaze. The selection unit selects a partial region located within the range from among the plurality of partial regions. This allows the information processing device to flexibly select an object according to the user's gaze by selecting a partial region corresponding to the range designated by the user's gaze.

[0179] [4. Hardware Configuration] The information devices such as the information processing devices 100 and 100A according to the above-described embodiments are realized by a computer 1000 having a configuration as shown in FIG. 18, for example. FIG. 18 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing devices such as the information processing devices 100 and 100A. The following description will be given taking the information processing device 100 according to the embodiment as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory), a Only Memory) 1300, HDD (Hard Disk Drive) 1400, Communication Interface The computer 1000 includes a processor 1500 and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0180] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0181] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0182] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0183] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0184] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. The medium may be, for example, a DVD (Digital Optical recording media such as a Versatile Disc (Variable Disc) or a Phase Change Rewritable Disc (PD), magneto-optical recording media such as a Magneto-Optical disc (MO), tape media, magnetic recording media, or Semiconductor memory, etc.

[0185] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 15 and the like. The information processing program according to the present disclosure and data in the storage unit 14 are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0186] The present technology can also be configured as follows. (1) an acquisition unit that acquires line-of-sight information indicating a user's gaze point and object information indicating a plurality of objects; a selection unit that selects one object from the plurality of objects based on a positional relationship between the user's gaze point and the plurality of objects; An information processing device comprising: (2) The selection unit Selecting the one object that is the nearest object from the user's gaze point The information processing device described in (1). (3) The acquisition unit obtaining the object information indicating boundaries of the plurality of objects; The selection unit Select a point on the boundary of the object that is closest to the user's gaze point. An information processing device according to (1) or (2). (4) The selection unit Selecting a plurality of points on the boundary of the one object in response to movement of the user's gaze point. (3) An information processing device according to the present invention. (5) a generating unit that generates an area bounded by lines connecting the plurality of points by connecting the plurality of points with lines in the selected order; The information processing device according to (4) is provided with: (6) The generation unit Modifying the region by changing the position of a point in the plurality of points (5) An information processing device according to the present invention. (7) The generation unit Modifying the region by adding a new point to the plurality of points. An information processing device according to (5) or (6). (8) The acquisition unit Obtaining the object information indicating boundaries of the plurality of objects detected from the image. The information processing device according to any one of (3) to (7). (9) The acquisition unit Obtaining the object information indicating boundaries of the plurality of objects detected from each of the plurality of images. (8) An information processing device according to (8). (10) Obtaining gaze information indicating a user's gaze point and object information indicating a plurality of objects; selecting one of the plurality of objects based on a positional relationship between the user's gaze point and the plurality of objects; An information processing method that performs processing. (11) an acquisition unit that acquires line-of-sight information indicating a user's gaze point and area information indicating a plurality of areas based on the arrangement of objects; a selection unit that selects one of the plurality of regions based on a positional relationship between the user's gaze point and the plurality of regions; An information processing device comprising: (12) The selection unit Selecting the one area where the user's gaze point is located from among the plurality of areas. (11) An information processing device according to (11). (13) The acquisition unit obtaining the region information indicating the plurality of regions based on a boundary of an object; The selection unit Selecting the one region corresponding to an object within whose boundary the user's gaze point is located. The information processing device according to (11) or (12). (14) The selection unit The one area is selected as an area to be processed for editing. (13) An information processing device according to (13). (15) a generation unit that generates a mesh that divides the one region into a plurality of subregions; The information processing device according to (14) is provided with: (16) The generation unit The mesh including the plurality of partial regions that are rectangularly divided within the one region is generated. (15) An information processing device according to (15). (17) The acquisition unit acquiring designation information indicating designation by the user for the plurality of partial regions; The selection unit Selecting a partial area corresponding to the user's designation from among the plurality of partial areas The information processing device according to (15) or (16). (18) The acquisition unit acquiring the designation information indicating the designation of the plurality of partial regions by the user's line of sight; The selection unit Selecting a partial area corresponding to the user's designation based on the user's line of sight (17) An information processing device according to (17). (19) The acquisition unit acquiring the designation information indicating a range within the one area designated by the user's line of sight; The selection unit Select a partial region located within the range from among the plurality of partial regions. The information processing device according to (17) or (18). (20) acquiring line-of-sight information indicating a user's gaze point and area information indicating a plurality of areas based on the arrangement of objects; selecting one of the plurality of regions based on a positional relationship between the user's gaze point and the plurality of regions; An information processing method that performs processing. [Explanation of symbols]

[0187] 100, 100A Information processing equipment 11 Communications Department 12 Input section 13 Display unit (display) 14 Storage section 15, 15A control section 151, 151A Acquisition Department 152 Detection unit 153 Selection Section 154 Generation part 155, 155A transmitter 16 Sensor section 161 Position and Orientation Sensor 162 Eye Sensor 20 Display device 50 Sensor Device

Claims

1. an acquisition unit that acquires line-of-sight information indicating a user's gaze point and object information indicating a plurality of objects; a selection unit that selects one object from the plurality of objects based on a positional relationship between the user's gaze point and the plurality of objects; Equipped with The acquisition unit obtaining the object information indicating boundaries of the plurality of objects; The selection unit A point on the boundary of the one object that is closest to the user's gaze point is selected, and a plurality of points on the boundary of the one object are selected in accordance with movement of the user's gaze point. Information processing device.

2. The selection unit Selecting the one object that is the nearest object from the user's gaze point The information processing device according to claim 1 .

3. a generating unit that generates an area bounded by lines connecting the plurality of points by connecting the plurality of points with lines in the selected order; The information processing device according to claim 1 , comprising:

4. The generation unit Modifying the region by changing the position of a point in the plurality of points The information processing device according to claim 3 .

5. The generation unit Modifying the region by adding a new point to the plurality of points. The information processing device according to claim 3 .

6. The acquisition unit Obtaining the object information indicating boundaries of the plurality of objects detected from the image. The information processing device according to claim 1 .

7. The acquisition unit Obtaining the object information indicating boundaries of the plurality of objects detected from each of the plurality of images. The information processing device according to claim 6 .

8. Obtaining gaze information indicating a user's gaze point and object information indicating a plurality of objects; selecting one object from the plurality of objects based on a positional relationship between the user's gaze point and the plurality of objects; obtaining the object information indicating boundaries of the plurality of objects; selecting a point on the boundary of the one object that is closest to the user's gaze point, and selecting a plurality of points on the boundary of the one object according to movement of the user's gaze point; An information processing method that performs processing.

Images