Information processing device, information processing method, and program

Abstract

To improve the convenience of contactless controllable devices used in medical settings and other similar environments. [Solution] The information processing device comprises a tracking unit that sets a target person based on the actions of a person in a medical setting where surgery, examination, or treatment is performed; a detection unit that detects the actions of the target person; and a control unit that controls equipment in the medical setting based on the actions of the target person. This technology can be applied, for example, to a system that controls equipment in a medical setting.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and more particularly to an information processing apparatus, an information processing method, and a program that can improve the convenience of devices that can be controlled non-contact.

Background Art

[0002] In a medical field where surgery, examination, or treatment is performed, it is necessary to maintain a clean area. Therefore, for example, in various medical devices, display devices, etc., devices that can be controlled non-contact are used. For example, in an image captured by a camera, a subject to be controlled of each device (hereinafter referred to as a tracking subject) is tracked, and each device is controlled by the line of sight, posture, voice, gesture, etc. of the tracking subject.

[0003] On the other hand, conventionally, a method has been proposed in which a template of a subject to be tracked is registered in advance and the subject is tracked in combination with distance information (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, for example, when the method described in Patent Document 1 is applied to devices in a medical field, when the tracking subject is lost or changed, registration of a new template is required, resulting in a decrease in convenience.

[0006] The present technology has been made in view of such a situation, and aims to improve the convenience of devices that can be controlled non-contact used in a medical field or the like. [Means for solving the problem]

[0007] One aspect of this technology is an information processing device comprising: a tracking unit that sets a target person based on the actions of a person in a medical setting where surgery, examination, or treatment is performed; a detection unit that detects the actions of the target person; and a control unit that controls equipment in the medical setting based on the actions of the target person.

[0008] One aspect of this technology is an information processing method which includes setting a target person to track based on the actions of a person in a medical setting where surgery, examination, or treatment is performed, detecting the actions of the target person to track, and controlling equipment in the medical setting based on the actions of the target person to track.

[0009] One aspect of this technology involves a program that causes a computer to perform processes including: setting a target person based on the actions of a person in a medical setting where surgery, examination, or treatment is being performed; detecting the actions of the target person; and controlling equipment in the medical setting based on the actions of the target person.

[0010] In one aspect of this technology, a person to be tracked is defined based on the actions of a person in a medical setting where surgery, examination, or treatment is performed, the actions of the person to be tracked are detected, and the equipment in the medical setting is controlled based on the actions of the person to be tracked. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing a first embodiment of an information processing system to which this technology is applied. [Figure 2] This is a flowchart illustrating the device control process performed by the information processing system shown in Figure 1. [Figure 3] This is a block diagram showing a second embodiment of an information processing system to which this technology is applied. [Figure 4] This is a flowchart illustrating a first embodiment of the device control processing performed by the information processing system shown in Figure 3. [Figure 5] It is a flowchart for explaining a second embodiment of device control processing executed by the information processing system of FIG. 3. [Figure 6] It is a diagram for explaining a method of changing a person to be tracked. [Figure 7] It is a block diagram showing a third embodiment of an information processing system to which this technology is applied. [Figure 8] It is a diagram showing a configuration example of a camera and a microphone. [Figure 9] It is a flowchart for explaining device control processing executed by the information processing system of FIG. 7. [Figure 10] It is a diagram for explaining control of voice directivity. [Figure 11] It is a diagram showing an example of a method of notifying a person to be tracked. [Figure 12] It is a diagram showing a configuration example of a computer.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out this technology will be described. The description will be made in the following order. 0. Background of this technology 1. First embodiment (example of executing display control) 2. Second embodiment (example of executing display control and operation control) 3. Third embodiment (example of adding voice control) 4. Variation 5. Others

[0013] <<0. Background of this technology>> First, the background of this technology will be described.

[0014] As described above, in a medical site where surgery, examination, or treatment is performed, since it is necessary to maintain a clean area, for example, devices that can be controlled non - contact are used by the line of sight, posture, voice, gesture, etc. of the person to be tracked.

[0015] On the one hand, in the medical field, since multiple staff members wear similar gowns, masks, protective goggles, etc., it is difficult to identify the person to be tracked only by visual features.

[0016] In contrast, for example, a method of attaching a wearable such as a marker to the person to be tracked can be considered. However, there are risks such as the wearable falling off or being forgotten, and when the person to be tracked changes, it becomes necessary to replace the wearable, so it is difficult to introduce it in the medical field.

[0017] Also, for example, a method of identifying the person to be tracked based on the position of the staff can be considered. However, in the medical field, the arrangement of the staff changes moment by moment according to the progress of the surgery, so it is difficult to introduce it in the medical field.

[0018] In contrast, the present technology is to easily identify the person to be tracked and improve the convenience of a non-contact and controllable device used in a medical field or the like.

[0019] <<1. First Embodiment>> Next, referring to FIGS. 1 and 2, the first embodiment of the present technology will be described.

[0020] <Configuration Example of Information Processing System 1> FIG. 1 shows a configuration example of an information processing system 1 which is a first embodiment of an information processing system to which the present technology is applied.

[0021] The information processing system 1 is, for example, a system that is introduced into a medical field where surgery, examination, or treatment is performed and controls the display of an image (moving image or still image) of the surgical site of a patient. The information processing system 1 includes a camera 11, an information processing unit 12, and a monitor 13.

[0022] The camera 11 is used, for example, to photograph a person (such as a medical staff) in the medical field. The camera 11 includes a photographing unit 21.

[0023] The imaging unit 21 takes photographs of the medical site and supplies the image data showing the images obtained from the imaging to the tracking unit 31 of the information processing unit 12.

[0024] The information processing unit 12 identifies and tracks the subject based on the captured image data and controls the monitor 13 based on the subject's actions. The information processing unit 12 comprises a tracking unit 31, a detection unit 32, and a control unit 33.

[0025] Furthermore, part or all of the information processing unit 12 may be provided in the camera 11 or the monitor 13. This also applies to the information processing unit of other embodiments described later.

[0026] The tracking unit 31 identifies and tracks the person to be tracked based on the captured image data. The tracking unit 31 is composed of, for example, an AI (Artificial Intelligence) edge sensor provided on the camera 11. The tracking unit 31 includes a specific action registration unit 41, an action detection unit 42, and a tracking target setting unit 43.

[0027] The specific action registration unit 41 registers specific actions, which are actions taken to identify the person to be tracked.

[0028] The motion detection unit 42 detects the movements of people in the medical setting based on the captured image data.

[0029] The tracking target setting unit 43 sets and desets the tracking target based on the detection results of the motion detection unit 42. The tracking target setting unit 43 supplies the captured image data and information indicating the range (target range) of the tracking target within the captured image to the detection unit 32.

[0030] The detection unit 32 performs motion detection processing for the person being tracked based on the captured image data. The detection unit 32 is composed of a more powerful computer than the tracking unit 31, for example, and can detect the motion of the person being tracked with higher accuracy. The detection unit 32 includes a face detection unit 51 and a pupil detection unit 52.

[0031] The face detection unit 51 performs face detection processing of the person being tracked based on the captured image data and supplies information indicating the detection result to the control unit 33.

[0032] The pupil detection unit 52 performs pupil detection processing of the person being tracked based on the captured image data and supplies information indicating the detection result to the control unit 33.

[0033] The control unit 33 controls the monitor 13 based on the detection results of the tracking subject's movements. The control unit 33 includes a display control unit 61.

[0034] The display control unit 61 controls the display of the image on the monitor 13 based on the detection result of at least one of the subject's face and pupils. For example, the display control unit 61 acquires display image data from an external source that shows the display image to be displayed on the monitor 13, and performs image processing on the display image based on the detection result of at least one of the subject's face and pupils. The display control unit 141 supplies the display image data showing the display image after image processing to the monitor 13.

[0035] The monitor 13 is a display device installed, for example, in a medical setting, and displays images of the patient's surgical site, etc. The monitor 13 includes a display unit 71.

[0036] The display unit 71 displays a display image based on the display image data.

[0037] Multiple cameras 11 and monitors 13 may be provided. This is also true for other embodiments described later.

[0038] Alternatively, the camera 11 may be built into the monitor 13. For example, the monitor 13 may be composed of an SRD (Spatial Reality Display), and the camera 11 may be composed of a camera built into the SRD. In this case, for example, the camera 11 captures the area in front of the monitor 13.

[0039] <Equipment control processing> Next, referring to the flowchart in Figure 2, we will explain the equipment control process performed by the information processing system 1.

[0040] This process starts, for example, when a command to start the device control process is input to the information processing unit 12, and ends when a command to end the device control process is input to the information processing unit 12.

[0041] In step S1, the specific operation registration unit 41 registers a specific operation.

[0042] The method for registering a specific action is not particularly limited. For example, a specific action can be registered by inputting video data representing the specific action into the specific action registration unit 41.

[0043] The video data may be pre-recorded or captured by camera 11 during registration. Alternatively, the video data may be generated using computer graphics (CG), for example.

[0044] For example, a specific operation may be registered by inputting a program or text data describing the specific operation into the specific operation registration unit 41.

[0045] The specific actions are not particularly limited; for example, any action desired by the user can be registered as a specific action. A specific action can be represented by, for example, at least one of a gesture and / or gaze. For example, an action not normally performed during surgery (e.g., raising both hands) may be registered as a specific action. For example, an action that is easy to perform during surgery (e.g., nodding) may be registered as a specific action.

[0046] Furthermore, a specific action may be a combination of two or more actions. For example, a combination of eye contact and a nodding motion may be registered as a specific action.

[0047] Specific actions include, for example, start and end actions. The start action is the action to set the person to be tracked and begin tracking. The end action is the action to remove the person from the tracking list and end tracking. The start and end actions can be the same, but it is preferable that they be different.

[0048] In step S2, the motion detection unit 42 performs bone estimation for the entire field of view. Specifically, the motion detection unit 42 acquires captured image data from the camera 11. The motion detection unit 42 performs person detection processing for the entire field of view of the captured image and performs bone estimation for each detected person.

[0049] In step S3, the motion detection unit 42 determines whether or not it has detected a person performing the initiation action. Specifically, the motion detection unit 42 detects the gestures and other actions of each person based on the bone estimation results. If the motion detection unit 42 determines that it has not detected a person performing the initiation action, the process returns to step S2.

[0050] Subsequently, the processes from steps S2 to S3 are repeatedly executed until it is determined in step S3 that a person performing the initiation action has been detected.

[0051] On the other hand, if it is determined in step S3 that a person performing the initiation action has been detected, the process proceeds to step S4.

[0052] In step S4, the tracking unit 31 starts tracking the person being tracked.

[0053] Specifically, the tracking target setting unit 43 sets the person performing the initiation action as the tracking target. Based on the captured image data supplied from the shooting unit 21, the tracking target setting unit 43 starts the process of setting the area containing the tracking target within the captured image as the ROI (Region of Interest).

[0054] The ROI may include all areas of the subject being tracked within the captured image, or it may include only specific areas (for example, only the upper body).

[0055] Furthermore, the tracking target setting unit 43 starts the process of supplying the captured image data and information indicating the location of the ROI within the captured image to the detection unit 32.

[0056] The motion detection unit 42 performs bone estimation within the ROI of the captured image and, based on the results of the bone estimation, starts the process of detecting the motion of the person being tracked.

[0057] In step S5, the detection unit 32 detects the movements of the person being tracked.

[0058] Specifically, the face detection unit 51 performs face detection processing of the person being tracked within the ROI of the captured image. For example, the face detection unit 51 detects the position, orientation, and movement of the person being tracked's face. The face detection unit 51 supplies information indicating the face detection result of the person being tracked to the control unit 33.

[0059] The pupil detection unit 52 performs pupil detection processing within the ROI of the captured image. For example, the pupil detection unit 52 detects the horizontal, vertical, and depth positions, movements, and gaze of the left and right pupils (eyes) of the subject. The pupil detection unit 52 supplies information indicating the pupil detection results of the subject to the control unit 33.

[0060] In step S6, the information processing system 1 displays an appropriate image to the person being tracked. Specifically, the display control unit 141 acquires display image data from an external source that indicates the display image to be displayed on the monitor 13. For example, the display image data includes a three-dimensional model of the subject, such as a patient.

[0061] The display control unit 141 performs various image processing on the display image displayed on the monitor 13 so that the tracked person can view the 3D model contained in the display image with the naked eye in 3D, based on the detection results of the position or gaze of both eyes (left and right pupils) of the tracked person. For example, the display control unit 141 generates a display image for the left eye (hereinafter referred to as the left-eye image) and a display image for the right eye (hereinafter referred to as the right-eye image) corresponding to the parallax of the tracked person, based on the horizontal, vertical, and depth positions of the left and right eyes of the tracked person. The left-eye image is an image showing the 3D model as seen from the position corresponding to the left eye of the tracked person, and the right-eye image is an image showing the 3D model as seen from the position corresponding to the right eye of the tracked person. The display control unit 141 supplies the display image data showing the left-eye image and the right-eye image to the monitor 13.

[0062] The display unit 71 displays an image based on the display image data. For example, the display unit 71 displays the left-eye image and the right-eye image so that the left-eye image is visible to the left eye of the person being tracked, and the left-eye image is visible to the right eye of the person being tracked.

[0063] This allows the person being tracked to view the displayed image in stereoscopic vision without the naked eye. In other words, when the left-eye image and the right-eye image are combined in the person being tracked, parallax occurs, and the 3D model in the displayed image is perceived in three dimensions.

[0064] In step S7, the motion detection unit 42 determines whether or not it has detected the tracking subject's termination action based on the detection results of the tracking subject's actions. If it is determined that the tracking subject's termination action has not been detected, the process returns to step S5.

[0065] Subsequently, the processes from steps S5 to S7 are repeatedly executed until it is determined in step S7 that the tracking target has terminated their actions.

[0066] On the other hand, if it is determined in step S7 that the tracking target has terminated the process, the process proceeds to step S8.

[0067] In step S8, the tracking unit 31 terminates the tracking of the person being tracked. That is, the tracking unit 31 terminates the tracking process for the person being tracked that was started in step S4.

[0068] After that, the process returns to step S2, and the processes from step S2 onward are executed.

[0069] In this way, the convenience of the monitor 13 used in medical settings can be improved.

[0070] For example, it is possible to easily and reliably identify and track a person being tracked without requiring prior registration of the person being tracked or the use of any equipment. Furthermore, the display of the displayed image can be controlled based on the actions of the person being tracked (e.g., eye or gaze movements). For example, even in an environment where multiple staff members are in front of monitor 13, the gaze of the person being tracked can be tracked, and the displayed image can be shown in a way that allows the person being tracked to view it in stereoscopic 3D without glasses.

[0071] For example, by having the tracked person perform a termination action and deselecting that person, it becomes possible to set up a new tracked person. This allows for easy changes to tracked people without false positives.

[0072] For example, because the range for bone estimation in captured images and detection of the subject's movements is limited to the ROI, high-speed and high-precision detection processing is achieved. This makes it possible to display effective images in medical settings where real-time performance and accuracy are required.

[0073] For example, since the user can arbitrarily set the start and end actions, simple and robust actions can be set as the start and end actions according to the surgical procedure and the user's preferences.

[0074] <<2. Second Embodiment>> Next, a second embodiment of this technology will be described with reference to Figures 3 to 6.

[0075] In the second embodiment, devices other than the monitor 13 can also be operated without contact by the person being tracked.

[0076] <Example configuration of information processing system 101> Figure 3 shows an example configuration of an information processing system 101, which is a second embodiment of an information processing system to which this technology is applied.

[0077] In the diagram, parts corresponding to the information processing system 1 in Figure 1 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0078] Comparing information processing system 101 with information processing system 1, they are similar in that they both have a camera 11 and a monitor 13, but they differ in that an information processing unit 111 is provided instead of an information processing unit 12, and equipment 112 is added.

[0079] Comparing the information processing unit 111 with the information processing unit 12, they are similar in that they both include a tracking unit 31, but they differ in that instead of a detection unit 32 and a control unit 33, a detection unit 121 and a control unit 122 are provided.

[0080] The detection unit 121 performs motion detection processing for the person being tracked based on the captured image data. The detection unit 121 is composed of a more powerful computer than the tracking unit 31, for example, and can detect the motion of the person being tracked with higher accuracy.

[0081] Comparing the detection unit 121 to the detection unit 32, they are similar in that they both have a face detection unit 51 and an eye detection unit 52, but they differ in that a hand detection unit 131 is added.

[0082] The hand detection unit 131 performs a hand detection process for the person being tracked based on the captured image data and supplies information indicating the detection result to the control unit 122.

[0083] The control unit 122 controls the monitor 13 and the device 112 based on the detection results of the tracking subject's movements. The control unit 122 includes a display control unit 141 and an operation control unit 142.

[0084] The display control unit 141 controls the display of the image on the monitor 13, similar to the display control unit 61 in Figure 1, based on the detection result of at least one of the following: the face, pupils, and hands of the person being tracked.

[0085] The operation control unit 142 operates the monitor 13 and the device 112 by controlling the monitor 13 and the device 112 based on the detection result of at least one of the face, pupils, and hands of the person being tracked.

[0086] The device 112 is composed of, for example, medical devices such as a C-arm. The device 112 includes an operating unit 151.

[0087] The operation unit 151 operates the device 112 under the control of the operation control unit 142.

[0088] Note that multiple devices 112 may be provided. If multiple devices 112 are provided, they may all be of different types, or they may all be of the same type. The same applies to other embodiments described later.

[0089] <First Embodiment of Device Control Processing> Next, with reference to the flowchart in Figure 4, a first embodiment of the device control processing performed by the information processing system 101 will be described.

[0090] This process starts, for example, when a command to start the device control process is input to the information processing unit 111, and ends when a command to end the device control process is input to the information processing unit 111.

[0091] In steps S51 to S54, the same process as in steps S1 to S4 in Figure 2 is executed.

[0092] In step S55, the detection unit 32 detects the movements of the person being tracked.

[0093] Specifically, the face detection unit 51 performs face detection processing of the person being tracked, similar to the process in step S5 of Figure 2, and supplies information indicating the detection result to the control unit 122. The pupil detection unit 52 performs pupil detection processing of the person being tracked, similar to the process in step S5 of Figure 2, and supplies information indicating the detection result to the control unit 122.

[0094] The hand detection unit 131 performs hand detection processing within the ROI of the captured image. For example, the hand detection unit 131 detects the position, posture, and movement of the hand of the person being tracked. The hand detection unit 131 supplies information indicating the detection result of the hand of the person being tracked to the control unit 122.

[0095] In step S56, the information processing system 101 controls each device based on the actions of the person being tracked.

[0096] Specifically, monitor 13 displays an appropriate image to the person being tracked, similar to the process in step S6 of Figure 2.

[0097] For example, the operation control unit 142 controls the operation unit 151 of the device 112 based on at least one of the tracked person's gestures, posture, and gaze. This allows the tracked person to operate the device 112 without physical contact.

[0098] In step S57, similar to the process in step S7 in Figure 2, it is determined whether or not the tracking target's termination action has been detected. If it is determined that the tracking target's termination action has not been detected, the process returns to step S55.

[0099] Subsequently, steps S55 to S57 are repeatedly executed until it is determined in step S57 that the tracking target has terminated the process.

[0100] On the other hand, if it is determined in step S57 that the tracking target has terminated the process, the process proceeds to step S58.

[0101] In step S58, the tracking of the person being tracked is completed, similar to the process in step S8 in Figure 2.

[0102] After that, the process returns to step S51, and the processes from step S51 onward are executed.

[0103] In this way, the convenience of the device 112 used in medical settings can be improved.

[0104] For example, similar to the first embodiment, the person to be tracked can be easily and reliably identified and tracked without requiring prior registration of the person to be tracked or the use of any attached devices. Furthermore, the device 112 can be operated non-contactually while maintaining a clean environment, based on the actions of the person to be tracked.

[0105] <Second Embodiment of Equipment Control Processing> Next, a second embodiment of the device control processing performed by the information processing system 101 will be described with reference to the flowchart in Figure 5.

[0106] This process starts, for example, when a command to start the device control process is input to the information processing unit 111, and ends when a command to end the device control process is input to the information processing unit 111.

[0107] In steps S101 to S106, the processes shown in steps S51 to S56 of Figure 4 are executed.

[0108] In step S107, similar to the process in step S57 in Figure 4, it is determined whether or not the tracking target's termination action has been detected. If it is determined that the tracking target's termination action has not been detected, the process proceeds to step S108.

[0109] In step S108, bone estimation for the entire field of view is performed, similar to the process in step S102.

[0110] In step S109, the motion detection unit 42 determines whether it has detected a person other than the person being tracked who is performing the initiation action. Specifically, the motion detection unit 42 detects the gestures and other actions of each person in the captured image based on the bone estimation results. If there is no person performing the initiation action in the captured image, or if there is a person performing the initiation action in the captured image but that person is the person being tracked, the motion detection unit 42 determines that it has not detected a person other than the person being tracked who is performing the initiation action, and the process returns to step S105.

[0111] Subsequently, steps S105 to S109 are repeatedly executed until it is determined in step S107 that a termination action by the person being tracked has been detected, or until it is determined in step S109 that a person other than the person being tracked has been detected performing an initiation action.

[0112] On the other hand, in step S109, if the motion detection unit 42 detects that a person performing an initiation action is present in the captured image and that person is not the person being tracked, it determines that it has detected a person other than the person being tracked performing an initiation action, and proceeds to step S110.

[0113] In step S110, the tracking target setting unit 43 changes the person being tracked. That is, the tracking target setting unit 43 changes the person being tracked from the current person to the person performing the initiation action.

[0114] For example, as shown in Figure 6, if the person being tracked is User B, and User C performs the start action, the person being tracked changes from User B to User C.

[0115] Subsequently, the process returns to step S104, and steps S104 to S110 are repeatedly executed until it is determined in step S107 that the tracking target has terminated their actions.

[0116] On the other hand, if it is determined in step S107 that the tracking target has terminated the process, the process proceeds to step S111.

[0117] In step S111, the tracking of the person being tracked is completed, similar to the process in step S58 in Figure 4.

[0118] After that, the process returns to step S102, and the processes from step S102 onward are executed.

[0119] As described above, it is possible to smoothly switch the person being tracked at any given time.

[0120] <<3. Third Embodiment>> Next, a third embodiment of this technology will be described with reference to Figures 7 to 10.

[0121] <Example Configuration of Information Processing System 201> Figure 7 shows an example configuration of an information processing system 201, which is a third embodiment of an information processing system to which this technology is applied.

[0122] In the diagram, parts corresponding to the information processing system 101 in Figure 3 are denoted by the same reference numerals, and their explanations are omitted as appropriate.

[0123] Comparing information processing system 201 with information processing system 101, they are similar in that they both include a camera 11, a monitor 13, and equipment 112. The differences are that information processing unit 212 is provided instead of information processing unit 111, and a microphone 211 is added.

[0124] The microphone 211 is used, for example, to collect sound in a medical setting. The microphone 211 includes a sound collection unit 221.

[0125] The sound collection unit 221 collects sounds such as conversations in a medical setting, and supplies audio data representing the collected sounds to the tracking unit 231.

[0126] The microphone 211 may also be equipped with multiple sound-collecting units 221, such as a beamforming microphone, and its directivity can be controlled.

[0127] For example, if microphone 211 is configured as a beamforming microphone, microphone 211 may be provided on camera 11, as shown in Figure 8.

[0128] Furthermore, multiple microphones 211 may be provided.

[0129] The information processing unit 212 includes a tracking unit 231, a detection unit 232, and a control unit 233.

[0130] The tracking unit 231 identifies and tracks the person to be tracked based on captured image data and audio data. The tracking unit 231 is composed of, for example, an AI edge sensor provided on the camera 11. The tracking unit 231 includes a specific action registration unit 241, an action detection unit 242, and a tracking target setting unit 243.

[0131] The specific action registration unit 241 registers specific actions, which are actions taken to identify the person to be tracked.

[0132] The motion detection unit 242 detects the movements of people in the medical setting based on captured image data and audio data.

[0133] The tracking target setting unit 243 sets and desets the tracking target based on the detection results of the motion detection unit 242. The tracking target setting unit 243 supplies the captured image data and information indicating the range of the tracking target within the captured image to the detection unit 232.

[0134] For example, the tracking target setting unit 243 controls the directivity of the microphone 211 so that its sensitivity to the direction of the tracking target is higher than its sensitivity to other directions. Alternatively, for example, the tracking target setting unit 243 extracts audio data indicating the sound from the direction of the tracking target from among multiple audio data indicating the sound from each direction supplied by the microphone 211. The tracking target setting unit 243 supplies the audio data indicating the sound from the direction of the tracking target to the detection unit 232.

[0135] The detection unit 232 performs motion detection processing based on captured image data and audio data. The detection unit 232 is configured with a more powerful processor than the tracking unit 231, for example, and can detect the motion of the person being tracked with higher accuracy.

[0136] Comparing the detection unit 232 with the detection unit 121 in Figure 3, they are identical in that they both include a face detection unit 51, an eye detection unit 52, and a hand detection unit 131, but differ in that a voice detection unit 251 has been added.

[0137] The voice detection unit 251 performs voice detection processing based on the voice data and supplies information indicating the detection result to the control unit 233.

[0138] The control unit 233 controls the monitor 13 and the device 112 based on the detection results of the tracking subject's movements. The control unit 233 includes a display control unit 261 and an operation control unit 262.

[0139] The display control unit 261 controls the display of the image on the monitor 13, similar to the display control unit 61 in Figure 1, based on the detection result of at least one of the following: the face, eyes, hands, and voice of the person being tracked.

[0140] The operation control unit 262 operates the monitor 13 and the device 112 by controlling them in the same way as the operation control unit 142 in Figure 3, based on the detection result of at least one of the following: the face, eyes, hands, and voice of the person being tracked. <Equipment control processing> Next, with reference to the flowchart in Figure 9, we will explain the details of the device control processing performed by the information processing system 201.

[0141] This process starts, for example, when a command to start the device control process is input to the information processing unit 212, and ends when a command to end the device control process is input to the information processing unit 212.

[0142] In step S151, the specific operation registration unit 241 registers a specific operation.

[0143] The method for registering a specific action is not particularly limited. For example, a specific action may be registered by inputting video data and audio data indicating the specific action into the specific action registration unit 241.

[0144] The video data may be captured and generated in advance, or it may be captured and generated by the camera 11 at the time of registration. Alternatively, for example, the video data may be generated using computer graphics (CG). The audio data may be generated in advance, or it may be generated by the microphone 211 at the time of registration.

[0145] For example, a specific operation may be registered by inputting a program or text data describing the specific operation into the specific operation registration unit 241.

[0146] The specific actions are not particularly limited; for example, a user can register any action as a specific action. In addition to the specific actions that can be registered in step S1 of Figure 2, it is also possible to register specific actions that combine spoken language. For example, it is possible to register a specific action that is represented by at least one of a gesture and / or gaze, along with spoken language.

[0147] Note that specific actions do not necessarily have to include spoken language.

[0148] In step S152, the motion detection unit 242 performs bone estimation for the entire field of view and sound detection in all directions.

[0149] Specifically, the motion detection unit 242 acquires captured image data from the camera 11. The motion detection unit 242 performs person detection processing across the entire field of view of the captured image and performs bone estimation for each detected person.

[0150] Furthermore, the motion detection unit 242 acquires audio data from the microphone 211. The motion detection unit 242 performs audio detection processing based on the audio data from all directions. This allows for the detection of speech from each person in the medical setting.

[0151] In step S153, the motion detection unit 242 determines whether or not it has detected a person performing the initiation action. Specifically, the motion detection unit 242 detects the actions of each person, such as gestures, posture, gaze, and speech, based on the results of bone estimation and voice detection. If the motion detection unit 242 determines that it has not detected a person performing the initiation action, the process returns to step S152.

[0152] Subsequently, the processes from steps S152 to S153 are repeatedly executed until it is determined in step S153 that a person performing the initiation action has been detected.

[0153] On the other hand, if it is determined in step S153 that a person performing the initiation action has been detected, the process proceeds to step S154.

[0154] In step S154, the tracking unit 231 starts tracking the person being tracked.

[0155] Specifically, the tracking target setting unit 243 sets the person performing the start operation as the tracking target. Similar to the process of the tracking target setting unit 43 in step S4 of Figure 2, the tracking target setting unit 243 starts the process of setting the area containing the tracking target in the captured image as the ROI (Region of Interest) based on the captured image data supplied from the shooting unit 21. The tracking target setting unit 243 starts the process of supplying the captured image data and information indicating the location of the ROI in the captured image to the detection unit 232.

[0156] Furthermore, the tracking target setting unit 243 starts processing to control the directionality of the audio data used for audio detection by the audio detection unit 251.

[0157] For example, the tracking target setting unit 243 starts a process to control the directivity of the sound collection unit 221 of the microphone 211 so that its sensitivity to the direction of the person being tracked is higher than its sensitivity to other directions.

[0158] For example, as shown in Figure 10, the directivity of the sound-collecting section 221 of the microphone 211 is controlled so that the sensitivity in the direction of sound source 301a where the person being tracked is located is higher than the sensitivity in the direction of sound sources 301b to 301d. For example, the directivity of the sound-collecting section 221 of the microphone 211 is controlled so that it collects sound from around sound source 301a.

[0159] The tracking target setting unit 243 starts the process of acquiring directionally controlled audio data from the microphone 211 and supplying it to the detection unit 232.

[0160] Alternatively, for example, the tracking target setting unit 243 starts the process of extracting audio data collected from the direction of the person being tracked, from among multiple audio data collected from each direction supplied by the microphone 211. The tracking target setting unit 243 then starts the process of supplying the extracted audio data (directionally controlled audio data) to the detection unit 232.

[0161] The motion detection unit 242 starts the process of estimating bones within the ROI of the captured image. The motion detection unit 242 starts the process of detecting spoken audio based on audio data whose directionality is controlled in the direction of the person being tracked. The motion detection unit 242 starts the process of detecting the motion of the person being tracked based on the results of bone estimation and audio detection.

[0162] In step S155, the detection unit 232 detects the movement of the person being tracked.

[0163] Specifically, the face detection unit 51 performs face detection processing of the person being tracked, similar to the process in step S5 of Figure 2, and supplies information indicating the face detection result of the person being tracked to the control unit 233. The pupil detection unit 52 performs pupil detection processing of the person being tracked, similar to the process in step S5 of Figure 2, and supplies information indicating the pupil detection result of the person being tracked to the control unit 233. The hand detection unit 131 performs hand detection processing of the person being tracked, similar to the process in step S55 of Figure 4, and supplies information indicating the hand detection result of the person being tracked to the control unit 233.

[0164] The voice detection unit 251 detects spoken voice based on voice data whose directionality is controlled to face the direction of the person being tracked, and supplies information indicating the detection result to the control unit 233.

[0165] In step S156, the information processing system 101 controls each device based on the actions of the person being tracked.

[0166] Specifically, for example, monitor 13 displays an appropriate image to the person being tracked, similar to the process in step S6 of Figure 2.

[0167] For example, the operation control unit 262 controls the operation unit 151 of the device 112 based on at least one of the following: the subject's gestures, posture, gaze, and spoken voice.

[0168] In step S157, the motion detection unit 242 determines whether or not it has detected the tracking subject's termination action based on the detection results of the tracking subject's actions. If it is determined that the tracking subject's termination action has not been detected, the process returns to step S155.

[0169] Subsequently, steps S155 to S157 are repeatedly executed until it is determined in step S157 that the tracking target has terminated the process.

[0170] On the other hand, if it is determined in step S157 that the tracking target has terminated the process, the process proceeds to step S158.

[0171] In step S158, the tracking unit 231 terminates the tracking of the person being tracked. That is, the tracking unit 231 terminates the tracking process of the person being tracked that was started in step S154.

[0172] After that, the process returns to step S152, and the processes from step S152 onward are executed.

[0173] As described above, it becomes possible to control the device 112 based on the voice of the person being tracked.

[0174] Furthermore, since only the voice of the person being tracked is detected, the device 112 is less likely to be misoperated by the voices or noise of people other than the person being tracked.

[0175] Furthermore, in the information processing system 201, as shown in the device control process in Figure 5, it is also possible to change the person being tracked by having someone other than the person being tracked perform the start operation.

[0176] <<4. Variation>> The following describes some modifications of the embodiments of the present technology described above.

[0177] For example, as shown in Figure 11, the tracked person may be displayed on the monitor 13 at any time, such as when the tracked person is identified, when the tracked person is changed, or when the tracked person is lost.

[0178] For example, the system may identify or change the person being tracked based on an action (e.g., pointing) by another person indicating the person being tracked.

[0179] This technology can also be applied to settings other than medical facilities, where the person being tracked controls devices that can be controlled without physical contact.

[0180] <<5. Others>> <Description of a computer using this technology> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0181] Figure 12 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0182] In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0183] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0184] The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory.

[0185] The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009.

[0186] The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0187] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0188] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0189] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0190] A program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, a program may be transferred to a remote computer and executed there.

[0191] When the above-described series of processes are performed by a computer executing a program, for example, the processor of the processing circuit 1001 functions as the information processing unit 12 in Figure 1, the information processing unit 111 in Figure 3, or the information processing unit 212 in Figure 7 by executing the program.

[0192] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of a processor and memory (for example, only the processing circuit 1001, or a combination of processing circuit 1001 to bus 1004, etc.), constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, an entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0193] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0194] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0195] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0196] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0197] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0198] <Examples of configuration combinations> This technology can also be configured as follows:

[0199] (1) A tracking unit sets up a target for tracking based on the actions of a person in a medical setting where surgery, examination, or procedure is performed. A detection unit that detects the actions of the person being tracked, A control unit that controls the equipment at the medical site based on the actions of the person being tracked. An information processing device equipped with the following features. (2) The tracking unit sets the person who initiated the action from among the people in the medical setting as the person to be tracked. The information processing device described in (1) above. (3) If a person other than the person being tracked performs the start action, the tracking unit will change the person being tracked to the person who performed the start action. The information processing device described in (2) above. (4) The tracking unit cancels the tracking target's settings when the tracking target performs a termination action. The information processing apparatus described in (2) or (3) above. (5) The aforementioned start operation can be registered by the user. An information processing device as described in any of (2) to (4) above. (6) The aforementioned initiation action is represented by at least one of a gesture and / or gaze. An information processing device according to any of (2) to (5) above. (7) The aforementioned initiation action is further represented by spoken audio. The information processing device described in (6) above. (8) The tracking unit detects the movements of people within the medical facility based on the images captured within the medical facility. The detection unit detects the movements of the person being tracked based on the captured image. An information processing device as described in any of (1) to (7) above. (9) The tracking unit detects the target area including the person being tracked in the captured image, The detection unit detects the movement of the person being tracked within the target area of ​​the captured image. The information processing device described in (8) above. (10) The detection unit detects at least one of the following: the subject's gestures, posture, gaze, and spoken voice. The control unit controls the medical equipment based on at least one of the following: the subject's gestures, posture, gaze, and speech. An information processing device according to any of (1) to (9) above. (11) The aforementioned medical equipment includes display devices, The control unit controls the display of the image on the display device based on the position of the eyes or gaze of the person being tracked. The information processing device described in (10) above. (12) The control unit controls the display of the images on the display device so that an image showing the 3D model as viewed from a position corresponding to the left eye of the person being tracked is viewed by the left eye of the person being tracked, and an image showing the 3D model as viewed from a position corresponding to the right eye of the person being tracked is viewed by the right eye of the person being tracked. The information processing device described in (11) above. (13) The control unit controls the operation of the medical equipment based on at least one of the following: the subject's gestures, posture, gaze, and speech. An information processing device as described in any of (10) to (12) above. (14) The detection unit detects the speech of the person being tracked based on audio data representing the sound within the medical facility. An information processing device as described in any of (10) to (13) above. (15) The tracking unit controls the directivity of the audio data used to detect the speech of the person being tracked. The information processing device described in (14) above. (16) The tracking unit controls the directivity of the microphones in the medical setting so that its sensitivity to the direction of the person being tracked is higher than its sensitivity to other directions. The information processing device described in (15) above. (17) The tracking unit extracts audio data collected from the direction of the person being tracked and supplies it to the detection unit. The information processing device described in (15) above. (18) The detection unit has higher accuracy in detecting human movement than the tracking unit. An information processing device as described in any of (1) to (17) above. (19) The tracking targets are defined based on the actions of individuals within the medical setting where surgery, examinations, or procedures are performed. To detect the actions of the person being tracked, Based on the actions of the person being tracked, the equipment at the medical facility is controlled. Information processing methods including (20) The tracking targets are defined based on the actions of individuals within the medical setting where surgery, examinations, or procedures are performed. To detect the actions of the person being tracked, Based on the actions of the person being tracked, the equipment at the medical facility is controlled. A program that causes a computer to perform a process that includes [a specific action].

[0200] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of Symbols]

[0201] 1 Information processing system, 11 Camera, 12 Information processing unit, 13 Monitor, 21 Shooting unit, 31 Tracking unit, 32 Detection unit, 33 Control unit, 41 Specific action registration unit, 42 Action detection unit, 43 Tracking target setting unit, 51 Face detection unit, 52 Eye detection unit, 61 Display control unit, 71 Display unit, 101 Information processing system, 111 Information processing unit, 112 Equipment, 121 Detection unit, 122 Control unit, 131 Hand detection unit, 141 Display control unit, 142 Operation control unit, 151 Operation unit, 201 Information processing system, 211 Microphone, 212 Information processing unit, 221 Sound collection unit, 231 Tracking unit, 232 Detection unit, 233 Control unit, 241 Specific operation registration unit, 242 operation detection unit, 243 tracking target setting unit, 251 voice detection unit, 261 display control unit, 262 operation control unit

Claims

1. A tracking unit sets up a target for tracking based on the actions of a person in a medical setting where surgery, examination, or procedure is performed. A detection unit that detects the actions of the person being tracked, A control unit that controls the equipment at the medical site based on the actions of the person being tracked. An information processing device equipped with the following features.

2. The tracking unit sets the person who initiated the action from among the people in the medical setting as the person to be tracked. The information processing apparatus according to claim 1.

3. If a person other than the person being tracked performs the start action, the tracking unit will change the person being tracked to the person who performed the start action. The information processing apparatus according to claim 2.

4. The tracking unit cancels the tracking target's settings when the tracking target performs a termination action. The information processing apparatus according to claim 2.

5. The aforementioned start operation can be registered by the user. The information processing apparatus according to claim 2.

6. The aforementioned initiation action is represented by at least one of a gesture and / or gaze. The information processing apparatus according to claim 2.

7. The aforementioned initiation action is further represented by spoken audio. The information processing apparatus according to claim 6.

8. The tracking unit detects the movements of people within the medical facility based on the images captured within the medical facility. The detection unit detects the movements of the person being tracked based on the captured image. The information processing apparatus according to claim 1.

9. The tracking unit detects the target area including the person being tracked in the captured image, The detection unit detects the movement of the person being tracked within the target area of ​​the captured image. The information processing apparatus according to claim 8.

10. The detection unit detects at least one of the following: the subject's gestures, posture, gaze, and spoken voice. The control unit controls the medical equipment based on at least one of the following: the subject's gestures, posture, gaze, and speech. The information processing apparatus according to claim 1.

11. The aforementioned medical equipment includes display devices, The control unit controls the display of the image on the display device based on the position of the eyes or gaze of the person being tracked. The information processing apparatus according to claim 10.

12. The control unit controls the display of the images on the display device so that an image showing the three-dimensional model as viewed from a position corresponding to the left eye of the person being tracked is viewed by the left eye of the person being tracked, and an image showing the three-dimensional model as viewed from a position corresponding to the right eye of the person being tracked is viewed by the right eye of the person being tracked. The information processing apparatus according to claim 11.

13. The control unit controls the operation of the medical equipment based on at least one of the following: the subject's gestures, posture, gaze, and speech. The information processing apparatus according to claim 10.

14. The detection unit detects the speech of the person being tracked based on audio data representing the sound within the medical facility. The information processing apparatus according to claim 10.

15. The tracking unit controls the directivity of the audio data used to detect the speech of the person being tracked. The information processing apparatus according to claim 14.

16. The tracking unit controls the directivity of the microphones in the medical setting so that its sensitivity to the direction of the person being tracked is higher than its sensitivity to other directions. The information processing apparatus according to claim 15.

17. The tracking unit extracts audio data collected from the direction of the person being tracked and supplies it to the detection unit. The information processing apparatus according to claim 15.

18. The detection unit has higher accuracy in detecting human movement than the tracking unit. The information processing apparatus according to claim 1.

19. The tracking targets are defined based on the actions of individuals within the medical setting where surgery, examinations, or procedures are performed. To detect the actions of the person being tracked, Based on the actions of the person being tracked, the equipment at the medical facility is controlled. Information processing methods including

20. The tracking targets are defined based on the actions of individuals within the medical setting where surgery, examinations, or procedures are performed. To detect the actions of the person being tracked, Based on the actions of the person being tracked, the equipment at the medical facility is controlled. A program that causes a computer to perform a process that includes [a specific action].

Images