Behavior determination device, behavior determination method, program, and recording medium

Abstract

To provide an action determination device which allows for reducing X-ray exposure.SOLUTION: An action determination device of the present invention comprises a gaze direction acquisition unit configured to acquire gaze information including gaze direction information of a determination target person, an action determination unit configured to determine whether or not the determination target person is looking at a determination target object on the basis of the gaze direction information, and a determination result output unit for outputting a determination result.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an action device, an action determination method, a program, and a recording medium.

Background Art

[0002] Endoscopic retrograde cholangiopancreatography (ERCP) is a medical examination method that combines endoscopy and X-ray diagnosis. Patent Document 1 discloses a technology that can reduce the burden on patients, avoid imposing complicated procedures on operators, shorten the operation time, further reduce the hospital introduction cost, and facilitate hygiene management by reducing the number of times of cleaning and disinfection, by performing ERCP and endoscopic ultrasound-guided fine needle aspiration (FNA) with a single endoscopic device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] Since ERCP involves X-ray imaging, there is a demand for reducing the X-ray exposure dose. This demand is not limited to ERCP, and is similarly applicable to other medical devices and other industrial devices.

[0005] Accordingly, an object of the present invention is to provide an action determination device, an action determination method, and a program capable of reducing the X-ray exposure dose.

Means for Solving the Problems

[0006] To achieve the above object, the action determination device of the present invention comprises: a line-of-sight information acquisition unit, an action determination unit, and a determination result output unit, The aforementioned gaze information acquisition unit acquires gaze information including the gaze direction information of the person to be judged, The behavior determination unit determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The judgment result output unit outputs the judgment result.

[0007] The behavior determination method of the present invention is This includes a process for acquiring eye-tracking information, a process for determining behavior, and a process for outputting the determination result. The aforementioned gaze information acquisition step acquires gaze information including the gaze direction information of the person to be judged, The aforementioned behavior determination step determines, based on the gaze direction information, whether the person to be determined is looking at the object to be determined. The judgment result output step outputs the judgment result.

[0008] The program of the present invention, This includes procedures for acquiring eye-tracking information, determining behavior, and outputting the determination result. The aforementioned eye-gaze information acquisition procedure acquires eye-gaze information including the gaze direction information of the person to be judged, The aforementioned behavior determination procedure determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The above-mentioned procedure for outputting the judgment result outputs the judgment result, Each of the aforementioned steps is a program that is executed by a computer.

[0009] The recording medium of the present invention is a computer-readable recording medium that stores the program of the present invention. [Effects of the Invention]

[0010] According to the present invention, it is possible to determine whether the person being judged is looking at the object being judged. If the present invention is applied to ERCP, for example, it is possible to determine whether the operator is looking at the monitor displaying the X-ray image. By turning on X-ray imaging when the operator is looking and turning it off when the operator is not looking, the X-ray irradiation time can be shortened, and as a result, the amount of X-ray exposure can be reduced. Brief Description of the Drawings

[0011] [Figure 1] Figure 1 is a block diagram illustrating an example configuration of the action determination device according to Embodiment 1. [Figure 2] Figure 2 is a block diagram illustrating an example hardware configuration of the action determination device according to Embodiment 1. [Figure 3] Figure 3 is a flowchart illustrating an example of processing in the action device according to Embodiment 1. [Figure 4] Figure 4 is a block diagram illustrating an example configuration of the action determination device according to Embodiment 2. [Figure 5] Figure 5 is a block diagram illustrating an example hardware configuration of the action determination device according to Embodiment 2. [Figure 6] Figure 6 is a flowchart illustrating an example of processing in the action device according to Embodiment 2. [Figure 7] Figure 7 is a block diagram illustrating an example configuration of the action determination device according to Embodiment 2. [Figure 8] Figure 8 is a block diagram illustrating an example hardware configuration of the action determination device according to Embodiment 2. [Figure 9] Figure 9 is a flowchart illustrating an example of processing in the action device according to Embodiment 2. [Figure 10] Figure 10 is an explanatory diagram illustrating an example in which the present invention is applied to ERCP. [Figure 11] Figure 11 is an explanatory diagram illustrating an example in which the present invention is applied to ERCP. [Figure 12] Figure 12 is an explanatory diagram illustrating an example in which the present invention is applied to ERCP. [Figure 13] Figure 13 is an explanatory diagram illustrating an example in which the present invention is applied to ERCP. Modes for Carrying Out the Invention

[0012] In the action determination device of the present invention, for example, the line-of-sight information acquisition unit further acquires distance information between the subject to be determined and the object to be determined, The action determination unit may be configured to determine, based on the line-of-sight direction information and the distance information, whether or not the subject to be determined is looking at the object to be determined.

[0013] The action determination device of the present invention further includes, for example, a line-of-sight direction determination unit, The line-of-sight direction determination unit may be configured to determine the line-of-sight direction based on the face image information of the subject to be determined and generate the line-of-sight direction information.

[0014] The action determination device of the present invention is, for example, further including a composite image generation unit, The composite image generation unit generates a marker image of the subject to be determined as a composite image, The line-of-sight information acquisition unit acquires the image of the subject to be determined, The determination result output unit may be configured to synthesize and display the marker image and the image of the subject to be determined when the action determination unit determines that the subject to be determined is looking at the object to be determined.

[0015] In the action determination device of the present invention, for example, the composite image generation unit generates a line-of-sight direction display image for each subject to be determined based on the line-of-sight direction information, The line-of-sight information acquisition unit acquires the image of the subject to be determined, The determination result output unit may be configured to synthesize and display the line-of-sight direction display image and the image of the subject to be determined.

[0016] In the action determination device of the present invention, for example, the object to be determined may be a display, more specifically a display of a medical device.

[0017] In the action determination method of the present invention, for example, the line-of-sight information acquisition step further acquires distance information between the subject to be determined and the object to be determined, The behavior determination step may also be configured to determine whether the person being determined is looking at the object being determined, based on the gaze direction information and the distance information.

[0018] The behavior determination method of the present invention further includes, for example, a gaze direction determination step, The gaze direction determination step may also be configured to determine the gaze direction based on the facial image information of the person to be determined and generate the gaze direction information.

[0019] The behavior determination method of the present invention is, for example, Furthermore, it includes a composite image generation process, The aforementioned composite image generation step generates the marker image of the person to be judged as a composite image, The aforementioned gaze information acquisition step acquires an image of the person to be judged, The judgment result output step may also be configured such that, when the action judgment step determines that the person being judged is looking at the object being judged, the sign image and the person being judged image are combined and displayed.

[0020] In the behavior determination method of the present invention, for example, the composite image generation step generates a gaze direction display image for each person to be determined based on the gaze direction information, The eye-tracking information acquisition step involves acquiring the image of the person to be judged, The judgment result output step may also be configured to display a composite image of the gaze direction display image and the image of the person to be judged.

[0021] In the behavior determination method of the present invention, for example, the object to be determined may be a display, and more specifically, a display for a medical device.

[0022] In the program of the present invention, for example, the line-of-sight information acquisition procedure further acquires distance information from the person to be judged to the object to be judged, The behavior determination procedure may also be configured to determine whether the person being determined is looking at the object being determined, based on the gaze direction information and the distance information.

[0023] The program of the present invention further includes, for example, a procedure for determining the direction of gaze. The gaze direction determination procedure may also be configured to determine the gaze direction based on the facial image information of the person to be determined and generate the gaze direction information.

[0024] The program of the present invention, for example, Furthermore, it includes a procedure for generating a composite image, The aforementioned composite image generation procedure generates the marker image of the person to be judged as a composite image, The aforementioned eye-tracking information acquisition procedure involves acquiring an image of the person to be judged, The judgment result output procedure may also be configured such that, when the action judgment procedure determines that the person being judged is looking at the object being judged, the sign image and the person being judged image are combined and displayed.

[0025] In the program of the present invention, for example, the composite image generation procedure generates a gaze direction display image for each person to be judged based on the gaze direction information, The eye-tracking information acquisition procedure involves acquiring the image of the person to be judged, The judgment result output procedure may also involve combining the gaze direction display image and the image of the person to be judged and displaying them together.

[0026] In the program of the present invention, for example, the object to be determined may be a display, and more specifically, a display for a medical device.

[0027] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably with those of the others, and unless otherwise specified, the configurations of each embodiment can be combined.

[0028] [Embodiment 1] Figure 1 is a block diagram showing an example configuration of the behavior determination device 10A of this embodiment. As shown in Figure 1, the device 10A includes a gaze information acquisition unit 11, a behavior determination unit 12, and a determination result output unit 13. Although not shown, the device 10A may also include a storage unit.

[0029] The device 10A may be, for example, a single device including the aforementioned parts, or it may be a device in which the aforementioned parts can be connected via a communication network. Furthermore, the device 10A can be connected to an external device described later via the communication network. The communication network is not particularly limited and can use a known network, for example, it may be wired or wireless. Examples of the communication network include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, LPWA, etc. The wireless communication may be in the form of direct communication between devices (Ad Hoc communication), infrastructure communication, indirect communication via an access point, etc. The device 10 may be, for example, incorporated into a server as a system. Furthermore, the device 10A may be, for example, a personal computer (PC, e.g., desktop or notebook type), smartphone, tablet terminal, etc., on which the program of the present invention is installed. The device 10A may also be in the form of cloud computing or edge computing, for example, in which at least one of the above-mentioned parts is on a server and the other parts are on a terminal.

[0030] Figure 2 illustrates a block diagram of the hardware configuration of the device 10A. The device 10A includes, for example, a central processing unit (CPU, GPU, etc.) 101, memory 102, bus 103, storage device 104, input device 105, output device 106, communication device 107, etc. Each part of the device 10A is interconnected via the bus 103 through its respective interface (I / F).

[0031] The central processing unit 101 operates in coordination with other components via controllers (system controller, I / O controller, etc.) and is responsible for the overall control of the device 10A. In the device 10A, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as a gaze information acquisition unit 11, an action determination unit 12, and a determination result output unit 13. The device 10A may also be equipped with other computing devices such as a CPU, GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), or a combination thereof.

[0032] Bus 103 can also be connected to external devices, for example. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, and external imaging devices. The device 10 can be connected to an external network (the aforementioned communication network) via a communication device 107 connected to bus 103, for example, and can also be connected to other devices via the external network.

[0033] Memory 102 may be, for example, main memory. When the central processing unit 101 performs processing, memory 102 reads various operational programs, such as the program of the present invention, stored in the storage device 104 (described later), and the central processing unit 101 receives data from memory 102 and executes the program. The main memory may be, for example, RAM (random access memory). Alternatively, memory 102 may be, for example, ROM (read-only memory).

[0034] The storage device 104 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 104 stores an operating program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes to the recording medium. The recording medium is not particularly limited and may be internal or external, for example, an HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD). If the device 10A includes the storage unit, for example, the storage device 104 functions as the storage unit. The storage unit can store, for example, the acquired gaze information, the judgment results, etc.

[0035] In this device 10A, the memory 102 and storage device 104 can also store various types of information, such as log information, information obtained from an external database (not shown) or external devices, information generated by this device 10A, and information used by this device 10A when executing processing. In this case, the memory 102 and storage device 104 may store, for example, information about the person to be judged (e.g., gaze information, identification information) and information about the object to be judged (device identification number, device type, etc.). At least some of the information may be stored on an external server other than the memory 102 and storage device 104, or it may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0036] The device 10A further includes, for example, an input device 105 and an output device 10. The input device 105 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; an audio input means such as a microphone; and so on. The output device 106 may include, for example, a display device such as an LED display or liquid crystal display; an audio output device such as a speaker; a printer; and so on. In this embodiment 1, the input device 105 and the output device 106 are configured separately, but the input device 105 and the output device 106 may be configured as an integrated unit, such as a touch panel display.

[0037] Next, an example of the behavior determination method of this embodiment will be described based on the flowchart S10A in Figure 3. The behavior determination method of this embodiment is carried out as follows, for example, using the behavior determination device 10A in Figure 1 or Figure 2. Note that the behavior determination method of this embodiment is not limited to the use of the device 10A in Figure 1 or Figure 2.

[0038] First, the gaze information acquisition unit 11 acquires gaze information, including gaze direction information (S11A, gaze information acquisition step). The gaze direction information can be obtained, for example, using a gaze direction determination device that determines the gaze direction from a face image. For example, a face image of the person to be determined is captured by a camera or the like, and the gaze direction is determined from the characteristics of the face image. Such gaze determination devices are commercially available or licensed. The gaze information acquisition unit 11 may acquire the gaze direction information determined by the gaze direction determination device via a communication network. In the present invention, the person to be determined may be one person or multiple people. In the present invention, the object to be determined may be one person or multiple objects. The gaze information acquisition unit 11 may also acquire, for example, distance information from the person to be determined to the object to be determined. The distance information can be obtained, for example, using a distance sensor that measures the distance between objects. For example, the distance from the object to be determined to the person to be determined may be measured using the distance sensor or the like, and the gaze information acquisition unit 11 may acquire the measured distance information, for example, via a communication network. The object to be determined is not particularly limited and can include, for example, a display device such as a display, and a specific example is a display for a medical device. The medical device is preferably, for example, a device related to X-ray imaging, and more preferably a display for a medical device used in ERCP.

[0039] Next, the behavior determination unit 12 determines whether the person being determined is looking at the object based on the gaze direction information (S12A, behavior determination step). For example, the behavior determination unit 12 determines that the person is "looking" if the object exists in the direction of the person's gaze based on the gaze direction information. Also, for example, the behavior determination unit 12 determines that the person is "not looking" if the object does not exist in the direction of the person's gaze. However, the processing by the behavior determination unit 12 is not limited to this, and the behavior determination unit 12 may, for example, determine whether the person is looking at the object based on the gaze direction information and the distance information. Specifically, the behavior determination unit 12 may, for example, calculate the coordinates of the space the person is looking at based on the gaze direction information and the distance information, and determine that the person is "looking" if the object exists in the spatial coordinates. Also, in this case, the behavior determination unit 12 may determine that the person is "not looking" if the object does not exist in the spatial coordinates. Furthermore, the behavior determination unit 12 may, for example, determine whether or not a person is looking at the object only if the distance information between the person and the object is within a threshold. The threshold is not particularly limited and can be set arbitrarily. This allows, for example, a physician performing ERCP to determine whether or not they are looking at the medical device's display, and prevents a non-physician (e.g., a nurse) from making an incorrect determination as to whether or not they are looking at the display.

[0040] The behavior determination unit 12 may, for example, determine that the person is "looking" if the object to be determined is present in the direction of the person's gaze for a certain period of time or longer, based on the gaze direction information. Alternatively, the behavior determination unit 12 may determine that the person is "not looking" if, for example, the object to be determined is present in the direction of the person's gaze for a certain period of time, and then the object to be determined is absent from that direction for a certain period of time. Furthermore, the behavior determination unit 12 may, for example, reset the count of the period for determining "not looking" if the object to be determined is absent from the direction of the gaze for a certain period of time, and the presence of the object to be determined in the direction of the person's gaze is detected before the period of time has elapsed. This configuration can suppress, for example, misjudgments due to so-called "glancing."

[0041] The judgment result output unit 13 then outputs a judgment result of "seeing" or "not seeing" (S13A, judgment result output step). The output may be, for example, an image output (e.g., displaying the judgment result on a display), an audio output (e.g., emitting a sound from a speaker when "seeing"), or a control signal output (controlling the object to be judged, e.g., turning the device to be judged on or off, turning the monitor on or off, etc.). In the case of an X-ray device such as an ERCP, the on / off of X-ray imaging may be controlled (automatically or manually) based on the output of the judgment result.

[0042] [Embodiment 2] A block diagram of the device 10B of this embodiment is shown in Figure 4, and a hardware configuration diagram of the device 10B of this embodiment is shown in Figure 5. As shown in Figures 4 and 5, the device 10B includes a gaze direction determination unit 14 and has the functions of the gaze direction determination unit device described in Embodiment 1. Other than this, this embodiment is the same as Embodiment 1.

[0043] Next, an example of the behavior determination method of this embodiment will be described based on the flowchart S10B in Figure 6. The behavior determination method of this embodiment is carried out as follows, for example, using the behavior determination device 10B in Figure 4 or Figure 5. Note that the behavior determination method of this embodiment is not limited to the use of the device 10B in Figure 4 or Figure 5.

[0044] First, the gaze direction determination unit 14 determines the gaze direction from the face image of the person to be determined, captured by a camera or the like, and generates gaze direction information (S11B, gaze direction determination step). The subsequent processing is the same as in Embodiment 1.

[0045] [Embodiment 3] A block diagram of the device 10C of this embodiment is shown in Figure 7, and a hardware configuration diagram of the device 10C of this embodiment is shown in Figure 8. As shown in Figures 7 and 8, the device 10C includes a composite image generation unit 15, the image synthesis unit 15 generates a marker image of the person to be judged as a composite image, the gaze information acquisition unit 11 acquires an image of the person to be judged, and the judgment result output unit 13 synthesizes the marker image and the image of the person to be judged and displays it when the behavior determination unit 12 determines that the person to be judged is looking at the object to be judged. Other than this, this embodiment is the same as Embodiment 1.

[0046] Next, an example of the behavior determination method of this embodiment will be described based on the flowchart S10C in Figure 9. The behavior determination method of this embodiment is carried out as follows, for example, using the behavior determination device 10C in Figure 7 or Figure 8. Note that the behavior determination method of this embodiment is not limited to the use of the behavior determination device 10C in Figure 7 or Figure 8.

[0047] First, the gaze information acquisition unit 11 acquires an image of the person to be judged (S11C, gaze information acquisition step). The image of the person to be judged is, for example, an image taken of the person to be judged, and may be the face image of the person to be judged. The gaze information acquisition unit 11 may acquire an image of the person to be judged by an imaging device such as a camera via a communication network. The gaze information acquisition unit 11 also acquires the gaze information, similar to the embodiment 1 described above.

[0048] Next, the composite image generation unit 15 generates a composite image of the marker image of the person to be judged (S11C, composite image generation step). The composite image is, for example, information for distinguishing the person to be judged from the person to be judged image described later, and may include, for example, an image of a frame line to surround the person to be judged. The marker image may be, for example, one type or two or more types. The composite image generation unit 15 may also generate a gaze direction display image for each person to be judged based on the gaze direction information. The gaze direction display image is, for example, an image that visualizes the gaze of the person to be judged, and may include, for example, a line connecting the eye of the person to be judged and the object to be judged (for example, also called a "gaze beam").

[0049] Next, the action determination unit 12 performs the action determination process in the same manner as S12A in Embodiment 1.

[0050] The judgment result output unit 13 then, if the behavior determination unit 12 determines that the person being judged is looking at the object being judged, combines the marker image and the image of the person being judged and displays it. Specifically, the judgment result output unit 13 can, for example, generate and display a combined image in which the person being judged who is determined to be looking at the object being judged is surrounded by the marker image (frame image). The position where the marker image is combined is not particularly limited, and the judgment result output unit 13 may, for example, recognize the face image of the person being judged included in the image of the person being judged using face recognition technology and combine the marker image at the position of the face image, or it may combine the marker image so as to surround the entire body of the person being judged. If there are two or more types of marker images, for example, different types of marker images may be combined and displayed for the person being judged who is determined to be looking at the object being judged and for the person being judged who is determined not to be looking at the object being judged. Specifically, this process involves surrounding individuals determined to be looking with a red border and individuals determined not to be looking with a white border. Furthermore, if the gaze direction display image has been generated, the judgment result output unit 13 may, for example, combine the gaze direction display image with the image of the person being judged and display it.

[0051] According to the device 10C of this embodiment, for example, the marker image can be synthesized and displayed for a person who is determined to be looking at the object to be judged, making it easier to determine whether or not the person is looking at the object to be judged.

[0052] [Embodiment 4] Next, Figures 10 to 13 show examples of applying the apparatus 10A of Embodiment 1, the apparatus 10B of Embodiment 2, or the apparatus 10C of Embodiment 3 (collectively referred to as apparatus 10) to ERCP. In this embodiment, since apparatus 10 is applied to ERCP, the object to be determined is, for example, the display of a medical device.

[0053] Figure 10 shows the relationship between the examination room monitors M1-M4, the imaging device (camera) C1, the main device 10, and the judgment result display device (display) D1. The four examination room monitors M1-M4 are arranged in pairs, one above the other and two to the left and right. Monitor M1 is a medical record screen display monitor, monitor M2 is an endoscopic image display monitor, monitor M3 is a recorded X-ray display monitor, and monitor M4 is a real-time X-ray display monitor. The "recorded X-ray display monitor" refers to a monitor that displays images captured (recorded) from X-ray images taken by an X-ray device outside the main device 10. In this embodiment, the judgment target "★" that the device 10 determines whether or not the person being judged is looking at is monitor M4. Camera C1 is positioned in the center of the four monitors M1-M4. Camera C1 is connected to the main device 10. The main device 10 is connected to display D1.

[0054] Figures 11 and 12 show how the system determines whether or not a person is looking at monitor M4. As shown in both figures, the system estimates the gaze direction of two people within the detection range and determines in real time whether or not they are looking at the lower right monitor M4. If they are looking at it, the word "In Use" is displayed on display D1 (Figure 11); if they are not looking at it, the word "No User" is displayed on display D1 (Figure 12). Although not shown in the figures, the system may also emit a signal sound from the speaker when the person is looking at it and not when they are not (or vice versa), or it may emit different signal sounds depending on whether the person is looking at it or not.

[0055] In Figure 13, the judgment status is displayed on monitor M4. In this figure, the person being judged is displayed with their face surrounded by a rectangular frame, and the rectangular frame is the marker image of the person being judged according to the present invention. Also in this figure, a line indicating the direction of the gaze of the person being judged is shown, and the line is the gaze direction display image according to the present invention. Although the judgment status is displayed on monitor M4 in Figure 13, the present invention is not limited to this, and for example, it may be displayed on display D1 in Figures 10 to 12.

[0056] [Embodiment 5] Furthermore, the present invention may also be in the following embodiments.

[0057] In this invention, in order to eliminate so-called "glancing," for example, if the monitor is looked at for less than a certain period of time, it may not be determined that the person is looking at it, but if the monitor is looked at for a certain period of time or longer, it may be determined that the person is looking at it. This prevents, for example, a false determination that a person being judged is "looking" if they "glancing" at the object being judged while looking at something else. Furthermore, for example, if it is determined that the monitor is being looked at for a certain period of time or longer, it may be determined that the person is "not looking" after a certain period of time has elapsed since their gaze left the monitor. In this case, for example, if the person being judged looks at the object being judged during the certain period of time after their gaze left the monitor before it is determined that they are "not looking," the count of the certain period of time for determining "not looking" may be reset. This prevents, for example, a false determination that the person being judged is "not looking" if they "glancing" at something other than the monitor being judged. In addition, if it is detected that the monitor has been staring at it for a certain period of time or longer, a red frame line and text indicating that it is in use may be displayed around the perimeter of the monitor, and an alarm sound may be output from the speaker, and the display and sound output may be stopped when the gaze leaves the monitor.

[0058] Furthermore, in this invention, in the image captured by the camera, people who are within a certain distance range (determination range) from the camera may be considered as subjects for determination, and people outside the determination range may not be considered as subjects for determination. The distance from the camera may be estimated from the distance between the eyes of the person in the image captured by the camera.

[0059] Furthermore, in the present invention, it may be determined whether the person detected across multiple frames in the image captured by the camera is the same person (tracking).

[0060] Furthermore, in the present invention, the gaze information of multiple frames in the image captured by the camera may be averaged (for example, averaging of the gaze angles) to suppress sudden false detections.

[0061] [Embodiment 6] The program of this embodiment is a program that causes a computer to execute each step of the behavior determination method described above. Specifically, the program of this embodiment is a program that causes a computer to execute the eye-tracking information acquisition procedure, the behavior determination procedure, and the determination result output procedure.

[0062] The aforementioned eye-gaze information acquisition procedure acquires eye-gaze information including the gaze direction information of the person to be judged, The aforementioned behavior determination procedure determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The procedure for outputting the judgment result outputs the judgment result.

[0063] Furthermore, the program of this embodiment can also be described as a program that causes a computer to function as a procedure for acquiring gaze information, a procedure for determining behavior, and a procedure for outputting the determination result.

[0064] The program of this embodiment can be based on the description in the aforementioned behavior determination device and behavior determination method of the present invention. Each of the aforementioned steps can be read as, for example, "step" instead of "process". The program of this embodiment may also be recorded on, for example, a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited and includes, for example, random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, floppy disk (FD), and the like.

[0065] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the configuration and details of the present invention can be understood by those skilled in the art within the scope of the present invention.

[0066] <Note> Some or all of the above embodiments may be described as follows, but are not limited to the following: (Note 1) It includes a gaze information acquisition unit, an action determination unit, and a determination result output unit. The aforementioned gaze information acquisition unit acquires gaze information including the gaze direction information of the person to be judged, The behavior determination unit determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The judgment result output unit outputs the judgment result. Behavior determination device. (Note 2) The gaze information acquisition unit further acquires distance information from the person to be judged to the object to be judged, The behavior determination device described in Appendix 1 determines whether the person to be determined is looking at the object to be determined, based on the gaze direction information and the distance information. (Note 3) Furthermore, it includes a gaze direction determination unit, The gaze direction determination unit determines the gaze direction based on the facial image information of the person to be determined and generates the gaze direction information. The behavior determination device described in Appendix 1 or 2. (Note 4) Furthermore, it includes a composite image generation unit, The composite image generation unit generates the marker image of the person to be judged as a composite image, The aforementioned gaze information acquisition unit acquires an image of the person to be judged, The judgment result output unit, when the behavior determination unit determines that the person being judged is looking at the object being judged, combines the sign image and the image of the person being judged and displays them. An action determination device as described in any of the appendices 1 to 3. (Note 5) The composite image generation unit generates a gaze direction display image for each person to be judged based on the gaze direction information. The gaze information acquisition unit acquires the image of the person to be judged, The judgment result output unit combines the gaze direction display image and the image of the person to be judged and displays them. A determination device as described in any of the appendices 1 to 4. (Note 6) The object to be determined is a display. A determination device as described in any of the appendices 1 to 5. (Note 7) The aforementioned display is a display for a medical device. The determination device described in Appendix 6. (Note 8) This includes a process for acquiring eye-tracking information, a process for determining behavior, and a process for outputting the determination result. The aforementioned gaze information acquisition step acquires gaze information including the gaze direction information of the person to be judged, The aforementioned behavior determination step determines, based on the gaze direction information, whether the person to be determined is looking at the object to be determined. The judgment result output step outputs the judgment result. Behavior judgment method. (Note 9) The aforementioned line-of-sight information acquisition step further acquires distance information from the person to be judged to the object to be judged, The behavior determination step is the behavior determination method described in Appendix 8, which determines whether the person to be determined is looking at the object to be determined based on the gaze direction information and the distance information. (Note 10) Furthermore, it includes a process for determining the direction of gaze, The gaze direction determination unit determines the gaze direction based on the facial image information of the person to be determined and generates the gaze direction information. The behavior determination method described in Appendix 8 or 9. (Note 11) Furthermore, it includes a composite image generation process, The aforementioned composite image generation step generates the marker image of the person to be judged as a composite image, The eye-tracking information acquisition step involves acquiring the image of the person to be judged, The judgment result output step, when the action judgment step determines that the person being judged is looking at the object being judged, combines and displays the sign image and the person being judged image. The method for determining the action described in Appendix 10. (Note 12) The aforementioned composite image generation step generates a gaze direction display image for each person to be judged based on the gaze direction information, The eye-tracking information acquisition step involves acquiring the image of the person to be judged, The judgment result output step involves combining the gaze direction display image and the image of the person to be judged and displaying them. The behavior determination method described in Appendix 10 or 11. (Note 13) The object to be determined is a display. The action determination method described in any of the appendices 8 to 12. (Note 14) The aforementioned display is a display for a medical device. The method for determining the action described in Appendix 13. (Note 15) This includes procedures for acquiring eye-tracking information, determining behavior, and outputting the determination result. The aforementioned eye-gaze information acquisition procedure acquires eye-gaze information including the gaze direction information of the person to be judged, The aforementioned behavior determination procedure determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The above-mentioned procedure for outputting the judgment result outputs the judgment result, A program in which each of the above steps is performed by a computer. (Note 16) The aforementioned gaze information acquisition procedure further acquires distance information from the person to be judged to the object to be judged, The behavior determination procedure is the program described in Appendix 15, which determines whether the person being determined is looking at the object being determined, based on the gaze direction information and the distance information. (Note 17) Furthermore, it includes a procedure for determining the direction of gaze, The gaze direction determination unit determines the gaze direction based on the facial image information of the person to be determined and generates the gaze direction information. The program described in Appendix 15 or 16. (Note 18) Furthermore, it includes a procedure for generating a composite image, The aforementioned composite image generation procedure generates the marker image of the person to be judged as a composite image, The eye-tracking information acquisition procedure involves acquiring the image of the person to be judged, The judgment result output procedure, when the action judgment procedure determines that the person being judged is looking at the object being judged, combines and displays the sign image and the image of the person being judged. The program described in any of the appendices 15 to 17. (Note 19) The aforementioned composite image generation procedure generates a gaze direction display image for each person to be judged based on the gaze direction information, The eye-tracking information acquisition procedure involves acquiring the image of the person to be judged, The judgment result output procedure involves combining the gaze direction display image and the image of the person to be judged and displaying them together. The program described in any of the appendices 15 to 18. (Note 20) The object to be determined is a display. The program described in any of the appendices 15 to 19. (Note 21) The aforementioned display is a display for a medical device. The program described in Appendix 20. (Note 22) A computer-readable recording medium containing a program described in any of the appendices 15 to 21. [Industrial applicability]

[0067] According to the present invention, it is possible to determine whether or not a subject is looking at an object, and it is widely useful in various fields such as devices with monitors like ERCP. [Explanation of symbols]

[0068] 10(10A, 10B, 10C) Behavior determination device 11 Gaze information acquisition unit 12 Behavior Judgment Department 13. Output section for judgment results 14 Gaze direction determination unit 15. Composite Image Generation Unit 101 Central Processing Unit 102 memory 103 Bus 104 Storage device 105 Input device 106 Output device 107 Communication devices

Claims

1. It includes a gaze information acquisition unit, an action determination unit, and a determination result output unit. The gaze information acquisition unit acquires gaze information, including the gaze direction information of the person to be judged, using a facial image of the person to be judged taken from the side of the object to be judged. The behavior determination unit determines, based on the gaze direction information, whether the person being determined is looking at the object being determined. The object to be determined is a display for equipment related to X-ray imaging. The aforementioned determination result output unit is: If the behavior determination unit determines that the person being determined is looking at the object being determined, it outputs information as a determination result that controls the X-ray imaging equipment to perform X-ray imaging. If the behavior determination unit determines that the person being determined is not looking at the object being determined, it outputs information as a determination result that controls the X-ray imaging equipment to stop the X-ray imaging. Behavior determination device.

2. The gaze information acquisition unit further acquires distance information from the person to be judged to the object to be judged, The behavior determination device according to claim 1, wherein the behavior determination unit determines, based on the distance information, whether the distance from the person to be determined to the object to be determined is within a threshold, and for the person to be determined whose distance is determined to be within a threshold, determines, based on the gaze direction information, whether the person to be determined is looking at the object to be determined.

3. Furthermore, it includes a gaze direction determination unit, The gaze direction determination unit determines the gaze direction based on the facial image information of the person to be determined and generates the gaze direction information. The behavior determination device according to claim 1 or 2.

4. Furthermore, it includes a composite image generation unit, The composite image generation unit generates the marker image of the person to be judged as a composite image, The aforementioned gaze information acquisition unit acquires an image of the person to be judged, The judgment result output unit, when the behavior determination unit determines that the person being judged is looking at the object being judged, combines the sign image and the image of the person being judged and displays them. The behavior determination device according to any one of claims 1 to 3.

5. The composite image generation unit generates a gaze direction display image for each person to be judged based on the gaze direction information. The gaze information acquisition unit acquires the image of the person to be judged, The judgment result output unit combines the gaze direction display image and the image of the person to be judged and displays them. The behavior determination device according to claim 4.

6. If it is determined that the person being judged is looking at the object being judged, the judgment result output unit outputs information to control the X-ray imaging process. The behavior determination device according to any one of claims 1 to 5, wherein if it is determined that the person to be determined is not looking at the object to be determined, the determination result output unit outputs information to control the stopping of the X-ray imaging.

7. The information for controlling the X-ray imaging and the information for controlling the X-ray imaging include at least one of the information for automatically controlling the equipment related to the X-ray imaging and the information for manually controlling it. The action determination device according to any one of claims 1 to 6, wherein the manually controlled information includes at least one of an image output and an audio output for manually controlling the on / off of X-ray imaging according to the determination result.

8. This includes a process for acquiring eye-tracking information, a process for determining behavior, and a process for outputting the determination result. The eye-line information acquisition step acquires eye-line information, including the direction of the gaze of the person to be judged, using a facial image of the person to be judged taken from the side of the object to be judged. The aforementioned behavior determination step determines, based on the gaze direction information, whether the person to be determined is looking at the object to be determined. The object to be determined is a display for equipment related to X-ray imaging. The aforementioned determination result output step is: In the aforementioned behavior determination step, if it is determined that the person being determined is looking at the object being determined, the determination result outputs information to control the X-ray imaging equipment to perform X-ray imaging. In the behavior determination step, if it is determined that the person being determined is not looking at the object being determined, the determination result outputs information that controls the X-ray imaging equipment to stop the X-ray imaging. A method for determining actions in which each step is executed by a computer.

9. This includes procedures for acquiring eye-tracking information, determining behavior, and outputting the determination result. The eye-line information acquisition procedure acquires eye-line information, including the direction of the gaze of the person being judged, using a facial image of the person being judged taken from the side of the object being judged. The aforementioned behavior determination procedure determines, based on the gaze direction information, whether the person to be determined is looking at the object to be determined. The object to be determined is a display for equipment related to X-ray imaging. The procedure for outputting the judgment result is as follows: In the aforementioned behavior determination procedure, if it is determined that the person being determined is looking at the object being determined, the determination result outputs information to control the X-ray imaging equipment to perform X-ray imaging. In the aforementioned behavior determination procedure, if it is determined that the person being determined is not looking at the object being determined, the determination result outputs information that controls the X-ray imaging equipment to stop the X-ray imaging.

10. A computer-readable recording medium that stores the program described in claim 9.

Images