A system for obtaining confident operability, a method for obtaining confident operability, and a program.

Abstract

This system provides a confidence-based operation acquisition processing system that can achieve a high level of confidence in operations performed with high precision in control systems that execute complex and precise movements operated by humans, at a low cost, and objectively and appropriately evaluate the confidence level of operation in real time. [Solution] In the confidence operation degree acquisition processing system 1000, the confidence operation degree acquisition processing device includes a gaze-leading amount acquisition processing device that acquires the gaze-leading amount when an operator operates an object in the direction of a target object while looking at the display screen of a display device, and a confidence operation degree acquisition processing device that acquires the degree of confidence in operation based on the confidence operation degree derived from the gaze-leading amount. When operating an object in the direction of a target object, the gaze-leading amount, which indicates how far ahead the operator's viewpoint is in the direction of the target object, is objective data that eliminates subjective elements, so the operator's confidence in operation can be evaluated by the confidence operation degree based on the gaze-leading amount.

Description

[Technical Field]

[0001] This invention relates to a technology for detecting the tactile sensation when a person is confidently manipulating an object. [Background technology]

[0002] In recent years, control systems have been developed that allow humans to perform complex and precise movements (for example, control systems using robots, robotic arms, drones, and control devices for manipulating objects). In such control systems, when humans perform operations requiring high precision, it is important to have a confident sense of operation (confident operation) that is gained through proficiency in the operation. If this confident operation can be evaluated and detected, it can be applied to training for skill improvement, so there is a need for technology to appropriately evaluate and detect this confident operation. This confident operation is difficult to evaluate objectively because it is strongly subjective, but for example, Non-Patent Document 1 discloses a technology that measures brain activity when a person is performing complex or precise movements in a control system and reads the operator's sense of operation. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ohata R, Asai T, Kadota H, Shigemasu H, Ogawa K, Imamizu H. Sense of Agency Beyond Sensorimotor Process: Decoding Self-Other Action Attribution in the Human Brain. Cereb Cortex. 2020;30(7):4076-4091. [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the technology disclosed in Non-Patent Document 1 requires the use of a large-scale fMRI (functional Magnetic Resonance Imaging) device to measure brain activity while a person performs a predetermined operation, making it difficult to implement at low cost. Furthermore, the technology disclosed in Non-Patent Document 1 requires reading the sensation of the operation from local fMRI signals of a specific time interval, which limits the information available for analysis. In addition, fMRI signals have a time delay of several seconds, making it difficult to obtain measurement results in real time.

[0005] In view of the above problems, the present invention aims to realize a confidence-based operation acquisition system, a confidence-based operation acquisition method, and a program that can achieve a confident operation feeling (confident operation feeling) when a person performs an operation requiring high precision, at low cost, and that can objectively and appropriately evaluate and detect it in real time, in a control system that can perform complex and precise operations by human operation. [Means for solving the problem]

[0006] To solve the above problems, a representative example (one aspect) of the invention disclosed in this application is a confidence level acquisition processing system for acquiring a confidence level when performing a predetermined operation by manipulating an object to a target, comprising: an imaging device that captures a three-dimensional space including the target object and the object to be manipulated; and a two-dimensional display device that displays the image captured by the imaging device on a display screen. The system includes a gaze detection device that detects the gaze of an operator who is operating an object while viewing the display screen of a 2D display device, and a confident operation degree acquisition processing device.

[0007] The confident operation degree acquisition processing device comprises a setting processing unit, a two-dimensional coordinate system acquisition processing unit, an object position detection unit, an operation target position detection unit, a line of sight detection processing unit, a line of sight preceding amount acquisition processing unit, and a confident operation degree acquisition processing unit.

[0008] The setting processing unit acquires and stores information about the location where the 2D display device is placed, information about the display screen, information about the target object, and information about the object to be operated on.

[0009] The 2D coordinate system acquisition processing unit sets a 2D coordinate system that defines the plane including the display screen, based on the location information of the 2D display device and the display screen information held in the setting processing unit.

[0010] The object position detection unit detects the target object's 2D coordinate position, which is the position of the target object in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target object held in the setting processing unit.

[0011] The target position detection unit detects the target's 2D coordinate position, which is the position of the target in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target held in the setting processing unit.

[0012] The gaze detection processing unit detects the 2D coordinate position of the gaze point, which is the position in the 2D coordinate system of the gaze point of the operator on the display screen, based on the intersection point between the operator's gaze detected by the gaze detection device and the display screen of the 2D display device.

[0013] The gaze-ahead quantity acquisition processing unit acquires the gaze-ahead quantity based on the positional relationship on the display screen between the target object's 2D coordinate position, the manipulated object's 2D coordinate position, and the gaze point's 2D coordinate position.

[0014] The confident operation degree acquisition processing unit acquires a confident operation degree, which indicates the degree to which the operator is operating the target object with confidence, based on the gaze-leading quantity. [Effects of the Invention]

[0015] According to the present invention, in a control system that can perform complex or precise operations through human operation, it is possible to realize a confident operation level acquisition system, a confident operation level acquisition method, and a program that can achieve a confident operation level (confident operation level) at low cost when a human is performing an operation requiring high precision, and that can objectively and appropriately evaluate and detect this level in real time. [Brief explanation of the drawing]

[0016] [Figure 1] A schematic diagram of the confident operation level acquisition processing system 1000 according to the first embodiment. [Figure 2] This diagram schematically shows the robot Rbt1, robot arm Rbt1.arm, and object Obj_tg, which are the targets of the operation in the confident operation degree acquisition processing system 1000 according to the first embodiment, as well as the 3D space SP1 on which the imaging device Cam1 is installed, and the 3D space SP2 on which the 2D display device Dev_disp, gaze measurement device Dev_vl, operator Psn1, and operation device (controller) Dev_ctrl are located. [Figure 3] A flowchart of the process performed by the confident operation level acquisition processing system 1000 according to the first embodiment. [Figure 4] This diagram schematically shows the situation when operator Psn1 is operating the robot arm Rbt1.arm using the control device (controller) Dev_ctrl while viewing the display screen of the 2D display device Dev_disp in the confident operation degree acquisition processing system 1000 according to the first embodiment. [Figure 5] This figure shows a graph (an example) illustrating the relationship between the confident operation score and time (number of operations) obtained by the confident operation score acquisition processing system 1000 according to the first embodiment. [Figure 6] This diagram schematically illustrates the situation in the confident operation degree acquisition processing system when operator Psn1 is operating the robot arm Rbt1.arm using the control device (controller) Dev_ctrl while viewing the display screen of the 2D display device Dev_disp. [Figure 7] A diagram showing the CPU bus configuration. [Modes for carrying out the invention]

[0017] [First Embodiment] The first embodiment will be described below with reference to the drawings.

[0018] <1.1: Configuration of the system for obtaining confident operational accuracy> Figure 1 is a schematic diagram of the confident operation level acquisition processing system 1000 according to the first embodiment.

[0019] Figure 2 is a schematic diagram showing the robot Rbt1, robot arm Rbt1.arm, and object Obj_tg, which are the targets of the operation in the confident operation degree acquisition processing system 1000 according to the first embodiment, as well as the 3D space SP1 in which the imaging device Cam1 is installed, and the 3D space SP2 in which the 2D display device Dev_disp, gaze measurement device Dev_vl, operator Psn1, and operation device (controller) Dev_ctrl are located.

[0020] The confident operability acquisition processing system 1000 includes, for example, an imaging device Cam1, a two-dimensional display device Dev_disp, an eye-tracking device Dev_vl, and a confident operability acquisition processing device 100, as shown in Figure 1. The imaging device Cam1, the two-dimensional display device Dev_disp, the eye-tracking device Dev_vl, and the confident operability acquisition processing device 100 may be connected by wire or wireless, or they may be connected via a transmission line, network, etc.

[0021] The imaging device Cam1 is an imaging device for capturing images of a three-dimensional space (for example, the three-dimensional space SP1 in Figure 2) in which a control system (for example, a robot, robotic arm, drone, or control device for manipulating an object) is installed, which can be operated by a human to perform complex or precise movements. The imaging device Cam1 is realized by an imaging device (for example, a camera) equipped with an image sensor (for example, a CMOS image sensor or a CCD image sensor) that can capture color images (moving images (video images)) of the three-dimensional space in which the control system is installed.

[0022] The imaging device Cam1 is positioned to capture both the object being operated on (for example, the robot arm Rbt1.arm of the robot Rbt in Figure 2) and the target object (for example, the target object Obj_tg in Figure 2), and captures a three-dimensional space including the object being operated on (for example, the robot arm Rbt1.arm of the robot Rbt in Figure 2) and the target object (for example, the target object Obj_tg in Figure 2). The imaging device Cam1 then outputs the video signal that forms the captured image (a video signal that forms an image composed of multiple frames (frame images)) as the video signal Din_video to the two-dimensional display device Dev_disp and the confident operation degree acquisition processing device 100.

[0023] The 2D display device Dev_disp has a display screen (a display screen for displaying 2D images (2D video)) and receives the video signal Din_video output from the imaging device Cam1 as input, and displays the video (frame images that make up a video) formed by the video signal Din_video on the display screen.

[0024] The eye-tracking device Dev_vl is a device for detecting a person's gaze and comprises an infrared irradiator (e.g., an infrared LED) and an infrared camera. The eye-tracking device Dev_vl irradiates a person's face (including the area of ​​the eyes) with infrared light (e.g., near-infrared light) using the infrared irradiator (e.g., an infrared LED) and captures the reflected light from the person's face (reflected light of the irradiated infrared light (e.g., near-infrared light)) with the infrared camera. The eye-tracking device Dev_vl then detects the cornea and pupil of the person's eye from the image (infrared image) captured by the infrared camera and detects the person's gaze from the positional relationship between the detected cornea and pupil. The eye-tracking device Dev_vl then outputs data including the detection result data of the person's gaze as data Din_vl to the confident operation degree acquisition processing device 100.

[0025] As shown in Figure 1, the confident operation degree acquisition processing device 100 comprises a setting processing device 1, a two-dimensional coordinate system acquisition processing device 2, an object position detection processing device 3, an operation target position detection processing device 4, a gaze detection processing device 5, a gaze preceding amount acquisition processing device 6, a confident operation degree acquisition processing device 7, and a confident operation degree time series analysis processing device 8.

[0026] The setting processing unit 1 is a functional unit that performs setting processing for various types of information, and for example, sets the following information. (1) Information about the 2D display device Dev_disp (information about the 3D space in which the 2D display device Dev_disp is set, and information about the position, size, orientation, etc. of the display screen of the 2D display device Dev_disp) (This information will be referred to as data Info_disp.) (2) Information about the object (information to identify the object (for example, information such as the size, shape, color, and location of the object)) (This information will be referred to as data Info_Obj_tg.) (3) Information about the object to be operated on (information to identify the object to be operated on (for example, information such as the size, shape, color, and position of the object to be operated on)) (This information will be referred to as data Info_ctrld.) The above information may be input to the setting processing unit 1 of the confidence operation level acquisition processing device 100 manually or automatically (for example, by taking a picture of the target and performing image recognition processing on the image acquired from the picture), for example, via an interface (not shown).

[0027] The setting processing unit 1 outputs data containing the information described in (1) above as data Info_disp to the 2D coordinate system acquisition processing unit 2, outputs data containing the information described in (2) above as data Info_Obj_tg to the object position detection processing unit 3, and outputs data containing the information described in (3) above as data Info_ctrld to the operation target position detection processing unit 4.

[0028] The 2D coordinate system acquisition processing unit 2 receives the data Info_disp output from the setting processing unit 1. Based on the data Info_disp, the 2D coordinate system acquisition processing unit 2 identifies the plane containing the display screen of the 2D display device Dev_disp, and acquires (sets) the 2D coordinate system that defines the identified plane. The 2D coordinate system acquisition processing unit 2 then outputs data including the acquired (set) 2D coordinate system data (data for identifying the 2D coordinate system (for example, data of the basis vectors of the 2D coordinate system)) as data D1_2D_crd to the object position detection processing unit 3, the operation target position detection processing unit 4, and the gaze detection processing unit 5.

[0029] The object position detection processing unit 3 receives the video signal Din_video output from the imaging device Cam1, the data Info_Obj_tg output from the setting processing unit 1, and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. The object position detection processing unit 3 identifies the object Obj_tg specified by the data Info_Obj_tg in the frame image formed by the video signal Din_video (for example, by performing image recognition processing on the frame image formed by the video signal Din_video), and detects (acquires) the position (position of the object Obj_tg) on ​​the 2D plane on the display screen of the 2D display device Dev_disp (let's call this plane plane PL_2D) that is identified by the data D1_2D_crd. The object position detection processing unit 3 then outputs the data including the detected (acquired) data of the position of the object Obj_tg on plane PL_2D to the gaze-ahead amount acquisition processing unit 6 as data D21.

[0030] The object position detection processing unit 4 receives the video signal Din_video output from the imaging device Cam1, the data Info_cntld output from the setting processing unit 1, and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. The object position detection processing unit 4 identifies the object to be operated (for example, the robot arm Rbt1.arm) identified by the data Info_Obj_cntld in the frame image formed by the video signal Din_video (for example, by performing image recognition processing on the frame image formed by the video signal Din_video), and detects (acquires) the position (position of the object to be operated) on the 2D plane on the display screen of the 2D display device Dev_disp (let's call this plane plane PL_2D) identified by the data D1_2D_crd. The object position detection processing unit 3 then outputs the data including the detected (acquired) data of the position of the object to be operated on plane PL_2D to the gaze-ahead amount acquisition processing unit 6 as data D22.

[0031] The gaze detection processing unit 5 receives the data Din_vl output from the gaze measurement device Dev_vl and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. Based on the data Din_vl, the gaze detection processing unit 5 detects the gaze position on the 2D plane PL_2D on the display screen of the 2D display device Dev_disp, which is identified by the data D1_2D_crd, and outputs the data including the detected gaze detection position as data D23 to the gaze preceding quantity acquisition processing unit 6.

[0032] The gaze-ahead quantity acquisition processing unit 6 receives data D21 output from the object position detection processing unit 3, data D22 output from the operation target position detection processing unit 4, and data D23 output from the gaze detection processing unit 5. Based on data D21, data D22, and data D23, the gaze-ahead quantity acquisition processing unit 6 executes a process to acquire the gaze-ahead quantity (gaze-ahead quantity acquisition process) and acquires the gaze-ahead quantity. Then, the gaze-ahead quantity acquisition processing unit 6 outputs the data including the acquired gaze-ahead quantity as data D3 to the confident operation degree acquisition processing unit 7.

[0033] The confident manipulativeness acquisition processing unit 7 receives data D3 output from the gaze-leading quantity acquisition processing unit 6, and based on data D3 (gaze-leading quantity), executes a process to acquire confident manipulativeness (confident manipulativeness acquisition process) and acquires confident manipulativeness. Then, the confident manipulativeness acquisition processing unit 7 outputs the data including the acquired confident manipulativeness as data D4 to the confident manipulativeness time series analysis processing unit 8.

[0034] The confident operationality time series analysis processing unit 8 receives data D4 output from the confident operationality acquisition processing unit 7, performs a time series analysis of confident operationality based on data D4 (confident operationality), and outputs the result data of this analysis as data Dout.

[0035] <1.2: Operation of the Confident Operational Ability Acquisition System> The operation of the confident operation level acquisition processing system 1000, configured as described above, will be explained below with reference to the drawings.

[0036] Figure 3 is a flowchart of the process performed by the confident operation level acquisition processing system 1000 according to the first embodiment.

[0037] Figure 4 schematically shows the situation in the confident operation degree acquisition processing system 1000 according to the first embodiment, when operator Psn1 is operating the robot arm Rbt1.arm using the control device (controller) Dev_ctrl while viewing the display screen of the 2D display device Dev_disp.

[0038] Figure 5 is a graph (an example) showing the relationship between the confident operation score and time (number of operations) obtained by the confident operation score acquisition processing system 1000 according to the first embodiment.

[0039] In the following explanation, for the sake of clarity, we will describe the operation of the confident operation degree acquisition processing system 1000 in the following case, where, as shown in Figure 2, an imaging device Cam1, a robot Rbt1 equipped with a robot arm Rbt1.arm (an example of a control system that can perform complex and precise movements by being operated by a person), and an object Obj_tg are installed in a three-dimensional space SP1 (the three-dimensional space shown in the right diagram of Figure 2), and in a three-dimensional space SP2 (the three-dimensional space shown in the left diagram of Figure 2) located away from the three-dimensional space SP1, a person (operator Psn1) operates the robot arm Rbt1.arm using the control device (controller) Dev_ctrl while looking at the two-dimensional display device Dev_disp, moving the robot arm Rbt1.arm from its initial position (home position) in the direction where the object Obj_tg is located, and performing an operation to grasp the object Obj_tg with the robot arm Rbt1.arm. Then, the robot arm Rbt1.arm is operated by the control device (controller) Dev_ctrl to move the robot arm Rbt1.arm from its initial position (home position) towards the object Obj_tg, grasp the object Obj_tg with the robot arm Rbt1.arm, and return the robot arm Rbt1.arm to its initial position (home position), and this operation is performed as a single operation sequence (this operation sequence Sq). iLet i be an integer, 1 ≤ i ≤ N, and N be a natural number, and repeat the operation sequence N times.

[0040] The following describes the processes performed by the confident operation level acquisition system 1000, referring to the flowchart in Figure 3.

[0041] (Step S1): In step S1, the configuration process is executed. Specifically, the following processes are performed.

[0042] The configuration processing unit 1 configures the information of the 2D display device Dev_disp. Specifically, the configuration processing unit 1 acquires information such as the position, size, and orientation of the display screen (plane PL_2D) of the 2D display device Dev_disp in the 3D space SP2 (see the left diagram in Figure 2). Then, the configuration processing unit 1 outputs the data containing the information of the 2D display device Dev_disp acquired (configured) above as data Info_disp to the 2D coordinate system acquisition processing unit 2.

[0043] Furthermore, the three-dimensional space SP2 is defined by the three-dimensional coordinate system shown in the left diagram of Figure 2. That is, the three-dimensional space SP2 is defined as having its origin at O ​​as shown in the left diagram of Figure 2. (3D) Let x (3D) Let the axis be the x-axis, and y (3D) Let the axis be the y-axis, z (3D) It shall be defined by a three-dimensional coordinate system with the z-axis as the axis.

[0044] Furthermore, the setting processing unit 1 acquires information about the target object (object) Obj_tg. Specifically, the setting processing unit 1 acquires information to identify the target object (object) Obj_tg (for example, information such as the size, shape, color, and position of the target object (object) Obj_tg). Then, the setting processing unit 1 outputs data containing the information about the target object Obj_tg acquired (set) as described above to the target object position detection processing unit 3 as data Info_Obj_tg.

[0045] In addition, the setting processing unit 1 acquires information on the operation target Rbt1.arm. Specifically, the setting processing unit 1 acquires information for specifying the operation target Rbt1.arm (for example, information such as the size, shape, color, position, etc. of the operation target Rbt1.arm). Then, the setting processing unit 1 outputs data including the information on the operation target Rbt1.arm acquired (set) as described above to the operation target position detection processing unit 4 as data Info_cntld.

[0046] (Step S2): In step S2, two-dimensional coordinate system acquisition processing is executed. Specifically, the following processing is executed.

[0047] Based on the data Info_disp output from the setting processing unit 1, the two-dimensional coordinate system acquisition processing unit 2 specifies a plane PL_2D including the display screen (display screen) of the two-dimensional display device Dev_disp, and acquires (sets) a two-dimensional coordinate system that defines the specified plane PL_2D. Then, the two-dimensional coordinate system acquisition processing unit 2 outputs data including the data of the acquired two-dimensional coordinate system (data for specifying the two-dimensional coordinate system (for example, data on the basis vectors of the two-dimensional coordinate system (the basis vectors of this two-dimensional coordinate system are represented as vectors in a three-dimensional coordinate system that defines the three-dimensional space SP2 (x (3D) component, y (3D) component, z (3D) component))) to the object position detection processing unit 3, the operation target position detection processing unit 4, and the line-of-sight detection processing unit 5 as data D1_2D_crd.

[0048] Note that, as shown in the left figure of FIG. 2, the two-dimensional coordinate system has an origin O (2D) (for example, the point at the upper left end of the display screen of the two-dimensional display device Dev_disp (the position of the pixel at the upper left end)), the x (2D) axis is the x-axis (for example, the horizontal axis of the display screen of the two-dimensional display device Dev_disp), and the y (2D) axis is the y-axis (for example, the vertical axis of the display screen of the two-dimensional display device Dev_disp)).

[0049] (Step S3): In step S3, loop 1 processing (the first loop processing) is started. Loop 1 processing is performed for each measurement period T j For a given (j: integer, 1 ≤ j ≤ M, M: natural number), the measurement is performed (starting from j=1 and incrementing j by +1 until j=M, with each measurement period T). j (Executed against).

[0050] (Step S4): In step S4, loop 2 processing (second loop processing) begins. Loop 2 processing is performed for each operation sequence Sq i For a given number (i: integer, 1 ≤ i ≤ N, N: natural number), the following operation sequence is performed (incrementing i by +1 from i=1 up to i=N): Sq i (Executed against).

[0051] (Step S5A): In step S5A, the object position detection process is executed. Specifically, the following processes are performed.

[0052] The object position detection processing unit 3 receives the video signal Din_video output from the imaging device Cam1, the data Info_Obj_tg output from the setting processing unit 1, and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. The object position detection processing unit 3 identifies the object Obj_tg specified by the data Info_Obj_tg in the frame image formed by the video signal Din_video. For example, the object position detection processing unit 3 performs image recognition processing on the frame image formed by the video signal Din_video to recognize the image region having the features specified by the data Info_Obj_tg as the object Obj_tg (detecting the object Obj_tg on the frame image) (for example, detecting (recognizing) the image region indicated by Obj_tg on the display screen (planar PL_2D) of the 2D display device Dev_disp in Figure 4 as the object Obj_tg).

[0053] Then, the object position detection processing unit 3 detects (acquires) the position of the detected object Obj_tg in the two-dimensional coordinate system specified by the data D1_2D_crd, that is, the two-dimensional coordinate position on the plane PL_2D which includes the display screen of the display device Dev_disp (the two-dimensional coordinate position of the object Obj_tg).

[0054] The object position detection processing unit 3 then outputs data including the detected (acquired) 2D coordinate position of the object Obj_tg on the plane PL_2D (which is denoted as "pos(Obj_tg)=(x0,y0)") to the line of sight advance amount acquisition processing unit 6 as data D21.

[0055] Furthermore, if the object Obj_tg has a predetermined size on the plane PL_2D (i.e., it becomes an image region of a predetermined size on the frame image), the center point (centroid) of the object Obj_tg is defined as the 2D coordinate position of the object Obj_tg pos(Obj_tg)=(x0,y0) (for example, the point shown as (x0,y0) in Figure 4).

[0056] (Step S5B): In step S5B, the operation target position detection process is executed. Specifically, the following processes are performed.

[0057] The target position detection processing unit 4 receives the video signal Din_video output from the imaging device Cam1, the data Info_cntld output from the setting processing unit 1, and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. The target position detection processing unit 4 identifies the target (robot arm Rbt1.arm) identified by the data Info_Obj_cntld in the frame image formed by the video signal Din_video. The target position detection processing unit 4 recognizes the image region having the features identified by the data Info_cntld as the target (robot arm Rbt1.arm) by, for example, performing image recognition processing on the frame image formed by the video signal Din_video (detecting the target (robot arm Rbt1.arm) on the frame image) (for example, detecting (recognizing) the image region indicated by Rbt1.arm on the display screen (plane PL_2D) of the 2D display device Dev_disp in Figure 4 as the target (robot arm Rbt1.arm)).

[0058] The operation target position detection processing unit 4 then detects (acquires) the position of the detected operation target (robot arm Rbt1.arm) in the 2D coordinate system specified by the data D1_2D_crd, that is, the 2D coordinate position on the plane PL_2D which includes the display screen of the display device Dev_disp (the 2D coordinate position of the operation target (robot arm Rbt1.arm)).

[0059] The operation target position detection processing unit 4 then outputs data including the detected (acquired) 2D coordinate position of the operation target (robot arm Rbt1.arm) on the plane PL_2D (which is denoted as "pos(Rbt1.arm)=(x1,y1)") to the gaze direction acquisition processing unit 6 as data D22.

[0060] Furthermore, if the object being operated (robot arm Rbt1.arm) has a predetermined size on the plane PL_2D (i.e., it becomes an image area of ​​a predetermined size on the frame image), the point that represents the object being operated (robot arm Rbt1.arm) (for example, the center point, the center of gravity, the point closest to the object, etc.) is defined as the 2D coordinate position pos(x1,y1) of the object being operated (robot arm Rbt1.arm). In this embodiment, the object-grabbing part at the tip of the robot arm Rbt1.arm, that is, the midpoint of the tip positions of the two gripping parts, is defined as the 2D coordinate position pos(Rbt1.arm)=(x1,y1) of the object being operated (robot arm Rbt1.arm) (for example, the point shown as (x1,y1) in Figure 4).

[0061] (Step S5C): In step S5C, gaze detection processing is performed. Specifically, the following processes are executed.

[0062] The eye-tracking device Dev_vl irradiates the face (including the eyes) of operator Psn1 with infrared light (e.g., near-infrared light) using an infrared irradiator (e.g., infrared LED), and captures the reflected light from operator Psn1's face (the reflected light of the irradiated infrared light (e.g., near-infrared light)) with an infrared camera (it is assumed that operator Psn1's eye position (position in 3D space SP2) is known, and the eye-tracking device Dev_vl has information on operator Psn1's eye position (position in 3D space SP2)). Then, the eye-tracking device Dev_vl detects the cornea and pupil of operator Psn1's eye from the image (infrared image) captured by the infrared camera, and detects operator Psn1's gaze (right eye gaze and left eye gaze) from the positional relationship between the detected cornea and pupil. The eye-tracking device Dev_vl then outputs data containing the detection results of the operator Psn1's gaze (right eye gaze and left eye gaze) as data Din_vl to the eye-tracking processing unit 5 of the confident operation level acquisition processing unit 100.

[0063] The gaze detection processing unit 5 receives the data Din_vl output from the gaze measurement device Dev_vl and the data D1_2D_crd output from the 2D coordinate system acquisition processing unit 2. Based on the data Din_vl (right eye line of sight and left eye line of sight), the gaze detection processing unit 5 detects the gaze position on the 2D plane PL_2D on the display screen of the 2D display device Dev_disp, which is identified by the data D1_2D_crd. In other words, the gaze detection processing unit 5 detects (1) the intersection point of the left eye line of operator Psn1 (a straight line identified in the 3D space SP2 (3D coordinate system)) and the plane PL_2D (let's call this intersection point p_vl_L), which are identified by the 3D coordinate system, and (2) the intersection point of the right eye line of operator Psn1 (a straight line identified in the 3D space SP2 (3D coordinate system)) and the plane PL_2D (let's call this intersection point p_vl).

[0064] The gaze detection processing unit 5 determines the viewpoint position pos(p_vl) on the plane PL_2D as the position in the 2D coordinate system of the intersection point p_vl_L (= intersection point p_vl_R) when the intersection point p_vl_L and the intersection point p_vl_R coincide (when the intersection point of the left eye line of operator Psn1 and the right eye line of operator Psn1 lies on the plane PL_2D (when the point of fixation of operator Psn1 (focal positions of the left and right eyes) lies on the plane PL_2D)). (For example, it detects (recognizes) the image area indicated by p_vl on the display screen (plane PL_2D) of the 2D display device Dev_disp in Figure 4 as the viewpoint position.)

[0065] On the other hand, if the intersection points p_vl_L and p_vl_R do not coincide, the gaze detection processing unit 5 sets the midpoint of the intersection points p_vl_L and p_vl_R as the viewpoint position pos(p_vl) on the plane PL_2D. In this case as well, the viewpoint position pos(p_vl) is written as (x2, y2) (where (x2, y2) is the 2D coordinate position of the midpoint of the intersection points p_vl_L and p_vl_R).

[0066] The gaze detection processing unit 5 then outputs the data including the gaze detection position pos(p_vl)=(x2,y2) detected as described above to the gaze preceding amount acquisition processing unit 6 as data D23.

[0067] Note that steps S5A, S5B, and S5C are executed in parallel. Steps S5B and S5C are executed in the operation sequence Sq i The process is executed continuously during the period in which it is performed, and data D22 and D23 are output continuously during the above period (assuming the frame rate of the video signal Din_video is fr1[fps] (fps: frames per second), the processing of steps S5B and S5C is executed with a period of, for example, k / fr1[s] (k: real number), and data D22 and D23 are output with the above period).

[0068] Furthermore, the processing in step S5A may be performed at the above-mentioned interval, and data D21 may be output at the above-mentioned interval, or data D21 may be output when the position of the object Obj_tg (position on the frame image) changes.

[0069] (Step S6): In step S6, the process of acquiring the gaze-ahead amount is executed. Specifically, the following processes are performed.

[0070] The gaze-ahead quantity acquisition processing unit 6 receives data D21 output from the object position detection processing unit 3, data D22 output from the operation target position detection processing unit 4, and data D23 output from the gaze detection processing unit 5. Based on data D21, data D22, and data D23, the gaze-ahead quantity acquisition processing unit 6 executes a process to acquire the gaze-ahead quantity (gaze-ahead quantity acquisition process). Specifically, the gaze-ahead quantity acquisition processing unit 6 executes the following processes. (1) The gaze-ahead quantity acquisition processing unit 6 acquires the unit vector vec_e_dir1 of the vector (direction vector) (the vector in the direction shown by dir1 in Figure 4) from the 2D coordinate pos(Rbt1.arm)=(x1,y1) (this data is acquired from data D22) of the initial position (home position) (or the current position of the target Rbt1.arm) to the 2D coordinate position pos(Obj_tg)=(x0,y0) (this data is acquired from data D21) of the object Obj_tg. (2) The line of sight advance quantity acquisition processing unit 6 acquires the 2D coordinate position pos(Rbt1.arm)=(x1,y1) of the target Rbt1.arm from data D22 in a continuous time sequence, and by differentiating the 2D coordinate position of the target Rbt1.arm with respect to time, it obtains the velocity v(Rbt1.arm)=(v_x1,v_y1)(v_x1: x component of the velocity v (velocity vector v) of the target Rbt1.arm in the 2D coordinate system). (2D) (components), v_y1: y component of the velocity v of the target Rbt1.arm (y (2D) Obtain the components.

[0071] Then, the gaze-ahead quantity acquisition processing unit 6, k1=v(Rbt1.arm)·vec_e_dir1 •: Operator for taking the dot product k1: dir1 direction component of velocity vector v(Rbt1.arm) v (dir1) (Rbt1.arm)=k1×vec_e_dir1 By performing a process equivalent to this, the velocity vector v (Rbt1.arm) is projected onto the dir1 direction. (dir1) Get (Rbt1.arm). (3) The line of sight advance quantity acquisition processing unit 6 acquires the viewpoint position pos(p_vl)=(x2,y2) on the plane PL_2D from the data D23 in a continuous time sequence, and by differentiating the viewpoint position with respect to time, it obtains the velocity of the viewpoint in the 2D coordinate system v(p_vl)=(v_x2,v_y2)(v_x2: x component of the velocity v (velocity vector v) of viewpoint p_vl (x (2D)Components), v_y2: y component of velocity v at viewpoint p_vl (y (2D) Obtain the components.

[0072] Then, the gaze-ahead quantity acquisition processing unit 6, k2 = v(p_vl) · vec_e_dir1 •: Operator for taking the dot product k2: dir1 component of velocity vector v(p_vl) v (dir1) (p_vl) = k² × vec_e_dir1 By performing a process equivalent to this, the velocity vector v(p_vl) is projected onto the dir1 direction. (dir1) Get (p_vl). (4) The gaze-ahead quantity acquisition processing unit 6 receives the velocity vector v of the target Rbt1.arm acquired in (2). (dir1) (Rbt1.arm) and the velocity vector of the viewpoint obtained in (3) v (dir1) The difference (velocity difference) vector v_diff from (p_vl) (dir1) =v (dir1) (p_vl)-v (dir1) Get (Rbt1.arm). (5) The gaze-ahead quantity acquisition processing unit 6 acquires the velocity difference vector v_diff in (4). (dir1) The magnitude (magnitude of the vector) is obtained as the line-of-sight leading quantity.

[0073] Note that (A) velocity difference vector v_diff (dir1) However, if the direction from the initial position of the target Rbt1.arm to the target object Obj_tg is the same as the direction of dir1, then the sign of the line-of-sight lead obtained above is set to positive (the line-of-sight lead is set to a positive value), and (B) the velocity difference vector v_diff (dir1) However, if the direction dir1 from the initial position of the target Rbt1.arm to the target object Obj_tg is in the opposite direction, the sign of the line-of-sight lead amount obtained above is set to negative (the line-of-sight lead amount is set to a negative value).

[0074] Then, the gaze-leading quantity acquisition processing unit 6 outputs the data including the gaze-leading quantity acquired as described above as data D3 to the confident operation degree acquisition processing unit 7.

[0075] (Step S7): In step S7, the termination determination process for loop 2 (the second loop process) is executed. That is, if the termination determination process determines that the termination condition has been met (i.e., if it is determined that loop 2 has been executed for all operation sequences Sqi), the process is terminated. On the other hand, if the termination determination process determines that the termination condition has not been met, the process returns to step S4, the variable i is incremented by +1, and loop 2 is repeatedly executed.

[0076] (Step S8): In step S8, the process for obtaining the degree of confidence in operation is executed. Specifically, the following processes are performed.

[0077] The confident operation degree acquisition processing unit 7 receives data D3 output from the gaze-ahead quantity acquisition processing unit 6 and executes a process to acquire the confident operation degree (confident operation degree acquisition process) based on data D3 (gaze-ahead quantity). Specifically, the following processes are executed.

[0078] Operation sequence Sq i In this process, the gaze-ahead quantity acquisition process is performed, N s pieces(N s Assuming that a gaze advance quantity of a natural number is obtained, the gaze advance quantity obtained on the kth step is val_advc k (k: integer, 1≦k≦N s When this is the case, the confident operation degree acquisition processing unit 7 performs the operation sequence Sq i The amount of line of sight preceding (this is val_advc(Sq i The value (represented as ) is obtained by performing a process equivalent to the following mathematical formula.

number

[0079] Then, the confident operation degree acquisition processing unit 7 performs the measurement period T j The degree of confidence in operation (this is val_sure_mnp(T j The value (represented as ) is obtained by performing a process equivalent to the following mathematical formula.

number

[0080] Then, the confident operation degree acquisition processing unit 7 uses the confident operation degree val_sure_mnp(T j The data including ) is output as data D4 to the confidence level of operation time series analysis processing unit 8.

[0081] (Step S9): In step S9, the termination determination process for loop 1 processing (the first loop processing) is executed. In other words, if the termination determination process determines that the termination condition has been met (for all measurement periods T), then the termination determination process is performed. j If it is determined that the loop 1 process has been executed, the process is terminated. On the other hand, if the termination determination process determines that the termination condition has not been met, the process returns to step S3, the variable j is incremented by +1, and the loop 1 process is repeatedly executed.

[0082] (Step S10): In step S10, a confident operationality time series analysis process is performed. Specifically, the following processes are performed.

[0083] The confidence in manipulative time series analysis processing unit 8 receives data D4 output from the confidence in manipulative acquisition processing unit 7 and performs time series analysis processing of confidence in manipulative based on data D4 (confidence in manipulative). Specifically, the confidence in manipulative time series analysis processing unit 8 analyzes the measurement period T1 to T M The confident operationality obtained is val_sure_mnp(T1)~val_sure_mnp(T M ) performs time series analysis to determine the skill level of operator Psn1 in operating the control system (operation of robot arm Rbt1.arm). For example, the confident operation degree time series analysis processing unit 8 calculates the confident operation degree val_sure_mnp(T1) ~ val_sure_mnp(T M A time series analysis is performed on the result, and the confidence level val_sure_mnp(T j If the value exceeds a predetermined threshold Th1, it is determined that the operator Psn1's proficiency in operating the control system (operating the robot arm Rbt1.arm) has reached a certain level.

[0084] For example, in the confident operation degree acquisition processing system 1000, if data is acquired showing the relationship between time (number of operations) and confident operation degree as shown in the graph in Figure 5, the confident operation degree time series analysis processing unit 8 will calculate the time time(T j0 )(Measurement period T j0The time immediately after the processing is completed is timetime(T j0 In this case, the threshold Th1 of confident operation is exceeded, so time time(T j0 ) thereafter (measurement period T j0 After the completion of the above process, it is determined that the operator Psn1's proficiency in operating the control system (operating the robot arm Rbt1.arm) has reached a certain level.

[0085] Furthermore, the confident operation degree time series analysis processing unit 8, after a predetermined time (after the time when a predetermined number of operations have been performed), (1) confident operation degree val_sure_mnp(T j ) exceeds a predetermined threshold, and (2) confidence level val_sure_mnp(T j If the amount of change in ) is less than or equal to a predetermined threshold Th2, it is determined that the operator Psn1's proficiency in operating the control system (operating the robot arm Rbt1.arm) has reached a certain level.

[0086] For example, in the confident operation degree acquisition processing system 1000, if data is acquired showing the relationship between time (number of operations) and confident operation degree as shown in the graph in Figure 5, the confident operation degree time series analysis processing unit 8 will calculate the time time(T j1 )(Measurement period T j1 The time immediately after the processing is completed is timetime(T j0 In ) ), the threshold Th1 of confident operation is exceeded, and the confident operation val_sure_mnp(T j Since the amount of change in ) is less than or equal to the predetermined threshold Th2, time time(T j1 ) thereafter (measurement period T j1 After the completion of the above process, it is determined that the operator Psn1's proficiency in operating the control system (operating the robot arm Rbt1.arm) has reached a certain level.

[0087] The confident operation degree time series analysis processing unit 8 then outputs the result data of the above determination process as data Dout.

[0088] Summary As described above, the confident operation degree acquisition processing system 1000 acquires the gaze-leading amount when the operator Psn1 operates the target object (in this embodiment, the robot arm Rbt1.arm) in the direction of the target object (object Obj_tg in this embodiment) while looking at the display screen of the 2D display device Dev_disp (for example, remote operation). The degree of confident operation can be acquired by the confident operation degree derived based on the acquired gaze-leading amount. The gaze-leading amount, which is data indicating how far ahead the operator's viewpoint (point of gaze on the display screen (plane PL_2D)) is in the direction of the target object when operating the target object in the direction of the target object, is objective data that eliminates subjective elements. Therefore, by evaluating the operator's confident operation degree based on the gaze-leading amount, the degree of the operator's confident operation can be objectively and appropriately detected and evaluated.

[0089] In the confident operation degree acquisition processing system 1000, the gaze measurement device Dev_vl detects the gaze of the operator Psn1, detects the operator's gaze point (viewpoint) p_vl on the display screen of the 2D display device Dev_disp, and on the display screen (plane PL_2D) of the 2D display device Dev_disp, which displays moving images of the object and the target being operated captured by the imaging device Cam1, the velocity of the gaze point (viewpoint) p_vl and the velocity of the target being operated are obtained from the positional relationship between the gaze point (viewpoint) p_vl, the object, and the target being operated, and the gaze lead amount is obtained based on the difference between the two. Therefore, the confident operation degree acquisition processing system 1000 does not require high-cost equipment such as an fMRI device, and can realize a system that acquires the gaze lead amount at low cost and evaluates the operator's confident operation degree based on the confident operation degree derived from the gaze lead amount.

[0090] Furthermore, in the confident operation degree acquisition processing system 1000, the process of detecting the operator Psn1's gaze using the gaze measurement device Dev_vl, and acquiring the speed of the gaze point (viewpoint) p_vl and the speed of the target object on the display screen (plane PL_2D) from the positional relationship of the operator's gaze point (viewpoint) p_vl, the target object, and the target object on the display screen Dev_disp, and acquiring the gaze lead amount based on the difference between the two, can be processed in real time (processing that is guaranteed to be completed within a predetermined time), so the process of acquiring the confident operation degree and evaluating the operator's confident operation sense based on the confident operation degree can also be processed in real time.

[0091] Thus, the confidence-based operation acquisition processing system 1000 can, in control systems where complex or precise operations can be performed by human operators, realize at low cost the confident feeling of operation (confident operation feeling) that occurs when a human performs operations requiring high precision, and can objectively and appropriately evaluate and detect it in real time.

[0092] ≪First Variation≫ Next, a first modified example of the first embodiment will be described. Note that parts similar to those in the above embodiment are denoted by the same reference numerals, and detailed descriptions are omitted.

[0093] In the confident operability acquisition system 1000 of the first embodiment, in step S6, the gaze-ahead quantity is acquired based on the difference between the direction dir1 (direction from the target object toward the objective) component of the velocity vector of the viewpoint p_vl on the two-dimensional plane PL_2D and the direction dir1 component of the velocity vector of the target object (robot arm Rbt1.arm). However, in the confident operability acquisition system of the first modified example of the first embodiment, in step S6, the gaze-ahead quantity is acquired based on the difference between the velocity of the viewpoint p_vl on the two-dimensional plane PL_2D and the velocity of the target object (robot arm Rbt1.arm). This is the only difference between the confident operability acquisition system of this modified example and the confident operability acquisition system 1000 of the first embodiment.

[0094] The following describes the process of step S6, which is performed in the confident operation degree acquisition processing system of this modified example.

[0095] (Step S6): In step S6, the process of acquiring the gaze-ahead amount is executed. Specifically, the following processes are performed.

[0096] The gaze-ahead quantity acquisition processing unit 6 receives data D21 output from the object position detection processing unit 3, data D22 output from the operation target position detection processing unit 4, and data D23 output from the gaze detection processing unit 5. Based on data D21, data D22, and data D23, the gaze-ahead quantity acquisition processing unit 6 executes a process to acquire the gaze-ahead quantity (gaze-ahead quantity acquisition process). Specifically, the gaze-ahead quantity acquisition processing unit 6 executes the following processes. (1) The line-of-sight advance quantity acquisition processing unit 6 acquires the 2D coordinate position pos(Rbt1.arm)=(x1,y1) of the target Rbt1.arm from data D22 in a continuous time sequence, and by differentiating the 2D coordinate position of the target Rbt1.arm with respect to time, it obtains the velocity v(Rbt1.arm)=(v_x1,v_y1)(v_x1: x component of the velocity v (velocity vector v) of the target Rbt1.arm in the 2D coordinate system). (2D) (components), v_y1: y component of the velocity v of the target Rbt1.arm (y (2D) Obtain the components. (2) The line of sight advance quantity acquisition processing unit 6 acquires the viewpoint position pos(p_vl)=(x2,y2) on the plane PL_2D from the data D23 in a continuous time sequence, and by differentiating the viewpoint position with respect to time, it obtains the velocity of the viewpoint in the 2D coordinate system v(p_vl)=(v_x2,v_y2)(v_x2: x component of the velocity v (velocity vector v) of viewpoint p_vl (x (2D) Components), v_y2: y component of velocity v at viewpoint p_vl (y (2D) Obtain the components. (3) The gaze-ahead quantity acquisition processing unit 6 is, v_diff=Abs(v(p_vl))-Abs(v(Rbt1.arm)) Abs(): A function to get the absolute value (magnitude) of a vector. The process equivalent to the above is performed to obtain the velocity difference v_diff, and the obtained velocity difference v_diff is then taken as the gaze-ahead quantity.

[0097] Then, the gaze-leading quantity acquisition processing unit 6 outputs the data including the gaze-leading quantity acquired as described above as data D3 to the confident operation degree acquisition processing unit 7.

[0098] The subsequent processing is the same as that of the confident operation degree acquisition processing system 1000 of the first embodiment.

[0099] Summary As described above, in the confident operation degree acquisition processing system of this modified example, the gaze-ahead quantity is obtained by subtracting the velocity (absolute value of the velocity vector) of the target being operated (robot arm Rbt1.arm) from the velocity (absolute value of the velocity vector) of the viewpoint p_vl on the 2D plane PL_2D. Therefore, the amount of computation can be further reduced compared to the confident operation degree acquisition processing system 1000 of the first embodiment. Note that the value obtained by subtracting the velocity (absolute value of the velocity vector) of the target being operated (robot arm Rbt1.arm) from the velocity (absolute value of the velocity vector) of the viewpoint p_vl on the 2D plane PL_2D tends to become a positive value (a large positive value) as the operator Psn1 becomes able to perform confident operations. For this reason, in the confident operation degree acquisition processing system 1000, by performing processing using the gaze-ahead quantity obtained by the above process, the degree of the operator's confident operation can be objectively and appropriately detected and evaluated.

[0100] ≪Second variation≫ Next, a second modified example of the first embodiment will be described. Note that parts similar to those in the above embodiment are denoted by the same reference numerals, and detailed descriptions are omitted.

[0101] In the confident operability acquisition processing system 1000 of the first embodiment, in step S6, the line of sight lead amount is acquired based on the difference between the direction dir1 (direction from the target object toward the objective) component of the velocity vector of the viewpoint p_vl on the two-dimensional plane PL_2D and the direction dir1 component of the velocity vector of the target object (robot arm Rbt1.arm). However, in the confident operability acquisition processing system of the second modified example of the first embodiment, in step S6, the line of sight lead amount is acquired based on the difference vector between the position vector of the viewpoint p_vl on the two-dimensional plane PL_2D and the position vector of the target object (robot arm Rbt1.arm). This is the only difference between the confident operability acquisition processing system of this modified example and the confident operability acquisition processing system 1000 of the first embodiment.

[0102] The following describes the process of step S6, which is performed in the confident operation degree acquisition processing system of this modified example.

[0103] (Step S6): In step S6, the process of acquiring the gaze-ahead amount is executed. Specifically, the following processes are performed.

[0104] The gaze-ahead quantity acquisition processing unit 6 receives data D21 output from the object position detection processing unit 3, data D22 output from the operation target position detection processing unit 4, and data D23 output from the gaze detection processing unit 5. Based on data D21, data D22, and data D23, the gaze-ahead quantity acquisition processing unit 6 executes a process to acquire the gaze-ahead quantity (gaze-ahead quantity acquisition process). Specifically, the gaze-ahead quantity acquisition processing unit 6 executes the following processes. (1) The gaze-ahead quantity acquisition processing unit 6 acquires the unit vector vec_e_dir1 of the vector (direction vector) (the vector in the direction shown by dir1 in Figure 4) from the 2D coordinate pos(Rbt1.arm)=(x1,y1) (this data is acquired from data D22) of the initial position (home position) (or the current position of the target Rbt1.arm) to the 2D coordinate position pos(Obj_tg)=(x0,y0) (this data is acquired from data D21) of the object Obj_tg. (2) The gaze-ahead quantity acquisition processing unit 6 acquires the 2D coordinate position pos(Rbt1.arm)=(x1,y1)(position vector) of the target Rbt1.arm from data D22 in a continuous time sequence. (3) The line of sight advance quantity acquisition processing unit 6 acquires the viewpoint position pos(p_vl)=(x2,y2)(position vector) on the plane PL_2D from the data D23 in a continuous time sequence. (4) The gaze-ahead quantity acquisition processing unit 6 obtains the vector diff_pos from the 2D coordinate position pos(Rbt1.arm)=(x1,y1) of the target Rbt1.arm to the viewpoint position pos(p_vl)=(x2,y2) (the difference vector between the two position vectors), diff_pos=pos(p_vl)-pos(Rbt1.arm) The dir1 direction component of the obtained position difference vector diff_pos is obtained by the following process. That is, the line-of-sight preceding quantity acquisition processing unit 6 k p =diff_pos·vec_e_dir1 •: Operator for taking the dot product k p : dir1 direction component of the position difference vector diff_pos By performing a corresponding process, the dir1-direction component of the position difference vector diff_pos is obtained. (5) The gaze-ahead quantity acquisition processing unit 6 acquires the dir1 direction component of the position difference vector diff_pos acquired in (4) as the gaze-ahead quantity.

[0105] Furthermore, (A) if the dot product of the position difference vector diff_pos and the unit vector vec_e_dir1 is positive, the sign of the gaze-ahead quantity obtained above is set to positive (the gaze-ahead quantity is set to a positive value), and (B) if the dot product of the position difference vector diff_pos and the unit vector vec_e_dir1 is negative, the sign of the gaze-ahead quantity obtained above is set to negative (the gaze-ahead quantity is set to a negative value).

[0106] Then, the gaze-leading quantity acquisition processing unit 6 outputs the data including the gaze-leading quantity acquired as described above as data D3 to the confident operation degree acquisition processing unit 7.

[0107] The subsequent processing is the same as that of the confident operation degree acquisition processing system 1000 of the first embodiment.

[0108] Summary As described above, in the confident operation degree acquisition processing system of this modified example, the value of the dot product of the vector from the position of the target (robot arm Rbt1.arm) on the 2D plane PL_2D to the position of the viewpoint p_vl (the position vector connecting the two) and the unit vector in the direction from the target (robot arm Rbt1.arm) toward the object Obj_tg is used as the gaze-ahead quantity. Therefore, the amount of computation can be further reduced compared to the confident operation degree acquisition processing system 1000 of the first embodiment. When operator Psn1 becomes able to perform confident operations, the vector from the position of the target (robot arm Rbt1.arm) on the 2D plane PL_2D to the position of the viewpoint p_vl becomes larger, and the value of the dot product of this vector and the unit vector vec_e_dir1 tends to become a large positive value. For this reason, in the confident operation degree acquisition processing system 1000, by performing processing using the gaze-ahead quantity acquired by the above processing, the degree of the operator's confident operation can be objectively and appropriately detected and evaluated.

[0109] [Other embodiments] In the above embodiment (including modified examples), the operation described was performed in which operator Psn1 operates the robot arm Rbt1.arm to grasp the target object Obj_tg and return it to its original position (initial position (home position)) in the confident operation degree acquisition processing system. However, the operation content that operator Psn1 is instructed to perform in the confident operation degree acquisition processing system is not limited to the above. For example, there may be multiple target objects, and the operation may involve following a predetermined trajectory toward the target objects. The operation content that operator Psn1 is instructed to perform in the confident operation degree acquisition processing system may, for example, be as shown in Figure 6, by operating the robot arm Rbt1.arm toward the target object Obj_tg(1) Operate to move it so that the robot arm Rbt1.arm reaches the position of the target object Obj_tg (1) Once it reaches the position of the target object Obj_tg, next, operate to move the robot arm Rbt1.arm to the target object Obj_tg (2) Once it reaches the position of the target object Obj_tg, it may be further operated to move the robot arm Rbt1.arm to the target object Obj_tg (2) Once it reaches the position of the target object Obj_tg, it may be further operated to move the robot arm Rbt1.arm to the target object Obj_tg (3) even if it is to be operated to move it to the target object Obj_tg

[0110] In such a case, in the certainty operation degree acquisition processing system, (1) When moving the operation target Rbt1.arm from the initial position to the target object Obj_tg (1) the direction indicated by dir1 in the above embodiment is set as the direction indicated by dir1 in FIG. 6 (1) (the direction from the initial position to Obj_tg (1) ), and (2) When moving the operation target Rbt1.arm from the position of the target object Obj_tg (1) to the target object Obj_tg (2) the direction indicated by dir1 in the above embodiment is set as the direction indicated by dir1 in FIG. 6 (2) (the direction from Obj_tg (1) to Obj_tg (2) ), and (3) When moving the operation target Rbt1.arm from the position of the target object Obj_tg (2) to the target object Obj_tg (3) the direction indicated by dir1 in the above embodiment is set as the direction indicated by dir1 in FIG. 6 (3) (the direction from Obj_tg (2) to Obj_tg (3) ), a unit vector is set, and the processing described in the above embodiment (including the modified example) may be performed

[0111] ​In this way (by setting the operation sequence as described above), the confident operation degree acquisition system can also perform confident operation degree acquisition processing for periodic operation content.

[0112] Furthermore, in the confident operability acquisition processing system and confident operability acquisition processing device described in the above embodiments (including modified examples), each block may be individually integrated into a single chip using a semiconductor device such as an LSI, or it may be integrated into a single chip including part or all of the blocks.

[0113] Although we have used the term LSI here, depending on the degree of integration, they may also be called IC, system LSI, super LSI, or ultra LSI.

[0114] Furthermore, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. After LSI manufacturing, FPGAs (Field Programmable Gate Arrays) that can be programmed, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells inside the LSI, may also be used.

[0115] Furthermore, some or all of the processing of each functional block in each of the above embodiments (including modified versions) may be implemented by a program. And some or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. The programs for each of these processes are stored in a storage device such as a hard disk or ROM, and are read from the ROM or RAM and executed.

[0116] Furthermore, each process of the above embodiments (including modified versions) may be implemented by hardware, or by software (including cases where it is implemented together with an OS (operating system), middleware, or a predetermined library). It may also be implemented by a mixed process of software and hardware. Furthermore, some or all of the processes of the above embodiments (including modified versions) may be implemented (executed) using, for example, one or more processors and / or one or more memories accessible from one or more processors. The processor may be implemented using, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit). The processor may also be a multiprocessor including two or more independent processors (two or more cores). Furthermore, the processor may include memory.

[0117] Furthermore, for example, when each functional part of the above embodiment (including modified versions) is implemented by software, the hardware configuration shown in Figure 7 (for example, a hardware configuration in which a CPU (which may be a GPU), ROM, RAM, input unit, output unit, etc. are connected by a bus) may be used to implement each functional part by software processing.

[0118] Furthermore, when each of the functional units of the above embodiment is implemented by software, the software may be implemented using a single computer having the hardware configuration shown in Figure 7, or it may be implemented by distributed processing using multiple computers.

[0119] Furthermore, the execution order of the processing method in the above embodiments is not necessarily limited to the description of the embodiments, and the execution order can be changed without departing from the spirit of the invention. Also, in the processing method in the above embodiments, some steps may be executed in parallel with other steps without departing from the spirit of the invention.

[0120] A computer program that causes a computer to execute the method described above, and a computer-readable recording medium on which such program is recorded, are included in the scope of the present invention. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, high-capacity DVDs, next-generation DVDs, and semiconductor memory.

[0121] The above-mentioned computer program is not limited to one recorded on the above-mentioned recording medium, but may also be transmitted via telecommunications lines, wireless or wired communication lines, networks such as the Internet, etc.

[0122] It should be noted that the specific configuration of the present invention is not limited to the embodiments described above, and various changes and modifications are possible without departing from the spirit of the invention.

[0123] [Note] The present invention can also be expressed as follows.

[0124] The first invention is a confidence level acquisition processing system for acquiring a confidence level when performing a predetermined operation by manipulating an object to a target, comprising: an imaging device that captures a three-dimensional space including the target object and the object to be manipulated; and a two-dimensional display device that displays the image captured by the imaging device on a display screen. The system includes a gaze detection device that detects the gaze of an operator who is operating an object while viewing the display screen of a 2D display device, and a confident operation degree acquisition processing device.

[0125] The confident operation degree acquisition processing device comprises a setting processing unit, a two-dimensional coordinate system acquisition processing unit, an object position detection unit, an operation target position detection unit, a line of sight detection processing unit, a line of sight preceding amount acquisition processing unit, and a confident operation degree acquisition processing unit.

[0126] The setting processing unit acquires and stores information about the location where the 2D display device is placed, information about the display screen, information about the target object, and information about the object to be operated on.

[0127] The 2D coordinate system acquisition processing unit sets a 2D coordinate system that defines the plane including the display screen, based on the location information of the 2D display device and the display screen information held in the setting processing unit.

[0128] The object position detection unit detects the target object's 2D coordinate position, which is the position of the target object in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target object held in the setting processing unit.

[0129] The target position detection unit detects the target's 2D coordinate position, which is the position of the target in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target held in the setting processing unit.

[0130] The gaze detection processing unit detects the 2D coordinate position of the gaze point, which is the position in the 2D coordinate system of the gaze point of the operator on the display screen, based on the intersection point between the operator's gaze detected by the gaze detection device and the display screen of the 2D display device.

[0131] The gaze-ahead quantity acquisition processing unit acquires the gaze-ahead quantity based on the positional relationship on the display screen between the target object's 2D coordinate position, the manipulated object's 2D coordinate position, and the gaze point's 2D coordinate position.

[0132] The confident operation degree acquisition processing unit acquires a confident operation degree, which indicates the degree to which the operator is operating the target object with confidence, based on the gaze-leading quantity.

[0133] This confident operation degree acquisition system acquires, for example, the amount of gaze leading when an operator operates an object in the direction of a target object while looking at the display screen of a 2D display device. The degree of confident operation is then obtained from the confident operation degree derived based on the acquired gaze leading amount. The gaze leading amount, which is data indicating how far ahead the operator's viewpoint (point of gaze on the display screen) is in the direction of the target object when operating the object in the direction of the target object, is objective data that eliminates subjective elements. Therefore, by evaluating the operator's confident operation degree derived from the gaze leading amount, the degree of the operator's confident operation can be objectively and appropriately detected and evaluated.

[0134] This confident operation degree acquisition system uses an eye-tracking device to detect the operator's gaze, detects the operator's point of gaze (viewpoint) on the display screen of a 2D display device, and acquires the gaze-leading quantity from the positional relationship between the point of gaze, the target object, and the operating object on the display screen of the 2D display device, which displays moving images of the target object and the operating object captured by an imaging device. Therefore, this confident operation degree acquisition system does not require expensive equipment such as an fMRI device, and can realize a system that acquires the gaze-leading quantity at low cost and evaluates the operator's confident operation degree derived from the gaze-leading quantity.

[0135] The second invention is the same as the first invention, wherein the gaze-ahead quantity acquisition processing unit acquires a value as the gaze-ahead quantity by subtracting the velocity of the 2D coordinate position of the manipulated object on the display screen from the velocity of the 2D coordinate position of the point of gaze on the display screen.

[0136] This allows the confident operation degree acquisition system to acquire the speed of the point of fixation (gaze) on the display screen and the speed of the object being operated, and to acquire the gaze-leading amount based on the difference between the two. Therefore, this confident operation degree acquisition system does not require expensive equipment such as an fMRI device, and can realize a system that acquires the gaze-leading amount at low cost and evaluates the operator's confident operation degree based on the confident operation degree derived from the gaze-leading amount.

[0137] Furthermore, in this confident operation degree acquisition processing system, the process of detecting the operator's gaze using an eye-tracking device, acquiring the speed of the gaze point (viewpoint) on the display screen and the speed of the target object from the positional relationship between the operator's gaze point (viewpoint), the target object, and the target object on the 2D display screen, and acquiring the gaze lead amount based on the difference between the two, can be processed in real time (processing that is guaranteed to be completed within a predetermined time). Therefore, the process of acquiring the confident operation degree and evaluating the operator's sense of confident operation based on the confident operation degree can also be processed in real time.

[0138] Thus, this confident operation level acquisition processing system can, in control systems where complex and precise actions can be performed by human operators, realize at low cost the confident feeling of operation (confident operation feeling) that occurs when a human performs operations requiring high precision, and can objectively and appropriately evaluate and detect it in real time.

[0139] The third invention is the same as the first invention, wherein the gaze-ahead quantity acquisition processing unit, on the display screen, defines the direction from the position of the 2D coordinate position of the object being operated to the 2D coordinate position of the target object as the first direction, and acquires a value as the gaze-ahead quantity by subtracting the first direction component of the velocity vector of the 2D coordinate position of the object being operated on the display screen from the first direction component of the velocity vector of the 2D coordinate position of the point of focus on the display screen.

[0140] As a result, this confident operation degree acquisition system can acquire a line-of-sight lead quantity based on the difference between the velocity of the 2D coordinate position of the point of focus and the velocity of the 2D coordinate position of the target object in the first direction from the position of the 2D coordinate position of the target object to the 2D coordinate position of the target object on the display screen, thereby enabling the acquisition of a line-of-sight lead quantity with higher accuracy.

[0141] The fourth invention is the first invention, wherein the gaze-ahead quantity acquisition processing unit, with the direction from the position of the 2D coordinate position of the object being operated to the 2D coordinate position of the target object on the display screen as the first direction, acquires the value of the first direction component of the position difference vector obtained by subtracting the position vector of the 2D coordinate position of the object being operated on the display screen from the position vector of the 2D coordinate position of the point of gaze on the display screen as the gaze-ahead quantity.

[0142] As a result, this confident operation degree acquisition system can acquire the line-of-sight lead quantity based on the value of the first directional component of the position difference vector between the 2D coordinate position of the point of focus on the display screen and the 2D coordinate position of the target of operation.

[0143] The fifth invention is any of the first to fourth inventions, wherein the gaze-line preceding amount acquisition processing unit is Multiple gaze-ahead quantities are acquired when performing a single operation sequence.

[0144] The confident operation score acquisition processing unit acquires one of the following as the confident operation score: the mean, median, or weighted mean of multiple gaze-leading quantities.

[0145] This allows the confident operation score acquisition system to obtain the confident operation score using statistical data from multiple gaze-leading quantities (the mean, median, and weighted mean of multiple gaze-leading quantities).

[0146] The sixth invention is the fifth invention, further comprising a confident operationality time series analysis processing unit that performs confident operationality time series analysis processing.

[0147] The confident operation acquisition processing unit acquires multiple confident operation values ​​in a time series.

[0148] The confidence operability time series analysis processing unit outputs data indicating that the operator's skill in the target operation has reached a certain level of proficiency when multiple confidence operability scores obtained by the confidence operability acquisition processing unit exceed a predetermined threshold.

[0149] As a result, this confident operation level acquisition system outputs data indicating that the operator's skill in performing the target operation has reached a certain level of proficiency when multiple confident operation levels acquired in a time series exceed a predetermined threshold. For example, it can notify the operator that they have reached a certain level of proficiency in the target operation.

[0150] The seventh invention is the fifth invention, further comprising a confident operationality time series analysis processing unit that performs confident operationality time series analysis processing.

[0151] The confident operation acquisition processing unit acquires multiple confident operation values ​​in a time series.

[0152] The confidence operability time series analysis processing unit outputs data indicating that the operator's operation of the target has reached a certain level of proficiency if the confidence operability acquired by the confidence operability acquisition processing unit exceeds the first threshold and the amount of change in the time series of confidence operability is less than or equal to the second threshold.

[0153] As a result, this confident operability acquisition system outputs data indicating that the operator's skill in performing the target operation has reached a certain level of proficiency when multiple confident operability scores acquired over time exceed a first threshold and the amount of change in the confident operability scores over time is less than or equal to a second threshold. For example, it can notify the operator that their skill in performing the target operation has reached a certain level of proficiency.

[0154] The eighth invention is a confidence operability acquisition processing system for acquiring a confidence operability when performing a predetermined operation by manipulating an object relative to a target, and is a confidence operability acquisition processing method executed using a confidence operability acquisition processing system comprising: an imaging device that photographs a three-dimensional space including a target object and an object; a two-dimensional display device that displays the image captured by the imaging device on a display screen; and a gaze detection device that detects the gaze of an operator who is operating the object while viewing the display screen of the two-dimensional display device. The confidence operability acquisition processing method comprises a setting processing step, a two-dimensional coordinate system acquisition processing step, an object position detection step, an object position detection step, a gaze detection processing step, a gaze preceding amount acquisition processing step, and a confidence operability acquisition processing step.

[0155] The setup process step acquires and stores information about the location where the 2D display device is placed, information about the display screen, information about the target object, and information about the object to be operated on.

[0156] The 2D coordinate system acquisition process step sets up a 2D coordinate system that defines the plane including the display screen, based on the location information of the 2D display device and the display screen information acquired in the setting process step.

[0157] The object position detection step detects the target object's 2D coordinate position, which is the position of the target object in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target object acquired in the setting process step.

[0158] The target position detection step detects the target's 2D coordinate position, which is the position of the target in a 2D coordinate system, within the frame image of the video captured by the imaging device, based on the information about the target acquired in the setting process step.

[0159] The gaze detection processing step detects the 2D coordinate position of the gaze point, which is the position in the 2D coordinate system of the gaze point of the operator on the display screen, based on the intersection point between the operator's gaze detected by the gaze detection device and the display screen of the 2D display device.

[0160] The gaze-ahead quantity acquisition process step acquires the gaze-ahead quantity based on the positional relationship on the display screen between the target object's 2D coordinate position, the operation target's 2D coordinate position, and the gaze point's 2D coordinate position.

[0161] The confident operation degree acquisition process step acquires a confident operation degree, which indicates the degree to which the operator is operating the target object with confidence, based on the gaze-leading amount.

[0162] This makes it possible to realize a confident operability acquisition method that produces the same effects as the first invention.

[0163] The ninth invention is a program for causing a computer to execute the confident operation degree acquisition method, which is the eighth invention.

[0164] This makes it possible to realize a program that causes a computer to execute a confident operation degree acquisition process that produces the same effects as the first invention. [Explanation of symbols]

[0165] 1000 Confident Operational Degree Acquisition Processing System 100 Confident Operational Degree Acquisition Processing Device 2. Two-dimensional coordinate system acquisition processing unit 3. Object position detection processing unit 4. Operation target position detection processing unit 5. Eye-tracking processing unit 6. Line-of-Eyes Priority Acquisition Processing Unit 7. Confident Operation Degree Acquisition Processing Unit 8. Confident Operational Degree Time Series Analysis Processing Unit Cam1 Imaging device Dev_disp 2D display device Dev_vl Eye-tracking device

Claims

1. A confidence-based operation acquisition system for obtaining the confidence level when performing a predetermined operation by manipulating an object with respect to a target, An imaging device that captures a three-dimensional space including the target object and the object being manipulated, A two-dimensional display device that displays images captured by the aforementioned imaging device on a display screen, A gaze detection device that detects the gaze of an operator who operates the object while viewing the display screen of the two-dimensional display device, A confident operation level acquisition processing device, Equipped with, The aforementioned confident operation level acquisition processing device is: A setting processing unit acquires and stores information about the location where the two-dimensional display device is positioned, information about the display screen, information about the target object, and information about the operation target. A two-dimensional coordinate system acquisition processing unit sets a two-dimensional coordinate system that defines a plane including the display screen, based on the information of the position where the two-dimensional display device is located and the information of the display screen, which are held in the setting processing unit. Based on the information about the target object held in the setting processing unit, the object position detection unit detects the target object's 2D coordinate position, which is the position of the target object in the 2D coordinate system, within the frame image of the video captured by the imaging device. Based on the information about the target to be operated that is held in the setting processing unit, the target to be operated position detection unit detects the target to be operated 2D coordinate position, which is the position of the target to be operated in the 2D coordinate system, within the frame image of the video captured by the imaging device. A gaze detection processing unit detects the position of the gaze point in the two-dimensional coordinate system of the operator's gaze point on the display screen, based on the intersection of the gaze line of the operator detected by the gaze detection device and the display screen of the two-dimensional display device, and the gaze point two-dimensional coordinate position, which is the position of the gaze point of the operator on the display screen in the two-dimensional coordinate system. A gaze-ahead quantity acquisition processing unit acquires a gaze-ahead quantity based on the positional relationship on the display screen between the two-dimensional coordinate position of the target object, the two-dimensional coordinate position of the object being operated on, and the two-dimensional coordinate position of the point of focus. A confidence operation degree acquisition processing unit that acquires a confidence operation degree indicating the degree to which the operator is operating the target with confidence, based on the aforementioned gaze-leading amount, Equipped with, A system for obtaining the degree of confidence in operation.

2. The aforementioned gaze-ahead quantity acquisition processing unit, The value obtained by subtracting the velocity of the 2D coordinate position of the object being operated on the display screen from the velocity of the 2D coordinate position of the point of focus on the display screen is obtained as the line of sight leading quantity. The confident operation level acquisition processing system according to claim 1.

3. The aforementioned gaze-ahead quantity acquisition processing unit, On the aforementioned display screen, if the direction from the position of the two-dimensional coordinate position of the object being operated on to the two-dimensional coordinate position of the target object is defined as the first direction, The value obtained by subtracting the first directional component of the velocity vector of the two-dimensional coordinate position of the object being operated on the display screen from the first directional component of the velocity vector of the two-dimensional coordinate position of the point of focus on the display screen is acquired as the line of sight lead quantity. The confident operation level acquisition processing system according to claim 1.

4. The aforementioned gaze-ahead quantity acquisition processing unit, On the aforementioned display screen, if the direction from the position of the two-dimensional coordinate position of the object being operated on to the two-dimensional coordinate position of the target object is defined as the first direction, The value of the first directional component of the position difference vector obtained by subtracting the position vector of the two-dimensional coordinate position of the object being operated on the display screen from the position vector of the two-dimensional coordinate position of the point of focus on the display screen is obtained as the line of sight leading quantity. The confident operation level acquisition processing system according to claim 1.

5. The aforementioned gaze-ahead quantity acquisition processing unit, When performing a single operation sequence, multiple preceding gaze quantities are acquired, The aforementioned confident operation level acquisition processing unit is: The mean, median, and weighted mean of the aforementioned multiple gaze-leading quantities are obtained as the degree of confidence. A confidence operation level acquisition processing system according to any one of claims 1 to 4.

6. The system further includes a confident operation time series analysis processing unit that performs confident operation time series analysis, The aforementioned confident operation level acquisition processing unit is: Obtain multiple degrees of confidence over time, The aforementioned confident operation degree time series analysis processing unit is: When the multiple confident operation levels acquired by the confident operation level acquisition processing unit exceed a predetermined threshold, the unit outputs data indicating that the operator's operation of the target has reached a certain level of proficiency. The confident operation level acquisition processing system according to claim 5.

7. The system further includes a confident operation time series analysis processing unit that performs confident operation time series analysis, The aforementioned confident operation level acquisition processing unit is: Obtain multiple degrees of confidence over time, The aforementioned confident operation degree time series analysis processing unit is: If the confidence level of operation acquired by the confidence level acquisition processing unit exceeds a first threshold and the amount of change in the confidence level of operation over time is less than or equal to a second threshold, the system outputs data indicating that the operator's operation of the target has reached a certain level of proficiency. The confident operation level acquisition processing system according to claim 5.

8. A confidence-based operation acquisition system for obtaining the confidence level when performing a predetermined operation by manipulating an object with respect to a target, An imaging device that captures a three-dimensional space including the target object and the object being manipulated, A two-dimensional display device that displays images captured by the aforementioned imaging device on a display screen, A gaze detection device that detects the gaze of an operator who operates the object while viewing the display screen of the two-dimensional display device, A method for obtaining confidence in operation that is performed using the confidence in operation acquisition processing system comprising the above, A setting process step that acquires and stores information about the location where the two-dimensional display device is positioned, information about the display screen, information about the target object, and information about the operation target. A two-dimensional coordinate system acquisition step, based on the information of the position where the two-dimensional display device is located and the information of the display screen obtained in the setting process step, sets a two-dimensional coordinate system that defines a plane including the display screen. A target position detection step in which, based on the information about the target object obtained in the setting processing step, the target object's 2D coordinate position, which is the position of the target object in the 2D coordinate system, is detected within the frame image of the video captured by the imaging device, An operation target position detection step, based on the information about the operation target obtained in the setting processing step, detects the operation target's 2D coordinate position, which is the position of the operation target in the 2D coordinate system, within the frame image of the video captured by the imaging device. A gaze detection processing step that detects the gaze of the operator detected by the gaze detection device and the display screen of the two-dimensional display device, based on the intersection of the gaze of the operator detected by the gaze detection device and the display screen of the two-dimensional display device, the gaze point two-dimensional coordinate position is the position of the gaze point of the operator on the display screen in the two-dimensional coordinate system, A gaze-ahead amount acquisition process step that acquires a gaze-ahead amount based on the positional relationship on the display screen between the two-dimensional coordinate position of the target object, the two-dimensional coordinate position of the object being operated on, and the two-dimensional coordinate position of the point of focus, A confidence operation degree acquisition process step that acquires a confidence operation degree indicating the degree to which the operator is confidently operating the target object, based on the aforementioned gaze-leading amount, A method for obtaining confidence in operation, comprising:

9. A program for causing a computer to execute the confident operation degree acquisition process method described in claim 8.

Images