Work estimating device, work estimating method, and computer-readable recording medium

Abstract

The job estimation device (200) includes a gaze area estimation unit (220), an object detection unit (210), a contact degree calculation unit (230), and a job estimation unit (250). The gaze area estimation unit (220) estimates the area the user is looking at, i.e., the gaze area, using information representing the user's line of sight. The object detection unit (210) detects the work object and at least one object candidate based on an image showing the work object being used by the user and at least one object candidate that is a candidate for the user's work. The contact degree calculation unit (230) calculates the contact degree, representing the degree of contact between the work object and each object candidate included in the at least one object candidate, based on the gaze area. The job estimation unit (250) estimates the user's job based on the work object and the contact degree.

Description

Technical Field

[0001] This invention relates to a job estimation apparatus, a job estimation method, and a computer-readable recording medium containing a job estimation program. Background Technology

[0002] In OJT (On-the-Job Training) education related to equipment maintenance, alternative educational systems are sought to reduce the burden on supervisors and educators, as well as the costs associated with the training. In such alternatives, these systems ideally provide users learning maintenance tasks with useful information related to the tasks they perform. To achieve this, the educational system needs to estimate the tasks being performed by the users.

[0003] Patent document 1 discloses the following technology: In the user's first-person viewpoint image, the detection results of objects and attention map are used to detect objects of interest that the user is paying attention to, and action recognition is performed based on the combination of information about objects of interest and non-objects of interest, thereby improving the accuracy of user action recognition.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-011526 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] The action recognition method disclosed in Patent Document 1 has the following problem: when multiple objects are reflected in the field of view of the image from the user's perspective, it may be mistakenly identified as the user performing an action even if the user has not performed any action. As a specific example, according to this action recognition method, when the user's work object and the props used by the user in the work are reflected in the image from the user's perspective, it may be judged that the user is performing a work even if the user is not using the props.

[0009] The purpose of this invention is to improve the accuracy of estimating user tasks by detecting the contact between the object being performed and the candidate object that is the target of the task.

[0010] Methods for solving problems

[0011] The task estimation apparatus of the present invention includes: a gaze area estimation unit that estimates the area the user is gazing at, i.e., the gaze area, using information representing the user's gaze; an object detection unit that detects the task object and the at least one object candidate object based on an image showing the task object being used by the user and at least one object candidate object that is a candidate for the user's task; a contact degree calculation unit that calculates a contact degree based on the gaze area, wherein the contact degree represents the degree of contact between the task object and each object candidate object included in the at least one object candidate object; and a task estimation unit that estimates the user's task based on the task object and the contact degree.

[0012] Invention Effects

[0013] According to the present invention, the contact degree calculation unit calculates the contact degree, representing the degree of contact between the work object and each of the candidate object included in at least one candidate object, based on the area being gazed at by the user, and the work estimation unit estimates the user's work based on the contact degree. Therefore, according to the present invention, by detecting the contact between the work object and the candidate object that is a candidate for work, the accuracy of estimating the user's work can be improved. Attached Figure Description

[0014] Figure 1 This is an example of the software structure of the job estimation device 200 in Implementation 1.

[0015] Figure 2 This is an example of the hardware structure of the job estimation device 200 in Implementation 1.

[0016] Figure 3 This is a flowchart illustrating the operation of the job estimation device 200 according to Embodiment 1.

[0017] Figure 4 This diagram illustrates the processing of the contact degree calculation unit 230 in Embodiment 1.

[0018] Figure 5 This is a flowchart illustrating the operation of the contact degree calculation unit 230 in Embodiment 1.

[0019] Figure 6 This diagram illustrates the processing of the contact degree calculation unit 230 in Embodiment 1.

[0020] Figure 7 This diagram illustrates the processing of the contact degree calculation unit 230 in Embodiment 1.

[0021] Figure 8 This diagram illustrates the processing of the contact degree calculation unit 230 in Embodiment 1.

[0022] Figure 9This is a specific example of the learning data D1 in Implementation Method 1.

[0023] Figure 10 This is a structural example of the learning device 400 in Embodiment 1.

[0024] Figure 11 This is a flowchart illustrating the operation of the learning device 400 according to Embodiment 1.

[0025] Figure 12 This is a software structure example of the job estimation device 200, which is a variation of Implementation 1.

[0026] Figure 13 This is a flowchart illustrating the operation of the job estimation device 200 in a modified example of Embodiment 1.

[0027] Figure 14 This is a hardware structure example of the job estimation device 200, which is a variation of Implementation 1. Detailed Implementation

[0028] In the description and accompanying drawings of the embodiments, the same reference numerals are used to label the same elements and corresponding elements. Descriptions of elements labeled with the same reference numerals are appropriately omitted or simplified. Arrows in the figures primarily indicate data flow or processing flow. Furthermore, the term "section" may be appropriately rewritten as "circuit," "process," "step," "processing," or "circuit."

[0029] Implementation Method 1

[0030] The following is a detailed description of this embodiment with reference to the accompanying drawings.

[0031] ***Structure Description***

[0032] Figure 1 The diagram shows a structural example of a job estimation system 90 and a software structural example of a job estimation device 200. The job estimation system 90 includes a job estimation device 200, a camera device 300, and a line-of-sight measuring device 350. In this diagram, black dots indicate lines connected to each other. When black dots are not shown at the intersections of multiple lines, the lines are not connected.

[0033] The camera device 300 is a device for capturing images of a user's work activities; specifically, it is a camera. The user is the subject of a task estimated by the task estimation system 90. Sometimes the user is not performing a task but is checking the task content or taking a break. The user may also be a robot or something similar. The camera device 300 sends the images captured by the camera device 300 as video images to the task estimation device 200. The video images can be moving images or more than one still image. The video images can also be RGB (Red-Green-Blue) images, depth images, or both. The camera device 300 may also consist of multiple devices.

[0034] The line-of-sight measurement device 350 is a device for measuring a user's line of sight. Specifically, it is a device equipped with a camera and worn on the user's head. The line-of-sight measurement device 350 sends line-of-sight measurement information, representing the measured result of the user's line of sight, to the job estimation device 200. The line-of-sight measurement device 350 may also consist of multiple devices.

[0035] The job estimation device 200 may not be directly connected to at least one of the camera device 300 and the line-of-sight measuring device 350. The job estimation device 200 may also be connected to an external recording device such as a recorder, and may also receive information that reproduces data pre-recorded by the recording device, wherein the external recording device records data sent to the job estimation device 200 by at least one of the camera device 300 and the line-of-sight measuring device 350.

[0036] The job estimation device 200 includes an object detection unit 210, a gaze area estimation unit 220, a contact degree calculation unit 230, a job estimation unit 250, and an estimation result storage unit 260. The job estimation device 200 estimates the job performed by the user based on information from the camera device 300 and the line of sight measurement device 350.

[0037] The object detection unit 210 detects objects and includes a task object detection unit 211 and a candidate object detection unit 215. An object is a general term for both candidate objects and task objects. Candidate objects are candidates for objects that can be used as the user's task objects. Task objects are objects used by the user during the task; for example, they may be the user's hand, a tool being used by the user, or both. A task object may also consist of multiple objects, such as the user's two hands or one hand and a tool. The object detection unit 210 detects the task object and at least one candidate object based on an image that reflects the task object being used by the user and at least one candidate object that can be used as the user's task object. Image and video are sometimes used interchangeably.

[0038] The work object detection unit 211 detects work objects. The work object detection unit 211 includes a tool detection unit 212 and a hand detection unit 213.

[0039] The prop detection unit 212 detects the props that the user is using based on the camera images.

[0040] The hand detection unit 213 detects the user's hand based on the camera image.

[0041] The candidate object detection unit 215 detects candidate objects. The candidate object detection unit 215 is also referred to as the object detection unit.

[0042] The gaze region estimation unit 220 estimates the gaze region using information measured by the gaze measurement device 350, which represents information about the user's gaze. The gaze region is the area the user gazes at. The gaze region can be a two-dimensional distribution of any shape, a distribution with a maximum value at the viewpoint position and a pre-defined distribution, or a heat map calculated using time-series data recording viewpoint positions. The viewpoint position is the position represented by the gaze measurement information; it is the position of the user's viewpoint. The gaze region estimation unit 220 can also estimate the gaze region using time-series data representing the user's viewpoint position. Furthermore, the gaze region can also be an area determined based on the distance between the positions of each candidate object and the viewpoint position. Specifically, within this area, the positions of each candidate object and the viewpoint position converge within a predetermined range.

[0043] The contact degree calculation unit 230 calculates the contact degree based on the gaze area, which represents the degree of contact between the working object and each of the at least one object candidate object. The contact degree calculation unit 230 is also referred to as the object position relationship value calculation unit within the gaze area. The contact degree calculation unit 230 can also calculate the weights corresponding to each of the at least one object candidate object based on the gaze area, and use the calculated weights to determine the contact degree. The contact degree calculation unit 230 can determine the contact degree based on the distance between the working object and each of the at least one object candidate object, based on the arrangement of the working object and the arrangement of each of the at least one object candidate object, or based on the overlapping area between the working object and each of the at least one object candidate object. The contact degree calculation unit 230 can also appropriately combine weights based on information representing the gaze area, as well as the working object and the object candidate objects, to determine the contact degree.

[0044] The job estimation unit 250 estimates the user's job based on the output of the object detection unit 210 and the output of the contact degree calculation unit 230. The job estimation unit 250 estimates the user's job based on the object to be worked on and the degree of contact. The job estimation unit 250 can also estimate the user's job using a rule-based estimation method or a learning model.

[0045] The estimation result storage unit 260 stores the results obtained by the job estimation unit 250 in estimating the job. The estimation result storage unit 260 is also called the job estimation result storage unit, and may be located outside the job estimation device 200.

[0046] Figure 2 An example of the hardware structure of the job estimation device 200 is shown. As shown in this figure, the job estimation device 200 is composed of a computer 100. The computer 100 is composed of a processing unit 101, a main storage device 102, an auxiliary storage device 103, a first interface 104, and a second interface 105, and is also referred to as a computer. The job estimation device 200 may also be composed of multiple computers 100.

[0047] The arithmetic unit 101 is an integrated circuit (IC) that performs computational processing and controls the hardware of the computer. Specifically, the arithmetic unit 101 may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The job estimation device 200 may also have multiple arithmetic units instead of the arithmetic unit 101. These multiple arithmetic units share the functions of the arithmetic unit 101.

[0048] The main storage device 102 is a device for temporarily storing signals of the arithmetic unit 101. As a specific example, the main storage device 102 is RAM (Random Access Memory). Data stored in the main storage device 102 is saved to the auxiliary storage device 103 as needed.

[0049] The auxiliary storage device 103 is a device for long-term storage of signals from the arithmetic unit 101. Specifically, the auxiliary storage device 103 may be ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 103 is loaded into the main storage device 102 as needed. The main storage device 102 and the auxiliary storage device 103 may also be integrated.

[0050] The first interface 104 is a device that receives signals from the camera device 300 connected to the computer 100. As a specific example, the first interface 104 is a USB (Universal Serial Bus) terminal, or a communication device such as a communication chip or NIC (Network Interface Card).

[0051] The second interface 105 is the same as the first interface 104, and is a device for receiving signals from the line-of-sight measuring device 350. The first interface 104 and the second interface 105 can also be configured as a single unit.

[0052] Auxiliary storage device 103 stores the job estimation program. The job estimation program is a program that enables the computer to implement the functions of each part of the job estimation device 200. The job estimation program is loaded into main storage device 102 and executed by arithmetic unit 101. The functions of each part of the job estimation device 200 are implemented by software.

[0053] Data used during the execution of the job estimation program and data obtained through the execution of the job estimation program are appropriately stored in a storage device. Each part of the job estimation device 200 appropriately utilizes a storage device. As a specific example, the storage device is composed of at least one of the main storage device 102, the auxiliary storage device 103, registers within the arithmetic unit 101, and cache memory within the arithmetic unit 101. Furthermore, data and information sometimes have the same meaning. The storage device may also be independent of the computer 100. The estimation result storage unit 260 is composed of a storage device.

[0054] The functions of the main storage device 102 and the auxiliary storage device 103 can also be implemented through other storage devices.

[0055] Job estimation programs can also be recorded on computer-readable non-volatile recording media. Specific examples of non-volatile recording media include optical discs or flash memory. Job estimation programs can also be provided as program products.

[0056] ***Instructions for Action***

[0057] The operation steps of the job estimation device 200 are equivalent to a job estimation method. Furthermore, the program that implements the operation of the job estimation device 200 is equivalent to a job estimation program.

[0058] Figure 3 This is a flowchart illustrating an example of the operation of the job estimation device 200. The operation of the job estimation device 200 will be explained with reference to this diagram. In the explanation of this flowchart, it is assumed that a job object and more than one candidate object are projected into the camera image.

[0059] (Step S101: Object detection processing)

[0060] The object detection unit 210 receives camera images from the camera device 300, detects the work object and candidate objects reflected in the received camera images, and calculates information corresponding to each detected candidate object. Specifically, this information includes attribute information representing the properties of each candidate object, and information representing the occupied region corresponding to each candidate object. The occupied region is the region corresponding to the area occupied by each object in the camera image; it can be a rectangular region containing each object, or it can be a set of pixels displaying each object. The object detection unit 210 detects candidate objects using a method that assigns labels to the candidate objects, or it can use a machine learning-based method that uses a pre-learned model. Furthermore, the object detection unit 210 calculates the occupied region corresponding to the work object.

[0061] (Step S102: Fixation Region Estimation Processing)

[0062] The gaze area estimation unit 220 receives gaze measurement information from the gaze measurement device 350 and uses the viewpoint position represented by the received gaze measurement information to estimate the gaze area.

[0063] (Step S103: Contact Degree Calculation Processing)

[0064] The contact degree calculation unit 230 calculates the contact index based on the work object detected by the object detection unit 210, the candidate objects, and the gaze area estimated by the gaze area estimation unit 220. The contact index is a quantification of the degree of contact between the work object and each candidate object.

[0065] As a specific example, the contact degree calculation unit 230 calculates the contact index based on any one of the following five factors: (1) distance, (2) overlapping area, (3) either distance or direction, (4) overlapping area and direction, and (5) either distance or direction. Distance represents the contact degree calculation unit 230 determining the contact index corresponding to each candidate object based on the distance between the working object and each candidate object. Overlapping area represents the contact degree calculation unit 230 determining the contact index corresponding to each candidate object based on the size of the area overlapping the occupied area corresponding to each candidate object and the occupied area corresponding to the working object. Direction represents the contact degree calculation unit 230 determining the contact index corresponding to each candidate object based on the direction of the working object relative to the candidate object and the direction of the working object itself. The contact degree calculation unit 230 may also calculate the contact index based on all three factors: distance, overlapping area, and direction. A specific example of calculating the contact index will be explained below. The direction of the working object corresponds to the configuration of the working object. The direction of the working object relative to the candidate object is based on the configuration of the candidate object and the configuration of the working object.

[0066] (1) Distance

[0067] The following is a specific example of how the contact degree calculation unit 230 calculates the contact index based on distance. The contact degree calculation unit 230 calculates the score as follows: the larger the area where the occupied area and the gaze area corresponding to the candidate object overlap, and the greater the degree of gaze at the overlapping gaze area, the higher the score value corresponding to the candidate object. Furthermore, the closer the work object and the candidate object are, the higher the score value corresponding to the candidate object. The score represents the degree of contact.

[0068] Figure 4 A specific example of the processing by the contact level calculation unit 230 is illustrated using a field-of-view image. A field-of-view image is an image representing at least a portion of the user's field of vision; a camera image can also be a field-of-view image. The gaze area estimation unit 220 or the contact level calculation unit 230, etc., can also generate a field-of-view image based on a camera image. Hereinafter, it will be assumed that the contact level calculation unit 230 performs processing based on the field-of-view image.

[0069] In this diagram, the occupied region A(i) (i = 1, 2, 3) is the occupied region corresponding to the candidate object C(i), and it is a rectangular region surrounding the candidate object C(i). The centroid go(i) is the centroid of the candidate object C(i). The object used in the task is a hand, and the centroid u is the position of the center of gravity of the hand. d(i) is the reciprocal of the distance from the centroid u to the centroid go(i).

[0070] Figure 5An example of the processing flow of the contact degree calculation unit 230 is shown. The processing of the contact degree calculation unit 230 will be explained with reference to this figure.

[0071] (Step S301)

[0072] The contact degree calculation unit 230 uses information representing the gaze region G to calculate the weights for each candidate object C(i) (i = 1, 2, ...).

[0073] As a specific example, firstly, the contact degree calculation unit 230 calculates the overlapping region Ov(i) (=A(i)∩G) related to each candidate object C(i). The overlapping region Ov(i) represents the region where the occupying region A(i) and the gaze region G overlap, and can also be a rectangular region surrounding the region where the occupying region A(i) and the gaze region G overlap. All regions are not limited to two-dimensional regions, and can also be three-dimensional regions. When each region is a three-dimensional region, the contact degree calculation unit 230 determines whether the stereoscopic objects corresponding to each region overlap each other when calculating the overlapping region. Next, assume that each region is two-dimensional, and set the function Gf as a function representing the user gaze degree in each pixel contained in the visual field image in the gaze region G. That is, the function Gf is a function representing the gaze distribution, and the function Gf(x,y) represents the user gaze degree in the pixel corresponding to the coordinate (x,y). As a specific example, the function Gf(x,y) is a function with the highest function value at the center of the gaze region G, and the function value gradually decreases as it moves towards the ends of the gaze region G. Furthermore, if the area of ​​the overlapping region Ov(i) is 0, the following steps may not be performed for the candidate object C(i) corresponding to the overlapping region Ov(i). That is, the contact degree calculation unit 230 may also filter the candidate object C(i) based on the gaze region G in this step.

[0074] Next, the contact degree calculation unit 230 calculates the weight W(i) corresponding to the candidate object C(i) as shown in [Equation 1]. The weight W(i) is calculated by dividing the integral value of the function Gf in the overlapping region Ov(i) by the number of pixels in the overlapping region Ov(i).

[0075] [Formula 1]

[0076] w (i) =∑ (x，y)∈v(i) Gf(x,y) / |Ov(i)|

[0077] (Step S302)

[0078] The contact degree calculation unit 230 calculates the value corresponding to the distance between the working object and each candidate object.

[0079] As a specific example, the contact degree calculation unit 230, as shown in [Equation 2], calculates the reciprocal d(i) of the distance value representing the distance as the value corresponding to the distance. In this example, the contact degree calculation unit 230 calculates the distance between the center of gravity u of the working object and the center of gravity go(i) as the distance between the working object and the candidate object C(i), and calculates the reciprocal d(i) of the calculated distance value.

[0080] [Formula 2]

[0081] d(i) = ||go(i) - u|| -1

[0082] (Step S303)

[0083] The contact degree calculation unit 230 calculates a score obtained by quantifying the contact degree.

[0084] As a specific example, the contact level calculation unit 230 uses the weight W(i) and the reciprocal of the distance value d(i) to calculate the score S(i) corresponding to the candidate object C(i) (=W(i)·d(i)). The score S(i) is an indicator of the probability that the candidate object C(i) is the user's task object. The larger the value of the score S(i), the higher the probability that the candidate object C(i) is the user's task object.

[0085] (Step S304)

[0086] The contact level calculation unit 230 outputs output information including the calculated score. The contact level calculation unit 230 can also rearrange the candidate objects in descending order based on the corresponding score S, and output each candidate object along with its corresponding score. The contact level calculation unit 230 can also output only scores above a predetermined baseline value.

[0087] Below, we assume that the output information from the contact degree calculation unit 230 includes attribute information of the candidate object and a score corresponding to the candidate object. The output information may also include information about the occupied area of ​​the object. As a specific example, the occupied area information is information representing the set of position coordinates that constitute the occupied area.

[0088] A specific example of the processing flow of the contact degree calculation unit 230 when the contact index is calculated based on "(1) distance" is as follows: Figure 5 As shown below. The main difference between the processing flow and that in “(1) Distance” is explained below.

[0089] (2) Overlapping regions

[0090] The following is a specific example of how the contact degree calculation unit 230 calculates the contact index based on the overlapping area. The contact degree calculation unit 230 calculates the score as follows: the larger the area of ​​overlap between the occupied area corresponding to the candidate object and the occupied area corresponding to the work object, the larger the score value corresponding to the candidate object.

[0091] Figure 6 This figure shows a specific example of the processing by the contact degree calculation unit 230. Figure 4 Same. The occupied area U is the occupied area corresponding to the hand, which is the object used for the task.

[0092] (Step S301)

[0093] The contact degree calculation unit 230 calculates the occupied area A(i) corresponding to each candidate object C(i).

[0094] (Step S302)

[0095] The contact degree calculation unit 230 calculates the size of the area where the occupied area A(i) and the occupied area U overlap.

[0096] As a specific example, the contact degree calculation unit 230 calculates the ratio A1(i) of the area of ​​the overlapping region of occupied region A(i) and occupied region U to the area of ​​occupied region A(i) (=|A(i)∩U| / |A(i)|).

[0097] (Step S303)

[0098] The contact level calculation unit 230 calculates the score.

[0099] As a specific example, the contact degree calculation unit 230 calculates the score S(i) (=W(i)·A1(i)) based on the ratio A1(i).

[0100] Alternatively, the contact degree calculation unit 230 may calculate the proportion A1(i) based on the area where the occupied area A(i) and the gaze area G and the occupied area U overlap (A(i)∩U), instead of the area (A(i)∩G)∩U) where the occupied area A(i) and the gaze area G overlap (A(i)∩G)∩U).

[0101] (3) Distance and direction, one of them

[0102] The following is a specific example of how the contact degree calculation unit 230 calculates the contact index based on distance and direction. The contact degree calculation unit 230 calculates the score as follows: it is a score that has the characteristics in "(1) distance", and the closer the direction of the working object relative to the candidate object is to the direction of the working object, the larger the score value corresponding to the candidate object.

[0103] Figure 7 This figure shows a specific example of the processing by the contact degree calculation unit 230. Figure 4 Same. Vector p i This represents the relative position of candidate object C(i) with respect to the position of the object being processed. As a specific example, vector p... i The direction is the direction originating from the center of gravity u and ending at the center of gravity go(i). Vector h is a unit vector representing the direction of the object used for the task. As a specific example, the contact degree calculation unit 230 can set the direction of vector h to the direction of the first intrinsic vector obtained by performing principal component analysis on the area representing the user's hand detected by the hand detection unit 213, or it can set the direction of vector h to the direction obtained using information representing the joint position of the user's fingers detected from the camera image or field of view image. When the object used for the task is a prop, the contact degree calculation unit 230 can also calculate the direction of vector h using a predefined direction, that is, the direction of the prop when the user uses the prop.

[0104] (Step S301)

[0105] The contact degree calculation unit 230 performs the same process as step S301 in “(1) Distance”.

[0106] (Step S302)

[0107] The contact degree calculation unit 230 calculates the values ​​corresponding to distance and direction, respectively.

[0108] As a specific example, firstly, the contact degree calculation unit 230 calculates the value corresponding to the distance in the same way as step S302 in "(1) Distance".

[0109] Next, the contact degree calculation unit 230 quantifies the contact degree between the candidate object and the work object by utilizing the orientation of the work object. Specifically, the contact degree calculation unit 230 calculates the dot product Δ, representing the difference between vector p and vector h, as shown in [Equation 3]. In this example, the closer the orientation of the work object is to the direction pointing towards the center of gravity of the candidate object C(i), the larger the value of the dot product Δ(i), which represents the contact degree. Furthermore, the dot product Δ(i) represents the degree to which the work object is oriented towards the candidate object C(i).

[0110] [Formula 3]

[0111] Δ(i)=p i .h / ||p i ||

[0112] (Step S303)

[0113] The contact level calculation unit 230 calculates the score.

[0114] As a specific example, the contact level calculation unit 230 calculates the score S(i) (=W(i)·f(d(i),Δ(i))) based on the weight W(i), the reciprocal of the distance value d(i), and the inner product Δ(i). Here, the function f is a function that associates the reciprocal of the distance value d(i) and the inner product Δ(i) as input variables. The function f can be a function that linearly associates the input variables, or it can be a function that non-linearly associates the input variables.

[0115] (4) Overlapping area and direction

[0116] The following is a specific example of how the contact degree calculation unit 230 calculates the contact index based on the overlapping area and direction. The contact degree calculation unit 230 calculates a score that has the characteristics of the score in "(2) Overlapping Area" and the characteristics of the score calculated based on the direction as shown in "(3) Distance and Direction".

[0117] Figure 8 This figure shows a specific example of the processing by the contact degree calculation unit 230. Figure 4 , Figure 6 and Figure 7 same.

[0118] (Step S301)

[0119] The contact degree calculation unit 230 performs the same process as step S301 in “(1) Distance”.

[0120] (Step S302)

[0121] The contact degree calculation unit 230 calculates the size of the overlapping area in the same way as step S302 in "(2) Overlapping Area". In addition, the contact degree calculation unit 230 quantifies the contact degree between the candidate object and the working object by using the orientation of the working object in the same way as step S302 in "(3) Distance and Direction".

[0122] In the following step, it is assumed that the contact degree calculation unit 230 calculates the ratio A1(i) and the inner product Δ(i).

[0123] (Step S303)

[0124] The contact level calculation unit 230 calculates the score.

[0125] As a specific example, the contact degree calculation unit 230 calculates the score S(i) (=W(i)·f(A1(i),Δ(i))) based on the ratio A1(i) and the inner product Δ(i). Here, the function f is the same as the function f mentioned above.

[0126] (5) Distance and direction, secondly

[0127] The contact degree calculation unit 230 can also calculate a multi-dimensional vector as a score. As a specific example, the score S(i) for object C(i) is a two-dimensional vector as shown below.

[0128] S(i)=[W(C(i)),f(d(i),Δ(i))]

[0129] Wherein, W(C(i)) represents the weight based on the gaze region, and f(d(i),Δ(i)) represents the calculated value representing the positional relationship between the working object and the object C(i). At this time, the contact degree calculation unit 230 can calculate W(C(i)) by the calculation method shown in (1) to (4) above, or by the calculation method shown below.

[0130] W(C(i))=|C(i)[x,y]-g(x,y)|

[0131] Where C(i)[x,y] represents the position of object C(i), and g(x,y) represents the viewpoint position. That is, the weights obtained by this formula are based on the distance between the position of each object and the viewpoint position.

[0132] (Step S104: Job estimation processing)

[0133] The task estimation unit 250 estimates the task being performed by the user using information representing the props output by the prop detection unit 212, information representing the user's hand output by the hand detection unit 213, and a score output by the contact degree calculation unit 230. Specifically, the task estimation unit 250 estimates the task being performed by the user using either a rule-based estimation method or a machine learning-based estimation method. Each estimation method will be explained in detail below. Furthermore, assuming the task object is the user's hand, objects A, B, and C are input as candidate objects, with object A having the highest score.

[0134] (i) Rule-based estimation methods

[0135] Consider the case where job tags are predefined to represent the combinations of each job object and each candidate object. In this case, the job estimation unit 250 can also retrieve the job tag corresponding to the combination of the user's "hand" and "object A" to estimate the user's job.

[0136] Furthermore, consider the case where no job label is defined, but a job label is pre-assigned for all combinations of input candidate objects and scores. In this case, the job estimation unit 250 can also estimate the job using all input candidate objects and the scores corresponding to each candidate object.

[0137] Furthermore, when both the hand and the prop appear in the camera image, the job estimation unit 250 can determine that there is a high probability that the prop is a job object and the hand is not a job object, and retrieve the job tag corresponding to the combination of the prop and the candidate object that the prop is in contact with, thereby estimating the user's job.

[0138] (ii) Machine learning-based estimation methods

[0139] The job estimation unit 250 inputs a combination of information about candidate objects appearing in the camera image and the scores corresponding to those candidate objects into a recognition machine that has undergone training through statistical machine learning, thereby estimating the user's job. The job estimation unit 250 may also utilize the action recognition method described in Patent Document 1.

[0140] The job estimation unit 250 calculates data that associates information about candidate objects with information about objects used in the job as feature quantities, and uses a learned model to infer the label corresponding to the job or action associated with the calculated feature quantities. Specifically, the job estimation unit 250 generates learning data D1 associated with graphical structures, etc., and inputs the generated learning data D1 into a recognition machine capable of processing graphical structures, such as a graph neural network or graph embedding, i.e., a learned recognition machine in statistical machine learning, thereby estimating the user's job.

[0141] Figure 9 A specific example of learning data D1 generated using graphics is shown. In this example, the nodes of the graphics are labels representing objects, the edge values ​​between the nodes representing the objects to be worked on and the nodes representing candidate objects are the scores calculated by the contact degree calculation unit 230, and the values ​​of the nodes between candidate objects are arbitrary fixed values ​​c. In addition to containing labels representing objects, nodes may also contain information such as the position of the object in the visual image and the size of the area occupied by the object. Furthermore, when representing a user's hand using nodes, the nodes may also contain information about the hand's position and orientation. If information representing finger joints is available, the nodes may also contain information about the joints. When representing a prop used by the user using nodes, the nodes may also contain information about the prop's position, orientation, and occupied area, based on the prop type.

[0142] Furthermore, when the job estimation device 200 acquires temporal data from the camera device 300 and the line-of-sight measuring device 350, the job estimation unit 250 can also generate data using the object detection results or object positional relationships corresponding to the data at each moment in each temporal data, employing a machine learning method that considers the temporal order of the generated data. As a specific example, the job estimation unit 250 can also use a temporal convolutional network.

[0143] Generally, when using machine learning-based estimation methods, it is necessary to collect and process learning data and generate learning models. Learning data is the data used to learn and generate the learning model. Figure 10 An example structure of a learning device 400 for performing these processes is shown. The learning device 400 includes a learning data acquisition unit 410, a learning model generation unit 420, and a learning model storage unit 430. The learning model storage unit 430 may also be located outside the learning device 400. The learning device 400 and the job estimation device 200 may also be integrated. The hardware structure of the learning device 400 may be the same as the hardware structure of the computer 100.

[0144] The learning data acquisition unit 410 acquires learning data D1 as learning data. Learning data D1 is set to be data that can be input into the job estimation unit 250. This data includes data that can also be used by the job estimation unit 250 when estimating the user's job.

[0145] The learning model generation unit 420 constructs a learning model capable of processing the data acquired by the learning data acquisition unit 410, performs learning based on the constructed learning model, and thereby generates a fully learned model. The learning model is also known as a machine learning model.

[0146] The learning model storage unit 430 stores the learned model generated by the learning model generation unit 420.

[0147] Figure 11 This diagram illustrates an example of the learning process. The processing of the learning device 400 will be explained with reference to this diagram.

[0148] (Step S501)

[0149] The learning data acquisition unit 410 acquires data that can be input into the job estimation unit 250 as learning data, and represents the acquired learning data as data representing the following information: information representing objects that can become job objects, information representing job objects, and information in the form of associating the information representing objects that can become job objects and the information representing job objects. The learning data includes at least one of information representing the gaze area and information representing the score corresponding to the object. When associating multiple pieces of information, the learning data acquisition unit 410, as a specific example, can use data that sets values ​​representing the positional relationships corresponding to each object as elements, or it can use... Figure 9 The graphical structure shown is illustrated. The learning data acquisition unit 410 assigns labels representing operational actions to the generated data.

[0150] (Step S502)

[0151] The learning model generation unit 420 processes the learning data acquired by the learning data acquisition unit 410 to generate a learning model.

[0152] As a specific example, consider the case where the learning data represents a graph structure. In this case, the learning data acquisition unit 410 can use a machine learning model such as a graph neural network capable of processing graph structures as the learning model, or it can use a model that vectorizes the learning data using a graph embedding method, and then processes the vectorized learning data. The learning model generation unit 420, when also considering the correlation between data at different time points in the time series data for learning, can, as a specific example, flexibly utilize models such as temporal convolutional networks.

[0153] (Step S503)

[0154] The learning model storage unit 430 stores the learning model that has been learned and generated by the learning model generation unit 420.

[0155] (Step S105: Storage and processing of estimation results)

[0156] The estimation result storage unit 260 stores the output of the job estimation unit 250.

[0157] ***Explanation of the effects of Implementation Method 1***

[0158] As described above, according to this embodiment, based on the combination of the work object and each candidate object existing in the image at the user's viewpoint, candidates for the work object are filtered using the user's gaze region. A score is calculated corresponding to the object that the user is touching with their hand or a prop among the filtered candidates, and the user's work is estimated based on the calculated score. Specifically, the contact degree calculation unit 230 performs a two-stage process in steps S301 to S304: filtering candidate objects based on the gaze region G and then detecting contact with the filtered candidate objects. Therefore, according to this embodiment, not only can the user's work be estimated with relatively high accuracy, but occlusion caused by the work object can also be prevented. Therefore, according to this embodiment, the robustness of the estimation of the user's work is improved.

[0159] Furthermore, when the equipment to be maintained and inspected is also the object of the work, the multiple candidate inspection parts are often close to each other. Therefore, it is difficult to estimate the inspection work corresponding to the identified parts simply by looking at the combination of objects. According to this embodiment, candidate objects are screened by the area of ​​gaze, and then contact between the hand or tool and the candidate object is detected. Therefore, even if the candidate objects are close to each other, the user's work for the object to be performed can be estimated with relatively high accuracy.

[0160] ***Other Structures***

[0161] <Variation Example 1>

[0162] The job estimation unit 250 can not only flexibly use scores to estimate jobs, but also flexibly use other information to estimate jobs. As a specific example, other information is at least one of the attribute information of each candidate object and the attribute information of the object used for the job. Here, as a specific example, attribute information is at least one of the object's position information, scale information, object's shape, and detection confidence (described later).

[0163] Furthermore, the case where other information is attribute information will be explained. The learning data acquisition unit 410 acquires information including the attribute information of each candidate object and the attribute information of the work object as learning data. The learning model generation unit 420 processes the learning data including the attribute information of each candidate object and the attribute information of the work object to generate a learning model.

[0164] <Variation Example 2>

[0165] The contact degree calculation unit 230 can also calculate a score based on the degree of contact between the user's hands and the object.

[0166] <Variation Example 3>

[0167] The contact degree calculation unit 230 can also take into account the degree of contact between the prop and the user's hand to calculate the score.

[0168] According to this variation, it is possible to prevent the work estimation device 200 from mistakenly identifying a prop that is only placed in the field of view as a prop that the user is using.

[0169] <Variation Example 4>

[0170] The job estimation unit 250 can also estimate the user's job when the user is using multiple job objects to perform the job.

[0171] According to this variation, even when the user is using both hands to perform the work, the work estimation unit 250 can also appropriately estimate the user's work.

[0172] <Variation Example 5>

[0173] Figure 12 The following shows a structural example of the job estimation system 90 and a software structural example of the job estimation device 200 of this modified example. The differences between Embodiment 1 and this modified example will be mainly explained below.

[0174] like Figure 12 As shown, the job estimation device 200 has a job action information calculation unit 240 based on the structural elements of the job estimation device 200 in Embodiment 1.

[0175] The object detection unit 210 calculates a detection confidence score for each of the at least one object candidate object. The detection confidence score is a value representing the accuracy of the estimated object candidate object; a higher detection confidence score indicates a more accurate estimation of the corresponding object candidate object. Specifically, the detection confidence score is the classification probability of an object calculated using common object detection methods such as SSD (Single Shot Multibox Detector) or Faster R-CNN (Convolutional Neural Network).

[0176] The operation action information calculation unit 240 updates the detection confidence level using the degree of contact, thereby calculating the updated confidence level. The operation action information calculation unit 240 is also referred to as the object score update unit. The updated confidence level is an indicator based on the degree of contact.

[0177] The job estimation unit 250 estimates the user's job based on the output of the object detection unit 210 and the output of the job action information calculation unit 240.

[0178] Figure 13This is a flowchart illustrating an example of the operation of the job estimation device 200. The operation of the job estimation device 200 will be explained with reference to this diagram.

[0179] (Step S101: Object detection processing)

[0180] The processing in this step is the same as that in step S101 of Embodiment 1. However, the object detection unit 210 calculates information including detection confidence as information corresponding to each detected candidate object.

[0181] (Step S111: Calculation and processing of operation information)

[0182] The operation action information calculation unit 240 uses the score output by the contact degree calculation unit 230, that is, the score associated with each object candidate, to update the detection confidence corresponding to each object candidate output by the object candidate detection unit 215, thereby calculating the updated confidence and outputting the calculated updated confidence as the score.

[0183] Through this step, the job estimation device 200 can estimate the user's job by considering not only the degree of contact between the user and the object being the job, but also the detection confidence level calculated by the candidate object detection unit 215. The job action information calculation unit 240 can also maintain both the detection confidence level and the score associated with the candidate object calculated by the contact degree calculation unit 230. Furthermore, when calculating and updating the confidence level, the job action information calculation unit 240 can use at least one of the position information and the scaling information of each object, or other information related to each object.

[0184] (Step S104: Job estimation processing)

[0185] The processing in this step is the same as that in step S104 of Embodiment 1. However, the job estimation unit 250 uses the score output by the job action information calculation unit 240 instead of the score output by the contact degree calculation unit 230.

[0186] <Variation Example 6>

[0187] Figure 14 This example illustrates the hardware structure of the job estimation device 200 in this modified example.

[0188] As shown in this figure, the job estimation device 200 replaces at least one of the arithmetic device 101, the main storage device 102 and the auxiliary storage device 103 and has a processing circuit 108.

[0189] The processing circuit 108 is hardware that implements at least a portion of the components of the job estimation device 200.

[0190] The processing circuit 108 can be dedicated hardware, or it can be a processor that executes programs stored in the main storage device 102.

[0191] In the case where the processing circuit 108 is dedicated hardware, as a specific example, the processing circuit 108 is a single circuit, a composite circuit, a programming processor, a parallel programming processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0192] The job estimation device 200 may also have multiple processing circuits that replace the processing circuit 108. The multiple processing circuits share the functions of the processing circuit 108.

[0193] In the job estimation device 200, some functions may be implemented through dedicated hardware, while the remaining functions may be implemented through software or firmware.

[0194] As a specific example, the processing circuit 108 is implemented by hardware, software, firmware, or a combination thereof.

[0195] The arithmetic unit 101, main storage unit 102, auxiliary storage unit 103, and processing circuit 108 are collectively referred to as the "processing circuit". That is, the functions of each functional structural element of the job estimation device 200 are realized through the processing circuit. Other devices described in this specification may also be the same as this variant.

[0196] ***Other Implementation Methods***

[0197] Implementation method 1 has been described; however, multiple parts of this implementation method can also be combined. Alternatively, a portion of this implementation method can be implemented. Furthermore, this implementation method can be modified as needed, and can be implemented in any combination as a whole or in part.

[0198] Furthermore, the described embodiments are essentially preferred examples and are not intended to limit the scope of the invention, its applications, and uses. The steps illustrated using flowcharts and other similar methods may also be appropriately modified.

[0199] Label Explanation

[0200] 90: Job estimation system; 100: Computer; 101: Computing device; 102: Main storage device; 103: Auxiliary storage device; 104: First interface; 105: Second interface; 108: Processing circuit; 200: Job estimation device; 210: Object detection unit; 211: Job object detection unit; 212: Prop detection unit; 213: Hand detection unit; 215: Candidate object detection unit; 220: Gaze area estimation unit; 230: Contact degree calculation unit; 240: Job action information calculation unit; 250: Job estimation unit; 260: Estimation result storage unit; 300: Camera device; 350: Line of sight measurement device; 400: Learning device; 410: Learning data acquisition unit; 420: Learning model generation unit; 430: Learning model storage unit; D1: Learning data.

Claims

1. A job estimation device, comprising: The gaze region estimation unit uses information representing the user's gaze to estimate the area the user is gazing at, i.e., the gaze region. The object detection unit detects the work object and the at least one object candidate object based on an image that reflects the work object being used by the user and at least one object candidate object that is a candidate for the user's work object. A contact degree calculation unit calculates the contact degree based on the gaze area, wherein the contact degree represents the degree of contact between the working object and each of the at least one candidate object included in the candidate object; and The job estimation unit estimates the user's job based on the job object and the degree of contact, wherein... The object detection unit calculates a detection confidence level for each of the at least one candidate object included in the candidate object list, representing the degree to which it is estimated to be the user's task object. The job estimation device further includes a job action information calculation unit, which updates the detection confidence level using the contact level, thereby calculating an updated confidence level. The job estimation unit uses the updated confidence level to estimate the user's job; The contact degree calculation unit calculates the weights corresponding to each object candidate object included in the at least one object candidate object based on the gaze area, and uses the calculated weights to determine the contact degree.

2. The job estimation device according to claim 1, wherein, The gaze region estimation unit uses temporal data representing the user's viewpoint position to estimate the gaze region.

3. The job estimation device according to claim 1 or 2, wherein, The contact degree calculation unit determines the contact degree based on the distance between the work object and each of the at least one object candidate object.

4. The job estimation device according to claim 1 or 2, wherein, The contact degree calculation unit calculates the contact degree based on the configuration of the work object and the configuration of each object candidate object included in the at least one object candidate object.

5. The job estimation device according to claim 1 or 2, wherein, The contact degree calculation unit determines the contact degree based on the overlapping area of ​​the work object and the at least one object candidate object.

6. The job estimation device according to claim 1 or 2, wherein, The object used for the task is the user's hand or a tool that the user is using.

7. The job estimation device according to claim 1 or 2, wherein, The job estimation unit estimates the user's job using a rule-based estimation method.

8. The job estimation device according to claim 1 or 2, wherein, The job estimation unit uses a learning model to estimate the user's job.

9. A job estimation method, wherein, The gaze region estimation unit uses information representing the user's gaze to estimate the area the user is gazing at, i.e., the gaze region. The object detection unit detects the work object and the at least one candidate object based on an image that shows the work object being used by the user and at least one candidate object that is a candidate for the user's work object. The contact degree calculation unit calculates the contact degree based on the gaze area, wherein the contact degree represents the degree of contact between the working object and each of the at least one candidate object included in the candidate object list. The job estimation unit estimates the user's job based on the object used for the job and the degree of contact. The object detection unit calculates a detection confidence level for each of the at least one candidate object included in the candidate object list, representing the degree to which it is estimated to be the user's task object. The operation action information calculation unit uses the contact level to update the detection confidence level, thereby calculating the updated confidence level. The job estimation unit uses the updated confidence level to estimate the user's job; The contact degree calculation unit calculates the weights corresponding to each object candidate object included in the at least one object candidate object based on the gaze area, and uses the calculated weights to determine the contact degree.

10. A computer-readable recording medium having recorded a job estimation program that causes a job estimation device, which is a computer, to perform the following processes: The gaze region estimation process uses information representing the user's gaze to estimate the area the user is gazing at, i.e., the gaze region. The object detection process detects the work object and the at least one candidate object based on an image that shows the work object being used by the user and at least one candidate object that is a candidate for the user's work object. The contact degree calculation process calculates the contact degree based on the gaze area, wherein the contact degree represents the degree of contact between the working object and each of the at least one candidate object included in the candidate object list; and The task estimation process estimates the user's task based on the object being worked on and the degree of contact. in, In the object detection process, for each of the at least one candidate object, a detection confidence level representing the degree to which it is estimated to be the user's task object is calculated. The job estimation procedure also causes the job estimation device to perform job action information calculation processing, in which the detection confidence is updated using the contact level, thereby obtaining an updated confidence level. In the job estimation process, the updated confidence level is used to estimate the user's job. In the contact degree calculation process, the weights corresponding to each object candidate object included in the at least one object candidate object are calculated based on the gaze region, and the contact degree is determined using the calculated weights.

Images