Information processing device

Abstract

To determine an operation regardless of the direction of an imaging device attached to a worker.SOLUTION: A video acquisition unit 101 acquires a video. A video storage unit 102 stores the acquired video in relation to the time of taking the video. A sensor measuring unit 202 measures a value used for detecting an eyeball movement. A motion detection unit 103 detects a motion of the eyeballs of the worker on the basis of the result of measurement by the sensor measuring unit 202. An interruption specifying unit 104 specifies a part of the acquired video that was taken when the rate of the detected motion of the eyeballs satisfies a predetermined condition as an interruption of the motion of the worker. For example, the condition is satisfied when the rate of a motion of the eyeballs that shows a blink is measured.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Patent Document 1 describes a technology that determines the start and end of an action from the absolute coordinates of an identification marker worn by a worker who appears in an image, and performs calculations using standard time master data to set a standard time corresponding to the distance when moving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-310617 A Summary of the Invention [Problem to be solved by the invention]

[0004] In addition to using a fixed imaging device to capture images of work, there is also a method of having the worker wear an imaging device. In the latter case, the identification marker is not always within the imaging range of the imaging device, so there is a possibility that the action cannot be identified. Therefore, an object of the present invention is to determine the movement of a worker regardless of the orientation of an imaging device worn by the worker. [Means for solving the problem]

[0005] The information processing device according to claim 1 of the present invention comprises: a processor, the processor acquiring a moving image captured by an imaging device worn by the worker, An image of a working area is captured in advance by an imaging device other than the imaging device worn by the worker, and the working area is associated with a type of motion of the worker; The eye movement of the worker is detected by a sensor worn by the worker, and a portion of the acquired video image captured when the speed of the detected eye movement satisfies a predetermined condition is identified as a break in the worker's movement. and determining a type of motion of the worker associated with the work area based on a position in the image of a gaze point indicated by the detected eye movement and a positional relationship between the work area and the gaze point. The present invention is characterized by the above.

[0006] The information processing device according to claim 2 of the present invention, in the aspect described in claim 1, is characterized in that it has a means for detecting blinking, and in addition to the above conditions, identifies the location photographed when a blink is detected as a break in the worker's movement.

[0007] An information processing device according to claim 3 of the present invention, in the aspect described in claim 1 or 2, is characterized in that the condition is satisfied when the speed of eye movement representing the fixation state of the worker is measured, and the start time or end time of the fixation state detected when the condition is satisfied is identified as a break in the worker's movement.

[0009] Claims of the invention 4 The information processing device according to the present invention is 1 In the aspect described above, the break in the worker's movement or the type of the movement is determined based on whether the position of the gaze point is included within a predetermined range around the work area.

[0010] Claims of the invention 5 The information processing device according to the present invention is 1 In the aspect described above, a representative position in the working area is determined, and a break in the worker's movement or the type of the movement is determined based on the distance between the representative position and the position of the gaze point.

[0011] Claims of the invention 6 The information processing device according to the present invention is 5 In the aspect described in the above, the distance is corrected in accordance with a moving direction of the position of the gaze point.

[0012] Claims of the invention 7 The information processing device according to the present invention comprises: 6 In any one of the aspects described above, the position of a work object is recognized from the acquired moving image, and a break in the worker's movement or the type of the movement is determined based on the positional relationship between the position of the point of gaze indicated by the detected eye movement and the position of the work object.

[0013] Claims of the invention 8 The information processing device according to the present invention is 7 In the aspect described above, the present invention is characterized in that a break in the worker's movement or the type of the movement is determined based on whether the position of the gaze point is included in a determined range around the position of the work object.

[0014] Claims of the invention 9 The information processing device according to the present invention is 7 In the aspect described above, coordinates representing the position of the work object are determined, and a break in the worker's movement or the type of the movement is determined based on the distance between the coordinates and the position of the gaze point.

[0015] Claim 1 of the present invention 0 The information processing device according to the present invention is 9 In the aspect described in the above, the distance is corrected in accordance with a moving direction of the position of the gaze point. Effect of the Invention

[0016] According to the invention as set forth in claim 1, the movement can be determined regardless of the orientation of the imaging device worn by the worker. Furthermore, according to the invention as defined in claim 1, a more appropriate type of action can be determined from an image including a working area, compared to a case in which the positional relationship of the present invention is not taken into consideration. According to the invention as defined in claim 2, it is possible to more reliably identify breaks in motion compared to a case in which blinking is not taken into consideration. According to the invention as defined in claim 3, it is possible to cut out a portion of a moving image in which a task is performed while gazing at a specific location. Claim 4 According to the present invention, more appropriate processing results can be obtained regardless of the shape of the work area. Claim 5 According to the present invention, even if the shape of the area required for the work is unclear, more appropriate processing results can be obtained. Claim 6 According to the present invention, it is possible to prevent erroneous processing results when a worker becomes accustomed to the work, compared to a case where the distance is not corrected. Claim 7 According to the present invention, even if a fixed imaging device is not provided, more appropriate processing results can be obtained compared to the case where the positional relationship of the present invention is not taken into consideration. Claim 8 According to the present invention, more appropriate processing results can be obtained regardless of the shape of the area required for work. Claim 9 According to the present invention, even if the shape of the area required for the work is unclear, more appropriate processing results can be obtained. Claim 1 0 According to the present invention, it is possible to prevent erroneous processing results when a worker becomes accustomed to the work, compared to a case where the distance is not corrected. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a work management system according to an embodiment. [Diagram 2] FIG. 1 shows the hardware configuration of a server device. [Diagram 3] A diagram showing the hardware configuration of a wearable device. [Figure 4] FIG. 1 is a diagram showing a functional configuration realized in an embodiment. [Diagram 5] A diagram to explain the speed of eye movement representing blinking [Figure 6] A diagram for explaining the speed of eye movement that indicates the fixation state [Figure 7] A diagram showing an example of output break data. [Figure 8] FIG. 1 is a diagram illustrating an example of an operation procedure in a specific process. [Figure 9] FIG. 13 is a diagram showing a functional configuration realized in a modified example. [Figure 10] A diagram showing an example of a work area [Figure 11] FIG. 13 is a diagram illustrating an example of a transition of a worker's gaze point. [Figure 12] FIG. 1 is a diagram illustrating an example of an operation table. [Figure 13] FIG. 13 is a diagram illustrating an example of a type of action that has been determined. [Figure 14]FIG. 13 is a diagram illustrating an example of stored information on a gaze point and a distance to a representative position. [Figure 15] FIG. 1 is a diagram illustrating an overall configuration of a work management system according to a modified example. [Figure 16] A diagram showing an example in which the gaze point stops in front of the working area [Figure 17] A magnified view of the area around the fixation point [Figure 18] FIG. 1 is a diagram showing an example of the relationship between the angle θ and the coefficient α. [Figure 19] FIG. 13 is a diagram illustrating an example of an operation table according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] [1] Example FIG. 1 shows the overall configuration of a work management system 1 according to an embodiment. The work management system 1 is a system for managing work performed by workers. Specifically, the work management system 1 captures work as video images and breaks down the captured video images into parts in which each work is being performed, so that the video images can be used to evaluate and improve each work. The work management system 1 includes a communication line 2, a server device 10, and a wearable device 20.

[0019] The communication line 2 is a communication system including a mobile communication network and the Internet, and relays data exchange between devices (devices, terminals, systems, etc.) communicating with the system itself. A server device 10 is connected to the communication line 2 via wired communication, and a wearable device 20 is connected to the communication line 2 via wireless communication. Note that communication between each device and the communication line 2 is not limited to the example shown in FIG. 1, and may be either wired communication or wireless communication.

[0020] The wearable device 20 is a device worn by a worker and is a glasses-type device in this embodiment. The wearable device 20 captures images of the worker's work and measures the worker's eye movements (=eyeball movements) using a sensor. The server device 10 identifies breaks in the worker's movements in the video based on the measurement results from the wearable device 20, and breaks down the work based on the identified breaks so that the results can be used to evaluate and improve each task.

[0021] 2 shows a hardware configuration of the server device 10. The server device 10 is a computer including a processor 11, a memory 12, a storage 13, and a communication device 14. The processor 11 includes, for example, an arithmetic unit such as a CPU (=Central Processing Unit), a register, and peripheral circuits. The memory 12 is a recording medium readable by the processor 11, and includes a RAM (=Random Access Memory), a ROM (=Read Only Memory), and the like.

[0022] The storage 13 is a recording medium readable by the processor 11, and includes, for example, a hard disk drive or a flash memory. The processor 11 uses the RAM as a work area to execute programs stored in the ROM or the storage 13, thereby controlling the operation of each piece of hardware. The communication device 14 includes an antenna, a communication circuit, etc., and is a communication means for communicating via the communication line 2.

[0023] 3 shows the hardware configuration of the wearable device 20. The wearable device 20 is a computer including a processor 21, a memory 22, a storage 23, a communication device 24, a UI (=User Interface) device 25, an imaging device 26, and a sensor device 27. The processor 21 to the communication device 24 are the same type of hardware as the processor 11 to the communication device 24 shown in FIG.

[0024] The UI device 25 is an interface provided to a user who uses the device. In this embodiment, the UI device 25 includes a lens portion of the device, which is a pair of glasses, and displays an image on the lens. The UI device 25 also has a touch panel or buttons provided on the rim, bridge, or temple of the device, and receives operations from the user.

[0025] The imaging device 26 has an optical system and an image sensor, and captures an image based on light that passes through the optical system and enters the image sensor. The sensor device 27 has a sensor for detecting the eye movement of the worker wearing the device. As the sensor, for example, a CMOS sensor (Complementary Metal Oxide Semiconductor) or a sensor for measuring eye movement called electrooculography (EOG) is used.

[0026] Specifically, when sensor device 27 has a CMOS sensor, eye movement is measured by analyzing an image captured by the CMOS sensor mounted on the frame of wearable device 20, which is the device itself. The measured eye movement is processed by a known method, and eye movement represented by coordinates on an image (first-person viewpoint image) captured by an imaging device in the direction in which the worker is gazing is measured.

[0027] Furthermore, if the sensor device 27 has an electro-oculometer, it has two electrodes and an amplifier provided on the left and right outer sides of the operator's eyes, and measures the potential of each electrode. The eye has a certain potential difference called the corneal retinal potential, with the cornea side being positive and the retina side being negative. Since the potential of each electrode is approximately proportional to the rotation angle of the eye, eye movement represented by the rotation angle of the eye is measured by using a rotation angle table that associates the potential of each electrode with the rotation angle of the eye.

[0028] In the work management system 1, the processor of each of the above devices executes a program to control each unit, thereby realizing each function described below. The operation performed by each function is also expressed as the operation performed by the processor of the device that realizes that function. 4 shows the functional configuration realized in this embodiment. The server device 10 includes a video acquisition unit 101, a video storage unit 102, a motion detection unit 103, a gap identification unit 104, and a gap output unit 105. The wearable device 20 includes an imaging unit 201 and a sensor measurement unit 202.

[0029] The imaging unit 201 of the wearable device 20 controls the imaging device 26 to capture moving images during work. The imaging device 26 is provided, for example, with a lens facing in front of the worker's face, and captures moving images that include much of the worker's field of vision. The imaging unit 201 transmits the captured moving images and moving image data indicating the capture time of each frame of the moving images to the server device 10.

[0030] The video acquisition unit 101 of the server device 10 acquires the video indicated by the transmitted video data, i.e., the video captured by the imaging device 26 worn by the worker and the capture time. The video acquisition unit 101 supplies the acquired video and the capture time of each frame to the video storage unit 102. The video storage unit 102 stores the supplied video in association with the capture time of each frame also supplied.

[0031] The sensor measurement unit 202 of the wearable device 20 controls the sensor device 27 to measure values ​​used for detecting eye movement. In this embodiment, the sensor measurement unit 202 measures the above-mentioned eye movement and transmits measurement data indicating the measurement result to the server device 10. The movement detection unit 103 of the server device 10 detects the eye movement of the worker based on the measurement result indicated by the transmitted measurement data, i.e., by the sensor worn by the worker.

[0032] As described above, the movement detection unit 103 measures eye movement by analyzing an image captured by a CMOS sensor mounted on the frame of the own device. The movement detection unit 103 processes the measured eye movement by a known method and measures the eye movement that indicates the gaze direction of the worker by coordinates on an image (first-person viewpoint image) captured by an imaging device. The movement detection unit 103 supplies the gap identification unit 104 with movement data indicating the measured coordinates and the measurement time of the coordinates.

[0033] The movement detection unit 103 may be configured to store a rotation angle table that associates the potential of each electrode with the rotation angles of the left and right eyeballs, and detect eye movements for the left and right eyeballs, respectively, represented by the rotation angles of the eyeballs associated with the measurement results in the rotation angle table. The movement detection unit 103 supplies the gap identification unit 104 with movement data indicating the rotation angles of the left and right eyeballs, which are the detection results of the eye movements, and the detection times of the rotation angles.

[0034] The gap identifying unit 104 identifies, among the moving images acquired by the moving image acquiring unit 101, a portion captured when the speed of eye movement detected by the movement detecting unit 103 satisfies a predetermined condition as a gap in the worker's movement. The gap identifying unit 104 measures the speed of eye movement, for example, by dividing the amount of change in the rotation angle indicated by the supplied movement data by the difference in detection time.

[0035] In this embodiment, the gap identifying unit 104 uses the following two conditions. The first condition (hereinafter, referred to as the "first condition") is satisfied when a blink is detected. The method of detecting a blink will be described with reference to FIG. 5. Fig. 5 is a diagram for explaining the speed of eye movement representing a blink, in which the horizontal axis indicates time and the vertical axis indicates the speed of eye movement.

[0036] The graph shows a threshold D1 for the velocity of eye movement. A broken line G1 showing the change in the velocity of eye movement exceeds the threshold D1 at time t1 (rising edge) and falls below the threshold D1 at time t2 (falling edge). Similarly, a rising edge is measured at time t3 and a falling edge is measured at time t4. The gap identification unit 104 detects that a blink has occurred in the period from the rising edge to the falling edge.

[0037] The gap identifying unit 104 is an example of the "means for detecting a blink" of the present invention. When a blink is detected, the gap identifying unit 104 determines that the first condition is satisfied, and identifies the location photographed when the first condition is satisfied as a gap in the worker's motion. For example, the gap identifying unit 104 identifies a location whose photographing time is the time when a rise or fall is measured as a location photographed when the first condition is satisfied, i.e., a gap in the worker's motion.

[0038] The second condition (hereinafter referred to as the "second condition") is met when the speed of eye movement that indicates a fixation state is measured. The fixation state is a state in which the gaze is fixed on one point. The speed of eye movement that indicates a fixation state will be described with reference to FIG. 6. Fig. 6 is a diagram for explaining the speed of eye movement, which indicates the fixation state, in which the horizontal axis indicates time and the vertical axis indicates the speed of eye movement.

[0039] The graph shows a broken line G2 indicating a change in the speed of eye movement. The speed of eye movement indicating a fixation state is, for example, a speed that is equal to or less than a threshold D2. The speed indicated by the broken line G2 is equal to or less than the threshold D2 at time t11, and is greater than the threshold D2 at time t12. The break identifying unit 104 determines the period from time t11 to t12 when the speed is equal to or less than the threshold D2 as the fixation state period, and identifies the point where the start time t11 of the fixation state is the shooting time as a break in the worker's movement.

[0040] In this way, the gap identifying unit 104 identifies the start time of the fixation state detected when the second condition is satisfied as a gap in the worker's movement. The gap identifying unit 104 may also identify the end time of the fixation state detected when the second condition is satisfied as a gap in the worker's movement. The gap identifying unit 104 supplies gap data indicating the identified gap to the gap output unit 105.

[0041] The break output unit 105 outputs the supplied break data, i.e., data indicating breaks in the actions of the worker identified by the break identification unit 104. The break output unit 105 outputs the break data to, for example, a user terminal (not shown) used by the work manager.

[0042] Fig. 7 shows an example of the output break data. In the example of Fig. 7, the user terminal displays, on the screen of the work management system, a character string saying "Please select a video of the disassembled work", the work time when the video was taken, a thumbnail image of the video, and a play button B11. When the play button B11 is pressed with any thumbnail image selected, the user terminal plays the part of the video that corresponds to the work. The work manager views the played video and checks the productivity, safety, and the like for each task.

[0043] Each device included in the work management system 1 performs a process of identifying a break in work by using the above-mentioned configuration. 8 shows an example of an operation procedure in the identification process. First, wearable device 20 (imaging unit 201) captures a moving image during work (step S11), and transmits moving image data indicating the captured moving image to server device 10 (step S12).

[0044] The server device 10 (video acquisition unit 101) acquires the video indicated by the transmitted video data (step S13). Next, the server device 10 (video storage unit 102) stores the acquired video in association with the shooting time (step S14). Next, the wearable device 20 (sensor measurement unit 202) measures a value used to detect eye movement (step S21) and transmits measurement data indicating the measurement result to the server device 10 (step S22).

[0045] The server device 10 (movement detection unit 103) detects the eye movement of the worker based on the measurement results indicated by the transmitted measurement data (step S23). Next, the server device 10 (break identification unit 104) identifies, from among the acquired video, a point captured when the speed of the detected eye movement satisfied a predetermined condition as a break in the worker's movement (step S24). Then, the server device 10 (break output unit 105) outputs data indicating the identified break in the worker's movement (step S25).

[0046] In this embodiment, as described above, the breaks in the motions in the video captured by the imaging device worn by the worker are identified based on the eye movements detected by the sensor worn by the worker, so that the motions can be identified regardless of the orientation of the imaging device worn by the worker.

[0047] In order to perform a task smoothly, a worker tends to blink as little as possible while working, and tends to blink between tasks. In this embodiment, as described above, by identifying the location photographed when a blink is detected as a break in the worker's motion, the break in the motion can be identified more reliably than when blinking is not taken into consideration.

[0048] In addition, some work is performed while gazing at a specific location. For example, visual inspection work is performed while gazing at a product delivered in front of one's eyes. In this embodiment, as described above, a location photographed when eye movement indicating a fixation state is detected is identified as a break in the worker's actions, and the portion of the work performed while gazing at a specific location is cut out from the video.

[0049] [2] Variations The above-described embodiment is merely one example of the implementation of the present invention, and may be modified as follows. Furthermore, the embodiment and each modification may be combined as necessary.

[0050] [2-1] Type of action In the embodiment, a break in a motion is identified, but the type of motion may also be determined. Fig. 9 shows a functional configuration realized in this modification. Fig. 9 shows a server device 10a including an action type determination unit 106 and an area image storage unit 107 in addition to the units shown in Fig. 4, and a wearable device 20.

[0051] The imaging unit 201 captures a video of the worker's work area. The imaging unit 201 transmits the captured video and video data indicating the capture time of each frame of the video to the server device 10a. The video acquisition unit 101 of the server device 10a acquires the video indicated by the transmitted video data, i.e., the video of the work area captured by the wearable device 20 worn by the worker and the capture time.

[0052] Area image storage unit 107 stores in advance a still image of the work area (hereinafter referred to as a "work area image") captured from directly above workbench A1 by a camera separate from wearable device 20. Wearable device 20 here is an example of an "imaging device worn by a worker" of the present invention, and a camera separate from wearable device 20 is an example of an "other imaging device" of the present invention.

[0053] The area image storage unit 107 may store as the work area image an image obtained by converting a still image captured by the imaging unit 201 of the wearable device 20 into a rectangular shape by identifying the edges and vertices of the work table. The area image storage unit 107 may also attach markers for position correction to predetermined positions in the work area, and store as the work area image an image obtained by converting the captured image into a rectangular work area image based on the markers.

[0054] The action type discrimination unit 106 of the server device 10a discriminates the type of action of the worker based on the position in the image of the gaze point indicated by the eye movement detected by the gap identification unit 104 and the positional relationship between the work area indicated by the work area image stored in the area image storage unit 107. The action type discrimination unit 106 discriminates the type of action of the worker based on, for example, whether or not the position of the gaze point is included in a predetermined range around the work area. This discrimination method will be described with reference to Figs. 10 to 13.

[0055] Fig. 10 shows an example of a work area. Fig. 10 shows a work area image of a work table A1. The work table A1 is provided with work areas C1, C2, C3 and work areas W1, W2, W3. Each work area is provided with markers B1, B2, B3, and B4 to facilitate identification of a position in space. Each marker is, for example, a QR code (registered trademark).

[0056] The work areas C1, C2, and C3 are areas where parts are placed and the action of picking up the parts is performed. The work areas W1, W2, and W3 are areas where the action of attaching the parts to the main body is performed. When a worker works on the workbench A1, the worker's gaze point moves in short bursts at high speed.

[0057] The action type discrimination unit 106 calculates a projective transformation matrix from the moving image to a still image based on the position of the marker for each frame of the moving image captured by the wearable device 20. Then, the action type discrimination unit 106 maps the gaze point position on the moving image onto the still image, and calculates the coordinates of the gaze point position on the still image.

[0058] Fig. 11 shows an example of the transition of the gaze point of a worker. In the example of Fig. 11, first, gaze point 1 appears in working area C1, and gaze points 2 and 3 appear in working area W1. Next, gaze point 4 appears in working area C2, and gaze points 5 and 6 appear in working area W2. Then, gaze point 7 appears in working area C3, and gaze point 8 appears in working area W3. When there is a working area including the calculated gaze point, the action type discrimination unit 106 discriminates the type of action associated with that working area. The action type discrimination unit 106 uses an action table in which working areas are associated with types of actions.

[0059] An example of the action table is shown in Fig. 12. In the example of Fig. 12, the working areas "C1", "C2", "C3", "W1", "W2", and "W3" are associated with the types of actions "action of picking up part E1 (Reach1)", "action of picking up part E2 (Reach2)", "action of picking up part E3 (Reach3)", "action of attaching part E1 (Attach1)", "action of attaching part E2 (Attach2)", and "action of attaching part E3 (Attach3)", respectively.

[0060] For example, when gaze point 1 included in working area C1 is calculated, the action type discrimination unit 106 discriminates the "action of picking up part E1" associated with working area C1 as the type of action. Also, when gaze point 5 or gaze point 6 included in working area W2 is calculated, the action type discrimination unit 106 discriminates the "action of attaching part E2" associated with working area W2 as the type of action.

[0061] Fig. 13 shows an example of the type of action that has been identified. In Fig. 13, the transition in the speed of eye movement, the timing at which eight fixation points from fixation point 1 to fixation point 8 appear, and the type of action that has been identified are shown in chronological order. For example, when fixation point 4 appears, Reach2 (= the action of picking up part E2) is identified. Also, when fixation point 8 appears, Attach3 (= the action of attaching part E3) is identified.

[0062] In this way, the action type discrimination unit 106 discriminates the type of action of the worker based on the position of the gaze point in the image and the positional relationship between the work area. As a result, a more appropriate processing result can be obtained from an image including the work area, that is, a more appropriate type of action can be discriminated, compared to a case where the positional relationship is not considered. Also, the action type discrimination unit 106 discriminates the type of action of the worker based on whether the position of the gaze point is included in a determined range around the work area. By determining this range according to the shape of the work area, a more appropriate processing result can be obtained regardless of the shape of the work area, that is, a more appropriate type of action can be discriminated.

[0063] The action type discrimination unit 106 may determine a representative position in the working area and discriminate the type of action of the worker based on the distance between the representative position and the position of the point of gaze. Each time the point of gaze moves, the action type discrimination unit 106 stores information about the point of gaze and the distance to the representative position in association with each other.

[0064] 14 shows an example of stored information on the gaze points and the distances to the representative positions. For example, it is stored that gaze point 1 starts at 0.0 seconds and ends at 1.2 seconds, the coordinates of the gaze point are (x, y) = (230, 400), and the distances between the gaze point and the representative positions of the working areas C1, W1, C2, W2, C3, and W3 are 100, 800, 1600, 1300, 1100, and 1000.

[0065] The action type discrimination unit 106 discriminates the type of action associated with a working area in which a representative position whose distance from the position of the gaze point is less than a threshold is determined. This threshold is hereinafter referred to as a "discrimination threshold." For example, if the action type discrimination unit 106 uses the action table of FIG. 12 and the discrimination threshold is 200, the action type discrimination unit 106 discriminates the type of action as "action of picking up part E1 (Reach1)" associated with working area C1 in the action table, since the working area in which the distance from the representative position of the gaze point 1 is less than the discrimination threshold is C1. In this case, even if the shape of the working area is unclear, a more appropriate processing result is obtained, that is, a more appropriate type of action is discriminated.

[0066] A fixed camera may be provided to capture images of the work area. Fig. 15 shows the overall configuration of a work management system 1a according to this modified example. The work management system 1a includes a fixed camera 30 in addition to the devices shown in Fig. 1. The fixed camera 30 is an imaging device fixed to a position capable of capturing an image of the worker's work area. The fixed camera 30 is installed, for example, vertically above the center of the work area and captures an image vertically below.

[0067] Fixed camera 30 is connected to communication line 2 via wired communication (or wireless communication) and exchanges data with each device. Fixed camera 30 has a function equivalent to that of imaging unit 201 of wearable device 20, and transmits video data capturing an image of the work area to server device 10a. Fixing the camera so that the work area is projected as a rectangle in this way eliminates the processing load of mapping and transforming the video image of the work area.

[0068] [2-2] Stopping the gaze point As a worker becomes accustomed to the task, he or she remembers the position of the next task area, and the gaze point may stop in front of the task area without moving the gaze to the task area. In such a case, the distance to the representative position of the next task area is equal to or greater than the discrimination threshold, but the distance to the representative position of another task area is less than the discrimination threshold, and the wrong type of motion may be identified.

[0069] Fig. 16 shows an example in which the gaze point stops in front of a working area. In the example of Fig. 16, the gaze point 4 stops in front of the representative position of the next working area C2. 17 shows an enlarged view of the periphery of fixation point 4. Fixation point 4 stops at a position where distance L1 from the representative position of working area C2 is equal to or greater than the discrimination threshold, and distance L2 from the representative position of working area C3 is less than the discrimination threshold.

[0070] In order to avoid misjudging the type of action in this state, the action type discrimination unit 106 corrects the distance between the representative position and the position of the gaze point according to the moving direction of the gaze point position. In Fig. 17, the arrow from gaze point 3 to gaze point 4 represents the moving direction F1 of the gaze point. The action type discrimination unit 106 calculates the angle θ between the direction from the previous gaze point toward the representative position of the working area and the moving direction F1.

[0071] 17, the action type discrimination unit 106 calculates an angle θ1 between the direction from the fixation point 3 toward the representative position of the working area C2 and the movement direction F1, and an angle θ2 between the direction from the fixation point 3 toward the representative position of the working area C3 and the movement direction F1. The action type discrimination unit 106 corrects the distance using a coefficient α according to the angle θ.

[0072] FIG. 18 shows an example of the relationship between the angle θ and the coefficient α. In the example of FIG. 18, when the angle θ is 0 degrees, the coefficient α is also 1, and the coefficient α increases in proportion to the angle θ until the angle θ becomes 45 degrees, at which point the coefficient α becomes 10. When the angle θ exceeds 45 degrees, the coefficient α is fixed at 10. For example, when the angle θ1 is 4.5 degrees, the angle θ2 is 18 degrees, the distance L1 is 0.2, and the distance L2 is 0.1, the motion type discrimination unit 106 calculates the corrected distance L1=(4.5 degrees÷45×9+1)×distance L1=1.9×0.2=0.38 and the corrected distance L2(18 degrees÷45×9+1)×distance L2=4.6×0.1=0.46. Here, the distances L1 and L2 are normalized with the screen width of the still image set to 1.

[0073] The motion type discrimination unit 106 judges whether the calculated corrected distance L1 and corrected distance L2 are equal to or greater than the discrimination threshold. In this example, when distance L1 is compared with distance L2, distance L2 is smaller, but when corrected distance L1 is compared with corrected distance L2, corrected distance L1 is smaller, and it can be determined that the gaze is directed toward L1. According to this modification, compared to a case where the distance is not corrected, it is possible to suppress erroneous processing results, i.e., to prevent an erroneous motion type from being determined, when the worker becomes accustomed to the work.

[0074] [2-3] Positional relationship with the work object When determining the type of action, the action type determination unit 106 first recognizes the position of the work object from the video acquired by the video acquisition unit 101. The action type determination unit 106 stores sample images of work objects such as parts, for example, and identifies the work object represented by the sample images having similar features used in image recognition.

[0075] The action type discrimination unit 106 prestores image features of the work object, and calculates the distance and direction to the work object based on the image features and position in the video of the identified work object.The action type discrimination unit 106 then calculates the coordinates of the work object in three-dimensional space based on the calculated distance and direction, and recognizes the position of the work object from the calculated coordinates.The action type discrimination unit 106 discriminates the type of action based on the positional relationship between the position in the video of the gaze point indicated by the eye movement detected by the gap identification unit 104 and the recognized position of the work object.

[0076] The action type discrimination unit 106, for example, assumes that the calculated distance to the work object is the distance to the gaze point, and specifies the position of the gaze point in the video image according to the eyeball rotation angle indicating the eyeball movement detected by the movement detection unit 103. The action type discrimination unit 106 discriminates the type of action of the worker, for example, based on whether or not the specified position of the gaze point is included in a predetermined range around the position of the recognized work object.

[0077] The action type discrimination unit 106 stores, for example, the shape and image features of the area required for work for each work object, and determines the area including the position of the recognized work object as the above-mentioned range. When the determined range includes the position of the specified gaze point, the action type discrimination unit 106 determines the type of action associated with the range. The action type discrimination unit 106 uses an action table that associates work objects with types of actions.

[0078] Fig. 19 shows an example of the action table of this modified example. In the example of Fig. 19, the task objects "part E1", "part E2", and "part E3" are associated with the action types "action of picking up part E1 and attaching it to the main body", "action of picking up part E2 and attaching it to the main body", and "action of picking up part E3 and attaching it to the main body", respectively.

[0079] For example, when gaze point 1 is calculated to be within a predetermined range around part E1, the action type discrimination unit 106 discriminates the type of action as "the action of picking up part E1 and attaching it to the main body" associated with part E1 when gaze point 2 is calculated to be within a predetermined range around part E2, the action type discrimination unit 106 discriminates the type of action as "the action of picking up part E2 and attaching it to the main body" associated with part E2 ...

[0080] Thus, in this modification, the motion type discrimination unit 106 discriminates the type of motion of the worker based on the positional relationship between the position of the gaze point in the video and the position of the work object. Since the position of the work object is recognized from the video, this modification can obtain more appropriate processing results, that is, a more appropriate type of motion can be discriminated, compared to a case where the positional relationship is not taken into consideration, even without providing a fixed imaging device.

[0081] Furthermore, the motion type discrimination unit 106 discriminates the type of motion of the worker based on whether the position of the gaze point is included in a determined range around the position of the work object. By determining this range according to the shape and size of the area required for work on the work object, more appropriate processing results can be obtained regardless of the shape of the area required for work, that is, a more appropriate type of motion can be discriminated.

[0082] The action type discrimination unit 106 may determine coordinates representing the position of the work object, and discriminate the type of action of the worker based on the distance between the coordinates and the position of the gaze point. The coordinates representing the position of the work object may be the coordinates of the position if there is only one recognized position of the work object, or may be a position shifted in a predetermined direction and distance from that position. Furthermore, the coordinates representing the position of the work object may be the coordinates of the center of the positions if there are multiple recognized positions of the work object.

[0083] The coordinates representing the position of the work object may be a position shifted in a predetermined direction and distance from the center of the positions of the recognized work objects. Each time the gaze point moves, the action type discrimination unit 106 stores information about the gaze point in association with the distance to the representative coordinates. The action type discrimination unit 106 discriminates the type of action associated with a work object having coordinates whose distance from the gaze point position is less than a discrimination threshold.

[0084] For example, when the work object whose representative coordinates are less than the discrimination threshold from the position of the gaze point are part E1, the action type discrimination unit 106 discriminates the type of action as "the action of picking up part E1 and attaching it to the main body" associated with part E1 in the action table of Fig. 19. In this case, even if the shape of the work area is unclear, a more appropriate processing result can be obtained, that is, a more appropriate type of action can be discriminated.

[0085] In this modified example, the motion type discrimination unit 106 may also correct the distance between the representative coordinate and the position of the fixation point according to the moving direction of the fixation point position in preparation for the case where the fixation point stops in front of the area required for the work. This correction may be performed by the method described in Fig. 16 to Fig. 18. This suppresses erroneous processing results, i.e., the determination of an erroneous type of motion, when the worker becomes accustomed to the work, compared to the case where the distance is not corrected.

[0086] [2-4] How to identify the break in motion The gap identifying unit 104 may identify a gap in a motion in different ways in different embodiments. For example, when a video image of a work area of ​​a worker is captured by the wearable device 20, the gap identifying unit 104 identifies a gap in a motion based on the position of a gaze point indicated by the detected eye movement in the image and the positional relationship between the work area.

[0087] The gap identifying unit 104 identifies a gap in an action based on, for example, whether or not the position of the gaze point is included in a predetermined range around the working area. Specifically, the gap identifying unit 104 determines that a predetermined condition is satisfied when the position of the gaze point is changed from a state in which it is not included in the predetermined range to a state in which it is included. This makes it possible to obtain a more appropriate processing result from an image including the working area, i.e., to identify a more appropriate gap in an action, compared to a case in which the positional relationship between the gaze point position in the image and the working area is not taken into consideration.

[0088] The gap identifying unit 104 may also identify a gap in an action based on the distance between the representative position in the work area and the position of the gaze point. Specifically, when the distance between the representative position and the position of the gaze point changes from a state in which the distance is equal to or greater than the discrimination threshold to a state in which the distance is less than the discrimination threshold, the gap identifying unit 104 determines that a predetermined condition is satisfied and identifies a gap in an action. This makes it possible to obtain a more appropriate processing result even if the shape of the area required for the work is unclear, that is, to identify a more appropriate gap in an action.

[0089] Furthermore, the gap identification unit 104 may correct the distance between the representative position and the position of the gaze point according to the moving direction of the gaze point position. The distance may be corrected in the same manner as in the above-mentioned modified example. This prevents erroneous processing results, that is, erroneous identification of gaps in an action, when the worker becomes accustomed to the work, compared to the case where the distance is not corrected.

[0090] Furthermore, the gap identifying unit 104 may recognize the position of a work object from the video acquired by the video acquiring unit 101, and identify a gap in the movement based on the positional relationship between the position of the recognized work object and the position of the gaze point in the video indicated by the eye movement detected by the gap identifying unit 104. The gap identifying unit 104 identifies a gap in the movement based on, for example, whether or not the position of the gaze point is included within a predetermined range around the recognized position of the work object.

[0091] As in the above modification, the gap identifying unit 104 stores the shape and size of the area required for each work object, and determines the area including the position of the recognized work object as the above range. Then, when the determined range changes from a state in which the position of the identified gaze point is not included in the determined range to a state in which it is included in the determined range, the gap identifying unit 104 determines that the determined condition is satisfied, and identifies the gap in the action.

[0092] As a result, as in the above-mentioned modified example, even without providing a fixed imaging device, more appropriate processing results can be obtained, i.e., more appropriate breaks in the movement can be identified, compared to the case where the positional relationship between the position of the gaze point in the moving image and the position of the work object is not taken into consideration. Also, by identifying the breaks based on whether the position of the gaze point is included in a predetermined range around the position of the work object, more appropriate processing results can be obtained, i.e., more appropriate breaks in the movement can be identified, regardless of the shape of the area required for the work.

[0093] The gap identifying unit 104 may also determine coordinates representing the position of the task target, and identify gaps in the motion based on the distance between the coordinates and the position of the gaze point. When the distance from the position of the gaze point changes from a state where the distance is equal to or greater than the discrimination threshold to a state where the distance is less than the discrimination threshold, the gap identifying unit 104 determines that a predetermined condition is satisfied, and identifies gaps in the motion. This makes it possible to obtain more appropriate processing results, i.e., to identify more appropriate gaps in the motion, even if the shape of the work area is unclear.

[0094] Also in this modified example, in preparation for the case where the gaze point stops in front of an area required for the work, the gap identification unit 104 may correct the distance between the representative coordinate and the gaze point position according to the moving direction of the gaze point position. This correction may be performed by the method described in Fig. 16 to Fig. 18. This suppresses erroneous processing results, that is, erroneous identification of a gap in an action, when the worker becomes accustomed to the work, compared to the case where the distance is not corrected.

[0095] [2-5] Functional configuration The functional configurations realized by the server apparatus 10, the wearable device 20, and the fixed camera 30 are not limited to those shown in Fig. 4 etc. For example, in the embodiment, the gap identification unit 104 identifies a gap using the first condition and a gap using the second condition, but these operations may be performed by separate functions.

[0096] For example, the operations performed by the video acquisition unit 101 and the video storage unit 102 and the operations performed by the break identification unit 104 and the break output unit 105 may each be performed by one function. The functions realized by the server device 10 may also be realized by two or more information processing devices or computer resources provided by a cloud service. In short, as long as the functions shown in Fig. 4 etc. are realized as a whole, the scope of operations performed by each function and the device that realizes each function may be freely determined.

[0097] [2-6] Processor In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0098] In addition, the operations of the processor in each of the above embodiments may be performed not only by one processor, but also by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate.

[0099] [2-7] Invention Categories The present invention can be understood as an information processing device, such as a server device and a wearable device, as well as an information processing method for implementing the processing performed by the information processing device, and as a program for causing a computer that controls the information processing device to function. This program may be provided in the form of a recording medium such as an optical disk on which it is stored, or may be provided in the form of a program that is downloaded to a computer via a communication line such as the Internet and installed to make it available. [Explanation of symbols]

[0100] 1...work management system, 10...server device, 20...wearable device, 30...fixed camera, 101...video acquisition unit, 102...video storage unit, 103...motion detection unit, 104...gap identification unit, 105...gap output unit, 106...motion type discrimination unit, 107...area image storage unit, 201...imaging unit, 202...sensor measurement unit.

Claims

1. a processor, the processor comprising: Acquire video images captured by an imaging device worn by the worker, An image of a work area is captured in advance by an imaging device other than the imaging device worn by the worker; Correlating the work area with a type of motion of the worker; Detecting eye movements of the worker using a sensor worn by the worker; Identifying, from among the acquired moving images, a portion captured when the detected speed of the eye movement satisfies a predetermined condition as a break in the worker's movement; A type of motion of the worker associated with the work area is determined based on a position in the image of a gaze point indicated by the detected eye movement and a positional relationship between the work area and the gaze point.

23. An information processing apparatus comprising:

2. The apparatus has a means for detecting blinking, and identifies a portion photographed when a blinking is detected as a break in the worker's movement in addition to the above conditions. The information processing device according to claim 1 .

3. The condition is satisfied when the speed of eye movement representing the fixation state of the worker is measured, and the start time or end time of the fixation state detected when the condition is satisfied is identified as a break in the worker's movement.

3. The information processing device according to claim 1 or 2.

4. A break in the motion of the worker or a type of the motion is determined based on whether the position of the gaze point is included in a predetermined range around the working area. The information processing device according to claim 1 .

5. A representative position in the working area is determined, and a break in the motion of the worker or a type of the motion is determined based on the distance between the representative position and the position of the gaze point. The information processing device according to claim 1 .

6. The distance is corrected according to a moving direction of the position of the gaze point. The information processing device according to claim 5 .

7. The position of the work object is recognized from the acquired moving image, and a break in the movement of the worker or a type of the movement is determined based on a positional relationship between the position of the gaze point indicated by the detected eye movement and the position of the work object. The information processing device according to claim 1 .

8. A break in the motion of the worker or a type of the motion is determined based on whether the position of the gaze point is included in a predetermined range around the position of the work object. The information processing device according to claim 7.

9. Coordinates representing the position of the work object are determined, and a break in the motion of the worker or a type of the motion is determined based on the distance between the coordinates and the position of the gaze point. The information processing device according to claim 7.

10. The distance is corrected according to a moving direction of the position of the gaze point. The information processing device according to claim 9.

Images