Information processing device, information processing system, and method

Abstract

To enable workers to perform their intended operations while minimizing a decrease in work efficiency. [Solution] An information processing device comprising: a display control unit that displays a guidance screen on a display unit that guides the procedure for a task using tools or parts; and an action estimation unit that estimates the actions of the worker based on the output from a terminal having an inertial sensor attached to the worker, wherein the display control unit changes the guidance screen displayed on the display unit according to the actions estimated by the action estimation unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a method.

Background Art

[0002] In recent years, labor shortages and talent shortages have become serious in various industries. For example, in the manufacturing industry, there are concerns about a decline in productivity due to factors such as the inability to train young workers and the lack of skills among workers, and companies are being forced to take countermeasures.

[0003] As a countermeasure against the lack of skills at the work site, a system for performing efficient maintenance inspections regardless of the experience of workers is disclosed in Patent Document 1. In the system disclosed in Patent Document 1, a worker wears smart glasses and performs work while browsing a manual projected inside the smart glasses. The system tracks the worker's line of sight with the smart glasses, and when the worker's line of sight is directed at the displayed work procedure manual for a predetermined time or longer, a confirmed mark is displayed and the next page is displayed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the system disclosed in Patent Document 1, if an operator is carefully looking at the work procedure manual, eye tracking may mark it as confirmed, causing the manual to move to the next page, which could disrupt the work. Also, in the system disclosed in Patent Document 1, an operator can operate the smart glasses with gestures, but the operator's gestures are recognized based on the image from the camera mounted on the smart glasses. Therefore, if the operator is holding tools or parts in both hands, the gestures may not be recognized correctly, and the operator may not be able to operate the smart glasses as intended. It should be noted that the operator can operate the smart glasses by releasing the tools or parts held in both hands and making gestures, but this reduces work efficiency.

[0006] Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to provide a technology that enables the operator to perform the intended operation while suppressing a decrease in work efficiency. [Means for solving the problem]

[0007] To solve the above problems, according to one aspect of the present invention, an information processing device is provided comprising: a display control unit that displays a guidance screen on a display unit that guides the procedure of work in which a tool or part is used; and an action estimation unit that estimates the actions of the worker based on the output from a terminal having an inertial sensor attached to the worker, wherein the display control unit changes the guidance screen displayed on the display unit according to the actions estimated by the action estimation unit.

[0008] The display control unit may change the content of the guidance screen to indicate a subsequent operation based on the operation estimation unit's estimation that the operator's operation is a first operation.

[0009] The display control unit may change the content of the guidance screen to content indicating the previous operation based on the operation estimation unit's estimation that the operator's operation is a second operation.

[0010] The information processing device further includes a viewpoint estimation unit that estimates the viewpoint position, which is the position from which the operator is viewing the guidance screen, and the display control unit may change the guidance screen displayed on the display unit in accordance with the viewpoint estimation result by the viewpoint estimation unit, in addition to the operation estimated by the operation estimation unit.

[0011] The display control unit may change the content of the guidance screen to content indicating a subsequent task based on the fact that the operation estimation unit estimates the operator's operation to be a first operation and that the position of the viewpoint on the guidance screen is estimated to be a first position.

[0012] The display control unit may change the content of the guidance screen to content indicating the previous operation based on the fact that the operation estimation unit estimates the operator's operation to be a second operation and that the position of the viewpoint on the guidance screen is estimated to be the second position.

[0013] Based on the fact that the operation estimation unit has estimated the operator's operation to be a third operation, the display control unit may reduce the range of the guidance screen corresponding to the position of the viewpoint.

[0014] Based on the fact that the operation estimation unit has estimated the operator's operation to be a fourth operation, the display control unit may enlarge the display of the range corresponding to the viewpoint position on the guidance screen.

[0015] Furthermore, in order to solve the above problems, according to another aspect of the present invention, there is an information processing system having a terminal having an inertial sensor to be worn by a worker, and an information processing device, wherein the information processing system comprises a display control unit that causes a guidance screen to be displayed on a display unit that guides the worker through the procedure of using a tool or part, and an action estimation unit that estimates the worker's actions based on the output from the terminal, and the display control unit changes the guidance screen to be displayed on the display unit according to the actions estimated by the action estimation unit, thereby providing an information processing device.

[0016] According to another aspect of the present invention to solve the above problems, a computer-executed method is provided, which includes causing a display unit to display a guidance screen for guiding a procedure of an operation in which a tool or a part is used, estimating an operation of the operator based on an output from a terminal having an inertial sensor worn by the operator, and changing the guidance screen to be displayed on the display unit according to the estimated operation.

Effect of the Invention

[0017] According to the present invention described above, it is possible to realize an operation intended by an operator while suppressing a decrease in work efficiency.

Brief Description of the Drawings

[0018] [Figure 1] It is an explanatory diagram showing a work support system according to an embodiment of the present invention. [Figure 2] It is an explanatory diagram showing a specific example of a guidance screen. [Figure 3] It is an explanatory diagram showing a configuration of a work support system according to an embodiment of the present invention. [Figure 4] It is an explanatory diagram showing a wearing state of the operator terminal 10. [Figure 5] It is a graph showing changes in acceleration in each of the X-axis, Y-axis, and Z-axis. [Figure 6] It is a graph showing changes in acceleration in each of the X-axis, Y-axis, and Z-axis. [Figure 7] It is a graph showing changes in acceleration in each of the X-axis, Y-axis, and Z-axis. [Figure 8] It is a graph showing changes in acceleration in each of the X-axis, Y-axis, and Z-axis. [Figure 9] It is an explanatory diagram showing a specific example of a work procedure table. [Figure 10] It is an explanatory diagram showing a specific example of a guidance screen. [Figure 11] It is a sequence diagram showing an operation of a work support system according to an embodiment of the present invention. [Figure 12]It is a sequence diagram showing the operation of a work support system according to an embodiment of the present invention following FIG. 11. [Figure 13] It is a block diagram showing an example of the hardware configuration 90.

Mode for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0020] An embodiment of the present invention relates to a work support system for supporting workers who perform inspections or assembly work, etc. Hereinafter, after explaining the outline of the work support system according to an embodiment of the present invention, the configuration and operation of the work support system will be sequentially described in detail.

[0021] <Outline of Work Support System> FIG. 1 is an explanatory diagram showing a work support system according to an embodiment of the present invention. As shown in FIG. 1, the work support system according to an embodiment of the present invention includes a worker terminal 10, a gaze detection device 20, a display device 30, and a work support device 40.

[0022] The worker terminal 10, the gaze detection device 20, and the display device 30 are communicably connected to the work support device 40. The worker terminal 10, the gaze detection device 20, and the display device 30 and the work support device 40 may be connected via the Internet, may be connected by a local network, may be connected by wire, or may be wirelessly connected by Bluetooth or a wireless LAN, etc.

[0023] Note that hereinafter, descriptions regarding power supplies, switches, etc. required for each device, synchronization processing between devices, processing blocks regarding selection of work procedures at the start of work, user registration, user identification mechanisms, and error processing, etc. will be omitted.

[0024] (Worker terminal 10) The worker terminal 10 is a terminal worn by the worker. In the example shown in Figure 1, the worker terminal 10 is a wearable terminal worn on the worker's arm (wrist). The worker terminal 10 has an inertial sensor such as an acceleration sensor or an angular velocity sensor, and transmits the data obtained from the inertial sensor to the work support device 40.

[0025] In the example shown in Figure 1, the worker holds part D in their left hand and tool G, an example of a tool, in their right hand. The worker assembles product O while checking the guidance screen displayed on the display device 30.

[0026] Although Figure 1 shows an example where the worker terminal 10 is attached to the worker's left wrist, the attachment site for the worker terminal 10 is not limited to the left wrist. For example, the attachment site for the worker terminal 10 may be the worker's right wrist, head, upper arm, shoulder, waist, thigh, or ankle. Furthermore, the number of worker terminals 10 attached to a worker is not limited to one; there may be two or more.

[0027] (Eye-tracking device 20) The gaze detection device 20 detects the worker's line of sight. For example, the gaze detection device 20 captures an image of the worker and detects the worker's line of sight based on the position of the worker's pupils. The gaze detection device 20 transmits the line of sight detection results to the work support device 40.

[0028] Although Figure 1 shows an example where the gaze detection device 20 is provided separately from the display device 30, the functions of the gaze detection device 20 and the display device 30 may be integrated into the same device. Furthermore, the gaze detection device 20 does not need to be a device dedicated to gaze detection; it may be shared with an imaging device that monitors and records the assembly and work status of product O.

[0029] (Display device 30) The display device 30 displays a guidance screen that guides the worker through the procedures for performing the task. The display device 30 can receive display data for the guidance screen (hereinafter sometimes simply referred to as the guidance screen) from the work support device 40 and display the guidance screen. A specific example of the guidance screen will be explained below with reference to Figure 2.

[0030] Figure 2 is an explanatory diagram showing a specific example of a guidance screen. As shown in Figure 2, the guidance screen includes a work number display C11, a work name display C12, an image of the work target C13, a tool information display C14, a guidance message C15, a back indicator P1, and a forward indicator P2.

[0031] In inspection and assembly operations, multiple tasks are sequentially performed, and the work support device 40 stores each task in association with a task number, task name, image, tool information, and guidance message. The tool information indicates the tool used in the corresponding task. The guidance message indicates points to pay attention to when performing the task.

[0032] On the guidance screen, the work number display C11 indicates the work number, the work name display C12 indicates the work name, the work target image C13 indicates the image, the tool information display C14 indicates the tool information, and the guidance message C15 indicates the guidance message. The back display P1 is used to change the content of the guidance screen to content showing the previous step. The forward display P2 is used to change the content of the guidance screen to content showing the next step.

[0033] Furthermore, as shown in Figure 2, the guidance screen includes a viewpoint indicator E. The viewpoint indicator E is positioned at the viewpoint where the worker is viewing the guidance screen, by the work support device 40, which will be described later.

[0034] (Work support device 40) The work support device 40 is an information processing device that generates the above-mentioned guidance screen and transmits the guidance screen to the display device 30, thereby displaying the guidance screen on the display device 30. The work support device 40 changes the guidance screen based on data received from the worker terminal 10 and the gaze detection results received from the gaze detection device 20. The detailed configuration of the work support system, and in particular the detailed configuration of the work support device 40, will be described below with reference to Figure 3.

[0035] <Configuration of the work support system> Figure 3 is an explanatory diagram showing the configuration of a work support system according to one embodiment of the present invention. As shown in Figure 3, the worker terminal 10 has an acceleration detection unit 110, the gaze detection device 20 has a gaze detection unit 210, and the display device 30 has a display unit 310. The acceleration detection unit 110 is an example of an inertial sensor and detects the three-axis movement of the worker at the part to which the worker terminal 10 is attached. The gaze detection unit 210 detects the worker's gaze. The display unit 310 displays a guidance screen. Although partially omitted in Figure 3, the worker terminal 10, gaze detection device 20, display device 30, and work support device 40 may have communication functions, control functions, memory functions, and operation detection functions.

[0036] The work support device 40 includes an action estimation unit 410, a viewpoint estimation unit 420, an operation determination unit 430, a procedure storage unit 440, a procedure selection unit 450, and a display control unit 460.

[0037] (Motion estimation unit 410) The motion estimation unit 410 acquires acceleration data from the acceleration detection unit 110, normalizes the acceleration values, and estimates what kind of action the worker performed. The following describes an example of motion estimation by the motion estimation unit 410 with reference to Figures 4 to 8.

[0038] Figure 4 is an explanatory diagram showing the mounting state of the worker terminal 10. In Figure 4, the worker terminal 10 is mounted on the worker's left arm W. The worker terminal 10 detects acceleration in three axes. In Figure 4, the positive directions of the X, Y, and Z axes that constitute these three axes are indicated by X1, Y1, and Z1, respectively.

[0039] Figures 5 to 8 are graphs showing the changes in acceleration along the X, Y, and Z axes, respectively. In the graph shown in Figure 5, the change in the positive direction of the Y axis is larger than the changes along the other axes, indicating that the worker's left wrist moved strongly in the positive direction of the Y axis. Thus, when the change in the positive direction of the Y axis is larger than the changes along the other axes, and the magnitude of the change in the Y axis (for example, the magnitude of the change in the Y axis relative to the changes along the other axes) meets a predetermined standard, the motion estimation unit 410 estimates that the worker twisted their left wrist inward (first motion).

[0040] In the graph shown in Figure 6, the change in the negative direction of the Y-axis is larger than the changes in the other axes, indicating that the worker's left wrist moved strongly in the negative direction of the Y-axis. Thus, when the change in the negative direction of the Y-axis is larger than the changes in the other axes, and the magnitude of the change in the Y-axis (for example, the magnitude of the change in the Y-axis relative to the changes in the other axes) meets a predetermined standard, the motion estimation unit 410 estimates that the worker twisted their left wrist outward (second motion).

[0041] In the graph shown in Figure 7, the change in the negative direction of the Z-axis is larger than the changes in the other axes. Thus, if the change in the negative direction of the Z-axis is larger than the changes in the other axes, and the magnitude of the change in the Z-axis (for example, the magnitude of the change in the Z-axis relative to the changes in the other axes) meets a predetermined standard, the motion estimation unit 410 estimates that the worker strongly flicked only the back of their left hand upward (third motion). However, if the worker raises their entire left arm, the change in the Z-axis reverses in sign, resulting in the graph shown in Figure 8.

[0042] In the graph shown in Figure 8, the change in the positive direction of the Z-axis is larger than the changes in the other axes. Thus, if the change in the positive direction of the Z-axis is larger than the changes in the other axes, and the magnitude of the change in the Z-axis (for example, the magnitude of the change in the Z-axis relative to the changes in the other axes) meets a predetermined standard, the motion estimation unit 410 estimates that the worker has strongly lowered only the back of their left hand (fourth motion). However, if the worker lowers their entire left arm, the change in the Z-axis will be reversed, resulting in the graph shown in Figure 7.

[0043] The motion estimation unit 410 outputs the estimated result of the worker's movements to the operation determination unit 430. The functions of the motion estimation unit 410 may also be implemented in the worker terminal 10.

[0044] (Viewpoint estimation unit 420) Referring to Figure 3, we return to the explanation of the work support device 40. The viewpoint estimation unit 420 estimates the position of the worker's viewpoint (the part the worker is looking at) on the guidance screen displayed on the display unit 310, based on the detection result of the worker's gaze obtained from the gaze detection unit 210. The viewpoint estimation unit 420 outputs the estimated position of the worker's viewpoint to the operation determination unit 430 and the display control unit 460.

[0045] (Operation determination unit 430) The operation determination unit 430 determines what operation the operator performed on the guidance screen based on the operator's actions estimated by the action estimation unit 410 and the position of the operator's viewpoint estimated by the viewpoint estimation unit 420.

[0046] For example, the operation determination unit 430 may determine that the operation performed by the operator is an instruction to transition to the next page, based on the assumption that the operator's action is a first action and that the operator's viewpoint remained within the range of the "proceed" display P2 on the guidance screen (the first position) for a predetermined period of time or longer. The content of the operation may not be a page transition, but rather scrolling to a section indicating the next task.

[0047] Furthermore, the operation determination unit 430 may determine that the operation performed by the operator is an instruction to transition to the previous page, based on the assumption that the operator's action is a second action and that the operator's viewpoint remained within the range of the back indicator P1 on the guidance screen (the second position) for a predetermined period of time or longer. Note that the content of the operation may not be a page transition, but rather scrolling to the section indicating the previous operation.

[0048] However, the operation determination unit 430 may determine that the operation performed by the worker is an instruction to transition to the next page, based on the assumption that the worker's action is a first action, without using information about the worker's viewpoint position. Similarly, the operation determination unit 430 may determine that the operation performed by the worker is an instruction to transition to the previous page, based on the assumption that the worker's action is a second action, without using information about the worker's viewpoint position.

[0049] Furthermore, the operation determination unit 430 may determine, based on the estimation that the operator's action is a third action, that the operation performed by the operator is an instruction to reduce the display within a range corresponding to the operator's viewpoint (for example, a range centered on the viewpoint).

[0050] Furthermore, the operation determination unit 430 may determine, based on the estimation that the operator's action is a fourth action, that the operation performed by the operator is an instruction to enlarge the display of a range corresponding to the operator's viewpoint (for example, a range centered on the viewpoint).

[0051] (Procedure storage unit 440) The procedure storage unit 440 stores multiple work procedure tables. Each work procedure table contains information about a series of operations. A specific example of a work procedure table will be explained below with reference to Figure 9.

[0052] Figure 9 is an explanatory diagram showing a specific example of a work procedure table. Figure 9 shows three work procedure tables T1 to T3. Each work procedure table is associated with a work number C1, work name C2, image C3, tool C4, and guidance message C5. The work number C1, work name C2, image C3, tool C4, and guidance message C5 correspond to the work number display C11, work name display C12, work target image C13, tool information display C14, and guidance message C15 in the guidance screen explained with reference to Figure 2.

[0053] (Procedure selection section 450) Referring to Figure 3, we return to the explanation of the work support device 40. The procedure selection unit 450 selects which work procedure to present to the worker on the guidance screen. The procedure selection unit 450 then reads the procedure (information) for the selected work from the procedure storage unit 440 and outputs the read information to the display control unit 460.

[0054] For example, based on the operation decision unit 430's determination that an operation to transition to the next page has been performed, the procedure selection unit 450 selects the next task for which the procedure is currently presented to the worker and reads the information of the selected task from the procedure storage unit 440. Also, based on the operation decision unit 430's determination that an operation to transition to the previous page has been performed, the procedure selection unit 450 selects the previous task for which the procedure is currently presented to the worker and reads the information of the selected task from the procedure storage unit 440.

[0055] (Display control unit 460) The display control unit 460 generates a guidance screen that guides the user through the procedure for using a tool or part, and displays the generated guidance screen on the display unit 310. At this time, the display control unit 460 may also place a viewpoint indicator (viewpoint indicator E shown in Figure 2) on the guidance screen that indicates the viewpoint of the worker estimated by the viewpoint estimation unit 420. Furthermore, the display control unit 460 changes the guidance screen displayed on the display unit 310 according to the operation determined by the operation determination unit 430. That is, the display control unit 460 changes the guidance screen displayed on the display unit 310 according to the output from the worker terminal 10. In addition, the display control unit 460 changes the guidance screen displayed on the display unit 310 according to the viewpoint estimation result by the viewpoint estimation unit 420.

[0056] For example, when the operation determination unit 430 determines that an operation to instruct a transition to the next page has been performed, and the procedure selection unit 450 reads out information for the next task, the display control unit 460 displays a guidance screen on the display unit 310 showing the information for the next task read out by the procedure selection unit 450. Also, when the operation determination unit 430 determines that an operation to instruct a transition to the previous page has been performed, and the procedure selection unit 450 reads out information for the previous task, the display control unit 460 displays a guidance screen on the display unit 310 showing the information for the previous task read out by the procedure selection unit 450.

[0057] Furthermore, based on the operation determination unit 430's determination that an operation to instruct enlarge display has been performed, the display control unit 460 enlarges the image C13 of the work target on the guidance screen. For example, if an operation to instruct enlarge display is performed when the worker's viewpoint display E is at the position shown in Figure 2, the display control unit 460 may enlarge the image C13 of the work target centered on the position of the worker's viewpoint, as shown in Figure 10. Similarly, based on the operation determination unit 430's determination that an operation to instruct reduced display has been performed in the state shown in Figure 10, the display control unit 460 may reduce the image C13 of the work target to the size shown in Figure 2.

[0058] <Operation> The configuration of the work support system according to one embodiment of the present invention has been described above. Next, the operation of the work support system according to one embodiment of the present invention will be summarized with reference to Figures 11 and 12.

[0059] Figure 11 is a sequence diagram showing the operation of a work support system according to one embodiment of the present invention. Figure 12 is a sequence diagram showing the operation of a work support system according to one embodiment of the present invention, following Figure 11. In Figures 11 and 12, steps related to initialization, abnormal cases, error handling, etc., are omitted.

[0060] While the worker is performing an action, the procedure selection unit 450 first obtains information about the target action from the procedure storage unit 440 (S1). Initially, the procedure selection unit 450 obtains information about the action with action number "1" from the procedure storage unit 440. The procedure selection unit 450 also outputs the magnification setting for image C3 to the display control unit 460 (S2). For example, the procedure selection unit 450 initially outputs 1x as the magnification setting to the display control unit 460.

[0061] Subsequently, the display control unit 460 generates a guidance screen based on the work information and magnification setting from the procedure selection unit 450, and displays the guidance screen on the display unit 310 of the display device 30 (S3). Then, the worker views the guidance screen (S4).

[0062] Subsequently, the gesture detection process shown in frame F1, the gaze detection process shown in frame F2, and the screen update process shown in frame F3 are performed. Frames F1 and F2 always run in parallel during the process. Frame F3 is executed depending on the results of frames F1 and F2.

[0063] Specifically, in frame F1, gesture detection processing is performed using the acceleration detection unit 110 of the worker terminal 10 worn by the worker. For example, the acceleration detection unit 110 detects the movement of the worker's arm (S5) and transmits acceleration data indicating the movement of the worker's arm to the motion estimation unit 410 (S6). Subsequently, the motion estimation unit 410 normalizes the acceleration data and estimates the worker's motion (S7). Then, the motion estimation unit 410 notifies the operation decision unit 430 of the estimation result of the worker's motion (S8).

[0064] Furthermore, in frame F2, gaze detection processing is performed using the gaze detection unit 210 to estimate the position of the worker's viewpoint during work. For example, the gaze detection unit 210 detects the worker's gaze (S9) and transmits the detection result of the worker's gaze to the viewpoint estimation unit 420 (S10). The viewpoint estimation unit 420 then estimates the position of the worker's viewpoint on the guidance screen displayed on the display unit 310 based on the detection result of the worker's gaze (S11). Subsequently, a viewpoint display is superimposed on the position of the viewpoint on the guidance screen (S12). Figure 11 shows an example in which the viewpoint display is transmitted from the viewpoint estimation unit 420 to the display unit 310, but the viewpoint display may also be transmitted to the display unit 310 via the display control unit 460. The viewpoint estimation unit 420 then notifies the operation decision unit 430 of the estimated viewpoint position (S13), and the operation decision unit 430 stores the viewpoint position (S14).

[0065] In frame F3, processing S15 to S18 is performed based on the assumption that the worker moved their left arm either up or down in frame F1 (condition I1), and processing S19 to S24 is performed based on the assumption that the worker twisted their left wrist either inward or outward in frame F1 (condition I2).

[0066] Specifically, if it is estimated that the worker moved their left arm up or down in frame F1, the operation determination unit 430 outputs the viewpoint position and a zoom in / out instruction (zoom in or zoom out instruction) to the display control unit 460 (S15). The display control unit 460 then changes the magnification of the guide screen image C3 around the viewpoint position and updates the image C3 on the guide screen displayed on the display unit 310 (S17). This allows the worker to view the image C3 with the changed magnification (S18).

[0067] Furthermore, if it is estimated that the worker twisted their left wrist inward or outward in frame F1, the operation determination unit 430 instructs the procedure selection unit 450 to transition to the next page or to the previous page depending on the direction of the twist (S19). Here, the operation determination unit 430 may also use the condition that the worker's viewpoint remains within the display P1 (return) or the display P2 (advance) for a predetermined time or longer as a condition for instructing the transition to the next page or to the previous page.

[0068] The procedure selection unit 450, in response to instructions from the operation determination unit 430, obtains information about the task on the destination page from the procedure storage unit 440 (S21), and outputs the task information to the display control unit 460 along with an instruction to reset the display magnification (S22). Subsequently, the display control unit 460 updates the guidance screen displayed on the display unit 310 based on the task information from the procedure selection unit 450 (S23), allowing the operator to view the updated guidance screen (S24). Note that resetting the display magnification in step S22 is not mandatory; the system may be configured to record the magnification level during the task and reproduce that level.

[0069] <Effects and Effects> According to the embodiment of the present invention described above, various effects and advantages can be obtained. For example, the work support system according to the present invention estimates the worker's movements based on acceleration data obtained from the acceleration detection unit 110 of the worker terminal 10 attached to the worker, and changes the guidance screen according to the estimated movements. With this configuration, the guidance screen can be operated even when the worker is holding parts or tools in both hands. Therefore, since the worker can operate the guidance screen without letting go of the parts or tools, a decrease in work efficiency can be suppressed when performing work while checking the work procedure.

[0070] Furthermore, the work support system according to one embodiment of the present invention changes the guidance screen in accordance with the estimated view of the worker, in addition to the worker's actions described above. With this configuration, it is possible to suppress the accidental change of the guidance screen due to actions taken by the worker for the purpose of work. In a system that changes the guidance screen based solely on the estimated view of the worker, if the worker is carefully looking at the guidance screen, the eye-tracking system may mark it as confirmed, causing the guidance screen to move to the next page, which may disrupt the work. In one embodiment of the present invention, since the guidance screen is controlled by a combination of the worker's actions and the estimated view of the worker, as described above, the above-mentioned disruption is less likely to occur.

[0071] With such a work support system, workers can, for example, enlarge or reduce image C3 on the guidance screen, or transition to the next or previous page, while minimizing a decrease in work efficiency and achieving their intended goals.

[0072] <Hardware Configuration> An embodiment of the present invention has been described above. The information processing, such as operation decision and display control described above, is realized through the cooperation of software and hardware. Below, an example of a hardware configuration that can be applied to the worker terminal 10, gaze detection device 20, display device 30, and work support device 40 will be described.

[0073] Figure 13 is a block diagram showing an example of a hardware configuration 90. The hardware configuration 90 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904. The hardware configuration 90 also includes a bridge 905, an external bus 906, an interface 907, an input device 908, a display device 909, an audio output device 910, a storage device (HDD) 911, a drive 912, and a network interface 915.

[0074] The CPU 901 functions as an arithmetic processing unit and control unit, controlling the overall operation according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and calculation parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and the like. Through the cooperation of the CPU 901, ROM 902, RAM 903 and software, functions such as the motion estimation unit 410, viewpoint estimation unit 420, operation decision unit 430, procedure selection unit 450, and display control unit 460 can be realized.

[0075] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0076] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, sensor, switch, and lever, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. By operating the input device 908, the user can input various types of data and instruct processing operations.

[0077] The display device 909 includes, for example, a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, and a lamp. The audio output device 910 includes an audio output device such as a speaker and headphones.

[0078] The storage device 911 is a data storage device configured as an example of a storage unit according to this embodiment. The storage device 911 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 911 is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or a memory having equivalent functionality. This storage device 911 drives the storage and stores programs executed by the CPU 901 and various data.

[0079] Drive 912 is a reader / writer for storage media and is either built into or external to the hardware configuration 90. Drive 912 reads information recorded on removable storage media 84, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to RAM 903 or storage device 911. Drive 912 can also write information to the removable storage media 84.

[0080] The network interface 915 is a communication interface composed of, for example, a communication device for connecting to a network. Furthermore, the network interface 915 may be a wireless LAN (Local Area Network) compatible communication device or a wired communication device for wired communication.

[0081] <Supplement> Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention.

[0082] For example, while the above description mainly focused on wrist movements that do not require large arm swings, the operator's movements for operating the guidance screen may be other movements. Other movements include, for example, moving the arm horizontally from the front to in front of the chest, and raising the arm horizontally from in front of the chest.

[0083] Furthermore, while the main examples of changing the guidance screen have been described as scaling image C3 and page transitions, if image C3 is a 3D image, the work support device 40 may rotate image C3 in three dimensions. In addition, the work support device 40 may place a help display area within the guidance screen, and a help request may be issued when the worker performs a predetermined action while looking at the help display area.

[0084] Furthermore, each process of the work support device 40 may be adjusted as appropriate. For example, the threshold values ​​for each axis of acceleration used by the motion estimation unit 410 to estimate whether the worker has moved their left arm up and down or twisted it may be adjusted according to the worker. Also, the time length used to determine whether the worker's viewpoint remained within a certain range (for example, the back indicator P1 or the forward indicator P2) may be adjusted according to the worker. These adjustment values ​​may be obtained by calibration and stored for each worker.

[0085] Furthermore, although the above describes an example in which an inertial sensor such as the acceleration detection unit 110 is used, other sensors may be used in addition to or instead of the inertial sensor. A sensor that detects contact with the other hand may be used in a terminal attached to the wrist or finger of one of the workers. With such a configuration, the worker can operate the guidance screen even if they are holding parts or tools in both hands.

[0086] Furthermore, although the above describes an example in which one embodiment of the present invention is applied mainly to assembly work, the present embodiment can also be applied to other types of work such as disassembly, inspection, and maintenance. In addition, in places such as factories where simple tasks are often repeated, the work support device 40 may display the first step page again after the last page of the work procedure. Moreover, if the ambient noise is not loud enough to interfere with speech recognition, commands using speech recognition may also be used.

[0087] Furthermore, each step in the processing of the work support system described herein does not necessarily have to be processed chronologically in the order described in the sequence diagram. For example, each step in the processing of the work support system may be processed in an order different from the order described in the sequence diagram, or may be processed in parallel.

[0088] Furthermore, it is possible to create computer programs that enable the CPU, ROM, RAM, and other hardware built into the worker terminal 10, gaze detection device 20, display device 30, and work support device 40 to perform functions equivalent to those of the aforementioned worker terminal 10, gaze detection device 20, display device 30, and work support device 40. A non-temporary storage medium on which such computer programs are stored is also provided.

[0089] Furthermore, the functions of the aforementioned motion estimation unit 410, viewpoint estimation unit 420, operation determination unit 430, procedure storage unit 440, procedure selection unit 450, and display control unit 460 may be implemented within a single housing or distributed across multiple housings. If the above functions are implemented within a single housing, the device having that housing may correspond to the work support device 40 or work support system. If the above functions are distributed across multiple housings, one or more combinations of the multiple devices having each housing may correspond to the work support device 40 or work support system. When the above functions are distributed across multiple housings, an interface for inputting and outputting information between the devices having each housing is also implemented. [Explanation of Symbols]

[0090] 10. Worker terminals 110 Acceleration detection unit 20 Eye-tracking device 210 Eye-tracking unit 30 Display device 310 Display section 40 Work support devices 410 Motion estimation section 420 Viewpoint Estimation Unit 430 Operation Determination Unit 440 Procedure storage unit 450 Procedure Selection Section 460 Display Control Unit

Claims

1. A display control unit that displays a guide screen on the display unit that guides the procedure for a task using a tool or part, A motion estimation unit that estimates the worker's movements based on the output from a terminal equipped with an inertial sensor attached to the worker, Equipped with, The display control unit is an information processing device that changes the guidance screen displayed on the display unit according to the operation estimated by the operation estimation unit.

2. The information processing apparatus according to claim 1, wherein the display control unit changes the content of the guidance screen to content indicating a subsequent operation based on the operation estimation unit's estimation that the operator's operation is a first operation.

3. The information processing apparatus according to claim 2, wherein the display control unit changes the content of the guidance screen to content indicating the previous operation based on the operation estimation unit's estimation that the operator's operation is a second operation.

4. The aforementioned information processing device is The system further includes a viewpoint estimation unit that estimates the viewpoint of the operator, which is the position the operator is looking at on the guidance screen. The information processing apparatus according to claim 3, wherein the display control unit changes the guidance screen to be displayed on the display unit in accordance with the viewpoint estimation result by the viewpoint estimation unit, in addition to the operation estimated by the operation estimation unit.

5. The information processing apparatus according to claim 4, wherein the display control unit changes the content of the guidance screen to content indicating a subsequent operation based on the fact that the operation estimation unit estimates the operator's operation to be a first operation and the position of the viewpoint on the guidance screen is estimated to be a first position.

6. The information processing apparatus according to claim 4, wherein the display control unit changes the content of the guidance screen to content indicating the previous operation based on the fact that the operation estimation unit estimates the operator's operation to be a second operation and the position of the viewpoint on the guidance screen is estimated to be a second position.

7. The information processing apparatus according to claim 4, wherein the display control unit, based on the estimation by the motion estimation unit that the worker's motion is a third motion, reduces the range on the guidance screen corresponding to the position of the viewpoint and displays it.

8. The information processing apparatus according to claim 4, wherein the display control unit expands and displays a range corresponding to the viewpoint position on the guidance screen based on the fact that the operation estimation unit has estimated the operator's operation to be a fourth operation.

9. An information processing system comprising a terminal having an inertial sensor to be worn by a worker, and an information processing device, The aforementioned information processing system is A display control unit that displays a guide screen on the display unit that guides the procedure for a task using a tool or part, An action estimation unit that estimates the worker's actions based on the output from the terminal, Equipped with, The display control unit is an information processing device that changes the guidance screen displayed on the display unit according to the operation estimated by the operation estimation unit.

10. To display a guide screen on the display unit that shows the procedure for using tools or parts, Based on the output from a terminal equipped with an inertial sensor attached to the worker, the worker's movements are estimated. The guidance screen displayed on the display unit is changed according to the estimated operation, A method performed by a computer, including [this].

Images