Work support system and method

Abstract

To enable both of viewpoint movement and work support by a hand motion, desired by a user.SOLUTION: An arithmetic device displays a three-dimensional work-target object in a three-dimensional coordinate space on a display device, and receives a motion signal that is a signal indicating a motion measured by any one of sensor units. When receiving the motion signal, the arithmetic device identifies the measured motion on the basis of the motion signal. When the identified motion is a motion relating to a space shift that is a shift of a user viewpoint in the three-dimensional space, the arithmetic device shifts the viewpoint in the three-dimensional space of the user according to the identified motion. When the identified motion is a hand motion relating to a work movement with respect to a work target, the arithmetic device outputs output information including information indicating the work movement according to the identified motion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention generally relates to techniques for work assistance. [Background technology]

[0002] Generally, an expert (an example of a supporter) supports a task performed by an unskilled person (an example of a support recipient). Techniques relating to task support are known, for example, from the techniques disclosed in Patent Documents 1 and 2.

[0003] According to Patent Document 1, an image of a work site where a worker (an example of a person being supported) is working is provided from a surrounding imaging device to a remote manager (an example of a supporter). The manager issues work instructions to the worker based on the image.

[0004] According to Patent Document 2, the hand motion of an expert (an example of a supporter) is superimposed on an image of the field of view of a worker (an example of a support recipient). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP2021-10101 [Patent Document 2] WO2015 / 060393 Summary of the Invention [Problem to be solved by the invention]

[0006] According to Patent Document 2, work assistance can be provided using hand motions. However, the image provided to the assistant is an image of the worker's field of vision, and therefore the assistant's field of vision is limited to the worker's field of vision.

[0007] On the other hand, according to Patent Document 1, the image of the surrounding image capturing device can be scrolled based on the manager's movements, that is, the manager's viewpoint can be moved. However, even in Patent Document 1, the field of view of the assistant is limited, specifically, it is limited to the capturing range of the surrounding image capturing device. Furthermore, the work assistance is not based on hand motion. [Means for solving the problem]

[0008] The work assistance system includes a display device, one or more sensor units that measure the motion of a user (typically an assistant), and a computing device. The computing device displays a three-dimensional work target object in a three-dimensional coordinate space on the display device and receives a motion signal that is a signal representing the motion measured by any of the sensor units. When the computing device receives the motion signal, it identifies the measured motion based on the motion signal. If the identified motion is a motion related to spatial movement, which is movement of the user's viewpoint in the three-dimensional coordinate space, the computing device moves the user's viewpoint in the three-dimensional coordinate space in accordance with the identified motion. If the identified motion is a hand motion related to a work action on a work target, the computing device outputs output information including information representing the work action in accordance with the identified motion. [Effects of the Invention]

[0009] According to the present invention, it is possible to both move the viewpoint as desired by the user and to provide work support using hand motions. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows an example of the overall configuration of a system according to a first embodiment. [Figure 2] 2 shows functional blocks of a computing device according to the first embodiment. [Figure 3] 10 shows functional blocks of a computing device according to a second embodiment. [Figure 4] 10 shows an example of a display mode of a hand object relating to spatial movement. [Figure 5] 10 shows an example of a display mode of a hand object relating to a work action. [Figure 6] 10 shows a modified example of the action identification information. [Figure 7] 10 shows an example of the overall configuration of a system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, an "interface apparatus" may refer to one or more interface devices. The one or more interface devices may be at least one of the following: An I / O interface device is one or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices for at least one of an I / O device and a remote display computer. The I / O interface device for the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. A communication interface apparatus that is one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0012] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0013] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and specifically may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0014] In the following description, the term "storage device" may refer to at least one of memory and persistent storage device.

[0015] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0016] In the following description, functions are sometimes described using the expression "yyy unit." However, the functions may be realized by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is realized by a program executed by a processor, the specified processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a non-transitory storage medium). The description of each function is merely an example; multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0017] In the following description, when elements of the same type are described without distinction, common reference symbols are used, and when elements of the same type are described with distinction, reference symbols are used.

[0018] Several embodiments will be described below. [First embodiment]

[0019] FIG. 1 shows an example of the overall configuration of a system according to the first embodiment.

[0020] The work assistance system 100 includes an input device 111, a display device 112, a motion sensor 102, and a computing device 104 communicatively connected thereto. A computer (for example, a laptop or tablet computer) including the input device 111, the display device 112, and the computing device 104 may also be employed.

[0021] The input device 111 may be a keyboard or a pointing device. The display device 112 may be a device having a display such as a flat panel display, a full-sphere display, or a half-sphere display.

[0022] The motion sensor 102 is an example of a sensor unit that measures the motion of the supporter 101 (an example of a user). The motion sensor 102 transmits (for example, wirelessly transmits) a motion signal that represents the measured motion.

[0023] The arithmetic device 104 includes an interface device 113, a storage device 114, and a processor 115 connected thereto. The interface device 113 communicates with the input device 111, the display device 112, and the motion sensor 102. The storage device 114 stores information illustrated in Fig. 2 and computer programs for implementing the functions illustrated in Fig. 2. The processor 115 executes the computer programs.

[0024] The arithmetic device 104 displays a three-dimensional work object 150 in a three-dimensional coordinate space on the display device 112. The work object 150 is an object that represents a work object such as a machine.

[0025] The computing device 104 receives motion signals representing motion measured by the motion sensor 102 (and the remote controller, described below).

[0026] When the calculation device 104 receives a motion signal, it identifies the measured motion based on the motion signal. If the identified motion is a motion related to spatial movement, it moves the viewpoint of the assistant 101 in the three-dimensional coordinate space in accordance with the identified motion. If the identified motion is a hand motion related to a work movement on a work object, it outputs output information including information representing the work movement in accordance with the identified motion.

[0027] The above-described computing device 104 allows both viewpoint movement as desired by the assistant and work assistance using hand motions. Specifically, the viewpoint of the assistant 101 can be moved within the range of the three-dimensional coordinate space in which the work target object 150 is placed. In addition, the motion related to the work action on the work target is a hand motion.

[0028] The computing device 104 may display a three-dimensional hand object 151 of one or both hands in addition to the work target object 150. The work target object 150 and the hand object 151 may both be in any format (for example, raster format or vector format).

[0029] Alternatively, one or more cameras may be provided at a site where the person being supported (e.g., an unskilled worker) and the object of work (e.g., a machine) for the person being supported are present, and a three-dimensional model of the site (e.g., a three-dimensional object of the site including the object of work) may be generated based on one or more images taken by the one or more cameras, and the three-dimensional model including the object of work 150 may be displayed on the display device 112. Alternatively, the object of work 150 may be an object created in advance using software such as CAD software.

[0030] In addition, in this embodiment, at least a motion sensor 102 is adopted as one or more sensor units, but instead of or in addition to the motion sensor 102, a sensor glove (for example, a glove-type sensor unit having a pressure sensor, a microphone, etc.) may be adopted, or a wearable sensor unit other than a sensor glove may be adopted, or other contact or non-contact sensor units may be adopted.

[0031] This embodiment will be described in detail below.

[0032] FIG. 2 shows functional blocks of the arithmetic unit 104.

[0033] The storage device 114 stores action identification information 231 , spatial movement information 232 , work action information 233 , and output information 234 .

[0034] The action identification information 231 includes information indicating the correspondence between a motion type and an action type. In this embodiment, an "action" is an action that is the purpose of the supporter 101, and a "motion" is an action that is a means for achieving the purpose of the supporter 101. According to the example shown in FIG. 2, the supporter 101 (e.g., an expert) makes a motion using the remote controller 103 to move through space. The supporter 101 also makes a hand motion for work (work assistance). The remote controller 103 has a sensor set (one or more sensors) and a wireless communication device. The remote controller 103 measures the motion through the sensor set and wirelessly transmits a motion signal of the measured motion through the wireless communication device.

[0035] The spatial movement information 232 includes information representing the calculated spatial movement. The spatial movement is the movement of the viewpoint of the supporter 101 in a three-dimensional coordinate space. In the spatial movement, one of the three-dimensional coordinate space and the viewpoint may be fixed while the other moves. The spatial movement information 232 may include information representing the start coordinates and end coordinates of the viewpoint and the speed of the viewpoint movement from the start coordinates to the end coordinates for each spatial movement.

[0036] The work movement information 233 includes information representing the calculated work movement. A work movement is a movement for working on a work object. The work movement information 233 may include, for each work movement, information representing the coordinates of three-dimensional hand objects representing the hands (for example, the position coordinates of each part of the hand object), the shape of the hand object, and animation of the hand object.

[0037] The output information 234 includes information to be output. For example, the output information 234 may include information representing one or more spatial movements, information representing one or more work actions, and information representing animation of a work target object, for a time period from a first time point to a second time point. For example, the output information 234 may be information serving as a work manual for a support recipient (e.g., an unskilled person).

[0038] The processor 115 executes a computer program to realize functions such as an input unit 210, a calculation unit 220, and an output unit 240. The input unit 210 includes a motion signal input unit 211 and a setting signal input unit 212. The calculation unit 220 includes an identification calculation unit 222, a spatial movement calculation unit 223, a task movement calculation unit 224, a movement identification setting unit 225, and an output information calculation unit 226. Below, the processing flow will be explained together with an explanation of each function.

[0039] The setting signal input unit 212 receives input of the action identification information from the input device 111, and inputs the received action identification information to the calculation unit 220. The action identification setting unit 225 receives input of the action identification information, and stores the action identification information 231 in the storage device 114.

[0040] The output information 234 includes information on the work target object and the hand object. The output information calculation unit 226 reads the information on the work target object from the output information 234, and the output unit 240 displays the work target object 150 on the display device 112.

[0041] The assistant 101 performs a motion for performing a task or moving through space on the task target object 150. Specifically, as described above, the motion is as follows. The assistant 101 performs a motion using the remote controller 103 for spatial movement (to change the viewpoint of the work target object 150). The remote controller 103 measures the motion and transmits a motion signal of the measured motion. The assistant 101 performs a hand motion for a task. The motion sensor 102 measures the motion and transmits a motion signal of the measured motion.

[0042] The motion signal input unit 211 receives motion signals from the motion sensor 102 or the remote controller 103 and inputs the received motion signals to the calculation unit 220. The identification calculation unit 222 receives the input motion signals and identifies the type of measured motion from the motion signals. The identification calculation unit 222 identifies the action type corresponding to the identified motion type from the action identification information 231. For example, if the identified motion type is "remote controller", the action type "spatial movement" is identified. If the identified motion type is "hand", the action type "work action" is identified.

[0043] If the identified action type is "spatial movement", the identification calculation unit 222 identifies the measured motion as a motion related to spatial movement. In this case, the spatial movement calculation unit 223 calculates the spatial movement. Specifically, the spatial movement calculation unit 223 calculates information representing the spatial movement of the supporter 101 by moving the viewpoint in the three-dimensional coordinate space in accordance with the identified motion, and includes the calculated information in the spatial movement information 232.

[0044] If the identified motion type is a "work motion," the identification calculation unit 222 identifies the measured motion as a motion related to a work motion for a work object. If the identified motion is a hand motion related to a work motion, the work motion calculation unit 224 calculates the work movement. Specifically, the work motion calculation unit 224 calculates information representing a work motion according to the identified motion, and includes the calculated information in the work motion information 233.

[0045] The output information calculation unit 226 reads information generated by the spatial movement calculation unit 224 or the work movement calculation unit 223 for the received motion signal from the spatial movement information 232 or the work movement information 233. The output information calculation unit 226 includes the read information in the output information 234, and calculates display information based on the read information. The output unit 240 reflects a display based on the calculated display information on the display device 112. This updates the display on the display device 112. Specifically, the viewpoint moves in the three-dimensional coordinate space, or an animation of the hand object 151 is displayed.

[0046] According to this embodiment, the motion related to the work movement is a hand motion, while the motion related to the spatial movement is a motion using the remote controller 103. This allows the arithmetic device 104 to accurately distinguish the type of movement that the support person 101 is aiming for. Note that the motion related to the spatial movement may be a motion of a part of the body other than the human hand (a part other than the part targeted by the motion related to the work movement) instead of a motion using the remote controller 103. For example, the motion related to the spatial movement may be a motion of moving the head in the direction of movement of the viewpoint. [Second embodiment]

[0047] The second embodiment will be described, focusing mainly on the differences from the first embodiment, and explanations of the commonalities with the first embodiment will be omitted or simplified.

[0048] FIG. 3 shows functional blocks of a computing device according to the second embodiment.

[0049] In the second embodiment, not only the motions related to the work actions but also the motions related to the spatial movement are hand motions. When not only the work actions but also the spatial movement are hand motions, operability (work support) for the supporter 101 is improved.

[0050] However, if hand motions are used not only for work movements but also for spatial movements, the motions may overlap, making it difficult to distinguish between the types of movements.

[0051] Therefore, the motions related to spatial movement are different types of hand motions from the hand motions related to work movements. This is expected to improve the ease of distinguishing between the types of movements. For example, the hand motions related to spatial movement may be any of the following: A hand motion that points your fingertip at the coordinates of the destination of the viewpoint. A hand motion that points a circle made with your fingers at the coordinates of the destination of the viewpoint. Hand motion that places a virtual marker at the coordinates of the destination of the viewpoint. A hand motion that moves (swipes) your finger in the opposite direction of the viewpoint. -Motion to move both hands in the direction of the viewpoint. A hand motion in which the hand is closed (no gaps between the fingers) and moves in the direction of the viewpoint.

[0052] In this embodiment, the hand motion related to spatial movement is a motion defined as a swimming motion. A swimming motion is intuitively easy to understand as a motion for spatial movement, and is unlikely to overlap with a motion for a work movement. Therefore, it is expected that both the operability and ease of distinguishing between types of movement for the support person 101 will be improved. According to the swimming motion, for example, the viewpoint may move in the direction of the stroke of the palm. In other words, moving the palm in the opposite direction to the direction of movement of the viewpoint may be a swimming motion.

[0053] Furthermore, in this embodiment, in order to improve the accuracy of distinguishing between action types, an action type "switching action" is provided in the action identification information 231. The "switching action" is a switch between spatial movement and work action. The action identification information 231 also includes information indicating whether the current action is spatial movement or work action. This information is, for example, a flag indicating whether or not an action of the type "spatial movement" and "work action" is the current action for each of the action types. "1" means that it is the current action, and "0" means that it is not the current action.

[0054] In the arithmetic device 104, the classification calculation unit 222 classifies the type of the measured motion and identifies the operation type corresponding to the classified motion type.

[0055] If the identified action type is a "switching action," the identification calculation unit 222 switches the current action from spatial movement to a work action, or from a work action to spatial movement. For example, if the spatial movement flag is "1" and the work action flag is "0," the identification calculation unit 222 updates the spatial movement flag to "0" and the work action flag to "1." On the other hand, if the spatial movement flag is "0" and the work action flag is "1," the identification calculation unit 222 updates the spatial movement flag to "1" and the work action flag to "0."

[0056] If the identified action type is not a switching action and the current action is a spatial movement, the identification calculation unit 222 identifies the measured motion as a motion related to a spatial movement.

[0057] If the identified action type is not a switching action and the current action is a work action, the identification calculation unit 222 identifies the measured motion as a motion related to a work action.

[0058] The motion type corresponding to the switching action may be an overlapping motion in which at least a part of one hand touches at least a part of the other hand, such as finger clasping. In other words, the motion related to the switching action is also a type of hand motion. This provides high operability. Furthermore, the overlapping motion is unlikely to overlap with either the hand motion of spatial movement or the hand motion of a work action. This is expected to prevent a decrease in the accuracy of distinguishing the action types.

[0059] The output unit 240 displays a hand object 151, which is a display object of the hand of the support person 101, in a three-dimensional coordinate space, but the display mode of the hand object for a hand motion related to spatial movement is different from the display mode of the hand object for a hand motion related to a work movement. This allows the support person 101 to understand at a glance which type of movement the support person 101's hand motion is recognized as by the calculation device 104. For example, the display mode of the hand object for a hand motion related to spatial movement may be a display of only the outline of the hand object (the hand object may not have a color), as exemplified in FIG. 4. The display mode of the hand object for a hand motion related to a work movement may be a filled-in display of the hand object, as exemplified in FIG. 5.

[0060] The information indicating the action type "switching action" and which of the action types "spatial movement" and "work action" is the current action may be provided in the first embodiment. For example, as shown in Fig. 6, the motion type corresponding to the switching action may be pressing a button on the remote controller 103 (an example of a predetermined motion using the remote controller 103). This allows the switching action to be accurately detected, and therefore the action type to be accurately distinguished. [Third embodiment]

[0061] The third embodiment will be described below, focusing mainly on the differences from the first embodiment, and explanations of the commonalities with the first embodiment will be omitted or simplified.

[0062] FIG. 7 shows an example of the overall configuration of a system according to the third embodiment.

[0063] The interface device 113 of the computing device 104 is communicatively connected to another computing device 704 via a communication network 750. The other computing device 704 has another display device 712. The supportee 701 views the display on the other display device 712.

[0064] The output unit 240 of the arithmetic device 104 outputs the output information 234 to another arithmetic device 704 via a communication network 750. The other arithmetic device 704 displays the work target object 150 and the work action for the work target (for example, animation of the hand object 151) on another display device 712 based on the received output information. In this way, work support can be provided to the support recipient 701.

[0065] Although several embodiments have been described above, these are merely examples for explaining the present invention, and it is not intended that the scope of the present invention be limited to these embodiments. The present invention can be implemented in various other forms. [Explanation of symbols]

[0066] 100: Work support system

Claims

1. A display device; One or more sensor units that measure the motion of a user who is a supporter; a computing device that displays a three-dimensional work object in a three-dimensional coordinate space on the display device and receives a motion signal that is a signal representing a motion measured by any one of the sensor units; Equipped with When the computing device receives the motion signal, (A) identifying the measured motion based on the motion signal; (B) if the identified motion is a motion related to spatial movement, which is a movement of a user's viewpoint in the three-dimensional coordinate space, moving the user's viewpoint in the three-dimensional coordinate space in accordance with the identified motion; (C) if the identified motion is a hand motion related to a work action on a work object, output information including information representing the work action according to the identified motion; The arithmetic device, in (A), (a1) identifying a type of the measured motion; (a2) identifying an action type corresponding to the identified motion type; (a3) if the identified action type is a switching action that is a switching between spatial movement and a work action, switching the current action from spatial movement to a work action or from a work action to spatial movement; (a4) if the identified action type is not a switching action and the current action is a spatial movement, identifying the measured motion as a motion related to a spatial movement; (a5) if the identified action type is not a switching action and the current action is a work action, identify the measured motion as a motion related to the work action; Work support system.

2. The motion related to spatial movement is hand motion. The work support system according to claim 1 .

3. Motions related to spatial movement are different types of hand motions from hand motions related to work operations. The work support system according to claim 1 .

4. The hand motion related to spatial movement is defined as a swimming motion. The work support system according to claim 3 .

5. The motion type corresponding to the switching operation is overlapping, in which at least a part of one hand touches at least a part of the other hand. The work support system according to claim 1 .

6. the computing device displays a hand object, which is a display object of the user's hand, in the three-dimensional coordinate space; the computing device differentiates the display mode of the hand object when the hand motion is related to spatial movement from the display mode of the hand object when the hand motion is related to a work operation; The work support system according to claim 1 .

7. the plurality of sensor units include a remote controller; The motion type corresponding to the switching operation is a predetermined motion using the remote controller. The work support system according to claim 1 .

8. the arithmetic device outputs the output information to another arithmetic device connected to the arithmetic device via a communication network; the other arithmetic device displays the work target object and the work action for the work target on another display device based on the output information; The work support system according to claim 1 .

9. A task assistance method performed by a computing device, comprising: a display step of displaying a three-dimensional object to be worked on in a three-dimensional coordinate space on a display device; a receiving step of receiving a motion signal that is a signal representing motion measured by any one of the sensor units for the user who is the assistant; an identification step of identifying the measured motion based on the motion signal; a moving step of moving the user's viewpoint in the three-dimensional coordinate space in accordance with the identified motion, when the identified motion is a motion related to spatial movement that is a movement of the user's viewpoint in the three-dimensional coordinate space; an output step of outputting output information including information representing a work movement according to the identified motion when the identified motion is a hand motion related to a work movement on a work object; and In the identifying step, Identifying a type of the measured motion; Identifying an action type corresponding to the identified motion type; When the identified action type is a switching action that is a switching between spatial movement and a work action, the current action is switched from spatial movement to a work action or from a work action to spatial movement; If the identified action type is not a switching action and the current action is a spatial movement, the measured motion is identified as a motion related to a spatial movement; If the identified action type is not a switching action and the current action is a work action, the measured motion is identified as a motion related to the work action. Work support method.

Images