Indication system and method for creating indication content

Abstract

To provide an indication content on the basis of indication items.SOLUTION: The indication system is for allowing a supervisor to indicate a deficiency in an operation by a trainee and, includes: an operation collection unit for collecting motion data of the trainee; an operation image providing unit for visually providing the motion data to the supervisor; an indication item collection unit for collecting indication items that are indications from the supervisor to the trainee; an indication content creation unit for creating an indication content, the indication content being data to be presented to the trainee based on the indication items; and an indication content display unit for presenting the indication content to the trainee.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a suggestion system and a suggestion content creation method. [Background technology]

[0002] Ideally, work with predefined procedures should be explained to workers in an easy-to-understand manner using an operation manual or the like. However, there are cases where the explanation is insufficient or the work is complicated, and prior explanation alone is not sufficient. Patent Document 1 discloses a method for maintaining equipment, which includes: acquiring an engineering model of an equipment installation location; creating and displaying a simulation scene based on the engineering model; and assisting a maintenance worker in identifying a target equipment in the simulation scene; and displaying, at the site where the target equipment is located, information related to the target equipment based on the simulation scene and assisting the maintenance worker in maintaining the target equipment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 136029 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 cannot generate pointed out content based on pointed out items. [Means for solving the problem]

[0005] A pointing out system according to a first aspect of the present invention is a pointing out system in which a supervisor points out deficiencies in work performed by a trainee, and includes: an action collection unit that collects action data of the trainee; an action video provision unit that visually provides the action data to the supervisor; a pointing out item collection unit that collects points of advice given to the trainee by the supervisor; a pointing out content creation unit that creates pointing out content, which is data to be presented to the trainee, based on the points of advice; and a pointing out content display unit that presents the pointing out content to the trainee. A method for creating suggested content according to a second aspect of the present invention is a method for creating suggested content executed by a suggested item reflection device to which suggested items pointed out by a supervisor with respect to a trainee's movement are input, the suggested items including a designation of an optimal state, which is an appropriate position or posture, for a suggested part, which is a body part of the trainee or a member used by the trainee, the method including a suggested item collection step for acquiring the suggested items, and a suggested content creation step for converting the optimal state into a three-dimensional optimal state expressed as a three-dimensional position or posture, and combining it with the suggested part to create suggested content. [Effects of the Invention]

[0006] According to the present invention, it is possible to generate pointed out content based on pointed out matters. [Brief explanation of the drawings]

[0007] [Figure 1] Overall configuration of the reporting system [Figure 2] Hardware configuration of the trainee device [Figure 3] Hardware configuration diagram of the device that reflects findings [Figure 4] Supervisor device hardware configuration diagram [Figure 5] Functional configuration diagram of trainee device [Figure 6] FIG. 10 is a diagram showing the display of task content on the trainee display device. [Figure 7] Functional configuration diagram of the device that reflects findings [Figure 8] Supervisor device functional configuration diagram [Figure 9] FIG. 10 shows the reproduction of trainee's movements on the supervisor display. [Figure 10] A diagram showing an example of a supervisor entering comments using pen input. [Figure 11] Flowchart showing the motion recording process of the trainee device [Figure 12] 1 is a flowchart showing the process performed by the supervisor device. [Figure 13] Flowchart showing the process of the device for reflecting pointed out items [Figure 14] Figure showing the pointed content confirmation screen [Figure 15] Flowchart showing the operation of the indicated content display program [Figure 16] A diagram showing the screen display of pointed out content from a third-person perspective [Figure 17] A diagram showing the screen display of pointed content from a first-person perspective DETAILED DESCRIPTION OF THE INVENTION

[0008] --Embodiment-- Hereinafter, embodiments of a suggestion system and a suggested content creating method will be described with reference to FIGS.

[0009] (Overall composition) FIG. 1 is an overall configuration diagram of a pointing out system 1. The pointing out system 1 includes a trainee device 10, a pointing out item reflecting device 20, and a supervisor device 40, which are connected via a network 2. A trainee action acquisition device 100 is connected to the trainee device 10. The pointing out system 1 is used by one trainee and one or more supervisors. If there are multiple supervisors, the pointing out system 1 includes multiple supervisor devices 40.

[0010] The trainee device 10 gives instructions to the trainee on movements and presents comments from the supervisor. The trainee action acquisition device 100 is a device for recording the trainee's movements, such as a combination of multiple inertial sensors or a tracker and a camera. Data acquired by the trainee action acquisition device 100 (hereinafter referred to as "sensor data") is transmitted to the trainee device 10. The supervisor device 40 receives the data recording the trainee's movements from the trainee device 10 and presents it to the supervisor. The supervisor device 40 records comments made by the supervisor on the trainee's movements and transmits them to the comment reflection device 20. The comment reflection device 20 aggregates the comments made by the supervisor and transmits them to the trainee device 10.

[0011] (Hardware configuration) 2 is a hardware configuration diagram of the trainee device 10. The trainee device 10 includes a first CPU 101, a first ROM 102, a first RAM 103, a trainee display device 104, a trainee input device 105, and a first communication unit 106. The first CPU 101 is a central processing unit. The first ROM 102 is a non-volatile storage device that stores programs executed by the trainee device 10. The first RAM 103 is a main storage device that is capable of high-speed reading and writing. The first CPU 101 loads the programs stored in the first ROM 102 into the first RAM 103 and executes them.

[0012] The trainee display device 104 is a liquid crystal display or a head-mounted display that displays images. The trainee input device 105 is an input device operated by the trainee, such as a mouse or keyboard. The first communication unit 106 is a communication device, such as a network interface card, that allows the trainee device 10 to communicate with other devices. However, the hardware for communicating with the suggested item reflection device 20 may be different from the hardware for communicating with the suggested item reflection device 20.

[0013] 3 is a hardware configuration diagram of the indicated item reflection device 20. The indicated item reflection device 20 includes a second CPU 201, a second ROM 202, a second RAM 203, and a second communication unit 206. The second CPU 201 is a central processing unit. The second ROM 202 is a non-volatile storage device that stores programs executed by the indicated item reflection device 20. The second RAM 203 is a main storage device that is capable of high-speed reading and writing. The first CPU 101 loads the programs stored in the second ROM 202 into the second RAM 203 and executes them. The second communication unit 206 is a communication device, such as a network interface card, that allows the indicated item reflection device 20 to communicate with other devices.

[0014] 4 is a hardware configuration diagram of the supervisor device 40. The supervisor device 40 includes a third CPU 401, a third ROM 402, a third RAM 403, a supervisor display device 404, a supervisor input device 405, and a third communication unit 406. The third CPU 401 is a central processing unit. The third ROM 402 is a non-volatile storage device that stores programs executed by the supervisor device 40. The third RAM 403 is a main storage device that is capable of high-speed reading and writing. The third CPU 401 loads the programs stored in the third ROM 402 into the third RAM 403 and executes them.

[0015] The supervisor display device 404 is a liquid crystal display or touch panel that displays images. The supervisor input device 405 is an input device operated by the trainee, such as a mouse, keyboard, or pen tablet. The third communication unit 406 is a communication device, such as a network interface card, that allows the supervisor device 40 to communicate with other devices.

[0016] (Functional configuration) Fig. 5 is a functional configuration diagram of the trainee device 10. Fig. 5 also shows data stored in the trainee device 10. The trainee device 10 includes an activity content display program 11, a trainee motion acquisition program 12, a trainee motion analysis program 13, a pointed content display program 14, an activity content database 81, a trainee motion database 82, and a pointed content database 83. The activity content database 81 stores pre-created activity instruction data for the trainee. The activity instructions may be in the form of text, images, audio, or video.

[0017] The trainee motion database 82 stores data on the trainee's motion (hereinafter referred to as "motion data") recorded by the trainee motion analysis program 13. The suggested content database 83 stores suggested content created by the suggested item reflecting device 20. The task content display program 11 displays the content stored in the task content database 81 on the trainee display device 104 and presents it to the trainee.

[0018] FIG. 6 is a diagram showing the display of task content on the trainee display device 104 by the task content display program 11. FIG. 6 shows a human figure 901 representing the trainee and a task object 902. The human figure 901 is a human model created in advance, and the task object 902 is a model of an object created in advance. An action instruction statement 903 for the trainee is shown at the top of FIG. 6, and the trainee performs the action described in this action instruction statement 903. The task content display program 11 may change the viewpoint of the human figure 901 and task object 902 displayed on the trainee display device 104 in response to an input operation by the trainee, for example, a mouse drag operation. Returning to FIG. 5, the explanation will continue.

[0019] The trainee motion acquisition program 12 acquires sensor data output by the trainee motion acquisition device 100. Hereinafter, the trainee motion acquisition program 12 will also be referred to as the "motion collection unit." The trainee motion analysis program 13 analyzes the movement of each joint of the trainee using the sensor data acquired by the trainee motion acquisition program 12 to obtain movement data. The trainee motion analysis program 13 may record the movement data resulting from the analysis in a trainee motion database 82, or may output the movement data to a pointed content display program 14. For example, the trainee motion acquisition program 12 records the three-dimensional positions and postures of the trainee's hands, feet, body, and head at each time point together with time data as movement data in the trainee motion database 82. The time data may be data indicating absolute time such as UTC (Coordinated Universal Time) or JST (Japan Standard Time), or may be relative data indicating the elapsed time from the start of recording. The pointed content display program 14 receives the pointed content database 83 created by the pointed matter reflection device 20 and displays it on the trainee display device 104.

[0020] Fig. 7 is a functional configuration diagram of the indicated item reflecting device 20. Fig. 7 also shows data stored in the indicated item reflecting device 20. The indicated item reflecting device 20 includes a work information collecting program 21, an indicated item collecting program 22, an indicated content creating program 23, an approval request program 24, an indicated content sending program 25, an indicated content database 83, and an indicated item database 84.

[0021] The work information collection program 21 acquires data stored in a trainee action database 82 from the trainee device 10. The suggested item collection program 22 acquires data stored in a suggested item database 84 from the supervisor device 40. Hereinafter, the suggested item collection program 22 will also be referred to as a "suggested item collection unit." The suggested content creation program 23 references the suggested item database 84 and creates a suggested content database 83 that includes one or more suggested contents. Hereinafter, the suggested content creation program 23 will also be referred to as a "suggested content creation unit."

[0022] The suggestions stored in the suggestion database 84 include specifications for the optimum state, which is the appropriate position or posture, of a part of the trainee's body (hereinafter referred to as the "suggested part"). However, since the suggested parts and the optimum state are often not sufficiently specified in the suggested parts, the suggested content creation program 23 specifies them. The suggested content creation program 23 specifies the optimum state as a three-dimensional position or posture. For example, the suggested content creation program 23 can specify the current position of the suggested part of the trainee from the movement data, and by combining this with the amount of correction included in the suggested parts, such as an utterance such as "move the thumb of your right hand 10 cm to the left," it can identify the optimum state by performing a numerical calculation with the three-dimensional position of the right thumb in the movement data. Also, for example, based on a pen input such as "your arm was bent" included in the suggested parts, it can identify the optimum state of "straighten your arm."

[0023] The approval request program 24 transmits the indicated content created by the indicated content creation program 23 to the supervisor device 40 and requests the supervisor to approve the indicated content. The indicated content transmission program 25 transmits the indicated content created by the indicated content creation program 23 and approved by the supervisor to the trainee device 10.

[0024] Fig. 8 is a functional configuration diagram of the supervisor device 40. However, Fig. 8 also shows data stored in the supervisor device 40. The supervisor device 40 includes a comment voice collection program 41, a comment pen input collection program 42, a comment action collection program 43, a trainee action reproduction program 44, a comment content approval response program 45, a trainee action database 82, and a comment item database 84. The trainee action database 82 is the same as the trainee action database 82 in the trainer device 10, and is acquired from the trainer device 10. The comment item database 84 is a collection of supervisor comments collected by the comment voice collection program 41, the comment pen input collection program 42, and the comment action collection program 43.

[0025] The suggestion voice collection program 41 collects the supervisor's voice regarding suggestions to the trainee and stores it in the suggestion database 84. The suggestion voice collection program 41 collects the supervisor's voice using a microphone provided in the supervisor device 40 and obtains a character string by performing voice recognition processing. The suggestion pen input collection program 42 collects the supervisor's pen input data regarding suggestions to the trainee. The suggestion pen input collection program 42 collects the supervisor's pen input data using a touch panel or pen tablet provided in the supervisor device 40.

[0026] The pointed-out action collection program 43 collects motion data of the supervisor relating to the points made to the trainee. The pointed-out action collection program 43 collects motion data of the supervisor using at least one of a camera that films the supervisor and an inertial sensor worn by the supervisor. Each collection program links the collected data with the time when the supervisor started the action and records it in the pointed-out item database 84. However, the "time" referred to here does not refer to the actual time, but rather to the time data recorded in the trainee action database 82. Below, the pointed-out voice collection program 41, the pointed-out pen input collection program 42, and the pointed-out action collection program 43 will each be referred to as an "pointed-out item collection unit."

[0027] The trainee motion reproducing program 44 refers to the trainee motion database 82, reproduces the trainee's motion in a virtual space using a three-dimensional model, and displays it on the supervisor display device 404. Specifically, the trainee motion reproducing program 44 first reads the position and posture data of the trainee's hands, feet, body, and head stored in the trainee motion database 82. Next, the trainee motion reproducing program 44 changes the position and posture of the three-dimensional model of the trainee in the virtual space based on the read data. Furthermore, the trainee motion reproducing program 44 creates an image obtained by photographing the virtual space from a preset camera viewpoint, i.e., performs rendering, and displays the virtual space including the 3D model of the trainee on the supervisor display device 404.

[0028] At this time, the supervisor may be able to freely change the camera viewpoint, or multiple viewpoints may be displayed simultaneously on the supervisor display device 404. The trainee motion reproducing program 44 successively reads data from the trainee motion database 82, reproduces the trainee's movements in chronological order in the virtual space, and displays them on the supervisor display device 404. The playback speed displayed by the trainee motion reproducing program 44 on the supervisor display device 404 may be the same speed as the original, or the supervisor may set the playback speed. Furthermore, the trainee motion reproducing program 44 may provide a user interface that enables the supervisor to instruct pausing and resuming playback. Hereinafter, the trainee motion reproducing program 44 will also be referred to as the "motion image providing unit."

[0029] When the pointed content approval response program 45 receives the pointed content created by the pointed item reflection device 20, it displays the following on the supervisor display device 404. That is, the pointed content approval response program 45 displays the pointed content, an inquiry message asking whether the pointed content is acceptable, and buttons showing the options available to the supervisor. The pointed content approval response program 45 transmits the results of the supervisor's selection of the options to the pointed item reflection device 20. A specific example will be described in detail later with reference to FIG. 14.

[0030] Figure 9 is a diagram showing the reproduction of the trainee's movements on the supervisor display device 404. The trainee movement reproduction program 44 references the trainee movement database 82, reproduces the trainee's movements in a virtual space, and displays them on the supervisor display device 404. Note that while Figure 9 displays from the same viewpoint as Figure 6 displayed by the task content display program 11, the two viewpoints may be different. Also displayed in Figure 9 are a voice input button 911, a pen input button 912, and a movement input button 913.

[0031] The supervisor presses one of the buttons and then makes the corresponding input. The supervisor presses the voice input button 911 when inputting instructions to the trainee by speech, i.e., verbal explanation. The supervisor presses the pen input button 912 when inputting instructions to the trainee by writing on the tablet terminal. The supervisor presses the action input button 913 when inputting instructions to the trainee by gesture. Note that the supervisor may press two or more buttons to input instructions using multiple methods. However, instead of displaying each button shown in Figure 9, each program may be constantly running, and each program may start recording in response to the supervisor's actions.

[0032] 10 is a diagram showing an example in which a supervisor inputs a comment using a pen. The supervisor inputs a comment target location 921 and comment content 922.

[0033] 11 is a flowchart showing the motion recording process executed by the task content display program 11, trainee motion acquisition program 12, and trainee motion analysis program 13 of the trainee device 10. First, in step S301, the task content display program 11 of the trainee device 10 displays task instructions on the trainee display device 104 of the trainee device 10. In the following step S302, the program starts recording the task motions of the trainee, specifically, the trainee motion acquisition program 12 acquires sensor data from the trainee motion acquisition device 100, the trainee motion analysis program 13 analyzes the sensor data, and records the analysis results in the trainee motion database 82, and then proceeds to step S303. In step S303, the trainee motion acquisition program 12 determines whether or not an instruction to stop recording has been received. If the trainee motion acquisition program 12 determines that no instruction to stop recording has been received, it remains in step S303, and if it determines that an instruction to stop recording has been received, it proceeds to step S304.

[0034] In step S304, the trainee motion acquisition program 12 ends recording of the work motions of the trainee and ends the processing shown in Fig. 11. That is, from the time step S302 is executed until step S304 is executed, the trainee motion acquisition program 12 continues to acquire sensor data from the trainee motion acquisition device 100, the trainee motion analysis program 13 continues to analyze the sensor data, and the trainee motion database 82 continues to record the analysis results. Note that the instruction to stop recording in step S303 may be replaced by the end of playback of the content displayed in step S301, or may be a result of voice recognition of an utterance of "stop recording" by the trainee, or may be a result of the trainee pressing a button (not shown).

[0035] 12 is a flowchart showing the processing executed by the supervisor device 40. First, in step S311, the supervisor device 40 acquires the trainee action database 82 from the trainee device 10 and starts displaying the trainee's actions using the trainee action reproduction program 44. In the following step S312, the supervisor device 40 starts each collection program, specifically the pointed-out voice collection program 41, the pointed-out pen input collection program 42, and the pointed-out action collection program 43, and proceeds to step S313. In step S313, the supervisor device 40 determines whether the trainee's action started in step S311 has ended. If the supervisor device 40 determines that the trainee's action has ended, it ends the processing shown in FIG. 12; if it determines that the trainee's action has not ended, it proceeds to step S314.

[0036] In step S314, the supervisor device 40 determines whether or not the supervisor has input any suggestions. If the supervisor device 40 determines that the supervisor has input any suggestions, the process proceeds to step S315; if the supervisor device 40 determines that the supervisor has not input any suggestions, the process returns to step S313. In step S315, the collection program corresponding to the input suggestions saves the input suggestions and transmits them to the suggested item reflection device 20, and the process returns to step S313. Note that the process shown in step S315 may be executed only when a positive determination is made in step S313. In this case, suggestions are not transmitted to the suggested item reflection device 20 every time they are input, but are transmitted all at once to the suggested item reflection device 20 at the end.

[0037] 13 is a flowchart showing the processing of the suggested item reflecting device 20. First, in step S321, the task information collection program 21 acquires the trainee action database 82 from the trainee device 10 and proceeds to step S322. In step S322, the suggested item collection program 22 acquires one suggested item from the suggested item database 84. In the following step S323, the suggested content creation program 23 classifies the suggested item acquired in step S322 into one of three categories: voice, pen input, and action. In the following step S324, the suggested content creation program 23 determines whether all suggested items have been acquired. If the suggested content creation program 23 determines that all suggested items have been acquired, it proceeds to step S325, and if it determines that there are still suggested items that have not been acquired, it returns to step S322.

[0038] In step S325, the pointed-out content creation program 23 organizes the pointed-out items and determines the priority for reflecting them in the content. For example, the priority may be set higher if the pointed-out items contain words from a predefined glossary, such as "danger," "top priority," or "important." Alternatively, the longer the supervisor points out the problem, the higher the priority may be. Furthermore, the supervisor's emotions may be estimated from the supervisor's comments, and a specific emotion, such as anger, may be given a higher priority. Note that emotion estimation can be achieved using a known algorithm, a large-scale language model, or the like.

[0039] In the next step S326, the suggested content creation program 23 creates suggested content. Specifically, the suggested content creation program 23 determines the paste position of the suggested content for the suggested content whose priority determined in step S325 is equal to or higher than a predetermined value. The paste position of the suggested content is a combination of the position on the time axis where the suggested content is displayed and the position on the screen. The position on the time axis may be, for example, the time when the supervisor starts inputting the suggested content, or if the suggested content covers the entire trainee's movements, all time may be the target. If the supervisor specifies a specific position using pen input or if the position can be identified from the supervisor's speech, the content is displayed at the specified or identified location. In particular, if the supervisor specifies a position, a marker indicating the specified position is included in the suggested content.

[0040] In the following step S327, the approval request program 24 transmits the content created in step S326 to the supervisor device 40, and then proceeds to step S328. In the following step S328, the approval request program 24 determines whether the supervisor has approved it, and if it determines that the supervisor has approved it, proceeds to step S329, and if it determines that the supervisor has not approved it, returns to step S322. In this step, the pointed content creation program 23 can determine the supervisor's approval, for example, based on whether the response from the supervisor device 40 is YES or NO. In step S329, the pointed content transmission program 25 transmits the pointed content approved by the supervisor to the trainee device 10, and then the processing shown in FIG. 13 ends.

[0041] 14 is a diagram showing a pointed content confirmation screen that the pointed content approval program 45 displays on the supervisor display device 404. The pointed content is a combination of a pointed portion 931 and pointed content 932. A yes button 934 and a no button 935 are displayed together with a query 933 asking the supervisor whether the pointed content is acceptable or not. The supervisor's selection of the yes button 934 or the no button 935 is transmitted to the pointed item reflecting device 20. That is, when the supervisor selects the yes button 934, the pointed item reflecting device 20 makes a yes determination in step S327, and when the supervisor selects the no button 935, the pointed item reflecting device 20 makes a no determination in step S327.

[0042] Fig. 15 is a flowchart showing the operation of the suggested content display program 14. Fig. 15 particularly shows the processing when the suggested content includes a position specification and the trainee uses a head-mounted display. The suggested content display program 14 displays the suggested content stored in the task content database 81. The suggested content display program 14 repeats the processing shown in Fig. 15 for each suggested item included in the suggested content database.

[0043] First, in step S331, the pointed content display program 14 acquires sensor data from the trainee motion acquisition device 100, and the trainee motion analysis program 13 analyzes the trainee's motion. The trainee's posture is identified from the analyzed trainee's motion, so the pointed content display program 14 can identify the trainee's viewpoint and field of view in real space. In the following step S332, the pointed content display program 14 determines whether the pointed position in the pointed content is within the trainee's field of view. As the trainee's field of view has been identified in step S331 as described above, it is easy to determine whether the pointed position is within the field of view. If the pointed content display program 14 determines that the pointed position is within the trainee's field of view, it proceeds to step S333, and if it determines that the pointed position is not within the trainee's field of view, it proceeds to step S334.

[0044] In step S333, the pointed out content display program 14 displays the pointed out position on the trainee display device 104, which is a head-mounted display, and then proceeds to step S335. In step S334, the pointed out content display program 14 displays the pointed out items and the current trainee's movements from a third-person perspective on the trainee display device 104, and then proceeds to step S335.

[0045] In step S335, the pointed content display program 14 determines whether the pointed position and the trainee's working position match. In this step, for example, when the arm position is specified in the pointed content, the pointed content display program 14 determines whether the worker's arm is at the specified position. If the pointed content display program 14 determines that the two match, it proceeds to step S336, and if it determines that the two do not match, it returns to step S331. In step S336, the pointed content display program 14 deletes the pointed display on the trainee display device 104 and ends the processing shown in FIG. 15.

[0046] Fig. 16 is a diagram showing a screen display of pointed out content from a third-person perspective, and Fig. 17 is a diagram showing a screen display of pointed out content from a first-person perspective. Fig. 16 and Fig. 17 show an example in which the pointed out content shown in Fig. 14 is judged by the supervisor to be acceptable, and similarly to Fig. 14, pointed out portion 941 and pointed out content 942 are shown.

[0047] However, while Fig. 14 shows the trainee's past movements created by the trainee movement reproduction program 44, Figs. 5 and 6 reflect the trainee's real-time movements, as will be explained below. The image of trainee 946 in Fig. 16 is an image obtained by rendering a model of the trainee reproduced in virtual space based on the movements of each joint of the trainee 946 analyzed by the trainer movement analysis program 13. Arm 951 in Fig. 17 is the actual trainee's arm, and is the arm seen through a transparent screen or the arm photographed by an external camera mounted on the HMD and displayed on the screen. When the trainee moves his or her arm and the correction target part 941 overlaps with the trainee's arm, the correction display is deleted.

[0048] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The suggestion system 1 creates suggestion content that allows a supervisor to point out deficiencies in the work of a trainee. The suggestion system 1 includes a trainee action acquisition program 12 (i.e., action collection unit) that collects trainee action data, and a trainee action reproduction program 44 (i.e., action video provision unit) that visually provides the action data to the supervisor. The suggestion system 1 also includes a suggestion voice collection program 41, a suggestion pen input collection program 42, a suggestion action collection program 43, and a suggestion collection program 22 (i.e., suggestion collection unit) that collect suggestions from the supervisor to the trainee. The suggestion system 1 also includes a suggestion content creation program 23 (i.e., suggestion content creation unit) that creates suggestion content, which is data to be presented to the trainee based on the suggestions, and a suggestion content display program 14 (i.e., suggestion content display unit) that presents the suggestion content to the trainee.

[0049] (2) The movement data includes at least one of data that quantifies the movement of the trainee's joints, data that quantifies the positions of the trainee's body parts, and data that quantifies the positions of the parts of the members used by the trainee.

[0050] (3) The trainee motion reproduction program 44 reproduces the trainee's motion in a three-dimensional model based on the motion data, so that the supervisor can check the trainee's motion in detail.

[0051] (4) The findings collection unit collects at least one of the supervisor's speech, the supervisor's pen input, and the supervisor's gesture as findings.

[0052] (5) The indication content includes a designation of an optimal state, which is an appropriate position or posture, for an indication part, which is a part of the trainee's body or a part of a member used by the trainee. The indication system 1 includes a trainee display device 104, which is a display device that can be used by the trainee. The indication content display program 14 displays an indication display indicating the optimal state of the indication part on the trainee display device.

[0053] (6) When the indicated part matches the optimum state, the indication content display program 14 deletes the indication, so that the trainee can recognize that the indicated part matches the optimum state.

[0054] (7) The method for creating suggested content executed by the suggested item reflecting device 20, to which the supervisor inputs suggested items regarding the trainee's movements, includes the following suggested item collecting step and suggested content creating step. The suggested items include a specification of an optimal state, which is an appropriate position or posture, for a suggested part, which is a body part of the trainee or a member used by the trainee. In the suggested item collecting step, the suggested item reflecting device 20 acquires suggested items from the supervisor device 40. In the suggested content creating step, the suggested item reflecting device 20 converts the optimal state into a three-dimensional optimal state expressed as a three-dimensional position or posture, and combines it with the suggested part to create suggested content. This makes it possible to create suggested content that is the supervisor's suggestions to the trainee.

[0055] (8) The indicated item reflecting device 20 includes a indicated content transmitting step (step S329 in FIG. 13) of transmitting the indicated content to the trainee device 10 used by the trainee.

[0056] (9) In the step of collecting findings, multiple findings are collected. In the step of creating findings, a priority is determined for each finding, and findings with a priority higher than a predetermined threshold, or a predetermined number of findings with the highest priority, are included in the findings content (step S326 in FIG. 13).

[0057] (Variation 1) The pointed out content creation program 23 created pointed out contents for the pointed out items with priorities equal to or higher than a predetermined value in step S326 of Fig. 13. However, the pointed out content creation program 23 may create pointed out contents for a predetermined number of pointed out items starting from the highest priority, or may create pointed out contents for all pointed out items.

[0058] (Variation 2) The suggestion system 1 in the above-described embodiment includes a trainer device 10, a suggested item reflecting device 20, and a supervisor device 40. However, it is not an essential component that the three devices, the trainer device 10, the suggested item reflecting device 20, and the supervisor device 40, exist independently, and two or more devices may be configured integrally. For example, the suggested item reflecting device 20 and the supervisor device 40 may be configured integrally, or the three devices may be configured integrally. Furthermore, the suggestion system 1 may be configured integrally with one or more additional supervisor devices 40.

[0059] (Variation 3) The supervisor device 40 includes a pointed-out voice collection program 41, a pointed-out pen input collection program 42, and a pointed-out action collection program 43. However, the supervisor device 40 only needs to include at least one of the above three programs.

[0060] (Variation 4) The indicated content display program 14 deletes the indication when the indicated portion matches the optimal state. However, when the indicated portion matches the optimal state, the indicated content display program 14 may add a display clearly indicating that the optimal state has been achieved while leaving the indication.

[0061] (Variation 5) The databases in the above-described embodiment may be synchronized at high speed. That is, task data showing the trainee's movements may be displayed in real time on the supervisor device 40, and instruction content may be instantly generated based on the supervisor's instructions and displayed on the trainee display device 104.

[0062] (Variation 6) In the above-described embodiment, the trainee motion analysis program 13 outputs the three-dimensional positions and postures of the trainee's hands, feet, body, and head at each time as motion data. However, the motion data may also include data on the three-dimensional positions and postures of the parts used by the trainee. In this case, the supervisor's instructions may include instructions regarding the parts used by the trainee, and the instruction content may also include the optimal state of the parts used by the trainee. Examples of parts used by the trainee include hand tools such as screwdrivers and wrenches, power tools, welding tools, switches and levers on the equipment being worked on, and doors that open and close.

[0063] In each of the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0064] In the above-described embodiments and modifications, the programs are stored in the ROM of each device. However, the programs may be stored in a non-volatile storage device. Furthermore, each device may have an input / output interface (not shown), and the programs may be loaded from another device via a medium available to the input / output interface when necessary. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Furthermore, some or all of the functions realized by the programs may be realized by a hardware circuit or FPGA.

[0065] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0066] 1: Pointing out system 10:Trainer device 11: Work content display program 12: Trainee movement acquisition program 14: Pointed content display program 20: Device for reflecting pointed out items 21: Work information collection program 22: Point collection program 23: Pointed content creation program 24: Approval request program 25: Pointed content sending program 40: Supervisor device 41: Audio recording program 42: Pointing pen input collection program 43: Program for collecting pointed out actions 44: Trainee movement reproduction program 45: Program for approving pointed out content 81: Working Content Database 82: Trainee movement database 83: Pointed Content Database 84: Database of findings 104: Trainee display device

Claims

1. A system for a supervisor to point out deficiencies in work performed by a trainee, a movement collecting unit that collects movement data of the trainee; a motion image providing unit that visually provides the motion data to the supervisor; a suggested matter collection unit that collects suggested matters that are suggestions from the supervisor to the trainee; a suggested content creation unit that creates suggested content, which is data to be presented to the trainee, based on the suggested items; a suggestion content display unit that presents the suggestion content to the trainee.

2. The indicating system according to claim 1, The motion data includes at least one of data that quantifies the movement of the trainee's joints, data that quantifies the positions of the trainee's body parts, and data that quantifies the positions of the parts of the members used by the trainee.

3. The indicating system according to claim 2, The motion image providing unit reproduces the trainee's motion as a three-dimensional model based on the motion data.

4. The indicating system according to claim 2, The indication system, wherein the indication collection unit collects at least one of the supervisor's speech, the supervisor's pen input, and the supervisor's gesture as the indication.

5. The indicating system according to claim 2, The instruction content includes a designation of an optimal state, which is an appropriate position or posture, for an instruction part, which is a part of the trainee's body or a part of a member used by the trainee, a trainee display device that is a display device that can be used by the trainee, The indication content display unit displays an indication display indicating the optimum state for the indicated part on the trainee display device.

6. 6. The indicating system according to claim 5, The indication system, wherein the indication content display unit deletes the indication display when the indicated portion matches the optimal state.

7. A method for creating suggested content executed by a suggested matter reflection device to which a suggested matter pointed out by a supervisor with respect to a trainee's movement is input, comprising: The instructions include a designation of an optimal state, which is an appropriate position or posture, for an instruction part that is a part of the trainee's body or a member used by the trainee, a finding collection step of acquiring the finding; and a step of converting the optimum state into a three-dimensional optimum state expressed as a three-dimensional position or posture, and combining the optimum state with the indicated part to create indicated content.

8. The method for creating pointed-out content according to claim 7, The method for creating suggested content further includes a suggested content transmitting step of transmitting the suggested content to a trainee device used by the trainee.

9. The method for creating pointed-out content according to claim 7, In the step of collecting findings, a plurality of findings are collected, In the pointed content creation step, a priority is determined for each pointed content, and the pointed content includes the pointed content whose priority is higher than a predetermined threshold, or a predetermined number of the pointed content with the highest priority.

Images