System for supporting creation of standard operation procedure

Abstract

To support creation of a standard operation procedure (SOP) based upon actual operations.SOLUTION: A system for supporting creation of a standard operation procedure comprises: a camera 21 and a microphone 22 which, in response to each of motion calls being voices indicating an operation motion and being uttered by a model operator M indicating operation models, photograph a series of states of operations and collect voices of the model operator, respectively; and a support computer 4 which analyzes the images photographed and the voices collected respectively by the camera 21 and microphone 22, generates operation motion character information showing the motion call in characters when the model operator M utters the motion call, and also extracts the operation motions corresponding respectively to the motion calls, as operation motion images, from images of a circumference of when the operation calls are uttered, and stores the operation motion character information and the operation motion images in association with each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a procedure creation support system that supports the creation of a procedure. [Background technology]

[0002] For example, when creating instructions (procedure manuals) for various tasks and operations at a power plant, traditionally, a person would record (take notes) the details of the task as it was being performed, stopping the work each time to record something. In addition, a person had to see, hear, and record the amount of operation of handles, dials, etc., and the changes in the surroundings due to the operation (operation results).

[0003] Meanwhile, there is known an instruction manual creation device that can efficiently create instruction manuals (see, for example, Patent Document 1). This device defines and holds data including text and images as module elements, defines the combination and application order of necessary elements among the module elements as a module configuration, and extracts and arranges the module elements according to the definition of the module configuration. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-235646 A Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, a person would record the operation details while actually working, which not only required a lot of time and effort, but also required the person to stop working every time they wanted to record, which interrupted the flow of the work and could lead to actions and timing that differed from normal. In addition, because the amount of operation and the results of operation were recorded by a person by sight and hearing, the recording was influenced by the person's senses, and there was a risk that the recording would not be appropriate.

[0006] Under these circumstances, there has been a demand for a technology that can easily create manuals based on actual work. On the other hand, the device in Patent Document 1 only defines module elements including text and images, and extracts and arranges the module elements according to the definition of the module configuration, and is not capable of creating manuals based on actual work.

[0007] Therefore, an object of the present invention is to provide a procedure creation support system capable of supporting the creation of a procedure based on an actual work. [Means for solving the problem]

[0008] In order to solve the above problem, the invention of claim 1 is a manual creation support system comprising: a photographing and sound collecting means for photographing a series of work situations performed by the model worker and collecting the sound, the photographing and sound collecting means storing the images photographed by the photographing and sound collecting means and the collected sound; a manual creating means for analyzing the sound stored in the image and sound storing means, and creating work action text information indicating the action name in text when the model worker utters the action name, and extracting a work action corresponding to the action name as a work action image from images around the time of the action name stored in the image and sound storing means; and a manual storing means for storing the work action text information and the work action image in association with each other.

[0009] The invention of claim 2 is characterized in that, in the instruction manual creation support system described in claim 1, the instruction creation means creates a work action illustration that illustrates the actions in the work action image based on the work action image, and the instruction storage means stores the work action illustration in association with the work action text information and the work action image.

[0010] The invention of claim 3 is characterized in that, in the procedure creation support system described in claim 1, the voice uttered by the model worker indicating the confirmation item is a confirmation call, the procedure creation means analyzes the voice stored in the image and audio storage means, and when the model worker utters a confirmation call, creates confirmation item text information indicating the confirmation call in text, and the procedure storage means stores the confirmation item text information.

[0011] The invention of claim 4 is characterized in that, in the procedure creation support system described in claim 3, the procedure creation means extracts, from the audio and images stored in the image and audio storage means, audio and images within a predetermined time from the time of the confirmation name as action result information, and the procedure storage means stores the action result information in association with the confirmation item text information.

[0012] The invention of claim 5 is characterized in that, in the procedure creation support system according to claim 1, the procedure creation means creates a procedure showing procedures for a series of work performed by the model worker by arranging the work action text information stored in the procedure storage means in chronological order.

[0013] The invention of claim 6 is characterized in that, in the procedure creation support system according to claim 5, it further comprises a procedure editing means for editing the procedure created by the procedure creation means.

[0014] The invention of claim 7 is characterized in that, in the instruction manual creation support system described in claim 1, the instruction manual creation means uses a work action learning model that has been machine-learned based on past performance data so that, when the voice stored in the image and voice storage means is input, it determines whether the model worker has uttered the action name or not, and if so, work action text information indicating the action name in text is output.

[0015] The invention of claim 8 is characterized in that, in the procedure creation support system described in claim 2, the procedure creation means uses a learning model for illustrations that has been machine-learned based on past performance data so that when the work action image is input, a work action illustration that illustrates the actions in the work action image is output.

[0016] The invention of claim 9 is characterized in that, in the procedure creation support system described in claim 3, the procedure creation means uses a learning model for confirmation items that is machine-learned based on past performance data so that, when the voice stored in the image and voice storage means is input, it determines whether the model worker has uttered the confirmation call or not, and if so, confirmation item text information indicating the confirmation call in text is output.

[0017] The invention of claim 10 is characterized in that, in the instruction manual creation support system described in claim 1, it comprises a glasses-type wearable terminal having a communication function and an AR function and worn by a user, and the wearable terminal displays at least one of the work action text information and the work action image stored in the instruction memory means as a virtual image in accordance with the user's operation.

[0018] The invention of claim 11 is characterized in that, in the instruction manual creation support system described in claim 2, it comprises a glasses-type wearable terminal having a communication function and an AR function and worn by a user, and the wearable terminal displays at least one of the work action text information, the work action image, and the work action illustration stored in the instruction memory means as a virtual image in accordance with the user's operation.

[0019] The invention of claim 12 is characterized in that, in the procedure creation support system described in claim 3, it comprises a glasses-type wearable terminal having a communication function and an AR function and worn by a user, and the wearable terminal displays the confirmation item text information stored in the procedure storage means as a virtual image in accordance with the user's operation.

[0020] The invention of claim 13 is characterized in that, in the procedure creation support system described in claim 4, a glasses-type wearable terminal is provided with a communication function, an AR function, and a speaker and is worn by a user, and the wearable terminal performs at least one of displaying the operation result information stored in the procedure memory means as a virtual image or outputting audio from the speaker in accordance with the user's operation. Effect of the Invention

[0021] According to the invention described in claim 1, when an exemplary worker calls out an action, work action text information showing the action name in text is created, and a work action image corresponding to the action name is extracted, and these are stored in association with each other. For example, when an exemplary worker calls out an action name "Next, do XX", work action text information showing this action name in text is created, and a work action image corresponding to the action "do XX" is extracted, and these are stored in association with each other.

[0022] In this way, text information and image information are created and stored in response to the action names (for example, pointing and calling out) made by the model worker during the actual work, making it possible to support the creation of manuals based on the actual work. Moreover, since the model worker only needs to perform the work and call out the actions in the same way as usual, not only is the burden on the model worker reduced, but it is also possible to create manuals that follow normal actions and their timing without interrupting the flow of a series of work.

[0023] According to the invention described in claim 2, a work action illustration is created by illustrating the actions in the work action image, so that it is possible to create an easy-to-understand manual by using this work action illustration.

[0024] According to the invention described in claim 3, when the model worker issues a confirmation call such as "After that, confirm that XX becomes XX," confirmation item text information showing this confirmation call in text is created and stored. In this way, text information is created and stored in response to the confirmation call issued by the model worker during actual work, making it possible to support the creation of a manual including the confirmation items that are actually being performed. Moreover, since the model worker only needs to perform work and issue a confirmation call in the same way as usual, not only is the burden on the model worker reduced, but it is also possible to create a manual that follows normal confirmations and their timing without interrupting the flow of the work series.

[0025] According to the invention described in claim 4, the voice and image within a predetermined time after the model worker issues a confirmation call are extracted as action result information and stored in association with the confirmation item text information. In other words, the voice and image showing what state is correct for the confirmation item (what result is correct for the action) are stored as action result information. This makes it possible to support the creation of appropriate and useful manuals that include the correct results and states for the actions.

[0026] According to the invention described in claim 5, a manual showing the procedure for a series of work performed by a model worker is created by arranging the work action text information in chronological order. In other words, the manual is automatically created by arranging the text information for each of a plurality of work actions performed by a model worker in chronological order, so that it is possible to easily create a manual based on actual work.

[0027] According to the invention described in claim 6, since the automatically created manual can be edited, it becomes possible to create a proper manual.

[0028] According to the invention described in claim 7, the machine-learned work motion learning model is used to output the work motion text information and whether the model worker uttered the action name, so that it is possible to more accurately capture the action name and create the work motion text information. As a result, it is possible to more accurately create the manual.

[0029] According to the invention described in claim 8, since a work action illustration is output by illustrating the actions in the work action image using a machine learning model for illustration, it becomes possible to create more appropriate work action illustrations. As a result, it becomes possible to create more appropriate manuals.

[0030] According to the invention described in claim 9, by using a machine-learned learning model for confirmation items, confirmation call-outs made by model workers and text information on confirmation items are output, so it is possible to more accurately capture confirmation calls and create confirmation item text information. As a result, it is possible to more accurately create manuals.

[0031] According to the invention described in claim 10, the work action text information and the work action image are displayed as a virtual image on the wearable device in accordance with the operation of the user wearing the wearable device. This enables the user to perform the work appropriately according to the virtual image.

[0032] According to the invention described in claim 11, work action text information, work action images, and work action illustrations are displayed as virtual images on the wearable device according to the operation of the user wearing the wearable device. This allows the user to perform work appropriately according to the virtual images. Moreover, since three virtual images can be displayed arbitrarily by the user's operation and selection, it is possible to display a virtual image suitable for the user and appropriately support the work.

[0033] According to the invention described in claim 12, confirmation item text information is displayed on the wearable device as a virtual image in accordance with the operation of the user wearing the wearable device. This allows the user to properly confirm the confirmation items in accordance with the virtual image.

[0034] According to the invention described in claim 13, the operation result information is displayed as a virtual image on the wearable terminal or output as a sound according to the operation of the user wearing the wearable terminal. Therefore, the user can know what the correct result or state is after the operation, and can make an appropriate decision. [Brief description of the drawings]

[0035] [Figure 1] 1 is a schematic configuration diagram showing a procedure creation support system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic perspective view showing smart glasses of the instruction manual creation support system of FIG. 1. [Diagram 3] FIG. 3 is a schematic configuration block diagram of the smart glasses of FIG. 2. [Figure 4] 2 is a schematic block diagram showing a support computer of the procedure creation support system of FIG. 1. [Diagram 5] 5 is a functional block diagram showing a schematic configuration of a work movement learning model of the support computer of FIG. 4. [Figure 6] FIG. 2 is a diagram showing an example of an image displayed on the smart glasses of the procedure creation support system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, the present invention will be described based on the illustrated embodiment.

[0037] 1 is a schematic diagram showing a procedure creation support system 1 according to an embodiment of the present invention. This procedure creation support system 1 is a system for supporting the creation of a procedure for a work and supporting the work by a worker (user) U, and mainly includes a camera 21 and a microphone 22 as an image capturing and sound collecting means, smart glasses (wearable terminal) 3, and a support computer 4.

[0038] Here, work includes all business actions, and broadly includes not only work and construction by workers, but also clerical work and actions by office staff, inspection work and actions by inspectors, etc. Also, instructions indicate at least one of the following: the content, procedure, order of the work (operation), confirmation items after work, and precautions, and include not only written documents but also electronic files and data.

[0039] In this embodiment, the following description will be mainly given as an example of creating a manual for a worker U at a power plant to perform various tasks and operations. Also, the manual is created based on the work actually performed by a model worker M who is skilled in the work and sets an example of the work. Note that Fig. 1 shows a scene where the model worker M turns the valve P1 with a Wilkie T1 at the work site P and checks the pressure value of the pressure gauge P2.

[0040] The camera 21 is a photographing device that photographs a series of work situations performed by the model worker M, and is attached to the front of the helmet 23 worn by the model worker M, so that it can photograph the scene seen by the model worker M. The microphone 22 is a device that collects sounds made by the model worker M and sounds generated in the surrounding area and converts them into electrical signals, and is placed from the helmet 23 worn by the model worker M to the mouth of the model worker M, so that it is easy to collect sounds made by the model worker M.

[0041] In this embodiment, a storage device (not shown) is attached to the helmet 23, and images (moving images, still images) captured by the camera 21 and sounds collected by the microphone 22 are stored in the storage device. The images and sounds stored (video and audio) in the storage device are then stored in an image and audio database 441 of the support computer 4, which will be described later. Alternatively, the images captured by the camera 21 and sounds collected by the microphone 22 may be sent directly to the support computer 4 via a communication device and stored in the image and audio database 441.

[0042] The smart glasses 3 are glasses-type wearable terminals equipped with a communication function, an AR (Augmented Reality) function, and an audio output function, and are worn by a worker U who performs work according to a manual, as shown in Fig. 2. As shown in Fig. 3, the smart glasses 3 mainly include a communication unit 31, a memory 32, an audio unit (speaker) 33, a stereo camera (photographing unit) 34, a display 35, an operation unit 36, an AR unit 37, and a central processing unit 38 that controls these units.

[0043] The communication unit 31 is an interface for communicating with the outside, and is capable of communicating with the support computer 4. The memory 32 is a storage unit that stores various information, and stores identification information of the smart glasses 3 and a manual (described later) received from the support computer 4. The audio unit 33 has a speaker function and a microphone function, and outputs the sound of the operation result information received from the support computer 4, and also collects the sound uttered by the worker U, converts it into an electrical signal, and transmits it to the support computer 4.

[0044] The stereo camera 34 is an imaging device that captures the scene that the worker U faces, and captures in real time the scene that the worker U sees when he or she turns their eyes. The display 35 is a display unit that displays the scene captured by the stereo camera 34 as an image for the worker U to view, and also displays virtual images and the like created by the AR unit 37, which will be described later. The operation unit 36 ​​is an interface that allows the worker U to perform various operations, and selects an image to be displayed from a display selection image G2, which will be described later.

[0045] In this embodiment, operation and selection can be performed by the line of sight and voice of the worker U, and the line of sight is detected to analyze the operation and selection by the worker U, and the voice of the worker U collected by the audio unit 33 is analyzed to obtain the operation content. Alternatively, a smartphone (multi-function mobile phone) communicably connected to the smart glasses 3 may be used as the operation unit 36, and operation and selection may be performed from this smartphone.

[0046] The AR unit 37 is a functional unit that executes and displays augmented reality, and displays work action text information (instructions) G3, work action images G4, work action illustrations G5, etc., which are received from the support computer 4 as virtual images at predetermined positions on the display 35.

[0047] The support computer 4 is a computer server for creating manuals and supporting work, and in this embodiment, is disposed in the management center C of the power plant. As shown in Fig. 4, the support computer 4 mainly comprises an input unit 41, a display unit 42, a communication unit 43, a memory unit 44, a manual creation task (manual creation means) 45, a manual editing task (manual editing means) 46, a work support task (work support means) 47, a learning task 48, and a central processing unit 49 for controlling these. Here, the support computer 4 may be composed of multiple computer servers.

[0048] The input unit 41 is an interface for inputting various information and commands, and specifically, inputs start commands for the procedure creation task 45 and the work support task 47, and inputs editing contents, which will be described later. The display unit 42 is a display for displaying various data and information, and specifically, displays work action text information, work action images, and instructions created in the procedure creation task 45. The communication unit 43 is an interface for communicating with the outside via the Internet network, a telephone communication network, and specifically, transmits work action text information G3, work action images G4, and the like to the smart glasses 3.

[0049] The storage unit 44 mainly includes an image and audio database (image and audio storage means) 441, a name database (name storage means) 442, a technique database (technique storage means) 443, learning models (learning models for work actions, learning models for illustrations, learning models for confirmation items) 444, and a track record database (track record database for work actions, illustrations, and confirmation items) 445. Here, the databases 441, 442, and 443 will be explained, and the learning model 444 and the track record database 445 will be explained later.

[0050] The image and audio database 441 is a database that stores images and audio showing a series of work situations (model work situations) by the model worker M. That is, as described above, the images and audio captured by the camera 21 and microphone 22 and stored in the storage device are stored in chronological order. The image and audio database 441 also stores images and audio showing a series of work situations for each series of tasks / operations.

[0051] The name database 442 is a database that stores examples of names (pointing and calling) uttered by the model worker M during work, and the names include action names and confirmation names. Here, action names are sounds that indicate work actions (each task / action content in a series of tasks), and include, for example, names such as "Next, do XX with XX" and "Use XX to do XX." Confirmation names are sounds that indicate confirmation items (contents to be confirmed), and include, for example, names such as "After that, make sure that XX becomes XX" and "Make sure that the XX meter shows XX."

[0052] Incidentally, these kinds of action calls and confirmation calls are also done in normal work, so model worker M only needs to carry out the work and call out the instructions in the same way as normal, and does not need to take the trouble of calling out the instructions in order to create a manual.

[0053] The procedure database 443 is a database that stores the work action text information, work action images, and work action illustrations created in the procedure creation task 45 described below in association with each other, stores confirmation item text information and action result information in association with each other, and also stores the created procedure documents.

[0054] The procedure creation task 45 is a program task that analyzes the voice stored in the image and voice database 441 and creates the procedure (points, essentials, and important points of the work) required for creating a procedure document based on the name uttered by the model worker M. In this embodiment, it is started when a storage range (a series of work data to be created) of the image and voice database 441 is specified from the input unit 41 and a start command is input. Alternatively, it may be started in parallel with the model work by the model worker M to create the procedure.

[0055] First, the voice stored in the image and voice database 441 is analyzed, and when the model worker M utters an action name, work action text information indicating the action name in text, a work action image corresponding to the action name, and a work action illustration illustrating the action in the work action image are created. That is, by referring to the example names stored in the name database 442, the action name uttered by the model worker M is extracted, and work action text information indicating this action name in text is created. For example, when the model worker M utters the action name "Next, open the valve," this name is converted into text to create work action text information "open the valve" (for example, step 5 of symbol G3 in FIG. 6).

[0056] Next, a work action / action image corresponding to the action name is extracted as a work action image from images around the time of the action name stored in the image and audio database 441. That is, since a work action corresponding to the action name is usually performed before and after the action name is uttered, an image corresponding to this work action is extracted from the image and audio database 441. For example, when the action name "Next, open the valve" is uttered, an image showing the work action of opening the valve is extracted as a work action image (for example, symbol G4 in FIG. 6).

[0057] Next, based on the extracted work action image, a work action illustration is created that illustrates the action in the work action image. That is, a work action illustration is created that illustrates the work action corresponding to the action name in an easy-to-understand manner. For example, when the action name "Next, open the valve" is uttered and a work action image of turning the valve P1 counterclockwise with the Wilkey T1 is extracted, a work action illustration (for example, symbol G5 in FIG. 6) that illustrates the action of turning the Wilkey T1 counterclockwise is created.

[0058] The work action text information, work action image, and work action illustration thus created are stored in the procedure database 443 in association with the action name uttered by the model worker M. Such processing is performed sequentially for a series of model works by the model worker M.

[0059] In parallel with this process, the voice stored in the image and voice database 441 is analyzed, and when the model worker M issues a confirmation call, confirmation item text information indicating the confirmation call in text is created. That is, by referring to example calls stored in the call database 442, the confirmation call issued by the model worker M is extracted, and confirmation item text information indicating this confirmation call in text is created. For example, when the model worker M issues a confirmation call such as "Confirm that the pressure gauge shows XX MPa," this call is converted into text and confirmation item text information (for example, step 6 of symbol G3 in FIG. 6) such as "Confirm that the pressure gauge shows XX MPa" is created.

[0060] Furthermore, from the voices and images stored in the image and voice database 441, voices and images within a predetermined time from the confirmation call are extracted as operation result information. That is, since a result (result of performing an action or operation) corresponding to the confirmation call or confirmation item usually occurs within a predetermined time after the confirmation call is issued, voices and images corresponding to this result are extracted from the image and voice database 441. For example, for a confirmation call such as "Check that the pressure gauge indicates XX MPa," an image showing the indicated value of the pressure gauge is extracted, and for a confirmation call such as "Check the sound of steam flowing," voice showing the sound of steam is extracted. Here, if only one of the voices or images is sufficient, the other may be deleted in the instruction manual editing task 46 described later.

[0061] The confirmation item text information and the action result information thus created are stored in the procedure database 443 in association with the confirmation call uttered by the model worker M. Such processing is performed sequentially for a series of model works by the model worker M.

[0062] Furthermore, the procedure creation task 45 uses the work action text information, work action image, work action illustration, check item text information, and action result information stored in the procedure database 443 in this way to create a procedure document showing the procedure for a series of work by the model worker M. That is, for each series of work, a procedure document showing the procedure and order of work and operation is created by combining text information, images, and illustrations showing the work actions in each work step, text information showing check items, and images and sounds (action result information), and is stored in the procedure database 443. Here, in this embodiment, the work action text information and the check item text information are arranged in chronological order (in the order of steps) to create the procedure document, and the work action image, work action illustration, and action result information corresponding to each step (work action and check item) of the procedure document are stored in association with each step. On the other hand, the procedure document may include check item text information, work action image, work action illustration, etc. in the work action text information.

[0063] When the voice stored in the image and voice database 441 is input, the procedure creation task 45 judges whether the model worker M has uttered an action name, and if so, outputs work action text information showing the action name in text, using a work action learning model that has been machine-learned based on past performance data, and this work action learning model is created by the learning task 48. Here, the learning model 444 includes a work action learning model, and the performance database 445 includes a work action performance database.

[0064] Furthermore, when a work action image is input, the procedure creation task 45 uses a learning model for illustration that is machine-learned based on past performance data so that a work action illustration that illustrates the actions in the work action image is output, and this learning model for illustration is created by the learning task 48. Here, the learning model 444 includes a learning model for illustration, and the performance database 445 includes a performance database for illustration.

[0065] Furthermore, when the voice stored in the image and voice database 441 is input, the procedure creation task 45 judges whether or not the model worker M has uttered a confirmation call, and if so, uses a learning model for confirmation items that has been machine-learned based on past performance data so as to output confirmation item text information indicating the confirmation call in text, and this learning model for confirmation items is created by the learning task 48. Here, the learning model 444 includes a learning model for confirmation items, and the performance database 445 includes a performance database for confirmation items.

[0066] Here, since the configuration of each learning model and the relationship with the learning task 48 are the same, the work motion learning model will be explained below.

[0067] That is, as shown in Fig. 5, the learning task 48 uses past performance data recorded and accumulated in the work performance database to create a learning model for work motions by a known machine learning algorithm such as a neural network. This work performance database is a database in which performance data including the results of voice analysis or the judgment by experts and skilled workers of power plant operations as to whether the model worker M uttered the action name based on the voice (voice collected by the microphone 22 of the model worker M) stored in the image and voice database 441 as input information, and work motion text information showing the action name in text if the model worker M uttered the action name. The past performance data includes the voice actually uttered by the model worker M, data created based on the actual judgment of experts and skilled workers, as well as data created in advance training, etc.

[0068] As shown in Fig. 5, this learning task 48 uses machine learning and deep learning using a neural network, and creates a neural network based on the performance data recorded in the work motion performance database, with the voice (voice uttered by the model worker M) stored in the image and voice database 441 as an input layer, work motion text information of whether the model worker M uttered the action name or not as an output layer, and analysis processing from the input layer to the output layer as an intermediate layer. Then, the learning task 48 uses the performance data of the work motion learning model as learning data to learn various parameters in the intermediate layer. That is, the learning task 48 learns various parameters in the intermediate layer based on the voice stored in the image and voice database 441 so that an appropriate judgment result is output.

[0069] Here, the illustration performance database is a database in which performance data including illustrations of work actions created by experts and skilled workers of power plant operations, etc., based on work action images as input information, is recorded and accumulated. Also, the confirmation item performance database is a database in which performance data including the results of whether the model worker M issued a confirmation call or not, determined by experts and skilled workers of power plant operations, etc., based on the voice stored in the image and voice database 441 as input information, and confirmation item text information indicating the confirmation call in text if the model worker M issued a confirmation call, is recorded and accumulated.

[0070] The manual editing task 46 is a program task for editing the manual created in the manual creation task 45, and is started when a start command is input from the input unit 41. The manual editing task 46 first displays a selection screen for selecting a manual stored in the manual database 443 on the display unit 42, and when a manual is selected by the input unit 41, displays the manual on the display unit 42. The manual can be edited by inputting the editing contents (corrections and additions) on the input unit 41. Furthermore, when the voice uttered by the worker U is transmitted to the support computer 4 via the smart glasses 3, the manual can be edited based on the voice. For example, when the worker utters a voice such as "It would be better to do XX here," this is stored and displayed, and the manual can be edited by referring to and referring to this voice.

[0071] The work support task 47 is a program task for supporting the work performed by the worker U based on the manuals stored in the manual database 443, and is started when a start command is input from the input unit 41. This work support task 47 displays and outputs the specified manual (the manual corresponding to the target work), the corresponding work action image, the work action illustration, and the action result information on the smart glasses 3 according to the operation of the worker U.

[0072] Here, the things displayed on the display 35 of the smart glasses 3 include the following, as shown in Fig. 6. First, it includes a view G1 seen by the worker U, which is captured by the stereo camera 34 of the smart glasses 3, and is displayed over the entire central part of the display 35. Second, it includes a display selection image G2 for selecting which virtual image to display. This display selection image G2 includes a "manual" button for selecting the manual G3, an "AR" button for selecting the work action illustration G5, and a "video" button for selecting the work action image G4.

[0073] Thirdly, it includes a manual (work action text information and confirmation item text information) G3 for the target work as a virtual image, and fourthly, it includes a work action image G4 as a virtual image. Here, the display selection image G2, the manual G3, and the work action image G4 are displayed at predetermined positions on the display 35, but the positions may be changeable and adjustable. Fifthly, it includes a work action illustration G5 as a virtual image, and is displayed in the center of the display 35 (so as to overlap the view G1).

[0074] Here, as described above, the worker U can operate and select with his / her gaze or voice, for example, he / she can select a button in the display selection image G2, or select the work action text information and the confirmation item text information in the instruction manual G3.

[0075] Any one or more of the instruction manual G3, work action image G4, and work action illustration G5 can be displayed simultaneously according to the operation and selection of the worker U. That is, the scene G1 and the display selection image G2 are always displayed on the smart glasses 3, and when one or more buttons are selected on the display selection image G2, the corresponding virtual image is displayed.

[0076] Specifically, when the "Instruction Manual" button is selected, the instruction manual G3, i.e., the work action text information and confirmation item text information, is displayed, when the "Video" button is selected, the work action image G4 is displayed, and when the "AR" button is selected, the work action illustration G5 is displayed. In addition, when multiple buttons are selected, all of the corresponding virtual images G3, G4, and G5 are displayed.

[0077] At this time, the work action image G4 and the work action illustration G5 that correspond to and are related to the work action text information and the confirmation item text information selected in the instruction manual G3 (or selected immediately before, even if the instruction manual G3 is not displayed) are displayed. When confirmation item text information is selected, at least one of the following is performed: an image of the action result information is displayed as the work action image G4, or the sound of the action result information is output from the audio unit 33 of the smart glasses 3.

[0078] Here, in this embodiment, the operation and selection of the worker U are sequentially sent from the smart glasses 3 to the support computer 4 (work support task 47), and in response to this, the work support task 47 transmits the manual G3 and the work action image G4 corresponding to the smart glasses 3. Then, the received manual G3 and the work action image G4 are displayed and output in the smart glasses 3. In contrast, when the work support task 47 is started and the manual (series of work) is specified, the work support task 47 may transmit all the manual G3 and the work action image G4 to the smart glasses 3, and the manual G3 and the work action image G4 may be displayed in the smart glasses 3 according to the operation and selection of the worker U. Note that any method of specifying the manual may be used, and for example, when the series of work to be performed by the worker U is decided, it may be specified by the input unit 41.

[0079] As described above, according to the instruction manual creation support system 1, when the model worker M utters an action name, work action text information showing the action name in characters is created, and a work action image corresponding to the action name is extracted, and these are associated and stored. For example, when the model worker M utters an action name "Next, do XX", work action text information showing this action name in characters is created, and a work action image corresponding to the action "do XX" is extracted, and these are associated and stored.

[0080] Furthermore, when the model worker M issues a confirmation call such as "After that, make sure that XX becomes XX," confirmation item text information indicating this confirmation call in text is created and stored. Furthermore, audio and images within a predetermined time after the model worker M issues the confirmation call are extracted as action result information and stored in association with the confirmation item text information. In other words, audio and images indicating what state is correct for the confirmation item (what result is correct for the action) are stored as action result information. This makes it possible to support the creation of appropriate and useful instructions that include the correct results and states for the actions.

[0081] In this way, text information and image information are created and stored in response to the action names and confirmation names (for example, pointing and calling out) given by the model worker M during actual work, making it possible to support the creation of manuals based on the work and confirmation items actually being performed. Moreover, since the model worker M only needs to perform the work in the same way as usual and call out the action names and confirmation names, not only is it possible to reduce the burden on the model worker M, but it is also possible to create manuals that follow normal actions, confirmations, and their timing without interrupting the flow of a series of work.

[0082] Furthermore, since a work action illustration is created by illustrating the actions in the work action image, it is possible to create an easy-to-understand manual by using this work action illustration.

[0083] In addition, a manual showing the procedure for a series of tasks performed by the model worker M is created by arranging text information and images of the tasks in chronological order. In other words, the manual is automatically created by arranging text information and images of each of a plurality of tasks and steps performed by the model worker M in chronological order, making it easy to create a manual based on actual tasks. Moreover, the automatically created manual can be edited, making it possible to create an appropriate manual.

[0084] On the other hand, by using the machine-learned learning model for work actions, whether the model worker M has issued an action name and work action text information are output, so it is possible to more accurately capture the action name and create work action text information. Similarly, by using the machine-learned learning model for illustrations, a work action illustration that illustrates the action in the work action image is output, so it is possible to create a more accurate work action illustration. Furthermore, by using the machine-learned learning model for confirmation items, whether the model worker M has issued a confirmation name and check item text information are output, so it is possible to more accurately capture the confirmation name and create check item text information. As a result of these, it is possible to more accurately create the manual.

[0085] In addition, according to the operation of the worker U wearing the smart glasses 3, work action text information, work action images, and work action illustrations are displayed as virtual images G3, G4, and G5 on the smart glasses 3. This enables the worker U to perform work appropriately according to the virtual images G3, G4, and G5. Moreover, since the three virtual images G3, G4, and G5 can be arbitrarily displayed by the operation and selection of the worker U, it is possible to display the virtual images G3, G4, and G5 suitable for the worker U and appropriately support the work.

[0086] Similarly, according to the operation of the worker U wearing the smart glasses 3, confirmation item text information is displayed as a virtual image G3 on the smart glasses 3, and operation result information is displayed as a virtual image G4 on the smart glasses 3 and output as audio. This allows the worker U to properly confirm the confirmation items according to the virtual images G3 and G4, and to know what the correct result / state is after the operation, thereby making an appropriate decision.

[0087] Although the embodiment of the present invention has been described above in detail, the specific configuration is not limited to this embodiment, and even if there are design changes within the scope of the present invention, they are included in the present invention. For example, the above embodiment has been described using an example of a manual for a worker U to perform various tasks and operations at a power plant, but it goes without saying that the present invention can also be applied to manuals for other tasks. [Explanation of symbols]

[0088] 1. Manual Creation Support System 21 Camera (means for recording and recording images) 22 Microphone (means for recording and recording sound) 3. Smart glasses (wearable devices) 31 Communication section (communication function) 33 Audio section (speaker) 34 Stereo camera (photography unit) 35 Display 37 AR section (AR function) 4 Support Computer 45 Procedure creation task (procedure creation means) 46 Manual editing task (manual editing method) 47 Work support tasks (work support means) 441 Image and audio database (image and audio storage means) 442 Name database (name storage means) 443 Procedure database (procedure storage means) 444 Learning Models (Learning Models for Work Operations, Learning Models for Illustrations, Learning Models for Confirmation Items) 445 Performance database (performance database for work actions, illustrations, and confirmation items) M Model worker U Worker (user) G1 Sight G2 Display Selection Image G3 instructions G4 Work operation image (including operation result information image) G5 Work movement illustration

Claims

1. The voice indicating the work action uttered by the model worker who shows the work example is defined as the action name, a photographing and sound collecting means for photographing a series of work situations performed by the model worker and collecting his / her voice; an image and sound storage means for storing the image captured by the image capturing and sound collecting means and the sound collected by the image capturing and sound collecting means; a procedure creating means for analyzing the voice stored in said image and voice storing means, and creating work action text information indicating said action name in text when said model worker utters said action name, and extracting a work action corresponding to said action name as a work action image from an image around the time when said action name is uttered that is stored in said image and voice storing means; a method storage means for storing the work action character information and the work action image in association with each other; A procedure creation support system comprising:

2. The procedure creation means creates a work action illustration based on the work action image by illustrating the action in the work action image, the technique storage means stores the work action illustration in association with the work action character information and the work action image; 2. The procedure creation support system according to claim 1.

3. The voice of the model worker indicating the items to be confirmed is used as the confirmation call, The procedure creation means analyzes the voice stored in the image and voice storage means, and creates confirmation item text information indicating a confirmation call in text when the model worker utters the confirmation call, The procedure storage means stores the confirmation item text information.

2. The procedure creation support system according to claim 1.

4. the procedure creation means extracts, from the voices and images stored in the image and voice storage means, the voices and images within a predetermined time from the time of the confirmation name call as action result information; The procedure storage means stores the operation result information in association with the confirmation item text information.

4. The procedure creation support system according to claim 3.

5. The procedure creation means creates a procedure document showing a procedure for a series of work performed by the model worker by arranging the work action text information stored in the procedure storage means in a chronological order.

2. The procedure creation support system according to claim 1.

6. A manual editing means for editing the manual created by the manual creation means is provided.

6. The procedure creation support system according to claim 5.

7. The procedure creation means, when the voice stored in the image and voice storage means is input, judges whether the model worker has uttered the action name or not, and uses a work action learning model that is machine-learned based on past performance data so that, if the model worker has uttered the action name, work action text information indicating the action name in text form is output.

2. The procedure creation support system according to claim 1.

8. The method creation means uses a learning model for illustrations that is machine-learned based on past performance data so that, when the work action image is input, a work action illustration that illustrates the action in the work action image is output.

3. The procedure creation support system according to claim 2.

9. the procedure creation means, when the voice stored in the image and voice storage means is input, judges whether the model worker has uttered the confirmation call, and if so, uses a learning model for confirmation items that has been machine-learned based on past performance data so as to output confirmation item text information indicating the confirmation call in text.

4. The procedure creation support system according to claim 3.

10. The device is equipped with a glasses-type wearable device that is equipped with communication functions and AR functions and is worn by the user; the wearable terminal displays at least one of the work action text information and the work action image stored in the technique storage means as a virtual image in accordance with an operation of the user; 2. The procedure creation support system according to claim 1.

11. The device is equipped with a glasses-type wearable device that is equipped with communication functions and AR functions and is worn by the user; the wearable terminal displays at least one of the work action text information, the work action image, and the work action illustration stored in the technique storage means as a virtual image in accordance with an operation of the user; 3. The procedure creation support system according to claim 2.

12. The device is equipped with a glasses-type wearable device that is equipped with communication functions and AR functions and is worn by the user; The wearable terminal displays the confirmation item text information stored in the instruction storage means as a virtual image in accordance with an operation of the user.

4. The procedure creation support system according to claim 3.

13. The present invention provides a wearable device that is equipped with a communication function, an AR function, and a speaker, and is configured as glasses that are worn by a user; The wearable terminal performs at least one of displaying the action result information stored in the instruction storage means as a virtual image and outputting the action result information from the speaker in accordance with an operation of the user.

5. The procedure creation support system according to claim 4.

Images