Device for creating a video instruction and method for creating a video instruction

Abstract

The present invention makes it possible to easily create a moving image guide that, in combination with a moving image of a worker's key operation, promotes recognition of the pressed key. This moving image guide creation device comprises: a machine operation panel registration unit that registers an image of the operation panel of a machine; a signal registration unit that registers each key on the operation panel in the image registered by the machine operation panel registration unit and a signal for controlling the operation panel by an operation of each key on the operation panel in conjunction with each other; a signal data change time extraction unit that extracts a time of change of the signal registered in the signal registration unit;a frame detection unit that detects, based on the time extracted by the signal data change time extraction unit, a frame in which a key corresponding to the changed signal was pressed by the worker from the moving image including an operation performed by the worker on the operation panel; and a moving image generation unit that generates a moving image indicating the key pressed by the worker on the image of the operation panel registered by the machine operation panel registration unit when the frame detected by the frame detection unit is displayed.

Description

Technical area

[0001] The present invention relates to a device for creating a video guide and a method for creating a video guide that displays methods for operating buttons of a machine. Background of the invention

[0002] For example, in a controller having a screen for enabling the operation of a molded product ejector or the like, a technique for reading a video guide and displaying the video guide on a screen is known as a video. The video guide includes video information that provides a description regarding the operation of the molded product ejector in the form of a video based on an actual video, a simulation video, or an animation. See, for example, Patent Document 1. Citation listPatent document

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2008-246712 Disclosure of the invention Problems to be solved by the invention

[0004] However, the technique described in Patent Document 1 has a problem that it takes time to create a video describing the operation of a machine because the video includes both an actually shot video of the machine and a simulation video, animation, or the like.

[0005] Additionally, in a video that includes an operation performed by a worker, it may be difficult to identify which button the worker is pressing.

[0006] When a signal corresponding to a worker pressing a button to control the machine is detected by the machine, there may be a delay that may result in a discrepancy between the time the worker presses the button in the video and the time the button is displayed as being pressed.

[0007] In addition, it is difficult and time-consuming to edit a long-term video recorded by a surveillance camera or similar.

[0008] Accordingly, it is desirable to create an easy-to-understand video tutorial that shows which key is pressed in combination with a video of a worker pressing the key. Means to solve the problems

[0009] One aspect of a video guidance creation device includes: a machine operation panel registration unit configured to register an image of an operation panel of a machine; a signal registration unit configured to register each key on the operation panel in the image registered by the machine operation panel registration unit and a signal for controlling the operation panel by an operation of each key on the operation panel in association with each other; a signal data change time extraction unit configured to extract a change time of the signal registered in the signal registration unit;a frame acquisition unit configured to acquire, based on the time extracted by the signal data change time extraction unit, a frame in which a key corresponding to the changed signal was pressed by the worker from a video including an operation performed by a worker on the operation panel; and a video creation unit configured to generate a video displaying the key pressed by the worker on the image of the operation panel registered by the machine operation panel registration unit when the frame acquired by the frame acquisition unit is displayed.

[0010] One aspect of a video instruction creation method is a video creation method for making a computer operate as a video instruction creation device, the video instruction creation method comprising: a machine operation panel registration step of registering an image of an operation panel of a machine; a signal registration step of registering each key on the operation panel in the registered image and a signal for controlling the operation panel by an operation of each key on the operation panel in association with each other; a signal data change time extraction step of extracting a time of change in the registered signal;a frame acquisition step of capturing, based on the extracted time, a frame in which a key corresponding to the changed signal was pressed by the worker from a video including an operation performed by a worker on the operation panel; and a video creation step of generating a video displaying the key pressed by the worker on the registered image of the operation panel when the captured frame is displayed. Short description of the drawings Fig. 1 is a functional block diagram showing a functional configuration example of a video guidance creation system 1 according to a first embodiment; Fig. 2 is a diagram showing an example of a registered image of an operation panel of a machine; Fig. 3 is a diagram showing an example of a registered signal table; Fig. 4 is a diagram showing an example of the relationship between a video and a timing of a signal change; Fig. 5 is a diagram showing an example of displaying a video guide; Fig. 6 is a flowchart illustrating a process for creating a video instruction with a video instruction creating device; Fig. 7 is a diagram showing an example of the configuration of a video guidance creation system according to a second embodiment; Fig. 8 is a diagram showing an example of an image area of the operation panel detected by an operation panel detecting unit and a joint position of a worker estimated by a joint position estimating unit; Fig. Figure 9 is a flowchart illustrating the process of creating a video tutorial; Fig. 10 is a diagram showing an example of the relationship between a panel usage time by the worker, a signal change, and a delay time; Fig. 11 is a diagram showing an example of the configuration of a video guidance creation system according to a third embodiment; and Fig. 12 is a flowchart illustrating the video creation process of the video instruction creation device. Preferred embodiment of the invention<Erste Ausführungsform>

[0011] First, the outline of the present embodiment will be described. In the present embodiment, a video guidance creation device registers an image of an operation panel of a machine, and also registers each key on the operation panel in the registered image and a signal for controlling the operation panel by an operation of each key on the operation panel in association with each other. The video guidance creation device extracts the timing of a signal change, and, based on the extracted time, recognizes, from the video including an operation performed by a worker on the operation panel, an image in which a key corresponding to the changed signal was pressed by the worker. The video guidance creation device creates a video displaying the key pressed by the worker on the registered image of the operation panel when the recognized image is displayed.

[0012] Therefore, according to the present embodiment, it is possible to easily create an easy-to-understand video guide that shows which key was pressed in combination with a video of the worker's key operation.

[0013] This concludes the description of the basic features of the present embodiment.

[0014] Now, the configuration of the present embodiment will be described in detail with reference to the drawings.

[0015] Fig. 1 is a functional block diagram showing a functional configuration example of the video instruction creation system 1 according to the first embodiment.

[0016] As in Fig. 1, the system for creating a video instruction 1 comprises a device for creating a video instruction 10, a camera 20 and a machine 30.

[0017] The video instruction device 10, the camera 20, and the machine 30 may be connected to each other via a network (not shown), such as a local area network (LAN) or the Internet. Alternatively, the video instruction device 10, the camera 20, and the machine 30 may also be connected directly via a connection interface (not shown).

[0018] The camera 20 is a surveillance camera or the like that is installed in a factory or other locations where the machine 30 is installed. The camera 20 preferably captures, for example, two-dimensional images in which an operation panel 31 included in the machine 30 and a worker operating the operation panel 31 are projected onto a plane perpendicular to an optical axis of the camera 20 at a predetermined frame rate (e.g., 30 fps, etc.). Without being limited thereto, the camera 20 may capture the operation panel 31 and the buttons or the like on the operation panel 31 from any angle. The camera 20 outputs the captured images as a video to the video instruction preparation device 10, which will be described later. The image data captured by the camera 20 may be visible light images such as RGB color images, grayscale images, and depth images.

[0019] Here, in Fig. 1, the device for creating a video instruction 10 is connected to a camera 20. However, without being limited thereto, the device for creating a video instruction 10 can also be connected to two or more cameras 20.

[0020] The machine 30 is, for example, a machine tool including a numerical control device (not shown) well known to those skilled in the art. The machine 30 operates based on a machining program executed by the numerical control device (not shown) and also operates based on an operation performed by the worker on the operation panel 31 of the machine 30. Furthermore, the machine 30 outputs a signal for controlling the operation panel 31, generated by a key operation performed by the worker on the operation panel 31, to the video guidance generating device 10 described later.

[0021] As described later, the machine 30 may also be a robot controller that controls a robot.

[0022] The control panel 31 has a plurality of buttons or the like arranged for operating or adjusting the machine 30. <Videoanleitung Vorrichtung zur Erstellung einer Videoanleitung 10>

[0023] The video instruction creation device 10 is a known computer comprising a control unit 11, an input unit 12, a display unit 13, and a storage unit 14. The control unit 11 includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, a frame acquisition unit 113, and a video creation unit 114. <Eingabeeinheit 12>

[0024] The input unit 12 is, for example, a keyboard, a touch panel in the display unit 13 described later, or the like, which receives inputs from a user. <Anzeigeeinheit 13>

[0025] The display unit 13 is, for example, a liquid crystal display or the like. The display unit 13 displays, for example, a video guide created by the video guide creation device 10 based on a user input operation via the input unit 12, as described later. <Speichereinheit 14>

[0026] The storage unit 14 is a solid-state drive (SSD), a hard disk drive (HDD), or the like, on which a machining program for operating the machine 30 and the like are stored. The storage unit 14 can also store an image of the operation panel 31 of the machine 30, which is registered by the machine operation panel registration unit 110 described later, and a signal table or the like, wherein the signal table includes each key of the operation panel 31 and a signal for controlling the operation panel 31 by operating each key, which are registered in association with each other by the signal registration unit 111 described later. The storage unit 14 also includes a video data storage unit 141 and a signal data storage unit 142.

[0027] The video data storage unit 141 stores video data recorded by the camera 20 as described above.

[0028] As described above, the signal data storage unit 142 stores the signal for controlling the operation panel 31 in response to the worker pressing a button on the operation panel 31 of the machine 30. Specifically, the video instruction creation device 10 acquires a signal for controlling the operation panel 31 for each button at a predetermined time interval (e.g., 1 second) via a network (not shown) and stores the acquired signals as signal data in the signal data storage unit 142. <Steuereinheit 11>

[0029] The controller 11, which is well known to those skilled in the art, includes a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), a complementary metal-oxide-semiconductor (CMOS) memory, or other components configured to communicate with each other via a bus.

[0030] The CPU is a processor that controls the entire video instruction creation device 10. The CPU reads a system program and application programs stored in the ROM via the bus and controls the entire video instruction creation device 10 in accordance with the system program and application programs. As shown in Fig. As shown in Figure 1, the control unit 11 is configured to perform the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the frame acquisition unit 113, and the video creation unit 114. The RAM stores various types of data such as temporary calculation data, display data, and the like. The CMOS memory is configured as a non-volatile memory, which is backed up by a battery (not shown) and maintains the stored state even when the video guidance creation device 10 is turned off.

[0031] The machine control panel registration unit 110 registers an image of the control panel 31 of the machine 30.

[0032] In particular, the machine control panel registration unit 110 can, for example, cut out an image area of the control panel 31 of the machine 30 from the images of the video captured by the camera 20 and store and register the cut-out image of the control panel 31 of the machine 30 in the storage unit 14.

[0033] Alternatively, the user can also use a digital camera, a smartphone, or the like to capture the operation panel 31 of the machine 30. The machine operation panel registration unit 110 can capture the image of the operation panel 31 of the machine 30 captured by the user's digital camera or the like, and store and register the captured image of the operation panel 31 of the machine 30 in the storage unit 14.

[0034] Fig. 2 shows an example of the registered image of the control panel 31 of the machine 30.

[0035] The signal registration unit 111 registers each key on the operation panel 31 in the image registered by the machine operation panel registration unit 110 and a signal for controlling the operation panel 31 by an operation of each key on the operation panel 31 in association with each other.

[0036] Specifically, the signal registration unit 111 displays, for example, an image of the operation panel 31 registered in the machine operation panel registration unit 110 on the display unit 13. The signal registration unit 111 arranges, based on the user's input via the input unit 12, a rectangular mark of a signal corresponding to the position of each key on the image of the operation panel 31, as shown in Fig. 3. The signal registration unit 111 receives, in response to a user input operation, key names for the arranged rectangular markers and signal names output when the keys are pressed, generates a registered signal table as shown in Fig. 3, and stores the generated registered signal table in the storage unit 14.

[0037] As in Fig. As shown in Figure 3, the registered signal table has storage areas for “rectangle mark”, “key name” and “signal name”.

[0038] For example, in the “Rectangle Mark” memory area in the registered signal table, a rectangle is stored for each key on the control panel 31. In Fig. 3, the line type of the rectangular marker is different for each key, although the line color (e.g., red, blue, etc.) of the rectangular marker can be different for each key. In the "Key Name" storage area in the registered signal table, a name such as "AUTO" or "EDIT" is stored for each key on which the rectangular marker is located based on user input.

[0039] In the storage area of "Signal Name" in the registered signal table, a signal name such as "Signal A" and "Signal B" is stored, and the signal name is output when each key on which the rectangle mark is arranged is pressed in response to the user's input operation.

[0040] The signal data change time extraction unit 112 extracts a time point of a signal change, e.g., a change from OFF to ON, from ON to OFF, or the like, in the signal data stored in the signal data storage unit 142.

[0041] The signal data change time extraction unit 112 can also identify the signal name of a changed signal.

[0042] The frame detection unit 113 detects, based on the time extracted by the signal data change time extraction unit 112, from the video including an operation performed by the worker on the operation panel 31, an image in which a key corresponding to the changed signal was pressed by the worker.

[0043] For example, in Fig. As shown in Figure 4, the frame detection unit 113 detects an image corresponding to the time ts of a signal change from the video including an operation performed by the worker on the control panel 31. The frame detection unit 113 stores the detected image in the storage unit 14.

[0044] The frame detection unit 113 may detect a frame closest to the signal change time ts, or it may detect a plurality of frames within a predetermined time range (e.g., 1 second, etc.) indicated by a dashed rectangle with the signal change time ts.

[0045] The video creation unit 114 creates a video that displays the button pressed by the worker on the image of the control panel 31 in Fig. 2 registered by the machine operation panel registration unit 110 when the image detected by the frame detection unit 113 is displayed.

[0046] Specifically, for example, the video creation unit 114 processes the video stored in the video creation unit 114 around each frame detected by the frame detection unit 113. The video creation unit 114 identifies, based on the signal name obtained from the signal data change time extraction unit 112 and the registered signal table in Fig. 3, the button pressed by the worker when each detected image is displayed. The video creation unit 114 creates a video that shows the position of the button pressed by the worker on the image of the machine control panel 31 in Fig. 2 registered by the machine operation panel registration unit 110 when each image is displayed with the edited image. The video creation unit 114 stores the created video as a video guide in the storage unit 14.

[0047] Fig. 5 shows an example of the video tutorial display.

[0048] As in Fig. As shown in Figure 5, the video guide displays a video of the worker in the left half of the screen. The video guide also indicates the position of the button pressed by the worker with a thick dashed rectangle, as shown in an enlarged view on the upper driving side, along with an image of the control panel 31 in the right half of the screen. <Videoerstellungsprozess der Vorrichtung zur Erstellung einer Videoanleitung 10>

[0049] Next, with reference to Fig. 6 the sequence of a process for creating a video instruction with the device for creating a video instruction 10 is described.

[0050] Fig. 6 is a flowchart illustrating the process of creating a video instruction with the video instruction creation device 10. The process shown here is executed upon receipt of a video creation instruction from the user.

[0051] In step S11, the machine control panel registration unit 110 registers an image of the control panel 31 of the machine 30.

[0052] In step S12, the signal registration unit 111 registers a registered signal table in which each key on the operation panel 31 in the image registered in step S11 and a signal for controlling the operation panel 31 by an operation of each key on the operation panel 31 are associated with each other.

[0053] In step S13, the signal data change time extraction unit 112 extracts a time of a signal change in the signal data stored in the signal data storage unit 142.

[0054] In step S14, the frame detection unit 113 detects, based on the time extracted in step S13, from the video including the operation performed by the worker on the operation panel 31, an image in which a key corresponding to the changed signal was pressed by the worker.

[0055] In step S15, the video creation unit 114 processes the video stored in the video data storage unit 141 around each image detected in step S14. Based on the signal table registered in step S12, the video creation unit 114 creates a video indicating the position of the button pressed by the worker on the screen of the operation panel 31 registered in step S11 when each image is displayed.

[0056] As described above, the video guidance creating apparatus 10 according to the first embodiment can easily create an easy-to-understand video guidance indicating which key is pressed in combination with a video of the worker's key operation.

[0057] This concludes the description of the first embodiment. <Modifikationsbeispiel der ersten Ausführungsform>

[0058] In the first embodiment, the video instruction creation device 10 creates a video that displays the position of the button pressed by the worker on the control panel screen, which is registered as each image is displayed. However, the video instruction creation device 10 is not limited to this. For example, the video instruction creation device 10 may create a video that displays only the button pressed by the worker as each image is displayed. <Zweite Ausführungsform>

[0059] A second embodiment is described below. In the first embodiment, the video guidance creation device 10 extracts the timing of a signal change and, based on the extracted timing, detects from the video including an operation performed by the worker on the operation panel 31 an image in which a key corresponding to the changed signal was pressed by the worker.However, the second embodiment differs from the first embodiment in that a video instruction making apparatus 10A detects the operation panel 31 and estimates a joint position of the worker from the video, detects a time of use of the operation panel 31 by the worker in the video that is before the extracted time of a signal change based on the detected image area of the operation panel 31 and the estimated joint position of the worker, and detects an image in which the worker pressed the button corresponding to the changed signal based on the detected time of use of the operation panel 31 by the worker.

[0060] Therefore, the video guidance creating apparatus 10A according to the second embodiment can easily create an easy-to-understand video guidance indicating which key was pressed in combination with a video of the worker's key operation.

[0061] The second embodiment is described below.

[0062] Fig. Figure 7 is a diagram showing an example of the configuration of a video instruction creation system according to the second embodiment. Note that the components that perform similar functions to the video instruction creation system 1 in Fig. 1 are designated by the same reference numerals to avoid a detailed description.

[0063] As in Fig. 7, the video instruction making system 1 according to the second embodiment includes the video instruction making device 10A, the camera 20, and the machine 30.

[0064] The camera 20 and the device 30 have similar functions to the camera 20 and the device 30 in the first embodiment.

[0065] The video instruction creation device 10A includes a control unit 11a, the input unit 12, the display unit 13, and the storage unit 14. The control unit 11a includes the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, an operation panel acquisition unit 115, a joint position estimation unit 116, an operation panel usage time acquisition unit 117, a frame acquisition unit 113a, and the video creation unit 114.

[0066] The input unit 12, the display unit 13 and the storage unit 14 have the same functions as the input unit 12, the display unit 13 and the storage unit 14 in the first embodiment.

[0067] The video data storage unit 141 and the signal data storage unit 142 have the same functions as the video data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0068] The control unit 11a, which is well known to those skilled in the art, includes a CPU, a ROM, a RAM, a CMOS memory and the like, which are configured to communicate with each other via a bus.

[0069] The CPU is a processor that controls the entire video instruction creation device 10A. The CPU reads a system program and application programs stored in the ROM via the bus and controls the entire video instruction creation device 10A in accordance with the system program and application programs. As shown in Fig. 7, the control unit 11a is configured to perform the functions of the machine panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the panel acquisition unit 115, the joint position estimation unit 116, the panel usage time acquisition unit 117, the frame acquisition unit 113a, and the video creation unit 114.

[0070] The machine panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the video creation unit 114 have similar functions to the machine panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the video creation unit 114 in the first embodiment.

[0071] The control panel detection unit 115 recognizes, for example, the control panel 31 of the machine 30 based on the video.

[0072] Specifically, the control panel detection unit 115 uses, for example, the video stored in the video data storage unit 141 and the image of the control panel 31 registered in the machine control panel registration unit 110 to detect the image area of the control panel 31 of the machine 30 from the video by publicly known image analysis (e.g., edge detection, etc.).

[0073] The joint position estimation unit 116 estimates a joint position of the worker from the video.

[0074] Specifically, the joint position estimation unit 116 estimates a coordinate, an angle, or the like of the joint of hands, fingers, or the like of the worker in each frame of the video stored in the video data storage unit 141, for example, by using publicly known methods (for example, "Motion detection and classification method from multidimensional time series data" by Kosuke Sugano, Kenta Oku, and Kyoji Kawagoe, DEIM Forum 2016 G4-5 or "Feature extraction using LSTM Autoencoder for multimodal sequential data" by Shohei Uezono and Satoshi Ono, Materials of the Japanese Society for Artificial intelligence, SIG-KBS-B802-01, 2018).

[0075] Fig. 8 shows an example of an image area 200 of the control panel detected by the control panel detection unit 115 and a joint position 210 of the worker estimated by the joint position estimation unit 116. In Fig. 8, the detected image area 200 of the control panel is shown by a bold rectangle and the joint position 210 of a finger of the worker is shown by black dots and a dashed line.

[0076] The control panel usage time acquisition unit 117 acquires, based on the image area 200 of the control panel 31 acquired by the control panel acquisition unit 115 and the joint position 210 of the worker estimated by the joint position estimation unit 116, a time point at which the worker uses the control panel in the video that is before the time point of a change in the signal extracted in the signal data change time extraction unit 112.

[0077] Specifically, the signal controlling the operation panel 31 of the machine 30 may be acquired with a delay from the actual time the worker uses the operation panel due to communication between the video guidance generation device 10A and the machine 30, or due to processing of the video guidance generation device 10, or for other reasons. Therefore, the operation panel usage time acquisition unit 117 determines, for example, whether or not the worker's joint position 210 estimated in the joint position estimation unit 116 exists in the image area 200 of the operation panel 31 acquired by the operation panel acquisition unit 115 in the video before the time of a signal change extracted in the signal data change time extraction unit 112.The operation panel usage time detection unit 117 determines that the worker uses the operation panel 31 when the worker's joint position 210 is within the image range 200 of the operation panel 31, and detects a time at which the corresponding image was captured as the time at which the worker uses the operation panel 31.

[0078] The frame detection unit 113a detects the frame at which the operation panel 31 was operated by the worker based on the time at which the worker used the operation panel 31 detected by the operation panel usage time detection unit 117. <Videoerstellungsprozess der Vorrichtung zur Erstellung einer Videoanleitung 10A>

[0079] Next, with reference to Fig. 9 describes the sequence of a process for creating a video instruction using the device for creating a video instruction 10A.

[0080] Fig. Figure 9 is a flowchart illustrating the process of creating a video with the video instruction creation device 10A. The process shown here is executed upon receiving a video creation instruction from the user.

[0081] Here, the processing in steps S21 to S23 and step S28 is the same as the processing in steps S11 to S13 and step S15 in Fig. 6, and the description of it is omitted.

[0082] In step S24, the control panel detection unit 115 uses the video stored in the video data storage unit 141 and the image of the control panel 31 registered in step S21 to recognize the image area 200 of the control panel 31 of the machine 30 from the video.

[0083] In step S25, the joint position estimation unit 116 estimates the joint position 210 of the worker's hands or fingers from the video stored in the video data storage unit 141.

[0084] In step S26, the control panel usage time detection unit 117 detects, based on the image area 200 of the control panel detected in step S24 and the worker's joint position 210 estimated in step S25, a time of use of the control panel by the worker in the video that is before the time of a signal change extracted in step S23.

[0085] In step S27, the frame detecting unit 113a detects a frame in which a key corresponding to the changed signal was pressed by the worker based on the time extracted in step S26.

[0086] As described above, the video guidance creating apparatus 10A according to the second embodiment can easily create an easy-to-understand video guidance indicating which key was pressed in combination with a video of the worker's key operation.

[0087] The video instruction creation device 10A can also automatically create a video in which the timing of a key press in the video coincides with the timing of the display of the pressed key by taking into account a delay in signal acquisition.

[0088] This concludes the description of the second embodiment. <Modifikationsbeispiel der zweiten Ausführungsform>

[0089] In the second embodiment, the video guidance generation device 10A recognizes, as the time of the worker using the operation panel 31, the time of an image captured when the estimated joint position 210 of the worker is included in the captured image area 200 of the operation panel 31 in the video, which is before the extracted time of a signal change. However, the video guidance generation device 10A is not limited to this.

[0090] As in Fig. For example, as shown in FIG. 10, the video guidance generation device 10A may calculate, as a signal detection delay time, the difference between an estimated end time of an operation of the control panel 31 based on the worker's wrist position and a time at which a last signal change occurred. The video guidance generation device 10A may match the time of displaying the pressed key by detecting an image in which a key corresponding to a changed signal was pressed by the worker, based on the time obtained by subtracting the calculated delay time from the time of a signal change. <Dritte Ausführungsform>

[0091] A third embodiment will now be described. In the first embodiment, the video guidance generating device 10 extracts the time of a signal change and, based on the extracted time, detects from the video including an operation performed by the worker on the operation panel 31 an image in which a key corresponding to the changed signal was pressed by the worker.The second embodiment differs from the first embodiment in that the video instruction creation device 10A detects the operation panel 31 and estimates the worker's joint position from the video, detects a time point in the video when the worker used the operation panel 31 that is before the extracted time point of a signal change based on the detected image area of the operation panel 31 and the estimated worker's joint position, and detects an image in which a button corresponding to the changed signal was pressed by the worker based on the detected time point of the operation panel. In the third embodiment, however, a video instruction creation device 10B captures images including the worker and also captures images from a time point of occurrence to a time point of an alarm in the machine 30.The video guidance creating apparatus 10B differs from the first and second embodiments in that the video guidance creating apparatus creates a video for the time of alarm occurrence in which the key operated by the worker is displayed when the image is displayed in a region of each cropped image.

[0092] Therefore, the video guidance creating apparatus 10B according to the third embodiment can easily create an easy-to-understand video guidance indicating which key was pressed in combination with a video of the worker's key operation.

[0093] The third embodiment is described below.

[0094] Fig. Figure 11 is a diagram showing an example of the configuration of a video instruction creation system according to the third embodiment. Note that the components that perform similar functions to the video instruction creation system 1 in Fig. 1 are designated by the same reference numerals to avoid a detailed description.

[0095] As in Fig. As shown in Fig. 11, the video instruction making system 1 according to the third embodiment includes the video instruction making device 10B, the camera 20, and the machine 30.

[0096] The camera 20 and the device 30 have similar functions to the camera 20 and the device 30 in the first embodiment.

[0097] The video guidance creating device 10B includes a control unit 11b, the input unit 12, the display unit 13, and the storage unit 14. The control unit 11b includes the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, a worker acquisition unit 118, a worker video clip unit 119, an alarm handling video clip unit 120, the frame acquisition unit 113, and a video creation unit 114b.

[0098] The input unit 12, the display unit 13 and the storage unit 14 have the same functions as the input unit 12, the display unit 13 and the storage unit 14 in the first embodiment.

[0099] The video data storage unit 141 and the signal data storage unit 142 have similar functions to the video data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0100] The control unit 11b, which is well known to those skilled in the art, includes a CPU, a ROM, a RAM, a CMOS memory, and the like, which are configured to communicate with each other via a bus.

[0101] The CPU is a processor that controls the entire video instruction creation device 10B. The CPU reads a system program and application programs stored in the ROM via the bus and controls the entire video instruction creation device 10B in accordance with the system program and application programs. As shown in Fig. 11, the control unit 11b is configured to perform the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the worker acquisition unit 118, the worker video clip unit 119, the alarm handling video clip unit 120, the frame acquisition unit 113, and the video creation unit 114b.

[0102] The machine panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 have similar functions to the machine panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 in the first embodiment.

[0103] The worker detection unit 118 detects the worker from a video including the operation performed on the machine 30 and stored in the video data storage unit 141, for example, by using a well-known image analysis (e.g., edge detection, etc.).

[0104] For example, the worker video clip unit 119 cuts out an image including the worker detected by the worker detection unit 118.

[0105] The alarm handling video clip unit 120 cuts images from a time of occurrence to a time of triggering an alarm in the machine 30.

[0106] Specifically, the alarm handling video clip unit 120 receives an instruction from the user via the input unit 12 to create a video guide on how to handle an alarm indicating, for example, an abnormality in a main shaft, interference between a tool and workpiece, or the like. The alarm handling video clip unit 120 clips the images from the time of occurrence to the time of triggering an alarm in the signal data stored in the signal data storage unit 142.

[0107] The video creation unit 144b creates a video indicating the button operated by the worker when the image acquired by the frame acquisition unit 113 is displayed in a range of the images clipped in the worker video clip unit 119 and the alarm handling video clip unit 120. <Videoerstellungsprozess der Vorrichtung zur Erstellung einer Videoanleitung 10B>

[0108] Next, with reference to Fig. 12 describes the flow of a process for creating a video instruction in the device for creating a video instruction 10B.

[0109] Fig. 12 is a flowchart illustrating the video instruction creation process by the video instruction creation device 10B. The process shown here is executed upon receiving a video creation instruction from the user.

[0110] Here, the processing in steps S31 to S33 is the same as the processing in steps S11 to S13 in Fig. 6, and a detailed description of it is omitted.

[0111] In step S34, the worker detection unit 118 recognizes the worker from a video stored in the video data storage unit 141, which includes an operation performed on the machine 30.

[0112] In step S35, the worker video clip unit 119 cuts out the images including the worker recognized in step S34.

[0113] In step S36, the alarm handling video clip unit 120 cuts the images from a time of occurrence to a time of triggering an alarm in the machine 30.

[0114] In step S37, the frame detection unit 113 detects, based on the time extracted in step S33, from the video including the operation performed by the worker on the operation panel (not shown), an image of the video in which a key corresponding to a changed signal was pressed by the worker.

[0115] In step S38, the video creation unit 114b creates a video indicating the key operated by the worker when the image detected in step S37 is displayed in a region of the images cropped in step S35 and the images cropped in step S36.

[0116] As described above, the video guidance creating apparatus 10B according to the third embodiment can easily create an easy-to-understand video guidance indicating which key is pressed in combination with a video of the worker's key operation.

[0117] The video instruction making device 10B can make a video instruction by automatically cutting out a part necessary for the instruction.

[0118] This concludes the description of the third embodiment.

[0119] As described in the first embodiment, the second embodiment, and the third embodiment, the video guidance creating devices 10, 10A, and 10B can easily create an easy-to-understand video guidance indicating which key is pressed in combination with a video of the worker's key operation. <modifikationsbeispiel>

[0120] The first embodiment, the second embodiment, and the third embodiment are configured to be a machine tool including, but not limited to, a numerical control device (not shown). The machine 30 may also be, for example, a robot or a controller that controls the robot.

[0121] In the first embodiment, the second embodiment, and the third embodiment, each function included in the video instruction creation devices 10, 10A, and 10B can be implemented by hardware, software, or a combination thereof. Here, "implemented by software" means "implemented by the computer reading and executing programs."

[0122] The programs can be stored on various types of non-transferable computer-readable media and supplied to the computer. Non-transitory computer-readable media includes various types of tangible storage media. Examples of non-transitory computer-readable media are magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), compact disc read-only memories (CD-ROMs), CD-Rs, CD-RWs, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAMs). The programs can be supplied to the computer via various types of transitory computer-readable media. Examples of transitory computer-readable media are electrical signals, optical signals, and electromagnetic waves.The transitory computer-readable media may deliver the programs to the computer over wired communication channels, such as wires and optical fibers, or over wireless communication channels.

[0123] It should be noted that the steps that describe a program to be recorded on a recording medium include both processes that are executed serially in a described order, as well as processes that are executed in parallel or individually and are not necessarily processed in chronological order.

[0124] Although the present disclosure has been described in detail, the present disclosure is not limited to any of the above-described embodiments. Various additions, replacements, modifications, partial deletions, and the like may be made to these embodiments without departing from the gist of the present disclosure or the conceptual spirit of the present disclosure, which can be understood from the contents defined in the claims and their equivalents. These embodiments may also be practiced in combination. For example, in the above-described embodiments, the order of each operation and the order of each processing are exemplary and not restrictive. This also applies to the case where numerical values or expressions are used in the description of the above-described embodiments.

[0125] In connection with the above embodiments and modification examples, the following additions are further disclosed. (Addendum 1)

[0126] A video instruction creation device (10) comprises: a machine operation panel registration unit (110) configured to register an image of an operation panel (31) of a machine (30); a signal registration unit (111) configured to register each key on the operation panel (31) in the image registered by the machine operation panel registration unit (110) and a signal for controlling the operation panel (31) by an operation of each key on the operation panel (31) in association with each other; a signal data change time extraction unit (112) configured to extract a time of change of the signal registered in the signal registration unit (111);a frame detection unit (113) configured to detect, based on the time extracted by the signal data change time extraction unit (112), a frame in which a key corresponding to the changed signal was pressed by the worker from a video including an operation performed by a worker on the operation panel (31); and a video creation unit (114) configured to generate a video displaying the key pressed by the worker on the image of the operation panel (31) registered by the machine operation panel registration unit (110) when the frame detected by the frame detection unit (113) is displayed. (Addendum 2)

[0127] The video instruction producing device (10A) in Appendix 1 comprises: an operation panel detection unit (115) configured to detect an image area of the operation panel (31) from the video; a joint position estimation unit (116) configured to estimate a joint position of the worker from the video;and an operation panel usage time acquisition unit (117) configured to acquire, based on the image area (200) of the operation panel (31) acquired by the operation panel acquisition unit (115) and the joint position (210) of the worker estimated by the joint position estimation unit (116), a time of the worker using the operation panel in the video that is earlier than the time extracted in the signal data change time extraction unit (112) based on the time acquired by the operation panel usage time acquisition unit (117), wherein the frame acquisition unit (113a) acquires a frame that records the operation performed by the worker. (Addendum 3)

[0128] The video instruction creating device (10B) in Appendix 1 comprises: a worker detection unit (118) configured to detect the worker from the video; a worker video clip unit (119) configured to cut out images including the worker detected by the worker detection unit (118); and an alarm handling video clip unit (120) configured to cut out frames from a time of occurrence to a time of triggering of an alarm in the machine (30), wherein the video creation unit (114b) creates a video indicating the key operated by the worker on the registered image of the operation panel (31) when the frame detected by the frame detection unit (113) is displayed in a range of the frames cut out in the worker video clip unit (119) and the frames cut out in the alarm handling video clip unit (120). (Addendum 4)

[0129] A video instruction creation method for causing a computer to operate as a video instruction creation device (10), the video instruction creation method comprising: a machine operation panel registration step for registering an image of an operation panel (31) of a machine (30); a signal registration step for registering each key on the operation panel (31) in the registered image and a signal for controlling the operation panel (31) by an operation of each key on the operation panel (31) in association with each other; a signal data change time extraction step for extracting a time of change in the registered signal; a frame detection step for detecting, based on the extracted time, from a video including an operation performed by a worker on the operation panel (31), a frame in which a key corresponding to the changed signal was pressed by the worker;and a video creating step of creating a video displaying the key operated by the worker on the registered image of the operation panel (31) when the detected frame is displayed; EXPLANATION OF REFERENCE NUMBERS 1 system for creating a video tutorial 10, 10A, 10B Device for creating a video tutorial 11 controllers 110 Control panel registration unit 111 Signal recording unit 112 Signal data change time extraction unit 113, 113a frame detection unit 114, 114b Video creation unit 115 Control panel detection unit 116 Joint position estimation unit 117 Control panel usage time recording unit 118 Worker Registration Unit 119 Worker Video Clip Unit 120 Alarm Handling Video Clip Unit 12 Input unit 13 Display unit 14 storage unit 141 Video data storage unit 142 Signal data storage unit 20 Camera 30 machines 31 Control panel QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2008-246712

[0003] Cited non-patent literature

[0000] Motion detection and classification method from multidimensional time series data" by Kosuke Sugano, Kenta Oku and Kyoji Kawagoe, DEIM Forum 2016 G4-5

[0074] Feature extraction using LSTM Autoencoder for multimodal sequential data' von Shohei Uezono und Satoshi Ono, Materials of the Japanese Society for Artificial intelligence, SIG-KBS-B802-01, 2018

[0074] < / modifikationsbeispiel>

Claims

[1] A device for creating a video instruction, comprising: a machine control panel registration unit configured to register an image of a control panel of a machine; a signal registration unit configured to register each key on the operation panel in the image registered by the machine operation panel registration unit and a signal for controlling the operation panel by an operation of each key on the operation panel in association with each other; a signal data change time extraction unit configured to extract a time of change in the signal registered in the signal registration unit; a frame detection unit configured to detect, based on the time extracted by the signal data change time extraction unit, from a video including an operation performed by a worker on the operation panel, a frame in which a key corresponding to the changed signal was pressed by the worker; and a video creation unit configured to create a video displaying the key pressed by the worker on the image of the operation panel registered by the machine operation panel registration unit when the image captured by the frame detection unit is displayed. [2] The video instruction creation apparatus of claim 1, further comprising: a control panel detection unit configured to detect an image area of the control panel from the video; a joint position estimation unit configured to estimate a joint position of the worker from the video; and a control panel usage time acquisition unit configured to acquire, based on the image area of the control panel acquired by the control panel acquisition unit and the joint position of the worker estimated by the joint position estimation unit, a time of use of the control panel by the worker in the video that is before the time extracted in the signal data change time extraction unit, wherein the frame detection unit detects a frame capturing the operation performed by the worker based on the time detected by the panel usage time detection unit. [3] The video instruction creation apparatus of claim 1, further comprising: a worker detection unit configured to detect the worker in the video; a worker video clipping unit configured to clip images comprising the worker captured by the worker capture unit; and an alarm handling video clip unit configured to clip images from a time of occurrence to a time of triggering of an alarm in the machine, wherein the video creation unit creates a video indicating the button operated by the worker on the registered image of the operation panel when the image acquired by the frame acquisition unit is displayed in a range of the images clipped in the worker video clip unit and the images clipped in the alarm handling video clip unit. [4] A method of creating a video tutorial for causing a computer to operate as a video tutorial creating device, the method of creating a video tutorial comprising: a machine operation panel registration step of registering an image of an operation panel of a machine; a signal registering step of registering each key on the operation panel in the registered image and a signal for controlling the operation panel by an operation of each key on the operation panel in association with each other; a signal data change time extraction step for extracting a time of change in the registered signal; a frame detection step of detecting, based on the extracted time, from a video including an operation performed by a worker on the control panel, a frame in which a key corresponding to the changed signal was pressed by the worker; and a video creation step of creating a video showing the button pressed by the worker on the registered image of the operation panel when the captured image is displayed.

Images