Video manual creation device and video manual creation method

Abstract

The present invention makes it possible to easily create a moving image manual which, in combination with a moving image of a button operation of a worker, promotes recognition of the pushed button.　This moving image manual creation device comprises: a machine operation panel registration unit that registers an image of the operation panel of a machine; a signal registration unit that registers each button on the operation panel in the image registered by the machine operation panel registration unit and a signal for controlling the operation panel by an operation of each button on the operation panel in association with each other; a signal data change time extraction unit that extracts a time of a change in the signal registered in the signal registration unit; a frame detection unit that detects, on the basis of the time extracted by the signal data change time extraction unit, from the moving image including an operation performed on the operation panel by the worker, a frame at which a button corresponding to the changed signal was operated by the worker; and a moving image creation unit that creates a moving image displaying the button that was operated by the worker on the image of the operation panel registered by the machine operation panel registration unit when the frame detected by the frame detection unit is displayed.

Description

Technical Field

[0001] The present invention relates to a video manual creation device for creating a video manual showing procedures for operating buttons of a machine, and a video manual creation method.

Background Art

[0002] For example, in a controller having a display screen and performing operations such as operation of a molded product take-out machine, a video manual including video information in which explanations regarding handling of the molded product take-out machine are given by a video such as actual shooting, simulation video, or animation of the molded product take-out machine is read out and video-displayed on the display screen. For example, see Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, since the video includes actual shooting of the machine, simulation video, animation, etc. for explanations regarding handling of the machine, there is a problem that it takes time to create the video. Also, in the video of the operator's operation, it may be difficult to understand which button was pressed. In addition, since there is a delay in acquiring a signal for controlling the machine corresponding to the operator's button operation from the machine, the timing of the operation where the operator presses the button in the video and the timing of displaying the pressed button may be misaligned. Moreover, it is very difficult and time-consuming to edit a long-time video acquired by a surveillance camera or the like.

[0005] Therefore, there is a need to easily create video manuals that clearly show which buttons were pressed, along with footage of the operator pressing the buttons. [Means for solving the problem]

[0006] One embodiment of the video manual creation device of the present disclosure includes: a machine control panel registration unit for registering an image of the machine's control panel; a signal registration unit for registering, in association with each of the buttons on the control panel in the image registered by the machine control panel registration unit and a signal that controls the control panel by operating each of the buttons on the control panel; a signal data change time extraction unit for extracting the time when the signal registered by the signal registration unit changes; a frame detection unit for detecting, based on the time extracted by the signal data change time extraction unit, a frame from a video including the operator's operation of the control panel, in which the operator operates the button corresponding to the changed signal; and a video creation unit for creating a video in which, when the frame detected by the frame detection unit is displayed, the button operated by the operator is displayed on the image of the control panel registered by the machine control panel registration unit.

[0007] One aspect of the video manual creation method of the present disclosure is a video manual creation method for causing a computer to function as a video manual creation device, comprising: a machine control panel registration step of registering an image of the machine's control panel; a signal registration step of registering each of the buttons on the control panel in the registered image and a signal that controls the control panel by operating each of the buttons on the control panel in association with each other; a signal data change time extraction step of extracting the time when the registered signal changes; a frame detection step of detecting, based on the extracted time, a frame from a video including the operator's operation of the control panel, in which the operator operates the button corresponding to the changed signal; and a video creation step of creating a video in which, when the detected frame is displayed, the button operated by the operator is shown on the registered image of the control panel. [Brief explanation of the drawing]

[0008] [Figure 1] This is a functional block diagram showing an example of the functional configuration of the video manual creation system 1 according to the first embodiment. [Figure 2] This figure shows an example of an image of the control panel of a registered machine. [Figure 3] This figure shows an example of a registered signal table. [Figure 4] This figure shows an example of the relationship between the time when the video and signal changed. [Figure 5] This diagram shows an example of how a video manual is displayed. [Figure 6] This is a flowchart explaining the video creation process of a video manual creation device. [Figure 7] This figure shows an example of the configuration of a video manual creation system according to the second embodiment. [Figure 8] This figure shows an example of the image area of ​​the control panel detected by the control panel detection unit and the joint position of the worker estimated by the joint position estimation unit. [Figure 9] This is a flowchart explaining the video creation process of a video manual creation device. [Figure 10] This figure shows an example of the relationship between the operator's control panel usage time, signal changes, and delay time. [Figure 11] This figure shows an example of the configuration of a video manual creation system according to the third embodiment. [Figure 12] This is a flowchart explaining the video creation process of a video manual creation device. [Modes for carrying out the invention]

[0009] <First Embodiment> First, the outline of this embodiment will be described. In this embodiment, the video manual creation device registers images of the machine's control panel and registers each button on the control panel in the registered images in association with the signals that control the control panel based on the operation of each button. The video manual creation device extracts the time when the signal changes and, based on the extracted time, detects the frame in the video, which includes the operator's operation of the control panel, when the operator operates the button corresponding to the changed signal. When the detected frame is displayed, the video manual creation device creates a video that displays the buttons operated by the operator on the registered control panel screen. As a result, according to this embodiment, it is possible to easily create a video manual that clearly shows which button was pressed, along with video footage of the operator's button operations. The above is an overview of this embodiment.

[0010] Next, the configuration of this embodiment will be described in detail with reference to the drawings. Figure 1 is a functional block diagram showing an example of the functional configuration of the video manual creation system 1 according to the first embodiment. As shown in Figure 1, the video manual creation system 1 includes a video manual creation device 10, a camera 20, and a machine 30. The video manual creation device 10, camera 20, and machine 30 may be interconnected via a network (not shown), such as a LAN (Local Area Network) or the Internet. Alternatively, the video manual creation device 10, camera 20, and machine 30 may be directly connected to each other via a connection interface (not shown).

[0011] The camera 20 is a surveillance camera or the like arranged in a factory or the like where the machine 30 is installed. The camera 20 preferably captures, at a predetermined frame rate (e.g., 30 fps or the like), a two-dimensional frame obtained by projecting an operation panel 31 included in the machine 30 and an operator operating the operation panel 31 onto a plane perpendicular to the optical axis of the camera 20. Note that the camera 20, not limited thereto, may capture the operation panel 31 and buttons or the like of the operation panel 31 from an arbitrary angle. The camera 20 outputs the captured frame as a video to a video manual creation device 10 described later. Note that the video data captured by the camera 20 may be a visible light image such as an RGB color image, a grayscale image, or a depth image. In FIG. 1, the video manual creation device 10 is connected to one camera 20, but is not limited thereto, and may be connected to two or more cameras 20.

[0012] The machine 30 is, for example, a machine tool including a numerical control device (not shown) known to those skilled in the art. The machine 30 operates based on a machining program executed by a numerical control device (not shown) and also operates based on an operation of the operation panel 31 of the machine 30 by an operator. Further, the machine 30 outputs a signal for controlling the operation panel 31 generated by a button operation of the operation panel 31 by the operator to a video manual creation device 10 described later. Note that the machine 30 may be a robot control device that controls a robot, as described later. The operation panel 31 has a plurality of buttons or the like for operating or setting the machine 30.

[0013] <Video manual creation device 10> The video manual creation device 10 is a known computer and includes a control unit 11, an input unit 12, a display unit 13, and a storage unit 14. The control unit 11 includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, a frame detection unit 113, and a video creation unit 114.

[0014] <Input unit 12> The input unit 12 is, for example, a keyboard or a touch panel located on the display unit 13 (described later), and accepts input from the user.

[0015] <Display section 13> The display unit 13 is, for example, a liquid crystal display. As will be described later, the display unit 13 displays a video manual created by the video manual creation device 10 based on user input operations via the input unit 12, for example.

[0016] <Storage section 14> The storage unit 14 is an SSD (Solid State Drive) or HDD (Hard Disk Drive), and stores processing programs and the like for operating the machine 30. The storage unit 14 may also store images of the machine's control panel 31, which are registered by the machine control panel registration unit 110 (described later), and signal tables, etc., which are registered by the signal registration unit 111 (described later) that associate each button on the control panel 31 with the signals that control the control panel 31 when each button is operated. The storage unit 14 also has a video data storage unit 141 and a signal data storage unit 142. As described above, the video data storage unit 141 stores video data captured by the camera 20. As described above, the signal data storage unit 142 stores signals that control the control panel 31 of the machine 30 when a button on the control panel 31 is operated by an operator. Specifically, the video manual creation device 10 acquires the control signals for each button on the control panel 31 at predetermined time intervals (for example, 1 second) via a network (not shown), and stores the acquired signals as signal data in the signal data storage unit 142.

[0017] <Control Unit 11> The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), CMOS (Complementary Metal-Oxide-Semiconductor) memory, etc., and these are configured to communicate with each other via a bus, as is well known to those skilled in the art. The CPU is the processor that controls the entire video manual creation device 10. The CPU reads the system program and application program stored in ROM via the bus and controls the entire video manual creation device 10 according to the system program and application program. As a result, as shown in Figure 1, the control unit 11 is configured to realize the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the frame detection unit 113, and the video creation unit 114. Various data such as temporary calculation data and display data are stored in the RAM. In addition, the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that retains its memory state even when the power to the video manual creation device 10 is turned off.

[0018] The machine control panel registration unit 110 registers an image of the control panel 31 of the machine 30. Specifically, the machine control panel registration unit 110 may, for example, extract an image area of ​​the machine control panel 31 from the video frame captured by the camera 20, and store and register the extracted image of the machine control panel 31 in the storage unit 14. Alternatively, the user may take an image of the control panel 31 of the machine 30 using a digital camera or smartphone. The machine control panel registration unit 110 may acquire the image of the control panel 31 of the machine 30 taken from the user's digital camera, etc., and store and register the acquired image of the control panel 31 of the machine 30 in the storage unit 14. Figure 2 shows an example of an image of the control panel 31 of the registered machine 30.

[0019] The signal registration unit 111 registers each button on the control panel 31 in the image registered by the machine control panel registration unit 110, and the signal that controls the control panel 31 when each button on the control panel 31 is operated, in association with each of these. Specifically, the signal registration unit 111 displays, for example, an image of the control panel 31 registered by the machine control panel registration unit 110 on the display unit 13. Based on user input via the input unit 12, the signal registration unit 111 places rectangular marks corresponding to the positions of each button on the displayed image of the control panel 31, as shown in Figure 3. Based on user input, the signal registration unit 111 receives the button name for the placed rectangular marks and the signal name output when the button is pressed, generates a registered signal table as shown in Figure 3, and stores the generated registered signal table in the storage unit 14. As shown in Figure 3, the registered signal table has storage areas for "rectangle mark," "button name," and "signal name." The "rectangle mark" storage area in the registered signal table stores, for example, the rectangles assigned to each button on the control panel 31. In Figure 3, the line type of the rectangle marks is shown to differ for each button, but the line color of the rectangle marks (for example, red or blue) may also be shown to differ for each button. The "Button Name" storage area in the registered signal table stores the names of each button, such as "AUTO" and "EDIT," which are assigned rectangular marks based on user input. The "Signal Name" storage area in the registered signal table stores the names of signals such as "Signal A" and "Signal B" that are output when each button with a rectangular mark is pressed based on user input.

[0020] The signal data change time extraction unit 112 extracts, for example, the time when a signal changes from OFF to ON, or from ON to OFF, etc., in the signal data stored in the signal data storage unit 142. Furthermore, the signal data change time extraction unit 112 may also be configured to identify the signal name of the changed signal.

[0021] The frame detection unit 113 detects, based on the time extracted by the signal data change time extraction unit 112, the frame in the video, including the operator's operation of the control panel 31, when the operator operates the button corresponding to the changed signal. Specifically, the frame detection unit 113 detects, for example, the time t in the video, which includes the operation of the control panel 31 by the operator, when the signal has changed, as shown in Figure 4. s The frame detection unit 113 detects the corresponding frame. The frame detection unit 113 stores the detected frame in the storage unit 14. The frame detection unit 113 detects the time t when the signal changes. s The closest frame may be detected, and the time t of the signal change may be detected. s Multiple frames within a predetermined time range (e.g., 1 second) indicated by a dashed rectangle containing the element may be detected.

[0022] When the frame detection unit 113 displays the frame, the video creation unit 114 creates a video showing the buttons operated by the operator on the screen of the control panel 31 in Figure 2, which is registered by the machine control panel registration unit 110. Specifically, the video creation unit 114 edits the video stored in the video data storage unit 141, for example, focusing on each frame detected by the frame detection unit 113. Based on the signal name identified by the signal data change time extraction unit 112 and the registered signal table in Figure 3, the video creation unit 114 identifies the button operated by the operator when each detected frame is displayed. The video creation unit 114 creates a video that, along with the edited frames, displays the position of the button operated by the operator on the screen of the control panel 31 in Figure 2, which is registered by the machine control panel registration unit 110, when each frame is displayed. The video creation unit 114 stores the created video as a video manual in the storage unit 14. Figure 5 shows an example of how a video manual is displayed. As shown in Figure 5, the video manual displays the operator's image on the left half of the screen. The video manual also displays an image of the control panel 31 on the right half of the screen, and as shown in the enlarged view in the upper right, the positions of the buttons operated by the operator are indicated by thick dashed rectangles.

[0023] <Video creation process of video manual creation device 10> Next, referring to Figure 6, the video creation process of the video manual creation device 10 will be explained. Figure 6 is a flowchart illustrating the video creation process of the video manual creation device 10. The flow shown here is executed when a video creation instruction is received from the user.

[0024] In step S11, the machine control panel registration unit 110 registers an image of the control panel 31 of the machine 30.

[0025] In step S12, the signal registration unit 111 registers a registered signal table that associates each of the buttons on the control panel 31 of the image registered in step S11 with the signals that control the control panel 31 based on the operation of each button on the control panel 31.

[0026] In step S13, the signal data change time extraction unit 112 extracts the time when the signal changed from the signal data stored in the signal data storage unit 142.

[0027] In step S14, the frame detection unit 113 detects, based on the time extracted in step S13, the frame in the video, which includes the operator's operation of the control panel 31, when the operator operates the button corresponding to the changed signal.

[0028] In step S15, the video creation unit 114 edits the video stored in the video data storage unit 141, focusing on each frame detected in step S14. Based on the registered signal table registered in step S12, the video creation unit 114 creates a video that displays the position of the button operated by the operator on the screen of the control panel 31, which was registered in step S11, when each frame is displayed.

[0029] As described above, the video manual creation device 10 according to the first embodiment can easily create a video manual that clearly shows which button was pressed, by combining it with video footage of the operator's button operations. The first embodiment has been described above.

[0030] <Modified form of the first embodiment> In the first embodiment, the video manual creation device 10 created a video that displayed the position of the button operated by the operator on the registered control panel screen when each frame was displayed, but it is not limited to this. For example, the video manual creation device 10 may create a video that displays only the button operated by the operator when each frame is displayed.

[0031] <Second Embodiment> Next, a second embodiment will be described. In the first embodiment, the video manual creation device 10 extracts the time when the signal changed and, based on the extracted time, detects the frame in the video, which includes the operator's operation of the control panel 31, when the operator operates the button corresponding to the changed signal. In contrast, in the second embodiment, the video manual creation device 10A detects the control panel 31 from the video and estimates the operator's joint positions. Based on the detected image region of the control panel 31 and the estimated operator's joint positions, it detects the time in the video prior to the time when the extracted signal changed when the operator was using the control panel 31, and, based on the detected time when the operator was using the control panel 31, it detects the frame in which the operator operates the button corresponding to the changed signal. This differs from the first embodiment. As a result, the video manual creation device 10A according to the second embodiment can easily create a video manual that clearly shows which button was pressed, by combining it with video footage of the operator's button operations. The second embodiment will be described below.

[0032] Figure 7 shows an example of the configuration of the video manual creation system according to the second embodiment. Elements having the same functions as those in the video manual creation system 1 of Figure 1 are denoted by the same reference numerals, and detailed explanations are omitted. As shown in Figure 7, the video manual creation system 1 according to the second embodiment includes a video manual creation device 10A, a camera 20, and a machine 30. The camera 20 and the machine 30 have the same functions as the camera 20 and the machine 30 in the first embodiment.

[0033] The video manual creation device 10A includes a control unit 11a, an input unit 12, a display unit 13, and a storage unit 14. The control unit 11a also includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, an operation panel detection unit 115, a joint position estimation unit 116, an operation panel usage time detection unit 117, a frame detection unit 113a, and a video creation unit 114. The input unit 12, display unit 13, and storage unit 14 have the same functions as the input unit 12, display unit 13, and storage unit 14 in the first embodiment. The video data storage unit 141 and the signal data storage unit 142 have the same functions as the video data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0034] The control unit 11a includes a CPU, ROM, RAM, CMOS memory, etc., which are configured to communicate with each other via a bus, and are known to those skilled in the art. The CPU is a processor that controls the entire video manual creation device 10A. The CPU reads the system program and application program stored in ROM via the bus and controls the entire video manual creation device 10A according to the system program and application program. As a result, as shown in Figure 7, the control unit 11a is configured to realize the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the operation panel detection unit 115, the joint position estimation unit 116, the operation panel usage time detection unit 117, the frame detection unit 113a, and the video creation unit 114. The machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the video creation unit 114 have the same functions as the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the video creation unit 114 in the first embodiment.

[0035] The control panel detection unit 115 detects, for example, the control panel 31 of the machine 30 from the video. Specifically, the control panel detection unit 115 uses, for example, the video stored in the video data storage unit 141 and the image of the control panel 31 registered by the machine control panel registration unit 110 to detect the image region of the machine control panel 31 from the video using known image analysis (e.g., edge detection).

[0036] The joint position estimation unit 116 estimates the joint positions of the worker from the video. Specifically, the joint position estimation unit 116 estimates the coordinates and angles of the joints of the worker's hands, fingers, etc., for each frame of the video stored in the video data storage unit 141, using, for example, known methods (e.g., Kosuke Sugano, Kenta Oku, Koji Kawagoe, "Motion Detection and Classification Method from Multidimensional Time Series Data", DEIM Forum 2016 G4-5, or Shohei Uezono, Satoshi Ono, "Feature Extraction of Multimodal Sequential Data Using LSTM Autoencoder", Artificial Information of the Japanese Society for Artificial Intelligence, SIG-KBS-B802-01, 2018). Figure 8 shows an example of the image region 200 of the control panel detected by the control panel detection unit 115 and the joint positions 210 of the worker estimated by the joint position estimation unit 116. In Figure 8, the detected image region 200 of the control panel is shown as a thick rectangle, and the joint positions 210 of the worker's fingers are shown as black circles and dashed lines.

[0037] The control panel usage time detection unit 117 detects the time when the operator was using the control panel in the video footage prior to the time when the signal extracted by the signal data change time extraction unit 112 changed, based on the image area 200 of the control panel 31 detected by the control panel detection unit 115 and the joint position 210 of the operator estimated by the joint position estimation unit 116. Specifically, the signals controlling the control panel 31 of the machine 30 may be acquired with a delay of several milliseconds to several seconds compared to the time the operator actually used the control panel, due to communication between the video manual creation device 10A and the machine 30, and processing by the video manual creation device 10. Therefore, the control panel usage time detection unit 117 determines, for example, whether the operator's joint position 210, estimated by the joint position estimation unit 116, is present in the image area 200 of the control panel 31 detected by the control panel detection unit 115, in the video footage prior to the time when the signal extracted by the signal data change time extraction unit 112 changed. If the operator's joint position 210 is present in the image area 200 of the control panel 31, the control panel usage time detection unit 117 determines that the operator is using the control panel 31 and detects the time when the frame was captured as the time the operator was using the control panel 31.

[0038] The frame detection unit 113a detects the frames operated by the operator based on the time the operator was using the control panel 31, as detected by the control panel usage time detection unit 117.

[0039] <Video creation process of video manual creation device 10A> Next, referring to Figure 9, we will explain the video creation process of the video manual creation device 10A. Figure 9 is a flowchart illustrating the video creation process of the video manual creation device 10A. The flow shown here is executed when a video creation instruction is received from the user. Note that the processes in steps S21 to S23 and step S28 are the same as the processes in steps S11 to S13 and step S15 in Figure 6, and therefore their explanation is omitted.

[0040] In step S24, the control panel detection unit 115 uses the video stored in the video data storage unit 141 and the image of the control panel 31 registered in step S21 to detect the image region 200 of the control panel 31 of the machine 30 from the video.

[0041] In step S25, the joint position estimation unit 116 estimates the joint positions 210 of the worker's hands, fingers, etc. from the video stored in the video data storage unit 141.

[0042] In step S26, the control panel usage time detection unit 117 detects the time when the operator was using the control panel in the video footage prior to the time when the signal extracted in step S23 changed, based on the image region 200 of the control panel detected in step S24 and the joint position 210 of the operator estimated in step S25.

[0043] In step S27, the frame detection unit 113a detects the frame in which the operator operated the button corresponding to the changed signal, based on the time detected in step S26.

[0044] As described above, the video manual creation device 10A according to the second embodiment can easily create a video manual that clearly shows which button was pressed, along with video footage of the operator's button operations. Furthermore, by taking into account the delay in signal acquisition, the video manual creation device 10A can create a video in which the timing of pressing a button in the video matches the timing of displaying the pressed button. The second embodiment has now been described.

[0045] <Modified form of the second embodiment> In the second embodiment, the video manual creation device 10A detected the time of a frame captured in the video footage prior to the time the extracted signal changed, when the estimated joint position 210 of the operator was within the image area 200 of the detected control panel 31, as the time the operator was using the control panel 31. However, it is not limited to this. For example, as shown in Figure 10, the video manual creation device 10A may calculate the delay time for signal acquisition by taking the difference between the time when the operation of the control panel 31, estimated by the joint position of the estimated worker, is completed and the time when the last change in the signal occurred. The video manual creation device 10A may also synchronize the timing of the display of the pressed button by detecting the frame in which the operator operated the button corresponding to the changed signal, based on the time obtained by subtracting the calculated delay time from the time when the signal changed.

[0046] <Third Embodiment> Next, a third embodiment will be described. In the first embodiment, the video manual creation device 10 extracted the time when the signal changed, and based on the extracted time, detected the frame in the video, which includes the operator's operation of the control panel 31, when the operator operated the button corresponding to the changed signal. In the second embodiment, the video manual creation device 10A detected the control panel 31 from the video and estimated the operator's joint positions, and based on the image area of ​​the detected control panel 31 and the estimated joint positions of the operator, detected the time when the operator was using the control panel in the video prior to the time when the extracted signal changed, and based on the detected time when the operator was using the control panel, detected the frame in which the operator operated the button corresponding to the changed signal. This differs from the first embodiment. In contrast, in the third embodiment, the video manual creation device 10B cuts out frames in which the operator is visible from the video, and also cuts out frames from when an alarm occurs in the machine 30 until it is resolved. The video manual creation device 10B differs from the first and second embodiments in that, in the video manual for when an alarm occurs, it creates a video that displays the buttons operated by the operator when each frame is displayed within the range of each extracted frame. As a result, the video manual creation device 10B according to the third embodiment can easily create a video manual that clearly shows which button was pressed, in conjunction with footage of the operator's button operations when an alarm occurs. The third embodiment will be described below.

[0047] Figure 11 shows an example of the configuration of a video manual creation system according to the third embodiment. Elements having the same functions as those in the video manual creation system 1 of Figure 1 are denoted by the same reference numerals, and detailed explanations are omitted. As shown in Figure 11, the video manual creation system 1 according to the third embodiment includes a video manual creation device 10B, a camera 20, and a machine 30. The camera 20 and the machine 30 have the same functions as the camera 20 and the machine 30 in the first embodiment.

[0048] The video manual creation device 10B includes a control unit 11b, an input unit 12, a display unit 13, and a storage unit 14. The control unit 11b also includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, a worker detection unit 118, a worker video extraction unit 119, an alarm response video extraction unit 120, a frame detection unit 113, and a video creation unit 114b. The input unit 12, display unit 13, and storage unit 14 have the same functions as the input unit 12, display unit 13, and storage unit 14 in the first embodiment. The video data storage unit 141 and the signal data storage unit 142 have the same functions as the video data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0049] The control unit 11b includes a CPU, ROM, RAM, CMOS memory, etc., which are configured to communicate with each other via a bus, and are known to those skilled in the art. The CPU is a processor that controls the entire video manual creation device 10B. The CPU reads the system program and application program stored in ROM via the bus and controls the entire video manual creation device 10B according to the system program and application program. As a result, as shown in Figure 11, the control unit 11b is configured to realize the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the worker detection unit 118, the worker video extraction unit 119, the alarm response video extraction unit 120, the frame detection unit 113, and the video creation unit 114b. The machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 have the same functions as the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 in the first embodiment.

[0050] The worker detection unit 118 detects a worker from video footage, including the operation of the machine 30, stored in the video data storage unit 141, using, for example, known image analysis (e.g., edge detection).

[0051] The worker image extraction unit 119 extracts, for example, frames in which a worker detected by the worker detection unit 118 is visible.

[0052] The alarm response video extraction unit 120 extracts frames from the time when the alarm occurs in machine 30 until it is resolved. Specifically, the alarm response video extraction unit 120 receives instructions from the user, for example via the input unit 12, to create a video manual on how to deal with alarms such as spindle abnormalities or interference between the tool and the workpiece. The alarm response video extraction unit 120 extracts frames from the signal data stored in the signal data storage unit 142, covering the time from when the received alarm occurred until it was resolved.

[0053] The video creation unit 114b creates a video that displays the buttons operated by the worker when it displays the frames detected by the frame detection unit 113 within the range of frames extracted by the worker video extraction unit 119 and frames extracted by the alarm response video extraction unit 120.

[0054] <Video creation process of video manual creation device 10B> Next, referring to Figure 12, the video creation process of the video manual creation device 10B will be explained. Figure 12 is a flowchart illustrating the video creation process of the video manual creation device 10B. The flow shown here is executed when a video creation instruction is received from the user. Note that the processes from steps S31 to S33 are the same as those from steps S11 to S13 in Figure 6, and therefore a detailed explanation is omitted.

[0055] In step S34, the worker detection unit 118 detects a worker from the video data stored in the video data storage unit 141, which includes the operation of the machine 30.

[0056] In step S35, the worker image extraction unit 119 extracts a frame in which the worker detected in step S34 is shown.

[0057] In step S36, the alarm response video extraction unit 120 extracts frames from the time when the alarm occurs in the machine 30 until it is resolved.

[0058] In step S37, the frame detection unit 113 detects, based on the time extracted in step S33, the frames of the video in which the operator operated the button corresponding to the changed signal, including the operator's operation of the control panel (not shown).

[0059] In step S38, the video creation unit 114b creates a video that displays the button operated by the operator when the frame detected in step S37 is displayed within the range between the frame extracted in step S35 and the frame extracted in step S36.

[0060] As described above, the video manual creation device 10B according to the third embodiment can easily create a video manual that clearly shows which button was pressed, in conjunction with video footage of the operator's button operations when an alarm occurs. Furthermore, the video manual creation device 10B can automatically extract the necessary sections of a manual and create a video manual. The third embodiment has now been described.

[0061] As described above, the video manual creation devices 10, 10A, and 10B of the present disclosure can easily create video manuals that clearly show which buttons were pressed, along with video footage of the operator's button operations.

[0062] <Variation> In the first, second, and third embodiments, the machine tool includes a numerical control device (not shown), but the invention is not limited thereto. For example, machine 30 may be a robot or a robot control device that controls the robot.

[0063] Furthermore, each function included in the video manual creation devices 10, 10A, and 10B in the first, second, and third embodiments can be implemented by hardware, software, or a combination thereof. Here, implementation by software means that it is implemented by a computer reading and executing a program.

[0064] Programs can be stored and supplied to a computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Programs may also be supplied to a computer using various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0065] Furthermore, the step of writing the program to be recorded on the recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually.

[0066] While this disclosure has been described in detail, it is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the gist of this disclosure or from the spirit of this disclosure derived from the claims and their equivalents. Furthermore, these embodiments can be implemented in combination. For example, the order of operations and processes in the embodiments described above are given as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0067] The following additional information is disclosed regarding the above embodiments and modifications. (Note 1) The video manual creation device (10) includes: a machine operation panel registration unit (110) for registering images of the operation panel (31) of the machine (30); a signal registration unit (111) for registering each of the buttons on the operation panel (31) in the images registered by the machine operation panel registration unit (110) and the signals that control the operation panel (31) by operating each of the buttons on the operation panel (31) in association with each other; a signal data change time extraction unit (112) for extracting the time when the signals registered by the signal registration unit (111) change; a frame detection unit (113) for detecting frames in the video including the operator's operation of the operation panel (31) when the operator operates the button corresponding to the changed signal, based on the time extracted by the signal data change time extraction unit (112); and a video creation unit (114) for creating a video that displays the buttons operated by the operator on the images of the operation panel (31) registered by the machine operation panel registration unit (110) when the frames detected by the frame detection unit (113) are displayed. (Note 2) The video manual creation device (10A) described in Appendix 1 includes a control panel detection unit (115) that detects the image region of the control panel (31) from the video, a joint position estimation unit (116) that estimates the joint positions of the operator from the video, and a control panel usage time detection unit (117) that detects the time when the operator was using the control panel in the video prior to the time extracted by the signal data change time extraction unit (112), based on the image region (200) of the control panel (31) detected by the control panel detection unit (115) and the joint positions (210) of the operator estimated by the joint position estimation unit (116). The frame detection unit (113a) detects the frame captured when the operator operated the control panel, based on the time detected by the control panel usage time detection unit (117). (Note 3) The video manual creation device (10B) described in Appendix 1 includes a worker detection unit (118) that detects workers from video footage, a worker video extraction unit (119) that extracts frames showing the workers detected by the worker detection unit (118), and an alarm response video extraction unit (120) that extracts frames from the time an alarm occurs in the machine (30) until it is resolved. The video creation unit (114b) displays frames detected by the frame detection unit (113) within the range of frames extracted by the worker video extraction unit (119) and frames extracted by the alarm response video extraction unit (120), and creates a video that displays the buttons operated by the worker on the registered control panel (31) image. (Note 4) A method for creating a video manual to enable a computer to function as a video manual creation device (10), comprising: a machine operation panel registration step of registering an image of the operation panel (31) of a machine (30); a signal registration step of registering each of the buttons on the operation panel (31) in the registered image and a signal that controls the operation panel (31) by operating each of the buttons on the operation panel (31) in association with each other; a signal data change time extraction step of extracting the time when the registered signal changes; a frame detection step of detecting, based on the extracted time, a frame from a video including the operator's operation of the operation panel (31) in which the operator operates the button corresponding to the changed signal; and a video creation step of creating a video in which, when the detected frame is displayed, the button operated by the operator is shown on the registered image of the operation panel (31). [Explanation of symbols]

[0068] 1. Video Manual Creation System 10, 10A, 10B Video Manual Creation Device 11 Control Unit 110 Control Panel Registration Section 111 Signal Registration Section 112 Signal data change time extraction unit 113, 113a Frame detection unit 114, 114b Video Production Department 115 Control panel detection unit 116 Joint position estimation unit 117 Control panel usage time detection unit 118 Worker detection unit 119 Worker video extraction section 120 Alarm response video extraction section 12 Input section 13 Display section 14 Storage section 141 Video data storage unit 142 Signal data storage unit 20 cameras 30 machines 31 Control panel

Claims

1. A machine control panel registration unit that registers images of the machine's control panel, A signal registration unit registers each of the buttons on the control panel in the image registered in the machine control panel registration unit, and the signals that control the control panel by operating each of the buttons on the control panel, in association with each of these. A signal data change time extraction unit extracts the time when the signal registered in the signal registration unit changed, Based on the time extracted by the signal data change time extraction unit, a frame detection unit detects the frame in the video, including the operator's operation of the control panel, when the operator operates the button corresponding to the changed signal. When the frame detected by the frame detection unit is displayed on the display unit, a video creation unit creates a video that displays on the display unit the buttons operated by the operator on the image of the control panel registered by the machine control panel registration unit, A video manual creation device equipped with the following features.

2. A control panel detection unit that detects the image region of the control panel from the aforementioned video, A joint position estimation unit that estimates the joint positions of the worker from the aforementioned video, The system includes a control panel usage time detection unit that detects the time when the operator was using the control panel in the video before the time extracted by the signal data change time extraction unit, based on the image region of the control panel detected by the control panel detection unit and the joint position of the operator estimated by the joint position estimation unit, The video manual creation apparatus according to claim 1, wherein the frame detection unit detects frames captured when the operator operated the control panel, based on the time detected by the control panel usage time detection unit.

3. A worker detection unit that detects workers from the aforementioned video, A worker image extraction unit extracts a frame in which the worker detected by the worker detection unit is shown, The machine includes an alarm response video extraction unit that extracts frames from the time from when an alarm occurs until it is resolved, The video creation unit creates a video in which, when the frame detected by the frame detection unit is displayed on the display unit within the range of the frame extracted by the worker video extraction unit and the frame extracted by the alarm response video extraction unit, the button operated by the worker on the registered image of the control panel is displayed on the display unit.

4. A method for creating a video manual, which causes a computer including a display unit to function as a video manual creation device, A machine control panel registration step to register an image of the machine's control panel, A signal registration step involves registering each of the buttons on the control panel in the registered image and the signals that control the control panel based on the operation of each of the buttons on the control panel, in association with each of these. A signal data change time extraction step that extracts the time when the registered signal changed, A frame detection step, based on the extracted time, detects a frame from the video including the operator's operation of the control panel when the operator operates the button corresponding to the changed signal. A video creation step that creates a video in which the buttons operated by the operator on the registered image of the control panel are displayed on the display unit when the detected frame is displayed on the display unit, A method for creating video manuals that includes the following features.

Images