Dynamic image manual making device and dynamic image manual making method

Abstract

According to the invention, a moving image manual which is easy to understand which button is pressed can be simply manufactured by combining with an image of a button operation of an operator. A moving image manual creation device includes: a machine operation panel registration unit that registers an image of an operation panel of a machine; a signal registration unit that registers each button of the operation panel of the image registered by the mechanical operation panel registration unit in association with a signal for controlling the operation panel by the operation of each button of the operation panel; a signal data change time extraction unit that extracts a time at which the signal registered by the signal registration unit changes; a frame detection unit that detects, on the basis of the time extracted by the signal data change time extraction unit, a frame when a button corresponding to the changed signal is operated by the operator in an image including the operation of the operator on the operation panel; and a moving image creation unit that, when the frame detected by the frame detection unit is displayed, creates a moving image in which the button operated by the operator is displayed on the image of the operation panel registered by the mechanical operation panel registration unit.

Description

Technical Field

[0001] The present invention relates to a dynamic image manual production device and a dynamic image manual production method for producing a dynamic image manual representing the steps of operating a button of a machine. Background Art

[0002] For example, there is known a technology in which a controller having a display screen and performing operations of a molded product take-out machine, for example, reads a dynamic image manual including dynamic image information for explaining the processing of the molded product take-out machine through dynamic images such as real shots, simulated images, or animations of the molded product take-out machine, and displays the dynamic images on the display screen. For example, refer to Patent Document 1.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-246712 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the technology described in Patent Document 1, regarding explanations related to the processing of the machine, the moving image includes real shots of the machine, simulated images, animations, etc., and therefore there is a problem that it takes a lot of time to create the moving image.

[0008] In addition, it is sometimes difficult to know which button is pressed in the video of the operator's operation.

[0009] Furthermore, since a delay occurs when a signal for controlling the machine corresponding to a button operation by an operator is obtained from the machine, the timing of the operator pressing a button and the display of the pressed button may be different in the video.

[0010] Furthermore, editing long-term images acquired by surveillance cameras or the like is extremely troublesome and time-consuming.

[0011] Therefore, it is desirable to be able to simply create a dynamic image manual that makes it easy to understand which button was pressed, in combination with an image of the operator's button operation.

[0012] Means for solving problems

[0013] One aspect of the moving image manual production device of the present disclosure includes: a mechanical operation panel registration unit that registers an image of a mechanical operation panel; a signal registration unit that registers by associating each button of the operation panel of the image registered by the mechanical operation panel registration unit with a signal that controls the operation panel through the operation of each button of the operation panel; a signal data change time extraction unit that extracts the time when the signal registered by the signal registration unit changes; a frame detection unit that, based on the time extracted by the signal data change time extraction unit, detects a frame in an image including an operator's operation of the operation panel when the operator operates a button corresponding to the changed signal; and a moving image production unit that, when the frame detected by the frame detection unit is displayed, produces a moving image in which the button operated by the operator is displayed on the image of the operation panel registered by the mechanical operation panel registration unit.

[0014] One aspect of the moving image manual production method of the present disclosure is a moving image manual production method for causing a computer to function as a moving image manual production device, and includes: a mechanical operation panel registration step of registering an image of a mechanical operation panel; a signal registration step of registering by associating each button of the operation panel of the registered image with a signal that controls the operation panel through the operation of each button of the operation panel; a signal data change time extraction step of extracting the time when the registered signal changes; a frame detection step of, based on the extracted time, detecting a frame in an image including an operator's operation of the operation panel when the operator operates a button corresponding to the changed signal; and a moving image production step of, when the detected frame is displayed, producing a moving image in which the button operated by the operator is displayed on the registered image of the operation panel. Description of the Drawings

[0015] Figure 1 It is a functional block diagram showing a functional configuration example of the moving image manual production system 1 of the first embodiment.

[0016] Figure 2 It is a diagram showing an example of an image of a registered mechanical operation panel.

[0017] Figure 3 It is a diagram showing an example of a registration signal table.

[0018] Figure 4 It is a diagram showing an example of the relationship between an image and the time of signal change.

[0019] Figure 5 It is a diagram showing a display example of a moving image manual.

[0020] Figure 6It is a flowchart for explaining the moving image production process of a moving image manual production device.

[0021] Figure 7 It is a diagram showing an example of the structure of the moving image manual production system according to the second embodiment.

[0022] Figure 8 It is a diagram showing an example of the image area of the operation panel detected by the operation panel detection unit and the joint positions of the operator estimated by the joint position estimation unit.

[0023] Figure 9 It is a flowchart for explaining the moving image production process of a moving image manual production device.

[0024] Figure 10 It is a diagram showing an example of the relationship between the operation panel usage time of the operator, the signal change, and the delay time.

[0025] Figure 11 It is a diagram showing an example of the structure of the moving image manual production system according to the third embodiment.

[0026] Figure 12 It is a flowchart for explaining the moving image production process of a moving image manual production device. Detailed Embodiment

[0027] <First Embodiment>

[0028] First, the outline of this embodiment will be described. In this embodiment, the moving image manual production device registers the image of the operation panel of the machine, and associates each button of the registered operation panel image with the signal that controls the operation panel through the operation of each button of the operation panel for registration. The moving image manual production device extracts the moment of signal change, and based on the extracted moment, detects the frame in the image including the operator's operation on the operation panel when the operator operates the button corresponding to the changed signal. When the detected frame is displayed, the moving image manual production device produces a moving image that displays the button operated by the operator on the registered operation panel screen.

[0029] Thus, according to this embodiment, it is possible to simply produce a moving image manual that is easy to understand which button has been pressed in combination with the image of the operator's button operation.

[0030] The above is the outline of this embodiment.

[0031] Next, the structure of this embodiment will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 It is a functional block diagram showing an example of the functional structure of the moving image manual production system 1 according to the first embodiment.

[0033] As Figure 1 shown, the moving image manual production system 1 includes a moving image manual production device 10, a camera 20, and a machine 30.

[0034] The moving image manual production device 10, the camera 20, and the machine 30 can also be interconnected via a network (not shown) such as a LAN (Local Area Network) or the Internet. Alternatively, the moving image manual production device 10, the camera 20, and the machine 30 can also be directly connected to each other via a connection interface (not shown).

[0035] The camera 20 is a surveillance camera or the like disposed in a factory or the like where the machine 30 is installed. For example, the camera 20 preferably captures a two-dimensional frame obtained by projecting an operation panel 31 included in the machine 30 and an operator operating the operation panel 31 onto a plane perpendicular to the optical axis of the camera 20 at a prescribed frame rate (e.g., 30 fps or the like). In addition, the camera 20 is not limited thereto, and may capture the operation panel 31 and buttons or the like of the operation panel 31 from an arbitrary angle. The camera 20 outputs the captured frame as an image to the moving image manual production device 10 described later. In addition, the image data captured by the camera 20 may also be a visible light image such as an RGB color image, a grayscale image, or a depth image.

[0036] In addition, in Figure 1 this case, the moving image manual production device 10 is connected to one camera 20, but is not limited thereto, and may also be connected to two or more cameras 20.

[0037] The machine 30 is, for example, a machine tool including a numerical control device (not shown) known to those skilled in the art. The machine 30 operates according to a machining program executed by a numerical control device (not shown) and also operates according to an operation of an operator on the operation panel 31 of the machine 30. In addition, the machine 30 outputs a signal for controlling the operation panel 31 generated by an operator's button operation on the operation panel 31 to the moving image manual production device 10 described later.

[0038] In addition, as described later, the machine 30 may also be a robot control device for controlling a robot.

[0039] The operation panel 31 is provided with a plurality of buttons or the like for operating or setting the machine 30.

[0040] <Moving image manual production device 10>

[0041] The dynamic image manual production device 10 is a well-known computer, which has a control unit 11, an input unit 12, a display unit 13, and a storage unit 14. The control unit 11 has a mechanical operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, a frame detection unit 113, and a dynamic image production unit 114.

[0042] <Input unit 12>

[0043] The input unit 12 is, for example, a keyboard, a touch panel arranged on the display unit 13 described later, etc., and accepts inputs from the user.

[0044] <Display unit 13>

[0045] The display unit 13 is, for example, a liquid crystal display, etc. As described later, the display unit 13 displays the dynamic image manual produced by the dynamic image manual production device 10 based on, for example, the input operation of the user via the input unit 12.

[0046] <Storage unit 14>

[0047] The storage unit 14 is an SSD (Solid State Drive) or an HDD (Hard Disk Drive), etc., and stores machining programs for operating the machine 30. In addition, the storage unit 14 can also store the image of the operation panel 31 of the machine 30 registered by the mechanical operation panel registration unit 110 described later, and the signal table, etc., in which each button of the operation panel 31 is associated with the signal for controlling the operation panel 31 by operating each button, registered by the signal registration unit 111 described later. In addition, the storage unit 14 has an image data storage unit 141 and a signal data storage unit 142.

[0048] In the image data storage unit 141, as described above, the image data captured by the camera 20 is stored.

[0049] As described above, the signals for controlling the operation panel 31 by operating the buttons of the operation panel 31 of the machine 30 by the operator are stored in the signal data storage unit 142. Specifically, the dynamic image manual production device 10 acquires the signals for controlling the operation panel 31 of each button at a prescribed time interval (for example, 1 second, etc.) via a network (not shown), and stores the acquired signals as signal data in the signal data storage unit 142.

[0050] <Control unit 11>

[0051] The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CMOS (Complementary Metal-Oxide-Semiconductor) memory, etc., which are configured to be able to communicate with each other via a bus, and are well-known to those skilled in the art.

[0052] The CPU is a processor that overall controls the moving image manual production device 10. The CPU reads out the system program and application program stored in the ROM via the bus, and controls the overall moving image manual production device 10 according to the system program and application program. Thus, as Figure 1 shown, the control unit 11 is configured to implement the functions of the mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the frame detection unit 113, and the moving image production unit 114. Various data such as temporary calculation data and display data are stored in the RAM. In addition, the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that maintains the storage state even when the power of the moving image manual production device 10 is turned off.

[0053] The mechanical operation panel registration unit 110 registers the image of the operation panel 31 of the machine 30.

[0054] Specifically, for example, the mechanical operation panel registration unit 110 can intercept the image area of the operation panel 31 of the machine 30 from the frames of the image captured by the camera 20, and store and register the intercepted image of the operation panel 31 of the machine 30 in the storage unit 14.

[0055] Alternatively, the user uses a digital camera or a smartphone, etc. to photograph the operation panel 31 of the machine 30. The mechanical operation panel registration unit 110 can also obtain the photographed image of the operation panel 31 of the machine 30 from the user's digital camera, etc., and store the obtained image of the operation panel 31 of the machine 30 in the storage unit 14 for registration.

[0056] Figure 2 is a diagram showing an example of the image of the operation panel 31 of the registered machine 30.

[0057] The signal registration unit 111 registers the correspondence between each button of the operation panel 31 of the image registered by the mechanical operation panel registration unit 110 and the signal for controlling the operation panel 31 by operating each button of the operation panel 31.

[0058] Specifically, for example, the signal registration unit 111 displays the image of the operation panel 31 registered by the mechanical operation panel registration unit 110 on the display unit 13. Based on the input operation of the user via the input unit 12, the signal registration unit 111 arranges rectangular marks of signals corresponding to the positions of the respective buttons on the image of the operation panel 31 displayed as shown in Figure 3 . Based on the input operation of the user, the signal registration unit 111 accepts the button name for the arranged rectangular mark and the signal name output by pressing the button, generates the registration signal table shown in Figure 3 , and stores the generated registration signal table in the storage unit 14.

[0059] As shown in Figure 3 , the registration signal table has storage areas for "rectangular mark", "button name", and "signal name".

[0060] In the storage area for "rectangular mark" in the registration signal table, for example, rectangles respectively arranged for the buttons of the operation panel 31 are stored. In addition, in Figure 3 , the line type of the rectangular mark varies for each button, but the color of the line of the rectangular mark (for example, red, blue, etc.) may also vary for each button.

[0061] In the storage area for "button name" in the registration signal table, names such as "automatic" and "edit" for each button with a rectangular mark arranged based on the input operation of the user are stored.

[0062] In the storage area for "signal name" in the registration signal table, signal names such as "signal A" and "signal B" output when each button with a rectangular mark arranged is pressed based on the input operation of the user are stored.

[0063] The signal data change time extraction unit 112 extracts, for example, the time when the signal changes from OFF to ON, from ON to OFF, etc. in the signal data stored in the signal data storage unit 142.

[0064] In addition, the signal data change time extraction unit 112 may also determine the signal name of the changing signal.

[0065] Based on the time extracted by the signal data change time extraction unit 112, the frame detection unit 113 detects the frame when the operator operates the button corresponding to the changing signal in the image including the operator's operation of the operation panel 31.

[0066] Specifically, for example, as shown in Figure 4 , the frame detection unit 113 detects the frame corresponding to the time t of the signal change in the image including the operator's operation of the operation panel 31. The frame detection unit 113 stores the detected frame in the storage unit 14. s

[0067] In addition, the frame detection unit 113 can detect the moment t closest to the signal change s of one frame, or can also detect multiple frames within a specified time range (e.g., 1 second, etc.) shown by the dotted rectangle including the moment t s of the signal change.

[0068] When the frame detected by the frame detection unit 113 is displayed, the moving image production unit 114 produces a moving image showing the buttons operated by the operator on the screen of the operation panel 31 registered by the mechanical operation panel registration unit 110 Figure 2 .

[0069] Specifically, the moving image production unit 114 edits the images stored in the video data storage unit 141 with each frame detected by the frame detection unit 113 as the center, for example. The moving image production unit 114 determines the buttons operated by the operator when each detected frame is displayed based on the signal name determined by the signal data change moment extraction unit 112 and Figure 3 the registered signal table. When the moving image production unit 114 displays each frame together with the edited frames, it produces a moving image showing the positions of the buttons operated by the operator on the screen of the operation panel 31 registered by the mechanical operation panel registration unit 110 Figure 2 . The moving image production unit 114 stores the produced moving image as a moving image manual in the storage unit 14

[0070] Figure 5 is a diagram showing a display example of the moving image manual

[0071] As Figure 5 shown, the moving image manual displays the image of the operator in the left half of the screen. In addition, the moving image manual shows the image of the operation panel 31 in the right half of the screen and, as shown in the enlarged view in the upper right, uses a thick dotted rectangle to indicate the positions of the buttons operated by the operator

[0072] <Moving image production process of the moving image manual production device 10>

[0073] Next, with reference to Figure 6 the process of the moving image production of the moving image manual production device 10 will be described

[0074] Figure 6 is a flowchart for explaining the moving image production process of the moving image manual production device 10. The process shown here is executed by receiving a moving image production instruction from the user

[0075] In step S11, the mechanical operation panel registration unit 110 registers the image of the operation panel 31 of the machine 30

[0076] In step S12, the signal registration unit 111 registers a registration signal table that associates each button of the operation panel 31 of the image registered in step S11 with a signal for controlling the operation panel 31 by operating each button of the operation panel 31.

[0077] In step S13, the signal data change time extraction unit 112 extracts the time when the signal changes from the signal data stored in the signal data storage unit 142.

[0078] In step S14, the frame detection unit 113 detects, based on the time extracted in step S13, a frame in the image including the operator's operation of the operation panel 31 when the operator operates the button corresponding to the changed signal.

[0079] In step S15, the moving image production unit 114 edits the image stored in the image data storage unit 141 centering on each frame detected in step S14. Based on the registration signal table registered in step S12, when displaying each frame, the moving image production unit 114 produces a moving image that displays the position of the button operated by the operator on the screen of the operation panel 31 registered in step S11.

[0080] Accordingly, the moving image manual production device 10 of the first embodiment can easily produce a moving image manual that is easy to understand which button has been pressed in combination with the image of the operator's button operation.

[0081] The above has described the first embodiment.

[0082] <Modification Example of the First Embodiment>

[0083] In the first embodiment, when displaying each frame, the moving image manual production device 10 produces a moving image that displays the position of the button operated by the operator on the screen of the registered operation panel. However, it is not limited thereto. For example, when displaying each frame, the moving image manual production device 10 may also produce a moving image that only displays the button operated by the operator.

[0084] <Second Embodiment>

[0085] Next, the second embodiment will be described. In the first embodiment, the moving image manual creation device 10 extracts the time of signal change, and based on the extracted time, detects the frame in the image including the operator's operation on the operation panel 31 when the operator operates the button corresponding to the changed signal. In contrast, in the second embodiment, the difference between the moving image manual creation device 10A and the first embodiment is that the operation panel 31 is detected from the image and the joint positions of the operator are estimated, and based on the detected image area of the operation panel 31 and the estimated joint positions of the operator, the time when the operator is using the operation panel 31 is detected in the image before the time of the extracted signal change. Based on the detected time when the operator is using the operation panel 31, the frame when the operator operates the button corresponding to the changed signal is detected.

[0086] Thus, the moving image manual creation device 10A of the second embodiment can simply create a moving image manual that is easy to understand which button has been pressed in combination with the image of the operator's button operation.

[0087] Hereinafter, the second embodiment will be described.

[0088] Figure 7 is a diagram showing an example of the configuration of the moving image manual creation system of the second embodiment. In addition, elements having the same functions as those of the elements of the moving image manual creation system 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. Figure 1 of the moving image manual creation system 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0089] As Figure 7 shown, the moving image manual creation system 1 of the second embodiment includes a moving image manual creation device 10A, a camera 20, and a machine 30.

[0090] The camera 20 and the machine 30 have the same functions as the camera 20 and the machine 30 in the first embodiment.

[0091] The moving image manual creation device 10A includes a control unit 11a, an input unit 12, a display unit 13, and a storage unit 14. In addition, the control unit 11a includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, an operation panel detection unit 115, a joint position estimation unit 116, an operation panel use time detection unit 117, a frame detection unit 113a, and a moving image creation unit 114.

[0092] The input unit 12, the display unit 13, and the storage unit 14 have the same functions as the input unit 12, the display unit 13, and the storage unit 14 in the first embodiment.

[0093] The image data storage unit 141 and the signal data storage unit 142 have the same functions as the image data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0094] The control unit 11a includes a CPU, a ROM, a RAM, a CMOS memory, etc., which are configured to be able to communicate with each other via a bus and are well-known to those skilled in the art.

[0095] The CPU is a processor that overall controls the moving image manual production device 10A. The CPU reads out the system program and the application program stored in the ROM via the bus and controls the overall moving image manual production device 10A according to the system program and the application program. Thus, as Figure 7 shown, the control unit 11a is configured to implement the functions of the mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the operation panel detection unit 115, the joint position estimation unit 116, the operation panel usage time detection unit 117, the frame detection unit 113a, and the moving image production unit 114.

[0096] The mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the moving image production unit 114 have the same functions as the mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the moving image production unit 114 in the first embodiment.

[0097] The operation panel detection unit 115 detects, for example, the operation panel 31 of the machine 30 from the image.

[0098] Specifically, the operation panel detection unit 115, for example, uses the image stored in the image data storage unit 141 and the image of the operation panel 31 registered by the mechanical operation panel registration unit 110, and detects the image area of the operation panel 31 of the machine 30 from the image through known image analysis (such as edge detection, etc.).

[0099] The joint position estimation unit 116 estimates the joint position of the operator based on the image.

[0100] Specifically, the joint position estimation unit 116 estimates the coordinates and angles of the joints of the operator's hands, fingers, etc. for each frame of the image stored in the image data storage unit 141 using, for example, a known method (e.g., Kosuke Sugano, Kenta Oku, Kyoji Kawagoe, "Motion detection / classification method based on multidimensional time series data", DEIM Forum 2016G4-5, or Shohei Uezono and Tomoji Ono, "Feature extraction of multimodal series data using LSTM Autoencoder", Artificial Intelligence Society Research Group Materials, SIG-KBS-B802-01, 2018).

[0101] Figure 8 2 is a diagram showing an example of an image area 200 of an operation panel detected by the operation panel detection unit 115 and a joint position 210 of an operator estimated by the joint position estimation unit 116. Figure 8 In FIG. 1 , a detected image region 200 of the operation panel is indicated by a thick rectangle, and joint positions 210 of the operator's fingers are indicated by black circles and dotted lines.

[0102] The operating panel usage time detection unit 117 detects the time when the operator is using the operating panel in the image before the signal change time extracted by the signal data change time extraction unit 112, based on the image area 200 of the operating panel 31 detected by the operating panel detection unit 115 and the joint position 210 of the operator estimated by the joint position estimation unit 116.

[0103] Specifically, due to the communication between the dynamic image manual production device 10A and the machine 30, the processing of the dynamic image manual production device 10, etc., the signal of the operating panel 31 for controlling the machine 30 may be obtained with a delay of, for example, several milliseconds to several seconds, from the time when the operator actually uses the operating panel. Therefore, the operating panel use time detection unit 117 determines whether the joint position 210 of the operator estimated by the joint position estimation unit 116 is in the image area 200 of the operating panel 31 detected by the operating panel detection unit 115, for example, in the image before the time of the signal change extracted by the signal data change time extraction unit 112. When the joint position 210 of the operator is present in the image area 200 of the operating panel 31, the operating panel use time detection unit 117 determines that the operator is using the operating panel 31, and detects the time when the frame is captured as the time when the operator is using the operating panel 31.

[0104] The frame detection unit 113 a detects a frame when the operator is operating, based on the time when the operator is using the operation panel 31 detected by the operation panel use time detection unit 117 .

[0105] <Dynamic Image Production Processing of the Dynamic Image Manual Production Device 10A>

[0106] Next, with reference to Figure 9 the process of the dynamic image production processing of the dynamic image manual production device 10A will be described.

[0107] Figure 9 It is a flowchart for explaining the dynamic image production processing of the dynamic image manual production device 10A. The process shown here is executed by receiving a dynamic image production instruction from the user.

[0108] In addition, the processing of steps S21 to S23 and step S28 is the same as the processing of steps S11 to S13 and step S15 of Figure 6 and the description thereof is omitted.

[0109] In step S24, the operation panel detection unit 115 uses the image of the operation panel 31 stored in the video data storage unit 141 and the image of the operation panel 31 registered in step S21 to detect the image area 200 of the operation panel 31 of the machine 30 from the video.

[0110] In step S25, the joint position estimation unit 116 estimates the joint positions 210 of the operator's hand, fingers, etc. from the video stored in the video data storage unit 141.

[0111] In step S26, the operation panel usage time detection unit 117 detects the time when the operator is using the operation panel in the video before the time of the signal change extracted in step S23 based on the image area 200 of the operation panel detected in step S24 and the joint positions 210 of the operator estimated in step S25.

[0112] In step S27, the frame detection unit 113a detects the frame when the button corresponding to the changed signal is operated by the operator based on the time detected in step S26.

[0113] Based on the above, the dynamic image manual production device 10A of the second embodiment can simply produce a dynamic image manual that is easy to understand which button has been pressed in combination with the video of the operator's button operation.

[0114] In addition, by considering the delay in obtaining the signal, the dynamic image manual production device 10A can produce a dynamic image in which the timing of the action of automatically pressing the button in the video and the display of the pressed button are consistent.

[0115] The above describes the second embodiment.

[0116] <Modification Example of the Second Embodiment>

[0117] In the second embodiment, the moving image manual production device 10A detects, as the time when the operator is using the operation panel 31, the time of the frame captured when the estimated joint position 210 of the operator enters the image area 200 of the detected operation panel 31 in the image before the time of the change in the extracted signal, but it is not limited thereto.

[0118] For example, as Figure 10 shown, the moving image manual production device 10A may also calculate the difference between the time when the operation of the operation panel 31 estimated based on the estimated joint position of the operator ends and the time when the last signal change occurs, as the delay time of the acquired signal. The moving image manual production device 10A may also detect the frame when the operator operates the button corresponding to the changed signal based on the time obtained by subtracting the calculated delay time from the time of the signal change, thereby making the timing of the display of the pressed button consistent.

[0119] <Third Embodiment>

[0120] Next, the third embodiment will be described. In the first embodiment, the moving image manual production device 10 extracts the time of the signal change, and based on the extracted time, detects the frame in the image including the operator's operation of the operation panel 31 when the operator operates the button corresponding to the changed signal. In addition, in the second embodiment, the difference between the moving image manual production device 10A and the first embodiment is that the operation panel 31 is detected from the image and the joint position of the operator is estimated, and based on the detected image area of the operation panel 31 and the estimated joint position of the operator, the time when the operator is using the operation panel is detected in the image before the time of the extracted signal change, and based on the detected time when the operator is using the operation panel, the frame when the operator operates the button corresponding to the changed signal is detected. In contrast, in the third embodiment, the moving image manual production device 10B extracts the frame in which the operator is reflected from the image, and extracts the frame from the occurrence to the elimination of the alarm in the machine 30. The moving image manual production device 10B is different from the first and second embodiments in that, in the moving image manual at the time of the alarm occurrence, when displaying the frames within the range of the extracted frames, a moving image showing the button operated by the operator is produced.

[0121] Thereby, the moving image manual production device 10B of the third embodiment can easily produce a moving image manual that is easy to understand which button has been pressed by combining with the image of the operator's button operation in the moving image manual at the time of the alarm occurrence.

[0122] Hereinafter, the third embodiment will be described.

[0123] Figure 11This is a diagram showing an example of the configuration of the moving image manual production system according to the third embodiment. In addition, elements having the same functions as the elements of the moving image manual production system 1 of Figure 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. Figure 1 Elements having the same functions as the elements of the moving image manual production system 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0124] As Figure 11 shown, the moving image manual production system 1 according to the third embodiment includes a moving image manual production device 10B, a camera 20, and a machine 30.

[0125] The camera 20 and the machine 30 have the same functions as the camera 20 and the machine 30 in the first embodiment.

[0126] The moving image manual production device 10B includes a control unit 11b, an input unit 12, a display unit 13, and a storage unit 14. In addition, the control unit 11b includes a machine operation panel registration unit 110, a signal registration unit 111, a signal data change time extraction unit 112, an operator detection unit 118, an operator image capture unit 119, an alarm response image capture unit 120, a frame detection unit 113, and a moving image production unit 114b.

[0127] The input unit 12, the display unit 13, and the storage unit 14 have the same functions as the input unit 12, the display unit 13, and the storage unit 14 in the first embodiment.

[0128] The image data storage unit 141 and the signal data storage unit 142 have the same functions as the image data storage unit 141 and the signal data storage unit 142 in the first embodiment.

[0129] The control unit 11b includes a CPU, a ROM, a RAM, a CMOS memory, etc., which are configured to be able to communicate with each other via a bus and are well-known to those skilled in the art.

[0130] The CPU is a processor that overall controls the moving image manual production device 10B. The CPU reads out the system program and application program stored in the ROM via the bus and controls the overall moving image manual production device 10B according to the system program and application program. Thus, as Figure 11 shown, the control unit 11b is configured to implement the functions of the machine operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, the operator detection unit 118, the operator image capture unit 119, the alarm response image capture unit 120, the frame detection unit 113, and the moving image production unit 114b.

[0131] The mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 have the same functions as the mechanical operation panel registration unit 110, the signal registration unit 111, the signal data change time extraction unit 112, and the frame detection unit 113 in the first embodiment.

[0132] The operator detection unit 118 detects an operator from the image including the operation of the machine 30 stored in the video data storage unit 141 by using, for example, known image analysis (such as edge detection, etc.).

[0133] The operator image extraction unit 119 extracts, for example, the frame showing the operator detected by the operator detection unit 118.

[0134] The alarm response image extraction unit 120 extracts the frames for the time from when an alarm is generated in the machine 30 until it is eliminated.

[0135] Specifically, the alarm response image extraction unit 120 receives, for example, an instruction for creating a moving image manual related to the response method for alarms such as abnormalities of the main shaft and interference between the tool and the workpiece from the user via the input unit 12. The alarm response image extraction unit 120 extracts the frames for the time from when the received alarm occurs until it is eliminated from the signal data stored in the signal data storage unit 142.

[0136] When the moving image production unit 114b displays the frames detected by the frame detection unit 113 within the frames extracted by the operator image extraction unit 119 and the frames extracted by the alarm response image extraction unit 120, it produces a moving image showing the buttons operated by the operator.

[0137] <Moving Image Production Process of the Moving Image Manual Production Device 10B>

[0138] Next, with reference to Figure 12 the flow of the moving image production process of the moving image manual production device 10B will be described.

[0139] Figure 12 It is a flowchart for explaining the moving image production process of the moving image manual production device 10B. The process shown here is executed by receiving a moving image production instruction from the user.

[0140] In addition, the processing from step S31 to step S33 is the same as the processing from step S11 to step S13 of Figure 6 and the detailed description thereof is omitted.

[0141] In step S34, the operator detection unit 118 detects an operator from the image including the operation of the machine 30 stored in the video data storage unit 141.

[0142] In step S35, the operator image capturing unit 119 captures the frame in which the operator is reflected, which is detected in step S34.

[0143] In step S36, the alarm response image capturing unit 120 captures the frames during the time from when an alarm is generated in the machine 30 until it is eliminated.

[0144] In step S37, based on the time extracted in step S33, the frame detection unit 113 detects the frame in the image containing the operator's operation on the operation panel (not shown) in which the operator has operated the button corresponding to the changed signal.

[0145] In step S38, when the frame detected in step S37 is displayed within the range of the frames captured in step S35 and the frames captured in step S36, the moving image production unit 114b produces a moving image showing the button operated by the operator.

[0146] Accordingly, the moving image manual production device 10B of the third embodiment can easily produce a moving image manual in which it is easy to understand which button has been pressed, in combination with the image of the operator's button operation, in the moving image manual at the time of an alarm.

[0147] In addition, the moving image manual production device 10B can automatically capture the parts required for the manual to produce the moving image manual.

[0148] The above has described the third embodiment.

[0149] As described above, as described in the first embodiment, the second embodiment, and the third embodiment, the moving image manual production devices 10, 10A, and 10B of the present disclosure can easily produce a moving image manual in which it is easy to understand which button has been pressed, in combination with the image of the operator's button operation.

[0150] <Modification Example>

[0151] In the first embodiment, the second embodiment, and the third embodiment, it is assumed to be a machine tool including a numerical control device (not shown), but it is not limited thereto. For example, the machine 30 may also be a robot and a robot control device for controlling the robot.

[0152] In addition, each function included in the moving image manual production devices 10, 10A, and 10B in the first embodiment, the second embodiment, and the third embodiment can be respectively realized by hardware, software, or a combination thereof. Here, realizing by software means realizing by a computer reading and executing a program.

[0153] The program can be stored using various types of non-transitory computer readable media and provided to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). Additionally, the program can also be provided to a computer via various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to a computer via wired communication paths such as wires and optical fibers or wireless communication paths.

[0154] Furthermore, the steps of describing a program recorded on a recording medium of course include processes that are performed in chronological order in that sequence, and also include processes that are executed in parallel or individually even if they are not necessarily performed in chronological order.

[0155] The present disclosure has been described in detail, but the present disclosure is not limited to the above-described respective embodiments. These embodiments can be subjected to various additions, substitutions, changes, partial deletions, etc. within the scope of not departing from the gist of the present disclosure, or within the scope of not departing from the gist of the present disclosure derived from the content described in the claimed technical solution and its equivalents. Additionally, these embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each action and the order of each process are shown as an example and are not limited thereto. Additionally, the same applies to cases where numerical values or mathematical formulas are used in the description of the above-described embodiments.

[0156] Regarding the above-described embodiments and variations, the following supplementary notes are also disclosed.

[0157] (Supplementary Note 1)

[0158] The dynamic image manual production device (10) includes: a mechanical operation panel registration unit (110) that registers an image of the operation panel (31) of the machine (30); a signal registration unit (111) that associates each button of the operation panel (31) of the image registered by the mechanical operation panel registration unit (110) with a signal for controlling the operation panel (31) through the operation of each button of the operation panel (31) and registers them; a signal data change time extraction unit (112) that extracts the time when the signal changes registered by the signal registration unit (111); a frame detection unit (113) that, based on the time extracted by the signal data change time extraction unit (112), detects a frame in the image including the operator's operation of the operation panel (31) when the operator operates the button corresponding to the changed signal; and a dynamic image production unit (114) that, when the frame detected by the frame detection unit (113) is displayed, produces a dynamic image in which the button operated by the operator is displayed on the image of the operation panel (31) registered by the mechanical operation panel registration unit (110).

[0159] (Appendix 2)

[0160] The dynamic image manual production device (10A) of Appendix 1 includes: an operation panel detection unit (115) that detects the image area of the operation panel (31) from the image; a joint position estimation unit (116) that estimates the joint positions of the operator from the image; and an operation panel use time detection unit (117) that, based on the image area (200) of the operation panel (31) detected by the operation panel detection unit (115) and the joint positions (210) of the operator estimated by the joint position estimation unit (116), detects the time when the operator is using the operation panel in the image before the time extracted by the signal data change time extraction unit (112), and the frame detection unit (113a) detects a frame obtained by photographing the operator's operation based on the time detected by the operation panel use time detection unit (117).

[0161] (Appendix 3)

[0162] The dynamic image manual production device (10B) of Appendix 1 includes: an operator detection unit (118) that detects the operator from the image; an operator image capture unit (119) that captures a frame showing the operator detected by the operator detection unit (118); and an alarm response image capture unit (120) that captures a frame for the time from when an alarm is generated in the machine (30) until it is eliminated. When the dynamic image production unit (114b) displays the frame detected by the frame detection unit (113) within the range of the frame captured by the operator image capture unit (119) and the frame captured by the alarm response image capture unit (120), it produces a dynamic image in which the button operated by the operator is displayed on the registered image of the operation panel (31).

[0163] (Supplementary Note 4)

[0164] A method for producing a dynamic image manual, which enables a computer to function as a dynamic image manual production device (10). The method for producing a dynamic image manual includes: a mechanical operation panel registration step of registering an image of an operation panel (31) of a machine (30); a signal registration step of registering each button of the operation panel (31) of the registered image in correspondence with a signal for controlling the operation panel (31) by operating each button of the operation panel (31); a signal data change time extraction step of extracting the time when the registered signal changes; a frame detection step of detecting, based on the extracted time, a frame in an image including an operator's operation on the operation panel (31) when the operator operates a button corresponding to the changed signal; and a dynamic image production step of producing, when the detected frame is displayed, a dynamic image in which the button operated by the operator is displayed on the registered image of the operation panel (31).

[0165] Explanation of Reference Numerals

[0166] 1 Dynamic image manual production system

[0167] 10, 10A, 10B Dynamic image manual production devices

[0168] 11 Control unit

[0169] 110 Operation panel registration unit

[0170] 111 Signal registration unit

[0171] 112 Signal data change time extraction unit

[0172] 113, 113a Frame detection units

[0173] 114, 114b Dynamic image production units

[0174] 115 Operation panel detection unit

[0175] 116 Joint position estimation unit

[0176] 117 Operation panel use time detection unit

[0177] 118 Operator detection unit

[0178] 119 Operator image capture unit

[0179] 120 Alarm response image capture unit

[0180] 12 Input unit

[0181] 13 Display unit

[0182] 14 Storage unit

[0183] 141 Image Data Storage Unit

[0184] 142 Signal Data Storage Unit

[0185] 20 Camera

[0186] 30 Machine

[0187] 31 Operation Panel.

Claims

1. A dynamic image manual production device, characterized in that, the dynamic image manual production device includes: a mechanical operation panel registration unit that registers an image of a mechanical operation panel; a signal registration unit that registers by associating each button of the operation panel of the image registered by the mechanical operation panel registration unit with a signal for controlling the operation panel through the operation of each button of the operation panel; a signal data change time extraction unit that extracts the time when the signal registered by the signal registration unit changes; a frame detection unit that, based on the time extracted by the signal data change time extraction unit, detects a frame in an image including an operator's operation on the operation panel when the operator operates a button corresponding to the changed signal; and a dynamic image production unit that, when displaying the frame detected by the frame detection unit, produces a dynamic image in which the button operated by the operator is displayed on the image of the operation panel registered by the mechanical operation panel registration unit.

2. The dynamic image manual production device according to claim 1, characterized in that, the dynamic image manual production device includes: an operation panel detection unit that detects an image area of the operation panel from the image; a joint position estimation unit that estimates the joint positions of the operator from the image; and an operation panel use time detection unit that, based on the image area of the operation panel detected by the operation panel detection unit and the joint positions of the operator estimated by the joint position estimation unit, detects the time when the operator is using the operation panel in the image before the time extracted by the signal data change time extraction unit, and the frame detection unit detects a frame obtained by photographing the operator's operation based on the time detected by the operation panel use time detection unit.

3. The dynamic image manual production device according to claim 1, characterized in that, the dynamic image manual production device includes: an operator detection unit that detects an operator from the image; an operator image capture unit that captures a frame showing the operator detected by the operator detection unit; and an alarm response image capture unit that captures frames for the time from when an alarm is generated in the machine until it is eliminated, and the dynamic image production unit produces a dynamic image in which the button operated by the operator is displayed on the registered image of the operation panel when displaying the frame detected by the frame detection unit within the range of the frame captured by the operator image capture unit and the frame captured by the alarm response image capture unit.

4. A dynamic image manual production method for causing a computer to function as a dynamic image manual production device, characterized in that, the dynamic image manual production method has: a mechanical operation panel registration step of registering an image of a mechanical operation panel; a signal registration step of registering by associating each button of the registered image of the operation panel with a signal for controlling the operation panel through the operation of each button of the operation panel; a signal data change time extraction step of extracting the time when the registered signal changes; A frame detection step of detecting, based on the extracted time, a frame in an image including an operator's operation on the operation panel when the operator operates a button corresponding to the changed signal; And A moving image production step of producing, when the detected frame is displayed, a moving image in which the button operated by the operator is displayed on the image of the registered operation panel.

Images