Programmable logic controller, dynamic image management device, dynamic image management system, dynamic image management method and recording medium

By introducing a clock element and a frame rate measurement unit into the programmable logic controller, and connecting the shooting device and the PLC with a general communication cable, the problem of mismatch between the dynamic image and the log time is solved, and the dynamic image and the PLC record time is realized. match.

CN116685914BActive Publication Date: 2024-11-26MITSUBISHI ELECTRIC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180088693.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-08
Publication Date
2024-11-26
Estimated Expiration
2041-06-08

AI Technical Summary

Technical Problem

In the FA field, dynamic images do not match the log time recorded by PLC, resulting in obstacles in the investigation of the cause of failure. The prior art requires dedicated low-latency shooting trigger lines, which limits the use of common shooting devices.

Method used

A programmable logic controller is designed, including a clock element, a logging unit, a dynamic image data acquisition unit, a frame rate measurement unit and a correction unit. The shooting device and the PLC are connected to the general communication cable to measure and correct the dynamic image. Frame speed ensures time matching.

Benefits of technology

Even if a normal shooting device is used, the time of the moving image can be matched with the log time recorded by the PLC, solving the mismatch caused by clock difference, fluctuations and delays.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116685914B_ABST
    Figure CN116685914B_ABST
Patent Text Reader

Abstract

The PLC (10) comprises: a clock element (120); a log recording unit (101) which records a log together with a recording time in a storage device (110) based on a clock signal emitted by the clock element; a dynamic image data acquisition unit (102) which acquires dynamic image data representing a dynamic image captured by the camera from a camera; a frame rate measurement unit (103) which measures a frame rate of a dynamic image represented by the dynamic image data based on a clock signal emitted by the clock element (120); a correction unit (104) which corrects a time added to the dynamic image represented by the dynamic image data based on the frame rate; and a dynamic image storage unit (105) which stores new dynamic image data (DM) representing a dynamic image to which the time corrected by the correction unit (104) is added in the storage device (110).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a programmable logic controller, a dynamic image management device, a dynamic image management system, a dynamic image management method and a recording medium. Background Art

[0002] In the field of FA (Factory Automation), the cause of a fault that occurred in a production device is sometimes investigated using logs recorded by a PLC (Programmable Logic Controller) and dynamic images of the production device captured by a camera. For example, when a production device performs an abnormal operation, a dynamic image of the abnormal operation is sometimes compared with the log at the time attached to the dynamic image to investigate the cause of the abnormal operation. In addition, as the time attached to the dynamic image, for example, the time when each frame of the dynamic image is captured by the camera, the time when the data of each frame of the dynamic image is obtained by the PLC, etc.

[0003] Here, if there is a mismatch between the time added to the moving image and the time recorded in the log during the cause investigation, there is a possibility that the cause investigation will be hindered, so a technology is needed to match the time added to the moving image with the time recorded in the log.

[0004] Patent document 1 discloses a technique for connecting a shooting device to a PLC via a dedicated low-latency shooting trigger line, thereby synchronizing the time added to a moving image with the time of recording a log with high precision. The technique described in patent document 1 enables the time added to a moving image to match the time of recording a log.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-134985 Summary of the invention

[0006] In the technology described in Patent Document 1, a dedicated imaging trigger line is required to connect the imaging device to the PLC. Therefore, a commonly used imaging device such as a network camera or a web camera cannot be used as the imaging device, and a dedicated imaging device corresponding to the imaging trigger line must be prepared.

[0007] In view of the above circumstances, an object of the present invention is to provide a programmable logic controller or the like which can match the time added to a moving image with the recording time of a log recorded by a PLC even when a conventional imaging device is used.

[0008] In order to achieve the above object, the programmable logic controller involved in the present invention has:

[0009] Clock components;

[0010] a log recording unit that records the log together with the recording time in the first storage device based on the clock signal emitted by the clock element;

[0011] a moving image data acquisition unit that acquires, from a photographing device, moving image data representing a moving image photographed by the photographing device;

[0012] a frame rate measuring unit for measuring a frame rate of a dynamic image represented by the dynamic image data based on a clock signal emitted by the clock element;

[0013] a correction unit that corrects a timing added to a dynamic image represented by the dynamic image data based on the frame rate; and

[0014] The moving image storage unit stores new moving image data representing the moving image to which the time corrected by the correction unit is added in the second storage device.

[0015] Effects of the Invention

[0016] According to the present invention, even if a general imaging device is used, it is possible to match the time added to a moving image with the recording time of a log recorded in a PLC. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a diagram showing the overall configuration of a moving image management system according to Embodiment 1 of the present invention.

[0018] Figure 2 This is a diagram for explaining the mismatch of moving images that should be avoided in the moving image management system according to the first embodiment of the present invention.

[0019] Figure 3 This is a diagram showing the configuration of a PLC according to Embodiment 1 of the present invention.

[0020] Figure 4 This is a diagram for explaining the correction of a moving image performed by the PLC according to the first embodiment of the present invention.

[0021] Figure 5 This is a diagram showing the configuration of a motion picture management device according to Embodiment 1 of the present invention.

[0022] Figure 6 This is a diagram showing an example of a log reference point designation screen displayed by the motion picture management device according to the first embodiment of the present invention.

[0023] Figure 7 This is a diagram showing an example of a moving image reference point designation screen displayed by the moving image management device according to the first embodiment of the present invention.

[0024] Figure 8 This is a flowchart showing an example of operation related to the timing correction of a moving image performed by the PLC according to the first embodiment of the present invention.

[0025] Fig. 9 This is a flowchart showing an example of the operation related to the linked playback of logs and moving images performed by the moving image management device according to the first embodiment of the present invention.

[0026] Fig.10 This is a diagram showing the configuration of a PLC according to Embodiment 2 of the present invention.

[0027] Fig.11 This is a diagram showing data stored in the storage device of the PLC according to the second embodiment of the present invention.

[0028] Fig.12 This is a flowchart showing an example of operation related to timing correction of a dynamic image based on a pattern performed by the PLC according to the second embodiment of the present invention.

[0029] Fig.13 This is a diagram showing an example of a log reference point notification screen displayed by a video management device according to a modified example of the present invention.

[0030] Fig.14 This is a diagram showing an example of a notification screen of a moving image reference point displayed by a moving image management device according to a modified example of the present invention. DETAILED DESCRIPTION

[0031] Below, while referring to the attached Figure 1 The following describes a video management system according to an embodiment of the present invention. In the drawings, the same or equivalent parts are denoted by the same reference numerals.

[0032] (Implementation Method 1)

[0033] One side reference Figure 1 The dynamic image management system 1 involved in the first embodiment is described. The dynamic image management system 1 is installed, for example, in a production site where production equipment is installed. The dynamic image management system 1 includes a PLC 10, a dynamic image management device 20, and a camera 30. As described later, according to the dynamic image management system 1, the time of the dynamic image attached to the production equipment captured by the camera 30 can be matched with the recording time of the log recorded by the PLC 10. The dynamic image management system 1 is an example of the dynamic image management system involved in the present invention.

[0034] The PLC 10 and the dynamic image management device 20 are connected so as to be communicable by, for example, a universal communication cable. Similarly, the PLC 10 and the camera 30 are connected so as to be communicable by, for example, a universal communication cable. The universal communication cable is, for example, a common communication cable such as a universal USB (Universal Serial Bus) cable or a universal network cable. These cables are not dedicated communication cables for specific purposes that achieve extremely low fluctuations, extremely low delays, etc.

[0035] The camera 30 is, for example, a common photographing device such as a commercially available network camera or web camera. Common photographing devices such as network cameras and web cameras are not manufactured with the idea of ​​transmitting dynamic image data with extremely low fluctuations and extremely low delays. The camera 30 continuously photographs the production equipment at a certain time interval. The camera 30 transmits dynamic image data representing dynamic images obtained by continuously photographing the production equipment to the PLC 10. Although Figure 1 Only one camera 30 is shown in the figure, but there may be a plurality of cameras 30. The camera 30 is an example of an imaging device according to the present invention.

[0036] PLC 10 is a programmable logic controller. In addition to being connected to the dynamic image management device 20 and the camera 30, the PLC 10 is also connected to sensors, actuators, etc. that are not shown in the figure. The PLC 10 obtains dynamic image data representing the dynamic image captured by the camera 30 from the camera 30. As described later, the PLC 10 records a log related to its own actions. In addition, as described later, the PLC 10 corrects the time of the dynamic image added to the acquired dynamic image data and saves the dynamic image data. The PLC 10 is an example of a programmable logic controller according to the present invention.

[0037] The dynamic image management device 20 reads the log recorded by the PLC 10 and the dynamic image data stored by the PLC 10, and plays the log and the dynamic image in a linked manner. Linked playback means that when playing the dynamic image, the log at the moment that is currently the playback object is displayed together with the dynamic image. Through this linked playback, for example, the user can confirm what state the production equipment was in when the data indicating the abnormality was recorded in the log by observing the dynamic image played in a linked manner. In addition, conversely, when the user finds that an abnormality has occurred in the production equipment in the dynamic image, he can observe the log played in a linked manner to confirm what data was recorded in the log at the moment the abnormality occurred. The dynamic image management device 20 is an example of a dynamic image management device involved in the present invention.

[0038] Although the details will be described later, in the first embodiment, the video management device 20 corrects the time difference between the log and the video based on the user's designation when performing linked playback, and performs linked playback.

[0039] Next, refer to Figure 2 Next, the mismatch of moving images that should be avoided in the moving image management system 1 will be described.

[0040] First, the "shooting time" layer is explained. In the "shooting time" layer, it is shown that the camera 30 shoots the production equipment at a certain time interval and continuously obtains the captured images. Each vertical line engraved on the "time" axis represents a captured image, that is, a frame in the dynamic image. In the "shooting time" layer, the vertical lines are arranged at a certain interval, so it shows that each frame of the dynamic image is obtained at a certain time interval. Once the camera 30 obtains each frame, it sends the image data representing each frame to the PLC 10.

[0041] However, each frame obtained at a certain time interval when the camera 30 is shooting is not necessarily received at a certain time interval when the PLC 10 is receiving it due to the occurrence of fluctuations. In addition, the clock element used by the camera 30 is different from the clock element used by the PLC 10, resulting in a clock difference, so the time interval between the frames when the camera 30 is shooting and the time interval between the frames when the PLC 10 is receiving are sometimes not equal. In addition, because the camera 30 and the PLC 10 are connected through a common communication cable, a delay is also generated in the transmission of each frame.

[0042] Because of the above situation, the time when the camera 30 captures each frame is usually inconsistent with the time when the PLC 10 receives the image data representing each frame. Therefore, even if you want to associate the dynamic image with the log based on the time when the PLC 10 receives each frame, the time associated with each frame and the time associated with the log are actually inconsistent, resulting in a mismatch. The clock difference, fluctuation, and delay that are the causes of the mismatch are described in detail below.

[0043] First, the clock difference will be described. Figure 2 The layer "When a clock difference occurs" shown indicates the frame interval when the PLC 10 receives when "fluctuation" and "delay" do not occur and only "clock difference" occurs. Although the interval between each frame is constant in the layer "When a clock difference occurs", the frame interval is different from that in "When shooting" due to the clock difference. For example, in the layer "When a clock difference occurs", it is shown that the time of 7 frames when shooting is compared with the time of 7 frames when receiving, and the time of 7 frames when receiving is longer. Therefore, the time of reception does not match the time of shooting, and a mismatch occurs when the dynamic image is associated with the log.

[0044] Next, fluctuations will be described. Figure 2 The "When Fluctuation Occurs" layer shown shows the frame interval when the PLC 10 receives when "clock difference" and "delay" do not occur and only "fluctuation" occurs. As shown in the "When Fluctuation Occurs" layer, the frame interval during reception becomes unstable due to fluctuations and fluctuates. Therefore, the time of reception is inconsistent with the time of shooting, and a mismatch occurs when the dynamic image is associated with the log.

[0045] Next, the delay is described. Figure 2 The "delay" layer shown shows the frame interval when the PLC 10 receives when "clock difference" and "fluctuation" do not occur, and only "delay" occurs. In the "delay" layer, although the frame interval is constant and equal to the interval when shooting, the reception time of each frame is later than when shooting. This "late", that is, delay, is caused by, for example, delays caused by communication cables and communication interfaces. Therefore, the time of reception is inconsistent with the time of shooting, which causes a mismatch when associating dynamic images with logs. In addition, because the delay is caused by communication cables, communication interfaces, etc., it can be expected that it is roughly constant.

[0046] exist Figure 2 In each layer of the present invention, the case where only one of "clock difference", "fluctuation" and "delay" occurs is described, but all three usually occur. Therefore, it is necessary to take measures to eliminate the mismatch.

[0047] As described later, the PLC 10 can handle “clock difference” and “fluctuation”, and the dynamic image management device 20 can handle “delay”.

[0048] Next, refer to Figure 3 The structure of the PLC 10 will be described. The PLC 10 includes a processor 100, a storage device 110, a clock element 120, and a communication interface 130. The processor 100, the storage device 110, the clock element 120, and the communication interface 130 are connected to each other via a bus B1 so as to be communicable.

[0049] The processor 100 is, for example, a CPU (Central Processing Unit) and realizes various functions described below by reading and executing a control program DP stored in the storage device 110 .

[0050] The storage device 110 includes, for example, a RAM (Random Access Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive) and other storage devices. The storage device 110 stores logs DL, dynamic image data DM, control programs DP and cache data DB. The logs DL, dynamic image data DM and cache data DB will be described later. In addition, the storage device 110 functions as a working memory when the processor 100 reads and executes the control program DP. The storage device 110 is an example of the first storage device and the second storage device according to the present invention.

[0051] The clock element 120 generates a clock signal used when the processor 100, the storage device 110, and the communication interface 130 operate. The clock element 120 is an example of a clock element according to the present invention.

[0052] The communication interface 130 is a common communication interface such as a USB interface or a network interface. By connecting the moving image management device 20 and the camera 30 to the communication interface 130 , the PLC 10 can communicate with the moving image management device 20 and the camera 30 .

[0053] The functions realized by the processor 100 reading and executing the control program DP are described below. The processor 100 reads and executes the control program DP, and includes a log recording unit 101, a moving image data acquisition unit 102, a frame rate measurement unit 103, a correction unit 104, and a moving image storage unit 105 as functional units.

[0054] The log recording unit 101 records the logs of sensor data obtained from the sensor not shown, data indicating the internal state of the PLC 10, etc., as logs DL in the storage device 110. When recording the logs, the log recording unit 101 records the logs together with the recording time based on the clock signal issued by the clock element 120. The log recording unit 101 is an example of the log recording unit involved in the present invention.

[0055] The dynamic image data acquisition unit 102 acquires dynamic image data representing the dynamic image captured by the camera 30, and saves it in the storage device 110 as a cache data DB, thereby caching the dynamic image data. At this time, the dynamic image data acquisition unit 102 also saves the reception time of each frame in the cache data DB. In addition, the reception time of each frame is obtained based on the clock signal emitted by the clock element 120. Here, an upper limit capacity is set for the cache data DB, and when the data size of the cache data DB reaches the upper limit capacity, the oldest data is deleted. The cache data DB is implemented by, for example, a data structure of a ring buffer. The data size of the cache data DB is, for example, a data size that can store 30 minutes of dynamic images. The dynamic image data acquisition unit 102 is an example of a dynamic image data acquisition unit involved in the present invention.

[0056] The frame rate measuring unit 103 reads the moving image data buffered in the storage device 110, and measures the frame rate of the moving image indicated by the moving image data based on the clock signal from the clock element 120. The frame rate measuring unit 103 is an example of a frame rate measuring means according to the present invention.

[0057] Below, while referring to Figure 4 The measurement of the frame rate is described. First, as a premise, it is assumed that each frame of the dynamic image is assigned a frame number that is connected in a time sequence. For example, the frame number assigned to the first frame captured after the camera 30 is activated is set to 0, the frame number of the frame captured next to the frame number 0 is set to 1, and the frame number of the next frame is set to 2...

[0058] Figure 4 The black dots shown are line graphs showing the correspondence between the frame number and the reception time of each frame in the dynamic image data stored in the buffer data DB. Figure 4 The straight line passing through the vicinity of the black dot shown is a straight line obtained by applying the least square method to each frame number and each reception time. This straight line is represented by the following equation (1), for example.

[0059] y=0.0169x+D…Formula (1)

[0060] Here, x is the frame number, y is the reception time, and D is the y intercept. The coefficient of x in this formula, "0.0169", is the average frame interval. That is, its reciprocal is the frame rate. Therefore, measuring the frame rate is equivalent to solving formula (1).

[0061] Here, the receiving time in the cache data DB is added based on the clock signal emitted by the clock element 120. Therefore, the above-mentioned formula (1) is also obtained based on the clock signal. Therefore, formula (1) is obtained based on the clock signal, and for the receiving time obtained by formula (1), the problem related to "clock difference" is eliminated. In addition, because formula (1) is a linear function of the frame number x, the difference in the receiving time obtained by formula (1) between each frame is constant. Therefore, for the receiving time obtained by formula (1), the problem related to "fluctuation" is also eliminated. That is, for the receiving time obtained by formula (1), the two problems of "clock difference" and "fluctuation" are dealt with.

[0062] As described later, since the problem of "delay" is handled by the dynamic image management device 20, it is only necessary to deal with the two problems of "clock difference" and "fluctuation" at this stage. In short, the frame rate measurement unit 103 measures the frame rate of the dynamic image represented by the dynamic image data based on the clock signal issued by the clock element 120 (that is, by solving the formula (1)), thereby dealing with the two problems of "clock difference" and "fluctuation".

[0063] Refer again Figure 3 The correction unit 104 corrects the reception time of each frame added to the dynamic image by using the formula (1) based on the frame rate measured by the frame rate measuring unit 103. Although the time corrected by the correction unit 104 does not take the delay into consideration, as described above, the delay is processed by the dynamic image management device 20. The correction unit 104 is an example of the correction unit involved in the present invention.

[0064] When a trigger is generated, the dynamic image storage unit 105 extracts dynamic image data near the trigger generation time from the dynamic image data contained in the cache data DB, and saves the new dynamic image data to which the reception time of each frame corrected by the correction unit 104 is attached as dynamic image data DM in the storage device 110. The trigger mentioned here refers to a signal for instructing the dynamic image storage unit 105 to save the dynamic image. For example, when a log related to an abnormality of production equipment is recorded in the log DL, there is a high possibility that a dynamic image before and after the abnormality of the production equipment is required, and a trigger is generated. Then, the dynamic image storage unit 105 saves the dynamic image corresponding to the generation of the trigger. The vicinity of the trigger generation time is, for example, 2 minutes before to 2 minutes after the trigger generation time. The dynamic image storage unit 105 is an example of a dynamic image storage unit involved in the present invention.

[0065] Next, refer to Figure 5The structure of the dynamic image management device 20 will be described. The dynamic image management device 20 includes a processor 200, a storage device 210, an input device 220, a display device 230, and a communication interface 240. The processor 200, the storage device 210, the input device 220, the display device 230, and the communication interface 240 are connected to each other via a bus B2 so as to be able to communicate. The dynamic image management device 20 is, for example, a computer such as a personal computer or a smart phone.

[0066] The processor 200 is, for example, a CPU. The processor 200 reads and executes a control program stored in the storage device 210 to implement each function described below.

[0067] The storage device 210 includes, for example, a RAM, a HDD, an SSD, and other storage devices. The storage device 210 stores a control program executed by the processor 200. The control program is, for example, a program of an engineering design tool provided by the manufacturer of the PLC 10. In addition, the storage device 110 functions as a working memory when the processor 200 reads and executes the control program.

[0068] The input device 220 outputs a signal based on an input operation from a user. The input device 220 is, for example, a touch screen integrated with a display device 230 described later. Alternatively, the input device 220 may be an input device such as a keyboard or a mouse.

[0069] The display device 230 displays information that the user should know based on the control of the display control unit 204 described later. The display device 230 is, for example, a touch screen integrated with the input device 220. Alternatively, the display device 230 may also be an independent display device. The display device 230 is an example of a display unit according to the present invention.

[0070] The communication interface 240 is a common communication interface such as a USB interface or a network interface. By connecting the PLC 10 to the communication interface 240 , the dynamic image management device 20 can communicate with the PLC 10 .

[0071] The functions realized by the processor 200 reading and executing the control program stored in the storage device 210 are described below. The processor 200 reads and executes the control program, and thus includes a data acquisition unit 201, a log reference point designation unit 202, a moving image reference point designation unit 203, and a display control unit 204 as functional units.

[0072] The data acquisition unit 201 communicates with the PLC 10 via the communication interface 240 , and acquires the log DL and the moving image data DM stored in the storage device 110 of the PLC 10 .

[0073] The log reference point designation unit 202 designates a log reference point, that is, a time-series reference point in the log DL, based on the user's input. For example, the log reference point designation unit 202 causes the display control unit 204, which will be described later, to display the following information: Figure 6 The screen shown. Figure 6 The screen shown is a screen that displays a log related to sensor values ​​in a time series as a line graph and prompts the user to specify a time as a log reference point. Figure 6 In the line graph shown in the screen shown, it is assumed that the time 12:00:13.000 is designated as the log reference point. In this case, the user can designate the time 12:00:13.000 as the log reference point by inputting "12:00:13.000" in milliseconds in the text box located to the left of the "OK" button and pressing the "OK" button. The log reference point designation unit 202 is an example of a log reference point designation unit according to the present invention.

[0074] The moving image reference point designation unit 203 designates a time-series reference point in the moving image represented by the moving image data DM, that is, a moving image reference point, based on the user's input. For example, the moving image reference point designation unit 203 causes the display control unit 204 described later to display the following Figure 7 The screen shown. Figure 7 The screen shown is a screen that plays a dynamic image obtained by shooting a production device and prompts the user to specify a time point as a reference point of the dynamic image. Figure 6 When the time of the frame played in the screen shown is designated as the moving image reference point, the time of the frame currently displayed can be designated as the moving image reference point by pressing the "Stop" button to stop the playing of the moving image and then selecting the "Yes" button at the bottom of the screen. The moving image reference point designation unit 203 is an example of a moving image reference point designation unit according to the present invention.

[0075] Here, the user should designate a time point that is considered to be the same as the time point designated as the log reference point as the dynamic image reference point. For example, the user should designate the time point when an abnormality is found in sensor data when checking the log as the log reference point, and the time point when an abnormality is found in production equipment when checking the dynamic image as the dynamic image reference point.

[0076] The display control unit 204 controls the display device 230 so that the log DL and the dynamic image indicated by the dynamic image data DM are played in conjunction with each other in such a manner that the log reference point and the dynamic image reference point are played in conjunction with each other. The display control unit 204 can correct the time difference caused by "delay" and play the log DL and the dynamic image synchronously by playing the log reference point and the dynamic image reference point in conjunction with each other in such a manner that the log reference point and the dynamic image reference point are played in conjunction with each other. Therefore, the problem of "delay" can be handled by the dynamic image management device 20. The display control unit 204 is an example of a display control unit according to the present invention.

[0077] Next, refer to Figure 8 An example of the operation of timing correction of a moving image executed by the PLC 10 will be described below.

[0078] The log recording unit 101 of the PLC 10 records the log DL together with the recording time in the storage device 110 based on the clock signal generated by the clock element 120 (step S101 ).

[0079] The moving image data acquisition unit 102 of the PLC 10 acquires the moving image data from the camera 30 and buffers the moving image data in the storage device 110 (step S102 ).

[0080] The frame rate measuring unit 103 of the PLC 10 measures the frame rate of the moving image represented by the buffered moving image data based on the clock signal generated by the clock element (step S103). Specifically, as described above, the average frame interval is obtained by the least square method.

[0081] The correction unit 104 of the PLC 10 corrects the reception time of each frame added to the moving image based on the measured frame rate (step S104). This correction solves the problems of "clock difference" and "jump".

[0082] The video image storage unit 105 of the PLC 10 determines whether a trigger has been generated (step S105 ). When it is determined that a trigger has not been generated (step S105 : No), the processor 100 of the PLC 10 repeats the operation from step S101 .

[0083] When determining that a trigger has been generated (step S105: Yes), the video storage unit 105 adds the corrected reception time and stores the video data around the trigger generation time as video data DM in the storage device 110 (step S106). Then, the processor 100 repeats the operation from step S101.

[0084] The time correction operation of the dynamic image executed by the PLC 10 eliminates the problems of "clock difference" and "fluctuation". Fig. 9 Next, an example of the operation of linking logs and video playback executed by the video management device 20 will be described. This operation solves the problem of "delay".

[0085] The data acquisition unit 201 of the dynamic image management device 20 reads the log DL from the PLC 10 (step S201 ).

[0086] The log reference point designation unit 202 of the dynamic image management device 20 waits for the user to designate a log reference point (step S202). The log reference point designation unit 202, for example, Figure 6 The screen shown is displayed, and the user is waiting to designate a log reference point.

[0087] The data acquisition unit 201 reads the moving image data DM from the PLC 10 (step S203 ).

[0088] The video reference point designation unit 203 of the video management device 20 waits for the user to designate a video reference point (step S204). The video reference point designation unit 203, for example, Figure 6 The screen shown is displayed, and the user is awaiting designation of a moving image reference point.

[0089] The display control unit 204 of the dynamic image management device 20 plays the dynamic image indicated by the dynamic image data DM and the log DL in a linked manner so that the log reference point and the dynamic image reference point are at the same time (step S205). Through this operation, the "delay" can be corrected and the dynamic image and the log DL can be played in a linked manner in synchronization. Then, when the playback of the dynamic image and the log DL is completed, the processor 200 of the dynamic image management device 20 ends the action of the linked playback.

[0090] The above is a description of the dynamic image management system 1 according to the first embodiment. According to the dynamic image management system 1, first, the frame rate measuring unit 103 of the PLC 10 measures the frame rate based on the clock signal of the clock element 120, and the correction unit 104 of the PLC 10 corrects the reception time of each frame of the dynamic image based on the frame rate, thereby eliminating the problems of "clock difference" and "fluctuation". Next, the log reference point specifying unit 202 of the dynamic image management device 20 allows the user to specify the log reference point, and the dynamic image reference point specifying unit 203 of the dynamic image management device 20 allows the user to specify the dynamic image reference point, and the display control unit 204 of the dynamic image management device 20 plays the dynamic image and the log DL in a linked manner so that the log reference point and the dynamic image reference point are the same time, thereby eliminating the problem of "delay". In addition, when eliminating these problems, the camera 30 can also be applied to a common shooting device. Therefore, according to the video management system 1 according to the first embodiment, even if a general imaging device is used, it is possible to match the time added to the video with the recording time of the log recorded by the PLC.

[0091] (Variation of Implementation Example 1)

[0092] In the first embodiment, the log DL, the dynamic image data DM and the cache data DB are stored in the same storage device 110, but they may be stored in different storage devices. For example, the log DL and the cache data DB may be stored in the storage device 110 of the PLC 10, and the dynamic image data DM may be stored in an external storage device.

[0093] In the first embodiment, the frame rate measuring unit 103 uses the least square method when calculating the frame rate. However, for example, when a large amount of dynamic image data is cached, the computational cost of applying the least square method to all frames is large. Therefore, the frame rate measuring unit 103 may select representative points in units of 50 frames, and apply the least square method only to the selected points. The representative point may be, for example, a point that is the central value among the points in the 50 frames, or an average value of the points may be used.

[0094] On the contrary, when only a small amount of dynamic image data is cached, sometimes the amount of data is too small, and even if the least square method is applied, the error becomes larger. In this case, if the shooting speed of the camera 30 is known, the frame rate measurement unit 103 can also assume the frame rate based on the shooting speed. For example, when the camera 30 can shoot 60 times in 1 second, the frame rate measurement unit 103 can also assume that the frame rate is 60fps. In this case, the coefficient of x in formula (1) is 0.16667. When the amount of data is small, even under such an assumption, the absolute error will not become larger, so it is difficult to cause problems.

[0095] In the first embodiment, the frame rate measuring unit 103 solves the formula (1) based on the frame number and the receiving time. However, when the camera 30 adds the count value obtained by timing the clock counter to each frame, it can also solve the same formula as the formula (1) based on the count value and the receiving time. For example, when the operating frequency of the clock counter of the camera 30 is 90kHz and the frame rate of the dynamic image is 60fps, the count value increases by 1500 for each frame. For example, when the frame rate of the dynamic image captured by the camera 30 is variable, since the frame interval becomes uneven, it is preferable to use the count value. When the count value is used, the unit of the coefficient of x in the formula (1) is not [seconds / frame] but [seconds / clock], and the unit of the frame rate is the reciprocal thereof, that is, [clock / second]. Therefore, although the frame rate obtained when the count value is used instead of the frame number is a unit different from the usual frame rate, it can be regarded as a frame rate expressed by other units.

[0096] In the first embodiment, the motion picture management device 20 includes the input device 220 and the display device 230 , but the input device 220 and the display device 230 may be external devices.

[0097] (Implementation Method 2)

[0098] Below, while referring to the attached Figure 1 The dynamic image management system 1 according to the second embodiment will be described. However, the overall structure of the dynamic image management system 1 is different from Figure 1 The same is true for the embodiment 1 shown.

[0099] In the first embodiment, in order to solve the problem of "delay", the user specifies the log reference point and the dynamic image reference point. However, requiring the user to specify the log reference point and the dynamic image reference point increases the user's workload. In addition, if the user fails to correctly specify the reference point, a mismatch occurs in the linked playback.

[0100] Therefore, in the second embodiment, the PLC 10 also automatically performs corrections for the "delay". In general, the display device displays a predetermined pattern and the camera 30 takes a picture, and the "delay" is corrected in such a way that the time point when the pattern is detected in the dynamic image and the time when the log is recorded when the pattern is displayed become the same. Therefore, in the second embodiment, the reference point designation by the dynamic image management device 20 is not necessary. The following is a specific description.

[0101] First, refer to Fig.10 as well as Fig.11 The differences between the PLC 10 according to the second embodiment and the first embodiment will be described. Fig.11 This is a diagram showing data stored in the storage device 110 of the PLC 10 according to the second embodiment.

[0102] First, the difference between PLC 10 and Embodiment 1 is that it has a display device 140 connected to bus B1. Display device 140 is, for example, a lamp provided on the housing surface of PLC 10. Alternatively, display device 140 may also be a display provided on the housing surface of PLC 10. Display device 140 is an example of a display device according to the present invention.

[0103] Here, the camera 30 needs to be installed at a position where it can capture an image of the display device 140. This is because, as described later, the PLC 10 according to the second embodiment corrects the "delay" using a pattern included in a moving image.

[0104] Next, the functions realized by the processor 100 reading and executing the control program DP are described. The difference from the first embodiment is that the processor 100 has a display control unit 106 and a pattern determination unit 107 as functional units. In addition, as described later, the function of the correction unit 104 is also slightly different.

[0105] In addition, for Fig.11 The data stored in the storage device 110 shown is different from that in Embodiment 1 in that pattern data DT is newly stored. As described above, the functions implemented by the processor 100 reading and executing the control program DP are different from those in Embodiment 1, so the control program DP is also different from that in Embodiment 1.

[0106] The display control unit 106 controls the display device 140 according to the pattern shown in the pattern data DT, so that the display device 140 displays a predetermined pattern. In the case where the display device 140 is a lamp, the pattern is, for example, a pattern that causes the lamp to flash 5 times within 3 seconds. As described later, the pattern determination unit 107 determines whether the dynamic image contains the pattern, so the pattern is preferably easy to determine. In addition, when displaying the pattern, the display control unit 106 outputs the time when the pattern is displayed to the correction unit 104. The display control unit 106 is an example of a display control unit according to the present invention.

[0107] The pattern determination unit 107 refers to the cache data DB and pattern data DT stored in the storage device 110, and determines whether the dynamic image shown by the cached dynamic image data contains a dynamic image obtained by photographing the pattern shown by the pattern data DT. For example, when the pattern data DT is a pattern of flashing a light 5 times within 3 seconds, the pattern determination unit 107 determines whether a dynamic image of flashing a light 5 times within 3 seconds is included in the dynamic image shown by the cached dynamic image data. In addition, when the pattern determination unit 107 determines that a dynamic image obtained by photographing the pattern is included, it outputs the time added to the frame in which the pattern is photographed to the correction unit 104. The pattern determination unit 107 is an example of a pattern determination unit according to the present invention.

[0108] The correction unit 104 according to the second embodiment corrects the time added to the dynamic image based on the time when the pattern is displayed output by the display control unit 106 and the time added to the frame in which the pattern is captured output by the pattern determination unit 107. Specifically, the correction unit 104 corrects the time added to the dynamic image so that the time when the pattern is displayed is the same as the time added to the frame in which the pattern is captured. By such correction, the correction unit 104 according to the second embodiment can also solve the problem of "delay".

[0109] One side reference Fig.12 An example of operation related to the timing correction of a moving image based on a pattern performed by the PLC 10 according to the second embodiment will be described. Fig.12 The actions shown are similar to Figure 8 The time adjustment operation according to the first embodiment shown is performed independently. Figure 8 In the second embodiment, by Figure 8 The problems of "clock difference" and "fluctuation" are eliminated by Fig.12 The action shown here eliminates the problem of "delay".

[0110] The display control unit 106 of the PLC 10 causes the display device 140 to display a pattern (step S301 ). Next, the display control unit 106 outputs the time when the pattern is displayed to the correction unit 104 (step S302 ).

[0111] The moving image data acquisition unit 102 of the PLC 10 acquires the moving image data from the camera 30 and buffers the moving image data in the storage device 110 (step S303 ).

[0112] The determination unit 107 refers to the buffered dynamic image data to determine whether a dynamic image obtained by photographing a pattern is included (step S304). If it is determined that a dynamic image obtained by photographing a pattern is not included (step S304: No), the processor 100 of the PLC 10 repeats the operation starting from step S301.

[0113] When it is determined that a moving image in which a pattern is photographed is included (step S304 : Yes), the pattern determination unit 107 outputs the time added to the frame in which the pattern is photographed to the correction unit 104 (step S305 ).

[0114] The correction unit 104 corrects the time added to the moving image so that the time when the pattern is displayed is the same as the time added to the frame capturing the pattern (step S306). The processor 100 then ends the operation of correcting the time of the moving image based on the pattern.

[0115] The above is a description of the dynamic image management system 1 according to the second embodiment. According to the dynamic image management system 1 according to the second embodiment, by correcting the time added to the dynamic image in such a manner that the time when the display device 140 displays the pattern is the same as the time added to the frame that captured the pattern, the problem of "delay" can be solved without adding any workload to the user. Therefore, according to the dynamic image management system 1 according to the second embodiment, the time added to the dynamic image can be matched with the recording time of the log recorded by the PLC without adding any workload to the user, compared with the first embodiment.

[0116] (Variation of Implementation Example 2)

[0117] In the second embodiment, the PLC 10 has the display device 140, but the PLC 10 may also display a pattern on an external display device without the display device 140. For example, when the camera 30 is set at a position where the PLC 10 cannot be photographed, the same effect as in the second embodiment can be obtained by setting an external display device at a position where the camera 30 can photograph and causing the display device to display a pattern.

[0118] (Implementation method 3)

[0119] In the first and second embodiments, the PLC 10 solves the problems of "clock difference" and "fluctuation". On the other hand, although the accuracy may be degraded, the dynamic image management device 20 can solve all three problems of "clock difference", "fluctuation" and "delay" as described below.

[0120] To briefly describe, in the video management system 1 according to Embodiment 3, the video management device 20 allows the user to specify two reference points for each log and video, and then plays the log and video in conjunction so that the two reference points are at the same time.

[0121] The structure of the dynamic image management device 20 according to the third embodiment Figure 5 The embodiment 1 shown is substantially the same, but differs as described below.

[0122] First, the difference from the first embodiment is that the log reference point designation unit 202 allows the user to designate two log reference points. Hereinafter, these log reference points are referred to as the first log reference point and the second log reference point, respectively.

[0123] Next, the difference from the first embodiment is that the dynamic image reference point designation unit 203 allows the user to designate two dynamic image reference points. Hereinafter, these dynamic image reference points are referred to as the first dynamic image reference point and the second dynamic image reference point, respectively. In addition, the first log reference point and the first dynamic image reference point may be collectively referred to as the first reference point, and the second log reference point and the second dynamic image reference point may be collectively referred to as the second reference point.

[0124] The display control unit 204 is different from Embodiment 1 in that the log and the moving image are played in conjunction so that the first log reference point and the first moving image reference point are the same time, and the second log reference point and the second moving image reference point are the same time. Unlike the case of Embodiment 1, by playing in conjunction based on two reference points, the first reference point and the second reference point, it is possible to eliminate not only the problem of "delay" but also the problems of "clock difference" and "fluctuation" as described below.

[0125] First, the problem caused by the "clock difference" is that the frame interval is different when shooting and when receiving. On the other hand, the display control unit 204 performs linked playback in such a way that the first log reference point and the first dynamic image reference point are the same time, and the second log reference point and the second dynamic image reference point are the same time, so the interval from the first log reference point to the second log reference point is the same as the interval from the first dynamic image reference point to the second dynamic image reference point. Therefore, the average frame interval can be regarded as the same when shooting and receiving. The reason is that the time when shooting corresponds to the time when the log is recorded.

[0126] Next, the problem caused by "fluctuation" is that the frame intervals vary. On the other hand, the display control unit 204 performs linked playback in such a way that the first log reference point and the first dynamic image reference point are at the same time, and the second log reference point and the second dynamic image reference point are at the same time, so that the linked playback can be performed in such a way that the frame intervals are substantially constant. For example, the average frame interval can be calculated based on the time from the first reference point to the second reference point and the total number of frames, so by displaying each frame based on the calculated average frame interval, the variation of the frame interval can be suppressed.

[0127] However, in any of the above methods, if the first reference point and the second reference point are too close in time, there is a possibility that the problems caused by the “clock difference” and the “fluctuation” cannot be fully solved.

[0128] The above is a description of the dynamic image management system 1 according to Embodiment 3. According to the dynamic image management system 1 according to Embodiment 3, by allowing the user to specify the first reference point and the second reference point, the time added to the dynamic image can be matched with the recording time of the log recorded by the PLC without relying on the PLC 10.

[0129] (Other Modifications)

[0130] In each embodiment, the log reference point designation unit 202 of the motion picture management device 20 designates the log reference point based on the user input. Alternatively, the log reference point designation unit 202 may designate the time when a trigger is generated as the log reference point without depending on the user input.

[0131] Specifically, when a trigger is generated, the log recording unit 101 of the PLC 10 records a log associated with the trigger. The time of recording the log associated with the trigger is the time when the trigger is generated. For example, the log recording unit 101 also records the trigger number that identifies the trigger in the log. The log reference point designation unit 202 designates the time when the trigger is generated as the log reference point. Then, the log reference point designation unit 202, for example, Fig.13 The notification screen shown in FIG. 1 is displayed to notify the user that the trigger generation time is designated as the log reference point. Then, the dynamic image reference point designation unit 203 performs the same operation as in the first and second embodiments. Figure 7 The screen shown is displayed.

[0132] In this case, unlike in the other embodiments, it is not necessary to Fig. 9 The waiting operation for designation of the log reference point in step S202 is shown. Therefore, the workload of the user can be reduced.

[0133] In addition, the log reference point designation unit 202 designates the log reference point based on the user's input, while the dynamic image reference point designation unit 203 may designate the time when the trigger is generated as the dynamic image reference point without relying on the user's input. Fig.14 The notification screen shown is displayed to notify the user that the time point currently being displayed is designated as the moving image reference point. The user designates the log reference point while referring to the displayed image.

[0134] Since the time added to the moving image by the moving image storage unit 105 at the current time point does not match the recording time of the log recorded by the log recording unit 101, the image displayed on the notification screen does not match the image captured by the camera 30 at the time of trigger generation. However, the displayed image is an image near the time of trigger generation, and the user can refer to the displayed image while viewing the log near the time of trigger generation, for example Figure 6 Therefore, even if the displayed image is not an image captured at the time of trigger generation, the user can specify a log reference point corresponding to the displayed image from around the time of trigger generation.

[0135] In this case, unlike in the other embodiments, it is not necessary to Fig. 9 The waiting action for designation of the dynamic image reference point in step S204 shown in FIG. 1 can thus reduce the workload of the user. In addition, in this case, Fig. 9 The operation of step S203 shown is performed before the operation of step S201.

[0136] The program used by the PLC 10 or the dynamic image management device 20 can be stored in a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), a USB flash drive, a memory card, or a HDD and distributed. Then, by installing the program on a dedicated computer, i.e., a programmable logic controller or a general-purpose computer, the programmable logic controller or the general-purpose computer can function as the PLC 10 or the dynamic image management device 20.

[0137] Furthermore, the program may be stored in a storage device of another server on the Internet, and downloaded from the server.

[0138] The present invention can be set as various embodiments and variations without departing from the broad spirit and scope of the present invention. In addition, the above-mentioned embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. That is, the scope of the present invention is not represented by the embodiments, but by the claims. Moreover, various variations implemented within the scope of the claims and within the scope of the disclosure equivalent thereto are considered to be within the scope of the present invention.

[0139] Description of the label

[0140] 1 dynamic image management system, 10 PLC, 20 dynamic image management device, 30 camera, 100 processor, 101 log recording unit, 102 dynamic image data acquisition unit, 103 frame rate measurement unit, 104 correction unit, 105 dynamic image storage unit, 106 display control unit, 107 pattern determination unit, 110 storage device, 120 clock element, 130 communication interface, 140 display device, 200 processor, 201 data acquisition unit, 202 log reference point designation unit, 203 dynamic image reference point designation unit, 204 display control unit, 210 storage equipment, 220 input device, 230 display device, 240 communication interface, B1, B2 buses, DB cache data, DL log, DM dynamic image data, DP control program, DT pattern data.

Claims

1. A programmable logic controller, comprising: Clock components; a log recording unit that records the log together with the recording time in the first storage device based on the clock signal emitted by the clock element; a moving image data acquisition unit that acquires, from a photographing device, moving image data representing a moving image photographed by the photographing device; a frame rate measuring unit for measuring a frame rate of a dynamic image represented by the dynamic image data based on a clock signal emitted by the clock element; a correction unit that corrects a time added to a dynamic image shown in the dynamic image data based on the frame rate; a dynamic image storage unit that stores new dynamic image data representing the dynamic image to which the time corrected by the correction unit is added in the second storage device; a display control unit that causes the display device to display a predetermined pattern; as well as a pattern determination unit for determining whether a dynamic image obtained by photographing the pattern is included in the dynamic image indicated by the dynamic image data, When the determination unit determines that the moving image indicated by the moving image data includes a moving image obtained by photographing the pattern, the correction unit corrects the timing added to the moving image indicated by the moving image data based on the timing at which the pattern is displayed by the display control unit.

2. A dynamic image management device, comprising: a log reference point designating unit for designating a log reference point which is a time-series reference point of the log recorded in the first storage device by the log recording unit of the programmable logic controller according to claim 1; a dynamic image reference point designating unit that designates, based on a user input, a dynamic image reference point, which is a temporal reference point of the dynamic image represented by the new dynamic image data stored in the second storage device by the dynamic image storage unit of the programmable logic controller; and The display control unit controls a display device to display the log and the moving image indicated by the new moving image data in a linked manner so that the log reference point and the moving image reference point become the same time.

3. A dynamic image management device, comprising: a log reference point designating unit for designating two time-series reference points of the log recorded in the first storage device by the programmable logic controller according to claim 1, namely, a first log reference point and a second log reference point; a dynamic image reference point designating unit that designates, based on a user input, two temporal reference points, namely, a first dynamic image reference point and a second dynamic image reference point, of the dynamic image represented by the dynamic image data stored in the second storage device by the programmable logic controller; and A display control unit controls a display unit to display the log and the moving image indicated by the moving image data in a linked manner so that the first log reference point and the first moving image reference point are at the same time and the second log reference point and the second moving image reference point are at the same time.

4. A dynamic image management system, comprising: The programmable logic controller of claim 1; and The dynamic image management device as claimed in claim 2.

5. A dynamic image management method, which is a method for managing dynamic images by a programmable logic controller. In the dynamic image management method, Based on a clock signal emitted by a clock element of the programmable logic controller, a log related to the programmable logic controller is recorded in a first storage device together with a recording time. acquiring, from a camera, moving image data representing a moving image captured by the camera, Based on the clock signal generated by the clock element, the frame rate of the dynamic image represented by the dynamic image data is measured, causing the display device to display a predetermined pattern, determining whether the dynamic image indicated by the dynamic image data includes a dynamic image obtained by photographing the pattern, When it is determined that the moving image shown in the moving image data includes a moving image obtained by photographing the pattern, the time at which the moving image shown in the moving image data is added is corrected based on the time at which the pattern is displayed, New moving image data showing the moving image to which the corrected time is added is stored in the second storage device.

6. A computer-readable recording medium storing a program for causing a computer to function as: Clock components; a log recording unit that records the log together with the recording time in the first storage device based on the clock signal emitted by the clock element; a moving image data acquisition unit that acquires, from a photographing device, moving image data representing a moving image photographed by the photographing device; a frame rate measuring unit for measuring a frame rate of a dynamic image represented by the dynamic image data based on a clock signal emitted by the clock element; a correction unit that corrects a time added to a dynamic image shown in the dynamic image data based on the frame rate; a dynamic image storage unit that stores new dynamic image data representing the dynamic image to which the time corrected by the correction unit is added in the second storage device; a display control unit that causes the display device to display a predetermined pattern; as well as a pattern determination unit for determining whether a dynamic image obtained by photographing the pattern is included in the dynamic image indicated by the dynamic image data, When the determination unit determines that the moving image indicated by the moving image data includes a moving image obtained by photographing the pattern, the correction unit corrects the timing added to the moving image indicated by the moving image data based on the timing at which the pattern is displayed by the display control unit.