Manufacturing line monitoring system

Abstract

This invention provides a manufacturing line monitoring system that enables production line managers and others to easily check the status of changes in a manufacturing line equipped with multiple imaging devices, thereby suppressing a decline in productivity. [Solution] The monitoring control device 111 of the manufacturing line monitoring system 1 continuously acquires images taken by at least two of the multiple monitoring cameras C1 to C4 and B and stores them as time-series data in the monitoring storage device 112. When it determines that it has acquired information indicating a change in the manufacturing line M, it performs event storage processing, which extracts time-series data for the change period from the time-series data associated with at least two monitoring cameras and stores it as clip videos P1 to P4 and Q. It also performs output processing, which outputs the video for a portion of the period of at least two clip videos as data that can be displayed in a list on the screen of the display device 114 and played back.

Description

Technical Field

[0001] The present invention relates to a manufacturing line monitoring system for monitoring a manufacturing line.

Background Art

[0002] Conventionally, in a factory's manufacturing line system, when there is any change in the manufactured product and process (including not only abnormalities but also cases where the state is different from normal within the normal range), state detection means (moving body detection sensors (e.g., optical sensors and ultrasonic sensors), abnormal sound detection sensors, cameras, temperature sensors, etc.) are used to obtain information indicating changes in the manufacturing line, and the fact that information indicating the change has been obtained is visually notified to managers, etc. by lighting a three-color lamp or a patrol lamp, which is a notification device (see Patent Documents 1 and 2).

[0003] Also, in such a manufacturing line system, although it is known that a change has occurred in the manufacturing line, there are cases where the state and cause of the change are not known, resulting in delays in the recovery and improvement of the manufacturing line, and productivity may decrease due to a decrease in the operating rate of the manufacturing line.

[0004] In order to solve such problems of decreased productivity, conventionally, when a manufacturing line is photographed by a monitoring camera and changes in the manufacturing line are detected using state detection means, a monitoring system has been disclosed that analyzes the state of changes in the manufacturing line by reviewing the video of the monitoring camera in order to clarify the cause of the change (see Patent Documents 1 to 3).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

[0006] However, in a manufacturing line equipped with multiple surveillance cameras, if information indicating changes in the manufacturing line is acquired using a state detection means, it is necessary to check the images from each of the multiple surveillance cameras to find the image of the manufacturing line state at the time the change occurred (for example, the image of the product and / or process at the time the change occurred). This can make it difficult to see the overall picture and analyze the state of changes in the manufacturing line.

[0007] Therefore, the present invention aims to provide a manufacturing line monitoring system that allows manufacturing line managers and others to easily check the state of change in a manufacturing line equipped with multiple imaging devices, thereby enabling them to analyze the state of change in the manufacturing line, identify the cause of the change, and suppress a decline in productivity. [Means for solving the problem]

[0008] (1) The present invention is a manufacturing line monitoring system for monitoring a manufacturing line that manufactures multiple products, Multiple imaging devices installed on the production line for monitoring the production line, A storage device that stores multiple images taken by at least two of the multiple imaging devices, Control device and Equipped with, The control device is Continuous storage processing involves continuously acquiring multiple images captured by at least two of the multiple imaging devices and storing them in the storage device as time-series data associated with each of the at least two imaging devices. (i) When it is determined that information indicating a change in the manufacturing line has been acquired, (ii) for at least two of the multiple imaging devices, in the continuous storage process, extract multiple image sets from the time series data associated with each of the at least two of the multiple imaging devices that is stored in the storage device, which includes the time of change, which is the time when information indicating a change in the manufacturing line was acquired, and which includes at least the time period before the time of change and the time period after the time of change, and (iii) store the extracted multiple image sets as multiple clip videos associated with each of the at least two of the multiple imaging devices, event storage processing, The system is characterized by performing an output process that outputs images of at least a portion of the change period of at least two of the plurality of clip images as data that can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or a system outside the manufacturing line monitoring system and played back.

[0009] According to the above configuration, when the control device of the manufacturing line monitoring system acquires information indicating a change in the manufacturing line, it extracts video footage of the change period, which includes the time when the information indicating the change in the manufacturing line was acquired and at least the time period before the change, from among multiple time-series data associated with each of at least two of the multiple cameras, and stores it as a clip video for each camera. In other words, triggered by the control device of the manufacturing line monitoring system determining that information indicating a change in the manufacturing line has been acquired, clip videos of the same period, which includes the time of change and at least the time period before the change, are simultaneously generated for at least two of the multiple cameras. Here, the change period includes the time of change, and at least the time period before the change among the time periods before and after the change. In other words, the change period may include the time of change and the time period before the change, or it may include the time of change, the time period before the change, and the time period after the change. Furthermore, at least two of the multiple imaging devices may all be imaging devices that photograph the manufacturing line, or they may include imaging devices that photograph the manufacturing line and imaging devices that photograph a notification device that notifies changes in the manufacturing line. In addition, information indicating changes in the manufacturing line may be obtained from the state detection means, or it may be obtained from images taken by at least one of the multiple imaging devices. Furthermore, based on the data output by the control device of the manufacturing line monitoring system, at least two of the multiple video clips can be displayed in a list on the display screen for at least a portion of the change period. Manufacturing line managers can then aggregate and play back the video of at least two video clips for at least a portion of the change period on the display screen of the manufacturing line monitoring system, or on a display screen of an external system. In other words, by viewing the video of at least two video clips for at least a portion of the change period in a list, manufacturing line managers can quickly confirm the state of the manufacturing line and products at the time of change and at least the time before the change. This allows manufacturing line managers to easily confirm the state of change in the manufacturing line based on the video clips, and by identifying and addressing the cause of the change in the manufacturing line, they can further suppress the decline in productivity. Furthermore, it is possible to identify signs immediately before changes occur in the manufacturing line and perform predictive maintenance in response to those changes. In addition, by analyzing the flow of movement in the manufacturing line process (movement of industrial products and people), it can be used to improve production efficiency (improvement of the manufacturing line layout, reduction of work time, etc.).

[0010] (2) Furthermore, the present invention relates to the manufacturing line monitoring system described in (1) above, In the output processing, the control device, The system may also be characterized by extracting videos from at least two of the multiple video clips that are from the same time period, displaying the extracted videos in a list on the screen of the display device, and outputting them as synchronized playback data that can be played back at the same time.

[0011] According to the above configuration, by playing back the outputted simultaneous playback data, it is possible to extract videos from multiple video clips that are from the same time period, display the extracted videos in a list on the display device screen, and play them back in synchronization with the same time. In other words, multiple video clips can be linked together so that their times are synchronized, and displayed and played back in a list on the display device screen. This allows production line managers and others to view multiple video clips simultaneously in a list, synchronized at the same time. They can then see a list of the state of the production line and products at the same time before and after a change occurs, and identify the location of the change and the product that caused it. Therefore, production line managers and others can more easily check the state of changes on the production line based on video clips, and by identifying and addressing the causes of changes on the production line, they can better suppress the decline in productivity.

[0012] (3) Furthermore, in the manufacturing line monitoring system described in (1) above, the present invention relates to the manufacturing line monitoring system described in (1) above. The aforementioned plurality of imaging devices include a plurality of manufacturing line imaging devices installed along the manufacturing line, with the manufacturing line as the object of imaging. The control device may be characterized in that, in the output processing, it extracts from a plurality of clip videos corresponding to at least two of the plurality of manufacturing line imaging devices videos of a specific product being transported on the manufacturing line at different time periods, and outputs the extracted plurality of videos as time-series playback data that can be displayed in a list on the screen of the display device and played back.

[0013] According to the above configuration, by playing back time-series playback data, videos from multiple video clips showing a specific product being transported on the manufacturing line at different time points are extracted, and these extracted videos are displayed and played back in a list on the display device screen. For example, by linking multiple videos so that the video of a specific product is in chronological order and displaying and playing them back in a list on the display device screen, manufacturing line managers can confirm the series of states in which a specific product is transported from upstream to downstream on the manufacturing line in chronological order. Here, videos from different time points refer to, for example, three videos: a video from the time point including the change in the specific product being filmed, a video from the time point before the change in the specific product being filmed, and a video from the time point after the change in the specific product being filmed. This allows production line managers and others to view the status of a specific product as it is transported from upstream to downstream along the production line in chronological order, to see a list of the status of a specific product when a change occurs on the production line, and to identify the location where a change occurred for a particular product on the production line. Therefore, production line managers and others can more easily check the state of changes on the production line based on video clips, and by identifying and addressing the causes of changes on the production line, they can better suppress the decline in productivity.

[0014] (4) Furthermore, in the manufacturing line monitoring system described in (3) above, the present invention relates to the manufacturing line monitoring system described in (3) above. In the event storage process described above, if it is determined that information indicating a change in the manufacturing line has been obtained from an image taken by at least one of the plurality of imaging devices, The aforementioned specific product may be characterized by being a product included in the image in which it was determined that a change had occurred in the manufacturing line.

[0015] Products captured in an image determined to have caused a change in the production line are likely to be the products that caused the change in the production line. According to the above configuration, a specific product that is captured in an image determined to have caused a change in the production line can reproduce a series of states in which the product is conveyed from upstream to downstream of the production line in chronological order. Thereby, the administrator of the production line or the like can view a series of states in which a specific product that is likely to have caused the change in the production line is conveyed from upstream to downstream of the production line in chronological order, and can check the states of the specific products when the change in the production line occurred in a list, and can check with a high probability the product that caused the change in the production line and the location where the change in the production line occurred. Therefore, the administrator of the production line or the like can more easily check the state of the change in the production line based on the clip video, and can more suppress the decrease in productivity by elucidating and responding to the cause of the change in the production line.

[0016] (5) Further, the present invention is in the production line monitoring system of (1) above, the plurality of imaging devices include a plurality of production line imaging devices that image the production line and are installed along the production line, the control device is in the output process, extracts images in which a plurality of the products grouped under predetermined conditions are imaged from a plurality of the clip videos corresponding to at least two of the plurality of production line imaging devices, and outputs the extracted plurality of images as group reproduction data that can be displayed and reproduced in a list on the screen of the display device. It may be characterized by this.

[0017] According to the above configuration, an administrator or the like can view a list of a plurality of videos in which a plurality of products grouped under predetermined conditions are photographed. For example, from a plurality of clip videos, videos in which a group of products of the same lot being conveyed on the production line are photographed are extracted, and a plurality of videos at the same or different times thus extracted are displayed and played back in a list on the screen of the display device. That is, for a plurality of videos, by associating and displaying in a list on the screen of the display device the videos of the group of products of the same lot and playing them back, an administrator or the like of the production line can check the state of the group of products being conveyed on the production line. Thereby, an administrator or the like can check the states of a plurality of grouped products in a list. Therefore, an administrator or the like of the production line can more easily check the state of changes in the production line based on the clip videos, and by elucidating and addressing the causes of the changes in the production line, it is possible to further suppress a decrease in productivity.

[0018] (6) Further, the present invention is in the production line monitoring system of (5) above, the extracted plurality of videos may at least include a video in which a product located on the most downstream side in the conveyance direction among the plurality of products grouped under the predetermined conditions is photographed, and a video in which a product located on the most upstream side in the conveyance direction among the plurality of products grouped under the predetermined conditions is photographed.

[0019] According to the above configuration, by playing back the group playback data, from a plurality of clip videos, videos in which products located on the most downstream side in the conveyance direction and products located on the most upstream side in the conveyance direction among the plurality of grouped products are photographed are extracted, and the extracted plurality of videos are displayed and played back in a list on the screen of the display device. That is, for a plurality of videos, by associating and displaying in a list on the screen of the display device the videos of the state in which the production line of the plurality of grouped products is conveyed from upstream to downstream and playing them back, an administrator or the like of the production line can check the state of the plurality of grouped products being conveyed on the production line. This allows production line managers to view a time-series overview of the entire process from upstream to downstream of the production line, including the timing of any changes that occurred. They can also identify the product and location that caused the change. For example, if a product that caused a change in the production line is part of a group of products, managers can view the state of the grouped products in a list, including how far the impact of that product extends (whether it affects the starting point of the group (the product located furthest downstream in the transport direction) or the ending point (the product located furthest upstream in the transport direction)). Therefore, production line managers and others can more easily check the state of changes on the production line based on video clips, and by identifying and addressing the causes of changes on the production line, they can better suppress the decline in productivity.

[0020] (7) Furthermore, in the manufacturing line monitoring system described in (5) above, the present invention relates to the manufacturing line monitoring system described in (5) above. In the event storage process described above, if it is determined that information indicating a change in the manufacturing line has been obtained from an image taken by at least one of the plurality of imaging devices, The extracted multiple images may be characterized by including at least an image that indicates that a change has occurred in the manufacturing line.

[0021] According to the above configuration, the series of events in which the product that caused the change in the manufacturing line, as captured in the image where a change in the manufacturing line was determined to have occurred, is transported from upstream to downstream of the manufacturing line can be replayed chronologically along with video footage that includes the image where a change in the manufacturing line was determined to have occurred. This allows production line managers to view a series of events in chronological order, including the timing of the change, when viewing images of the product that caused the change in the production line as it is transported from upstream to downstream. This enables them to identify the product and location that caused the change in the production line. Therefore, production line managers and others can more thoroughly examine the state of changes on the production line based on video clips, and by identifying and addressing the causes of changes on the production line, they can better suppress the decline in productivity.

[0022] (8) Furthermore, the manufacturing line monitoring system of the present invention described in (1) above is The control device is further equipped with an input device connected to the control device to which instructions are input, The aforementioned plurality of imaging devices include a plurality of manufacturing line imaging devices installed along the manufacturing line, with the manufacturing line as the object of imaging. In the output processing, the control device, From a plurality of clip videos corresponding to at least two of the plurality of manufacturing line imaging devices, videos from different time periods in which a specific product being transported on the manufacturing line is captured are extracted, and the extracted videos are output as time-series playback data that can be displayed in a list on the screen of the display device and played back. From a plurality of clip videos corresponding to at least two of the plurality of shooting devices, videos from the same time period are extracted, and the extracted videos are displayed in a list on the screen of the display device and output as synchronized playback data that can be played back at the same time, or From a plurality of clip videos corresponding to at least two of the plurality of manufacturing line imaging devices, videos in which a plurality of products that are grouped according to predetermined conditions are captured are extracted, and the extracted plurality of videos are output as group playback data that can be displayed in a list on the screen of the display device and played back, or The system may also be characterized by selecting based on instructions input to the input device.

[0023] According to the above configuration, the production line manager or other personnel can arbitrarily select and instruct the production line monitoring system, via an input device, whether to output multiple video clips corresponding to at least two of the multiple camera devices as time-series playback data, as simultaneous playback data, or as group playback data. This allows managers and others to select the data output by output processing based on the most suitable video clips to analyze the causes of changes in the manufacturing line. Specifically, the data output by output processing based on video clips can be used for purposes such as predictive maintenance in response to changes by identifying signs immediately preceding changes in the manufacturing line, analyzing the flow of processes on the manufacturing line (movement of industrial products and people), and improving production efficiency (improving the layout of the manufacturing line, shortening work time, etc.). Therefore, production line managers and others can more easily check the state of changes on the production line based on video clips, and by identifying and addressing the causes of changes on the production line, they can better suppress the decline in productivity.

[0024] (9) Furthermore, the present invention relates to any of the manufacturing line monitoring systems described in (1) to (8) above, The control device is The system may also be characterized by further performing a real-time output process that outputs the multiple images captured by each of the multiple imaging devices, which are continuously acquired in the continuous storage process, as data that can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or a system outside the manufacturing line monitoring system.

[0025] With the above configuration, multiple images captured by each of the multiple imaging devices that continuously acquire data through continuous storage processing can be aggregated and displayed in real time on the screen of a display device owned by the manufacturing line monitoring system, or on a display device owned by a system outside the manufacturing line monitoring system. This allows administrators to view a list of images captured by multiple cameras, aggregated and played back on the display screen, enabling them to check the real-time status of the manufacturing line at a glance. Therefore, production line managers and others can easily check not only the state of changes on the production line based on video clips, but also the real-time state of the production line. By identifying the causes of changes on the production line and taking appropriate action, the decline in productivity can be further suppressed.

[0026] (10) Furthermore, the present invention relates to any of the manufacturing line monitoring systems described in (1) to (3), (5), (6), and (8) above, The aforementioned plurality of imaging devices include an imaging device for imaging an imaging device that visually notifies an imaging device of changes in the manufacturing line, In the event storage processing, the control device, If the notification device camera determines from the image captured by the notification device that the notification device has visually notified a change in the manufacturing line, then it is determined that information indicating a change in the manufacturing line has been acquired. The aforementioned at least two imaging devices may or may not include the notification device imaging device.

[0027] According to the above configuration, the control device of the manufacturing line monitoring system can determine, through image analysis, whether or not a change in the manufacturing line has been visually notified from the image captured by the notification device camera, which photographs the notification device that visually notifies changes in the manufacturing line, and thereby obtain information indicating changes in the manufacturing line. The manufacturing line manager can then aggregate and play back images of at least two clips, covering at least a portion of the change period, on the display screen of the manufacturing line monitoring system or a display screen of an external system. In other words, by viewing images of at least two clips covering at least a portion of the change period in a list, the manufacturing line manager can quickly confirm the state of the manufacturing line and products at the time of the change and at least the time before the change. This allows the manufacturing line manager to more easily confirm the state of changes in the manufacturing line based on the clip images, and by identifying and addressing the causes of changes in the manufacturing line, they can more effectively suppress the decline in productivity. In image analysis, a pattern matching analysis method that compares with pre-stored images may be applied, or a machine learning analysis method that is updated sequentially may be applied. Furthermore, at least two of the multiple imaging devices are manufacturing line imaging devices that are used in a manufacturing line monitoring system to image the manufacturing line, and are imaging devices that are used to store clipped images. Other imaging devices that image the manufacturing line, other than those used to store clipped images, may be placed on the manufacturing line. Additionally, the control device of the manufacturing line monitoring system may or may not store clipped images based on a set of images representing a change period extracted from images captured by the notification device imaging device. [Effects of the Invention]

[0028] This manufacturing line monitoring system allows managers and others to easily check the state of change on a manufacturing line equipped with multiple imaging devices. By analyzing the state of change on the manufacturing line, it is possible to identify the cause of the change and suppress the decline in productivity. [Brief explanation of the drawing]

[0029] [Figure 1] This diagram illustrates the manufacturing line monitoring system according to this embodiment and the manufacturing line system including the manufacturing line on which the manufacturing line monitoring system is installed. [Figure 2] This is a block diagram of the manufacturing line monitoring system and the manufacturing line system according to this embodiment. [Figure 3] This is a flowchart of the manufacturing line control process. [Figure 4] This is a flowchart of the manufacturing line monitoring process by the manufacturing line monitoring system according to this embodiment. [Figure 5] This is an explanatory diagram illustrating an example of playback data from the same time. [Figure 6] Figure 5 is an explanatory diagram illustrating how the data is displayed on the device that displays the playback data at the same time. [Figure 7] This is an explanatory diagram illustrating an example of time-series playback data. [Figure 8] Figure 7 is an explanatory diagram illustrating how the time-series playback data is displayed on the display device. [Figure 9] This is an explanatory diagram of an example of group playback data. [Figure 10] Figure 9 is an explanatory diagram illustrating how the group playback data is displayed on the display device. [Figure 11] This is an explanatory diagram illustrating a specific example of a notification device that visually notifies users of changes in a manufacturing line. [Modes for carrying out the invention]

[0030] (Embodiment) As shown in Figure 1, the manufacturing line monitoring system 1 according to this embodiment is installed on a manufacturing line M that transports a large number of products (industrial products Y1, etc.) by belt conveyors M1 to M4 according to a processing or assembly procedure.

[0031] (Manufacturing Line M) First, let's explain the manufacturing line M that will be monitored by the manufacturing line monitoring system 1. As shown in Figure 2, the manufacturing line M is included in the manufacturing line system 2. As shown in Figures 1 and 2, the manufacturing line system 2 includes belt conveyors M1 to M4, a state detection sensor (state detection means) A, an alarm device D including a tricolor light D1, a manufacturing line management device 21 that controls the manufacturing line M, a manufacturing line input device 24, and a manufacturing line display device 25. As shown in Figure 2, the manufacturing line management device 21 includes a manufacturing line control device 22 and a manufacturing line storage device 23.

[0032] (Belt conveyor M1~M4) The belt conveyor M1 has a structure in which a conveyor belt is wrapped between a drive pulley connected to a drive motor and a driven pulley, but it may also be a roller conveyor or the like. Belt conveyors M2 to M4 have the same structure as belt conveyor M1. In this embodiment, for the sake of explanation, the layout is as shown in Figure 1, where the four belt conveyors M1 to M4 are arranged in a straight line from upstream to downstream. However, the layout is not limited to a straight line and can be of various shapes. In this embodiment, for the sake of explanation, the manufacturing line M includes only belt conveyors M1 to M4, but other belt conveyors or work equipment may be present before and after belt conveyors M1 to M4. The industrial products Y1 and the like placed on belt conveyor M1 are transported by the belt conveyors M1 to M4 from belt conveyor M1 to belt conveyor M4 (i.e., from upstream to downstream).

[0033] (Status detection sensor A) The state detection sensor A is a sensor that detects the state of multiple industrial products being transported in the manufacturing line M, which includes belt conveyors M1 to M4. The signal from the state detection sensor A is input to the manufacturing line control device 22 of the manufacturing line management device 21, which will be described later. The manufacturing line control device 22 of the manufacturing line management device 21 then acquires information indicating changes in the manufacturing line M based on the input signal from the state detection sensor A.

[0034] In this embodiment, the state detection sensor A uses an optical sensor, which is an example of a motion detection sensor, and is configured to measure the distance between adjacent industrial products of multiple industrial products being transported on belt conveyors M1 to M4. The distance between adjacent industrial products is between industrial product Y1 and industrial product Y2, or between industrial product Y5 and industrial product Y6, as shown in Figure 1. The signal of the distance between adjacent industrial products detected by the state detection sensor A is input to the manufacturing line control device 22 of the manufacturing line management device 21. When the situation changes from one where a predetermined distance is maintained between adjacent industrial products, such as between industrial product Y1 and industrial product Y2, as shown in Figure 1, to one where the predetermined distance is not maintained between adjacent industrial products, such as between industrial product Y5 and industrial product Y6, due to industrial product Y6 colliding with industrial product Y5, the manufacturing line control device 22 of the manufacturing line management device 21 acquires information indicating a change in the manufacturing line M based on the signal from the state detection sensor A.

[0035] Here, the changes in the manufacturing line M detected by the manufacturing line control device 22 of the manufacturing line management device 21 are not limited to changes in distance between adjacent industrial products, but also include changes to situations different from those assumed in advance. For example, changes in the manufacturing line M detected by the manufacturing line control device 22 of the manufacturing line management device 21 may include cases where an industrial product being transported on the manufacturing line M is deviated from a preset position, where the orientation of the transported industrial product is different from the preset orientation (including cases where it has fallen over), where the type of transported industrial product is different from the type of industrial product set in advance, or where an unusual noise occurs. Furthermore, state detection sensor A is not limited to motion detection sensors such as optical sensors; depending on the object being detected, state detection means such as abnormal sound detection sensors or temperature sensors can also be employed.

[0036] (Notification device D) Notification device D is a device that visually notifies of changes in the manufacturing line M. Notification device D is controlled by the manufacturing line control device 22 of the manufacturing line management device 21, and when the manufacturing line control device 22 acquires information indicating a change in the manufacturing line M, it visually notifies of the change in the manufacturing line M. In this embodiment, the tricolor lamp D1 is described as an example of the notification device D. However, the notification device D is not limited to the tricolor lamp D1, but may also be a meter D2 that displays a numerical value (see Figure 11(A)), a monitor screen D3 that displays a numerical value (see Figure 11(B)), a lamp D4 that can be lit or flashed (see Figure 11(C)), etc. In other words, the visual notification of changes in the manufacturing line M by the notification device D is not limited to notification by the color of the lighting of the tricolor lamp D1 (which is also notification by the lighting of the red lamp of the tricolor lamp D1), but may also be notification by the lighting or flashing of the lamp D4, or notification by a numerical value displayed on the meter D2 or monitor screen D3.

[0037] The three-color light D1 in this embodiment is a warning light having a red lamp, a yellow lamp, and a blue lamp from top to bottom. It has the function of notifying changes in the manufacturing line M and those around the manufacturing line M (managers, users, etc.) based on whether it is lit or not and the color of the lights. For example, when information indicating a change in the manufacturing line M is acquired using the state detection sensor A, the red lamp of the three-color light D1 lights up, notifying those around the manufacturing line M of the change in the manufacturing line M. Note that the lighting of the blue lamp on the three-color light D1 is not a notification of a change in the manufacturing line M, but rather a notification that the manufacturing line M is operating normally, and the lighting of the yellow lamp on the three-color light D1 is a notification that a worker is working on the manufacturing line M.

[0038] (Manufacturing line control device 21) As described above, the manufacturing line management device 21 includes a manufacturing line control device 22 and a manufacturing line storage device 23. The manufacturing line management device 21 is also connected to the belt conveyors M1 to M4, the status detection sensor A, the notification device D (three-color light D1), the manufacturing line input device 24, and the manufacturing line display device 25 by wiring or communication.

[0039] The manufacturing line control device 22 performs various computer controls on the manufacturing line M and is composed of, for example, a CPU. For example, the manufacturing line control device 22 is connected to each drive motor (not shown) of the belt conveyors M1 to M4 of the manufacturing line M and controls their drive. The manufacturing line control device 22 is also connected to the state detection sensor A and acquires information indicating changes in the manufacturing line M based on the signal from the state detection sensor A. Specifically, the manufacturing line control device 22 determines whether or not there is a change in the manufacturing line M based on the signal from the state detection sensor A. Furthermore, if the manufacturing line control device 22 detects a change in the manufacturing line M, it will illuminate the three-color lamp D1 in red. Furthermore, if the manufacturing line control device 22 detects a change in the manufacturing line M, it stops the drive motors of the belt conveyors M1 to M4, thereby stopping the conveyance on the belt conveyors M1 to M4.

[0040] The manufacturing line memory device 23 stores programs and other information that perform various computer controls on the manufacturing line M. The manufacturing line input device 24 is a device used by managers of the manufacturing line M to input various information and instructions, such as a keyboard or mouse. The manufacturing line display device 25 is a device that displays operation screens, output screens, etc., based on commands from the manufacturing line control device 22, and is, for example, a display.

[0041] (Manufacturing line control processing) Next, the manufacturing line control process performed in manufacturing line M will be explained based on the flowchart in Figure 3.

[0042] (Start of transport process (S1)) First, the manager or other personnel activates the manufacturing line management device 21. When the manufacturing line input device 24 receives input for starting the manufacturing line M, the manufacturing line control device 22 starts the conveying process (S1). Specifically, in the conveying process, the manufacturing line control device 22 starts the belt conveyors M1 to M4 by driving the respective drive motors of the belt conveyors M1 to M4. As a result, industrial products X1-X10, Y1-Y10, Z1-Z10, etc., placed on belt conveyor M1 are transported from belt conveyor M1 to belt conveyor M4, that is, from upstream to downstream. Note that only industrial products X9-X10, Y1-Y10, and Z1-Z2 are shown in Figure 1.

[0043] (Detection of changes in manufacturing line M (S2)) Next, the manufacturing line control device 22 detects changes in the manufacturing line M based on the signal from the state detection sensor A, that is, it acquires information indicating changes in the manufacturing line M based on the signal from the state detection sensor A (S2).

[0044] In this embodiment, for multiple industrial products being transported on the belt conveyors M1 to M4 of the manufacturing line M, information indicating changes in the manufacturing line M is obtained from the distance between adjacent industrial products detected by the state detection sensor A, which is a motion detection sensor (optical sensor).

[0045] Then, if the manufacturing line control device 22 determines that a predetermined distance is maintained between adjacent industrial products in the manufacturing line M, it determines that no change is detected in the manufacturing line M (S2: NO) and repeats the process in S2. On the other hand, if the manufacturing line control device 22 determines that a predetermined distance is not maintained between adjacent industrial products in the manufacturing line M, it acquires information indicating a change in the manufacturing line M (S2: YES).

[0046] (Transportation stop process (S3)) When the manufacturing line control device 22 acquires information indicating a change in the manufacturing line M (S2:YES), the manufacturing line control device 22 performs a transport stop process (S3). In this transport stop process, the manufacturing line control device 22 stops the drive motors of each of the belt conveyors M1 to M4, thereby stopping the drive of the belt conveyors M1 to M4. As a result, the transport of industrial products on the manufacturing line M is stopped.

[0047] (Notification processing (S4)) When the manufacturing line control device 22 executes the transport stop process (S3), it performs an alert process (S4). In this alert process, the manufacturing line control device 22 illuminates the three-color lamp D1 in red.

[0048] Furthermore, when the manufacturing line control device 22 acquires information indicating a change in the manufacturing line M (S2:YES), it may execute the transport stop process (S3) and the notification process (S4) at the same time, or it may execute the notification process (S4) first and then the transport stop process (S3). In addition, the notification process (S4) may not only illuminate the three-color lamp D1 in red, but may also notify the surrounding area of ​​the change in the manufacturing line M with a notification sound, or it may send an email to the manager or other person notifying them of the change in the manufacturing line M.

[0049] (Decision to resume (S5)) Next, the manufacturing line control device 22 determines whether or not the manufacturing line M has been restarted by the manufacturing line input device 24 (S5). For example, by analyzing the state of change in the manufacturing line M and identifying its cause, managers can resolve the change in the manufacturing line M. If the manufacturing line M has been restarted by the manufacturing line input device 24 (S5: YES), the manufacturing line control device 22 turns off the red light of the three-color lamp D1, returns to S1, and resumes the transport process. On the other hand, if the manufacturing line M is not restarted, such as by performing a termination operation on the manufacturing line management device 21 using the manufacturing line input device 24 (S5:NO), the manufacturing line control process is terminated.

[0050] (Manufacturing line monitoring system 1) Next, we will describe the manufacturing line monitoring system 1 that monitors the manufacturing line M. As shown in Figures 1 and 2, the manufacturing line monitoring system 1 includes manufacturing line monitoring cameras C1 to C4 (corresponding to imaging devices; manufacturing line monitoring camera C4 is not shown in Figure 1), notification device monitoring camera B (corresponding to notification device imaging device), manufacturing line monitoring device 11, monitoring input device 113, and monitoring display device 114. The manufacturing line monitoring system 1 has the function of monitoring (displaying on screen, recording) the industrial products manufactured by manufacturing line M and manufacturing line M itself using manufacturing line monitoring cameras C1 to C4 and notification device monitoring camera B. Manufacturing line monitoring cameras C1 to C4 and notification device monitoring camera B are multiple imaging devices installed on manufacturing line M for monitoring manufacturing line M.

[0051] Here, as shown in Figure 2, the manufacturing line monitoring system 1 is not electrically connected to the manufacturing line system 2 having the manufacturing line M, and is an electrically independent system from the manufacturing line system 2 having the manufacturing line M. However, the manufacturing line monitoring system 1 may be an electrically connected system to the manufacturing line system 2 having the manufacturing line M. If the manufacturing line monitoring system 1 is an electrically connected system to the manufacturing line system 2 having the manufacturing line M, the manufacturing line control device 22 may be configured to output information indicating changes in the manufacturing line M, acquired in the detection of changes in the manufacturing line M (S2), to the monitoring control device 111 via the communication unit 115 of the manufacturing line monitoring system 1 for acquisition.

[0052] (Manufacturing line surveillance cameras C1~C4) The manufacturing line monitoring cameras C1 to C4 are capable of capturing images and are installed along the manufacturing line M, with the manufacturing line M being the target of the images. The manufacturing line monitoring cameras C1 to C4 are installed at equal intervals along the belt conveyors M1 to M4M and are installed and configured to capture predetermined shooting ranges of the manufacturing line M. In this embodiment, as shown in Figure 1, the manufacturing line monitoring camera C1 includes the conveying area downstream of belt conveyor M1 and the conveying area upstream of belt conveyor M2 in its shooting range. The manufacturing line monitoring camera C2 includes the conveying area downstream of belt conveyor M2 and the conveying area upstream of belt conveyor M3 in its shooting range. The manufacturing line monitoring camera C3 includes the conveying area downstream of belt conveyor M3 and the conveying area upstream of belt conveyor M4 in its shooting range. The manufacturing line monitoring camera C4 (see Figure 2) includes the conveying area downstream of belt conveyor M4 in its shooting range.

[0053] Furthermore, while it is sufficient for multiple production line monitoring cameras to be installed and configured to include at least a portion of production line M in their shooting range, it is preferable that they be installed along production line M so that all of the belt conveyors of production line M are included in their shooting range. Furthermore, the manufacturing line monitoring camera may be configured to allow its shooting range to be arbitrarily adjusted within a predetermined range by zooming (enlarging or reducing).

[0054] (Surveillance camera B) The notification device monitoring camera B is capable of capturing images and is configured to capture images of the notification device D. In this embodiment, the notification device monitoring camera B is installed in a position that includes the tricolor light D1 in its shooting range in order to capture images of the tricolor light D1. In other words, the notification device monitoring camera B corresponds to a notification device imaging device that photographs the notification device D (three-color light D1), which visually notifies that information indicating changes in the manufacturing line M has been acquired.

[0055] The configuration of the notification device monitoring camera B is not particularly limited, and a camera with the same configuration as the manufacturing line monitoring cameras C1 to C4 may be used. Furthermore, at least some of the manufacturing line monitoring cameras C1 to C4 may also serve as the notification device monitoring camera B. Furthermore, the notification device surveillance camera B may be configured to allow its shooting range to be arbitrarily adjusted within a predetermined range by zooming (enlarging or reducing). Furthermore, if the manufacturing line M is equipped with other notification devices D besides the three-color lamp D1, such as a meter D2 that displays a numerical value (see Figure 11(A)), a monitor screen D3 that displays a numerical value (see Figure 11(B)), or a lamp D4 that can be lit or flashed (see Figure 11(C)), the notification device monitoring camera B may be installed in a position that includes the notification device D selected from these notification devices D within its shooting range. Furthermore, in this embodiment, there is one notification device monitoring camera B, but if there are multiple notification devices D, multiple notification device monitoring cameras B may be installed, and multiple notification device monitoring cameras B may be configured to each photograph multiple notification devices D.

[0056] (Manufacturing line monitoring device 11) As shown in Figure 2, the manufacturing line monitoring device 11 comprises a monitoring control device 111, a monitoring storage device 112, and a communication unit 115. The communication unit 115 of the manufacturing line monitoring device 11 is connected to the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B by wired or wireless communication. The monitoring control device 111 is connected to the monitoring input device 113 and the monitoring display device 114.

[0057] The monitoring and control device 111 performs various computer controls on the manufacturing line monitoring system 1 and is composed of, for example, a CPU.

[0058] For example, the monitoring control device 111 processes the images captured by each of the manufacturing line monitoring cameras C1 to C4 to be sequentially (continuously) stored in the memory area of ​​the monitoring storage device 112 corresponding to each of the manufacturing line monitoring cameras C1 to C4 (details will be described later). Furthermore, the monitoring control device 111 processes the video of the three-color light D1 captured by the notification device monitoring camera B to be continuously stored in the memory area of ​​the monitoring storage device 112 corresponding to the notification device monitoring camera B.

[0059] Furthermore, the monitoring and control device 111 determines the color of the three-color light D1, which is the target of the notification device monitoring camera B, by performing image analysis (pattern matching that compares the image with a pre-stored image of the three-color light D1 lit in red, or by sequentially updated machine learning, etc.) on the image captured by the notification device monitoring camera B. For example, if the monitoring and control device 111 determines that the three-color light D1 is lit in red, that is, if it determines that the notification device D has visually notified of a change in the manufacturing line M, it determines that it has acquired information indicating a change in the manufacturing line M.

[0060] The monitoring storage device 112 stores programs and other information used for various computer controls in the manufacturing line monitoring system 1. The monitoring storage device 112 also has a rewritable storage area for necessary information. For example, the images captured by the manufacturing line monitoring cameras C1 to C4 are stored in their corresponding storage areas.

[0061] The monitoring input device 113 is a device used by managers of the manufacturing line M to input various information and instructions, such as a keyboard or mouse. The monitoring and display device 114 is a device for displaying operation screens, output screens, etc., based on commands from the monitoring and control device 111, and is, for example, a display.

[0062] (Manufacturing line monitoring process) Next, the manufacturing line monitoring process performed in the manufacturing line monitoring system 1 will be explained based on the flowchart in Figure 4 and the explanatory diagrams in Figures 5 to 10.

[0063] First, the administrator or other personnel activate the manufacturing line monitoring device 11. The monitoring input device 113 receives input for the activation of the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C4.

[0064] (Start of continuous memory processing (S10)) When a start operation is input, the monitoring control device 111 starts continuous storage processing (S10). In this continuous storage processing, the monitoring control device 111 sequentially (continuously) acquires multiple images taken by each of the manufacturing line monitoring cameras C1 to C4 and stores them as multiple time-series data associated with each of the manufacturing line monitoring cameras C1 to C4 in the storage area of ​​the monitoring storage device 112 corresponding to each of the manufacturing line monitoring cameras C1 to C4. For example, time-series data from manufacturing line monitoring camera C1 is stored in folder 112C1 (not shown) of the monitoring storage device 112, time-series data from manufacturing line monitoring camera C2 is stored in folder 112C2 (not shown) of the monitoring storage device 112, time-series data from manufacturing line monitoring camera C3 is stored in folder 112C3 (not shown) of the monitoring storage device 112, and time-series data from manufacturing line monitoring camera C4 is stored in folder 112C4 (not shown) of the monitoring storage device 112.

[0065] Furthermore, in continuous storage processing, the monitoring control device 111 sequentially (continuously) acquires multiple images captured by the notification device monitoring camera B and stores them as time-series data associated with the notification device monitoring camera B in the storage area of ​​the monitoring storage device 112 corresponding to the notification device monitoring camera B. For example, time-series data from the notification device surveillance camera B is stored in folder 112B (not shown) of the monitoring storage device 112.

[0066] Furthermore, the time-series data captured by the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B, stored in folders 112C1 to 112C4 and folder 112B (not shown), respectively, will be overwritten after a predetermined number of days to prevent exceeding the storage capacity of the monitoring storage device 112.

[0067] (Start of real-time output processing (S11)) The monitoring and control device 111 starts real-time output processing at the same time as the start of continuous storage processing (S11). In real-time output processing, the monitoring and control device 111 outputs multiple images taken by the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B, which are continuously acquired during continuous storage processing, as data that can be displayed in a list on the screen of a display device. This display device is, for example, the monitoring and display device 114 of the manufacturing line monitoring system 1. The display device may also be a display device of a system outside of the manufacturing line monitoring system 1. Note that the display device of a system outside of the manufacturing line monitoring system 1 may or may not include the manufacturing line display device 25 of the manufacturing line M. In real-time output processing, for example, multiple frames may be displayed in software on one screen of the monitoring and display device 114, and the images taken by the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C4 may be displayed in real time in these multiple frames. Here, the number of frames (frames for multiple videos) displayed on one screen of the monitoring display device 114 during real-time output processing can be set arbitrarily.

[0068] (Information showing changes in manufacturing line M is obtained (S12)) Next, the monitoring and control device 111 performs a change detection process (S12) to determine whether or not it has acquired information indicating a change in the manufacturing line M, based on whether or not it has visually notified the change in the manufacturing line M. First, the system continuously acquires images of the notification device D from the notification device monitoring camera B, and determines from the images of the notification device D whether or not the notification device D has visually notified of a change in the manufacturing line M. In this embodiment, the monitoring control device 111 determines from the images of the tricolor light D1 that the notification device monitoring camera B continuously acquires whether or not the tricolor light D1 is lit red.

[0069] Specifically, the monitoring control device 111 determines whether or not the tricolor light D1, which is the target of the notification device monitoring camera B, is lit red by performing image analysis on the image of the tricolor light D1 captured by the notification device monitoring camera B (S12).

[0070] Then, the monitoring and control device 111 determines that it has acquired information indicating a change in the manufacturing line M when it determines that the notification device D has visually notified of a change in the manufacturing line M. In this embodiment, in the notification process (S4) described above, if the tricolor light D1 is lit red, the monitoring and control device 111 performs image analysis on the image of the tricolor light D1 captured by the notification device monitoring camera B, and determines that the tricolor light D1, which is the subject of the notification device monitoring camera B's image, is lit red, and determines that it has acquired information indicating a change in the manufacturing line M (S12: YES). In other words, in the S12 process, a change detection process is performed in which information indicating a change in the manufacturing line M is obtained by performing image analysis on the visual notification image of the three-color light D1 captured by the notification device monitoring camera B.

[0071] Furthermore, if the notification device D captured by the notification device monitoring camera B is not the three-color light D1, the change detection process will be executed as follows. When the notification device D photographed by the notification device monitoring camera B is lamp D4 (see Figure 11(C)), if the "status" lamp of lamp D4 is lit, the monitoring control device 111 analyzes the image of lamp D4 taken by the notification device monitoring camera B and determines that the "status" lamp of lamp D4, which is the target of the notification device monitoring camera B, is lit, and determines that information indicating a change in the manufacturing line M has been acquired. Furthermore, if the notification device D captured by the notification device monitoring camera B is a meter D2 or a monitor screen D3, then by performing image analysis (such as comparing the image of the numerical value of meter D2 or the image of the numerical value of monitor screen D3 captured by the notification device monitoring camera B with the numerical value of monitor screen D3, if it is determined that there has been a change compared to the numerical value range indicating a normal state, then it is determined that information indicating a change in the manufacturing line M has been acquired.

[0072] (Event memory processing (S13)) Next, when the monitoring and control device 111 determines that it has acquired information indicating a change in the manufacturing line M (S12: YES), it performs event storage processing (S13).

[0073] In this event memory processing, the monitoring control device 111, for all of the multiple manufacturing line monitoring cameras C1 to C4, extracts multiple image groups for each of the change periods (for example, 1 minute of video including the time of change and the time before and after the change) from multiple time-series data associated with each of the manufacturing line monitoring cameras C1 to C4, which are stored in folders 112C1 to 112C4 (not shown) of the monitoring storage device 112, during the change detection processing (S12:YES). These images are then stored as clip videos P1 to P4 corresponding to each of the manufacturing line monitoring cameras C1 to C4 in predetermined folders of the monitoring storage device 112. In other words, in the event memory processing (S13), triggered by the determination that information indicating a change in the manufacturing line M has been obtained from the image of the three-color light D1 acquired by the notification device monitoring camera B, clip videos P1 to P4 of the same period, which is the change period including the time of change and at least the time before the time of change, are simultaneously generated for all of the multiple manufacturing line monitoring cameras C1 to C4 installed along the manufacturing line M. In this embodiment, the change period includes the time of change (when it is determined that information indicating a change in the manufacturing line M has been obtained) when it is determined by image analysis that the three-color light D1 is lit red, and includes the time before and after the time of change. Note that the change period only needs to include the time of change and at least the time before the time of change, and does not need to include the time of change and the time before the time of change but not the time after the time of change.

[0074] Furthermore, in event memory processing, the monitoring control device 111 may extract multiple image groups representing the change period from the time-series data associated with the notification device monitoring camera B stored in folder 112B (not shown) of the monitoring storage device 112 during continuous memory processing, and store them as clip video Q corresponding to the notification device monitoring camera B in a predetermined folder of the monitoring storage device 112. In other words, in event memory processing, the monitoring control device 111 extracts multiple image groups representing the change period from multiple time-series data associated with at least each of the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B, and stores them as multiple clip video in the monitoring storage device 112.

[0075] For example, clip video P1 extracted from time-series data associated with manufacturing line monitoring camera C1 is stored in folder 112P1 (not shown) of the monitoring storage device 112; clip video P2 extracted from time-series data associated with manufacturing line monitoring camera C2 is stored in folder 112P2 (not shown) of the monitoring storage device 112; clip video P3 extracted from time-series data associated with manufacturing line monitoring camera C3 is stored in folder 112P3 (not shown) of the monitoring storage device 112; clip video P4 extracted from time-series data associated with manufacturing line monitoring camera C4 is stored in folder 112P4 (not shown) of the monitoring storage device 112; and clip video Q extracted from time-series data captured by notification device monitoring camera B is stored in folder 112Q (not shown) of the monitoring storage device 112. Furthermore, since video clips P1-P4 and Q are saved in folders 112P1-112P4 and 112Q, which are different from folders 112C1-112C4 and 112B, they will not be overwritten even after a specified number of days have passed.

[0076] Furthermore, when the monitoring and control device 111 determines that it has acquired information indicating a change in the manufacturing line M, it may cause the monitoring and control device 112 to record a flag F1 indicating the time when the information indicating a change in the manufacturing line M was acquired, in synchronization with clip videos P1 to P4 and clip video Q, respectively.

[0077] On the other hand, if the monitoring and control device 111 does not determine that it has acquired information indicating a change in the manufacturing line M (S12: NO), the process in S12 is repeated to continue determining whether or not the three-color light D1 is lit red based on the video footage captured by the notification device monitoring camera B.

[0078] (Output processing (S14)) Next, after the event storage processing in S13, the monitoring control device 111 performs output processing (S14). In the output processing, at least a portion of the video clips (at least two of the video clips P1 to P4 and video clip Q) corresponding to at least two of the imaging devices among the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B are output as data that can be displayed in a list on the screen of a display device and played back. At least a portion of the video clips refers to the video for at least a portion of the change period of the video clips. This display device is, for example, the monitoring display device 114 of the manufacturing line monitoring system 1. The display device may also be a display device of a system outside of the manufacturing line monitoring system 1. Note that the display device of a system outside of the manufacturing line monitoring system 1 may or may not include the manufacturing line display device 25 of the manufacturing line M. The imaging devices, consisting of at least two of the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B, may be at least two of the manufacturing line monitoring cameras C1 to C4, or at least one of the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B.

[0079] In output processing (S14), the monitoring control device 111 selects which of the three types of playback data to output based on the instruction input to the monitoring input device 113: simultaneous playback data, time-series playback data, and group playback data. In output processing (S14), the monitoring control device 111 may output pre-set playback data (for example, simultaneous playback data) immediately after event storage processing (S13), and then output playback data based on the input instruction if an instruction is input to the monitoring input device 113 thereafter. The playback data that is automatically output immediately after event storage processing (S13) and the playback data that is output based on the instruction input after event storage processing may be different types of playback data from the three types of playback data, or they may be the same type but with different settings. Alternatively, in output processing (S14), the monitoring control device 111 may output playback data based on the input instruction after an instruction is input to the monitoring input device 113 following event storage processing (S13).

[0080] The following describes the processing when simultaneous playback data, time-series playback data, and group playback data are output during the output processing (S14). In the following explanation, the case in which data is output to the monitoring display device 114 is used as an example, but the display device to which the data is output is not limited to the monitoring display device 114 as described above.

[0081] (Output of playback data from the same time) The monitoring and control device 111 extracts video footage from at least two of the clip videos P1 to P4 and clip video Q that cover the same time period, and outputs the extracted video footage to the monitoring and display device 114 as synchronized playback data that can be displayed in a list on the screen of the monitoring and display device 114 and played back in sync at the same time. Preferably, the at least two clip videos that form the basis of the synchronized playback data include at least two of the clip videos P1 to P4. The at least two clip videos that form the basis of the synchronized playback data may or may not include clip video Q. Specific examples of simultaneous playback data and specific examples of methods for playing simultaneous playback data will be described later.

[0082] (Output of time-series playback data) The monitoring control device 111 extracts images from at least two of the clip images P1 to P4 (i.e., from multiple clip images corresponding to at least two of the multiple manufacturing line monitoring cameras C1 to C4) that capture a specific industrial product being transported on the manufacturing line M at different time points in time. The extracted images are then output to the monitoring display device 114 as time-series playback data that can be displayed and played back in a list on the screen of the monitoring display device 114. In other words, the time points of the images extracted from at least two clip images included in the time-series playback data are all different. The specific industrial product is set (selected) based on information or instructions input to the monitoring input device 113. For example, if the simultaneous playback data is output to the monitoring display device 114 and displayed and played back, allowing an administrator to identify an industrial product that is presumed to have caused a change on the manufacturing line M, the administrator may input information or instructions to the monitoring input device 113 so that this industrial product becomes the specific industrial product. The time periods of the footage extracted from at least two of the clip footage P1 to P4 may be consecutive or partially overlapping. Multiple footage of a specific industrial product at different time periods, extracted from at least two clip footage, includes footage of a first time period including the time of change, and footage of a second time period including the time before the time of change. The second time period may partially overlap with the first time period, or it may be consecutive with the first time period. Preferably, the second time period is not separate from the first time period. The second time period may or may not include time periods after the time of change. Multiple footage of a specific industrial product at different time periods, extracted from at least two clip footage, may include, in addition to the footage of the first and second time periods, footage of a third time period that includes time periods after the first time period but does not include time periods before the first time period. In this case, time-series playback data is generated from at least three of the clip footage P1 to P4. The monitoring control device 111 may output data that allows the monitoring display device 114 to display and play back a list of multiple videos from different time periods extracted from at least two of the clip videos P1 to P4 included in the time-series playback data, along with at least a portion of the video from clip video Q. In other words, the data output when the monitoring input device 113 selects to output time-series playback data may include the time-series playback data and at least a portion of the video from clip video Q. In this case, the time period of at least a portion of the video from clip video Q included in the output data is a time period that includes the time of change, and may be, for example, the time period of the entire set of videos included in the time-series playback data, or the first time period described above. Specific examples of time-series playback data and specific examples of methods for playing back time-series playback data will be described later.

[0083] (Output of group playback data) The monitoring and control device 111 extracts images of multiple industrial products grouped according to predetermined conditions from at least two clip images from among the clip images P1 to P4 (that is, from multiple clip images corresponding to at least two of the multiple manufacturing line monitoring cameras C1 to C4), and outputs the extracted images to the monitoring and display device 114 as group playback data that can be displayed and played back in a list on the screen of the monitoring and display device 114. Grouping according to predetermined conditions may also be done by setting multiple industrial products included in the same lot as one group. Generally, the distance between an industrial product located at one end of a lot and an industrial product located at the end of an adjacent lot is wider than the distance between adjacent industrial products included in the same lot. The monitoring and control device 111 may, for example, determine the lot boundaries from images captured by the manufacturing line monitoring cameras C1 to C4. By determining the two lot boundaries, the start and end points of the group are identified. The monitoring and control device 111 may then determine which video clips to extract from the clip videos P1 to P4 to generate group playback data, based on the information of the start and end points of the group and the transport speed of the manufacturing line M. The group that serves as the basis for generating group playback data is set (selected) based on information or instructions input to the monitoring input device 113. If the clip videos P1 to P4 stored in the event storage processing (S13) do not capture multiple groups but capture a single group, the group that serves as the basis for generating group playback data may be automatically set (selected) without waiting for instructions or information input from the monitoring input device 113. Multiple videos extracted from at least two of the clip videos P1 to P4 include videos of a time period that includes the time of change. Multiple videos extracted from at least two of the clip videos P1 to P4 may include at least one of the following: a video capturing an industrial product located at the downstream end of the transport direction among the multiple industrial products belonging to the group, and a video capturing an industrial product located at the upstream end of the transport direction among the multiple industrial products belonging to the group.The video footage of the industrial product located at the downstream end of the transport direction among multiple industrial products belonging to the group, and the video footage of the industrial product located at the upstream end of the transport direction among multiple industrial products belonging to the group, do not necessarily have to include the time period in which the change occurred. The monitoring control device 111 may output data that allows the monitoring display device 114 to display and play back a list of multiple videos from different time periods extracted from at least two of the clip videos P1 to P4 included in the group playback data, along with at least a portion of the video from clip video Q. In other words, the data output when the monitoring input device 113 selects to output the group playback data may include the group playback data and at least a portion of the video from clip video Q. In this case, the time period of at least a portion of the video from clip video Q included in the output data is a time period that includes the time of change, and may be, for example, the time period of all the videos included in the group playback data, or the same time period as any of the videos included in the group playback data. Specific examples of group playback data and specific examples of how to play back group playback data will be described later.

[0084] (Decision to resume (S15)) After output processing (S14), the monitoring control device 111 determines whether the manufacturing line monitoring system 1 has been restarted by the monitoring input device 113 (S15). For example, in manufacturing line M, after acquiring information indicating a change in manufacturing line M, the cause of the change is resolved, and after the manager of manufacturing line M confirms safety, etc., the manufacturing line monitoring system 1 is restarted by the monitoring input device 113 (S15: YES), the process returns to S11. On the other hand, if no restart operation is performed, such as ending the manufacturing line monitoring system 1 using the monitoring input device 113 (S15:NO), the manufacturing line monitoring process is terminated. In the manufacturing line monitoring process performed by the monitoring control device 111 of this embodiment, the continuous storage process is configured not to stop regardless of whether or not it is determined in the change detection process (S12) that information indicating a change in the manufacturing line M has been acquired. However, in the manufacturing line monitoring process performed by the monitoring control device 111, the continuous storage process may be configured to stop when it is determined in the change detection process (S12) that information indicating a change in the manufacturing line M has been acquired (S12: YES), and to resume upon a restart determination (S15: YES).

[0085] Next, specific examples of simultaneous playback data, time-series playback data, and group playback data, and their playback modes will be described. In the following explanation, the case in which simultaneous playback data, time-series playback data, and group playback data are output to the monitoring display device 114 will be used as an example, but as mentioned above, the display device to which the output is sent is not limited to the monitoring display device 114.

[0086] (Playback of data played at the same time) Figure 6 shows an example of the screen of the monitoring display device 114 when simultaneous playback data is output to and displayed on the monitoring display device 114. In the example in Figure 6, the simultaneous playback data includes four videos from the same time period, extracted from clip videos P1-P3 and clip video Q, out of clip videos P1-P4 and clip video Q. In Figure 6, four frames (a screen divided into four sections) are displayed software-wise on one screen of the monitoring display device 114, and the four videos included in the simultaneous playback data are displayed in a list in these four frames. These four videos are then played back in sync at the same time. In Figure 6, the playback bar displayed at the bottom of the screen of the monitoring display device 114 visually indicates the total playback time of the four videos, and the playback position indicator F2 on the playback bar represents the current playback time common to all four videos. By sliding the playback position indicator F2 on the playback bar of the monitoring display device 114 by operating the monitoring input device 113, the playback time of the four videos can be changed and they can be played back while maintaining their synchronized and linked state at the same time. In this embodiment, one screen of the monitoring display device 114 is divided into four frames, and the clip video Q corresponding to the notification device monitoring camera B and the clip videos P1 to P3 corresponding to the manufacturing line monitoring cameras C1 to C3 are displayed, but this is not limited to this configuration. In the configuration in which the images are displayed in a list on the screen of the monitoring display device 114, one screen of the monitoring display device 114 may be divided into multiple frames, and the required number of clip videos from the clip video Q corresponding to the notification device monitoring camera B and the clip videos P1 to P4 corresponding to the manufacturing line monitoring cameras C1 to C4 may be displayed.

[0087] Figure 5 is a diagram illustrating the simultaneous playback data shown in the example in Figure 6. As shown in Figure 5, the change period of the clip video is 1 minute including the time of change and the time before and after, and the clip video P1 is divided into 15 units (4-second units) to form unit videos P1 / T1, P1 / T2, P1 / T3...P1 / T15, the clip video P2 is divided into 15 units (4-second units) to form unit videos P2 / T1, P2 / T2, P2 / T3...P2 / T15, and the clip video P3 is divided into 15 units (4-second units) to form unit videos From P3 / T1, P3 / T2, P3 / T3...P3 / T15, and from the unit videos Q / T1, Q / T2, Q / T3...Q / T15 obtained by dividing clip video Q into 15 segments (4-second units), the unit videos P1 / T5~T9, P2 / T5~T9, P3 / T5~T9, and Q / T5~T9, which include the unit videos P1 / T8, P2 / T8, and P3 / T8 at the time of change, are extracted to generate playback data for the same time. In Figures 7 and 9, which will be described later, the change period of the clip video is 1 minute including the time of change, and the unit videos at the time of change when clip videos P1~P3 and clip video Q are each divided into 15 segments (4-second units) are P1 / T8, P2 / T8, P3 / T8, and Q / T8. In this embodiment, the change period of the clip video is set to 60 seconds, but it is not limited to this and can be set arbitrarily. Furthermore, in this embodiment, the unit video of the clip is divided into 15 segments (4-second segments), but it is not limited to this and can be set arbitrarily.

[0088] (Playback of time-series playback data) Figure 8 shows an example of the screen of the monitoring display device 114 when time-series playback data and at least a portion of the clip video Q are output to and displayed on the monitoring display device 114. In the example in Figure 8, the time-series playback data includes three videos taken at different time periods, extracted from clip videos P1 to P3 of clip videos P1 to P4, in which a specific industrial product Y6 is filmed. In Figure 8, four frames (a screen divided into four sections) are displayed software-wise on one screen of the monitoring display device 114, and the three videos included in the time-series playback data and at least a portion of the clip video Q are displayed in a list in these four frames. These four videos are then played back in chronological order. In Figure 8, the playback bar displayed at the bottom of the screen of the monitoring display device 114 visually indicates the total playback time of the four videos, and the playback position indicator F2 on the playback bar represents the current playback time of one of the four videos. Alternatively, instead of the playback position indicator F2 on the playback bar shown in Figure 8, a playback bar and playback position indicator may be displayed at the bottom of each of the four videos. In Figure 8, the video extracted from clip video P2 is playing, the video extracted from clip video P1 is stopped at the end of playback, and the video extracted from clip video P3 is stopped at the start of playback.

[0089] Figure 7 illustrates the time-series playback data and at least a portion of the clip video Q displayed in the example in Figure 8. As shown in Figure 7, the unit video P2 / T7~T9 of the first time period including the time of change is extracted from clip video P2, the unit video P1 / T5~T7 of the second time period including the time period before the time of change is extracted from clip video P1, and the unit video P3 / T9~T11 of the third time period including the time period after the time of change is extracted from clip video P3 to generate the time-series playback data. Industrial product Y6 is filmed in unit video P2 / T7~T9, unit video P1 / T5~T7, and unit video P3 / T9~T11. Also, as shown in Figure 7, at least a portion of the clip video Q displayed in the list as well as the three videos included in the time-series playback data is unit video Q / T5~T11, which is the same time period as the entire three videos extracted from clip videos P1~P3.

[0090] (Playback of group playback data) Figure 10 shows an example of the screen of the monitoring display device 114 when group playback data and at least a portion of the clip video Q are output to and displayed on the monitoring display device 114. In the example in Figure 10, the group playback data includes three videos from clip videos P1 to P3, which are extracted from clip videos P1 to P4 and depict 10 industrial products Y1 to Y10 belonging to one group. In Figure 10, four frames (a screen divided into four sections) are displayed software-wise on one screen of the monitoring display device 114, and the three videos included in the group playback data and at least a portion of the clip video Q are displayed in a list in these four frames. These four videos are then played back in chronological order. In Figure 10, the playback bar displayed at the bottom of the screen of the monitoring display device 114 visually indicates the total playback time of the four videos, and the playback position indicator F2 on the playback bar represents the current playback time of any of the four videos. Alternatively, instead of the playback position indicator F2 on the playback bar shown in Figure 10, a playback bar and playback position indicator may be displayed at the bottom of each of the four videos. In Figure 10, the video extracted from clip video P2 at times T7 to T11 is playing, playback is waiting at the video at T12, which is the first unit video extracted from clip video P1, and playback is stopped at the video at T6, which is the last unit video extracted from clip video P3.

[0091] Figure 9 illustrates the group playback data and at least a portion of the clip video Q displayed in the example in Figure 10. As shown in Figure 9, unit videos P1 / T12~T15 are extracted from clip video P1, unit videos P2 / T7~T11, which include the time of change, are extracted from clip video P2, and unit videos P3 / T2~T6 are extracted from clip video P3 to generate the group playback data. Unit videos P3 / T2~T6 capture industrial product Y1, which is located at the downstream end of the transport direction among industrial products Y1~Y10. Unit videos P1 / T12~T15 capture industrial product Y10, which is located at the upstream end of the transport direction among industrial products Y1~Y10. Also, as shown in Figure 9, at least a portion of the three videos included in the group playback data and the clip video Q displayed in the list are unit videos Q / T2~T15, which are videos from the same time period as the entire three videos extracted from clip videos P1~P3.

[0092] According to the above-described manufacturing line monitoring system 1, when the monitoring control device 111 acquires information indicating a change in the manufacturing line M, it extracts video of the change period from a plurality of time-series data associated with at least two of the monitoring cameras (all monitoring cameras in this embodiment) among the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B, and stores them as clip video P1 to P4 and clip video Q, respectively, in folders 112C1 to 112C4 and 112B of the monitoring storage device 112, which are associated with the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B. In other words, triggered by the monitoring control device 111 of the manufacturing line monitoring system 1 determining that it has acquired information indicating a change in the manufacturing line M, clip video of the same period, which includes the time of change and at least the time before the time of change, is generated simultaneously for the manufacturing line monitoring cameras C1 to C4 and the notification device monitoring camera B. Furthermore, based on the data output by the monitoring control device 111 of the manufacturing line monitoring system 1, at least two of the multiple video clips P1 to P4 and video clip Q can be displayed in a list on the display device screen for at least a portion of the change period. The manager of the manufacturing line M can then aggregate and play back the video of at least two of the multiple video clips P1 to P4 and video clip Q for at least a portion of the change period on the monitoring display device 114 of the manufacturing line monitoring system 1, or on the display device of an external system. In other words, by viewing a list of the video of at least two of the multiple video clips P1 to P4 and video clip Q for at least a portion of the change period, the manager of the manufacturing line M can quickly confirm the state of the manufacturing line and products at the time of change and at least the time before the change. This allows the manager of the manufacturing line M to easily confirm the state of change in the manufacturing line M based on the video clips, and by identifying and addressing the cause of the change in the manufacturing line M, the decline in productivity can be further suppressed. Furthermore, predictive maintenance can be performed by identifying signs immediately before changes occur in the manufacturing line M. In addition, the flow analysis of the manufacturing line M process (movement of industrial products and people) can be used to improve production efficiency (improvement of the layout of the manufacturing line M, reduction of work time, etc.).

[0093] Furthermore, according to the above-mentioned manufacturing line monitoring system 1, the manager of the manufacturing line M can arbitrarily select and instruct the manufacturing line monitoring system 1 to output at least two of the clip videos P1 to P4, or at least two of the clip videos P1 to P4 and clip video Q, as time-series playback data, as simultaneous playback data, or as group playback data, using the monitoring input device 113. This allows managers and others to select the data output by output processing based on the most suitable video clips to analyze the causes of changes in manufacturing line M. Specifically, the data output by output processing based on video clips can be used for purposes such as predictive maintenance in response to changes by identifying signs immediately preceding changes in manufacturing line M, process flow analysis of manufacturing line M (movement of industrial products and people), and improvement of production efficiency (improvement of the layout of manufacturing line M, reduction of work time, etc.). Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0094] Furthermore, according to the above-mentioned manufacturing line monitoring system 1, by playing back the outputted simultaneous playback data, it is possible to extract videos from multiple video clips (at least two of the video clips P1 to P4) that are from the same time period, display the extracted videos in a list on the screen of the monitoring display device 114, and play them back in synchronization with the same time. In other words, multiple video clips can be linked together so that their times are synchronized, and displayed in a list on the screen of the monitoring display device 114 for playback. This allows managers of manufacturing line M to view multiple video clips simultaneously in a list, synchronized to the same time. They can then see a list of the state of manufacturing line M and industrial products at the same time before and after a change occurs on manufacturing line M, and identify the location of the change on manufacturing line M and the industrial product that caused the change. Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0095] Furthermore, according to the above-mentioned manufacturing line monitoring system 1, by playing back time-series playback data, it extracts videos from multiple video clips (at least two of the video clips P1 to P4) showing a specific industrial product being transported on the manufacturing line M at different time points in time, and displays and plays back the extracted videos in a list on the screen of the monitoring display device 114. For example, by linking the videos of multiple videos so that the videos of a specific industrial product are in chronological order and displaying and playing them back in a list on the screen of the monitoring display device 114, the manager of the manufacturing line M can confirm the series of states in which a specific industrial product is transported from upstream to downstream on the manufacturing line M in chronological order. This allows managers of production line M to view the status of specific industrial products as they are transported from upstream to downstream along production line M in chronological order, to check the status of specific industrial products at the time of any changes in production line M, and to identify the location where changes occurred for specific industrial products within production line M. Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0096] Furthermore, according to the above-described manufacturing line monitoring system 1, by playing back group playback data, managers and others can view a list of multiple videos in which multiple industrial products grouped under predetermined conditions are filmed. In this embodiment, from multiple clip videos (at least two of the clip videos P1 to P4), videos in which groups of industrial products of the same lot being transported on the manufacturing line M are filmed are extracted, and the extracted videos from the same or different times are displayed in a list on the screen of the monitoring display device 114 and played back. In other words, by linking the videos of groups of industrial products of the same lot for multiple videos and displaying and playing them back in a list on the screen of the monitoring display device 114, managers and others on the manufacturing line M can check the status of groups of industrial products of the same lot being transported on the manufacturing line M. This allows administrators and others to view the status of multiple grouped industrial products in a single list. Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0097] Furthermore, according to the above-mentioned manufacturing line monitoring system 1, by playing back group playback data, the system extracts videos from multiple clip videos (at least two of the clip videos P1 to P4) showing the industrial product located at the downstream end of the transport direction and the industrial product located at the upstream end of the transport direction among the grouped industrial products, and displays and plays back the extracted videos in a list on the screen of the monitoring display device 114. In other words, by linking the videos of the grouped industrial products being transported along the manufacturing line M from upstream to downstream and displaying and playing them back in a list on the screen of the monitoring display device 114, the manager of the manufacturing line M can check the status of the grouped industrial products being transported along the manufacturing line M. This allows the manager of manufacturing line M to view a time-series overview of the entire process of transporting products along manufacturing line M from upstream to downstream, including the timing of any changes that occurred on the line. This allows them to identify the industrial product and location that caused the change on manufacturing line M. For example, if a group of industrial products contains one product that caused a change on manufacturing line M, the manager can view the state of the grouped industrial products in a list, including how far the impact of that product extends (whether it affects the starting point of the group (the product located furthest downstream in the transport direction) or the ending point (the product located furthest upstream in the transport direction)). Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0098] Furthermore, since the monitoring control device 111 performs real-time output processing, it can aggregate and display in real time multiple images taken by the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C4, which are continuously acquired through continuous storage processing, on the screen of the monitoring display device 114 of the manufacturing line monitoring system 1, or on the screen of a display device of a system outside the manufacturing line monitoring system 1. This allows administrators to view a list of all the video footage captured by the notification device monitoring camera B and the manufacturing line monitoring cameras C1-C4, which are aggregated and played back on the display screen, enabling them to check the real-time status of the manufacturing line M in a single view. Therefore, managers of manufacturing line M can easily check not only the state of change in manufacturing line M based on video clips, but also the real-time state of manufacturing line M. By identifying the causes of changes in manufacturing line M and taking appropriate measures, the decline in productivity can be further suppressed.

[0099] Furthermore, according to the above-described manufacturing line monitoring system 1, the monitoring control device 111 can obtain information indicating changes in manufacturing line M by determining, through image analysis, whether or not a change in manufacturing line M has been visually notified from the image captured by the notification device monitoring camera B, which photographs the notification device D that visually notifies changes in manufacturing line M. The manager of manufacturing line M can then aggregate and play back images of at least two clip videos, covering at least a portion of the change period, on the monitoring display device 114 of the manufacturing line monitoring system 1, or on the screen of a display device of an external system. In other words, the manager of manufacturing line M can view images of at least two clip videos, covering at least a portion of the change period, in a list, allowing them to quickly confirm the state of manufacturing line M and industrial products at the time of change and at least the time prior to the change. This makes it easier for the manager of manufacturing line M to confirm the state of changes in manufacturing line M based on the clip videos, and by identifying and addressing the causes of changes in manufacturing line M, they can better suppress the decline in productivity.

[0100] (Other embodiments) (1) In the above embodiment, it was explained that the data output in the output process (S14) may or may not include at least a portion of the video clip Q corresponding to the notification device surveillance camera B. However, in the output process (S14), the video clips of at least two of the video clips P1 to P4 and the video of the notification device surveillance camera B, which is continuously acquired in the continuous storage process, may be output as data that can be displayed and played back in a list on the screen of the display device.

[0101] (2) In the above embodiment, the monitoring control device 111 determines that it has acquired information indicating a change in the manufacturing line M when it determines from the image captured by the notification device monitoring camera B that the notification device D has visually notified of a change in the manufacturing line M. However, the monitoring and control device 111 is not limited to this; it may also determine whether a change has occurred in the manufacturing line M by analyzing images taken by the manufacturing line monitoring cameras C1 to C4, and if it determines that a change has occurred in the manufacturing line M, it may determine that it has acquired information indicating a change in the manufacturing line M. The image analysis may also be an analysis method using machine learning by AI (Artificial Intelligence). More specifically, by learning patterns using a large amount of data and establishing rules for discriminating unknown data, it may be possible to use DNN (Deep Neural Network) or CNN (Convolutional Neural Network) to perform processes such as determining the image content and estimating the posture of industrial products. In this case, the AI-based image analysis can automatically and accurately acquire information indicating a change in the manufacturing line M when industrial products being transported on the manufacturing line M deviate from a preset position, when the orientation of the transported industrial products differs from a preset orientation (including cases where they are lying down), or when the type of transported industrial product differs from a preset type of industrial product. Furthermore, as described above, if the manufacturing line monitoring system 1 is electrically connected to the manufacturing line system 2 having the manufacturing line M, the monitoring control device 111 may determine that it has acquired information indicating a change in the manufacturing line M when it receives information indicating a change in the manufacturing line M based on the signal from the state detection sensor A via the communication unit 115 from the manufacturing line control device 22.

[0102] (2-1) If the monitoring control device 111 determines from the images taken by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, the manufacturing line monitoring system does not need to be equipped with the notification device monitoring camera B. In this case, the manufacturing line monitoring system does not need to be equipped with the state detection sensor A.

[0103] (2-2) When the monitoring control device 111 determines from images taken by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, the time-series playback data may include multiple videos taken at different time periods, extracted from at least two of the clip videos P1 to P4, in which the industrial product included in the image in which the change in the manufacturing line M was determined to have occurred is captured. In other words, the specific industrial product mentioned above may be any industrial product included in the image in which the change in the manufacturing line M was determined to have occurred.

[0104] Industrial products captured in images where a change is determined to have occurred on manufacturing line M are highly likely to be the industrial products that caused the change on manufacturing line M. According to the above configuration, it is possible to reconstruct the series of events in chronological order as a specific industrial product, which is captured in images where a change is determined to have occurred on manufacturing line M, is transported from upstream to downstream on manufacturing line M. This allows managers of production line M to view a series of states in chronological order as a specific industrial product, which is highly likely to have caused a change in production line M, is transported from upstream to downstream of production line M. They can then check the state of the specific industrial product at the time the change occurred in production line M in a list, and with a high degree of probability, identify the industrial product that caused the change in production line M and the location where the change occurred in production line M. Therefore, managers of manufacturing line M can more easily confirm the state of changes in manufacturing line M based on video clips, and by identifying and addressing the causes of changes in manufacturing line M, the decline in productivity can be more effectively suppressed.

[0105] (2-3) If the monitoring control device 111 determines from the images captured by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, the multiple videos included in the group playback data may include the video in which the change in the manufacturing line M has been determined to have occurred. If the monitoring control device 111 determines from the images captured by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, the multiple videos included in the group playback data may be videos of different products. If the monitoring and control device 111 determines from the images captured by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, the multiple videos included in the group playback data may include a video of an industrial product located at the downstream end in the transport direction among multiple industrial products grouped under predetermined conditions, and a video of an industrial product located at the downstream end in the transport direction among multiple industrial products belonging to the same group.

[0106] When the monitoring and control device 111 determines from images captured by the manufacturing line monitoring cameras C1 to C4 that a change has occurred in the manufacturing line M, and the multiple videos included in the group playback data include videos containing the images in which the change in the manufacturing line M was determined to have occurred, the following effect can be obtained: The series of events in which the industrial product that caused the change in the manufacturing line M, captured in the images in which the change in the manufacturing line M was determined to have occurred, is transported from upstream to downstream of the manufacturing line M can be played back in chronological order along with the videos containing the images in which the change in the manufacturing line M was determined to have occurred. This allows the manager of manufacturing line M to view a series of events in which the industrial product that caused the change in manufacturing line M, as captured in the image, is transported from upstream to downstream of manufacturing line M, in a chronological order, including the timing of the change in manufacturing line M. This enables them to identify the industrial product and location that caused the change in manufacturing line M. Therefore, managers of manufacturing line M can more thoroughly examine the state of changes in manufacturing line M based on the video clips, and by identifying and addressing the causes of changes in manufacturing line M, they can better suppress the decline in productivity.

[0107] (3) In the above embodiment, the monitoring control device 111 selects which of the simultaneous playback data, time-series playback data, and group playback data to output in the output processing (S14) based on the instructions input to the monitoring input device 113. However, the monitoring control device 111 may be configured to output only one of the playback data from simultaneous playback data, time-series playback data, and group playback data in the output processing (S14). Alternatively, the monitoring control device 111 may be configured to select which of the two playback data from simultaneous playback data, time-series playback data, and group playback data to output in the output processing (S14) based on the instructions input to the monitoring input device 113. Furthermore, the monitoring control device 111 may output data in a format different from any of the simultaneous playback data, time-series playback data, and group playback data in the output processing (S14).

[0108] (4) In the above embodiment, multiple industrial products included in the same lot are grouped together, but the conditions for grouping are not limited to this. Several specific examples of grouping conditions are described below.

[0109] (4-1) For example, the monitoring and control device 111 may set the time when it determines that the tricolor light D1 has lit up blue (i.e., the time when it determines that transport on the manufacturing line M has started) as the starting point of the group, and the time when it determines that the tricolor light D1 has lit up red (i.e., the time when it determines that a change has occurred on the manufacturing line M) as the ending point of the group, and set multiple industrial products transported between the starting point and the ending point of the group as one group.

[0110] (4-2) For example, the monitoring and control device 111 may set the time when it determines that an operator has performed a predetermined task as the start point of a group, based on images taken by the manufacturing line monitoring cameras C1 to C4 or the notification device monitoring camera B, and the time when it determines that an operator has performed the same or other predetermined task as the end point of a group, and set multiple industrial products transported between the start point and the end point of a group as one group.

[0111] (4-3) For example, the monitoring and control device 111 may set up multiple industrial products to be transported between predetermined times as a single group. The predetermined time may be a time arbitrarily selected by an administrator or the like using the monitoring input device 113.

[0112] (5) The manufacturing line monitoring system 1 of the above embodiment may be operated only within the factory or company (LAN (Local Area Network) or WAN (Wide Area Network)) of the manufacturing line M, using a personal computer or the like in the manufacturing line monitoring device 11. Furthermore, the manufacturing line monitoring system 1 can also be implemented as a cloud service by connecting the manufacturing line monitoring device 11 to an external network and installing the monitoring storage device 112, etc., on an external server. Furthermore, the manufacturing line monitoring system 1 may have a control device and a storage device for each of the multiple imaging devices, including the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C3, and may perform some functions of the monitoring control device and some functions of the monitoring storage device. For example, in continuous storage processing, the manufacturing line monitoring system 1 may, based on a command from the monitoring control device 111, have the control device of each of the multiple imaging devices store multiple images captured by each of the multiple imaging devices as time-series data in the storage device of each of the multiple imaging devices. Then, in event storage processing, the manufacturing line monitoring system 1 may, based on a command from the monitoring control device 111, have the control device of each of the multiple imaging devices extract multiple image groups for the change period from multiple time-series data associated with each of the multiple imaging devices stored in the storage device of each of the multiple imaging devices, and store them in the monitoring storage device 112 as clip images corresponding to each of the multiple imaging devices.

[0113] Although embodiments of the present invention have been described above with reference to the drawings, it should be understood that the specific configuration is not limited to these embodiments and examples. Furthermore, the scope of the present invention is indicated not only by the above-described embodiments and examples but also by the claims, and includes all modifications within the meaning and scope of equivalence to the claims. [Explanation of Symbols]

[0114] 1: Manufacturing line monitoring system, 111: Monitoring control device (control device), 112: Monitoring storage device (storage device), 113: Monitoring input device (input device), 114: Monitoring display device (display device of the manufacturing line monitoring system), B: Notification device monitoring camera (notification device imaging device), C1~C4: Manufacturing line monitoring camera (manufacturing line imaging device), D: Notification device, D1: Three-color light (notification device), D2: Meter (notification device), D3: Monitor screen (notification device), D4: Lamp (notification device), M: Manufacturing line, P1~P4, Q: Clipped video

Claims

1. A manufacturing line monitoring system that monitors a manufacturing line that produces multiple products, Multiple surveillance cameras installed on the production line for monitoring the production line, A storage device that stores multiple images taken by at least two of the aforementioned multiple surveillance cameras, A control device is provided, The control device is Continuous storage processing involves continuously acquiring multiple images captured by at least two of the multiple surveillance cameras and storing them in the storage device as time-series data associated with each of the at least two of the multiple surveillance cameras, (i) When it is determined that information indicating a change in the manufacturing line has been acquired, (ii) for at least two of the multiple surveillance cameras, an event storage process is performed, in which, from the time-series data associated with each of the at least two of the multiple surveillance cameras stored in the storage device in the continuous storage process, a plurality of image sets are extracted that include the change time, which is the time when information indicating a change in the manufacturing line was acquired, and the change period includes at least the time before the change time among the time periods before and after the change time, and (iii) the extracted plurality of image sets are stored in the storage device as a plurality of clip images associated with each of the at least two of the multiple surveillance cameras, The system performs an output process that outputs images of at least two of the multiple clip images for at least a portion of the change period as data that can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or a system outside the manufacturing line monitoring system, and plays them back. The aforementioned multiple surveillance cameras include multiple production line surveillance cameras installed along the production line, with the production line being the target of the surveillance. In the event storage processing, the control device, (a) From the images captured by the multiple manufacturing line monitoring cameras, it is determined by image analysis whether or not a change has occurred in the manufacturing line, and if it is determined that a change has occurred in the manufacturing line, it is determined that information indicating a change in the manufacturing line has been acquired, or (b) A manufacturing line monitoring system characterized in that, when a notification device monitoring camera, which is included in the plurality of monitoring cameras and photographs a notification device that visually notifies changes in the manufacturing line, determines from the image taken by the notification device that visually notifies changes in the manufacturing line, it determines that information indicating changes in the manufacturing line has been acquired.

2. In the output processing, the control device, The manufacturing line monitoring system according to claim 1, characterized in that it extracts videos from at least two of the multiple video clips that are from the same time period, and outputs the extracted videos as simultaneous playback data that can be displayed in a list on the screen of the display device and played back in sync at the same time.

3. The control device, in the output processing, The manufacturing line monitoring system according to claim 1, characterized in that it extracts images from multiple clip images corresponding to at least two of the multiple manufacturing line monitoring cameras, showing a specific product being transported on the manufacturing line at different time points, and outputs the extracted images as time-series playback data that can be displayed in a list on the screen of the display device and played back.

4. In the event storage process described above, if it is determined that information indicating a change in the manufacturing line has been obtained from an image taken by at least one of the multiple manufacturing line monitoring cameras, The manufacturing line monitoring system according to claim 3, characterized in that the specific product is a product included in an image in which a change has been determined to have occurred in the manufacturing line.

5. The control device, in the output processing, The manufacturing line monitoring system according to claim 1, characterized in that it extracts images of multiple products that are grouped according to predetermined conditions from multiple clip images corresponding to at least two of the multiple manufacturing line monitoring cameras, and outputs the extracted multiple images as group playback data that can be displayed in a list on the screen of the display device and played back.

6. The manufacturing line monitoring system according to claim 5, characterized in that the extracted multiple images include at least an image of a product located at the downstream end in the transport direction among the multiple products grouped according to the predetermined conditions, and an image of a product located at the upstream end in the transport direction among the multiple products grouped according to the predetermined conditions.

7. In the event storage process described above, if it is determined that information indicating a change in the manufacturing line has been obtained from an image taken by at least one of the multiple manufacturing line monitoring cameras, The manufacturing line monitoring system according to claim 5, characterized in that the extracted plurality of images include at least an image in which it is determined that a change has occurred in the manufacturing line.

8. The control device is further equipped with an input device connected to the control device to which instructions are input, In the output processing, the control device, From multiple clip videos corresponding to at least two of the multiple manufacturing line monitoring cameras, extract videos from different time periods in which a specific product being transported on the manufacturing line is captured, and output the extracted videos as time-series playback data that can be displayed and played back in a list on the screen of the display device. From multiple video clips corresponding to at least two of the multiple surveillance cameras, extract video clips from the same time period, and output the extracted video clips as synchronized playback data that can be displayed in a list on the screen of the display device and played back in sync at the same time, or From multiple clip videos corresponding to at least two of the multiple manufacturing line monitoring cameras, the system extracts videos in which multiple products that are grouped according to predetermined conditions are captured, and outputs the extracted videos as group playback data that can be displayed in a list on the screen of the display device and played back. The manufacturing line monitoring system according to claim 1, characterized in that it selects based on instructions input to the input device.

9. The control device is A manufacturing line monitoring system according to any one of claims 1 to 8, further characterized by performing a real-time output process that outputs a plurality of images taken by each of the plurality of surveillance cameras, which are continuously acquired by the continuous storage process, as data that can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or a system outside the manufacturing line monitoring system.

10. In the event storage processing, if it is determined that information indicating a change in the manufacturing line has been obtained from the image captured by the notification device monitoring camera, The manufacturing line monitoring system according to any one of claims 1 to 3, 5, 6, or 8, characterized in that the plurality of clip videos stored in the event storage process include or do not include the clip videos corresponding to the notification device monitoring camera.

Images