Manufacturing line monitoring system

Abstract

The present invention provides a manufacturing line monitoring system capable of suppressing any decline in productivity by allowing, inter alia, a manager of a manufacturing line to easily check the state of change in the manufacturing line in which a plurality of imaging devices are installed.　A monitoring control device (111) of a manufacturing line monitoring system (1) executes: a constant storage process that constantly acquires images captured by at least two among a plurality of monitoring cameras (C1-C4) and (B), and storing the images as time-series data in a monitoring storage device (112); and an event storage process that, when it is determined that information indicating a change in a manufacturing line (M) has been acquired, extracts time-series data of a change period from the time-series data associated with the at least two monitoring cameras, and storing the time-series data as clip videos (P1-P4) and (Q).

Description

Production line monitoring system

[0001] The present invention relates to a production line monitoring system for monitoring a production line.

[0002] Conventionally, in a factory production line system, when there is any change in the product or process being manufactured (not only an abnormality but also a state that deviates from normal conditions even within the normal range), a status detection means (a motion detection sensor (e.g., an optical sensor or ultrasonic sensor), an abnormal sound detection sensor, a camera, a temperature sensor, etc.) is used to obtain information indicating the change in the production line, and the acquisition of the information indicating the change is visually notified to a manager or the like by lighting up an alarm device such as a three-color light or a patrol lamp (see Patent Documents 1 and 2).

[0003] Furthermore, in such manufacturing line systems, although it is possible to detect that a change has occurred on the manufacturing line, it may not be possible to determine the state or cause of the change, which may result in delays in the recovery and improvement of the manufacturing line, a decrease in the operating rate of the manufacturing line, and a decline in productivity.

[0004] In order to solve this problem of reduced productivity, conventionally, monitoring systems have been disclosed that use a surveillance camera to capture images of the production line, and when a change in the production line is detected using a status detection means, analyze the status of the change in the production line by reviewing the surveillance camera footage in order to clarify the cause of the change (see Patent Documents 1 to 3).

[0005] International Publication No. 2022 / 239123 Japanese Patent Application Publication No. 2023-24346 Japanese Patent Application Publication No. 2023-075578

[0006] However, in a production line where multiple surveillance cameras are installed, if information indicating changes in the production line is obtained using a status detection means, it is necessary to check the footage from each of the multiple surveillance cameras to find footage of the state of the production line at the time the change occurred in the production line (for example, footage of the product and / or process at the time the change occurred in the production line), which makes it difficult to view at a glance and may make it difficult to analyze the state of the change in the production line.

[0007] On the other hand, when installing a new monitoring system that captures images of an existing production line using a monitoring camera or other imaging device to analyze changes in the production line and stores the captured images when changes are detected using a status detection means, it is necessary to electrically connect the monitoring system to the existing production line in which the status detection means for detecting changes in the production line is installed. However, depending on the configuration of the production line system, it may be difficult to electrically connect the monitoring system, making it less versatile.

[0008] In this regard, as mentioned above, many existing production line systems use status detection means to detect changes in the production line, and then use an alarm device to visually notify those in the vicinity that a change has been detected.

[0009] Therefore, the present invention aims to provide a highly versatile production line monitoring system that can be installed on a variety of production lines, allowing production line managers and others to easily check the state of changes in a production line on which multiple imaging devices are installed, thereby analyzing the state of changes in the production line, identifying the cause of the changes, and preventing a decline in productivity.

[0010] (1) The present invention provides a production line monitoring system for monitoring a production line that manufactures a plurality of products, the system comprising: a plurality of image capturing devices installed on the production line for monitoring the production line; a storage device that stores a plurality of images captured by at least two of the plurality of image capturing devices; and a control device, wherein the control device performs a continuous storage process that constantly acquires a plurality of images captured by at least two of the plurality of image capturing devices and stores them in the storage device as time-series data associated with each of the at least two of the plurality of image capturing devices; (i) when it is determined that information indicating a change in the production line has been acquired, (ii) for at least two of the plurality of image capturing devices, from the time series data associated with each of the at least two image capturing devices among the plurality of image capturing devices, which has been stored in the storage device in the continuous storage process, extracting a plurality of image groups for a change period that includes the change time, which is the time when the information indicating the change in the production line was acquired, and that includes at least the time period before the change time among the time periods after the change time, and (iii) storing the extracted plurality of image groups as a plurality of clip images associated with each of the at least two image capturing devices among the plurality of image capturing devices.

[0011] According to the above configuration, when the control device of the production line monitoring system acquires information indicating a change in the production line, the control device extracts, from the multiple time-series data associated with at least two of the multiple image capture devices, video of a change period that includes the time of change, which is the time when the information indicating a change in the production line was acquired, and at least a time period before the change time, and stores the extracted video as clip video for each image capture device. In other words, when the control device of the production line monitoring system determines that it has acquired information indicating a change in the production line, clip video of the same period, which is a change period that includes the time of change and at least a time period before the change time, is simultaneously generated for at least two of the multiple image capture devices. Here, the change period includes at least the time period before the change time among the time period before the change time and the time period after the change time. In other words, the change period may include the time of change and a time period before the change time, or it may include the time of change, the time period before the change time, and a time period after the change time. Furthermore, at least two of the multiple image capture devices may all be image capture devices that capture the production line, or may include an image capture device that captures the production line and an image capture device that captures an alarm device that notifies changes in the production line. Furthermore, information indicating changes in the production line may be obtained from a status detection means or may be obtained from an image captured by at least one of the multiple image capture devices.

[0012] (2) Furthermore, the present invention is characterized in that, in the manufacturing line monitoring system of (1) above, the control device executes an output process to output images of at least a portion of the change period of at least two clip images among the plurality of clip images as data that can be displayed and played back in a list on a screen of a display device possessed by the manufacturing line monitoring system or a system external to the manufacturing line monitoring system.

[0013] According to the above configuration, the data output by the output processing of the control device of the production line monitoring system can be used to display a list of at least two clip videos from at least a portion of the change period on the screen of a display device. A production line manager or the like can aggregate and play back at least a portion of the change period for at least two clip videos on the screen of a display device included in the production line monitoring system or a display device included in a system external to the production line monitoring system. In other words, by viewing at least a portion of the change period for at least two clip videos in a list, a production line manager or the like can at a glance confirm the status of the production line and the products at the time of the change and at least the time period prior to the change. This allows a production line manager or the like to easily confirm the status of changes on the production line based on the clip videos, identify the cause of the change, and take appropriate action to further prevent a decline in productivity. Furthermore, predictive maintenance can be performed by identifying signs of changes immediately before they occur on the production line. Furthermore, analysis of the flow of production line processes (the movement of people and objects such as industrial products) can be used to improve production efficiency (improving the layout of the production line, shortening work times, etc.).

[0014] (3) Furthermore, in the manufacturing line monitoring system of (2) above, the present invention may be characterized in that, in the output processing, the control device extracts images from the same time period from at least two of the plurality of clip images, and outputs the extracted plurality of images as simultaneous playback data that can be displayed in a list on the screen of the display device and played back in synchronization at the same time.

[0015] According to the above configuration, by playing back the output simultaneous playback data, video from the same time period can be extracted from multiple video clips, and the extracted video clips can be displayed in a list on the display device screen and played back synchronously at the same time. In other words, multiple video clips can be linked and synchronized so that their times are synchronized and displayed in a list on the display device screen for playback. This allows production line managers and others to simultaneously view multiple video clips synchronized to the same time, allowing them to see at a glance the status of the production line and products at the same time before and after a change in the production line, and to identify the location of the change in the production line and the product that caused the change. Therefore, production line managers and others can more easily check the status of changes in the production line based on the video clips, and by identifying and addressing the cause of the change in the production line, they can further reduce productivity declines.

[0016] (4) Furthermore, in the production line monitoring system of (2) above, the plurality of photographing devices may include a plurality of production line photographing devices that photograph the production line and are installed along the production line, and the control device, in the output processing, may extract, from a plurality of clip images corresponding to at least two of the plurality of production line photographing devices, images of a specific product being transported on the production line at different time periods, and output the extracted plurality of images as time-series playback data that can be displayed and played back in a list on the screen of the display device.

[0017] According to the above configuration, by playing back time-series playback data, footage of a specific product being transported on the production line, captured at different times, is extracted from multiple clips, and the extracted footage is displayed and played back in a list on the display device screen. For example, by linking the multiple footage so that the footage of the specific product is in chronological order and displaying and playing back the list on the display device screen, a production line manager or the like can check the series of states of the specific product being transported from upstream to downstream along the production line in chronological order. Here, footage from different time periods refers to, for example, three types of footage: footage from a time period including the time when the specific product was captured, footage from a time period before the time when the specific product was captured, and footage from a time period after the time when the specific product was captured. This allows a production line manager or the like to view the state of the specific product being transported from upstream to downstream along the production line in chronological order, check the state of the specific product when a change occurs on the production line in a list, and identify the location where the change occurred for the specific product on the production line. Therefore, production line managers and the like can more easily check the state of changes in the production line based on the clip footage, and by identifying the causes of changes in the production line and taking appropriate measures, they can further prevent declines in productivity.

[0018] (5) Furthermore, in the production line monitoring system of (4) above, the present invention may be characterized in that, when it is determined in the event storage process that information indicating a change in the production line has been acquired from an image captured by at least one of the plurality of photographing devices, the specific product is the product included in the image in which it is determined that a change in the production line has occurred.

[0019] The product captured in an image in which a change in the production line is determined to have occurred is likely to be the product that caused the change in the production line. According to the above configuration, a series of events in which a specific product, the product captured in an image in which a change in the production line is determined to have occurred, is transported from upstream to downstream along the production line can be played back in chronological order. This allows a production line manager or the like to view a series of events in chronological order in which a specific product, which is likely to have caused the change in the production line, is transported from upstream to downstream along the production line. By checking the status of the specific product at the time of the change in the production line in a list, the product that caused the change in the production line and the location where the change occurred can be identified with a high degree of probability. Therefore, a production line manager or the like can more easily check the status of the change in the production line based on the clip video, and by identifying the cause of the change in the production line and taking appropriate action, a decline in productivity can be further suppressed.

[0020] (6) Furthermore, in the production line monitoring system of (2) above, the present invention may be characterized in that the plurality of photographing devices include a plurality of production line photographing devices that photograph the production line and are installed along the production line, and the control device, in the output processing, extracts images of a plurality of products that are grouped under predetermined conditions from a plurality of the clip images corresponding to at least two of the plurality of production line photographing devices, and outputs the extracted plurality of images as group playback data that can be displayed and played back in a list on the screen of the display device.

[0021] According to the above configuration, a manager or the like can view, at a glance, multiple videos of multiple products grouped under predetermined conditions. For example, from multiple video clips, videos of products from the same lot being transported on the production line can be extracted, and the extracted videos, taken at the same or different times, can be displayed and played back in a list on the screen of a display device. In other words, by linking multiple videos of products from the same lot and displaying and playing them in a list on the screen of a display device, a production line manager or the like can check the status of products from the same lot being transported on the production line. This allows a manager or the like to check the status of multiple grouped products at a glance. Therefore, a production line manager or the like can more easily check the status of changes in the production line based on the video clips, and by identifying and addressing the causes of changes in the production line, a decline in productivity can be further reduced.

[0022] (7) Furthermore, in the manufacturing line monitoring system of (6) above, the present invention may be characterized in that the extracted multiple images include at least an image of a product located at the most downstream side in the conveying direction among the multiple products grouped under the specified conditions, and an image of a product located at the most upstream side in the conveying direction among the multiple products grouped under the specified conditions.

[0023] According to the above configuration, by playing back the group playback data, footage of the product located furthest downstream in the conveying direction and the product located furthest upstream in the conveying direction among the grouped products is extracted from the multiple clip videos, and the extracted multiple videos are displayed and played back in a list on the screen of a display device. In other words, by linking the multiple videos of the grouped products being transported from upstream to downstream along the production line and displaying and playing them in a list on the screen of a display device, a production line manager or the like can check the status of the grouped products being transported along the production line. This allows the production line manager to check the series of statuses of the grouped products being transported along the production line from upstream to downstream in a list in chronological order, including the timing of changes in the production line, and to identify the products and locations that caused changes in the production line. For example, if a product among multiple grouped products is the cause of a change in the production line, a manager or the like can check the status of the multiple grouped products at a glance, such as how far the effect of the product that caused the change in the production line has spread to the multiple grouped products (whether it has affected the start point of the group (the product located furthest downstream in the conveying direction) or the end point of the group (the product located furthest upstream in the conveying direction)). Therefore, a manager or the like of the production line can more easily check the status of changes in the production line based on the clip video, and by identifying the cause of the change in the production line and taking action, a decline in productivity can be further suppressed.

[0024] (8) Furthermore, in the production line monitoring system of (6) above, the present invention may be characterized in that, when it is determined in the event storage process that information indicating a change in the production line has been acquired from an image captured by at least one of the plurality of photographing devices, the extracted plurality of images may include at least an image including an image in which it is determined that a change has occurred in the production line.

[0025] According to the above configuration, a series of states of a target product, which is the cause of a change in the production line and is captured in an image in which it is determined that a change has occurred in the production line, being transported from upstream to downstream along the production line can be played back in chronological order along with video including the image in which it is determined that a change has occurred in the production line. This allows a production line manager, when viewing a series of states of a target product, which is the cause of a change in the production line and is captured in an image in which it is determined that a change has occurred in the production line, being transported from upstream to downstream along the production line in chronological order, to view a list in chronological order, including the timing of the change in the production line, and to identify the product and its location that caused the change in the production line. Therefore, a production line manager, etc., can check the state of the change in the production line in more detail based on the video clip, and by identifying the cause of the change in the production line and taking appropriate action, a decline in productivity can be further reduced.

[0026] (9) The production line monitoring system of the present invention described in (2) above further comprises an input device connected to the control device for inputting instructions, wherein the plurality of photographing devices include a plurality of production line photographing devices that photograph the production line and are installed along the production line, and wherein the control device, in the output processing, extracts, from a plurality of clip videos corresponding to at least two of the plurality of production line photographing devices, images of specific products being transported on the production line taken at different time periods, and outputs the extracted plurality of images as time-series playback data that can be displayed and played back in a list on the screen of the display device, or extracts, from a plurality of clip videos corresponding to at least two of the plurality of photographing devices, images of the same time period, and outputs the extracted plurality of images as simultaneous playback data that can be displayed and played back in a list on the screen of the display device, The method may be characterized in that, based on an instruction input to the input device, the method selects whether to extract footage of a plurality of products grouped under predetermined conditions from a plurality of clip footage corresponding to at least two of the plurality of production line photography devices, and output the extracted plurality of footage as group playback data that can be displayed and played back in a list on the screen of the display device.

[0027] According to the above configuration, a production line manager or the like can use an input device to select and instruct whether to output multiple video clips corresponding to at least two of the multiple image capture devices as time-series playback data, simultaneous playback data, or group playback data in the production line monitoring system. This allows the manager or the like to select data output by the output process based on the optimal video clips for analyzing the cause of changes in the production line. Specifically, the data output by the output process based on the video clips can be used for purposes such as predictive maintenance of changes by identifying signs of changes immediately before they occur in the production line, analyzing the flow of objects and people in the production line (such as industrial products), and improving production efficiency (improving the production line layout, reducing work time, etc.). Therefore, production line managers or the like can more easily check the status of changes in the production line based on the video clips, and by identifying and addressing the causes of changes in the production line, they can further reduce productivity declines.

[0028] (10) Furthermore, the present invention may be characterized in that, in any of the manufacturing line monitoring systems described above in (1) to (9), the control device further executes real-time output processing to output multiple images captured by each of the multiple imaging devices that are constantly acquired by the continuous storage processing as data that can be displayed in a list on a screen of a display device possessed by the manufacturing line monitoring system or a system external to the manufacturing line monitoring system.

[0029] According to the above configuration, multiple images captured by each of the multiple image capture devices, which are constantly acquired through continuous storage processing, can be aggregated and displayed in real time on the screen of a display device included in the production line monitoring system or a display device included in a system external to the production line monitoring system. This allows managers and others to check the real-time status of the production line by viewing a list of each video captured by the multiple image capture devices aggregated and played back on the display screen. Therefore, production line managers and others can easily check not only the status of changes in the production line but also the real-time status of the production line based on the clipped video, and by identifying the cause of changes in the production line and taking appropriate action, productivity declines can be further reduced.

[0030] (11) The present invention also provides a production line monitoring system according to any one of (1) to (4), (6), (7), and (9) above, wherein the plurality of photographing devices include an alarm device photographing device that photographs an alarm device that visually notifies a change in the production line, and the control device, in the event storage process, determines that information indicating a change in the production line has been acquired when it is determined from an image photographed by the alarm device photographing device that the alarm device has visually notified a change in the production line, and the at least two photographing devices may or may not include the alarm device photographing device.

[0031] According to the above configuration, the control device of the production line monitoring system can determine whether a change in the production line has been visually notified from images captured by the notification device image capture device, which captures images of the notification device that visually notifies changes in the production line, through image analysis, and acquire information indicating the change in the production line. Then, a production line manager or the like can consolidate and play back at least a portion of the change period for at least two video clips on a display device included in the production line monitoring system or a display device included in a system external to the production line monitoring system. In other words, by viewing at least a portion of the change period for at least two video clips in a list, a production line manager or the like can at a glance confirm the state of the production line and the products at the time of the change and at least the time period prior to the change. This allows a production line manager or the like to more easily confirm the state of the change in the production line based on the video clips, identify the cause of the change in the production line, and address it, thereby further reducing productivity declines. Note that the image analysis may employ a pattern matching method that compares images with pre-stored images, or an analysis method using machine learning that is updated sequentially. Furthermore, at least two of the multiple image capturing devices are production line image capturing devices in the production line monitoring system that capture the production line and are image capturing devices that are targeted to store clip video. An image capturing device that captures the production line other than the production line image capturing device that is targeted to store clip video may be disposed on the production line. Furthermore, the control device of the production line monitoring system may or may not store clip video based on a group of multiple images of a change period extracted from images captured by the alarm device image capturing device.

[0032] (12) Furthermore, in the production line monitoring system of the present invention described above in (11), the control device constantly acquires images of the notification device captured by the notification device photographing device, and determines from the acquired images of the notification device by image analysis whether the notification device has visually notified a change in the production line, and if it determines that the notification device has visually notified a change in the production line, executes the event storage process.

[0033] According to the above configuration, by installing an alarm device camera that captures an image of an alarm device that visually notifies changes in an existing production line, the control device of the production line monitoring system can detect changes in the production line by determining through image analysis whether a change in the production line has been visually notified from the image captured by the alarm device camera. Note that the image analysis may employ an analysis method using pattern matching that compares the image with pre-stored images, or an analysis method using machine learning that is updated sequentially. Furthermore, the multiple production line camera devices are camera devices that capture the production line in the production line monitoring system and capture images that become clip videos. Camera devices that capture the production line other than the production line camera device may be disposed on the production line. Furthermore, the alarm device camera device is at least one camera device. If there is one alarm device on the production line, the one camera that captures the single alarm device is the alarm device camera device. If there are multiple alarm devices on the production line, the multiple camera devices that capture each of the multiple alarm devices are the alarm device camera devices. When the control device of the production line monitoring system detects a change in the production line, it extracts, from the multiple time-series data associated with each of the multiple production line photography devices installed along the production line, video footage from the change period, which includes the time when the change is determined to have visually notified the change in the production line and at least the time period before the change, and stores the extracted video footage as a clip for each production line photography device. In other words, when the control device of the production line monitoring system detects a change in the production line from images acquired by the notification device photography device, it triggers the simultaneous generation of clip footage from all of the multiple production line photography devices installed along the production line, which is the change period, including the time when the change occurred and at least the time period before the change. This allows a production line manager or the like to analyze the state of the change in the production line captured by the multiple production line photography devices by going back in time from the time of the change based on the stored clip footage, thereby identifying the cause of the change and preventing a decline in productivity.Here, the change period includes the time of change and at least the time period before the change among the time period before the change and the time period after the change. In other words, the change period may include the time of change and the time period before the change, or may include the time of change, the time period before the change, and the time period after the change. The change time may be the time when an image that is determined to have visually notified a change in the production line is captured, or the time when the control device of the production line monitoring system determines that a change in the production line has visually notified. Furthermore, the production line photography device, notification device photography device, storage device, and control device included in the production line monitoring system can be installed independently of an existing production line without being electrically connected to the existing production line. This makes the system highly versatile and allows it to be installed on a variety of production lines.

[0034] By enabling production line managers and the like to easily check the status of changes in a production line where multiple imaging devices are installed, the status of changes in the production line can be analyzed, the causes of the changes can be clarified, and declines in productivity can be suppressed, thereby providing a highly versatile production line monitoring system that can be installed on a variety of production lines.

[0035] FIG. 1 is an explanatory diagram of a production line system including a production line monitoring system according to an embodiment of the present invention and a production line on which the production line monitoring system is installed. FIG. 2 is a block diagram of the production line monitoring system and the production line system according to an embodiment of the present invention. FIG. 3 is a flowchart of production line control processing. FIG. 4 is a flowchart of production line monitoring processing by the production line monitoring system according to an embodiment of the present invention. FIG. 4 is an explanatory diagram of an example of simultaneous playback data. FIG. 5 is an explanatory diagram of a form in which the simultaneous playback data shown in FIG. 5 is displayed on a display device. FIG. 7 is an explanatory diagram of an example of time-series playback data. FIG. 9 is an explanatory diagram of a form in which the time-series playback data shown in FIG. 9 is displayed on a display device. FIG. 10 is an explanatory diagram of an example of group playback data. FIG. 11 is an explanatory diagram of a form in which the group playback data shown in FIG. 9 is displayed on a display device. FIG. 12 is an explanatory diagram of a specific example of an alarm device that visually notifies changes in a production line.

[0036] (Embodiment) As shown in FIG. 1, a manufacturing line monitoring system 1 according to this embodiment is installed on a manufacturing line M where a large number of products (such as industrial product Y1) are transported by belt conveyors M1 to M4 according to processing or assembly procedures.

[0037] (Production Line M) First, we will explain the production line M that is the target of monitoring by the production line monitoring system 1. As shown in Figure 2, the production line M is included in the production line system 2. As shown in Figures 1 and 2, the production line system 2 includes belt conveyors M1 to M4, a status detection sensor (status detection means) A, an alarm device D including a three-color light D1, a production line management device 21 that controls the production line M, a production line input device 24, and a production line display device 25. As shown in Figure 2, the production line management device 21 includes a production line control device 22 and a production line storage device 23.

[0038] (Belt conveyors M1-M4) Belt conveyor M1 has a structure including a conveyor belt wound between a drive pulley connected to a drive motor and a driven pulley, but may also be a roller conveyor. Belt conveyors M2-M4 have a structure similar to that of belt conveyor M1. In this embodiment, for convenience of explanation, as shown in FIG. 1, four belt conveyors M1-M4 are arranged in a straight line in order from upstream to downstream, but various layout shapes are possible. In this embodiment, for convenience of explanation, the production line M includes only belt conveyors M1-M4, but other belt conveyors or work devices may be present before and after the belt conveyors M1-M4. Industrial products Y1, etc., placed on belt conveyor M1 are transported from belt conveyor M1 to belt conveyor M4 (i.e., from upstream to downstream) by the belt conveyors M1-M4.

[0039] (Status detection sensor A) Status detection sensor A is a sensor that detects the status of multiple industrial products transported on production line M, which includes belt conveyors M1 to M4. The signal from status detection sensor A is input to a production line control device 22 of production line management device 21, which will be described later. Then, based on the input signal from status detection sensor A, production line control device 22 of production line management device 21 obtains information indicating changes on production line M.

[0040] In this embodiment, the status detection sensor A uses an optical sensor, which is an example of a motion detection sensor, and is configured to measure the distance between adjacent industrial products among multiple industrial products transported on the belt conveyors M1 to M4. The adjacent industrial products are, as shown in FIG. 1 , between industrial products Y1 and Y2, or between industrial products Y5 and Y6. A signal indicating the distance between adjacent industrial products detected by the status detection sensor A is input to the production line control device 22 of the production line management device 21. When the situation changes from one in which a predetermined distance is maintained between adjacent industrial products, such as between industrial products Y1 and Y2, to one in which the predetermined distance is no longer maintained, such as between industrial products Y5 and Y6, due to, for example, industrial product Y6 colliding with industrial product Y5, the production line control device 22 of the production line management device 21 acquires information indicating a change in the production line M based on the signal from the status detection sensor A.

[0041] Here, the change in the production line M detected by the production line control device 22 of the production line management device 21 is not limited to a change in the distance between adjacent industrial products, but also includes a change to a situation that differs from a pre-determined situation. For example, the change in the production line M detected by the production line control device 22 of the production line management device 21 may be when an industrial product being transported on the production line M is shifted from a pre-determined position, when the orientation of the transported industrial product is different from the pre-determined orientation (including when the product has fallen over), when the type of industrial product being transported is different from the pre-determined type of industrial product, or when an abnormal sound is generated. Note that the state detection sensor A is not limited to a motion detection sensor such as an optical sensor, and state detection means such as an abnormal sound detection sensor or a temperature sensor can be used depending on the detection target.

[0042] (Notification Device D) The notification device D is a device that visually notifies changes in the production line M. The notification device D is controlled by the production line control device 22 of the production line management device 21, and visually notifies changes in the production line M when the production line control device 22 acquires information indicating a change in the production line M. In the present embodiment, the three-color lamp D1 is described as an example of the notification device D. However, the notification device D is not limited to the three-color lamp D1, and may be a meter D2 (see FIG. 11A) that displays numerical values, a monitor screen D3 (see FIG. 11B) that displays numerical values, or a lamp D4 that can be lit or flashed (see FIG. 11C). In other words, the visual notification of a change in the production line M by the notification device D is not limited to a notification by the color of the three-color lamp D1 (which also includes a notification by the red lamp of the three-color lamp D1 being lit), but may be at least one of a notification by the lighting or flashing of the lamp D4, or a notification by a numerical value displayed on the meter D2 or the monitor screen D3.

[0043] The tri-color light D1 in this embodiment is a patrol lamp having, from top to bottom, a red light, a yellow light, and a blue light, and has the function of notifying people (managers, users, etc.) around the production line M of changes in the production line M depending on whether or not the light is on and the color of the light. For example, when information indicating a change in the production line M is acquired using the status detection sensor A, the red light of the tri-color light D1 lights up to notify people around the production line M of the change in the production line M. Note that the lighting of the blue light of the tri-color light D1 does not notify of a change in the production line M but rather that the production line M is operating normally, and the lighting of the yellow light of the tri-color light D1 notifies that a worker is working on the production line M.

[0044] (Production line management device 21) As described above, the production line management device 21 includes the production line control device 22 and the production line storage device 23. The production line management device 21 is also connected to the belt conveyors M1 to M4, the status detection sensor A, the notification device D (three-color light D1), the production line input device 24, the production line display device 25, etc., by wiring or communication.

[0045] The production line controller 22 performs various computer controls for the production line M and is comprised of, for example, a CPU. For example, the production line controller 22 is connected to each drive motor (not shown) of the belt conveyors M1 to M4 of the production line M to control their drive. The production line controller 22 is also connected to a status detection sensor A and acquires information indicating changes in the production line M based on a signal from the status detection sensor A. Specifically, the production line controller 22 determines whether or not there is a change in the production line M based on the signal from the status detection sensor A. If the production line controller 22 detects a change in the production line M, it turns on the three-color light D1 in red. If the production line controller 22 detects a change in the production line M, it stops each drive motor of the belt conveyors M1 to M4, thereby stopping transport on the belt conveyors M1 to M4.

[0046] The production line storage device 23 stores programs and the like for performing various computer controls on the production line M. The production line input device 24 is a device, such as a keyboard or mouse, through which a manager of the production line M or the like inputs various information and instructions. The production line display device 25 is a device, such as a display, that displays an operation screen, output screen, etc. based on commands from the production line control device 22.

[0047] (Manufacturing Line Control Processing) Next, the manufacturing line control processing executed in the manufacturing line M will be described with reference to the flowchart of FIG.

[0048] (Start of conveying process (S1)) First, a manager or the like starts the production line management device 21. When a start operation for the production line M is input by the production line input device 24, the production line control device 22 starts the conveying process (S1). Specifically, in the conveying process, the production line control device 22 starts the belt conveyors M1 to M4 by driving the respective drive motors of the belt conveyors M1 to M4. As a result, industrial products X1 to X10, Y1 to Y10, Z1 to Z10, etc. placed on the belt conveyor M1 are conveyed from the belt conveyor M1 to the belt conveyor M4, i.e., from the upstream side to the downstream side. Note that only industrial products X9 to X10, Y1 to Y10, and Z1 to Z2 are shown in FIG. 1.

[0049] (Detection of changes in production line M (S2)) Next, the production line control device 22 detects changes in the production line M based on the signal of the status detection sensor A, that is, acquires information indicating changes in the production line M based on the signal of the status detection sensor A (S2).

[0050] In this embodiment, for multiple industrial products transported on belt conveyors M1 to M4 of the production line M, information indicating changes in the production line M is obtained from the distance between adjacent industrial products detected by a status detection sensor A, which is a motion detection sensor (optical sensor).

[0051] If the production line control device 22 determines that a predetermined distance is maintained between adjacent industrial products on the production line M, it determines that no change has been detected on the production line M (S2: NO) and repeats the process of S2. On the other hand, if the production line control device 22 determines that a predetermined distance is not maintained between adjacent industrial products on the production line M, it acquires information indicating a change on the production line M (S2: YES).

[0052] (Transport Stop Processing (S3)) When the production line control device 22 acquires information indicating a change in the production line M (S2: YES), the production line control device 22 performs a transport stop processing (S3). In this transport stop processing, the production line control device 22 stops the drive motors of the belt conveyors M1 to M4, thereby stopping the driving of the belt conveyors M1 to M4. This stops the transport of industrial products on the production line M.

[0053] (Notification Process (S4)) After executing the transport stop process (S3), the production line control device 22 performs the notification process (S4). In this notification process, the production line control device 22 lights up the three-color lamp D1 in red.

[0054] When the production line control device 22 acquires information indicating a change in the production line M (S2: YES), the production line control device 22 may execute the transport stop process (S3) and the notification process (S4) at the same time, or may execute the notification process (S4) before executing the transport stop process (S3). Furthermore, in the notification process (S4), in addition to lighting the three-color light D1 in red, the change in the production line M may be notified to those nearby by an alarm sound, or an email may be sent to a manager or the like notifying them of the change in the production line M.

[0055] (Resume Determination (S5)) Next, the production line control device 22 determines whether a resume operation for the production line M has been performed via the production line input device 24 (S5). For example, by analyzing the state of a change in the production line M and identifying its cause, a manager or the like can resolve the change in the production line M. Then, if a resume operation for the production line M has been performed via the production line input device 24 (S5: YES), the production line control device 22 turns off the red illumination of the three-color light D1, returns to S1, and resumes the conveyance process. On the other hand, if a resume operation for the production line M has not been performed via the production line input device 24, such as by performing an operation to shut down the production line management device 21 (S5: NO), the production line control process is terminated.

[0056] (Production Line Monitoring System 1) Next, the production line monitoring system 1 that monitors the production line M will be described. As shown in Figures 1 and 2, the production line monitoring system 1 includes production line monitoring cameras C1 to C4 (corresponding to imaging devices; production line monitoring camera C4 is not shown in Figure 1), an alarm device monitoring camera B (corresponding to an alarm device imaging device), a production line monitoring device 11, a monitoring input device 113, and a monitoring display device 114. The production line monitoring system 1 has the function of monitoring (displaying on a screen and recording) industrial products manufactured by the production line M and the production line M that manufactures them using the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B. The production line monitoring cameras C1 to C4 and the alarm device monitoring camera B are multiple imaging devices installed on the production line M for monitoring the production line M.

[0057] 2 , the production line monitoring system 1 is not electrically connected to the production line system 2 having the production line M, and is an electrically independent system from the production line system 2 having the production line M. However, the production line monitoring system 1 may be a system that is electrically connected to the production line system 2 having the production line M. When the production line monitoring system 1 is a system that is electrically connected to the production line system 2 having the production line M, the production line control device 22 may be configured to output information indicating the change in the production line M, which is acquired in the detection of the change in the production line M (S2), to the monitoring control device 111 via the communication unit 115 of the production line monitoring system 1, so that the information is acquired.

[0058] (Production Line Monitoring Cameras C1-C4) The production line monitoring cameras C1-C4 are capable of capturing images, are installed along the production line M, and are configured to capture images of the production line M. The production line monitoring cameras C1-C4 are installed at equal intervals along the belt conveyors M1-M4M, and are each installed and configured to capture a predetermined capture range of the production line M. In this embodiment, as shown in FIG. 1 , the production line monitoring camera C1 covers within its capture range the transport area downstream of the belt conveyor M1 and the transport area upstream of the belt conveyor M2. The production line monitoring camera C2 covers within its capture range the transport area downstream of the belt conveyor M2 and the transport area upstream of the belt conveyor M3. The production line monitoring camera C3 covers within its capture range the transport area downstream of the belt conveyor M3 and the transport area upstream of the belt conveyor M4. The production line monitoring camera C4 (see FIG. 2) has an imaging range that includes the transport area on the downstream side of the belt conveyor M4.

[0059] The multiple production line monitoring cameras may be installed and configured so as to include at least a portion of the production line M in their imaging range, but it is preferable that they be installed along the production line M so as to include all of the belt conveyors of the production line M in their imaging range. The production line monitoring cameras may also be configured so that their imaging range can be arbitrarily adjusted within a predetermined range by zooming (enlarging / reducing) or the like.

[0060] (Alarm device monitoring camera B) The alarm device monitoring camera B is capable of capturing images and is configured to capture images of the alarm device D. In this embodiment, the alarm device monitoring camera B is installed in a position that includes the tricolor light D1 in its capture range in order to capture the tricolor light D1. In other words, the alarm device monitoring camera B corresponds to an alarm device capture device that captures the alarm device D (tricolor light D1), which visually notifies that information indicating a change in the production line M has been acquired.

[0061] The configuration of the alarm device monitoring camera B is not particularly limited, and a camera with the same configuration as the production line monitoring cameras C1 to C4 may be used. Furthermore, at least some of the production line monitoring cameras C1 to C4 may also serve as the alarm device monitoring camera B. The alarm device monitoring camera B may be configured to allow its capture range to be arbitrarily adjusted within a predetermined range by zooming (enlarging / reducing), etc. Furthermore, if the production line M includes alarm devices D other than the three-color light D1, such as a meter D2 (see FIG. 11A) that displays numerical values, a monitor screen D3 (see FIG. 11B) that displays numerical values, or a lamp D4 that can light or blink (see FIG. 11C), the alarm device monitoring camera B may be installed in a position that includes a selected alarm device D within its capture range. Furthermore, although the present embodiment uses one alarm device monitoring camera B, if there are multiple alarm devices D, multiple alarm device monitoring cameras B may be installed, and multiple alarm device monitoring cameras B may be configured to capture multiple alarm devices D, respectively. Here, the alarm device monitoring camera B may be installed so that all of the alarm devices D (the three-color light D1, the meter D2, the monitor screen D3, and the lamp D4) are included in the shooting range. In other words, if there are multiple alarm devices D, one alarm device monitoring camera B may be configured to shoot one alarm device D, or may be configured to shoot multiple alarm devices D.

[0062] 2, the production line monitoring device 11 includes a monitoring control device 111, a monitoring storage device 112, and a communication unit 115. The communication unit 115 of the production line monitoring device 11 is connected to the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B by wired or wireless communication. The monitoring control device 111 is connected to a monitoring input device 113 and a monitoring display device 114.

[0063] The monitoring and control device 111 performs various computer controls for the production line monitoring system 1, and is configured by, for example, a CPU.

[0064] For example, the monitoring control device 111 performs processing such as sequentially (always) storing the images captured by the production line monitoring cameras C1 to C4 in storage areas of the monitoring storage device 112 corresponding to the production line monitoring cameras C1 to C4 (details will be described later). Also, the monitoring control device 111 performs processing such as sequentially (always) storing the video of the three-color light D1 captured by the alarm device monitoring camera B in a storage area of ​​the monitoring storage device 112 corresponding to the alarm device monitoring camera B.

[0065] Furthermore, the monitoring control device 111 performs image analysis (such as pattern matching, which compares the image captured by the alarm device monitoring camera B with pre-stored images of the tri-color light D1 lit in red, or successively updated machine learning) to determine the lighting color of the tri-color light D1, which is the subject of the image captured by the alarm device monitoring camera B. Then, for example, when the monitoring control device 111 determines that the tri-color light D1 is lit in red, that is, when the monitoring control device 111 determines that the alarm device D has visually notified of a change in the production line M, it determines that it has acquired information indicating a change in the production line M.

[0066] The monitoring storage device 112 stores programs and the like that control various computers in the production line monitoring system 1. The monitoring storage device 112 also has a storage area where necessary information and the like can be rewritten. For example, each of the images captured by the production line monitoring cameras C1 to C4 is stored in the corresponding storage area.

[0067] The monitoring input device 113 is a device, such as a keyboard or a mouse, through which a manager of the production line M or the like inputs various information and instructions. The monitoring display device 114 is a device, such as a display, for displaying an operation screen, an output screen, etc. based on instructions from the monitoring control device 111.

[0068] (Manufacturing Line Monitoring Process) Next, the manufacturing line monitoring process executed in the manufacturing line monitoring system 1 will be described with reference to the flowchart of FIG. 4 and the explanatory diagrams of FIGS.

[0069] First, an administrator or the like starts up the production line monitoring device 11. The monitoring input device 113 inputs a start-up operation for the alarm device monitoring camera B and the production line monitoring cameras C1 to C4. At this time, for example, four frames (a screen divided into four parts) are displayed by software on a single screen of the monitoring and display device 114, and the image of the tri-color light D1 captured by the alarm device monitoring camera B and any four of the images captured by the production line monitoring cameras C1 to C3 are displayed in real time in the four frames. Note that the manner in which the images are displayed in a list on the screen of the monitoring and display device 114 is merely an example and is not limited to the manner shown in the figure. Alternatively, the single screen of the monitoring and display device 114 may be divided into multiple frames, and a required number of images from the image of the tri-color light D1 captured by the alarm device monitoring camera B and the images captured by the production line monitoring cameras C1 to C3 may be displayed.

[0070] (Start of Continuous Storage Process (S10)) When a start operation is input, the monitoring control device 111 starts the continuous storage process (S10). In this continuous storage process, the monitoring control device 111 sequentially (constantly) acquires multiple images captured by each of the production line monitoring cameras C1 to C4 and stores them as multiple pieces of time-series data associated with each of the production line monitoring cameras C1 to C4 in storage areas of the monitoring storage device 112 corresponding to each of the production line monitoring cameras C1 to C4. For example, the time-series data from the production line monitoring camera C1 is stored in folder 112C1 (not shown) of the monitoring storage device 112, the time-series data from the production line monitoring camera C2 is stored in folder 112C2 (not shown) of the monitoring storage device 112, and the time-series data from the production line monitoring camera C3 is stored in folder 112C3 (not shown) of the monitoring storage device 112. The time-series data from the production line monitoring camera C4 is stored in folder 112C4 (not shown) of the monitoring storage device 112.

[0071] Furthermore, in a continuous storage process, the monitoring control device 111 sequentially (constantly) acquires a plurality of images taken by the alarm device monitoring camera B and stores them in a storage area of ​​the monitoring storage device 112 corresponding to the alarm device monitoring camera B as time-series data associated with the alarm device monitoring camera B. For example, the time-series data of the alarm device monitoring camera B is stored in a folder 112B (not shown) of the monitoring storage device 112.

[0072] In addition, the time series data captured by the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B, which are stored in folders 112C1 to 112C4 and folder 112B (not shown), respectively, are overwritten after a predetermined number of days so as not to exceed the storage capacity of the monitoring storage device 112.

[0073] (Start of Real-Time Output Processing (S11)) The monitoring control device 111 starts the real-time output processing together with the start of the continuous storage processing (S11). In the real-time output processing, the monitoring control device 111 outputs multiple images captured by the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B, which are constantly acquired in the continuous storage processing, as data that can be displayed in a list on the screen of a display device. This display device is, for example, the monitoring and display device 114 of the production line monitoring system 1. The display device may also be a display device of a system external to the production line monitoring system 1. Note that the display device of the system external to the production line monitoring system 1 may or may not include the production line display device 25 of the production line M. In the real-time output processing, for example, multiple frames may be displayed by software on one screen of the monitoring and display device 114, and the images captured by the alarm device monitoring camera B and the production line monitoring cameras C1 to C4 may be displayed in real time in the multiple frames. Here, the number of frames (frames for multiple images) displayed on one screen of the monitor / display device 114 in real-time output processing can be set arbitrarily.

[0074] (Acquiring Information Indicating a Change in the Production Line M (S12)) Next, the monitoring control device 111 executes a change detection process to determine whether or not information indicating a change in the production line M has been acquired based on a determination of whether or not a change in the production line M has been visually notified (S12). First, images of the alarm device D are constantly acquired from the alarm device monitoring camera B, and it is determined from the images of the alarm device D whether or not the alarm device D has visually notified a change in the production line M. In this embodiment, the monitoring control device 111 determines whether the tricolor light D1 is lit in red from images of the tricolor light D1 constantly acquired by the alarm device monitoring camera B.

[0075] Specifically, the monitoring control device 111 performs image analysis on the image of the three-color light D1 captured by the alarm device monitoring camera B to determine whether the three-color light D1, which is the subject of the image captured by the alarm device monitoring camera B, is lit in red (S12).

[0076] Then, when the monitoring control device 111 determines that the alarm device D has visually notified a change in the production line M, it determines that it has acquired information indicating a change in the production line M. Here, in this embodiment, when the tri-color light D1 is lit red in the above-mentioned alarm processing (S4), the monitoring control device 111 performs image analysis of the image of the tri-color light D1 captured by the alarm device monitoring camera B, determines that the tri-color light D1, which is the subject of the image captured by the alarm device monitoring camera B, is lit red, and determines that it has acquired information indicating a change in the production line M (S12: YES). That is, in the processing of S12, a change detection processing is performed in which it is determined that it has acquired information indicating a change in the production line M by image analysis of the visual alarm image of the tri-color light D1 captured by the alarm device monitoring camera B.

[0077] If the alarm device D photographed by the alarm device monitoring camera B is not the three-color lamp D1, the change detection process is executed as follows: If the alarm device D photographed by the alarm device monitoring camera B is lamp D4 (see FIG. 11C ), and the “status” lamp of lamp D4 is lit, the monitoring control device 111 performs image analysis of the image of lamp D4 photographed by the alarm device monitoring camera B, and determines that the “status” lamp of lamp D4 photographed by the alarm device monitoring camera B is lit, and determines that information indicating a change in the production line M has been acquired. Furthermore, when the alarm device D photographed by the alarm device monitoring camera B is a meter D2 or a monitor screen D3, the image of the numerical value of the meter D2 or the image of the numerical value of the monitor screen D3 photographed by the alarm device monitoring camera B is subjected to image analysis (such as comparing it with a numerical range indicating normality and a numerical range (error range) indicating a change), and if it is determined that the numerical value of the meter D2 or the numerical value of the monitor screen D3 has changed compared to the numerical range indicating normality, it is determined that information indicating a change in the production line M has been obtained.

[0078] (Event Storage Process (S13)) Next, when the monitoring control device 111 determines that it has acquired information indicating a change in the production line M (S12: YES), it performs event storage process (S13).

[0079] In this event storage process, for all of the multiple production line monitoring cameras C1 to C4, the monitoring control device 111 performs a constant storage process, extracting multiple groups of images from multiple time series data associated with each of the production line monitoring cameras C1 to C4, which are stored in folders 112C1 to 112C4 (not shown) in the monitoring memory device 112, during the change detection process (S12: YES), for a change period including the time when the monitoring control device 111 determines that the alarm device D has visually notified the change on the production line M (for example, one minute of video footage including the time of change and the time before and after the change), and stores these in a specified folder in the monitoring memory device 112 as clip videos P1 to P4 corresponding to the production line monitoring cameras C1 to C4, respectively. That is, the event storage process (S13) is triggered by determining that information indicating a change in the production line M has been acquired from the image of the tricolor light D1 captured by the alarm device monitoring camera B. Clip images P1 to P4 covering the same period, which is a change period including the time of the change and at least the time period before the change, are simultaneously generated from all of the multiple production line monitoring cameras C1 to C4 installed along the production line M. In this embodiment, the change period includes the time when it is determined through image analysis that the tricolor light D1 is lit red (the time when it is determined that information indicating a change in the production line M has been acquired), and also includes the time period before and the time period after the change. Note that the change period may include the time of the change and at least the time period before the change, among the time periods before and after the change, and may include the time period before the change, but not the time period after the change.

[0080] Furthermore, in the event storage process, the monitoring control device 111 may extract multiple groups of images during a change period from the time-series data associated with the alarm device monitoring camera B that was stored in folder 112B (not shown) of the monitoring storage device 112 in the continuous storage process, and store the extracted images in a predetermined folder in the monitoring storage device 112 as clip video Q corresponding to the alarm device monitoring camera B. In other words, in the event storage process, the monitoring control device 111 extracts multiple groups of images during a change period from the multiple sets of time-series data associated with at least each of the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B, and stores the extracted images in the monitoring storage device 112 as clip video.

[0081] For example, clip video P1 extracted from the time series data associated with production line monitoring camera C1 is stored in folder 112P1 (not shown) of the monitoring storage device 112, clip video P2 extracted from the time series data associated with production line monitoring camera C2 is stored in folder 112P2 (not shown) of the monitoring storage device 112, clip video P3 extracted from the time series data associated with production line monitoring camera C3 is stored in folder 112P3 (not shown) of the monitoring storage device 112, clip video P4 extracted from the time series data associated with production line monitoring camera C4 is stored in folder 112P4 (not shown) of the monitoring storage device 112, and clip video Q extracted from the time series data captured by alarm device monitoring camera B is stored in folder 112Q (not shown) of the monitoring storage device 112. It should be noted that the clip images P1 to P4 and clip image Q are stored in folders 112P1 to 112P4 and 112Q that are different from the folders 112C1 to 112C4 and 112B, and therefore will not be overwritten even after a predetermined number of days have passed.

[0082] In addition, when the monitoring control device 111 determines that it has acquired information indicating a change in the production line M, it may record a flag F1 indicating the time at which the information indicating a change in the production line M was acquired in the monitoring storage device 112 so that the flag F1 is synchronized with the clip images P1 to P4 and the clip image Q, respectively.

[0083] On the other hand, if the monitoring control device 111 does not determine that it has acquired information indicating a change in the production line M (S12: NO), it repeats the processing of S12 to continue determining whether the three-color light D1 is lit in red based on the image captured by the alarm device monitoring camera B.

[0084] (Output Process (S14)) Next, after the event storage process of S13, the monitoring control device 111 performs an output process (S14). In the output process, at least some of the clip videos (at least two of clip videos P1-P4 and clip video Q) corresponding to at least two of the production line monitoring cameras C1-C4 and the alarm device monitoring camera B are output as data that can be displayed in a list on the screen of a display device and played back. "At least some of the clip videos" refers to video of at least a part of a change period of the clip videos. This display device is, for example, the monitoring and display device 114 included in the production line monitoring system 1. The display device may be included in a system external to the production line monitoring system 1. The display device included in the system external to the production line monitoring system 1 may or may not include the production line display device 25 included in the production line M. The at least two imaging devices among the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B may be at least two production line monitoring cameras among the production line monitoring cameras C1 to C4, or at least one production line monitoring camera among the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B.

[0085] In the output process (S14), the monitoring control device 111 selects which of three types of playback data, i.e., simultaneous playback data, time-series playback data, and group playback data, to output based on an instruction input to the monitoring input device 113. In the output process (S14), the monitoring control device 111 may output preset playback data (e.g., simultaneous playback data) immediately after the event storage process (S13), and then, when an instruction is input to the monitoring input device 113, output playback data based on the input instruction. The playback data automatically output immediately after the event storage process (S13) and the playback data output based on an instruction input after the event storage process may be different types of playback data from the three types of playback data, or may be playback data of the same type but with different settings. Furthermore, in the output process (S14), the monitoring control device 111 may output playback data based on the input instruction after an instruction is input to the monitoring input device 113 after the event storage process (S13).

[0086] The following describes the output process (S14) when simultaneous playback data, time-series playback data, and group playback data are output. In the following description, an example is given in which data is output to the monitoring and display device 114, but the display device to which the data is output is not limited to the monitoring and display device 114, as described above.

[0087] (Output of simultaneous playback data) The monitoring control device 111 extracts video from the same time period from at least two clip videos out of clip videos P1 to P4 and clip video Q, and outputs the extracted videos to the monitoring display device 114 as simultaneous playback data that can be displayed in a list on the screen of the monitoring display device 114 and played back synchronously at the same time. It is preferable that the at least two clip videos that form the source of the simultaneous playback data include at least two clip videos out of clip videos P1 to P4. The at least two clip videos that form the source of the simultaneous playback data may or may not include clip video Q. Specific examples of simultaneous playback data and specific examples of how the simultaneous playback data is played back will be described later.

[0088] (Output of Time-Series Playback Data) The monitoring control device 111 extracts, from at least two of the clip videos P1-P4 (i.e., from multiple clip videos corresponding to at least two of the multiple production line monitoring cameras C1-C4), images of a specific industrial product being transported on the production line M, captured at different time periods, and outputs the extracted multiple images to the monitoring display device 114 as time-series playback data that can be displayed and played back in a list on the screen of the monitoring display device 114. In other words, the time periods of the images extracted from the at least two clip videos included in the time-series playback data are different from each other. The specific industrial product is set (selected) based on information or instructions input to the monitoring input device 113. For example, if a manager or other person identifies an industrial product that is presumed to have caused a change on the production line M by outputting, displaying, and playing back the simultaneous playback data to the monitoring display device 114, the manager or other person may input information or instructions to the monitoring input device 113 to identify this industrial product as the specific industrial product. The time periods of the images extracted from at least two of the clip images P1 to P4 may be continuous or may partially overlap. The multiple images of a specific industrial product captured in different time periods extracted from at least two clip images include images of a first time period including the time of change and images of a second time period including the time period before the time of change. The second time period may partially overlap with the first time period or may be continuous with the first time period. It is preferable that the second time period is not separated from the first time period. The second time period may or may not include a time period after the time of change. The multiple images of a specific industrial product captured in different time periods extracted from at least two clip images may include, in addition to the images of the first time period and the second time period, images of a third time period including a time period after the first time period but excluding a time period before the first time period. In this case, time-series playback data is generated from at least three of the clip images P1 to P4.The monitoring control device 111 may output multiple videos of different time periods extracted from at least two of the clip videos P1 to P4 included in the time-series playback data, and at least a portion of the video of clip video Q, as data that can be displayed and played back in a list on the screen of the monitoring display device 114. In other words, the data output when the output of time-series playback data is selected by the monitoring input device 113 may include the time-series playback data and at least a portion of the video of clip video Q. In this case, the time period of at least a portion of the video of clip video Q included in the output data is a time period that includes a change time, and may be, for example, the time period of all the multiple videos included in the time-series playback data, or the first time period described above. Specific examples of time-series playback data and specific examples of how the time-series playback data is played back will be described later.

[0089] (Output of Group Playback Data) The monitoring control device 111 extracts video images of multiple industrial products grouped under predetermined conditions from at least two of the clip images P1-P4 (i.e., from multiple clip images corresponding to at least two of the multiple production line monitoring cameras C1-C4), and outputs the extracted video images to the monitoring display device 114 as group playback data that can be displayed and played back in a list on the screen of the monitoring display device 114. Grouping under predetermined conditions may involve setting multiple industrial products included in the same lot as one group. Generally, the distance between an industrial product located at the end of a lot and an industrial product located at the end of an adjacent lot is wider than the distance between adjacent industrial products included in the same lot. The monitoring control device 111 may, for example, determine lot boundaries from images captured by the production line monitoring cameras C1-C4. By determining the two boundaries between the lots, the start and end points of the groups are identified. The monitoring control device 111 may then determine which clip videos P1 to P4 to extract to generate group playback data based on information about the start and end points of the group and the conveyance speed of the production line M. The group serving as the basis for generating group playback data is set (selected) based on information or instructions input to the monitoring input device 113. If the clip videos P1 to P4 stored in the event storage process (S13) do not capture multiple groups but only a single group, the group serving as the basis for generating group playback data may be automatically set (selected) without waiting for instructions or information input from the monitoring input device 113. The multiple videos extracted from at least two clip videos P1 to P4 include videos from a time period that includes a change. The multiple videos extracted from at least two clip videos P1 to P4 may include at least one of a video of an industrial product located at the most downstream side in the conveyance direction among the multiple industrial products belonging to the group and a video of an industrial product located at the most upstream side in the conveyance direction among the multiple industrial products belonging to the group.The video of the industrial product located furthest downstream in the conveying direction among the multiple industrial products belonging to the group and the video of the industrial product located furthest upstream in the conveying direction among the multiple industrial products belonging to the group may or may not be video of a time period that includes a change time. The monitoring control device 111 may output multiple videos of different time periods extracted from at least two of the clip videos P1 to P4 included in the group playback data, and at least a portion of the video of clip video Q, as data that can be displayed and played back in a list on the screen of the monitoring display device 114. In other words, the data output when output of group playback data is selected by the monitoring input device 113 may include the group playback data and at least a portion of the video of clip video Q. In this case, the time period of at least a portion of the video of clip video Q included in the output data is a time period that includes a change time, and may be, for example, the time period of all the multiple videos included in the group playback data, or the same time period as any of the multiple videos included in the group playback data. Specific examples of group playback data and specific examples of how to play back the group playback data will be described later.

[0090] (Resume Determination (S15)) After the output process (S14), the monitoring control device 111 determines whether a resume operation of the production line monitoring system 1 has been performed via the monitoring input device 113 (S15). For example, after acquiring information indicating a change in the production line M, if the cause of the change is resolved, and a manager or the like of the production line M confirms safety, and then a resume operation of the production line monitoring system 1 is performed via the monitoring input device 113 (S15: YES), the process returns to S11. On the other hand, if a resume operation is not performed, such as by shutting down the production line monitoring system 1 via the monitoring input device 113 (S15: NO), the production line monitoring process is terminated. Note that in the production line monitoring process executed by the monitoring control device 111 of this embodiment, the continuous storage process is configured not to stop the process, regardless of whether it is determined that information indicating a change in the production line M has been acquired in the change detection process (S12). However, in the production line monitoring process executed by the monitoring control device 111, the continuous storage process may be configured to be stopped when it is determined in the change detection process (S12) that information indicating a change in the production line M has been acquired (S12: YES), and to be resumed upon a resume determination (S15: YES).

[0091] Next, specific examples of simultaneous playback data, time-series playback data, and group playback data and their playback modes will be described. In the following description, the simultaneous playback data, time-series playback data, and group playback data will be described as examples in which they are output to the monitoring and display device 114, but the display device to which they are output is not limited to the monitoring and display device 114, as described above.

[0092] (Playback of Simultaneous Playback Data) Figure 6 shows an example of the screen of the monitoring and display device 114 when simultaneous playback data is output to and displayed on the monitoring and display device 114. In the example of Figure 6, the simultaneous playback data includes four videos from the same time period that have been extracted from clip videos P1 to P3 and clip video Q, among clip videos P1 to P4 and clip video Q. In Figure 6, four frames (a screen divided into four) are displayed by software on one screen of the monitoring and display device 114, and the four videos included in the simultaneous playback data are displayed in a list in the four frames. These four videos are then played back in synchronization at the same time. In Figure 6, a playback bar displayed at the bottom of the screen of the monitoring and display device 114 visually indicates the total playback time of the four videos, and a playback position indicator F2 on the playback bar indicates the current playback time common to all four videos. By operating the monitor input device 113 to slide the playback position indicator F2 of the monitor / display device 114 along the playback bar, the playback time can be changed and the four videos can be played back while maintaining a synchronized and linked state at the same time. In this embodiment, one screen of the monitor / display device 114 is divided into four frames, and clip video Q corresponding to the alarm device monitoring camera B and clip videos P1 to P3 corresponding to the production line monitoring cameras C1 to C3 are displayed, but this is not limited to this. As a form in which the videos are displayed in a list on the screen of the monitor / display device 114, one screen of the monitor / display device 114 may be divided into multiple frames, and a required number of videos from clip video Q corresponding to the alarm device monitoring camera B and clip videos P1 to P4 corresponding to the production line monitoring cameras C1 to C4 may be displayed.

[0093] Fig. 5 is a diagram illustrating the simultaneous playback data displayed in the example of Fig. 6. As shown in Fig. 5, the clip video change period is one minute including the time of change, and the clip video P1 is divided into 15 units (4 seconds each) into unit videos P1 / T1, P1 / T2, P1 / T3, ... P1 / T15, the clip video P2 is divided into 15 units (4 seconds each) into unit videos P2 / T1, P2 / T2, P2 / T3, ... P2 / T15, and the clip video P3 is divided into 15 units (4 seconds each). From unit videos P1 / T5-T9, P2 / T5-T9, P3 / T5-T9, and Q / T5-T9, including unit videos P1 / T8, P2 / T8, and P3 / T8 at the time of change, are extracted from P3 / T1, P3 / T2, P3 / T3...P3 / T15, and unit videos Q / T1, Q / T2, Q / T3...Q / T15 obtained by dividing clip video Q into 15 sections (4-second intervals), and simultaneous playback data is generated. Note that in Figures 7 and 9 (described below), the clip video change period is one minute, including both before and after the change, and the unit videos at the time of change when clip videos P1-P3 and clip video Q are each divided into 15 sections (4-second intervals) are P1 / T8, P2 / T8, P3 / T8, and Q / T8. Note that, although the clip video change period is set to 60 seconds in this embodiment, it is not limited to this and can be set as desired. In this embodiment, the unit video of the clip video is divided into 15 sections (4-second sections), but this is not limitative and can be set arbitrarily.

[0094] (Playback of Time-Series Playback Data) FIG. 8 shows an example of the screen of the monitoring and display device 114 when the time-series playback data and at least a portion of the video of clip video Q are output to and displayed on the monitoring and display device 114. In the example of FIG. 8, the time-series playback data includes three videos of a specific industrial product Y6 taken from clip video P1-P3 among clip videos P1-P4, captured in different time periods. In FIG. 8, four frames (a screen divided into four parts) are displayed by software on one screen of the monitoring and display device 114, and the three videos included in the time-series playback data and at least a portion of the video of clip video Q are displayed in a list in the four frames. These four videos are then played back in chronological order. In FIG. 8, a playback bar displayed at the bottom of the screen of the monitoring and display device 114 visually indicates the total playback time of the four videos, and a playback position indicator F2 on the playback bar indicates the current playback time of one of the four videos. It should be noted that a playback bar and a playback position indicator may be displayed below each of the four videos instead of the playback position indicator F2 on the playback bar shown in Fig. 8. In Fig. 8, the video extracted from clip video P2 is currently being played, the video extracted from clip video P1 is stopped in the state it was in when playback ended, and the video extracted from clip video P3 is stopped in the state it was in when playback started.

[0095] FIG. 7 is a diagram illustrating the time-series playback data and at least some of the images of clip video Q displayed in the example of FIG. 8. As shown in FIG. 7, unit videos P2 / T7-T9 of a first time period including the time of change are extracted from clip video P2, unit videos P1 / T5-T7 of a second time period including the time period before the time of change are extracted from clip video P1, and unit videos P3 / T9-T11 of a third time period including the time period after the time of change are extracted from clip video P3, thereby generating time-series playback data. Unit videos P2 / T7-T9, unit videos P1 / T5-T7, and unit videos P3 / T9-T11 depict industrial product Y6. Also, as shown in FIG. 7, at least some of the images of clip video Q displayed in a list with the three images included in the time-series playback data are unit videos Q / T5-T11, which are images from the same time period as the entire time period of the three images extracted from clip videos P1-P3.

[0096] (Playback of Group Playback Data) Figure 10 shows an example of the screen of the monitoring and display device 114 when the group playback data and at least a portion of the video of clip video Q are output to and displayed on the monitoring and display device 114. In the example of Figure 10, the group playback data includes three videos of ten industrial products Y1 to Y10 belonging to one group, extracted from clip videos P1 to P3 among clip videos P1 to P4. In Figure 10, four frames (a screen divided into four) are displayed by software on one screen of the monitoring and display device 114, and the three videos included in the group playback data and at least a portion of the video of clip video Q are displayed in a list in the four frames. These four videos are then played back in chronological order. In Figure 10, a playback bar displayed at the bottom of the screen of the monitoring and display device 114 visually indicates the playback time of the entire four videos, and a playback position indicator F2 on the playback bar indicates the current playback time of one of the four videos. It is also possible to display a playback bar and a playback position indicator below each of the four videos instead of the playback position indicator F2 on the playback bar shown in Fig. 10. In Fig. 10, the video extracted from clip video P2 is being played from time T7 to time T11, playback is on standby for video T12, which is the first unit video extracted from clip video P1, and playback has stopped for video T6, which is the last unit video extracted from clip video P3.

[0097] FIG. 9 is a diagram illustrating the group playback data and at least some of the images in clip video Q displayed in the example of FIG. 10. As shown in FIG. 9, unit videos P1 / T12-T15 are extracted from clip video P1, unit videos P2 / T7-T11 in a time period including a transition are extracted from clip video P2, and unit videos P3 / T2-T6 are extracted from clip video P3 to generate group playback data. Unit videos P3 / T2-T6 are shot of industrial product Y1, which is located at the most downstream side in the conveying direction among industrial products Y1-Y10. Unit videos P1 / T12-T15 are shot of industrial product Y10, which is located at the most upstream side in the conveying direction among industrial products Y1-Y10. Also, as shown in FIG. 9, at least some of the images in clip video Q displayed in a list with the three images included in the group playback data are unit videos Q / T2-T15, which are images from the same time period as the entire time period of the three images extracted from clip videos P1-P3.

[0098] According to the above-described production line monitoring system 1, when the monitoring control device 111 acquires information indicating a change in the production line M, it extracts video footage of a period of change from a plurality of time-series data associated with at least two of the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B (all of the monitoring cameras in this embodiment), and stores the extracted video footage as clip video P1 to P4 and clip video Q associated with the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B in folders 112C1 to 112C4 and 112B of the monitoring storage device 112. In other words, triggered by the determination by the monitoring control device 111 of the production line monitoring system 1 that it has acquired information indicating a change in the production line M, clip video footage of the same period of change, which includes the time of the change and at least the time period before the change, is generated simultaneously for the production line monitoring cameras C1 to C4 and the alarm device monitoring camera B in a synchronized manner. Then, from the data output by the output processing of the monitoring and controlling device 111 of the production line monitoring system 1, the images of at least two of the clip videos P1-P4 and clip video Q for at least a portion of the change period can be displayed in a list on the screen of a display device. A manager of the production line M can consolidate and play back the images of at least two of the clip videos P1-P4 and clip video Q for at least a portion of the change period on the monitoring and controlling device 114 of the production line monitoring system 1 or on the screen of a display device of a system external to the production line monitoring system 1. In other words, by viewing a list of images of at least two of the clip videos P1-P4 and clip video Q for at least a portion of the change period, the manager of the production line M can confirm at a glance the state of the production line and the products at the time of the change and at least the time period prior to the change. This allows the manager of the production line M to easily check the state of the change in the production line M based on the clip videos, and by identifying the cause of the change in the production line M and taking appropriate action, productivity declines can be further reduced.It is also possible to perform predictive maintenance for changes by grasping signs immediately before they occur on the production line M. Furthermore, analysis of the process flow of the production line M (movement of people and objects such as industrial products) can be utilized to improve production efficiency (improvement of the layout of the production line M, reduction of work time, etc.).

[0099] Furthermore, according to the above-described production line monitoring system 1, a manager of the production line M can use the monitoring input device 113 to select and instruct whether at least two of the clip videos P1-P4, or at least two of the clip videos P1-P4 and clip video Q, are to be output as time-series playback data, simultaneous playback data, or group playback data. This allows the manager to select data output by the output process based on the optimal clip video for analyzing the cause of a change in the production line M. Specifically, the data output by the output process based on the clip video can be used for purposes such as predictive maintenance of changes in the production line M by identifying signs of changes immediately before they occur, analyzing the flow of people and objects (such as industrial products), and improving production efficiency (improving the layout of the production line M, reducing work time, etc.). Therefore, the manager of the production line M can more easily check the status of changes in the production line M based on the clip video, and by identifying and addressing the causes of changes in the production line M, productivity declines can be further reduced.

[0100] Furthermore, according to the above-described production line monitoring system 1, by playing back the output simultaneous playback data, video images from the same time period are extracted from multiple clip videos (at least two of clip videos P1 to P4), and the extracted multiple clip videos are displayed in a list on the screen of the monitoring and display device 114 and played back synchronously at the same time. In other words, the multiple clip videos can be linked so that their times are synchronized and displayed in a list on the screen of the monitoring and display device 114. This allows managers of the production line M, etc., to simultaneously view multiple clip videos synchronized to the same time and check the status of the production line M and industrial products at the same time before and after a change in the production line M, and identify the location of the change in the production line M and the industrial products that caused the change. Therefore, managers of the production line M, etc., can more easily check the status of changes in the production line M based on the clip videos, and by identifying and addressing the cause of the change in the production line M, productivity declines can be further reduced.

[0101] Furthermore, according to the above-described production line monitoring system 1, by playing back time-series playback data, footage of a specific industrial product being transported on the production line M, captured at different time periods, is extracted from a plurality of clip videos (at least two of the clip videos P1 to P4), and the extracted footage is displayed and played back in a list on the screen of the monitoring and display device 114. For example, by linking the plurality of videos so that the footage of the specific industrial product is in chronological order and displaying and playing back the footage in a list on the screen of the monitoring and display device 114, the manager of the production line M or the like can check the series of states of the specific industrial product being transported from upstream to downstream on the production line M in chronological order. This allows the manager of the production line M or the like to view the state of the specific industrial product being transported from upstream to downstream on the production line M in chronological order, check the state of the specific industrial product when a change occurs on the production line M in a list, and check the location on the production line M where a change occurred for the specific industrial product. Therefore, managers of the production line M can more easily check the state of changes in the production line M based on the clip footage, and by identifying the cause of the changes in the production line M and taking appropriate measures, they can further prevent a decline in productivity.

[0102] Furthermore, with the above-described production line monitoring system 1, by playing back group playback data, managers and the like can view, at a glance, multiple videos of multiple industrial products grouped under predetermined conditions. In this embodiment, from multiple clip videos (at least two clip videos among clip videos P1 to P4), videos of industrial products of the same lot being transported on the production line M are extracted, and the extracted multiple videos, either taken at the same time or at different times, are displayed and played back in a list on the screen of the monitoring and display device 114. In other words, by linking multiple videos of industrial products of the same lot and displaying and playing them in a list on the screen of the monitoring and display device 114, managers and the like of the production line M can check the status of the industrial products of the same lot being transported on the production line M. This allows managers and the like to check the status of multiple grouped industrial products at a glance. Therefore, managers and the like of the production line M can more easily check the status of changes in the production line M based on the clip videos, and by identifying and addressing the causes of changes in the production line M, productivity declines can be further reduced.

[0103] Furthermore, according to the above-described production line monitoring system 1, by playing back the group playback data, from the plurality of clip videos (at least two clip videos out of clip videos P1 to P4), videos of the industrial product located furthest downstream in the conveyance direction and the industrial product located furthest upstream in the conveyance direction among the plurality of grouped industrial products are extracted, and the extracted plurality of videos are displayed and played back in a list on the screen of the monitoring and display device 114. In other words, by linking the plurality of videos, videos of the plurality of grouped industrial products being transported from upstream to downstream along the production line M, and displaying and playing back the linked videos in a list on the screen of the monitoring and display device 114, the manager of the production line M, etc., can check the status of the plurality of grouped industrial products being transported along the production line M. This allows the manager of the production line M to check the series of states of the plurality of grouped industrial products being transported along the production line M in a list in chronological order, including the timing of changes in the production line M, and can identify the industrial products and locations that caused changes in the production line M. For example, if there is an industrial product among the grouped industrial products that has caused a change in the production line M, the manager or the like can check the status of the grouped industrial products at a glance, such as how far the influence of the industrial product that caused the change in the production line M has spread to the grouped industrial products (whether it has affected the start point of the group (the industrial product located furthest downstream in the conveying direction) or the end point of the group (the industrial product located furthest upstream in the conveying direction)). Therefore, the manager or the like of the production line M can more easily check the status of the change in the production line M based on the clip video, and by identifying the cause of the change in the production line M and taking action, it is possible to further prevent a decline in productivity.

[0104] Furthermore, because the monitoring and control device 111 performs real-time output processing, multiple images captured by the alarm device monitoring camera B and the production line monitoring cameras C1-C4, which are constantly acquired through continuous storage processing, can be aggregated and displayed in real time on the monitor and display device 114 of the production line monitoring system 1 or on the screen of a display device of a system external to the production line monitoring system 1. This allows managers and others to check the real-time status of the production line M by viewing a list of the images captured by the alarm device monitoring camera B and the production line monitoring cameras C1-C4 aggregated and played back on the display screen. Therefore, managers and others of the production line M can easily check not only the status of changes in the production line M but also the real-time status of the production line M based on the clipped images, and by identifying the cause of the changes in the production line M and taking appropriate action, a decline in productivity can be further suppressed.

[0105] Furthermore, according to the above-described production line monitoring system 1, the monitoring control device 111 can acquire information indicating a change in the production line M by analyzing images captured by the notification device monitoring camera B, which captures the notification device D that visually notifies changes in the production line M, to determine whether a change in the production line M has been visually notified. The supervisor of the production line M can then consolidate and play back at least a portion of the change period for at least two video clips on the monitoring and display device 114 of the production line monitoring system 1 or on a display device of a system external to the production line monitoring system 1. In other words, by viewing at least a portion of the change period for at least two video clips at a glance, the supervisor of the production line M can confirm at a glance the status of the production line M and the industrial products at the time of the change and at least the time period prior to the change. This allows the supervisor of the production line M to more easily confirm the status of the change in the production line M based on the video clips, and by identifying the cause of the change in the production line M and taking appropriate action, a decline in productivity can be further suppressed.

[0106] This allows managers of the production line M to easily view the stored clip images P1-P4 and check the status of the production line M and industrial products Y1-Y10, etc., before and after a change is detected on the production line M. In other words, based on the stored clip images P1-P3, changes in the production line M can be analyzed retroactively from the time of the change, identifying the cause of the change and preventing a decline in productivity. Furthermore, the existing production line M equipped with an alarm device D that visually notifies changes on the production line M and the newly installed production line monitoring system 1 equipped with multiple production line monitoring cameras C1-C3 and an alarm device monitoring camera B can be installed independently of the existing production line M without electrical connection. This allows the system to be installed on a variety of production lines, making it highly versatile.

[0107] (Other embodiments) (1) In the above embodiment, it was explained that the data output in the output process (S14) may or may not include at least a portion of the video of the clip video Q corresponding to the alarm device monitoring camera B. However, in the output process (S14), a portion of the video of at least two of the clip videos P1 to P4 and the video of the alarm device monitoring camera B that is constantly acquired by the constant storage process may be output as data that can be displayed in a list on the screen of a display device and played back.

[0108] (2) In the above embodiment, the monitoring control device 111 determines that it has acquired information indicating a change in the production line M when it determines that the notification device D has visually notified a change in the production line M from the image captured by the notification device monitoring camera B. However, this is not limited to this. The monitoring control device 111 may determine whether a change has occurred in the production line M through image analysis from the images captured by the production line monitoring cameras C1 to C4, and when it determines that a change has occurred in the production line M, it may determine that it has acquired information indicating a change in the production line M. The image analysis may be an analytical method using machine learning by AI (artificial intelligence). More specifically, by learning patterns using large amounts of data and establishing rules for discriminating unknown data, it may be possible to use a deep neural network (DNN) or a convolutional neural network (CNN) to determine the content of the image and estimate the posture of an industrial product. In this case, by image analysis using AI, when an industrial product being transported on the production line M is shifted from a preset position, when the orientation of the transported industrial product is different from the preset orientation (including when the product has fallen over), or when the type of the transported industrial product is different from the preset type of industrial product, it is possible to automatically and accurately obtain information indicating a change in the production line M. Furthermore, as described above, when the production line monitoring system 1 is a system electrically connected to the production line system 2 having the production line M, the monitoring control device 111 may determine that it has obtained information indicating a change in the production line M when it obtains information indicating a change in the production line M based on a signal from the status detection sensor A from the production line control device 22 via the communication unit 115.

[0109] (2-1) When the monitoring control device 111 determines that a change has occurred on the production line M from images captured by the production line monitoring cameras C1 to C4, the production line monitoring system does not need to include the alarm device monitoring camera B. In this case, the production line monitoring system does not need to include the status detection sensor A.

[0110] (2-2) When the monitoring control device 111 determines that a change has occurred on the production line M from images captured by the production line monitoring cameras C1 to C4, the time-series playback data may include multiple images taken in different time periods, extracted from at least two clip images of the clip images P1 to P4, in which the industrial product included in the image in which it has been determined that a change has occurred on the production line M. In other words, the above-mentioned specific industrial product may be the industrial product included in the image in which it has been determined that a change has occurred on the production line M.

[0111] The industrial product captured in the image in which it is determined that a change has occurred in the production line M is likely to be the industrial product that caused the change in the production line M. According to the above configuration, it is possible to chronologically play back the series of events in which a specific industrial product, which is the industrial product captured in the image in which it is determined that a change has occurred in the production line M, is transported from upstream to downstream along the production line M. This allows the manager of the production line M, etc., to chronologically view the series of events in which a specific industrial product that is likely to have caused a change in the production line M is transported from upstream to downstream along the production line M, and by checking the status of the specific industrial product at the time of the change in the production line M in a list, it is possible to identify with a high degree of probability the industrial product that caused the change in the production line M and the location on the production line M where the change occurred. Therefore, the manager of the production line M, etc., can more easily check the status of the change in the production line M based on the clip video, and by identifying the cause of the change in the production line M and taking action, it is possible to further suppress a decline in productivity.

[0112] (2-3) When the monitoring control device 111 determines from images captured by the production line monitoring cameras C1 to C4 that a change has occurred on the production line M, the multiple videos included in the group playback data may include videos containing images in which it has been determined that a change has occurred on the production line M. When the monitoring control device 111 determines from images captured by the production line monitoring cameras C1 to C4 that a change has occurred on the production line M, the multiple videos included in the group playback data may be videos of different products. When the monitoring control device 111 determines from images captured by the production line monitoring cameras C1 to C4 that a change has occurred on the production line M, the multiple videos included in the group playback data may include a video of an industrial product located furthest downstream in the conveying direction among multiple industrial products grouped under predetermined conditions, and a video of an industrial product located furthest downstream in the conveying direction among multiple industrial products belonging to the same group.

[0113] When the monitoring control device 111 determines that a change has occurred on the production line M from images captured by the production line monitoring cameras C1 to C4, and the multiple videos included in the group playback data include a video containing an image in which it has been determined that a change has occurred on the production line M, the following effect can be achieved. A series of states in which the target industrial product, which is the cause of the change on the production line M and captured in the image in which it has been determined that a change has occurred on the production line M, is transported from upstream to downstream on the production line M can be played back in chronological order along with the video containing the image in which it has been determined that a change has occurred on the production line M. As a result, when a manager of the production line M views a series of states in chronological order in which the target industrial product, which is the cause of the change on the production line M and captured in the image in which it has been determined that a change has occurred on the production line M, is transported from upstream to downstream on the production line M, the manager can check the list in chronological order, including the timing of the change on the production line M, and can identify the industrial product and location that caused the change on the production line M. Therefore, the manager of the production line M can check the state of the change on the production line M in more detail based on the clip video, and by identifying the cause of the change on the production line M and taking action, a decline in productivity can be further suppressed.

[0114] (3) In the above embodiment, the monitoring control device 111 selects which of the simultaneous playback data, the time-series playback data, and the group playback data to output in the output process (S14) based on an instruction input to the monitoring input device 113. However, the monitoring control device 111 may be configured to output only one of the simultaneous playback data, the time-series playback data, and the group playback data in the output process (S14). Also, the monitoring control device 111 may be configured to select which of two of the simultaneous playback data, the time-series playback data, and the group playback data to output in the output process (S14) based on an instruction input to the monitoring input device 113. Also, the monitoring control device 111 may output data in a format different from any of the simultaneous playback data, the time-series playback data, and the group playback data in the output process (S14).

[0115] (4) In the above embodiment, multiple industrial products included in the same lot are grouped together, but the grouping conditions are not limited to this. Several specific examples of grouping conditions will be described below.

[0116] (4-1) For example, the monitoring control device 111 may set the point at which it determines, based on an image captured by the alarm device monitoring camera B, that the three-color light D1 has turned blue (i.e., the point at which it determines that transportation on the production line M has started) as the start point of the group, and the point at which it determines that the three-color light D1 has turned red (i.e., the point at which it determines that a change has occurred on the production line M) as the end point of the group, and may set multiple industrial products transported between the start point and the end point of the group as one group.

[0117] (4-2) For example, the monitoring control device 111 may set the time when it determines that a worker has performed a specified task based on images taken by the production line monitoring cameras C1 to C4 or the alarm device monitoring camera B as the start point of the group, and the time when it determines that the worker has performed the same or a different specified task as the end point of the group, and may set multiple industrial products transported between the start point and the end point of the group as one group.

[0118] (4-3) For example, the monitoring control device 111 may group together multiple industrial products that are transported between predetermined times. The predetermined time may be a time arbitrarily selected by an administrator or the like using the monitoring input device 113.

[0119] (5) In the above embodiment, when the visual notification of a change in the production line M by the alarm device D is the numerical value of the meter D2 or the numerical value on the monitor screen D3 captured by the alarm device monitoring camera B, the monitoring control device 111 performs image analysis of the image of the numerical value of the meter D2 or the image of the numerical value on the monitor screen D3 captured by the alarm device monitoring camera B, and if it is determined that a change has occurred when the numerical value of the meter D2 or the numerical value on the monitor screen D3 is compared with a numerical range indicating a normal state, then the monitoring control device 111 detects a change in the production line M. In this case, if it is determined, as a result of analyzing the image of the numerical value of the meter D2 or the image of the numerical value on the monitor screen D3 captured by the alarm device monitoring camera B, that the photographed numerical value of the meter D2 or the numerical value on the monitor screen D3 falls within a numerical range indicating a change (such as an abnormality or error), then the monitoring control device 111 detects a change in the production line M. In this way, by checking the clip video immediately before the photographed numerical value of the meter D2 or the numerical value on the monitor screen D3 enters the numerical range indicating a change, it is possible to find signs of an upcoming change. By being able to predict changes in the production line M in this way (predict that an abnormality will soon occur), the production line M can be stopped before a change in the production line M is detected (before the value on the meter D2 or the value on the monitor screen D3 enters the numerical range indicating a change), thereby further suppressing a decline in productivity.

[0120] (6) The production line monitoring system 1 of the above embodiment may be operated solely within the factory or company of the production line M (via a local area network (LAN) or a wide area network (WAN)) using a personal computer or the like as the production line monitoring device 11. The production line monitoring system 1 may also be implemented as a cloud service in which the production line monitoring device 11 is connected to an external network and the monitoring storage device 112 or the like is provided on an external server. The production line monitoring system 1 may also include a control device and a storage device for each of the multiple image capturing devices, including the alarm device monitoring camera B and the production line monitoring cameras C1 to C3, and perform some of the functions of the monitoring control device and the monitoring storage device. For example, in the continuous storage process of the production line monitoring system 1, the control device of each of the multiple image capturing devices may store multiple images captured by each of the multiple image capturing devices as time-series data in the storage device of each of the multiple image capturing devices based on a command from the monitoring control device 111. In the event storage process, the production line monitoring system 1 may be configured such that, based on instructions from the monitoring control device 111, the control device possessed by each of the multiple imaging devices extracts multiple groups of images for the change period from multiple time series data associated with each of the multiple imaging devices, which is stored in the memory device of each of the multiple imaging devices, and stores the extracted groups of images in the monitoring memory device 112 as clip footage corresponding to each of the multiple imaging devices.

[0121] Although the embodiments of the present invention have been described above based on the drawings, the specific configurations should not be considered to be limited to these embodiments and examples. The scope of the present invention is indicated not only by the description of the above-mentioned embodiments and examples but also by the claims, and further includes all modifications within the meaning and scope of the claims. This application is based on Japanese Patent Application No. 2024-149919 filed on August 30, 2024, and Japanese Patent Application No. 2024-149923 filed on August 30, 2024, the contents of which are incorporated herein by reference.

[0122] 1: production line monitoring system, 111: monitoring control device (control device), 112: monitoring storage device (storage device), 113: monitoring input device (input device), 114: monitoring display device (display device possessed by production line monitoring system), B: alarm device monitoring camera (alarm device photographing device), C1 to C4: production line monitoring camera (production line photographing device), D: alarm device, D1: three-color light (alarm device), D2: meter (alarm device), D3: monitor screen (alarm device), D4: lamp (alarm device), M: production line, P1 to P4, Q: clip video

Claims

1. A production line monitoring system for monitoring a production line that manufactures a plurality of products, comprising: a plurality of image capturing devices installed on the production line for monitoring the production line; a storage device that stores a plurality of images captured by at least two of the plurality of image capturing devices; and a control device, wherein the control device performs a continuous storage process that constantly acquires a plurality of images captured by at least two of the plurality of image capturing devices and stores the images in the storage device as time-series data associated with each of the at least two of the plurality of image capturing devices; (i) upon determining that information indicating a change in the production line has been acquired, (ii) for at least two of the plurality of image capturing devices, extracting a plurality of image groups for a change period that includes a change time, which is the time when the information indicating the change in the production line was acquired, from time series data associated with each of the at least two image capturing devices, which is stored in the storage device in the continuous storage process, and that includes at least a time period before the change time among time periods after the change time, and (iii) storing the extracted plurality of image groups as a plurality of clip videos associated with each of the at least two image capturing devices.

2. The manufacturing line monitoring system of claim 1, characterized in that the control device executes an output process that outputs video of at least a portion of the change period of at least two clip videos out of the plurality of clip videos as data that can be displayed and played back in a list on the screen of a display device possessed by the manufacturing line monitoring system or a system external to the manufacturing line monitoring system.

3. The manufacturing line monitoring system of claim 2, wherein the control device, in the output processing, extracts images from the same time period from at least two of the plurality of clip images, and outputs the extracted images as simultaneous playback data that can be displayed in a list on the screen of the display device and played back in synchronization at the same time.

4. The manufacturing line monitoring system of claim 2, wherein the plurality of photographing devices include a plurality of production line photographing devices that photograph the manufacturing line and are installed along the manufacturing line, and wherein the control device, in the output processing, extracts images of different time periods in which a specific product being transported on the manufacturing line is photographed from a plurality of clip images corresponding to at least two of the plurality of manufacturing line photographing devices, and outputs the extracted plurality of images as time-series playback data that can be displayed and played back in a list on the screen of the display device.

5. The production line monitoring system of claim 4, characterized in that, in the event storage process, when it is determined that information indicating a change in the production line has been acquired from an image captured by at least one of the plurality of photographing devices, the specific product is the product included in the image in which it is determined that a change in the production line has occurred.

6. The manufacturing line monitoring system of claim 2, wherein the plurality of photographing devices include a plurality of production line photographing devices that photograph the manufacturing line and are installed along the manufacturing line, and wherein the control device, in the output processing, extracts images of a plurality of products that are grouped under predetermined conditions from a plurality of the clip images corresponding to at least two of the plurality of manufacturing line photographing devices, and outputs the extracted plurality of images as group playback data that can be displayed and played back in a list on the screen of the display device.

7. The manufacturing line monitoring system described in claim 6, characterized in that the extracted multiple images include at least an image of the product located at the most downstream side in the conveying direction among the multiple products grouped under the specified conditions, and an image of the product located at the most upstream side in the conveying direction among the multiple products grouped under the specified conditions.

8. The manufacturing line monitoring system of claim 6, wherein, in the event storage process, when it is determined that information indicating a change in the manufacturing line has been acquired from an image captured by at least one of the plurality of photographing devices, the extracted plurality of images includes at least an image containing an image that is determined to indicate that a change has occurred in the manufacturing line.

9. An input device connected to the control device for inputting instructions is further provided, wherein the plurality of image capturing devices include a plurality of production line image capturing devices that capture the production line and are installed along the production line, and the control device, in the output processing, extracts images of specific products transported on the production line captured at different time periods from a plurality of clip images corresponding to at least two of the plurality of production line image capturing devices, and outputs the extracted images as time-series playback data that can be displayed and played back in a list on the screen of the display device, or extracts images of the same time period from a plurality of clip images corresponding to at least two of the plurality of image capturing devices, and outputs the extracted images as simultaneous playback data that can be displayed and played back in a list on the screen of the display device, or extracts images of multiple products grouped under predetermined conditions from a plurality of clip images corresponding to at least two of the plurality of production line image capturing devices, and outputs the extracted images as group playback data that can be displayed and played back in a list on the screen of the display device, 3. The manufacturing line monitoring system according to claim 2, wherein the selection is made based on an instruction input to the input device.

10. A manufacturing line monitoring system as described in any one of claims 1 to 9, characterized in that the control device further performs real-time output processing to output multiple images taken by each of the multiple imaging devices that are constantly acquired by the continuous storage processing as data that can be displayed in a list on the screen of a display device possessed by the manufacturing line monitoring system or a system external to the manufacturing line monitoring system.

11. The manufacturing line monitoring system of any one of claims 1 to 4, 6, 7 and 9, wherein the plurality of photographing devices include an alarm device photographing device that photographs an alarm device that visually notifies of a change in the manufacturing line, and the control device determines in the event storage process that it has acquired information indicating a change in the manufacturing line when it is determined from the image photographed by the alarm device photographing device that the alarm device has visually notified of a change in the manufacturing line, and the at least two photographing devices may or may not include the alarm device photographing device.

12. The production line monitoring system described in claim 11, characterized in that the control device constantly acquires images of the alarm device captured by the alarm device photographing device, determines from the acquired images of the alarm device through image analysis whether the alarm device has visually notified a change in the production line, and if it determines that the alarm device has visually notified a change in the production line, executes the event storage process, assuming that information indicating a change in the production line has been acquired.

Images