Manufacturing line monitoring system

Abstract

By analyzing the state of change in the manufacturing line, the cause of the change is identified, thereby suppressing the decline in productivity, and a highly versatile manufacturing line monitoring system that can be installed on various manufacturing lines is provided. [Solution] The manufacturing line monitoring system 1 installed on the existing manufacturing line M has a monitoring control device 111 that performs continuous storage processing to store multiple images taken by manufacturing line monitoring cameras C1 to C3 as time-series data in the monitoring storage device 112, change detection processing to detect changes in the manufacturing line M from visual notifications of the three-color light D1 taken by the notification device monitoring camera B, and event storage processing to extract video of the change period, including the time of change, from the time-series data stored in the continuous storage processing and store it in the monitoring storage device 112 as clipped video P1 to P3.

Description

Technical Field

[0001] The present invention relates to a manufacturing line monitoring system for monitoring a manufacturing line.

Background Art

[0002] Conventionally, in a manufacturing line system of a factory, when there is any change (including not only an abnormality but also a state different from normal even within the normal range) in the manufactured product and process, state detection means (moving body detection sensors (e.g., optical sensors and ultrasonic sensors), abnormal sound detection sensors, cameras, temperature sensors, etc.) are used to detect changes in the manufacturing line, and the fact that such a change has been detected is visually notified to an administrator or the like by lighting a three-color lamp or a patrol lamp, which is a notification device (see Patent Documents 1 to 2).

[0003] Also, in such a manufacturing line monitoring system, although it is known that a change has occurred in the manufacturing line, there are cases where the state and cause of the change are not known, resulting in delays in the recovery and improvement of the manufacturing line and a decrease in productivity due to a decrease in the operating rate of the manufacturing line.

[0004] In order to solve such a problem of decreasing productivity, conventionally, when a manufacturing line is photographed by a monitoring camera and changes in the manufacturing line are detected using state detection means, a monitoring system for analyzing the state of the change in the manufacturing line by reviewing the video of the monitoring camera to clarify the cause of the change has been disclosed (see Patent Documents 3 to 5).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

[0006] On the other hand, if a new monitoring system is to be installed to analyze changes in a manufacturing line, which involves photographing the existing manufacturing line with cameras or other imaging devices and saving the video footage when changes are detected in the manufacturing line using a state detection means, then the monitoring system needs to be electrically connected to the existing manufacturing line where the state detection means for detecting changes in the manufacturing line is installed. However, depending on the configuration of the manufacturing line system, it may be difficult to electrically connect the monitoring system, resulting in low versatility.

[0007] In this regard, as mentioned above, many existing manufacturing line systems use state detection means to detect changes in the manufacturing line and notify the surrounding area of ​​the detected change through a notification device that visually indicates the change.

[0008] Therefore, the present invention aims to provide a highly versatile manufacturing line monitoring system that can be installed on various manufacturing lines, while also suppressing a decline in productivity by analyzing the state of change in the manufacturing line and clarifying the cause of the change. [Means for solving the problem]

[0009] The present invention is a manufacturing line monitoring system for monitoring a manufacturing line that produces products, The aforementioned manufacturing line is the subject of the photographs, and a plurality of manufacturing line photographing devices are installed along the aforementioned manufacturing line, A notification device photographing device that photographs a notification device that visually notifies changes in the aforementioned manufacturing line, A storage device that stores multiple images taken by the aforementioned multiple manufacturing line imaging devices, Control device and Equipped with, The control device is Continuous storage processing involves continuously acquiring multiple images captured by each of the multiple manufacturing line imaging devices and storing them in the storage device as multiple time-series data associated with each of the multiple manufacturing line imaging devices. The notification device camera continuously acquires images of the notification device, and from the acquired images of the notification device, it is determined by image analysis whether the notification device has visually notified of a change in the manufacturing line. If it is determined that the notification device has visually notified of a change in the manufacturing line, a change detection process is performed to detect the change in the manufacturing line. When the change detection process detects a change in the manufacturing line, the event storage process extracts a set of images from the multiple time-series data associated with each of the multiple manufacturing line cameras stored in the storage device during the continuous storage process. These images include the time of change that is determined to have visually indicated a change in the manufacturing line, and also include at least the time period before the time of change among the time periods before and after the time of change. The extracted sets of images are then stored in the storage device as clip images corresponding to each of the multiple manufacturing line cameras. It is characterized by performing the following.

[0010] According to the above configuration, by installing a notification device imaging device that images notification devices that visually notify changes in the existing manufacturing line, the control device of the manufacturing line monitoring system can detect changes in the manufacturing line by determining, through image analysis, whether or not a change in the manufacturing line has been visually notified from the image captured by the notification device imaging device. In the image analysis, an analysis method using pattern matching that compares with pre-stored images may be applied, or an analysis method using machine learning that is updated sequentially may be applied. Furthermore, the multiple manufacturing line imaging devices are imaging devices that target the manufacturing line in the manufacturing line monitoring system and capture images that become clipped video. It is also possible that imaging devices other than the manufacturing line imaging device are placed on the manufacturing line to photograph the manufacturing line. Furthermore, the notification device imaging device is at least one imaging device, and if there is one notification device on the manufacturing line, the one imaging device that photographs that one notification device is the notification device imaging device, and if there are multiple notification devices on the manufacturing line, the multiple imaging devices that photograph each of the multiple notification devices are the notification device imaging devices. When the control unit of the manufacturing line monitoring system detects a change in the manufacturing line, it extracts video footage from multiple time-series data associated with each of the multiple manufacturing line cameras installed along the manufacturing line, specifically the video of the change period, which includes the time of the change and at least the time before the change, and stores this footage as a clip for each manufacturing line camera. In other words, the control unit of the manufacturing line monitoring system, triggered by the detection of a change in the manufacturing line from the image of the notification device acquired by the notification device imaging device, simultaneously generates clips of the same period, which includes the time of the change and at least the time before the change, for all of the multiple manufacturing line cameras installed along the manufacturing line. This allows manufacturing line managers and others to analyze the state of change in the manufacturing line as captured by the multiple manufacturing line cameras, working backward from the time of the change, based on the stored clips, thereby identifying the cause of the change and suppressing a decline in productivity. Here, the change period includes the time of the change, and at least the time before the change, as well as the time after the change. In other words, the change period may include the time of change and the time period before the change, or it may include the time of change, the time period before the change, and the time period after the change. The time of change may be the time when an image is taken that is judged to have visually notified a change in the manufacturing line, or it may be the time when the control device of the manufacturing line monitoring system determines that it has visually notified a change in the manufacturing line. Furthermore, the manufacturing line monitoring system's manufacturing line imaging device, notification device imaging device, memory device, and control device can be installed independently of the existing manufacturing line without being electrically connected to it. Therefore, it can be installed on a variety of manufacturing lines and is highly versatile.

[0011] Furthermore, the present invention may be characterized in that, in the above-mentioned manufacturing line monitoring system, the visual notification of changes in the manufacturing line by the notification device is at least one of the following: notification by lighting or flashing a lamp, notification by the color of the lamp's illumination, or notification by a numerical value displayed on an instrument or monitor.

[0012] According to the above configuration, the method of visually notifying changes in the manufacturing line through the illumination or flashing of lamps, the color of the lamps, or numerical values ​​on instruments or monitors, places a low processing burden on the control device of the manufacturing line monitoring system through image analysis. Therefore, regardless of the specifications of the control device of the manufacturing line monitoring system, it is possible to accurately detect, through image analysis, whether or not a change in the manufacturing line has been visually notified from the image captured by the notification device's imaging device. This makes it possible to analyze the state of the change in the manufacturing line at an appropriate time when a change occurs, clarify the cause of the change, and further suppress the decline in productivity.

[0013] Furthermore, in the above-mentioned manufacturing line monitoring system, The memory device stores the images captured by the notification device and the imaging device. The control device is In the continuous storage process, multiple images captured by each of the multiple manufacturing line imaging devices and the notification device imaging device are continuously acquired and stored in the storage device as multiple time-series data associated with each of the multiple manufacturing line imaging devices and the notification device imaging device. In the event storage processing, when a change in the manufacturing line is detected by the change detection processing, a plurality of image groups for the change period are extracted from the plurality of time-series data associated with each of the plurality of manufacturing line imaging devices and the notification device imaging devices, which are stored in the storage device in the continuous storage processing, for all of the plurality of manufacturing line imaging devices and the notification device imaging devices, and the extracted plurality of image groups are stored in the storage device as clip images corresponding to each of the plurality of manufacturing line imaging devices and the notification device imaging devices.

[0014] According to the above configuration, when the control device of the manufacturing line monitoring system detects a change in the manufacturing line, it can extract video footage of the period of change from the time-series data associated with at least one manufacturing line camera installed along the manufacturing line, as well as each notification device camera, and store it as a clip video in the storage device for each camera. This allows manufacturing line managers and others to analyze not only the state of change in the manufacturing line, but also the state of change in the notification device, working backward in time from the time of the change, based on the stored clip video. For example, malfunctions and time lags in the notification device can also be analyzed. As a result, the cause of the change can be resolved with high accuracy, and the decline in productivity can be further suppressed.

[0015] Furthermore, in the above-mentioned manufacturing line monitoring system, The control device is The event storage process may further be characterized by outputting data such that, for at least two of the multiple clip videos stored in the event storage process, the video for at least a portion of the change period can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or a system outside the manufacturing line monitoring system and played back.

[0016] According to the above configuration, from the data output by the output processing of the control device of the production line monitoring system, for at least two clip videos among a plurality of clip videos, the videos of at least a part of the change period can be displayed in a list on the screen of the display device. Then, an administrator of the production line or the like can, for at least two clip videos, collectively view the videos of at least a part of the change period, and can immediately check the state of the production line and the products at the time of change and at least the time zone before the change time when there was a change in the production line. As a result, an administrator of the production line or the like can, based on the clip videos, clarify the cause of the change in the production line and take countermeasures, thereby further suppressing the reduction in productivity.

Effect of the Invention

[0017] By analyzing the state of changes in the production line, it is possible to clarify the cause of the changes and suppress the reduction in productivity, and to provide a highly versatile production line monitoring system that can be installed for various production lines.

Brief Description of the Drawings

[0018] [Figure 1] It is an explanatory diagram of a production line system including the production line monitoring system according to the present embodiment and the production line where the production line monitoring system is installed. [Figure 2] ​​​​​​​​​​​​​​​This is an explanatory diagram illustrating a specific example of a notification device that visually notifies users of changes in a manufacturing line. [Modes for carrying out the invention]

[0019] (Embodiment) As shown in Figure 1, the manufacturing line monitoring system 1 according to this embodiment is installed on a manufacturing line M that transports a large number of products (industrial products Y1, etc.) by belt conveyors M1 to M4 according to a processing or assembly procedure.

[0020] (Manufacturing Line M) First, let's explain the manufacturing line M that will be monitored by the manufacturing line monitoring system 1. As shown in Figure 2, the manufacturing line M is included in the manufacturing line system 2. As shown in Figures 1 and 2, the manufacturing line system 2 includes belt conveyors M1 to M4, a state detection sensor (state detection means) A, an alarm device D including a tricolor light D1, a manufacturing line management device 21 that controls the manufacturing line M, a manufacturing line input device 24, and a manufacturing line display device 25. As shown in Figure 2, the manufacturing line management device 21 includes a manufacturing line control device 22 and a manufacturing line storage device 23.

[0021] (Belt conveyor M1~M4) The belt conveyor M1 has a structure in which a conveyor belt is wrapped between a drive pulley connected to a drive motor and a driven pulley, but it may also be a roller conveyor or the like. Belt conveyors M2 to M4 have the same structure as belt conveyor M1. In this embodiment, for the sake of explanation, the layout is as shown in Figure 1, where the four belt conveyors M1 to M4 are arranged in a straight line from upstream to downstream. However, the layout is not limited to a straight line and can be of various shapes. In this embodiment, for the sake of explanation, the manufacturing line M includes only belt conveyors M1 to M4, but other belt conveyors or work equipment may be present before and after belt conveyors M1 to M4. The industrial products Y1 and the like placed on belt conveyor M1 are transported by the belt conveyors M1 to M4 from belt conveyor M1 to belt conveyor M4 (i.e., from upstream to downstream).

[0022] (Status detection sensor A) The state detection sensor A is a sensor that detects the state of multiple industrial products being transported in the manufacturing line M, which includes belt conveyors M1 to M4. The signal from the state detection sensor A is input to the manufacturing line control device 22 of the manufacturing line management device 21, which will be described later. The manufacturing line control device 22 of the manufacturing line management device 21 then detects changes in the manufacturing line M based on the input signal from the state detection sensor A.

[0023] In this embodiment, the state detection sensor A uses an optical sensor, which is an example of a motion detection sensor, and is configured to measure the distance between adjacent industrial products of multiple industrial products being transported on belt conveyors M1 to M4. The distance between adjacent industrial products is between industrial product Y1 and industrial product Y2, or between industrial product Y5 and industrial product Y6, as shown in Figure 1. The signal of the distance between adjacent industrial products detected by the state detection sensor A is input to the manufacturing line control device 22 of the manufacturing line management device 21. When the situation changes from one where a predetermined distance is maintained between adjacent industrial products, such as between industrial product Y1 and industrial product Y2, as shown in Figure 1, to one where the predetermined distance is not maintained between adjacent industrial products, such as between industrial product Y5 and industrial product Y6, for example, if industrial product Y6 collides with industrial product Y5, the manufacturing line control device 22 of the manufacturing line management device 21 detects a change in the manufacturing line M based on the signal from the state detection sensor A.

[0024] Here, the changes in the manufacturing line M detected by the manufacturing line control device 22 of the manufacturing line management device 21 are not limited to changes in distance between adjacent industrial products, but also include changes to situations different from those assumed in advance. For example, changes in the manufacturing line M detected by the manufacturing line control device 22 of the manufacturing line management device 21 may include cases where an industrial product being transported on the manufacturing line M is deviated from a preset position, where the orientation of the transported industrial product is different from the preset orientation (including cases where it has fallen over), where the type of transported industrial product is different from the type of industrial product set in advance, or where an unusual noise occurs. Furthermore, state detection sensor A is not limited to motion detection sensors such as optical sensors; depending on the object being detected, state detection means such as abnormal sound detection sensors or temperature sensors can also be employed.

[0025] (Notification device D) Notification device D is a device that visually notifies of changes in the manufacturing line M. Notification device D is controlled by the manufacturing line control device 22 of the manufacturing line management device 21, and when the manufacturing line control device 22 detects a change in the manufacturing line M, it visually notifies of the change in the manufacturing line M. In this embodiment, the tricolor lamp D1 is described as an example of the notification device D. However, the notification device D is not limited to the tricolor lamp D1, but may also be a meter D2 that displays a numerical value (see Figure 7(A)), a monitor screen D3 that displays a numerical value (see Figure 7(B)), a lamp D4 that can be lit or flashed (see Figure 7(C)), etc. In other words, the visual notification of changes in the manufacturing line M by the notification device D is not limited to notification by the color of the lighting of the tricolor lamp D1 (which is also notification by the lighting of the red lamp of the tricolor lamp D1), but may also be notification by the lighting or flashing of the lamp D4, or notification by a numerical value displayed on the meter D2 or monitor screen D3.

[0026] The three-color light D1 in this embodiment is a warning light having a red lamp, a yellow lamp, and a blue lamp from top to bottom. It has the function of notifying changes in the manufacturing line M and those around the manufacturing line M (managers, users, etc.) based on whether or not it is lit and the color of the light. For example, if a change in the manufacturing line M is detected using the state detection sensor A, the red lamp of the three-color light D1 will light up, notifying those around the manufacturing line M of the change in the manufacturing line M. Note that the lighting of the blue lamp on the three-color light D1 is not a notification of a change in the manufacturing line M, but rather a notification that the manufacturing line M is operating normally, and the lighting of the yellow lamp on the three-color light D1 is a notification that a worker is working on the manufacturing line M.

[0027] (Manufacturing line control device 21) As described above, the manufacturing line management device 21 includes a manufacturing line control device 22 and a manufacturing line storage device 23. The manufacturing line management device 21 is also connected to the belt conveyors M1 to M4, the status detection sensor A, the notification device D (three-color light D1), the manufacturing line input device 24, and the manufacturing line display device 25 by wiring or communication.

[0028] The manufacturing line control device 22 performs various computer controls on the manufacturing line M and is composed of, for example, a CPU. For example, the manufacturing line control device 22 is connected to the drive motors (not shown) of the belt conveyors M1 to M4 of the manufacturing line M and controls their drive. The manufacturing line control device 22 is also connected to the state detection sensor A and detects changes in the manufacturing line M based on the signal from the state detection sensor A. Specifically, the manufacturing line control device 22 determines whether or not there is a change in the manufacturing line M based on the signal from the state detection sensor A. Furthermore, if the manufacturing line control device 22 detects a change in the manufacturing line M, it will illuminate the three-color lamp D1 in red. Furthermore, if the manufacturing line control device 22 detects a change in the manufacturing line M, it stops the drive motors of the belt conveyors M1 to M4, thereby stopping the conveyance on the belt conveyors M1 to M4.

[0029] The manufacturing line memory device 23 stores programs and other information that perform various computer controls on the manufacturing line M. The manufacturing line input device 24 is a device used by managers of the manufacturing line M to input various information and instructions, such as a keyboard or mouse. The manufacturing line display device 25 is a device that displays operation screens, output screens, etc., based on commands from the manufacturing line control device 22, and is, for example, a display.

[0030] (Manufacturing line control processing) Next, the manufacturing line control process performed in manufacturing line M will be explained based on the flowchart in Figure 3.

[0031] (Start of transport process (S1)) First, the manager or other personnel activates the manufacturing line management device 21. When the manufacturing line input device 24 receives input for starting the manufacturing line M, the manufacturing line control device 22 starts the conveying process (S1). Specifically, in the conveying process, the manufacturing line control device 22 starts the belt conveyors M1 to M4 by driving the respective drive motors of the belt conveyors M1 to M4. As a result, industrial products X1-X10, Y1-Y10, Z1-Z10, etc., placed on belt conveyor M1 are transported from belt conveyor M1 to belt conveyor M4, that is, from upstream to downstream. Note that only industrial products X9-X10, Y1-Y10, and Z1-Z2 are shown in Figure 1.

[0032] (Detection of changes in manufacturing line M (S2)) Next, the manufacturing line control device 22 detects a change in the manufacturing line M based on the signal from the state detection sensor A (S2).

[0033] In this embodiment, changes in the manufacturing line M are detected from the distance between adjacent industrial products, which is detected by a state detection sensor A, which is a motion detection sensor (optical sensor), for multiple industrial products being transported on the belt conveyors M1 to M4 of the manufacturing line M.

[0034] Then, if the manufacturing line control device 22 determines that a predetermined distance is maintained between adjacent industrial products in the manufacturing line M, it determines that no change is detected in the manufacturing line M (S2: NO) and repeats the process in S2. On the other hand, if the manufacturing line control device 22 determines that a predetermined distance is not maintained between adjacent industrial products in the manufacturing line M, a change is detected in the manufacturing line M (S2: YES).

[0035] (Transportation stop process (S3)) When the manufacturing line control device 22 detects a change in the manufacturing line M (S2:YES), the manufacturing line control device 22 performs a transport stop process (S3). In this transport stop process, the manufacturing line control device 22 stops the drive motors of each of the belt conveyors M1 to M4, thereby stopping the drive of the belt conveyors M1 to M4. As a result, the transport of industrial products on the manufacturing line M is stopped.

[0036] (Notification processing (S4)) When the manufacturing line control device 22 executes the transport stop process (S3), it performs an alert process (S4). In this alert process, the manufacturing line control device 22 illuminates the three-color lamp D1 in red.

[0037] Furthermore, when the manufacturing line control device 22 detects a change in the manufacturing line M (S2:YES), it may perform the transport stop process (S3) and the notification process (S4) at the same time, or it may perform the notification process (S4) first and then the transport stop process (S3). In addition, the notification process (S4) may not only illuminate the three-color lamp D1 in red, but may also notify the surrounding area of ​​the change in the manufacturing line M with an alarm sound, or it may send an email to the manager or other person notifying them of the change in the manufacturing line M.

[0038] (Decision to resume (S5)) Next, the manufacturing line control device 22 determines whether or not the manufacturing line M has been restarted by the manufacturing line input device 24 (S5). For example, by analyzing the state of change in the manufacturing line M and identifying its cause, managers can resolve the change in the manufacturing line M. If the manufacturing line M has been restarted by the manufacturing line input device 24 (S5: YES), the manufacturing line control device 22 turns off the red light of the three-color lamp D1, returns to S1, and resumes the transport process. On the other hand, if the manufacturing line M is not restarted, such as by performing a termination operation on the manufacturing line management device 21 using the manufacturing line input device 24 (S5:NO), the manufacturing line control process is terminated.

[0039] (Manufacturing line monitoring system 1) Next, we will describe the manufacturing line monitoring system 1 that monitors the manufacturing line M. As shown in Figures 1 and 2, the manufacturing line monitoring system 1 includes manufacturing line monitoring cameras C1 to C3 (manufacturing line imaging devices), notification device monitoring camera B (notification device imaging device), manufacturing line monitoring device 11, monitoring input device 113, and monitoring display device 114. The manufacturing line monitoring system 1 has the function of monitoring (displaying on screen, recording) the industrial products manufactured by manufacturing line M and manufacturing line M itself using manufacturing line monitoring cameras C1 to C3, etc.

[0040] Here, as shown in Figure 2, the manufacturing line monitoring system 1 is not electrically connected to the manufacturing line system 2 which has the manufacturing line M, and is an electrically independent system from the manufacturing line system 2 which has the manufacturing line M.

[0041] (Manufacturing line surveillance cameras C1~C3) The manufacturing line monitoring cameras C1 to C3 are capable of capturing images, targeting the manufacturing line M, and are installed along the manufacturing line M. The manufacturing line monitoring cameras C1 to C3 are installed at equal intervals along the belt conveyors M1 to M4M, and are installed and configured to capture predetermined shooting ranges of the manufacturing line M. In this embodiment, as shown in Figure 1, the manufacturing line monitoring camera C1 includes the conveying area downstream of belt conveyor M1 and the conveying area upstream of belt conveyor M2 in its shooting range. The manufacturing line monitoring camera C2 includes the conveying area downstream of belt conveyor M2 and the conveying area upstream of belt conveyor M3 in its shooting range. The manufacturing line monitoring camera C3 includes the conveying area downstream of belt conveyor M3 and the conveying area upstream of belt conveyor M4 in its shooting range.

[0042] Furthermore, while it is sufficient for multiple production line monitoring cameras to be installed and configured to include at least a portion of production line M in their shooting range, it is preferable that they be installed along production line M so that all of the belt conveyors of production line M are included in their shooting range. Furthermore, the manufacturing line monitoring camera may be configured to allow its shooting range to be arbitrarily adjusted within a predetermined range by zooming (enlarging or reducing).

[0043] (Surveillance camera B) The notification device monitoring camera B is a camera capable of capturing images and is configured to capture images of the notification device D. In this embodiment, the notification device monitoring camera B is installed in a position that includes the tricolor light D1 in its shooting range in order to capture images of the tricolor light D1. In other words, the notification device monitoring camera B corresponds to a notification device imaging device that photographs the notification device D (three-color light D1), which visually notifies the detection of changes in the manufacturing line M.

[0044] The configuration of the notification device monitoring camera B is not particularly limited, and a camera with the same configuration as the manufacturing line monitoring cameras C1 to C3 may be used. Furthermore, at least some of the manufacturing line monitoring cameras C1 to C3 may also serve as the notification device monitoring camera B. Furthermore, the notification device surveillance camera B may be configured to allow its shooting range to be arbitrarily adjusted within a predetermined range by zooming (enlarging or reducing). Furthermore, if the manufacturing line M is equipped with other notification devices D besides the tricolor light D1, such as a meter D2 that displays numerical values ​​(see Figure 7(A)), a monitor screen D3 that displays numerical values ​​(see Figure 7(B)), and a lamp D4 that can be lit or flashed (see Figure 7(C)), the notification device monitoring camera B may be installed in a position that includes a selected notification device D from among these notification devices D in its shooting range, or it may be installed so that all notification devices D (tricolor light D1, meter D2, monitor screen D3, and lamp D4) are included in its shooting range. In other words, if there are multiple notification devices D, one notification device monitoring camera B may be configured to photograph one notification device D, or it may be configured to photograph multiple notification devices D. Furthermore, in this embodiment, there is one notification device monitoring camera B, but if there are multiple notification devices D, multiple notification device monitoring cameras B may be installed, and multiple notification device monitoring cameras B may be configured to each photograph multiple notification devices D.

[0045] (Manufacturing line monitoring device 11) As shown in Figure 2, the manufacturing line monitoring device 11 comprises a monitoring control device 111, a monitoring storage device 112, and a communication unit 115. The communication unit 115 of the manufacturing line monitoring device 11 is connected to the manufacturing line monitoring cameras C1 to C3 and the notification device monitoring camera B by wired or wireless communication. The monitoring control device 111 is connected to the monitoring input device 113 and the monitoring display device 114.

[0046] The monitoring and control device 111 performs various computer controls on the manufacturing line monitoring system 1 and is composed of, for example, a CPU.

[0047] For example, the monitoring control device 111 processes the images captured by the manufacturing line monitoring cameras C1 to C3 to be sequentially (continuously) stored in the memory areas of the monitoring storage device 112 corresponding to each of the manufacturing line monitoring cameras C1 to C3 (details will be described later).

[0048] Furthermore, the monitoring and control device 111 determines the color of the three-color light D1, which is the target of the notification device monitoring camera B, by performing image analysis (pattern matching that compares the image with a pre-stored image of the three-color light D1 lit in red, or by continuously updated machine learning, etc.). Then, for example, if the monitoring and control device 111 determines that the three-color light D1 is lit in red, that is, if it determines that the notification device D has visually notified of a change in the manufacturing line M, it detects the change in the manufacturing line M.

[0049] The monitoring storage device 112 stores programs and other information used for various computer controls in the manufacturing line monitoring system 1. The monitoring storage device 112 also has a rewritable storage area for necessary information. For example, the images captured by the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C3 are stored in their respective storage areas.

[0050] The monitoring input device 113 is a device used by managers of the manufacturing line M to input various information and instructions, such as a keyboard or mouse. The monitoring and display device 114 is a device for displaying operation screens, output screens, etc., based on commands from the monitoring and control device 111, and is, for example, a display.

[0051] (Manufacturing line monitoring process) Next, the manufacturing line monitoring process performed in the manufacturing line monitoring system 1 will be explained based on the flowchart in Figure 4 and the explanatory diagrams in Figures 5 and 6.

[0052] First, the administrator or other personnel activate the manufacturing line monitoring device 11. The monitoring input device 113 inputs the start operation for the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C3. In this case, for example, four frames (a screen divided into four sections) are displayed in software so that they can be viewed as a list on one screen of the monitoring display device 114, and the video of the tricolor light D1 captured by the alarm device monitoring camera B and the videos captured by the manufacturing line monitoring cameras C1 to C3 are displayed in real time in these four frames. Note that the configuration in which one screen of the monitoring display device 114 is divided into four sections and displays the video of the tricolor light D1 captured by the alarm device monitoring camera B and the videos captured by the manufacturing line monitoring cameras C1 to C3 as a list on the screen of the monitoring display device 114 is just one example and is not limited to this. Alternatively, as a configuration in which the video can be viewed as a list on the screen of the monitoring display device 114, one screen of the monitoring display device 114 may be divided into multiple frames, and the required number of videos from the video of the tricolor light D1 captured by the alarm device monitoring camera B and the videos captured by the manufacturing line monitoring cameras C1 to C3 may be displayed.

[0053] (Start of continuous memory processing (S11)) When a start operation is input, the monitoring control device 111 starts continuous storage processing (S11). In this continuous storage processing, the monitoring control device 111 sequentially (continuously) acquires multiple images taken by each of the manufacturing line monitoring cameras C1 to C3 and stores them as multiple time-series data associated with each of the manufacturing line monitoring cameras C1 to C3 in the storage area of ​​the monitoring storage device 112 corresponding to each of the manufacturing line monitoring cameras C1 to C3. For example, time-series data from manufacturing line monitoring camera C1 is stored in folder 112C1 (not shown) of the monitoring storage device 112, time-series data from manufacturing line monitoring camera C2 is stored in folder 112C2 (not shown) of the monitoring storage device 112, and time-series data from manufacturing line monitoring camera C3 is stored in folder 112C3 (not shown) of the monitoring storage device 112. Furthermore, the time-series data captured by the manufacturing line monitoring cameras C1 to C3, stored in each folder 112C1 to 112C3, will be overwritten after a predetermined number of days to prevent exceeding the storage capacity of the monitoring storage device 112. The monitoring control device 111 continuously acquires images captured by the notification device monitoring camera B. In continuous storage processing, the monitoring control device 111 may store the images captured by the notification device monitoring camera B, which it has acquired continuously, as time-series data associated with the notification device monitoring camera B in the storage area of ​​the monitoring storage device 112 corresponding to the notification device monitoring camera B, or it may not store them in the monitoring storage device 112.

[0054] (Detection of changes in manufacturing line M (S12)) Next, the monitoring and control device 111 performs a change detection process to detect changes in the manufacturing line M, based on its determination of whether or not it has visually notified the changes in the manufacturing line M (S12). First, the system continuously acquires images of the notification device D from the notification device monitoring camera B, and determines from the images of the notification device D whether or not the notification device D has visually notified of a change in the manufacturing line M. In this embodiment, the monitoring control device 111 determines from the images of the tricolor light D1 that the notification device monitoring camera B continuously acquires whether or not the tricolor light D1 is lit red.

[0055] Specifically, the monitoring control device 111 determines whether or not the tricolor light D1, which is the target of the notification device monitoring camera B, is lit red by performing image analysis on the image of the tricolor light D1 captured by the notification device monitoring camera B (S12).

[0056] Then, the monitoring and control device 111 detects a change in the manufacturing line M when it determines that the notification device D has visually notified it of a change in the manufacturing line M. In this embodiment, in the notification process (S4) described above, if the tricolor light D1 is lit red, the monitoring and control device 111 performs image analysis on the image of the tricolor light D1 captured by the notification device monitoring camera B, and determines that the tricolor light D1, which is the target of the notification device monitoring camera B, is lit red, and detects a change in the manufacturing line M (S12:YES). In other words, in process S12, a change detection process is performed to detect changes in the manufacturing line M by performing image analysis on the visual notification image of the three-color light D1 captured by the notification device monitoring camera B.

[0057] Furthermore, if the notification device D captured by the notification device monitoring camera B is not the three-color light D1, the change detection process will be executed as follows. When the notification device D photographed by the notification device monitoring camera B is lamp D4 (see Figure 7(C)), if the "status" lamp of lamp D4 lights up, the monitoring control device 111 analyzes the image of lamp D4 taken by the notification device monitoring camera B to determine that the "status" lamp of lamp D4, which is the target of the notification device monitoring camera B, is lit, and detects a change in the manufacturing line M. Furthermore, if the notification device D captured by the notification device monitoring camera B is a meter D2 or a monitor screen D3, the system performs image analysis (comparing the image of the numerical value of meter D2 or the image of the numerical value of monitor screen D3 captured by the notification device monitoring camera B with a numerical range indicating normality and a numerical range indicating a change (error range)). If it is determined that there has been a change in the numerical value of meter D2 or monitor screen D3 compared to the numerical range indicating normality, the system detects a change in the manufacturing line M.

[0058] (Event memory processing (S13)) Next, when the monitoring and control device 111 detects a change in the manufacturing line M (S12:YES), it performs event storage processing (S13).

[0059] In this event memory processing, the monitoring control device 111, for all of the multiple manufacturing line monitoring cameras C1 to C3, extracts multiple image groups from the multiple time-series data associated with each of the manufacturing line monitoring cameras C1 to C3, which are stored in folders 112C1 to 112C3 (not shown) of the monitoring storage device 112, during the change detection processing (S12:YES). These groups include the change period (for example, 1 minute of video including the change and the time before and after the change) in which the notification device D determined that a change in the manufacturing line M had been visually notified. These groups are then stored as clip videos P1 to P3, corresponding to each of the manufacturing line monitoring cameras C1 to C3, in predetermined folders of the monitoring storage device 112. In other words, in the event memory processing (S13), the detection of a change in the manufacturing line M from the image of the three-color light D1 acquired by the notification device monitoring camera B is used as a trigger, and clip videos P1 to P3 of the same period, which is the change period including the time of the change and at least the time before the change, are simultaneously generated for all of the multiple manufacturing line monitoring cameras C1 to C3 installed along the manufacturing line M. In this embodiment, the change period includes the time of the change (when a change in the manufacturing line M is detected) when it is determined from image analysis that the three-color light D1 is lit red, and includes the time before and after the change. Note that the change period only needs to include the time of the change and at least the time before the change, and does not need to include the time of the change and the time before the change but not the time after the change.

[0060] For example, clip video P1 extracted from time-series data associated with manufacturing line monitoring camera C1 is stored in folder 112P1 (not shown) of the monitoring storage device 112, clip video P2 extracted from time-series data associated with manufacturing line monitoring camera C2 is stored in folder 112P2 (not shown) of the monitoring storage device 112, and clip video P3 extracted from time-series data associated with manufacturing line monitoring camera C3 is stored in folder 112P3 (not shown) of the monitoring storage device 112. Note that video clips P1-P3 are saved in folders 112P1-112P3, which are different from folders 112C1-112C3, and therefore will not be overwritten even after a specified number of days have passed. Furthermore, if time-series data associated with the notification device surveillance camera B is stored in the memory area corresponding to the notification device surveillance camera B during continuous memory processing, the monitoring control device 111 may, during event memory processing, extract multiple image groups for the change period from this time-series data and store them as clip video Q corresponding to the notification device surveillance camera B in a predetermined folder of the monitoring storage device 112.

[0061] On the other hand, if the monitoring and control device 111 does not detect any changes in the manufacturing line M (S12: NO), the process in S12 is repeated, and the determination of whether or not the three-color light D1 is lit red is continued based on the video footage captured by the notification device monitoring camera B.

[0062] (Output processing (S14)) Next, after the event storage processing in S13, the monitoring control device 111 performs output processing (S14). In the output processing, at least a portion of the video clips (at least two of the video clips P1 to P3) corresponding to at least two of the manufacturing line monitoring cameras C1 to C3 are output as data that can be displayed in a list on the screen of a display device and played back. At least a portion of the video clips refers to video from at least a portion of the change period of the video clips. This display device is, for example, the monitoring display device 114 of the manufacturing line monitoring system 1. The display device may also be a display device of a system outside of the manufacturing line monitoring system 1. Note that the display device of a system outside of the manufacturing line monitoring system 1 may or may not include the manufacturing line display device 25 of the manufacturing line M. Note that the output processing (S14) may be performed immediately after the event storage processing (S13), or it may be performed after the event storage processing (S13) at an appropriate time when the manager of the manufacturing line M can respond. In the output processing of this embodiment, immediately after the event storage processing in S13, the monitoring control device 111 outputs to the monitoring display device 114 data such that at least a portion of the video clips P1 to P3 corresponding to the manufacturing line monitoring cameras C1 to C3 stored in the monitoring storage device 112 in the processing of S13 can be displayed in a list on the screen of the monitoring display device 114 of the manufacturing line monitoring system 1 and played back. Furthermore, in event memory processing, if multiple image groups representing the change period are extracted from the time-series data associated with the notification device monitoring camera B and stored in a predetermined folder of the monitoring storage device 112 as clip video Q corresponding to the notification device monitoring camera B, the monitoring control device 111 outputs at least some of the video from the multiple clip videos (clip video Q and at least two clip videos from clip videos P1 to P3) corresponding to the notification device monitoring camera B and at least two of the monitoring cameras C1 to C3 of the manufacturing line as data that can be displayed and played back in a list on the display device screen.

[0063] In this embodiment, for example, the data output to the monitoring display device 114 is displayed in a list on the screen of the monitoring display device 114 of the manufacturing line monitoring system 1 and played back (see Figure 6). Figure 6 is an example in which the video for a portion of the change period of clip video P1 to P3 is output as data that can be displayed in a list on the screen of the monitoring display device 114 of the manufacturing line monitoring system 1 and played back. Furthermore, in the example of Figure 6, the data output in the output process that can be displayed in a list on the display device of the manufacturing line monitoring system 1 and played back also includes at least a portion of the video of clip video Q corresponding to the notification device monitoring camera B. Specifically, in the example of Figure 6, four frames (a screen divided into four sections) are displayed in software so that they can be displayed in a list on one screen of the monitoring display device 114, and the clip video Q corresponding to the notification device monitoring camera B and the clip videos P1 to P3 corresponding to the manufacturing line monitoring cameras C1 to C3 are displayed in these four frames. Figure 5 is a diagram illustrating the data output in the example of Figure 6. As shown in Figure 5, the transition period for clip videos P1 to P3 and clip video Q is 1 minute including the transition time, with clip video P1 divided into 15 segments (4-second units) as unit videos P1 / T1, P1 / T2, P1 / T3...P1 / T15, clip video P2 divided into 15 segments (4-second units) as unit videos P2 / T1, P2 / T2, P2 / T3...P2 / T15, and clip video P3 divided into 15 segments The unit video P3 / T1, P3 / T2, P3 / T3...P3 / T15 are divided (in 4-second units), and the unit video Q / T1, Q / T2, Q / T3...Q / T15 are divided into 15 segments (in 4-second units), with unit videos P1 / T5~T9, P2 / T5~T9, P3 / T5~T9, and Q / T5~T9 being output, including the unit video P1 / T8, P2 / T8, P3 / T8 at the time of change. In this embodiment, one screen of the monitoring display device 114 is divided into four frames, and the clip video Q corresponding to the notification device monitoring camera B and the clip videos P1~P3 corresponding to the manufacturing line monitoring cameras C1~C3 are displayed, but it is not limited to this.In the manner in which the images are displayed in a list on the screen of the monitoring display device 114, one screen of the monitoring display device 114 may be divided into multiple frames, and the necessary number of clip images from the clip images Q corresponding to the notification device monitoring camera B and the clip images P1 to P3 corresponding to the manufacturing line monitoring cameras C1 to C3 may be displayed. In this embodiment, the change period of the clip images is set to 60 seconds, but it is not limited to this and can be set arbitrarily. In this embodiment, the unit image of the clip images is divided into 15 segments (4-second units), but it is not limited to this and can be set arbitrarily. Figures 5 and 6 show an example where some of the clip videos P1 to P3 included in the data output by the output process, which can be displayed and played back in a list on the screen of the monitoring display device 114 of the manufacturing line monitoring system 1, are of the same period, but this is not limited to this example. For example, some of the clip videos P1 to P3 included in the data output by the output process, which can be displayed and played back in a list on the display device of the manufacturing line monitoring system 1, may be of different periods. If some of the clip videos P1 to P3 included in the data output by the output process, which can be displayed and played back in a list on the display device of the manufacturing line monitoring system 1, are of different periods, there may or may not be overlapping periods among those different periods. Furthermore, the data output by the output process may include images taken by the notification device monitoring camera B instead of at least some of the clip videos Q corresponding to the notification device monitoring camera B. The images taken by the notification device monitoring camera B may be images taken at the time of change, or real-time images. The data output during the output process does not necessarily have to include at least a portion of the video clip Q or the images captured by the notification device surveillance camera B.

[0064] (Decision to resume (S15)) Next, the monitoring control device 111 determines whether or not the manufacturing line monitoring system 1 has been restarted by the monitoring input device 113 (S15). For example, in manufacturing line M, if, after detecting a change in manufacturing line M, the cause is resolved, and the manager of manufacturing line M or the like confirms safety, the manufacturing line monitoring system 1 has been restarted by the monitoring input device 113 (S15: YES), the process returns to S12. On the other hand, if no restart operation is performed, such as ending the manufacturing line monitoring system 1 using the monitoring input device 113 (S15:NO), the manufacturing line monitoring process is terminated. In the manufacturing line monitoring process performed by the monitoring control device 111 of this embodiment, the continuous storage process is configured not to stop regardless of whether or not a change is detected in the manufacturing line M by the change detection process (S12). However, in the manufacturing line monitoring process performed by the monitoring control device 111, the continuous storage process may be configured to stop when a change is detected in the manufacturing line M by the change detection process (S12) (S12:YES), and to resume upon a restart decision (S15:YES).

[0065] According to the above-described manufacturing line monitoring system 1, by installing a notification device monitoring camera B that photographs a notification device D that visually notifies changes in the existing manufacturing line M, the monitoring control device 111 can detect changes in the manufacturing line M by determining, through image analysis, whether or not a change in the manufacturing line M has been visually notified from the image captured by the notification device monitoring camera B. Then, when the monitoring control device 111 detects a change in the manufacturing line M from the image captured by the notification device monitoring camera B, it extracts video footage of the change in the manufacturing line M, including the period before and after the change, from the time-series data associated with each of the manufacturing line monitoring cameras C1 to C3 that include the manufacturing line M in their shooting range, and stores it as clipped video P1 to P3 in folders 112P1 to 112P3 (not shown) of the monitoring storage device 112. This allows managers of the manufacturing line M to easily check the state of the manufacturing line M and industrial products Y1-Y10, etc., at the time of the detected change and before and after by viewing the stored clip videos P1-P3. In other words, based on the stored clip videos P1-P3, the state of change in the manufacturing line M can be analyzed retrospectively from the time of the change, allowing for the identification of the cause of the change and the suppression of productivity decline. Furthermore, the existing manufacturing line M, which is equipped with an alerting device D that visually notifies changes in the manufacturing line M, and the newly installed manufacturing line monitoring system 1, which has multiple manufacturing line monitoring cameras C1 to C3 and an alerting device monitoring camera B, can be installed independently of the existing manufacturing line M without electrical connection. Therefore, it can be installed on various manufacturing lines and is highly versatile.

[0066] According to the above-described manufacturing line monitoring system 1, the method of visually notifying changes in the manufacturing line M through the lighting and color of the three-color light D1, the values ​​of the meter D2, the values ​​on the monitor screen D3, and the lighting and flashing of the lamp D4 has a low processing burden on image analysis. Therefore, regardless of the specifications of the monitoring control device 111, it is possible to accurately detect, through image analysis, whether or not changes in the manufacturing line M have been visually notified from the images captured by the notification device monitoring camera B.

[0067] According to the above-described manufacturing line monitoring system 1, at least a portion of the clipped video footage P1 to P3 can be displayed in a list on the display device screen from the data output by the output processing of the monitoring display device 114. Then, the manager of the manufacturing line M can view at least a portion of the clipped video footage P1 to P3 together to quickly check the state of the manufacturing line M and industrial products Y1 to Y10, etc., at the time of change and at least the time before the change. As a result, the manager of the manufacturing line M can identify the cause of the change in the manufacturing line M based on the clipped video footage P1 to P3 and take appropriate action, thereby further suppressing the decline in productivity.

[0068] (Other embodiments) (1) In the above embodiment, when the visual notification of a change in the manufacturing line M by the notification device D is the numerical value of meter D2 captured by the notification device monitoring camera B or the numerical value on the monitor screen D3, the monitoring control device 111 performs image analysis on the image of the numerical value of meter D2 captured by the notification device monitoring camera B or the image of the numerical value on the monitor screen D3. If it determines that there has been a change when comparing the numerical value of meter D2 or the numerical value on the monitor screen D3 with the range indicating normal, it detects a change in the manufacturing line M. In this case, if the system analyzes the images of the meter D2 reading and the monitor screen D3 reading captured by the notification device monitoring camera B, and determines that the captured readings of meter D2 and monitor screen D3 have entered a range indicating a change (abnormality, error, etc.), then a change in the manufacturing line M is detected. This allows the system to identify signs of an impending change by reviewing the video clips taken just before the captured readings of meter D2 and monitor screen D3 enter the range indicating a change. By predicting changes in the manufacturing line M in this way (forecasting that an abnormality will occur soon), the manufacturing line M can be stopped before a change is detected (before the readings of meter D2 and monitor screen D3 enter the range indicating a change), thereby further suppressing the decline in productivity.

[0069] (2) The manufacturing line monitoring system 1 of the above embodiment may be operated only within the factory or company (LAN (Local Area Network) or WAN (Wide Area Network)) of the manufacturing line M, using a personal computer or the like in the manufacturing line monitoring device 11. Furthermore, the manufacturing line monitoring system 1 can also be implemented as a cloud service by connecting the manufacturing line monitoring device 11 to an external network and installing the monitoring storage device 112, etc., on an external server. Furthermore, the manufacturing line monitoring system 1 may have a control device and a storage device for each of the multiple imaging devices, including the notification device monitoring camera B and the manufacturing line monitoring cameras C1 to C3, and may perform some functions of the monitoring control device and some functions of the monitoring storage device. For example, in continuous storage processing, the manufacturing line monitoring system 1 may, based on a command from the monitoring control device 111, have the control device of each of the multiple imaging devices store multiple images captured by each of the multiple imaging devices as time-series data in the storage device of each of the multiple imaging devices. Then, in event storage processing, the manufacturing line monitoring system 1 may, based on a command from the monitoring control device 111, have the control device of each of the multiple imaging devices extract multiple image groups for the change period from multiple time-series data associated with each of the multiple imaging devices stored in the storage device of each of the multiple imaging devices, and store them in the monitoring storage device 112 as clip images corresponding to each of the multiple imaging devices.

[0070] Although embodiments of the present invention have been described above with reference to the drawings, it should be understood that the specific configuration is not limited to these embodiments and examples. Furthermore, the scope of the present invention is indicated not only by the above-described embodiments and examples but also by the claims, and includes all modifications within the meaning and scope of equivalence to the claims. [Explanation of Symbols]

[0071] 1: Manufacturing line monitoring system, 111: Monitoring and control device (control device), 112: Monitoring and storage device (storage device), B: Notification device monitoring camera (notification device imaging device), C1~C3: Manufacturing line monitoring camera (manufacturing line imaging device), D: Notification device, D1: Three-color light (notification device), D2: Meter (notification device), D3: Monitor screen (notification device), D4: Lamp (notification device), M: Manufacturing line, P1~P3, Q: Clipped video

Claims

1. A manufacturing line monitoring system that monitors the manufacturing line for producing products, The aforementioned manufacturing line is the subject of the filming, and multiple manufacturing line surveillance cameras are installed along the aforementioned manufacturing line, A notification device monitoring camera that photographs a notification device that visually notifies changes in the aforementioned manufacturing line, A storage device that stores multiple images taken by the aforementioned multiple manufacturing line monitoring cameras, A control device is provided, The control device is Continuous storage processing involves continuously acquiring multiple images captured by each of the multiple manufacturing line monitoring cameras and storing them in the storage device as multiple time-series data associated with each of the multiple manufacturing line monitoring cameras. The system continuously acquires images of the notification device captured by the notification device monitoring camera, and uses image analysis to determine whether the notification device visually notified of a change in the manufacturing line from the acquired images of the notification device. If it is determined that the notification device visually notified of a change in the manufacturing line, the system detects the change in the manufacturing line through a change detection process. A manufacturing line monitoring system characterized by the following: when the change detection process detects a change in the manufacturing line, the system performs an event storage process, for all of the multiple manufacturing line monitoring cameras, extracting from the multiple time-series data associated with each of the multiple manufacturing line monitoring cameras stored in the storage device in the continuous storage process, a set of images that includes the time of change that is determined to have visually notified the change in the manufacturing line, and for the change period that includes at least the time before the time of change among the time periods before and after the time of change, and storing the extracted set of images as clip images corresponding to each of the multiple manufacturing line monitoring cameras in the storage device.

2. The manufacturing line monitoring system according to claim 1, characterized in that the visual notification of changes in the manufacturing line by the notification device is at least one of the following: notification by lighting or flashing a lamp, notification by the color of the lamp's illumination, or notification by a numerical value displayed on an instrument or monitor.

3. The aforementioned storage device stores images captured by the notification device monitoring camera. The control device is In the continuous storage process described above, multiple images captured by each of the multiple manufacturing line monitoring cameras and the notification device monitoring cameras are continuously acquired and stored in the storage device as time-series data associated with each of the multiple manufacturing line monitoring cameras and the notification device monitoring cameras. The manufacturing line monitoring system according to claim 1, characterized in that, in the event storage processing, when a change in the manufacturing line is detected by the change detection processing, a plurality of image groups for the change period are extracted from the time-series data associated with each of the plurality of manufacturing line monitoring cameras and the notification device monitoring cameras stored in the storage device in the continuous storage processing, and the extracted plurality of image groups are stored in the storage device as clip images corresponding to each of the plurality of manufacturing line monitoring cameras and the notification device monitoring cameras.

4. The control device is A manufacturing line monitoring system according to any one of claims 1 to 3, further characterized in that, for at least two of the multiple clip videos stored in the event storage processing, the video for at least a portion of the change period is output as data that can be displayed in a list on the screen of a display device owned by the manufacturing line monitoring system or owned by a system outside the manufacturing line monitoring system and played back.

Images