Visual sense monitoring system and visual sense monitoring method

Abstract

To eliminate the need for connection coordination with a facility PLC in a visual sense monitoring system for monitoring an image during stopping for an object that repeats operation and the stopping.SOLUTION: A visual sense monitoring system 10 includes: an entire camera 1 that repeatedly images an entire image of an object 70 that repeats operation and stopping; a stopping determination unit 3 that on the basis of the entire image of the object 70, determines whether or not the object 70 is stopped; a detail camera 2 that is configured by a camera being the same as the entire camera 1 or another camera and images a specific area 77 being part of the object 70 when the stopping determination unit 3 determines that the object 70 is stopped; an agreement determination unit 4 that determines whether or not an image of the specific area 77 agrees with a master image; and a measurement recording unit 5 that when agreement of the image of the specific area 77 with the master image is determined by the agreement determination unit 4, measures a position of a measurement point in the specific area 77 and records the position over time.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a visual surveillance system and a visual surveillance method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, visual monitoring system devices are known that visually monitor changes that are precursors of abnormalities in manufacturing processes or the like using images taken by a camera.

[0003] For example, Patent Document 1 discloses a process monitoring device that uses image recognition processing to collect operation data and monitor the operation status of a manufacturing process executed by a control device such as a PLC. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-108238 A Summary of the Invention [Problem to be solved by the invention]

[0005] In a system that constantly stores all videos captured by a surveillance camera such as a vehicle drive recorder and reviews the entire video when an abnormality or an alert occurs, the amount of data to be stored and the amount of work to review the videos are enormous. For example, consider a case where an object that moves intermittently on a production line repeatedly moves and stops, and signs of abnormality are predicted by monitoring the trend of changes in the position when it stops. In this case, it is conceivable to connect a surveillance system including a camera to the facility PLC at the installation site, and have the camera take an image of the object when it stops every time a production line stop signal is received from the facility PLC, and monitor the trend.

[0006] However, in order to connect the monitoring system to the PLC of the facility where the system is to be installed and coordinate their operations, in addition to installing the cameras and peripheral devices, it is necessary to investigate the control of the facility where the system is to be installed, create a monitoring system program and install it on-site, check the imaging operation and modify the control, and configure the display screen, which requires a significant amount of work.

[0007] It's one thing if the monitoring system is fixed to one facility, but if the monitoring system is used to patrol multiple facilities with different PLC specifications, it is inefficient to have to spend a lot of time introducing the system every time a connection is made. Also, depending on the PLC specifications, it may not be possible to introduce it to the facility.

[0008] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a visual monitoring system and a visual monitoring method for monitoring images of an object that repeatedly moves and stops when it is stopped, which does not require connection and coordination with an equipment PLC. [Means for solving the problem]

[0009] (First aspect) The visual surveillance system of the present invention comprises a panoramic camera (1), a stop determination unit (3), a match determination unit (4), and a measurement and recording unit (5). The panoramic camera repeatedly captures a panoramic image of an object (70) that repeatedly moves and stops.

[0010] The stop determination unit determines whether the object is stopped based on the entire image of the object. If the stop determination unit determines that the object is stopped, the match determination unit determines whether an image of a specific region (77), which is a part of the object, matches the master image.

[0011] When the matching determination unit determines that the image of the specific region matches the master image, the measurement record unit measures the positions of the measurement points in the specific region and records them over time. The stop determination unit determines that the object is stopped when, for example, the brightness value difference between the image of the entire camera and the previous frame is less than a threshold value.

[0012] The visual monitoring system of the present invention can monitor images of a specific area using the stop determination unit's determination that an object has stopped as a trigger, even without receiving a production line stop signal from the equipment PLC. Since monitoring is possible with the system alone, it is only necessary to perform routine tasks such as "installing the camera and peripheral devices," registering various parameters, checking operation, and setting up the screen for each piece of equipment to be introduced. This reduces the significant amount of work required for connection and coordination with the equipment PLC. In addition, the system is highly versatile as it can be introduced to any equipment regardless of the PLC specifications.

[0013] Preferably, the visual surveillance system further includes a detail camera (2) configured as a camera separate from the overall camera, which captures an image of a specific area when the stop determination unit determines that the object is stopped.

[0014] Compared to the resolution required for matching and ensuring accuracy in position measurement, the images from the overall camera used for stop detection do not require high resolution. On the other hand, a higher frame rate for the images from the overall camera is better for quickly recognizing the timing of stopping. Therefore, by separating the overall camera from the detailed camera, it becomes easier to select cameras with specifications that meet the respective required characteristics.

[0015] (Second aspect) The visual monitoring method of the present invention includes an overall camera imaging step (S11), a stop determination step (S14), a match determination step (S27), and a measurement and recording step (S31, S32). In the overall camera imaging step, an overall image of an object (70) that repeatedly moves and stops is repeatedly captured by the overall camera (1). In the stop determination step, it is determined whether the object is stopped or not based on the overall image of the object.

[0016] If it is determined that the object is stationary, a match determination step determines whether an image of a specific area (77) that is a part of the object matches the master image. If it is determined that the image of the specific area matches the master image, a measurement recording step measures the positions of measurement points within the specific area and records them over time.

[0017] Preferably, when the stop determination unit determines that the object is stopped, the image of the specific area is captured by a detail camera (2) configured as a camera separate from the overall camera. According to this visual monitoring method, the same effects as those of the visual monitoring system described above can be obtained. [Brief description of the drawings]

[0018] [Figure 1] 1A and 1B are schematic plan and front views of a production line to which a visual monitoring system according to an embodiment is applied; [Diagram 2] FIG. 1 is an overall view of an electronic circuit board, which is an example of a target object. [Diagram 3] Magnified view of a particular area. [Figure 4] FIG. 1 is a block diagram of a visual surveillance system according to one embodiment. [Diagram 5] 13 is a flowchart of a stop determination thread according to an embodiment of the visual monitoring method. [Figure 6] 6 is a time chart showing a change in an evaluation value ΔLm_V of a luminance value difference. [Figure 7] 13 is a flowchart of a match determination and measurement recording thread according to an embodiment of the visual monitoring method. [Figure 8] 13 is a flowchart of a match determination and measurement recording thread according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (One embodiment) A visual monitoring system and a visual monitoring method according to an embodiment of the present invention will be described with reference to the drawings. The visual monitoring system and the visual monitoring method according to the present embodiment visually monitor changes that are a sign of abnormalities in a manufacturing process or the like using images from a camera.

[0020] 1(a) and (b) show a production line 90 to which a visual monitoring system 10 is applied. A slider 92 is movable on a conveying path 91 of the production line 90. An object 70 is placed on a pallet 93 of the slider 92 and is conveyed so as to move intermittently. The visual monitoring system 10 is composed of an overall camera 1, a detailed camera 2, and a processor 30. The overall camera 1 and the detailed camera 2 are installed above the conveying path 91, and capture images of the object 70 which repeatedly moves and stops.

[0021] FIG. 2 shows an electronic circuit board of an in-vehicle motor control ECU as an example of the object 70. Electronic elements such as ICs, power elements, and capacitors are mounted on this electronic circuit board, and a plurality of mounting hole seats are formed at the four corners and at the intermediate positions along the long sides. For example, the seat at the lower left of FIG. 2 is selected as a "specific area 77 that is a part of the object 70". FIG. 3 shows an enlarged view of the specific area 77. For example, a pattern to be compared with the master image is drawn on the surface of the specific area 77. Alternatively, even if there is no special dedicated pattern, the shape or pattern of the object 70 itself in the specific area 77 may be compared with the master image. In addition, the center point of the specific area 77 is set as a measurement point M for position measurement.

[0022] In this example, the position of the specific area 77 relative to the entire object 70 is fixed, and if the position of the seat, which is the specific area 77, deviates from the reference position, it is determined that the entire object 70 deviates from the reference position. In other words, by monitoring the trend of change in the position of the specific area 77 of the object 70 over time, the change in the setting position of the object 70 on the production line 90 is estimated. The purpose of the visual monitoring system 10 is to monitor this trend and prevent a decrease in productivity due to line stoppages or breakdowns caused by equipment factors.

[0023] The visual monitoring system 10 first repeatedly captures images of the object 70 and determines whether the object 70 is stationary on the conveying path 91 of the production line 90. If the visual monitoring system 10 determines that the object 70 is stationary, it measures the position of a measurement point M within the specific area 77 based on the image of the specific area 77.

[0024] The configuration of a visual monitoring system 10 according to an embodiment will be described with reference to FIG. 4. The visual monitoring system 10 includes an overall camera 1, a detailed camera 2, a stop determination unit 3, a match determination unit 4, and a measurement and recording unit 5. The stop determination unit 3, the match determination unit 4, and the measurement and recording unit 5 are included in a processing device 30. The overall camera 1 and the detailed camera 2 communicate signals with the processing device 30 via wire or wirelessly. The processing device 30 transmits an imaging trigger signal to the detailed camera 2. The processing device 30 also receives and processes image data captured by the overall camera 1 and the detailed camera 2. The processing device 30 is not necessarily installed near the production line 90, and image data may be transmitted from the cameras 1 and 2 to the processing device 30 in a remote location via the cloud, for example.

[0025] While the systems described below as “other embodiments” have a single overall camera, the visual surveillance system 10 of one embodiment has two cameras: an overall camera 1 and a detailed camera 2 which is constructed separately from the overall camera 1.

[0026] The overall camera 1 repeatedly captures an overall image of the object 70. The detailed camera 2 captures an image of a specific area 77. The detailed camera 2 is configured with a camera with a higher resolution (more pixels) than the overall camera 1. In addition, the frame rate of the overall camera 1 (e.g., 30 fps) is higher than the frame rate of the detailed camera 2 (e.g., about 6 fps). By separating the overall camera 1 and the detailed camera 2, it becomes easier to select cameras with specifications that meet the required characteristics of each.

[0027] The stop determination unit 3 determines whether the object 70 is stopped based on the entire image of the object 70. In detail, the stop determination unit 3 detects the brightness value of the entire image, and determines that the object 70 is stopped when the brightness value difference from the previous frame is less than a threshold value. When it is determined that the object 70 is stopped, an imaging trigger is output to the detail camera 2.

[0028] If it is determined that the object 70 is stationary, the detail camera 2 captures an image of the specific area 77. Furthermore, the match determination unit 4 determines whether or not the image of the specific area 77 matches the master image. If the match determination unit 4 determines that the image of the specific area 77 matches the master image, the measurement and recording unit 5 measures the positions of a total of M points within the specific area 77 and records them over time.

[0029] The flow of a visual monitoring method according to an embodiment will be described with reference to the flowcharts in Fig. 5 and Fig. 7, and Fig. 6. This visual monitoring method includes a stop determination thread (Fig. 5) and a match determination / measurement recording thread (Fig. 7). In the explanation of the flowchart, the symbol "S" means a step. In addition, the name of each processing time associated with the progress of the step is added.

[0030] In the stop determination thread, the stop determination unit 3 determines whether the object 70 is stopped or not based on the overall image of the object 70 captured by the overall camera 1. In the overall camera imaging step S11, the overall camera 1 repeatedly captures an overall image of the object 70 at a frame rate of, for example, 30 fps. S11 to S14 are repeated for each cycle of the frame rate until YES is determined in S14.

[0031] In S12, the stop determination unit 3 detects luminance at N points (N is an integer equal to or greater than 2) of the entire image, and calculates luminance value differences ΔLm1 to ΔLmN from the previous frame. In S13, the root mean square (RMS) of the luminance value differences ΔLm1 to ΔLmN of the N points is calculated as a positive value as an evaluation value ΔLm_V of the luminance value differences, for example, by the following formula:

[0032]

number

[0033] In the stop determination step S14, it is determined whether the evaluation value ΔLm_V of the brightness value difference is less than the threshold value. When the object 70 is moving in a parallel motion or the like, the brightness of the entire image changes significantly, so the evaluation value ΔLm_V of the brightness value difference is equal to or greater than the threshold value. Therefore, the determination in S14 is NO, and the process returns to S11. When the object 70 stops moving, the evaluation value ΔLm_V of the brightness value difference becomes less than the threshold value, and the determination in S14 is YES.

[0034] 6 shows an example of change in the evaluation value ΔLm_V of the brightness value difference over time. When the object 70 is moving, the evaluation value ΔLm_V of the brightness value difference exceeds the threshold, and when the object 70 is stationary, the evaluation value ΔLm_V of the brightness value difference falls below the threshold. Pt1 and Pt2 are frames in which the object 70 switches from moving in the previous frame to stationary.

[0035] In S15, it is determined whether the previous stop confirmation flag is OFF. If the answer is YES in S14 and the previous stop confirmation flag is OFF (S15: YES), the stop confirmation flag is changed from OFF to ON in S16, and the frame discrimination start trigger is turned ON in S17. When the frame discrimination start trigger is turned ON, the process is taken over to S21 of the match determination / measurement recording thread, and the stop determination thread is returned.

[0036] On the other hand, if the previous stop confirmation flag is ON (S15: NO), that is, if the stopped state continues, the process returns without turning the frame discrimination start trigger ON again. As shown in FIG. 6, the next frame discrimination start trigger is turned ON in the first frame Pt2 after frame Pt1 in which the evaluation value ΔLm_V of the brightness value difference exceeds the threshold once and then falls below the threshold again. Note that the next frame discrimination start trigger may be turned ON under the condition that a predetermined number of frames in which the evaluation value ΔLm_V of the brightness value difference is less than the threshold occur consecutively, or that a timer setting time has elapsed since the evaluation value ΔLm_V of the brightness value difference became less than the threshold. In other words, the idea of ​​being able to specify the image for stop determination on the time axis may be used.

[0037] In the match determination / measurement recording thread, the match determination unit 4 determines whether or not the image of the specific area 77 captured by the detailed camera 2 matches the master image. If it is determined that the image of the specific area 77 matches the master image, the measurement recording unit 5 measures the position of the measurement point M in the specific area 77 and records it over time.

[0038] In S21, it is determined whether the frame discrimination start trigger is ON, and if YES, the process proceeds to S22. In S22, it is determined whether the object 70 is present in the overall camera image. If the object 70 is present, the result is YES in S23 and the process proceeds to S24. If the object 70 is not present, the result is NO in S23 and the process returns. This avoids unnecessary capture of detailed camera images when the overall camera image is determined to be stopped because the object 70 was not supplied, such as when the production line has finished operating.

[0039] In S24, the detailed camera imaging trigger is turned ON, and the detailed camera 2 images the specific area 77 of the object 70. Since the detailed camera 2 with high resolution acquires data with a large number of pixels, the imaging time of the detailed camera 2 is longer than the imaging time of the overall camera 1. When the transfer of the detailed camera image from the detailed camera 2 to the match determination unit 4 is completed (S25: YES), the process proceeds to S26.

[0040] In S26, frame discrimination is performed using pattern matching. "Frame discrimination" means discriminating whether the captured frame is suitable as a frame to be used for position measurement. Specifically, the suitability of the frame is determined by pattern matching between the image of the specific area 77 and the master image of the registered model. Therefore, the term "frame discrimination" is almost synonymous with the term "match determination" for detailed camera images.

[0041] In S27, the match determination unit 4 determines whether the image of the specific region 77 matches the master image. Specifically, if the matching rate of the pattern matching is equal to or greater than a predetermined value, it is determined to match (YES), and the process proceeds to S31. If the matching rate is less than a predetermined value, it is determined to not match (N0), and the process returns. Note that if the object 70 shifts along the depth direction of the image captured by the detail camera 2, it may be out of focus and may result in an erroneous determination.

[0042] In measurement step S31, the position of measurement point M is measured by the measurement recording unit 5. The position of measurement point M may be expressed, for example, by orthogonal two-axis (XY) coordinates relative to a reference point. Alternatively, it may be expressed by rθ coordinates relative to the reference point, that is, the distance r from the reference point and the angle θ relative to a reference direction. The resolution of the detailed camera 2 is set so that the dynamic accuracy of the measurement position is equal to or higher than the resolution.

[0043] In a recording step S32, the position of the measured measurement point M is recorded over time. The measurement recording unit 5 adds and updates the latest data to a graph with the horizontal axis representing time, for example, and displays the graph data on a screen, or outputs the graph data to an output device such as an external monitor or printer. After S32, the match determination and measurement recording thread is returned.

[0044] This is the end of the visual monitoring system's functionality, and the tendency of position change based on the graph is evaluated by an operator or judgment means. For example, the judgment means may compare the measured value and its rate of change with a threshold value and display it on a PC screen, or may present the judgment result on a signal tower, etc. If there is a sign of an abnormality, for example, an operator may review past measurement images and investigate the cause of the abnormality.

[0045] A supplementary explanation about the relationship between the stopping time of the object 70 and the time of each process is provided. The stopping time of the object 70 must be equal to or greater than the "sum of the overall camera imaging time of the stopping judgment thread, and the object presence / absence determination time and detailed camera imaging time of the matching judgment / measurement recording thread." In other words, the processing conditions of each step are determined so that the sum of the overall camera imaging time, the object presence / absence determination time, and the detailed camera imaging time is equal to or less than the stopping time of the object 70, which is determined from the cycle time.

[0046] (effect) In a comparative example where the PLC of the equipment to be introduced is connected to the monitoring system, and the monitoring system monitors when the equipment is stopped using a stop signal from the equipment PLC as a trigger, a large amount of man-hours are required to connect and link with the equipment. For example, if the man-hours for each process are estimated as shown below, a total of 60.0 hours of man-hours are required for each piece of equipment to be introduced. Also, depending on the specifications of the PLC, it may not be possible to introduce it into the equipment.

[0047] (a) Installation of cameras and peripheral devices 4.0H (b) Survey of facility control at the site of introduction 16.0H (c) Programming of monitoring system and on-site implementation 24.0H (g) Check the imaging operation and correct the control 8.0H (h) Display screen settings 8.0H

[0048] In contrast, in the visual monitoring system 10 of this embodiment, even if a production line stop signal is not received from the equipment PLC, the image of the specific area 77 can be monitored by being triggered by the determination of the stop of the object by the stop determination unit 3. The required man-hours for each piece of equipment to be introduced are estimated as follows. (a) requires the same man-hours as the comparative example. (c*), (g*), and (h*) are processes corresponding to the comparative example, but the man-hours are reduced compared to the comparative example. (d), (e), and (f) are registrations of various parameters not present in the comparative example.

[0049] (a) Installation of cameras and peripheral devices 4.0H (c*) Adding a monitoring system program 0.2H (d) Stop judgment registration 0.3H (e) Master image registration 0.5H (f) Measurement point registration 1.0H (g*) Check the imaging operation, correct the control 1.0H (h*) Display screen settings 1.0H

[0050] The total man-hours in this embodiment are 8.0 hours, which is approximately 87% less than the 60.0 hours in the comparative example. In this embodiment, the system can monitor independently by performing stoppage judgment based on the overall camera image, and the considerable man-hours required for connection and coordination with the equipment PLC can be reduced.

[0051] (Other embodiments) (1) The system is not limited to a configuration having two cameras, an overall camera 1 and a detailed camera 2 configured as a camera separate from the overall camera 1, and one overall camera may also function as a detailed camera. The function of the overall camera is to capture an overall image used for stop determination. The function of the detailed camera is to capture an image of a specific area used for matching determination and position measurement of a measurement point. In one pattern, the detailed camera image may be captured separately after the overall camera 1 captures the overall image, in accordance with the flowcharts of Figures 5 and 7.

[0052] Alternatively, in another pattern, if the resolution of the entire image is sufficient for position measurement, an image of a portion of the specific region 77 in the entire image may be used as a detailed camera image. In that case, S24 and S25 in Fig. 7 are skipped, and the entire camera image is used to determine the presence or absence of an object, followed by a match determination and position measurement. Specifically, it is sufficient to set the measurement point M in the position measurement to be registered with respect to the entire camera image.

[0053] (2) In another pattern, after the object presence / absence determination and the match determination are performed using the overall camera image including the image of the specific area 77, a detailed camera image for position measurement may be captured and the position measurement may be performed. This pattern may also be adopted in a configuration in which the overall camera 1 and the detailed camera 2 are configured as two separate cameras. FIG. 8 shows a flowchart of the match determination / measurement recording thread according to this pattern as a modified example. After frame determination (S26) is performed using the image for stop determination and the match determination (S27) is determined as YES, the detailed camera image for position measurement is captured (S28, S29).

[0054] (3) The shape of the specific area 77 in the object 70 may be any shape that allows easy determination of whether it matches or does not match the master image. Also, the specific area is not limited to one location for one object, and multiple locations may be set. The image from the detailed camera is collated with the master image corresponding to each specific area to identify which specific area it is, and the position of the measurement point M within that specific area is measured.

[0055] (4) The visual monitoring system and method of the above embodiment are used with the goal of monitoring the positional deviation of the entire object 70 based on the measured position of the specific area 77, on the premise that the position of the specific area 77 relative to the entire object 70 is fixed. On the other hand, if there is a possibility that the position of the specific area 77 relative to the entire object 70 may fluctuate, the visual monitoring system and method of the present invention may be used for the purpose of monitoring the positional deviation of the specific area 77.

[0056] (5) The key point of the present invention is not to capture an image of a stationary object, but to capture an image of an object that repeatedly moves and stops, and to determine whether it has stopped. The movement of the object is not limited to translation on a linear production line, but also includes rotation on an index table, turning around to switch the front and back, vibration, etc.

[0057] The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit of the present invention. [Explanation of symbols]

[0058] 10 Visual surveillance system, 1. Overall camera, 2. Detail camera, 3... Stop determination unit, 4... Match determination unit, 5... Measurement and recording unit, 70... object, 77... specific area, S11: Whole camera imaging step, S14: Stop determination step, S27: Match determination step; S31: Measurement step, S32: Recording step.

Claims

1. An overall camera (1) that repeatedly captures an overall image of an object (70) that repeatedly moves and stops; a stop determination unit (3) that determines whether the object is stopped based on a whole image of the object; a match determination unit (4) that determines whether an image of a specific area (77) that is a part of the object matches a master image when the stop determination unit determines that the object is stopped; a measurement recorder (5) that measures positions of measurement points within the specific area and records them over time when the match determiner determines that the image of the specific area matches the master image; A visual surveillance system comprising:

2. The visual surveillance system of claim 1, further comprising a detail camera (2) configured as a camera separate from the overall camera and configured to capture an image of the specific area when the stop determination unit determines that the object is stopped.

3. an overall camera imaging step (S11) of repeatedly capturing an overall image of an object (70) that repeatedly moves and stops by an overall camera (1); a stop determination step (S14) of determining whether or not the object is stopped based on the entire image of the object; a match determination step (S27) for determining whether an image of a specific area (77) that is a part of the object matches a master image when the object is determined to be stationary; a measurement and recording step (S31, S32) of measuring positions of measurement points in the specific area and recording them over time when it is determined that the image of the specific area matches the master image; A visual surveillance method comprising:

4. The visual surveillance method according to claim 3, wherein when the stop determination unit determines that the object is stopped, an image of the specific area is captured by a detail camera (2) configured as a camera separate from the overall camera.

Images