Surveillance system

Abstract

To provide a monitoring system with which it is possible to accurately grasp the motion of humans or things and accurately monitor manufacturing processes, thus contributing to improving work efficiency.SOLUTION: This monitoring system comprises a camera for monitoring humans, things or devices, an analysis unit for analyzing image data obtained by the camera, and a display unit for displaying a result of analysis performed by the analysis unit. The analysis unit includes change point detection means for detecting a change point in the image data and analyzing the state of the human or the thing or the device on the basis of the change point. The analysis includes a step for removing blurs of the image data after having acquired the image data, and a step for referring to an image included in the image data in a library and recognizing the type of the image.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a monitoring system for monitoring humans, objects, and devices at the manufacturing site of objects.

Background Art

[0002] In the manufacturing site of objects, a system for monitoring humans involved in manufacturing, objects to be manufactured, and devices or machines (manufacturing devices, inspection devices, packaging devices, etc.) for executing manufacturing processes is known, and further improvements are being made (see, for example, Patent Documents 1 to 3).

[0003] For example, in the manufacturing site of the assembly industry, a product is produced by aggregating a plurality of parts and combining operations (for example, welding, cutting, assembly, etc.) along a predetermined assembly drawing, work procedure manual, etc. In the case of a complex product, predetermined operations are carried out at a plurality of work sites, and each part (unit) is put together to produce a more complex product. In such work, when the types of parts (objects) to be aggregated are numerous, the work procedures, on-site layout, part aggregation work, etc. become complicated, and work waiting, part stagnation, deviation in part aggregation timing, misplacement of workers (humans), etc. are likely to occur, and a problem of reduced efficiency of the assembly work arises. However, especially in the assembly industry, since the use of equipment and machinery is less and the assembly work by humans is central, the logging environment by sensors is not well organized. For this reason, a monitoring system for accurately grasping and monitoring the states of humans and objects at the manufacturing site is desired. In other words, it is desired to construct a digital twin of the assembly industry and utilize it to improve management numerical values and realize monitoring.

[0004] However, when quantifying and recording the working state of a human, it is necessary for the worker to stop working to record the working state, or for another worker to record while observing the actions of the worker, which has the problem of being time-consuming and costly.

[0005] Also, regarding the monitoring of the state of things, when the thing is inside a machine, it is possible to record and monitor the operations of the machine. However, when the thing moves from a machine at one work site to a machine at another work site, its state cannot be fully monitored.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] The present invention provides a monitoring system that can accurately grasp the movements of humans and things, accurately monitor the manufacturing process, and contribute to improving work efficiency.

Means for Solving the Problems

[0008] To solve the above problems, a monitoring system according to the present invention is a monitoring system that monitors humans, things, or devices at a manufacturing site, and includes a camera that monitors the humans, things, or devices, an analysis unit that analyzes image data obtained by the camera, and a display unit that displays the analysis result by the analysis unit. The analysis unit includes change point detection means for detecting change points in the image data and analyzing the state of the human, thing, or device based on the change points. The analysis includes a step of removing blurring of the image data after acquiring the image data, and a step of referring to an image included in the image data in a library and recognizing the type of the image.

Effects of the Invention

[0009] According to the monitoring system of the present invention, it is possible to provide a monitoring system that can accurately monitor the manufacturing process and contribute to the improvement of work efficiency.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Mode for Carrying Out the Invention

[0011] Hereinafter, this embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may sometimes be denoted by the same number. Note that the accompanying drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are by no means used for limitingly interpreting the present disclosure. The description in this specification is merely a typical example and does not limit the scope of the claims or application examples of the present disclosure in any sense.

[0012] In this embodiment, although the description is made in sufficient detail for those skilled in the art to implement the present disclosure, other implementations and forms are also possible, and it is necessary to understand that configuration and structural changes and replacement of various elements can be made without departing from the scope and spirit of the technical idea of the present disclosure. Therefore, the following description should not be construed as being limited thereto.

[0013] Hereinafter, an embodiment related to the process of assembling a product (final product and unit as an intermediate product) composed of a plurality of parts will be described. Parts ordered and delivered from the backbone system or the ordering system are carried into and stored in the parts warehouse. On the other hand, when a manufacturing work order is issued in accordance with the manufacturing plan of the product, the parts are conveyed to the assembly shop in accordance with a predefined manufacturing plan (including the assembly drawing and assembly procedure manual created by the design department) for each work order. In this assembly shop, assembly personnel and necessary equipment and jigs are arranged in accordance with the manufacturing plan. The assembly personnel carry out predetermined assembly work using the necessary equipment and jigs according to the case in accordance with the manufacturing plan.

[0014] The product on which the specified assembly work has been carried out is conveyed to the next assembly shop. By repeating this and integrating, the final product is completed, the specified shipping inspection is carried out, and it is stored in the shipping warehouse. This embodiment provides a means and method for recording the above-mentioned assembly process and the progress of work proceeding in the factory. Specifically, for each warehouse, dispensing shelf, and assembly shop, it records the identification information, start and completion times, and arrival and departure times of each component, each device / jig, and each worker. In the conventional technology, in order to detect the presence or absence of product components, it was necessary to individually install sensors according to the work content for each work and for each shop and the configuration of the shop. This was because, for example, whenever the work content and the size and shape of the components changed due to product changes, etc., it was necessary to change the installation position of the sensors. According to the embodiment described below, as long as components, jigs, and workers exist within the imaging range of the camera, there is no need to change the installation position of the camera. Thereby, the installation cost at the time of system introduction can be significantly reduced.

[0015] [First Embodiment] An example of the monitoring system 1 according to the first embodiment will be described with reference to FIG. 1. This monitoring system 1 can be configured, as an example, from a human monitoring camera 11, an object monitoring camera 12, a device monitoring camera 13, a transfer computer (PC) 14, a collection / accumulation server 15, an analysis server 16, and a display computer (PC) 17.

[0016] The human monitoring camera 11 is a camera for monitoring the movements of a human (worker) performing work at the manufacturing site. The human monitoring camera 11 has, for example, information on the shape of a human and characteristics of movements stored in advance, and can be configured to be able to quickly and surely detect human movements.

[0017] The object monitoring camera 12 is a camera for monitoring the movement of objects manufactured and moving at the manufacturing site. As an example, the object monitoring camera 12 can be a camera that images and identifies identification information (such as one-dimensional codes, two-dimensional codes, color charts, etc.) attached to a cart for placing and transporting an object, in order to monitor the movement of the cart. By attaching different identification information to a plurality of carts, the types of carts can be identified. However, this is only an example, and the object monitoring camera 12 is not limited to monitoring the movement of the cart by means of identification information. For example, it may recognize the shape of the object itself or identify various identification information attached to the object. Also, instead of recognizing the cart, for example, the movement of a tray placed on a belt conveyor may be monitored by the object monitoring camera 12. When the object monitoring camera 12 is configured to read a two-dimensional code, the object monitoring camera 12 can be equipped with an application for two-dimensional code analysis inside it. Of course, it goes without saying that the application for two-dimensional code analysis may also be on the cloud.

[0018] The device monitoring camera 13 is a camera for monitoring the movement of various devices (manufacturing devices, inspection devices, packaging devices, etc.) installed at the manufacturing site. The device monitoring camera 13 can be different according to the characteristics of the device to be imaged. For example, in the case of the device monitoring camera 13 for a device that requires human intervention in operation, it can be a camera that images the operation panel of the device and monitors human operations. By imaging the operation panel, it is possible to determine whether the device is in operation or stopped, etc.

[0019] In recent years, facilities and equipment can detect parameters indicating the state inside the equipment using sensors arranged inside the equipment. However, in the case of facilities without these sensors, it is necessary to modify the equipment to install these sensors. However, in the case of skilled workers who use the facilities and equipment, they can monitor whether the processing, assembly, and other processes of the device are in any process stage, and whether there is any abnormality in the state of the equipment by visual inspection and listening to sounds. Therefore, even in the monitoring of facilities, it means that the state can be monitored by utilizing a TV camera. By introducing this monitoring with a TV camera, it is possible to eliminate the introduction of a high-cost monitoring system such as modification for sensor installation and monitoring by skilled workers.

[0020] In the case of an automatic device without human intervention, the device monitoring camera 13 for the device can be, for example, a camera for monitoring the input port of a mono (part) or the movement of an operating part (for example, the blade of a cutting device). Furthermore, the device monitoring camera 13 can also be used for inspecting the content of various operations inside the device. For example, in a device for harness connection, a harness connection confirmation inspection is executed, and from the image, it can be determined whether the correct connection has been made, and if not, errors such as incorrect wiring can be detected.

[0021] Also, in this embodiment, inspection target detection means for detecting an inspection target and defect detection means for detecting defects can be provided in the cameras 11 to 13 or an externally attached controller. The inspection target detection means has a function of detecting the inspection target part in the image of a good product. As an example, the inspection target detection means can be composed of a deep learning network that has learned the inspection target part in advance. By providing this means, it is possible to avoid misdetecting parts with large deformations but not regarded as abnormal (for example, parts such as cables) as defects.

[0022] By inputting the detected part as the part to be inspected into the defect detection means, defects in the inspection target can be detected. As an example, this means can be configured by pre-training a plurality of normal parts (images of normal products) in an autoencoder. By creating a difference image between the image reconstructed by the autoencoder and the input image, if there is an abnormal part, the image signal at that position will have a large value and will be detected as abnormal through threshold processing. The detected defect information (defect image, product information, etc.) can be stored in the defect information storage means.

[0023] When a new detection target is provided, its normal image can be learned by the inspection target detection means and the defect detection means (autoencoder). This learning may be performed using the information of 3D CAD generated at the time of product design.

[0024] Also, the information (network parameters) learned for the new detection target can be reflected by performing a weighted average with the corresponding parameters of the already learned network. By this method, it is possible to handle cases where each user does not want to upload information regarding a new detection target to the cloud. That is, by sending the parameters of the learned network shared on the cloud to the user's site and performing a weighted average at that site, it is possible to configure and provide highly accurate defect detection means and inspection target detection means specific to the user.

[0025] Note that the device monitoring camera 13 can use various lenses (ultra-close-up lens, fisheye lens, wide-angle lens, telephoto lens, etc.) according to the characteristics of the imaging target and the distance to the imaging target.

[0026] The progress of the assembly work at the assembly process site can be visually confirmed and recorded by a skilled person or administrator with specific skills. From this perspective, in this embodiment, appropriate data processing described below is performed only on the imaging information by a TV camera that captures images with visible light, and the attachment / detachment times to be recorded are calculated.

[0027] Also, when the assembly work at each assembly shop during the assembly process is completed, not only the completion time is recorded, but also the start and completion times for each process of the assembly work, and defects in the intermediate products after assembly are detected and recorded. The sampling time of this TV camera does not necessarily have to be the normal 30fps or 60fps, and can be thinned out to numerical values such as 1fps or 0.2fps as needed. Conversely, high-speed imaging such as 120fps or 2000fps may also be used. The former has the effect of reducing the total amount of stored data, and the latter is required when collecting data with high-speed movement.

[0028] The transfer PC 14 has a function of reading video data (which may include both moving images and still images) captured by various cameras 11 to 13 and transferring it to the collection and storage server 15. The transfer PC 14 can also be configured to perform a data compression operation by a well-known method prior to the transfer operation. In addition to the data compression operation, it is also possible to perform image processing such as image processing (for example, leaving only the parts of the images of humans and objects that are the monitoring targets and deleting unnecessary images, or increasing the resolution of the monitoring target images and decreasing the resolution of the images of other objects). Note that the transfer PC 14 may be wirelessly connected to the collection and storage server 15 or may be wired-connected.

[0029] The collection and storage server 15 is a computer for collecting and storing the video data transferred from the transfer PC 14 in a storage unit (not shown). Although the analysis of the collected video data is performed by the analysis server 16, it is also possible for the collection and storage server 15 to be responsible for a part of the analysis process, for example, a part of the analysis of the video data.

[0030] The analysis server 16 analyzes various video data stored in the collection and storage server 15, analyzes the operations of humans, objects, and devices, and generates analysis data as the result of the analysis. The generated analysis data is output to the display PC 17 and visualized and displayed in a form recognizable by the operator. Note that the analysis server 16 can also be configured to analyze the difference between the analysis data as the result of analyzing the operations of humans, objects, and devices and the work plan data.

[0031] The analysis items in the analysis server 16 can include items that a human (such as a site supervisor) can discover during daily supervision work, but it is not necessary to be limited to this. For example, the signal A included in the video data acquired and stored (learned) in the collection and storage server 15 over a long period of time can be compared with the signal A' included in the newly acquired video data, and the comparison result can also be used as an analysis item. When there is a difference of a predetermined value or more between the signals A and A' obtained in this way, the analysis server 16 functions as a change point detection means for identifying the location where this difference occurred as a "change point" indicating a significant change point in the operation of a human, an object, or a device.

[0032] In addition, by identifying such a change point, the analysis server 16 can detect the time when significant operations of humans, objects, and devices occur, which are likely to be overlooked in human monitoring, and can be used as an indicator of factors inhibiting the work efficiency of the manufacturing process. Also, in the case of human monitoring, it is necessary to separate the data when the correct operation is being performed from the data when an abnormal operation is being performed. While it is easy to prepare the correct data, it takes time to prepare the incorrect data. According to this system, it is possible to easily introduce the system in a short period without the need to prepare incorrect data. The analysis server 16 associates the time and place where such a change point occurred and mathematically defines it as "working hours at a specific work location".

[0033] Generally, just because a human and objects are present at a predetermined location does not mean that the work is actually proceeding properly. However, by comparing and analyzing the work time data obtained as described above, for example, (a) when the current work deviates from the average work time, (b) when the variation in work time is large among multiple works, (c) when a human or object stays outside a specific work area etc. can be "mechanically" detected. These are different from the "work time" defined or recognized by humans, but can detect the inhibiting factors of the manufacturing lead time. That is, (a) the fact that it "deviates from the average time" is an inhibiting factor, (b) there is an inhibiting factor of "large variation" in the work itself. (c) there are inhibiting factors of "waiting for work" and "staying of objects".

[0034] Referring to FIG. 2, the operation of the human monitoring camera 11 will be schematically described. FIG. 2 shows a state where a plurality of machines ((1) to (9)) are arranged at the manufacturing site and a human (worker) H moves along the arrow, which is imaged by the human monitoring camera 11. The human monitoring camera 11 can store, as an example, information on the physique of the human H, the properties (color, shape, etc.) of the clothes usually worn by the human H, and the characteristics of the movement (average walking speed, stride). The human monitoring camera 11 can identify the image of the human H included in the video data by comparing the information with the video data. The identified image of the human H is surrounded by a bounding box BB, and the video data can be recorded in a form including the bounding box BB.

[0035] In this way, when video data is obtained by the human monitoring camera 11, the video data is analyzed in the analysis server 16, and it is possible to analyze what work the human H performs in which time period (dwelling time), in which time period the human does not perform work but moves or the like (moving time), or is located outside the shooting range, according to the change points of the identified human actions. The change points of the human actions can be identified according to the time when the human arrives in front of a predetermined machine, the time of departure, and other information. The bar display in the lower right of FIG. 2 shows the working times (including the start time and the end time) related to the operations (4) to (8). Such a bar display can also be displayed on the display screen of the display PC 17.

[0036] In this embodiment, the worker detection means is constituted by the human monitoring camera 11 or an external controller. The human monitoring camera 11 as the worker detection means detects the worker in the acquired image and can include a deep learning network that has learned the image of the worker in advance. By this worker detection means, the worker is detected in the captured image. From the detected position, it is detected whether the worker exists at a predetermined position, that is, the entry time and the exit time to the work location. In this embodiment, the captured image is acquired from a plurality of directions, and the existence position (direction of the viewpoint) of the worker in each two-dimensional image is known, and the intersection position in that direction is detected as the existence position of the worker in the three-dimensional space. This method of detecting the existence position of the worker from a plurality of images is effective in cases where it is desired to improve the detection accuracy of the depth. There is also an effect that detection is made possible by using a plurality of images in a case where the worker becomes the shadow of equipment or other objects and does not move into the image.

[0037] Referring to FIG. 3, the operations of the object monitoring camera 12 and the machine monitoring camera 13 will be schematically described. In FIG. 2, as an example, an example in which three object monitoring cameras 12 (12A to 12C) and three machine monitoring cameras 13 (13A to 13C) are arranged is described, but this is merely an example, and the numbers of the object monitoring cameras 12 and the machine monitoring cameras 13 are not limited to specific numbers. Further, the object monitoring camera 12 is not limited to a specific form, but preferably, as schematically shown in FIG. 3, it can be a camera that captures a two-dimensional code 22 attached to a cart 21 on which an object (such as a part) is placed.

[0038] The cart 21 moves between a plurality of machines (11) to (13), for example, as shown in FIG. 3. For example, when the cart 21 finishes the previous process than the machine (11) and moves in front of the machine (11), the two-dimensional code 22 attached to the cart 21 is imaged by the object monitoring camera 12A. Thereby, the arrival of the cart 21 at the machine (11) is detected.

[0039] When the work at the machine (11) is finished and the cart 21 leaves the machine (11) and moves toward the next machine (12), the object monitoring camera 12A detects the departure of the cart 21 from the machine (11) by detecting that the two-dimensional code 22 is no longer imaged. When the cart 21 arrives in front of the machine (12), the two-dimensional code 22 attached to the cart 21 is imaged by the object monitoring camera 12B. Thereby, the arrival of the cart 21 at the machine (12) is detected. By repeating such operations, based on the video data of the object monitoring camera 12, the movement of the cart 21, that is, the object (such as a part) can be detected.

[0040] Note that the change points of the operation of the object can be specified by the start time and the departure time at a certain machine (see FIG. 4). The movement time (waiting time) from the upper machine (process) to the lower machine (process) can be calculated by the difference between the departure time from the upper machine and the arrival time at the lower machine. Note that the change points of the operation of the machine can also be specified in the same manner by the time when a certain machine starts a specific operation and the time when it ends.

[0041] In addition, when the component intermediate product placed on the cart 21 is placed on the cart 21 in the food service shelf, a correspondence relationship is given with the product manufacturing number etc. in which the cart number, component number, and component number are linked. The correspondence relationship is also given as information to the two-dimensional code 22, and when the two-dimensional code 22 is read, the information on these correspondence relationships is automatically collected. By this series of automatic collection, the cart 21, components, operator, shop, and change time are linked. This linking is also made among other shops, other products, and other operators. As a result, at a specific time, it is recognized where the components, operator, equipment, and jig are located, and the vector data thereof is stored in the storage means. Further, the detection of the time of the state change does not necessarily have to be based on the image captured by the TV camera, and it may be a combination with information from other sensors, already installed sensors, etc.

[0042] With reference to the flowchart of FIG. 5, the operation of the monitoring system 1 according to the embodiment will be described. In this system, the human monitoring camera 11, the object monitoring camera 12, and the machine monitoring camera 13 operate independently to monitor the operations of humans, objects, and machines. The obtained video data is transferred to and read by the transfer PC 14, and the transfer PC 14 performs a predetermined data compression operation on the video data and then transfers the video data to the collection / accumulation server 15. The collection / accumulation server 15 receives and stores the transferred data. The data stored in the collection / accumulation server 15 is appropriately transmitted to the analysis server 16 and is made the subject of analysis.

[0043] Based on the stored data, the analysis server 16 identifies the change points of humans, objects, and devices as described above, identifies the time when the change points occurred, and further the location where the change points occurred, and stores that information in a storage unit (not shown). By identifying a plurality of change points, it is possible to identify the start time and end time of significant operations of each machine of humans and objects.

[0044] When the change points, the times and locations at which the change points occurred are specified, the states of people and things in each process can be understood. As a result, the time required for each process can be understood, and among a plurality of processes, it can be determined in which process there is room (margin) to shorten the lead time. That is, the analysis server 16 can function as a margin determination means for determining the margin in each process based on the specified change points and the data at the time when a significant operation occurred. Regarding the margin, it can be visually displayed on the display PC 17.

[0045] FIG. 6 is an example of a display screen when the analysis result in the analysis server 16 is displayed on the display of the display PC 17. In this display example of FIG. 6, the horizontal axis is the time axis, and for each different production number, a Gantt chart is used to compare the time line of the production plan and the time lines of each process (Process 1 to 3) in the actual work (performance) by bar display. According to this display, the differences between the production plan and the actual work on each working day are visualized. According to this screen display, it can be visualized which production number is in which process, and the progress in a large process that is executed over several days can be grasped for each small process.

[0046] As shown here, the monitoring system of this embodiment exhibits a great effect when monitoring the progress of the manufacturing plan. When the progress of the manufacturing plan lags, it means that labor costs, depreciation costs of equipment, etc. are accumulated, leading to an increase in manufacturing costs. This is the reason why the effect of progress management is significant. Also, when a large delay occurs, by reexamining the manufacturing plan (manufacturing schedule), an appropriate manufacturing output can be obtained. This is the reason why replanning (rescheduling) of the manufacturing plan is necessary. At this time, the magnitude of the delay can be set in advance, for example, as a 20% increase or a 30% increase in the lead time set initially, or, for example, as a 20% increase or a 30% increase in the tact time of a specific process that is a bottleneck. That is, when determining the timing when replanning is necessary, the monitoring system of this embodiment plays an important role. In other words, by combining and using the monitoring system and the scheduler of the present invention, a greater effect can be produced. At this time, the scheduler to be tried can be a stacking type automatic scheduler or a scheduler equipped with some optimization algorithm, and a certain effect can be expected.

[0047] Figure 7 is another example of a display screen when the analysis result in the analysis server 16 is displayed on the display of the display PC. In the upper display example of this Figure 7, the horizontal axis represents the elapsed time, and the breakdown of the working time of each process (Process 1 to 3) for different lot numbers is displayed as the length of the bar. According to this display, the variation in the total working time for each lot number is visualized, and the variation in the required time for each process is also visualized. This makes it easy to grasp the lot numbers that require a long working time and to investigate the causes.

[0048] Also, in the lower display example of Figure 7, although the horizontal axis represents the elapsed time, for each of the plurality of carts, the moving time between different processes is displayed as the length of the bar. According to this display, it is possible to visualize which cart has a long moving time between which processes, and it is possible to facilitate the investigation of the causes.

[0049] Figure 8 graphs the operating time and the time under operation in each process (Processes 1 to 3). Focusing only on a single process, the higher the ratio of the operating time to the total (operating rate), the higher the working efficiency. However, if the operating rate is high in a certain process while it is low in an adjacent process, the working efficiency of the entire process cannot be improved. Thus, when there is a difference in the operating rate between adjacent processes, it can be determined that a bottleneck has occurred in the work. When a bottleneck occurs, the working efficiency can be improved by changing the manufacturing plan including the personnel arrangement.

[0050] Note that in the embodiments described above, there may be a case where, for security reasons, the user wants to avoid uploading the information in the facility (information on the processing of the captured data after acquisition in this embodiment, the detection of the time of state changes of people, objects, and equipment, and the processing results of subsequent data processing) to the cloud. In this case, the transition time, product information, and other information can be encoded and decoded in advance with the password set by the user. As a result, these information can be handled as text information whose content has meaning to people in the facility, and as information that has no meaning without the password on the cloud.

[0051] Also, by performing identification using a hash function before and after using the data on the cloud, it can be ensured that the data on the cloud has not been tampered with, and the security of the user data can be provided. This method adds the pre-hash value to the stored data to generate the next hash value, resulting in a situation where tampering takes a long time and is substantially impossible, and as a result, the security can be further improved. It may also utilize blockchain technology.

[0052] In addition, in a system that uses the images of a TV camera, the operator may feel resistance to being imaged or may be concerned about the leakage of personal information. In contrast, at an initial stage, means for performing image conversion such as Fourier transform on the captured image and converting it into information that cannot be judged by human eyes can be provided in a controller or the like. In subsequent processes such as learning, by using such a converted image, the same image conversion can be executed in subsequent processes such as operator detection.

[0053] By acquiring data of various times including the work completion time detected in this way for each product, the lead time from the start to the completion of the product can be analyzed as the work time for each work. Also, for example, at the end of the work time of a day, a report can be made by summarizing the progress of the work, the arrangement positions of parts, etc.

[0054] Regarding the utilization system and method of various time data such as the arrival / departure time and start / completion time detected in this way, an explanation will be given below. According to this system, by focusing on a specific product, the progress time of each work can be described. When calculating the variation of the above progress time in a specific product or a specific shop, if the variation is large due to an operator or other disturbances, it can be determined that the work is not yet mature. In that case, for example, · Replace the operator of a specific work with a member with a high work proficiency · Analyze the cause of the variation and prepare a jig for separating the work and other countermeasures can be taken.

[0055] Also, in the above case, by monitoring the variation situation of the start / completion time, the deterioration of the work quality can be known in real time.

[0056] In addition, by preparing an environment having the monitoring system, production planning system, and display system shown in FIGS. 6 to 8 of the present embodiment on the cloud, the following new values can be generated. For example, when an algorithm capable of detecting human change points with higher accuracy and high time resolution is developed, the value of the entire system can be improved by changing only this algorithm. For example, since the process at the time of process change can be decomposed more finely, by changing their combinations, the lead time reduction by process redesign can be made more accurate. In this way, by having a monitoring system, a production planning system, or a display system, new technologies and programs can be easily and quickly applied to the field. In other words, the present invention has an aspect of providing a "foundation; platform" for improving the efficiency of the production site.

[0057] Furthermore, by counting the usage time of each of the technologies and programs thus applied to the field, the total usage time of those technologies can be measured. According to this usage time, the remuneration to the system provider can be automatically calculated by a machine. This is the mechanization of administrative processing, but also in the manufacturing site, such mechanization and efficiency improvement of administrative processing have an important meaning in terms of cost reduction. Also, by having this type of usage time measurement system and paying remuneration according to the usage time, compared to introducing the system with a lump sum payment, the amount of temporary cash payment can be reduced, and an effect of improving the so-called cash flow can be expected.

[0058] In addition, according to this monitoring system, it is possible to detect and report deviations from the average working time and the assumed work completion time. In a remote monitoring system using a simple monitoring camera or the like, the monitor needs to continuously monitor the images during shooting, whereas an effect can be expected that a report can be received from the system only when deviating from the assumption.

[0059] [Effect] As described above, according to the system of the present embodiment, the operations of humans, objects, and devices are monitored by cameras 11 to 13, and according to the change points in the video data, changes in the significant operations of humans, objects, and devices are detected, analyzed in the analysis server 16, and the results can be visualized and presented. Therefore, the movements of humans, objects, and devices can be monitored by an operator visually or the like without having to monitor them one by one and input them one by one, and without spending time on condition setting, and the manufacturing process can be monitored accurately and at low cost, improving work efficiency. In other words, the same response can be achieved only by imaging with cameras 11 to 13 for the matters that are sensuously judged by the eyes of the management supervisor. As a result, the monitoring of the manufacturing site can be efficiently executed by this system without arranging a dedicated system person in charge. In addition, by visualizing the manufacturing site, the realization of visualization can be achieved, and it becomes possible to provide the solution to the management issues beyond that in one stop.

[0060] The cameras 11 to 13 used can appropriately change the installation location, change the type of camera, or increase the number, and it is easy to respond to changes in the manufacturing site. Specifically, with this system, it is possible to record the states or results of sites where it is not easy to leave records of the states or results, such as assembly work by humans, visual inspection by humans, conveyance of objects by humans, and the operation status of devices by humans. As a result, the state of the work site by humans can be recorded in the same way as a factory site composed of automatic machines having a mechanism for recording data indicating the state.

[0061] [Second Embodiment] Next, a monitoring system according to the second embodiment will be described. Referring to FIG. 9, an example of a monitoring system 1A according to the second embodiment will be described. The monitoring system of the second embodiment is different from that of the first embodiment (FIG. 1) in that it is configured to monitor a cart on which parts and intermediate products (mono) in a manufacturing process are placed. FIG. 9 only shows the configuration for monitoring the cart, but the same configuration and operation as those of the first embodiment (FIG. 1) can be adopted for monitoring humans and devices. The analysis server 16 analyzes various video data stored in the collection / accumulation server 15, analyzes the operations of humans, mono, and devices, and generates analysis data as the result of the analysis. The generated analysis data is output to the display PC 17 and visualized and displayed in a form recognizable by the operator. Then, change point detection, identification of the time / location of significant operations, detection / display of spare locations, etc. (FIG. 5) can be executed in the same manner as in the first embodiment.

[0062] In recent years, the IoT implementation in factories has been promoted, and along with it, the automation of various operations has been advanced. However, in factories with small-lot production of multiple varieties, carts are often used, and parts are taken one by one from the shelves of the cart, transported to the processing location, and assembly, etc. are carried out. It is difficult to automate such operations, and many operations are performed manually. In such factories that use carts, in addition to grasping the work processes and progress status of each worker, it is required to grasp the position of the cart, and thereby grasp the delivery status of the parts and intermediate products, etc. mounted on the cart.

[0063] From such a perspective, the second embodiment, similar to the first embodiment, grasps the position of the cart 21 to which the two-dimensional code 22 is attached. As shown in FIG. 9, as an example, three object monitoring cameras 12 (12A to 12C) are arranged. This is merely an example, and the number of object monitoring cameras 12 is not limited to a specific number. The object monitoring cameras 12A to 12C can be cameras that photograph the two-dimensional code 22 attached to the cart 21 on which an object (such as a part) is placed. Note that the cart 21 may simply move along the passage and the two-dimensional code 22 may be read during the movement, or as shown in FIG. 9, the cart 21 may be moved to a pit (a stop location) not shown in the figure, and the two-dimensional code 22 may be read while the cart 21 is stopped at the pit, or a combination thereof may be used.

[0064] Similar to the first embodiment, when parts and intermediate products placed on this cart 21 are placed on the cart 21 in the food distribution shelf, a correspondence relationship is given with a product manufacturing number or the like in which the cart number, part number, and part number are associated. The correspondence relationship is also given as information to the two-dimensional code 22, and when the two-dimensional code 22 is read, the information on these correspondence relationships is automatically collected. Through this series of automatic collection, the cart 21, parts, workers, shops, and change times are associated. This association is also made among other shops, other products, and other workers.

[0065] The two-dimensional code 22 attached to the cart 21 is preferably small enough (for example, about 3 cm to 4 cm square) so that imaging of the two-dimensional code 22 by the monitoring camera 12 is not blocked by workers or the like, and in view of the structure of the cart 21. As a specific example of the two-dimensional code 22, an ArUco marker as shown in FIG. 10 is suitable. Since the ArUco marker has as few as 1000 patterns and can be read in real time from a long distance, even when the size of the marker is reduced as described above, reliable reading is possible. As long as it is small in size and can be read sufficiently from a distance, two-dimensional codes other than the ArUco marker (QR code, Chameleon code, etc. (all trademarks)) can also be used. In addition to two-dimensional codes, various identification marks can be used.

[0066] It is also conceivable to improve the design of the existing ArUco marker or adopt a marker with a new design (such as a circle). As a result, it is expected that the amount of information will increase and the determination accuracy of the progress status of the manufacturing process in the system will be improved. However, as the types of markers increase, it becomes difficult to identify markers with a size of about 3 cm × 3 cm, and an increase in size is required. In addition, in the case of a marker with a new design, there is a problem that the number of available marker IDs decreases. Therefore, in this second embodiment, while using the existing ArUco marker without modification, the following method is adopted to obtain sufficient recognition accuracy while adopting a small marker size that can be attached to the cart 21.

[0067] Figure 11 shows an example of the execution procedure of data processing when the analysis server 16 reads the two-dimensional code 22. First, the analysis server 16 acquires image data from the video data collected by the collection / accumulation server 15 (step S11), and detects the ArUco marker from the image data using deep learning (step S12). In the deep learning here, images of several (for example, about 1 to 20) ArUco markers out of 1000 ArUco markers are given to the analysis server 16 as learning data, and based on the learning data, an image determined to be an ArUco marker is specified from the image data. Therefore, in this step S11, the analysis server 16 simply determines whether the obtained image contains an ArUco marker, and does not perform the detection of its type (1000 types). The detection of the type of ArUco marker is performed in step S16 described later.

[0068] In marker detection using deep learning, it is preferable to use Faster-R-CNN, which is an object detection method. Faster-R-CNN has a model structure that identifies whether the content of a certain rectangle is an object or a background, and classifies the detected regions. Compared with conventional object detection methods such as R-CNN and Fast-R-CNN, since a CNN structure called RPN (Region Proposal Network) is used when extracting candidates for the object region, the processing time can be significantly shortened. Marker detection can be achieved by fine-tuning the pre-trained model of Faster-R-CNN with a custom dataset created for markers. In creating the custom dataset, it is preferable to use Siam Mask, which is an object tracking method at the mask unit, in order to efficiently create a large amount of labeled data. If the position of the object to be tracked is given in the first frame of the video, the position of the target can be estimated in all subsequent frames. This method is faster than other object tracking methods using deep learning and enables real-time operation.

[0069] After step S12, the analysis server 16 executes a cutting process to cut out the image around the detected ArUco marker (step S13). Specifically, the coordinates of the four vertices of the rectangle of the detected marker are detected, and the ArUco marker and its surroundings are cut out according to the coordinates. By executing such a cutting process, only the image of the ArUco marker to be detected remains, and unnecessary data is deleted, so that the subsequent data processing can be speeded up and the accuracy can be improved.

[0070] In addition, in order to remove the blur of the image caused by imaging the moving cart 21, a blur removal process for the image data is also executed (step S15). The blur removal process estimates and generates a non-blurred image from the obtained blurred ArUco marker image by using the above-mentioned learning data of the ArUco marker. As an example, DeblurGAN-v2, a method in which a generative adversarial network (GAN) is applied to the blur removal process, can be used for the blur removal process. In the model structure of DeblurGAN-v2, the generator creates a generated image while adding a high-dimensional feature map that has been upsampled and a low-dimensional feature map. In the discriminator, Patch GAN is introduced, and the generated image is divided at the patch level to determine whether it is real or fake. By doing so, it is a model that focuses on the blur of fine parts in the image. In addition, the blur removal process can also be performed on a blurred image that is not in the learning data. The blur removal process may be executed on the entire image after the image data is acquired, or may be executed after the cutting process (see FIG. 12).

[0071] The analysis server 16 further performs predetermined preprocessing on the cut-out image of the ArUco marker (step S14). The preprocessing includes, as an example, smoothing, sharpening, or both, and may also include other processes such as normalization processing. By performing smoothing and / or sharpening processing, the brightness and darkness of the ArUco marker become clear, and the accuracy of recognizing the ArUco marker is improved. For the smoothing process, for example, a Gaussian filter can be used.

[0072] Then, the analysis server 16 refers to the image of the ArUco marker after preprocessing in the OpenCV (registered trademark) library and identifies the type of the ArUco marker (1000 types) (step S16). Note that the preprocessing can also be omitted depending on the situation of the image.

[0073] As shown in FIG. 12, in the present embodiment, in the data processing when reading the two-dimensional code 22 (ArUco marker), the execution order or execution position (whole or after cutting out) of the blur removal process can be appropriately changed (method (6) or methods (3), (5) in FIG. 12), and the presence or absence of preprocessing can also be appropriately selected (method (3)). Also, after performing the blur removal process on the entire image, steps S12 to S14 can be omitted (method (2)). That is, methods (2), (3), (5), and (6) shown in FIG. 12 can be selected.

[0074] Next, the analysis results of the effects of each method are shown. FIG. 13 is a graph showing the change in the detection success rate of the ArUco marker with respect to the detection distance when using methods (1) and (4). The upper graph in FIG. 13 is a graph when the number of pixels of the image obtained by the camera is 1280×720, and the lower graph is a graph when the number of pixels is 720×400. The "detection success" graph in both graphs shows the change in the identification success rate in OpenCV when using method (4).

[0075] In method (1), the analysis server 16 only acquires the images from the surveillance camera 12 (without performing blur removal processing, detection / cutting out, and preprocessing), and the detection of the presence and type of the ArUco marker is executed by comparing an external OpenCV library with the acquired images. In method (4), the analysis server 16 similarly does not perform blur removal processing, and only executes the detection and cutting out of the ArUco marker by deep learning and the preprocessing (smoothing and / or sharpening processing). It can be seen that method (4) has an improved overall recognition success rate and an extended recognizable distance compared to method (1).

[0076] Referring to FIG. 14, the effects of methods (1) to (3) are compared. The upper graph in FIG. 14 shows the result of measuring the relationship between the speed [m / s] of the cart 21 and the recognition success rate [%] of the two-dimensional code 22 (ArUco marker) when the distance between the surveillance camera 12 and the cart 21 is 1 m. The lower graph in FIG. 14 shows the result of measuring the relationship between the speed [m / s] of the cart 21 and the recognition success rate [%] of the two-dimensional code 22 (ArUco marker) when the distance between the surveillance camera 12 and the cart 21 is 2 m. In method (1), that is, the method of inputting image data into OpenCV without performing any of the blur removal process, the detection process and the extraction process of the two-dimensional code 22, and the preprocessing, it was found that the recognition success rate was less than 40% even at a distance of 1 m and a speed of 0.2 [m / s]. On the other hand, when blur removal is performed as in methods (2) and (3), it was found that the recognition success rate is nearly 100% at a distance of 1 m and a speed of 0.2 [m / s]. It was also found that the recognition success rate is around 70% even at a distance of 2 m and a speed of 0.2 [m / s]. At a distance of 1 m, the result was that the recognition success rate was higher when the blur removal process was applied to the entire image as in method (2) than when the blur removal process was applied after extraction as in method (3). On the other hand, at a distance of 2 m, conversely, method (3) had a higher recognition success rate. This is presumably due to the characteristics of Delblur-GAN-v2 used in the blur removal process. From this result, it can be seen that the blur removal process may be performed on the entire image depending on the situation, or may be performed after the detection and extraction processes of the ArUco marker are executed.

[0077] Referring to Fig. 15, the effects of methods (4) to (6) will be described. The upper graph in Fig. 15 shows the relationship between the speed [m / s] of the cart 21 and the recognition success rate [%] of the two-dimensional code 22 (ArUco marker) when the distance between the monitoring camera 12 and the cart 21 is 1 m, for each of methods (4) to (6), as a result of measurement. The lower graph in Fig. 15 shows the relationship between the speed [m / s] of the cart 21 and the recognition success rate [%] of the two-dimensional code 22 (ArUco marker) when the distance between the monitoring camera 12 and the cart 21 is 2 m, for each of methods (4) to (6), as a result of measurement. The measurement results of method (1) are also shown together.

[0078] In method (4), that is, the method of inputting image data into OpenCV by performing the detection / extraction process of the two-dimensional code 22 and the pre-processing without performing the blur removal process, it was found that the recognition success rate was less than 50% even at a distance of 1 m and a speed of 0.2 [m / s]. On the other hand, when the blur removal process is performed before or after the detection / extraction process of the ArUco marker, as in methods (5) and (6), it was found that the recognition success rate was nearly 100% at a distance of 1 m and a speed of 0.2 [m / s]. It was also found that the recognition success rate was around 80% even at a distance of 2 m and a speed of 0.2 [m / s]. Thus, it can be seen that the recognition success rate is improved by performing the blur removal process before and after the detection and extraction process of the ArUco marker.

[0079] Next, a method for learning ArUco markers to generate learning data will be described. As described above, there are 1000 types of ArUco markers even with only existing markers, but it is difficult to have all of them learned by the analysis server 16, and if all are to be learned, the management cost will increase. Therefore, in the system of the second embodiment, a configuration is adopted in which only some of the 1000 types of ArUco markers are learned as learning data. For example, out of the 1000 types of ArUco markers, about 1 to 20 types can be learned as learning data. In this way, even with the learning of some ArUco markers, it is possible to execute the identification of ArUco markers in the analysis server 16. The determination of which type of ArUco marker among the 1000 types can be executed by referring to the OpenCV library.

[0080] Figures 16 and 17 show the experimental results of the marker detection operation using less learning data. Figure 16 shows 20 types of ArUco markers prepared for learning and 4 types of ArUco markers for testing. Figure 17 shows the relationship between the number of types of markers learned and the detection success rate. In the example of Figure 16, 20 types with ID = 00~015, 20, 30, 40, 50 were used as learning data, but this is just an example. The ArUco markers for testing include ArUco markers not included in the learning data (ID = 98, 99). In the detection of the test data, the aforementioned SiamMask was used. Also, the resolution was set to 1280×720, and the ArUco markers were appropriately arranged at a distance of 0.3~3m, and imaging was performed by the surveillance cameras 12A~12C to conduct the experiment.

[0081] As shown in the graph of Figure 17, it was found that even when about 1 to 20 types of ArUco markers are used as learning data, the final detection success rate can generally reach 90% or more.

[0082] As described above, according to the monitoring system of the present embodiment, by performing blur removal processing on the obtained image of the two-dimensional code 22 and then referring to a library such as OpenCV, it becomes possible to detect a two-dimensional code such as an ArUco marker with high accuracy. Therefore, according to this system, it is possible to provide a monitoring system that can accurately grasp the movements of people and objects, accurately monitor the manufacturing process, and contribute to the improvement of work efficiency.

[0083] The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, for a part of the configuration of each embodiment, it is possible to add, delete, or replace other configurations.

Explanation of Reference Numerals

[0084] 1, 1A... Monitoring system, 11... Human monitoring camera, 12, 12A to 12C... Object monitoring cameras, 13... Equipment monitoring camera, 14... Transfer computer (PC), 15... Collection / accumulation server, 16... Analysis server, 17... Display computer (PC), 21... Cart, 22... Two-dimensional code, 31... Target image extraction unit, 32... Distance learning data generation / acquisition unit, 33... Step determination unit.

Claims

1. In a monitoring system for monitoring humans, objects, or devices at a manufacturing site, a camera for monitoring the human, object, or device, an analysis unit for analyzing the image data obtained by the camera, and a display unit for displaying the analysis result by the analysis unit are provided, the analysis unit includes change point detection means for detecting change points in the image data and analyzing the state of the human, object, or device based on the change points, the analysis includes, after acquiring the image data, a step of removing blurring of the image data, a step of referring to an image included in the image data in a library and recognizing the type of the image and includes, the step of removing blurring of the image data is executed after or at least either before executing the process of detecting and cutting out an image of a recognition target included in the image data by deep learning, when the distance from the camera to the human, object, or device is a first distance, the step of removing blurring of the image data is executed for the entire image data without executing the cutting-out process, when the distance from the camera to the human, object, or device is a second distance different from the first distance, the step of removing blurring of the image data is executed after executing the cutting-out process, A monitoring system.

2. The analysis unit, includes operation time point detection means for specifying a time point at which a significant operation of the human, the object, or the device has occurred from time data of a plurality of the change points The monitoring system according to claim 1.

3. The monitoring system according to claim 2, wherein the analysis unit further includes a margin location determination unit that determines a margin location as a margin for shortening the lead time in each process based on the detection result of the operation timing detection means.

4. The monitoring system according to claim 1, wherein the deep learning is executed for some of the plurality of types of the recognition target and is used as learning data.

Images