surveillance system

Abstract

To provide a monitoring system with which it is possible to accurately grasp the motion of humans or things and accurately monitor manufacturing processes, thus contributing to improving work efficiency.SOLUTION: This monitoring system comprises a camera for monitoring things, an analysis unit for analyzing video data obtained by the camera, and a display unit for displaying a result of analysis performed by the analysis unit. The analysis unit is constituted so as to detect a change point in the video data and analyze the state of a human or a thing or a device on the basis of the change point, and is constituted so as to divide a thing manufacturing process into a plurality of steps, apply distance learning to the video data having been acquired for each of the plurality of steps so as to acquire feature quantity spatial data, and compare the feature quantity spatial data with the video data to be assessed so as to determine a step for the video data to be assessed.SELECTED DRAWING: Figure 9A

Description

[Technical Field]

[0001] The present invention relates to a monitoring system for monitoring people, things, and machines at a manufacturing site. [Background technology]

[0002] Systems are known that monitor people involved in manufacturing, the products being manufactured, and the machines that carry out the manufacturing process (manufacturing equipment, inspection equipment, packaging equipment, etc.) at manufacturing sites, and further improvements to these systems are being made (see, for example, Patent Documents 1 to 3).

[0003] For example, in manufacturing sites in the assembly industry, multiple parts are aggregated and combined to produce a product through operations (e.g., welding, cutting, assembly, etc.) according to prescribed assembly drawings and work procedures. In the case of complex products, multiple work sites perform their respective tasks, and each unit is combined to create a more complex product. In such operations, when a large number of parts (items) are aggregated, the work procedures, site layout, and part aggregation process become complicated. This can lead to issues such as waiting for tasks, parts backlogs, mistimed part aggregation, and worker placement errors, resulting in reduced assembly efficiency. However, particularly in the assembly industry, where equipment and machinery are rarely used and assembly work is primarily performed by humans, a sensor-based logging environment is not yet in place. For this reason, there is a demand for monitoring systems that can accurately grasp and monitor the status of people and items at manufacturing sites. In other words, there is a need to build a digital twin of the assembly industry and utilize it to improve and monitor management figures.

[0004] However, when quantifying and recording a person's work status, the worker must stop what they are doing and record the work status, or another worker must observe the worker's movements and record them, which is time-consuming and costly.

[0005] Furthermore, with regard to monitoring the status of an object, when the object is inside a machine, it is possible to record and monitor the machine's operations, but when an object is moved from a machine at one work site to a machine at another work site, its status cannot be adequately monitored. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 5416322 [Patent Document 2] Japanese Patent Application Publication No. 2019-139570 [Patent Document 3] Japanese Patent Application Publication No. 2019-16226 Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention provides a monitoring system that can accurately grasp the movements of people and things, accurately monitor the manufacturing process, and contribute to improving work efficiency. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention provides a monitoring system for monitoring people, objects, or equipment at a manufacturing site, comprising: a camera for monitoring the object; an analysis unit for analyzing video data acquired by the camera; and a display unit for displaying the analysis results by the analysis unit. The analysis unit is configured to detect change points in the video data and analyze the state of the person, object, or equipment based on the change points, divide the manufacturing process of the object into a plurality of steps, apply metric learning to the video data acquired for each of the plurality of steps to obtain feature space data, and compare the feature space data with video data to be evaluated to determine the step of the video data to be evaluated. [Effects of the Invention]

[0009] According to the monitoring system of the present invention, it is possible to provide a monitoring system that can accurately monitor the manufacturing process and contribute to improving work efficiency. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a monitoring system 1 according to a first embodiment. [Figure 2] 1 is a schematic diagram for explaining the operation of the people monitoring camera 11. FIG. [Figure 3] 1 is a schematic diagram for explaining the operation of the object monitoring camera 12 and the device monitoring camera 13. FIG. [Figure 4] 1 is a schematic diagram for explaining the operation of the object monitoring camera 12 and the device monitoring camera 13. FIG. [Figure 5] 4 is a flowchart illustrating the operation of the monitoring system 1A according to the embodiment. [Figure 6] 10 is an example of a display screen on the display PC 17. [Figure 7] 10 is an example of a display screen on the display PC 17. [Figure 8] 10 is an example of a display screen on the display PC 17. [Figure 9A] FIG. 10 is a schematic diagram showing an example of the configuration of a monitoring system 1 according to a second embodiment. [Figure 9B] An example of various image processing and processing of video data on the transfer PC is shown below. [Figure 10] An example of division of the assembly process of a desktop PC will be described. [Figure 11] FIG. 10 is a conceptual diagram illustrating an overview of distance learning in the second embodiment. [Figure 12] FIG. 10 is a conceptual diagram illustrating an overview of distance learning in the second embodiment. [Figure 13] FIG. 10 is a conceptual diagram illustrating an overview of distance learning in the second embodiment. [Figure 14] FIG. 10 is a conceptual diagram illustrating an overview of distance learning in the second embodiment. [Figure 15]FIG. 10 is a conceptual diagram illustrating a step determination method in the second embodiment. [Figure 16] The adjustment of the margin m in the third embodiment will be described. [Figure 17] 10 is a graph illustrating the effect of adjusting the margin m. [Figure 18] 10 is a graph illustrating the effect of adjusting the margin m. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0012] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to implement the present disclosure, it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0013] The following describes an embodiment of the process for assembling products (final products and units as intermediate products) composed of multiple parts. Parts ordered and delivered from the core system or ordering system are brought in and stored in a parts warehouse. Meanwhile, when a manufacturing production number is issued in accordance with the product's manufacturing plan, the parts are transported to the assembly shop in accordance with the manufacturing plan (including assembly drawings and assembly procedures created by the design department) predefined for each production number. In this assembly shop, assembly personnel and necessary equipment and jigs are allocated in accordance with the manufacturing plan. Assembly personnel perform the specified assembly work using the necessary equipment and jigs depending on the case, in accordance with the manufacturing plan.

[0014] After the required assembly work is completed, the product is transported to the next assembly shop. This process is repeated and integrated to complete the final product, which undergoes the required shipping inspection and is then stored in a shipping warehouse. This embodiment provides a means and method for recording the assembly process and work progress as they occur within a factory. Specifically, the identification information of each part, each piece of equipment / jig, each worker, the start / end time, and the take-off / removal time are recorded for each warehouse, serving shelf, and assembly shop. In conventional technology, in order to detect the presence or absence of product parts, sensors had to be individually installed for each work, each shop, and each shop configuration. This required the installation location of the sensor to be changed whenever the work content or the size and shape of the part changed, for example, due to a product change. According to the embodiment described below, the installation location of the camera does not need to be changed as long as the part, jig, or worker is within the camera's imaging range. This significantly reduces the installation cost when introducing the system.

[0015] [First embodiment] An example of a monitoring system 1 according to a first embodiment will be described with reference to Fig. 1. This monitoring system 1 may be composed of, for example, a people monitoring camera 11, an object monitoring camera 12, a device monitoring camera 13, a transfer computer (PC) 14, a collection and storage server 15, an analysis server 16, and a display computer (PC) 17.

[0016] People monitoring camera 11 is a camera for monitoring the movements of people (workers) performing work at a manufacturing site. People monitoring camera 11 stores information about, for example, people's shapes and movement characteristics in advance, and can be configured to quickly and reliably detect human movements.

[0017] The object monitoring camera 12 is a camera for monitoring the movement of objects manufactured and moving at a manufacturing site. For example, the object monitoring camera 12 may be a camera that captures and identifies identification information (one-dimensional code, two-dimensional code, color chart, etc.) attached to a cart to monitor the movement of the cart on which the object is placed and transported. By attaching different identification information to each of multiple carts, the type of cart can be identified. However, this is merely an example, and the object monitoring camera 12 is not limited to monitoring the movement of carts using identification information. For example, the shape of the object itself may be recognized, or various identification information attached to the object may be identified. Furthermore, instead of recognizing carts, the object monitoring camera 12 may monitor the movement of trays placed on a belt conveyor, for example. If the object monitoring camera 12 is configured to read two-dimensional codes, the object monitoring camera 12 may be equipped with an application for analyzing two-dimensional codes. Of course, the application for analyzing two-dimensional codes may also be stored on the cloud.

[0018] The equipment monitoring camera 13 is a camera for monitoring the operation of various devices (manufacturing devices, inspection devices, packaging devices, etc.) installed at the manufacturing site. The equipment monitoring camera 13 can be different depending on the characteristics of the device being imaged. For example, an equipment monitoring camera 13 for a device that requires human operation can be a camera that images the device's operation panel and monitors the human operation. By imaging the operation panel, it is possible to determine whether the device is operating or stopped.

[0019] Today's facilities and equipment can detect parameters that indicate the state of the equipment using sensors installed inside the equipment. However, if the equipment does not have these sensors, it is necessary to modify the equipment to install these sensors. However, skilled workers who work with the equipment can monitor the process stage of the equipment (machining, assembly, or other processing) by visual inspection and listening to the sounds, and can also monitor the state of the equipment to see if there are any abnormalities. Therefore, the status of the equipment can also be monitored by utilizing TV cameras. Introducing monitoring using TV cameras can eliminate the need for expensive monitoring systems, such as modifications to install sensors and monitoring by skilled workers.

[0020] In the case of an automatic device that does not require human operation, the device monitoring camera 13 for that device can be a camera for monitoring, for example, the input port for items (components) or the movement of moving parts (for example, the blade of a cutting device). Furthermore, the device monitoring camera 13 can also be used to inspect the content of various operations within the device. For example, in a device for connecting harnesses, a harness connection confirmation inspection can be performed, and from the images, it can be determined whether the connections are correct or not, and if not, errors such as incorrect wiring can be detected.

[0021] In this embodiment, cameras 11 to 13 may be provided with an inspection object detection means for detecting an inspection object and a defect detection means for detecting defects, either within the cameras 11 to 13 or in an external controller. The inspection object detection means has a function of detecting the inspection object portion in an image of a non-defective product. As an example, the inspection object detection means may be configured with a deep learning network that has previously learned the inspection object portion. By providing this means, it is possible to avoid falsely detecting as defective a portion that is significantly deformed but not considered abnormal (for example, a component such as a cable).

[0022] Then, by inputting the part detected as the part to be inspected into defect detection means, defects in the object to be inspected can be detected. As an example, this means can be configured by having an autoencoder learn multiple normal parts (images of normal products) in advance. By creating a difference image between the image reconstructed by the autoencoder and the input image, if there is an abnormal part, the image signal at that position will have a large value and will be detected as an abnormality through threshold processing. Detected defect information (defect images, product information, etc.) can be stored in defect information storage means.

[0023] When a new detection target is provided, the normal image of the new detection target can be trained by the inspection target detection means and the defect detection means (autoencoder). This training may be performed using 3D CAD information generated during product design.

[0024] Furthermore, the information (network parameters) learned for a new detection target can be reflected by performing a weighted average with the corresponding parameters of an already trained network. This method also accommodates cases where users do not want to upload information about new detection targets to the cloud. In other words, by sending the trained network parameters shared on the cloud to the user's location and performing a weighted average at that location, it is possible to configure and provide a highly accurate defect detection method and inspection target detection method specific to the user.

[0025] The device monitoring camera 13 can use various lenses (ultra close-up lens, fisheye lens, wide-angle lens, telephoto lens, etc.) depending on the characteristics of the object to be imaged and the distance to the object.

[0026] The progress of assembly work at the assembly process site can be identified and recorded by visual confirmation by an expert or manager with specific skills. From this perspective, this embodiment is configured to calculate the takeoff and landing times to be recorded by performing appropriate data processing, as described below, only on the image information captured by a TV camera using visible light.

[0027] Furthermore, when assembly work is completed at each assembly shop during the assembly process, not only is the completion time recorded, but the start and completion times for each assembly process, as well as any defects in the intermediate products that have been assembled, are also detected and recorded. The sampling rate of this TV camera does not need to be the usual 30 fps or 60 fps, but can be thinned to values ​​such as 1 fps or 0.2 fps, as necessary. Conversely, high-speed imaging such as 120 fps or 2000 fps is also acceptable. The former has the effect of reducing the total amount of data stored, while the latter is necessary when collecting data involving high-speed movement.

[0028] The transfer PC 14 has the function of reading video data (which may include both video and still images) captured by the various cameras 11-13 and transferring it to the collection and storage server 15. The transfer PC 14 can also be configured to perform data compression using a well-known method prior to the transfer operation. In addition to the data compression operation, it can also perform image processing and manipulation (for example, image processing such as leaving only the image portion of the person or object being monitored and deleting unnecessary images, or image processing such as increasing the resolution of the image of the monitored object and decreasing the resolution of the image of other objects). The transfer PC 14 may be connected to the collection and storage server 15 wirelessly or via a wired connection.

[0029] The collection and storage server 15 is a computer that collects and stores in a storage unit (not shown) the video data transferred from the transfer PC 14. The analysis of the collected video data is executed by the analysis server 16, but it is also possible for the collection and storage server 15 to be responsible for part of the analysis process, for example, part of the analysis of the video data.

[0030] The analysis server 16 analyzes the various types of video data stored in the collection and storage server 15, analyzes the movements of people, objects, and machines, and generates analysis data as the analysis results. The generated analysis data is output to the display PC 17 and displayed in a visualized form that can be recognized by an operator. The analysis server 16 can also be configured to compare the analysis data as the result of analyzing the movements of people, objects, and machines with the work plan data and analyze the differences.

[0031] The analysis items in the analysis server 16 may include, but are not limited to, items that a person (e.g., a site supervisor) may discover in the course of daily supervisory duties. For example, a signal A contained in video data acquired and accumulated (learned) over a long period of time in the collection and storage server 15 may be compared with a signal A' contained in newly acquired video data, and the comparison result may be used as the analysis item. If there is a difference between the signals A and A' thus acquired that is equal to or greater than a predetermined value, the analysis server 16 identifies the location where this difference occurred as a "change point" that indicates a significant change in the behavior of a person, object, or machine.

[0032] Identifying such change points makes it possible to detect when significant behaviors of people, objects, and machines occur that are often overlooked by human monitoring, and can serve as an indicator of factors impeding work efficiency in the manufacturing process. Furthermore, with human monitoring, it is necessary to separate data showing correct behavior from data showing abnormal behavior. While correct data is easy to prepare, preparing incorrect data is time-consuming. This system eliminates the need to prepare incorrect data, allowing for easy system implementation in a short period of time. The analysis server 16 links the time and location of such change points and mathematically defines them as "work time at a specific work location."

[0033] Generally, just having people and things in a specific place does not necessarily mean that work is progressing properly. However, by comparing and analyzing the work time data obtained as described above, it is possible to, for example, (a) If the work currently being done deviates from the average work time, (b) When there is a large variation in work time between multiple tasks, (c) When people or things are staying outside of a specific work location These can be detected "mechanically." These are different from the "work time" that humans define or recognize, but they can detect factors that hinder manufacturing lead time. That is, (a) is a hindrance caused by "deviating from the average time," while (b) is a hindrance caused by "large variations" in the work itself. (c) is a hindrance caused by "waiting for work" and "backlog of goods."

[0034] The operation of the human monitoring camera 11 will be explained briefly with reference to FIG. 2. FIG. 2 shows a manufacturing site where multiple machines ((1) to (9)) are arranged, and how a human (worker) H moves along the arrows, captured by the human monitoring camera 11. As an example, the human monitoring camera 11 can store information on the physique of the human H, the characteristics (color, shape, etc.) of the clothes usually worn by the human H, and movement characteristics (average walking speed, stride length). The human monitoring camera 11 can identify the image of the human H included in the video data by comparing this information with the video data. The identified image of the human H is enclosed by a bounding box BB, and the video data can be recorded in a form that includes the bounding box BB.

[0035] In this way, when video data is obtained by the human monitoring camera 11, the video data is analyzed in the analysis server 16, and it is possible to analyze which tasks the person H performed at which time periods (stay time), and which time periods the person H did not perform the task but moved around (movement time), or was in a position outside the capture range, according to the identified change points in the person's behavior. The change points in the person's behavior can be identified according to the time the person arrived in front of a specific machine, the time they left, and other information. The bar display in the lower right of Figure 2 indicates the task times (including start and end times) for tasks (4) to (8). Such bar displays can also be displayed on the display screen of the display PC 17.

[0036] In this embodiment, the worker detection means is configured by the people monitoring camera 11 or an external controller. The people monitoring camera 11 as the worker detection means detects workers in captured images and may include a deep learning network that has previously learned images of workers. This worker detection means detects workers in captured images. From the detected position, it is determined whether the worker is present at a predetermined position, i.e., the time of entry and exit from the work area. In this embodiment, captured images are acquired from multiple directions, and the worker's location (viewpoint direction) in each 2D image is determined. The intersection of these directions is detected as the worker's location in 3D space. This method of detecting the worker's location from multiple images is effective in cases where high depth detection accuracy is desired. Furthermore, using multiple images also has the effect of enabling detection in cases where the worker is hidden in the image due to being shadowed by equipment or other objects.

[0037] The operation of the object monitoring camera 12 and the machine monitoring camera 13 will be explained briefly with reference to Fig. 3. Fig. 2 illustrates an example in which three object monitoring cameras 12 (12A to 12C) and three machine monitoring cameras 13 (13A to 13C) are arranged, but this is merely an example, and the numbers of object monitoring cameras 12 and machine monitoring cameras 13 are not limited to a specific number. Furthermore, the object monitoring camera 12 is not limited to a specific type, but can preferably be a camera that photographs a two-dimensional code 22 attached to a cart 21 on which an object (part, etc.) is placed, as shown schematically in Fig. 3.

[0038] The cart 21 moves between a plurality of machines (11) to (13), as shown in Fig. 3. For example, when the cart 21 completes a process preceding the machine (11) and moves to the front of the machine (11), the two-dimensional code 22 attached to the cart 21 is captured by the object monitoring camera 12A. This allows the arrival of the cart 21 at the machine (11) to be detected.

[0039] When work on machine 11 is finished and cart 21 leaves machine 11 and moves toward the next machine 12, object monitoring camera 12A detects that the two-dimensional code 22 is no longer captured, thereby detecting that cart 21 has left machine 11. When cart 21 arrives in front of machine 12, object monitoring camera 12B captures an image of the two-dimensional code 22 attached to cart 21. This detects that cart 21 has arrived at machine 12. By repeating this operation, the movement of cart 21, i.e., the object (parts, etc.), can be detected based on the video data from object monitoring camera 12.

[0040] Change points in the behavior of an object can be identified by the start and end times of a certain machine (see Figure 4). The travel time (waiting time) from a higher-level machine (process) to a lower-level machine (process) can be calculated by the difference between the departure time from the higher-level machine and the arrival time at the lower-level machine. Change points in machine behavior can also be identified by the start and end times of a certain machine's specific operation.

[0041] Furthermore, when parts are placed on the cart 21 from the serving shelf, the intermediate part products placed on the cart 21 are assigned a correspondence relationship with the cart number, part number, and product serial number linked to the part number. The correspondence relationship is also assigned as information to the two-dimensional code 22, and by reading the two-dimensional code 22, this correspondence information is automatically collected. This series of automatic collection links the cart 21, part, worker, shop, and change time. This linking is also performed between other shops, other products, and other workers. As a result, the locations of parts, workers, facilities, equipment, and jigs at a specific time are recognized, and the vector data is stored in the storage means. Furthermore, the detection of the time of a status change does not necessarily have to be based on images captured by a TV camera, but can also be combined with information from other sensors, already installed sensors, etc.

[0042] The operation of the monitoring system 1 according to the embodiment will be described with reference to the flowchart in Figure 5. In this system, the people monitoring camera 11, the object monitoring camera 12, and the machine monitoring camera 13 each operate independently to monitor the operations of people, objects, and machines. The obtained video data is transferred to and read by the transfer PC 14, which then performs a predetermined data compression operation on the video data and transfers the video data to the collection and storage server 15. The collection and storage server 15 receives and stores the transferred data. The data stored in the collection and storage server 15 is sent to the analysis server 16 as appropriate and is used for analysis.

[0043] Based on the stored data, the analysis server 16 identifies the change points of people, things, and machines as described above, identifies the time when the change points occurred, and further identifies the location where the change points occurred, and stores this information in a storage unit (not shown). By identifying multiple change points, it is possible to identify the start time and end time of significant actions of people and each machine of things.

[0044] By identifying change points and the time and place where the change points occurred, the status of people and things in each process can be determined, which not only determines the time required for each process but also determines which of multiple processes has room for shortening the lead time (room for margin). In other words, the analysis server 16 can function as a room for margin determination means for determining room for margin in each process based on the identified change points and data at the time when significant actions occurred. Room for margin can be visually displayed on the display PC 17.

[0045] FIG. 6 is an example of a display screen when the analysis results in the analysis server 16 are displayed on the display of the display PC 17. In the display example of FIG. 6, the horizontal axis is the time axis, and it is a Gantt chart in which the timeline of the manufacturing plan and the timeline of each process (processes 1 to 3) in the actual work (performance) are compared by bar display for each different production number. This display visualizes the difference between the manufacturing plan and the actual work on each work day. This screen display visualizes which production number is in which process, and the progress of each sub-process within a large process that takes place over several days can be grasped.

[0046] As shown here, the monitoring system of this embodiment is highly effective in monitoring the progress of a manufacturing plan. A delay in the progress of a manufacturing plan leads to increased manufacturing costs due to increased labor costs, equipment depreciation, and other factors. This is why progress management is so effective. Furthermore, when a significant delay occurs, appropriate manufacturing output can be obtained by reassessing the manufacturing plan (manufacturing schedule). This is why rescheduling of the manufacturing plan is necessary. In this case, the magnitude of the delay can be preset, for example, a 20% or 30% increase from the initial lead time. Alternatively, the takt time of a specific process that is causing a bottleneck can be preset, for example, a 20% or 30% increase. In other words, the monitoring system of this embodiment plays an important role in determining when rescheduling is necessary. In other words, using the monitoring system of this embodiment in combination with a scheduler can produce even greater benefits. In this case, a certain degree of effectiveness can be expected whether the scheduler being tried out is an automatic stacking scheduler or a scheduler equipped with some kind of optimization algorithm.

[0047] FIG. 7 is another example of a display screen when the analysis results from the analysis server 16 are displayed on the display of a display PC. In the display example at the top of FIG. 7, the horizontal axis represents elapsed time, and the breakdown of the work time for each process (processes 1 to 3) for each different production number is displayed as the length of the bar. This display visualizes the variation in the total work time for each production number, as well as the variation in the time required for each process. This makes it easy to identify production numbers that require long work times and to investigate the causes.

[0048] In the example display at the bottom of Fig. 7, the horizontal axis represents elapsed time, but the travel time between different processes for each of multiple carts is displayed as the length of the bar. This display makes it possible to visualize which carts and which processes have long travel times, making it easy to identify the causes.

[0049] Figure 8 is a graph of the operating time and non-operating time for each process (processes 1 to 3). When focusing on a single process, the higher the ratio of operating time to the total (operating rate), the higher the work efficiency. However, if a process has a high operating rate while an adjacent process has a low operating rate, the work efficiency of the entire process cannot be improved. When there is a difference in operating rate between adjacent processes in this way, it can be determined that a bottleneck has occurred in the work. When a bottleneck occurs, work efficiency can be improved by changing the manufacturing plan, including personnel allocation.

[0050] In the embodiment described above, for security reasons, there may be cases where the user wants to avoid uploading on-site information (information on the processing of image data after acquisition in this embodiment, time detection of changes in the status of people, objects, and equipment, and the results of subsequent data processing) to the cloud. In this case, the time of change, product information, and other information can be encoded and decoded in advance using a password set by the user. This allows this information to be treated as text information whose content has meaning to different people within the site, and as information that has no meaning without a password on the cloud.

[0051] Furthermore, by identifying data on the cloud using a hash function before and after use, it is possible to ensure that the data on the cloud has not been tampered with, thereby ensuring the security of user data. This method may also utilize blockchain technology, which adds a previous hash value to stored data to generate the next hash value, thereby making it virtually impossible to tamper with the data, thereby further improving security.

[0052] Furthermore, in systems that use images from TV cameras, workers may be resistant to being photographed or may be concerned about their personal information being leaked. To address this issue, a controller or other device can be provided with a means for converting captured images into information that cannot be interpreted by the human eye, such as by applying image transformations such as Fourier transforms, at an early stage. By using such converted images in subsequent learning and other processes, similar image transformations can be performed in subsequent processes, such as worker detection.

[0053] By acquiring various time data, including the detected work completion time, for each product, the lead time from start to finish of the product can be analyzed as the work time for each work. Also, for example, at the end of a day's work, a report can be created that compiles the progress of work, the placement positions of parts, etc.

[0054] A system and method for utilizing various time data such as take-off and arrival times and start and completion times detected in this way will be explained below. With this system, it is possible to describe the progress time of each task by focusing on a specific product. When the variance in the progress time is calculated for a specific product or a specific shop, if the variance is large due to the worker or other disturbances, it can be determined that the task is not mature. In such a case, for example, · Change the worker for a specific task to a member with higher work skills Analyze the causes of variation and prepare jigs to separate the work Countermeasures such as the following can be taken.

[0055] In the above case, by monitoring the variability in start-to-completion times, it is possible to know in real time whether the quality of work is deteriorating.

[0056] Furthermore, by preparing an environment on the cloud that includes the monitoring system, production planning system, and display system shown in Figures 6 to 8 of this embodiment, the following new value can be created. For example, when an algorithm is developed that can detect human change points with greater accuracy and higher time resolution, the value of the entire system can be improved by simply changing this algorithm. For example, because a process can be broken down into smaller parts when it is changed, changing the combination of these parts can further improve the accuracy of lead time reduction through process redesign. In this way, having a monitoring system, production planning system, or display system allows new technologies and programs to be easily and quickly applied to the workplace. In other words, the present invention has the aspect of providing a "platform" for improving the efficiency of the production site.

[0057] Furthermore, by counting the usage time of each technology and program applied on-site, it is possible to measure the total usage time of those technologies. The system provider's compensation can be automatically calculated based on this usage time. This is the mechanization of administrative processes, but even in manufacturing sites, the mechanization and efficiency of such administrative processes is important in terms of cost reduction. Furthermore, by having this type of usage time measurement system and paying compensation according to usage time, one-time cash payments can be reduced compared to introducing a system with a lump-sum payment, which is expected to have the effect of improving cash flow.

[0058] Furthermore, this monitoring system can detect and report deviations from the average work time and expected work completion time. In a remote monitoring system using a simple surveillance camera, the monitor must continuously monitor the images being captured, whereas the system can be expected to report only when deviations from the expected time occur.

[0059] [effect] As described above, according to the system of this embodiment, the movements of people, objects, and machines are monitored by cameras 11-13. Significant changes in the movements of people, objects, and machines are detected based on change points in the video data. The analysis server 16 then analyzes the results and visualizes them for display. Therefore, the movements of people, objects, and machines can be monitored visually by an operator, eliminating the need for time-consuming input and setting of conditions. This allows for accurate and low-cost monitoring of the manufacturing process, improving work efficiency. In other words, matters that are currently judged intuitively by a supervisor can be handled with the same level of accuracy by capturing images using cameras 11-13. As a result, this system allows for efficient monitoring of the manufacturing site without the need for a dedicated system engineer. Furthermore, visualization of the manufacturing site allows for visualization, which in turn enables one-stop solutions to business issues.

[0060] The cameras 11-13 used can be installed at different locations, with different types or numbers being increased as needed, making it easy to adapt to changes in the manufacturing site. Specifically, this system can record the status or results of work sites where it is difficult to record the status or results, such as human assembly work, human visual inspection, human transport of goods, and human machine operation. As a result, the status of a human work site can be recorded in the same way as a factory site composed of automated machines that has a mechanism for recording data indicating the status.

[0061] [Second embodiment] Next, a monitoring system according to a second embodiment will be described. An example of a monitoring system 1A according to the second embodiment will be described with reference to FIG. 9A. The second embodiment relates to a system that primarily monitors the assembly process of an object. However, it goes without saying that it is possible to appropriately combine the contents of the first embodiment to monitor people and equipment as well. As an example, this monitoring system 1A can be composed of monitoring cameras 11A to 13A, a transfer computer (PC) 14, a collection and storage server 15, an analysis server 16, and a display computer (PC) 17.

[0062] One example of the object monitored by the surveillance cameras 11A-13A is a desktop computer being manufactured (assembled) at a manufacturing site. The surveillance cameras 11A-13A are cameras for capturing images of the desktop computer (object) being assembled and monitoring the progress of the assembly process. Here, there are three surveillance cameras 11A-13A, and they can be installed, for example, behind, in front of, and on the ceiling of an assembly workbench. The illustrated example is merely an example, and the number of cameras and their installation positions are not limited to any particular ones. Furthermore, the surveillance cameras 11A-13A can use various lenses (ultra-close-up lenses, fisheye lenses, wide-angle lenses, telephoto lenses, etc.) depending on the characteristics of the object to be imaged and the distance to the object to be imaged.

[0063] The transfer PC 14 has the function of reading video data (which may include both video and still images) captured by the various cameras 11A-13A and transferring it to the collection and storage server 15. The transfer PC 14 may be configured to perform a data compression operation using a well-known method prior to the transfer operation. In addition to the data compression operation, it may also perform image processing (e.g., rotating, projecting, or color-changing the image of the monitored object, cropping unnecessary portions (see FIG. 9B), or image processing such as increasing the resolution of the monitored object image and decreasing the resolution of the images of other objects). The transfer PC 14 may be connected to the collection and storage server 15 wirelessly or via a wired connection.

[0064] The collection and storage server 15 is a computer that collects and stores in a storage unit (not shown) the video data transferred from the transfer PC 14. The analysis of the collected video data is executed by the analysis server 16, but it is also possible for the collection and storage server 15 to be responsible for part of the analysis process, for example, part of the analysis of the video data.

[0065] The analysis server 16 analyzes various types of video data accumulated in the collection and storage server 15, analyzes images of the desktop computer, which is the manufacturing target (object), and generates analysis data as the analysis results. The analysis server 16 is configured to detect change points in the video data and analyze the status of a person, object, or device based on the detected change points. The generated analysis data is output to the display PC 17 and displayed in a visualized form that can be recognized by an operator. The analysis server 16 can also be configured to compare the analysis data with work plan data and analyze the differences. The analysis items in the analysis server 16 include analyzing which of multiple assembly processes (from the start of assembly to completion) the desktop computer, which is the monitored object (object) of assembly work, has progressed to. Specifically, the analysis server 16 includes, for example, a target image extraction unit 31, a distance learning data generation / acquisition unit 32, and a step determination unit 33.

[0066] The target image extraction unit 31 has a function of extracting an image of a desktop computer that is the monitoring target from the data captured by the surveillance cameras 11A to 13A and stored in the collection and storage server 15. As an example, the target image extraction unit 31 can extract an image of the desktop computer that is the monitoring target by fine-tuning an object detection method such as Faster-RCNN.

[0067] The distance learning data generation / acquisition unit 32 has a function of generating or acquiring distance learning data (feature space data) to be used in the step determination unit 33. The step determination unit 33 has a function of using the feature space data to determine which of multiple steps the desktop computer to be monitored is currently in.

[0068] The distance learning data generation / acquisition unit 32 receives images of each manufacturing process of the desktop PC to be monitored and labeling data that is the name of the process. As will be described later, the distance learning data generation / acquisition unit 32 applies distance learning to the images and labeling data to generate feature space data. By applying distance learning, images of different steps can be sufficiently separated in feature space, thereby improving the accuracy of judgment by the step judgment unit 33. The distance learning data generation / acquisition unit 32 may generate distance learning data based on image data obtained from the monitoring cameras 11A to 13A installed in this system 1A, or may simply acquire distance learning data obtained from another system.

[0069] An outline of the determination operation in this first embodiment will be described below. As shown in Fig. 10, the desktop PC assembly starts with the preparation of the chassis (start of assembly: Step 0), followed by the installation of the power supply unit (Step 1), the motherboard (Step 2), the CPU (Step 3), the CPU fan (Step 4), the memory (Step 5), the GPU (Step 6), the case fan (Step 7), the hard disk drive (Step 8), and the cable connection (Step 9).

[0070] In this way, the assembly work of a desktop PC can be divided into a plurality of steps (for example, 10 steps) and grasped. According to the monitoring system of the second embodiment, it is possible to know which of Steps 0 to 9 the progress of the assembly work of a desktop PC in a certain assembly booth is at, based on the images captured by the monitoring cameras 11A to 13A and the analysis results of the images captured by the analysis server 16. In the above example, the number of steps (number of divisions) in the manufacturing process is 10, but this is merely an example and is not limited to this. Furthermore, the number of divisions does not need to be fixed and can be changed according to the situation.

[0071] The monitoring system of the second embodiment divides the desktop PC manufacturing process into multiple steps as described above. It is preferable to divide the steps by taking into consideration the work time for each step and the number of components to be installed in that step. The monitoring system 1 then determines which step the desktop PC is currently being assembled in. To achieve this determination, the system of the second embodiment collects image data of the desktop PC for each step and applies metric learning to the image data obtained for each step to generate metric learning data (feature space data). First, as shown in FIG. 11, multiple pieces of image data Di for each of steps 0 to 9 are acquired, and metric learning is performed using this data. The acquired image data Di may be either still images or video. In metric learning, the images for each step are input as positive data (target data), anchor data, and negative data, and metric learning is performed (see FIG. 12).

[0072] Specifically, for example, one image data Di is selected as positive data to be used for distance learning from the image data Di related to one of steps Step 0 to 9. At this time, one anchor data as reference data is selected from the multiple image data Di belonging to the same step, and one image data of a step different from the one to which the positive data belongs is selected as negative data.

[0073] As shown in Figure 13, this positive data, anchor data, and negative data are input into a convolutional neural network, and distance learning is performed using a method called triplet loss, which learns these three pieces of data as a set. In this triplet loss, since the positive data and anchor data belong to the same step (e.g., Step 1), feature space data is calculated so that they are located close to each other in the feature space (so that the distance is short). On the other hand, since the positive data and negative data belong to different steps, feature space data is calculated so that they are located farther apart in the feature space. In this way, feature space data is calculated so that data belonging to the same step are close to each other in the feature space, and so that data belonging to different steps are sufficiently far apart in the feature space.

[0074] Specifically, as shown in FIG. 14, the distance between the anchor data and the positive data is d p , the distance between the anchor data and the negative data is d n When the margin is m (here, m is a fixed value), the loss function L shown in the following [Equation 1] is triplet approaches 0 (distance d p and distance d n the difference between them is the margin m), and the distance d p and d n is optimized.

[0075]

number

[0076] In this way, feature space data of positive data and negative data is calculated. The above procedure is repeated for all image data. In this case, each image data is selected as anchor data once in one epoch. Note that the procedure for selecting positive data and negative data may be irregular (random).

[0077] Next, with reference to FIG. 15, the step determination based on the feature space data in the step determination unit 33 will be described. The obtained image data to be determined is input to a convolutional neural network to calculate its feature. Then, a nearest neighbor search (kNN) is used to determine which feature in the feature space data the feature most closely resembles. The step in the feature space data that corresponds to the feature closest to the feature of the determination data can be determined to be the step for the image data. If the distance between the feature of the determination data and the feature in the feature space data is equal to or greater than a predetermined value, it can be determined to be a determination error, and the determination of the determination data can be suspended.

[0078] As described above, the monitoring system of the second embodiment makes it possible to accurately determine the progress of assembly of an item (desktop computer, etc.) manufactured in an assembly process.

[0079] [Third embodiment] Next, a monitoring system 1A according to a third embodiment will be described with reference to FIGS. 16 to 18. The overall configuration of the monitoring system 1A according to the second embodiment may be the same as that of the first embodiment. The operation of the analysis server 16 is also basically the same. However, in the third embodiment, adaptive triplet loss, which appropriately changes the margin m, is applied in the metric learning data generation / acquisition unit 32, which differs from the second embodiment in this respect. Here, as in the second embodiment, it is assumed that the assembly process of a desktop PC is divided into multiple steps (e.g., 10 steps, Steps 0 to 9). Furthermore, when performing metric learning, images from each step are input as positive data (learning target data), anchor data, and negative data, and metric learning is performed, as in the second embodiment (see FIG. 12).

[0080] FIG. 16 is a graph showing an example of how to change the margin m ([Equation 1]) in adaptive triplet loss according to the third embodiment. The horizontal axis of the graph represents the label n of the step to which the data selected as negative data belongs. n The vertical axis is the margin m. The graph (a) on the right side of Figure 16 shows the label n of the step to which the data selected as positive data belongs. a If is 1 (n a The graph on the left (b) shows the change in the margin m when the data selected as positive data belongs to the label n a If n is 5, a = 5). As shown in graphs (a) and (b), the margin changes for the label n a For each different label n n The way in which the margin m changes with changes in n is different. a and n b If the values ​​are close, the margin m is set to a large value, and n a and n b The further away the value is, the smaller the margin m is. a The same applies when the margin m is a value other than 1 or 5. The graph in Fig. 16 follows a normal distribution, but it does not have to be a normal distribution, and other distributions may be used as long as the margin m changes appropriately.

[0081] When applying triplet loss, positive and negative data are selected from steps close to each other (e.g., adjacent steps), and n a and n b are close to each other, the positive data image and the negative data image are a and n bIn this case, when the margin m is set to a small value, the feature amounts of the positive data and the negative data are similar to each other, and as a result, the feature amounts of the images of adjacent steps are assigned values ​​that are similar to each other. This can cause erroneous determinations when identifying images of adjacent steps. In the first embodiment, if the margin m is set to a fixed value and an appropriate fixed value is set, appropriate distance learning can be performed overall. However, when m is a fixed value, the value of m may be too small in some steps (especially in steps where no significant changes occur in the image), and appropriate distance learning may not be performed.

[0082] Therefore, in the third embodiment, as shown in FIG. 16, a Depending on the value of n The margin m is variable for the change of the label n a and n b The closer the value is (n a and n b The margin m is controlled so that the value of m increases as the difference between the values ​​of σ and σ decreases. In other words, the closer the step to which the positive data belongs to and the step to which the negative data belongs, the larger the margin m is set to. As an example, the variance σ 2 The margin m can be calculated by multiplying the normal distribution formula f(x) ([Equation 2]) by a constant a. Figure 16 is a graph when a=5000 and σ=2.

[0083]

number

[0084] Fig. 17 is a graph showing the effect of the third embodiment. The upper graph in Fig. 17 shows the step determination accuracy (accuracy) for each number of distance learning epochs when the margin m is fixed. The lower graph shows the change in determination accuracy when the margin m is made variable according to the third embodiment. Overall, it can be seen that the determination accuracy is improved when the margin m is made variable.

[0085] FIG. 18 shows the effect of the third embodiment using a map in which feature space data (128 dimensions) is displayed two-dimensionally using T-SNE, a dimension reduction algorithm. The upper graph in FIG. 18 shows feature space data when the margin m is fixed. The lower graph shows feature space data when the margin m is made variable according to the third embodiment. In the upper graph, approximate values ​​are given as features for some data, while in the lower graph, the features of data from different steps are generally far apart, making it possible to improve the accuracy of step determination. Variance σ 2 By appropriately changing the constant a, a more appropriate margin m can be given, and the judgment accuracy can be further improved. 2 The change of the constant a may be performed by an operator at an input unit (not shown) of the analysis server 16, or may be performed automatically by the analysis server 16 according to the verification result of the accuracy of the step determination.

[0086] As described above, according to the third embodiment, the margin m is variable, which further optimizes the feature space data in distance learning and further improves the accuracy of step determination.

[0087] [others] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0088] 1, 1A... monitoring system, 11... human monitoring camera, 12... object monitoring camera, 11A to 13A... monitoring camera, 13... device monitoring camera, 14... transfer computer (PC), 15... collection and storage server, 16... analysis server, 17... display computer (PC), 21... cart, 22... two-dimensional code, 31... target image extraction unit, 32... distance learning data generation / acquisition unit, 33... step judgment unit.

Claims

[Claim 1] In a monitoring system for monitoring people, things, or equipment at a manufacturing site, a camera for monitoring the object; an analysis unit that analyzes the video data obtained by the camera; a display unit that displays the analysis results by the analysis unit; Equipped with the analysis unit is configured to detect a change point in the video data and analyze a state of the person, object, or device based on the change point; Dividing a manufacturing process of the product into a plurality of steps, and applying metric learning to video data obtained for each of the plurality of steps to acquire feature space data; The feature space data is compared with video data to be determined, and a step of the video data to be determined is determined; The analysis unit When the distance learning is performed, Selecting one piece of image data obtained at each step as positive data; Selecting data belonging to the same step as the positive data as anchor data; Selecting data belonging to a step different from the positive data as negative data; performing distance learning so that a distance in a feature space between the positive data and the anchor data is a first distance and a distance in the feature space between the positive data and the negative data is a second distance greater than the first distance; The analysis unit sets a margin, performing distance learning such that the difference between the first distance and the second distance is the margin; the analysis unit sets the margin to a larger value as the step to which the positive data belongs and the step to which the negative data belongs are closer to each other; The analysis unit sets, for each step to which the positive data belongs, a change amount of the margin relative to a change amount of the step to which the negative data belongs. A monitoring system characterized by:

Images