Production line monitoring method and monitoring system

Abstract

A production line monitoring method and a monitoring system thereof. The monitoring system is used to: obtain multiple images of an operator; determine the action type of the operator in the multiple images based on an image recognition model; determine the occurrence time and action cycle of the action type; and record the occurrence time and action cycle of the action type.

Description

Technical Field

[0001] The present invention relates to a production line monitoring method and a monitoring system thereof, and more specifically, to a production line monitoring method and a monitoring system thereof based on machine learning technology. Background Art

[0002] In the production process of traditional industrial products, the assembly of parts of various devices still requires manual assistance. Specifically, a single device often needs to be equipped with many parts, and the assembly of different parts on the device is usually completed manually by operators at various stations on the factory production line.

[0003] However, errors in manual operation or delays caused by various factors often cause bottlenecks in production line output. Therefore, monitoring devices are needed on the production line to record and determine the causes of the production line output bottleneck in order to improve subsequent efficiency.

[0004] However, conventional monitoring devices mostly have only the function of recording images. Therefore, when a situation occurs on a production line, it is generally necessary to manually search the images recording the production line and determine the factors of error or delay. Summary of the invention

[0005] Some embodiments of the present invention provide a production line monitoring method for a monitoring system, comprising: acquiring multiple images of an operator; judging the action type of the operator in the multiple images based on an image recognition model; determining the occurrence time and action cycle of the action type; and recording the action type, occurrence time and action cycle.

[0006] Some embodiments of the present invention provide a production line monitoring method for a monitoring system, comprising: acquiring a video, wherein the video includes multiple video clips; determining an action type of each video clip based on an image recognition model; receiving a user setting to change the action type of a first video clip of the multiple video clips; and adjusting the image recognition model according to the action type of the first video clip.

[0007] Some embodiments of the present invention provide a monitoring system for production line monitoring, comprising a processor and a storage unit. The storage unit stores a program and an image recognition model. When the program is executed, the processor: obtains multiple images of an operator; based on the image recognition model, determines the action type of the operator in the multiple images; determines the occurrence time and action cycle of the action type; and records the action type, occurrence time and action cycle.

[0008] The above has been a fairly extensive overview of the features and technical advantages of the present invention so that the following embodiments of the present invention may be better understood. Additional features and advantages of the present invention will be described below, and these features and advantages form the subject of the claims of the present invention. It will be appreciated by those skilled in the art that the disclosed concepts and specific embodiments may be easily used as a basis for modifying or designing other structures or processes for achieving the same purpose of the present invention. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the present invention as set forth in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] When read together with the accompanying drawings, the various aspects of the present invention can be best understood according to the following embodiments. It should be noted that various features are not drawn to scale according to standard practice in the industry. In fact, the size of various features can be arbitrarily increased or reduced for the sake of clarity of discussion.

[0010] A more complete understanding of the present invention may be derived by referring to the embodiments and claims when considered in conjunction with the accompanying drawings, wherein like reference numerals refer to similar elements throughout the several views.

[0011] Figure 1A is a block diagram of a monitoring system of some embodiments of the present invention.

[0012] Figure 1B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

[0013] Figure 2A is a block diagram of a monitoring system of some embodiments of the present invention.

[0014] Figure 2B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

[0015] Figures 2C to 2F is a schematic diagram of pictures captured by the image capturing device according to some embodiments of the present invention.

[0016] Figure 3A is a block diagram of a monitoring system of some embodiments of the present invention.

[0017] Figure 3B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

[0018] Figures 3C to 3F is a schematic diagram of pictures captured by the image capturing device according to some embodiments of the present invention.

[0019] Figures 4A to 4B is a flowchart of a production line monitoring method according to some embodiments of the present invention.

[0020] Figures 5A to 5G is a flowchart of a production line monitoring method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0021] Embodiments or examples of the invention shown in the accompanying drawings are now described using specific language. It should be understood that no limitation of the scope of the invention is intended herein. Any changes or modifications to the described embodiments and any further application of the principles described in this document are considered to occur generally to one of ordinary skill in the art to which the invention relates. Reference numerals may be repeated throughout the embodiments, but this does not necessarily mean that one or more features of one embodiment are applicable to another embodiment, even if such embodiments share the same reference numerals.

[0022] It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers or parts, such elements, components, regions, layers or parts are not limited by such terms. On the contrary, such terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the teachings of the present invention, the first element, component, region, layer or part discussed below may be referred to as the second element, component, region, layer or part.

[0023] The terms used herein are only for describing specific example embodiments and are not intended to limit the inventive concept. As used herein, the singular forms "a / an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that when used in this specification, the terms "comprises and comprising" indicate the presence of stated features, integers, steps, operations, elements or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups thereof.

[0024] Delays caused by manual errors or other factors on the production line often lead to bottlenecks in the production line output. However, conventional monitoring devices on the production line only record images, so it is still necessary to manually search for images to find the factors of errors or delays. This error detection method has very low efficiency and flexibility, and cannot effectively improve the bottleneck of the production line output. Therefore, in order to more quickly and accurately find the factors that cause errors or delays on the production line, and thus improve the efficiency of the production line output, innovative monitoring methods and monitoring systems are needed.

[0025] Please refer to Figure 1A, which is a block diagram of a monitoring system 1 of some embodiments of the present invention. The monitoring system 1 includes a processor 11 and a storage unit 13. The storage unit 13 stores a program 130 and an image recognition model 132. The image recognition model 132 may include a model related to machine learning technology. Specifically, the image recognition model 132 is mainly a machine learning model generated by using a plurality of training data according to a machine learning algorithm.

[0026] Specifically, in some embodiments, some image data and the action types actually corresponding to these image data can be used as training data to train the image recognition model 132 based on the machine learning algorithm (i.e., to generate the image recognition model 132). In this way, the image recognition model 132 can be used to receive image data and output the action type of the operator in the image. For example, after receiving the image sequence of the operator, the image recognition model 132 determines that the operator is performing the action of "picking up" or "putting down", and outputs the action type as "picking up" or "putting down".

[0027] The processor 11 and the storage unit 13 are electrically connected via a communication bus 17. Through the communication bus 17, the processor 11 can execute a program 130 stored in the storage unit 13. When the program 130 is executed, one or more interrupts, such as a software interrupt, can be generated to enable the processor 11 to execute the program 130 having a production line monitoring function. The function of the program 130 will be further described below.

[0028] Please refer to Figure 1B , which is a schematic diagram of the use of the monitoring system 1 of some embodiments of the present invention. In detail, when the operation of the production line machine 92 needs to be monitored and analyzed, the image capture device 91 can be installed in the environment where the production line machine 92 is located to capture images related to the production line machine 92. Among them, the monitoring system 1 can be connected to the image capture device 91 through a network connection (wired network or wireless network).

[0029] In some embodiments, when the operator 93 operates on the production line machine 92, the image capture device 91 can capture multiple images 910 of the operator 93 according to the position of the production line machine 92, and transmit the multiple images 910 to the monitoring system 1 through the network. In other words, the monitoring system 1 can obtain the multiple images 910 of the operator 93 from the image capture device 91 through the network.

[0030] Next, using the image recognition model 132 generated and stored in the storage unit 13, the processor 11 of the monitoring system 1 can determine the action type of the operator 93 of the plurality of images 910. Since the plurality of images 910 carry relevant information of timestamps, the processor 11 can determine the capture time of the plurality of images 910, and further determine the occurrence time and action cycle of the action type represented by the plurality of images 910. The processor 11 can record the action type, occurrence time and action cycle in the storage unit 13 for subsequent use.

[0031] Please refer to Figure 2A , which is a block diagram of a monitoring system 2 of some embodiments of the present invention. The monitoring system 2 includes a processor 21, a storage unit 23, and an input device 25. The storage unit 23 stores a program 230, an image recognition model 232, and training data 234. The image recognition model 232 may include a model related to machine learning technology, which is used to receive video data (i.e., image sequence data) and output the action type of the operator in the video.

[0032] The processor 21, the storage unit 23, and the input device 25 are electrically connected via the communication bus 27. Through the communication bus 27, the processor 21 can execute the program 230 stored in the storage unit 23. When the program 230 is executed, one or more interrupts, such as software interrupts, can be generated to enable the processor 21 to execute the program 230 with the production line monitoring function. The function of the program 230 will be further described below.

[0033] In some embodiments, the image recognition model 232 is mainly a machine learning model generated based on a machine learning algorithm using a plurality of training data 234. Specifically, some video data and the action types actually corresponding to these video data can be used as training data to train the image recognition model 232 based on the machine learning algorithm (i.e., generate the image recognition model 232).

[0034] In more detail, each training data 234 may include: (1) video data; and (2) an action type corresponding to the video data. When the program 230 is executed, the processor 21 retrieves the training data 234 from the storage unit 23 and uses a machine learning algorithm to train the image recognition model 232 based on the multiple training data 234.

[0035] In other words, the video data of the plurality of training data 234 can be used as training input data during the training phase, and the action types of the plurality of training data 234 can be used as training output data during the training phase. After the processor 21 generates the image recognition model 232, the image recognition model 232 can be stored in the storage unit 23 for subsequent use.

[0036] It should be noted that in some embodiments, the machine learning algorithm mainly introduces a convolutional neural network (CNN) algorithm to construct an image recognition model 232 for determining the action type based on the training data 234. In some examples, the CNN algorithm may include image processing and image recognition algorithms such as the YOLO (you only look once) algorithm and the R3D (ResNet 3D) algorithm, but they are not intended to limit the aspects of the machine learning algorithm in the present invention.

[0037] In some embodiments, in the program code of the CNN algorithm for training the image recognition model 232, there is a training function for training the image recognition model 232. During the training of the image recognition model 232, the training function may include a portion for receiving the training data 234.

[0038] Furthermore, the video data can be used as training input data, and the action type corresponding to the video data can be used as training output data. Then, the training function can be executed after the main function of the program code of the CNN algorithm is executed to train the image recognition model 232. After the image recognition model 232 is generated based on the CNN algorithm and using the training data, the image recognition model 232 can be used to determine the action type corresponding to the input video.

[0039] Please refer to Figure 2B , which is a schematic diagram of the use of the monitoring system 2 of some embodiments of the present invention. In detail, when the operation of the production line machine 82 needs to be monitored and analyzed, the image capture device 81 can be installed in the environment where the production line machine 82 is located to capture the video related to the production line machine 82. Among them, the monitoring system 2 can be connected to the image capture device 81 through a network connection (wired network or wireless network).

[0040] In some embodiments, when the operator 83 operates on the production line machine 82, the image capture device 81 can capture the video 810 (e.g., video streaming) of the operator 83 in real time according to the position of the production line machine 82, and transmit the video 810 to the monitoring system 2 via the network. In other words, the monitoring system 2 can obtain the video 810 of the operator 83 from the image capture device 81 via the network.

[0041] In some embodiments, in order to increase the accuracy of the conversion of the image recognition model 232, the video captured on-site by the production machine 82 can be used as feedback data to adjust the image recognition model 232. In detail, the video 810 can include multiple video clips, and the processor 21 of the monitoring system 2 can determine the action type of the operator 83 in each video clip by using the image recognition model 232 generated and stored in the storage unit 23.

[0042] After the processor 21 uses the image recognition model 232 to determine the action type of the operator 83 in each video clip of the video 810, the monitoring system 2 can provide the video clip and the corresponding action type to the user, so that the user can determine whether there is a conversion error of the image recognition model 232. In some embodiments, the monitoring system 2 can provide the video clip and the corresponding action type to the user through a display (not shown) and a graphical user interface (GUI).

[0043] Next, if the user determines that the specific video clip and its corresponding action type are conversion errors of the image recognition model 232, the user can input user settings through the input device 25 to change the action type of the specific video clip to a correct action.

[0044] Subsequently, the processor 21 can use the specific video clip and the corrected action type to update the training data 234, and re-use the updated multiple training data 234 to generate the image recognition model 232. In more detail, the processor 21 can use the original training data 234, at least one specific video clip, and at least one action type corresponding to the at least one specific video clip to generate the image recognition model 234 based on the machine learning algorithm.

[0045] In this way, since the training data re-used for the image recognition model 232 includes relevant data for the production line machine 82 and the operator 83 (i.e., at least one specific video clip and at least one action type corresponding to this at least one specific video clip), the updated image recognition model 232 will have a higher conversion accuracy when applied to the environment of the production line machine 82.

[0046] The technology of adjusting the image recognition model 232 by using the video captured on-site at the production machine 82 as feedback data can be more clearly understood through the following example. For example, the video 810 includes ten video clips "C1-C10". Using the image recognition model 232 generated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type (e.g., "pick up" action or "put down" action) of the operator 83 in each video clip "C1-C10".

[0047] After the processor 21 uses the image recognition model 232 to determine the action type of the video segment "C1-C10", the monitoring system 2 provides the user with the video segments "C1-C10" and their corresponding action types, through the display and GUI, so that the user can determine whether there is a conversion error in the image recognition model 232.

[0048] In this example, the action types of the video segments "C1" and "C8" are respectively determined by the monitoring system 2 to be "pick-up action" and "put down action". However, the user determines that the action types of the video segments "C1" and "C8" should be "put down action" and "pick-up action" respectively. Therefore, the user inputs the user setting through the input device 25 to correct the action types of the video segments "C1" and "C8" to "put down action" and "pick-up action" respectively. Subsequently, the processor 21 updates the training data 234 with the video segments "C1" and "C8" and the corrected action types, and reuses the updated multiple training data 234 to generate the image recognition model 232.

[0049] In some embodiments, after the image recognition model 232 is updated through the aforementioned steps, when the operator 83 continues to operate on the production line machine 82, the image capture device 81 can capture the video 812 of the operator 83 according to the position of the production line machine 82, and transmit the video 812 to the monitoring system 2 through the network. In other words, the monitoring system 2 can obtain the video 812 of the operator 83 from the image capture device 81 through the network. The video 812 includes multiple video clips.

[0050] Next, using the image recognition model 232 that has been updated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type of each video clip of the video 812. Since each video clip carries relevant information of a timestamp, the processor 21 can determine the capture time of each video clip, and further determine the occurrence time and action cycle of the action type represented by each video clip. The processor 21 can record the action type and action cycle in the storage unit 23 for subsequent use.

[0051] In some embodiments, the processor 21 can determine whether the action period of the corresponding action type exceeds the period threshold value for each video clip stored in the storage unit 23. If so, the action type and the corresponding video clip are marked, and the action type, occurrence time and occurrence period of the corresponding video clip are recorded in the log file. In this way, the user can use the log file to efficiently call out the marked video clips in the video 812, and further understand the reasons why the action period of the action type in these video clips exceeds the period threshold value, so as to quickly eliminate the factors causing the delay.

[0052] For example, the default "pick up" action should be completed within 3 seconds, then the processor 21 determines whether the action period of all video clips corresponding to the "pick up" action exceeds the 3 second value. If so, the action type and the corresponding video clip are marked, and the action type, occurrence time and occurrence period of the video clip are recorded in the record file. Then the user can use the record file to efficiently call out the marked video clips in the video 812, and further understand the reasons why the action period of the action type in these video clips exceeds 3 seconds, so as to quickly eliminate the factors causing the delay.

[0053] In some embodiments, the processor 21 can determine whether the time difference between the occurrence times of the corresponding two action types exceeds the time threshold for all two consecutive video clips stored in the storage unit 23. If so, the two action types and the corresponding two video clips are marked, and the action types, occurrence times, and occurrence cycles of the two video clips are recorded in the record file. In this way, the user can use the record file to efficiently call out the marked two video clips in the video 812, and further understand the reason why the time difference between the occurrence times of the corresponding two action types exceeds the time threshold, so as to quickly eliminate the factors causing the delay.

[0054] For example, it is assumed that the related parts configuration operation should be completed within 10 seconds between the consecutive "pick up" action and the "put down" action. The processor 21 determines whether the time difference of the two video clips of the consecutive "pick up" action and the "put down" action exceeds 10 seconds. If so, the two action types and the corresponding two video clips are marked, and the action types, occurrence times and occurrence cycles of the two video clips are recorded in the record file. In this way, the user can use the record file to efficiently call out the two marked video clips in the video 812, and further understand the reason why the time difference of the occurrence times of the corresponding two action types exceeds 10 seconds, so as to quickly eliminate the factors causing the delay.

[0055] Please refer to Figure 2C , which is a schematic diagram of a screen captured by the image capture device 81 in some embodiments of the present invention. In some embodiments, since the image or video captured by the image capture device 81 has a larger range, the processor 21 will consume more hardware resources and time when processing the image or video using the image recognition model 232.

[0056] However, not all images or videos captured by the image capture device 81 need to be monitored. Therefore, a smaller area that needs to be monitored can be defined for the captured images or videos, and the processor 21 only needs to use the image recognition model 232 to process the images or videos in the smaller area. In this way, the processing speed can be greatly accelerated.

[0057] Please refer to Figure 2D , which is another schematic diagram of a screen captured by the image capture device 81 of some embodiments of the present invention. Specifically, the user can input user settings through the input device 25 to define the monitoring area 80A on the image range captured by the image capture device 81, and the processor 21 only needs to process the image or video of the monitoring area 80A using the image recognition model 232. In this way, since the image or video size of the monitoring area 80A is small, the processing speed of the monitoring system 2 can be greatly accelerated.

[0058] In some embodiments, when the environment of the production line machine 82 changes (for example, the angle of the image capture device 81 is adjusted, the operator configuration is changed, the operator position is changed, etc.), the originally intended monitoring range may deviate from the monitoring area 80A, and cause the conversion error of the image recognition model 232 to increase. At this time, the user can directly adjust the position of the monitoring area 80A to reduce the deviation caused by the environmental changes of the production line machine 82.

[0059] Please refer to Figure 2E , which is another schematic diagram of the images captured by the image capture device 81 of some embodiments of the present invention. Specifically, due to the change in the environment of the production line machine 82, the image or video in the monitoring area 80A is not the content to be monitored, which may cause the conversion error of the image recognition model 232 to increase.

[0060] Please refer to Figure 2F , which is another schematic diagram of a screen captured by the image capture device 81 of some embodiments of the present invention. Specifically, the user can input another user setting through the input device 25 to move the monitoring area 80A on the image range captured by the image capture device 81 so that the area to be monitored returns to normal.

[0061] In some embodiments, the images captured by the image capture device 81 may be first transmitted to the monitoring system 2. Subsequently, the monitoring system 2 may display these images through a general display (not shown) and receive user settings through an input device 25 such as a keyboard or a mouse, so that the monitoring system 2 can complete related operations.

[0062] In some embodiments, the images captured by the image capture device 81 may be first transmitted to the monitoring system 2. Subsequently, the monitoring system 2 may transmit the images to a remote display (e.g., a handheld smart device, a notebook computer, etc.) via a network, and receive user settings via an input device 25 such as a network interface, so that the monitoring system 2 can complete related operations.

[0063] Please refer to Figure 3A, which is a block diagram of a monitoring system 3 of some embodiments of the present invention. The monitoring system 3 includes a processor 31, a storage unit 33, and an input device 35. The storage unit 33 stores a program 330, an image recognition model 332A, an image recognition model 332B, and training data 334A, 334B. Among them, the image recognition models 332A, 332B may include models related to machine learning technology, which are used to determine the type of operator's action or the change in the number of objects in the video data (i.e., image sequence data).

[0064] The processor 31, the storage unit 33, and the input device 35 are electrically connected via a communication bus 37. Through the communication bus 37, the processor 31 can execute the program 330 stored in the storage unit 33. When the program 330 is executed, one or more interrupts, such as software interrupts, can be generated to enable the processor 31 to execute the program 330 with the production line monitoring function. The function of the program 330 will be further described below.

[0065] In some embodiments, the image recognition model 332A is mainly a machine learning model generated by a plurality of training data 334A based on a machine learning algorithm. Specifically, some video data and the action types actually corresponding to these video data can be used as training data to train the image recognition model 332A based on the machine learning algorithm (i.e., to generate the image recognition model 332A).

[0066] In more detail, each training data 334A may include: (1) video data; and (2) an action type corresponding to the video data. When the program 330 is executed, the processor 31 retrieves the training data 334A from the storage unit 33 and uses a machine learning algorithm to train the image recognition model 332A based on the multiple training data 334A.

[0067] In other words, the video data of the plurality of training data 334A can be used as training input data during the training phase, and the action types of the plurality of training data 334A can be used as training output data during the training phase. After the processor 31 generates the image recognition model 332A, the image recognition model 332A can be stored in the storage unit 33 for subsequent use.

[0068] In some embodiments, the video data used as training input data in the plurality of training data 334A includes image data of the operator's actions. The action type corresponding to the video data can be used as training output data. Then, the program code of the CNN algorithm can be executed to train the image recognition model 332A. After the image recognition model 332A is generated based on the CNN algorithm and using the training data, the image recognition model 332A can be used to determine the action type corresponding to the input video.

[0069] In some embodiments, the image recognition model 332B is mainly a machine learning model generated by a plurality of training data 334B according to a machine learning algorithm. Specifically, some video data and the actual change in the number of objects corresponding to these video data can be used as training data to train the image recognition model 332B based on the machine learning algorithm (i.e., to generate the image recognition model 332B).

[0070] In more detail, each training data 334B may include: (1) video data; and (2) a change in the number of objects corresponding to the video data (e.g., increase or decrease). When the program 330 is executed, the processor 31 retrieves the training data 334B from the storage unit 33 and uses a machine learning algorithm to train the image recognition model 332B based on the multiple training data 334B.

[0071] In other words, the video data of the plurality of training data 334B can be used as training input data during the training phase, and the object quantity changes of the plurality of training data 334B can be used as training output data during the training phase. After the processor 31 generates the image recognition model 332B, the image recognition model 332B can be stored in the storage unit 33 for subsequent use.

[0072] In some embodiments, the video data used as training input data in the plurality of training data 334B includes image data of object quantity changes. The object quantity changes corresponding to the video data can be used as training output data. Then, the program code of the CNN algorithm can be executed to train the image recognition model 332B. After the image recognition model 332B is generated based on the CNN algorithm and using the training data, the image recognition model 332B can be used to determine the object quantity changes corresponding to the input video.

[0073] In more detail, the video data records the change in the quantity of specific objects (e.g., product components), and the change in the quantity of specific objects can represent different action behaviors. For example, when the quantity of specific objects decreases in the video data, it means that the operator's action is more likely to "pick up" the specific object. When the quantity of specific objects increases in the video data, it means that the operator's action is more likely to "put down" the specific object. Based on this, using the change in the quantity of specific objects in the image data can help improve the accuracy of the action type judgment result.

[0074] Please refer to Figure 3B, which is a schematic diagram of the use of the monitoring system 3 of some embodiments of the present invention. In detail, when the operation of the production line machine 72 needs to be monitored and analyzed, the image capture device 71 can be installed in the environment where the production line machine 72 is located to capture the video related to the production line machine 72. Among them, the monitoring system 3 can be connected to the image capture device 71 through a network connection (wired network or wireless network).

[0075] In some embodiments, when the operator 73 operates on the production line machine 72, the image capture device 71 can capture the video 710 (e.g., video stream) of the operator 73 in real time according to the position of the production line machine 72, and transmit the video 710 to the monitoring system 3 via the network. In other words, the monitoring system 3 can obtain the video 710 of the operator 73 from the image capture device 71 via the network. The video 710 includes multiple video clips.

[0076] Next, the user can input user settings through the input device 35 to define monitoring areas 70A and 70B on the image range captured by the image capture device 71, and the processor 31 only needs to process the images or videos of the monitoring areas 70A and 70B using the image recognition models 332A and 332B.

[0077] Subsequently, using the image recognition model 332A stored in the storage unit 33, the processor 31 of the monitoring system 3 can determine the action type of the monitoring areas 70A and 70B in each video segment of the video 710. Since each video segment carries relevant information of a timestamp, the processor 31 can determine the capture time of each video segment, and further determine the occurrence time and action cycle of the action type represented by the monitoring areas 70A and 70B in each video segment. The processor 31 can record the action type and action cycle in the storage unit 33 for subsequent use.

[0078] In some embodiments, for each surveillance area 70A and 70B of each video clip, the processor 31 of the surveillance system 3 may further use the image recognition model 332B to determine the change in the number of objects and update the action type of the operator 73 accordingly. FIG. 3C to FIG. 3D , which is a schematic diagram of a screen captured by the image capture device 71 of some embodiments of the present invention. For example, for the monitoring area 70A of a specific video segment, the processor 31 of the monitoring system 3 can first use the image recognition model 332A to determine that the action type is "pick up".

[0079] Next, for the monitoring area 70A of the specific video clip, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine that the number of objects 74 in the video clip has decreased. Accordingly, since the action type of the specific video clip in the monitoring area 70A is "pick up", and the decrease in the number of objects 74 is indeed caused by "pick up", the specific action type can be accurately confirmed as "pick up".

[0080] It should be noted that for the monitoring area 70A of the specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "put down", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip has decreased, it means that the judgment of the image recognition model 332A may be wrong. Accordingly, based on the judgment that the number of objects 74 in the video clip has decreased using the image recognition model 332B, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video clip from "put down" to "pick up".

[0081] Please refer to FIG. 3E to FIG. 3F , which is a schematic diagram of a screen captured by the image capture device 71 of some embodiments of the present invention. For example, for the monitoring area 70B of a specific video clip, the processor 31 of the monitoring system 3 can first use the image recognition model 332A to determine that the action type is "put down".

[0082] Next, for the monitoring area 70B of the specific video clip, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine the increase in the number of objects 74 in the video clip. Accordingly, since the action type of the specific video clip in the monitoring area 70B is "putting down", and the increase in the number of objects 74 is indeed caused by "putting down", the specific action type can be accurately confirmed as "putting down".

[0083] Similarly, for the monitoring area 70B of the specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "pick up", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip has increased, it means that the judgment of the image recognition model 332A may be wrong. Accordingly, based on the judgment that the number of objects 74 in the video clip has increased using the image recognition model 332B, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video clip from "pick up" to "put down".

[0084] It should be noted that in the aforementioned embodiment, when the processor uses the image recognition model to determine the action type of the image or video data, it can mainly use the image recognition model to first identify and track the operator's hands, and further determine the action type of the operator in the image or video through the operator's hand movements.

[0085] Some embodiments of the present invention include a production line monitoring method, the flow chart of which is as follows: Figures 4A to 4B The production line monitoring methods of these embodiments are implemented by a monitoring system (such as the monitoring system of the aforementioned embodiments). The detailed operation of the method is as follows.

[0086] First, the monitoring system executes step S401 to obtain multiple images of the operator. The monitoring system may obtain multiple images of the operator through an image capture device installed on a production line machine. The monitoring system executes step S402 to determine the action type of the operator in the multiple images based on an image recognition model. The image recognition model may include a model related to machine learning technology, which is used to receive image data and output the action type of the operator in the image.

[0087] Next, since the multiple images may carry relevant information of the time stamp, the monitoring system executes step S403 to determine the occurrence time and action cycle of the action type according to the action type of the operator in the multiple images. The monitoring system executes step S404 to record the action type, occurrence time and action cycle for subsequent use.

[0088] In some embodiments, after step S402 , in order to increase the accuracy of the determination, the monitoring system may execute step S402 ′ to update the action type according to the change in the number of objects in the plurality of images based on another image recognition model.

[0089] Some embodiments of the present invention include a production line monitoring method, the flow chart of which is as follows: Figures 5A to 5F The production line monitoring methods of these embodiments are implemented by a monitoring system (such as the monitoring system of the aforementioned embodiments). The detailed operation of the method is as follows.

[0090] In some embodiments, the production line monitoring method needs to provide an image recognition model including machine learning technology for receiving image data and outputting the operator's action type in the image. Therefore, the image recognition model needs to be trained and generated using training data.

[0091] Please refer to Figure 5A , which is a flowchart of the generation of an image recognition model of a production line monitoring method of some embodiments of the present invention. The monitoring system executes step S501 to generate an image recognition model based on a machine learning algorithm using multiple training data. Each training data includes a training input and a training output. The training input includes a training video clip, and the training output includes a training action type corresponding to the training video clip. The monitoring system executes step S502 to store the image recognition model for subsequent use.

[0092] In some embodiments, in order to increase the accuracy of the image recognition model, the video captured on-site at the production machine can be used as feedback to adjust the image recognition model. Figure 5B , which is a flowchart of the image recognition model update of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S503 to obtain a video. The monitoring system can obtain the operator's video from an image capture device installed on the production line machine, and the video includes multiple video clips.

[0093] The monitoring system executes step S504 to determine the action type of the operator in each video clip based on the image recognition model generated above. The monitoring system executes step S505, where the user provides the video clip and the corresponding action type, so that the user can determine whether there is a conversion error of the image recognition model.

[0094] When the user determines that there is a conversion error in the image recognition model, resulting in a mismatch between the specific video clip and its corresponding action type, the user can change the specific video clip and its corresponding action type. The monitoring system executes step S506 to receive the user's setting to change the action type of the video clip.

[0095] After all video clips are judged, the monitoring system executes step S507 to adjust the image recognition model according to the specific video clip and the changed action type. Specifically, the monitoring system generates the image recognition model according to the original training data, the specific video clip, the action type corresponding to the specific video clip, and the machine learning algorithm.

[0096] In this way, since the retraining of the image recognition model contains relevant information for the production line machines and operators (i.e., the specific video clips and the action type corresponding to the specific video clips), the updated image recognition model will have higher accuracy when applied to the judgment of the production line machines.

[0097] In some embodiments, the status of production line machines can be monitored based on the updated image recognition model. Figure 5C , which is a flow chart of a production line monitoring method according to some embodiments of the present invention. The monitoring system executes step S508 to obtain a video of an operator of a production line machine. The video includes multiple video clips. The monitoring system executes step S509 to determine the action type of the operator of each video clip based on the image recognition model.

[0098] Next, since multiple video clips may carry relevant information of time stamps, the monitoring system executes step S510 to determine the occurrence time and action period of the action type according to the action type of the operator in each video clip. The monitoring system executes step S511 to record the action type, occurrence time and action period for subsequent use.

[0099] In some embodiments, the monitoring system can determine whether the video action is delayed. Figure 5D , which is a flow chart of a production line monitoring method according to some embodiments of the present invention. The monitoring system executes step S512 to determine whether the action cycle exceeds the cycle threshold value for the action type of the operator in each video clip. If so, the monitoring system executes step S513 to mark the action type and the corresponding video clip, and record the action type, occurrence time and occurrence cycle of the corresponding video clip in the record file, so that the user can efficiently call out the marked video clip in the video using the record file. If not, the monitoring system repeats step S512 to determine whether the action cycle exceeds the cycle threshold value for the action type of the operator in the next video clip.

[0100] In some embodiments, the monitoring system can determine whether there is a delay between actions in the video. Figure 5E , which is a flow chart of a production line monitoring method of some embodiments of the present invention. The monitoring system executes step S514 to calculate the time difference between the occurrence times of the action types of the two video clips. The monitoring system executes step S515 to determine whether the time difference exceeds the time threshold value. If so, the monitoring system executes step S516 to mark the action type and the corresponding two video clips, and record the action type, occurrence time and occurrence cycle of the two video clips in the record file, so that the user can use the record file to efficiently call out the marked video clips in the video. If not, the monitoring system repeats step S514 to calculate the time difference between the occurrence times of the corresponding action types for the next pair of video clips, and determine whether the action cycle exceeds the cycle threshold value.

[0101] In some embodiments, the following steps may be optionally added to speed up the speed and efficiency of image processing. Fig. 5F , which is a flow chart of a production line monitoring method of some embodiments of the present invention. The monitoring system executes step S517 to receive user settings to define a monitoring area on the video to be captured. In other words, the user settings are used to define the monitoring area on the image range captured by the image capture device. Among them, since the image or video size of the monitoring area is small, the processing speed of the monitoring system can be greatly accelerated.

[0102] In some embodiments, the following steps can be optionally added to reduce the deviation caused by environmental changes on the production line machine site. Figure 5G , which is a flow chart of a production line monitoring method according to some embodiments of the present invention. The monitoring system executes step S518 to receive a user setting for moving the monitoring area. In other words, the user setting is used to move the monitoring area within the image range captured by the image capture device.

[0103] The monitoring system and production line monitoring method of the present invention can more quickly and accurately identify factors causing errors or delays on the production line through automation and artificial intelligence, thereby improving the efficiency of the production line output and effectively improving the bottleneck of the production line output.

[0104] It should be particularly understood that the processor mentioned in the above embodiments may be a central processing unit (CPU), other hardware circuit elements capable of executing relevant instructions, or a combination of computing circuits known to those skilled in the art based on the above disclosure. In addition, the storage unit mentioned in the above embodiments may include a memory (such as ROM, RAM, etc.) or a storage device (such as flash memory, HDD, SSD, etc.) for storing data.

[0105] Furthermore, the communication bus mentioned in the above embodiments may include a communication interface for transmitting data between components such as a processor, a storage unit, an input device, etc., and may include an electrical bus interface, an optical bus interface, or even a wireless bus interface. However, such description is not intended to limit the hardware implementation example of the present invention.

[0106] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and replacements may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims. For example, many of the processes discussed above may be implemented in different ways and replaced by other processes or combinations thereof.

[0107] Furthermore, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods, and steps described in this specification. As will be readily understood by one of ordinary skill in the art from the disclosure of the present invention, currently existing or subsequently developed processes, machines, manufactures, material compositions, components, methods, or steps that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein may be utilized in accordance with the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufactures, material compositions, components, methods, or steps.

[0108] Explanation of symbols:

[0109] 1: Monitoring system

[0110] 11: Processor

[0111] 13: Storage Unit

[0112] 130: Procedure

[0113] 132: Image Recognition Model

[0114] 17: Communication Bus

[0115] 2: Monitoring system

[0116] 21: Processor

[0117] 23: Storage Unit

[0118] 230: Procedure

[0119] 232: Image Recognition Model

[0120] 234: Training Materials

[0121] 25: Input Devices

[0122] 27: Communication Bus

[0123] 3: Monitoring system

[0124] 31: Processor

[0125] 33: Storage Unit

[0126] 330: Procedure

[0127] 332A: Image Recognition Model

[0128] 332B: Image Recognition Model

[0129] 334: Training Materials

[0130] 35: Input Devices

[0131] 37: Communication Bus

[0132] 70A: Surveillance Area

[0133] 70B: Surveillance Area

[0134] 71: Image Capture Device

[0135] 710: Video

[0136] 72: Production line machine

[0137] 73: Operator

[0138] 74: Object

[0139] 80A: Surveillance Area

[0140] 81: Image Capture Device

[0141] 810: Video

[0142] 812: Video

[0143] 82: Production line machine

[0144] 83: Operator

[0145] 91: Image Capture Device

[0146] 910: Image

[0147] 92: Production line machine

[0148] 93: Operator

[0149] S401-S404: Steps

[0150] S501-S518: Steps

Claims

1. A production line monitoring method for a monitoring system, comprising: Acquiring multiple images of an operator; Determining an action type of the operator of the plurality of images based on a first image recognition model; Based on a second image recognition model, updating the action type of the operator in the plurality of images according to a change in the number of objects in the plurality of images; Determine an occurrence time and an action period of the action type; and Record the action type, the time of occurrence and the action cycle; The step of updating the action type of the operator in the plurality of images according to the change in the number of objects in the plurality of images based on the second image recognition model comprises: When the second image recognition model determines that the change in the number of objects is a decrease, updating the action type of the operator from putting down to picking up; and When the second image recognition model determines that the change in the number of objects is an increase, the action type of the operator is updated from picking up to putting down.

2. The production line monitoring method of claim 1, further comprising: Determine whether the action cycle exceeds a cycle threshold; When the action period exceeds the period threshold, the action type is marked.

3. The production line monitoring method of claim 1, further comprising: Calculate a time difference between the occurrence time of the action type and the occurrence time of another action type; Determine whether the time difference exceeds a time threshold; When the time difference exceeds the time threshold, the action type and the other action type are marked.

4. The production line monitoring method of claim 1, further comprising: receiving a user setting for defining a monitoring area on the plurality of images; in, The step of determining the action type of the operator of the plurality of images based on the first image recognition model further comprises: Based on the first image recognition model, the action type within the monitoring area of ​​the multiple images is determined.

5. The production line monitoring method of claim 4, further comprising: Another user setting is received to move the monitoring area defined on the plurality of images.

6. The production line monitoring method according to claim 4, wherein: The multiple images have an image size, the monitoring area has an area size, and the monitoring area size is smaller than the image size.

7. The production line monitoring method of claim 1, further comprising: Based on the first image recognition model, at least one hand of the operator in the plurality of images is recognized, and the action type of the at least one hand of the operator is determined.

8. The production line monitoring method of claim 1, further comprising: Based on a machine learning algorithm, the first image recognition model is generated using a plurality of training data, wherein each training data includes a training input and a training output, the training input includes a plurality of training images, and the training output includes a training action type corresponding to the plurality of training images.

9. A production line monitoring method for a monitoring system, comprising: Get a video, where The video contains multiple video clips; Determining an action type of each of the video clips based on a first image recognition model; Based on a second image recognition model, updating the action type of each of the video clips according to a change in the number of objects in the plurality of video clips; receiving a user setting to change the motion type of a first video segment of the plurality of video segments; and Adjusting the first image recognition model according to the action type of the first video clip; The step of updating the action type of each video segment according to the change in the number of objects in the plurality of video segments based on the second image recognition model comprises: When the second image recognition model determines that the change in the number of objects is a decrease, updating the operator's action type from putting down to picking up; and When the second image recognition model determines that the change in the number of objects is an increase, the action type of the operator is updated from picking up to putting down.

10. The production line monitoring method of claim 9, further comprising: The first image recognition model is generated using a plurality of training data and a machine learning algorithm, wherein: Each training data includes a training input and a training output. The training input includes a training video segment, and the training output includes a training action type corresponding to the training video segment.

11. The production line monitoring method according to claim 10, wherein: The step of adjusting the first image recognition model according to the action type of the first video clip further comprises: The first image recognition model is generated using the plurality of training data, the first video segment, the action type corresponding to the first video segment, and the machine learning algorithm.

12. A monitoring system for production line monitoring, comprising: a processor; and A storage unit stores a program and a first image recognition model, wherein when the program is executed, the processor: Acquiring multiple images of an operator; Determining an action type of the operator of the plurality of images based on the first image recognition model; Based on a second image recognition model, updating the action type of the operator in the plurality of images according to a change in the number of objects in the plurality of images; Determine an occurrence time and an action period of the action type; and Record the action type, the time of occurrence and the action cycle; in: When the second image recognition model determines that the change in the number of objects is a decrease, the processor updates the action type of the operator from putting down to picking up; as well as When the second image recognition model determines that the change in the number of objects is an increase, the processor updates the action type of the operator from picking up to putting down.

13. The monitoring system of claim 12, wherein: The program, when executed, further causes the processor to: Determine whether the action cycle exceeds a cycle threshold; When the action period exceeds the period threshold, the action type is marked.

14. The monitoring system of claim 12, wherein: The program, when executed, further causes the processor to: Calculate a time difference between the occurrence time of the action type and the occurrence time of another action type; Determine whether the time difference exceeds a time threshold; When the time difference exceeds the time threshold, the action type and the other action type are marked.

15. The monitoring system of claim 12, further comprising: An input device for receiving a user setting; in, The program, when executed, further causes the processor to: Defining a monitoring area on the plurality of images according to the user setting; Based on the first image recognition model, the action type within the monitoring area of ​​the multiple images is determined.

16. The monitoring system of claim 15, wherein: The input device is further configured to receive another user setting. When the program is executed, the processor is further configured to: The monitoring area defined on the plurality of images is moved according to the setting of another user.

17. The monitoring system of claim 12, wherein: The program, when executed, further causes the processor to: Based on a machine learning algorithm, the first image recognition model is generated using a plurality of training data, wherein each training data includes a training input and a training output, the training input includes a plurality of training images, and the training output includes a training action type corresponding to the plurality of training images.

18. The monitoring system of claim 17, further comprising: An input device for receiving a user setting; in, The program, when executed, further causes the processor to: Changing the action type of the operator of the plurality of images according to the user setting; The first image recognition model is adjusted according to the action type of the operator in the plurality of images.

19. The monitoring system of claim 17, wherein: The program, when executed, further causes the processor to: Based on the machine learning algorithm, the first image recognition model is generated using the plurality of training data, the plurality of images, and the action type of the operator corresponding to the plurality of images.

Images