Electronic billboard automatic monitoring and production abnormality real-time reaction system and method thereof

Abstract

An electronic display automatic monitoring and production abnormality real-time reaction system and method thereof, a monitoring device analyzes image patterns of a monitoring image to analyze at least one analysis region and extracts a corresponding image to be analyzed, at least one image analysis method is set according to the image pattern of the image to be analyzed to convert the image to be analyzed into corresponding image features, when the image features are compared with the preset features and are abnormal, the monitoring device generates, records and sends warning messages and / or issues alarms, thereby achieving the technical effect of providing automatic monitoring of electronic displays and real-time reaction to production abnormalities.

Description

Technical Field

[0001] A monitoring and real-time response system and method, particularly a system and method for analyzing the image pattern of an image to be analyzed in order to set an image analysis method accordingly, and for a monitoring device to react in real time when the image features are abnormal. Background Technology

[0002] With the development of computer technology, the use of electronic Kanban visualization methods to monitor the production process has become a common practice in enterprise manufacturing. Through electronic Kanban visualization, producers and managers can not only efficiently and conveniently grasp production information and save on communication and management costs, but also identify production anomalies in real time, reducing anomaly handling time and minimizing enterprise losses.

[0003] However, it is not practical for people to monitor electronic dashboards around the clock. On the one hand, it requires dedicated personnel to be on duty, which increases costs. On the other hand, people are affected by fatigue, which can easily lead to misjudgments and oversights during the monitoring process.

[0004] In summary, it is clear that existing technologies have long suffered from the problem of misjudgment and oversight due to manual monitoring of electronic dashboards. Therefore, it is necessary to propose improved technical methods to solve this problem. Summary of the Invention

[0005] In view of the problems of misjudgment and oversight that arise from manual monitoring of electronic dashboards in existing technologies, this invention discloses an automatic monitoring system and method for electronic dashboards and real-time response to production anomalies, wherein:

[0006] The electronic signboard automatic monitoring and real-time response system for production anomalies disclosed in this invention is applicable to electronic signboards that display monitoring information. It includes: an electronic signboard and a monitoring device. The monitoring device further includes: an image capture module, a selection capture module, an analysis method setting module, an image analysis module, a generation module, and an alert module.

[0007] The electronic dashboard displays the monitoring information that has been set for monitoring; the image capture module of the monitoring device captures monitoring images from the monitoring information; the selection capture module of the monitoring device analyzes the image patterns of the monitoring images to identify at least one analysis area corresponding to the image pattern and capture it as at least one image to be analyzed; the analysis method setting module of the monitoring device sets at least one image analysis method according to the image pattern of at least one image to be analyzed; the image analysis module of the monitoring device converts at least one image to be analyzed into corresponding image features according to the set at least one image analysis method; the generation module of the monitoring device generates a warning message when the image features are found to be abnormal compared with preset features; and the warning module of the monitoring device records the warning message, sends the warning message, and / or issues an alarm when the generation module generates a warning message.

[0008] The automatic monitoring and real-time response method for production anomalies disclosed in this invention is applicable to electronic dashboards that display monitoring information, and includes the following steps:

[0009] First, the monitoring device establishes a connection with the electronic dashboard; then, the monitoring device captures monitoring images from the monitoring information; next, the monitoring device analyzes the image patterns of the monitoring images to identify at least one analysis area corresponding to the image pattern and captures it as at least one image to be analyzed; next, the monitoring device sets at least one image analysis method according to the image pattern of the at least one image to be analyzed; next, the monitoring device converts the at least one image to be analyzed into corresponding image features according to the set at least one image analysis method; next, when the image features are found to be abnormal compared with preset features, the monitoring device generates an alert message; finally, when the monitoring device generates an alert message, it records the alert message, sends the alert message, and / or issues an alarm.

[0010] The system and method disclosed in this invention are as described above. The difference between them and the prior art is that the monitoring device analyzes the image pattern of the monitored image to analyze at least one analysis area and captures it as the corresponding image to be analyzed. At least one image analysis method is set according to the image pattern of the image to be analyzed to convert the image to be analyzed into the corresponding image features. When the image features are found to be abnormal when compared with the preset features, the monitoring device generates, records and sends a warning message and / or issues an alarm.

[0011] Through the above-mentioned technical means, the present invention can achieve the technical effect of providing automatic monitoring of electronic dashboards and real-time response to production anomalies. Attached Figure Description

[0012] Figure 1 The diagram illustrates the system block diagram of the electronic Kanban automatic monitoring and real-time response system for production anomalies of the present invention.

[0013] Figure 2 The illustration shows a monitoring image of the electronic dashboard of the present invention, which automatically monitors and responds to production anomalies in real time.

[0014] Figure 3 The diagram illustrates the analysis area for automatic monitoring and real-time response to production anomalies on the electronic dashboard of this invention.

[0015] Figures 4A to 4C The illustration is a schematic diagram of the analysis of the electronic Kanban system for automatic monitoring and real-time response to production anomalies according to the present invention.

[0016] Figure 5A as well as Figure 5B The illustration shows an anomaly template for automatic monitoring and real-time response to production anomalies using the electronic dashboard of this invention.

[0017] Figure 6 The diagram illustrates the method flowchart for automatic monitoring of electronic dashboards and real-time response to production anomalies according to the present invention.

[0018] The annotations in the attached figures are explained as follows:

[0019] 10: Electronic Kanban

[0020] 20: Monitoring device

[0021] 21: Image Capture Module

[0022] 22: Select the capture module

[0023] 23: Analysis Method Setting Module

[0024] 24: Image Analysis Module

[0025] 25: Generating Modules

[0026] 26: Warning Module

[0027] 41: Monitoring video

[0028] 421: First Analysis Region

[0029] 422: Second Analysis Region

[0030] 423: Third Analysis Region

[0031] 431: First image to be analyzed

[0032] 432: Second image to be analyzed

[0033] 433: Third image to be analyzed

[0034] 511: First Anomaly Template

[0035] 512: Second Anomaly Template

[0036] Step 101: The electronic dashboard displays the monitoring information that has been set to be monitored.

[0037] Step 102: Establish a connection between the monitoring device and the electronic dashboard.

[0038] Step 103: The monitoring device captures monitoring images from the monitoring information.

[0039] Step 104: The monitoring device selects at least one analysis area from the monitored images and captures it as at least one image to be analyzed.

[0040] Step 105: The monitoring device analyzes the image pattern of at least one image to be analyzed, and sets at least one image analysis method according to the analyzed image pattern.

[0041] Step 106: The monitoring device converts at least one image to be analyzed into corresponding image features according to at least one preset image analysis method.

[0042] Step 107: When the image features are found to be abnormal compared to preset features, the monitoring device generates an alert message.

[0043] Step 108: When the monitoring device generates an alert message, the monitoring device records the alert message, sends the alert message, and / or issues an alarm. Detailed Implementation

[0044] The following will describe in detail the implementation of the present invention with reference to the accompanying drawings and embodiments, thereby enabling a full understanding of how the present invention uses technical means to solve technical problems and achieve technical effects, and allowing for its implementation.

[0045] The following section will first describe the electronic Kanban automatic monitoring and real-time production anomaly response system disclosed in this invention, and please refer to [reference needed]. Figure 1 As shown, Figure 1 The diagram illustrates the system block diagram of the electronic Kanban automatic monitoring and real-time response system for production anomalies of the present invention.

[0046] The electronic signboard automatic monitoring and real-time response system for production anomalies disclosed in this invention is applicable to an electronic signboard 10 that displays monitoring information. It includes a monitoring device 20, which further includes an image capture module 21, a selection capture module 22, an analysis method setting module 23, an image analysis module 24, a generation module 25, and an alert module 26.

[0047] The electronic dashboard 10 is used to display the monitoring information that has been set to be monitored. The monitoring information can be presented in the form of histograms, line graphs, tables, array color indicators, etc. The electronic dashboard 10 can display monitoring information in the form of videos, or it can display monitoring information in the form of images and update the display of the monitoring information in the form of images at regular intervals. This is only an example and is not intended to limit the scope of application of the present invention.

[0048] The monitoring device 20 and the electronic signboard 10 are connected via wired or wireless transmission. The aforementioned wired transmission methods include, for example, cable networks, fiber optic networks, etc., and the aforementioned wireless transmission methods include, for example, Wi-Fi, mobile communication networks (e.g., 3G, 4G, 5G, etc.). These are merely examples and are not intended to limit the scope of application of the present invention.

[0049] Please refer to Figure 2 As shown, Figure 2 The illustration shows a monitoring image of the electronic dashboard of this invention, which automatically monitors and responds to production anomalies in real time. Figure 2 In the monitoring device 20, the image capturing module 21 can capture the monitoring image 41 from the monitoring information. If the monitoring information is in video format, the image capturing module 21 of the monitoring device 20 can capture the video to obtain the monitoring image 41. If the monitoring information is in video format, the image capturing module 21 of the monitoring device 20 can receive the image from the electronic display board 10 to obtain the monitoring image 41.

[0050] Please refer to Figure 3 As shown, Figure 3The diagram illustrates the analysis area for automatic monitoring and real-time response to production anomalies on the electronic dashboard of this invention. After the image acquisition module 21 of the monitoring device 20 acquires the monitoring image 41, the selection acquisition module 22 of the monitoring device 20 analyzes the image style of the monitoring image 41. Image styles include, for example, histograms, line graphs, array color charts, etc. A histogram has multiple rectangular blocks, horizontal and vertical axis image features; a line graph has multiple turning points, multiple line segments, horizontal and vertical axis image features; and an array color chart has multiple color indicators arranged in an array image features. The selection acquisition module 22 of the monitoring device 20 analyzes the image style of the monitoring image 41 based on the aforementioned image features to monitor the image. The monitoring image 41 analyzes a first analysis region 421 corresponding to the image patterns of the histogram and line graph, a second analysis region 422 corresponding to the image pattern of the histogram, and a third analysis region 423 corresponding to the image pattern of the array color chart. The first analysis region 421, the second analysis region 422, and the third analysis region 423 are then extracted from the monitoring image 41 as the first image to be analyzed 431, the second image to be analyzed 432, and the third image to be analyzed 433, respectively. This is merely an example and does not limit the scope of the invention. Please refer to the diagrams of the first image to be analyzed 431, the second image to be analyzed 432, and the third image to be analyzed 433 for further details. Figures 4A to 4C As shown, Figures 4A to 4C The illustration is a schematic diagram of the analysis of the electronic Kanban system for automatic monitoring and real-time response to production anomalies according to the present invention.

[0051] The analysis method setting module 23 of the monitoring device 20 sets at least one corresponding image analysis method based on the image styles corresponding to the first image to be analyzed 431, the second image to be analyzed 432, and the third image to be analyzed 433. It is worth noting that the image analysis method can be pre-established in the monitoring device 20, or the image analysis method can be provided to the monitoring device 20 for computing services through an external server. Furthermore, the image analysis method can be expanded and updated in the monitoring device 20 or the external server to provide more diversified image analysis methods.

[0052] Specifically, the analysis method setting module 23 of the monitoring device 20 sets the image analysis method as "match_historgram" based on the image style of the first image to be analyzed 431 as a histogram and a line graph, sets the image analysis method as "match_cluster" based on the image style of the second image to be analyzed 432 as a histogram, and sets the image analysis method as "match_template" based on the image style of the third image to be analyzed 433 as an array color mark. This is only an example and is not intended to limit the scope of application of the present invention.

[0053] After the analysis method setting module 23 of the monitoring device 20 analyzes the image style of the first image to be analyzed 431, the second image to be analyzed 432, and the third image to be analyzed 433, and sets the image analysis method according to the analyzed image style, the image analysis module 24 of the monitoring device 20 converts at least one image to be analyzed into corresponding image features according to at least one set image analysis method.

[0054] Specifically, the third image to be analyzed, 433, uses the image analysis method set to "match_template" as follows:

[0055] The first anomaly template 511 and the second anomaly template 512 are pre-established. Please refer to the diagrams of the first anomaly template 511 and the second anomaly template 512. Figure 5A as well as Figure 5B As shown, Figure 5A as well as Figure 5B The diagram illustrates the abnormal templates for automatic monitoring and real-time response to production anomalies using the electronic dashboard of this invention. The first abnormal template 511 and the second abnormal template 512 are sequentially compared with the third image 433 to be analyzed from left to right and from top to bottom. The mean square error (MSE) is used to calculate the similarity of the third image 433 to be analyzed. The similarity of the third image 433 to be analyzed is the image feature transformed from the third image 433 to be analyzed. The formula for the mean square error is as follows:

[0056]

[0057] Where T represents the first abnormal template 511 or the second abnormal template 512, I represents the third image to be analyzed 433, R represents the similarity function between the first abnormal template 511 or the second abnormal template 512 and the third image to be analyzed 433, (x,y) represents the array unit, (x',y') represents the movement process variable, and the closer the calculated R is to 1, the more similar the third image to be analyzed 433 is to the first abnormal template 511 or the second abnormal template 512.

[0058] Specifically, the first image to be analyzed, 431, uses the following image analysis method based on the set image analysis method "match_historgram":

[0059] Each pixel of the first image 431 to be analyzed has a corresponding [B,G,R] matrix. [B,G,R] are the blue, green and red values ​​commonly used in image processing. Converting the first image 431 from a color image to a grayscale image means converting the [B,G,R] matrix into a single grayscale value. The ratio of the average grayscale value of the first image 431 to the normal or abnormal grayscale value is the image feature of the first image 431 after conversion.

[0060] Specifically, the second image to be analyzed, 432, uses the image analysis method "match_cluster" as set as follows:

[0061] The second image to be analyzed, 432, is converted into a grayscale image, and then the image features of the converted grayscale image, 432, are calculated using the Histogram of Oriented Gradient (HOG).

[0062] The directional gradient histogram can be obtained by dividing each sample image of each category into interconnected cells after converting it into a grayscale image, collecting the gradient of each pixel in the cell or the directional gradient histogram of the edge, and combining all the directional gradient histograms to obtain the image features of the second image to be analyzed 432.

[0063] The directional gradient histogram can also expand the cell unit into a block, calculate the density of each histogram in this block, and then normalize each cell unit in the block according to this density. Finally, the image features of the second image to be analyzed 432 can be obtained by combining all the directional gradient histograms after contrast normalization.

[0064] Next, the generation module 25 of the monitoring device 20 generates an alert message when the image features are compared with preset features and are found to be abnormal. Specifically, when the image analysis method is "match_historgram" and "match_template", an alert message is generated when the calculated image features of the third image to be analyzed 433 and the image features of the first image to be analyzed 431 are compared with preset features and are found to be abnormal.

[0065] If the image analysis method is "match_cluster", based on the pre-established sample images of "bad", "good" and "modest" (e.g., 10 sample images for each category), where different categories represent the corresponding degree of abnormality, each sample image of each category is converted into a grayscale image. Then, the oriented gradient histogram is used to calculate the image features of each sample image of each category after it has been converted into a grayscale image. Please refer to the above description for the calculation of image features, which will not be repeated here.

[0066] Image features of the same category are clustered together, and the cluster center of image features of the same category is the prototype of the sample image for each category. Then, the distance between the image features of the second image to be analyzed 432 and the prototype of the sample image is calculated. When the distance is less than or equal to a preset threshold value, the image features of the second image to be analyzed 432 are the category corresponding to the prototype of the sample image.

[0067] In addition, sample images categorized as "good" and "modest" can have their image features further assigned a feature label of 0, and sample images categorized as "bad" can have their image features further assigned a feature label of 1. The feature labels of the sample images are used to train a support vector machine (SVM). The image features of the second image to be analyzed 432 are then input into the trained SVM. If the output is 0, it indicates that the image features of the second image to be analyzed 432 are in the "good" or "modest" category. If the output is 1, it indicates that the image features of the second image to be analyzed 432 are in the "bad" category. When the image features of the second image to be analyzed 432 are in the "bad" category, it means that the image features are abnormal compared with the preset features, and the generation module 25 of the monitoring device 20 will generate a warning message.

[0068] The image analysis module 24 and the generation module 25 can flexibly use various image analysis methods. Each image analysis method uses computer vision technology to combine manual feature extraction and deep learning feature extraction methods according to the actual situation to achieve feature extraction. Then, according to the actual situation, it combines statistical analysis methods (e.g., determining anomalies based on distribution), supervised learning methods (e.g., training the model after labeling multiple anomaly categories), and unsupervised learning methods (e.g., learning the pattern of normal samples based on the distribution of samples) to achieve good detection results.

[0069] The warning module 26 of the monitoring device 20 records the warning message, sends the warning message, and / or issues an alarm when the generation module 25 of the monitoring device 20 generates a warning message. The warning module 26 of the monitoring device 20 can send the warning message via email or SMS. The warning module 26 of the monitoring device 20 can issue a voice alarm or an alarm via sound effect. This is only an example and is not intended to limit the scope of application of the present invention.

[0070] The warning module 26 of the monitoring device 20 further sends warning messages and / or issues alarms according to the warning strategy, specifically:

[0071] Within a specific time period (e.g., 5 minutes, 10 minutes, etc.), when the types of warning messages generated by the same electronic signboard 10 are similar, the warning module 26 of the monitoring device 20 may send a maximum of two warning messages and / or issue a second alarm; within a specific time period (e.g., 5 minutes, 10 minutes, etc.), when different electronic signboards 10 generate warning messages using the same image analysis method, the warning module 26 of the monitoring device 20 may send a maximum of two warning messages and / or issue a second alarm; within a specific duration (e.g., 1 hour, 2 hours, etc.), when the types of warning messages generated by the same electronic signboard 10 are similar, the monitoring device 20 may send a maximum of two warning messages and / or issue a second alarm. The warning module 26 of the control device 20 may send a maximum of two warning messages and / or issue a maximum of two alarms. Within a specific time period (e.g., 1 hour, 2 hours, etc.), when different electronic billboards 10 generate warning messages using the same image analysis method, the warning module 26 of the monitoring device 20 may send a maximum of two warning messages and / or issue a maximum of two alarms. Furthermore, it may provide a strategy for merging warning messages and / or alarms, a strategy for preventing false alarms of warning messages and / or alarms, and a notification for missed warning messages and / or alarms. These are merely examples and are not intended to limit the scope of application of the present invention.

[0072] Furthermore, the monitoring device 20 can add the corresponding electronic signboard 10 to the monitoring list. When the selected acquisition module 22 of the monitoring device 20 acquires the image to be analyzed, the image analysis module 24 of the monitoring device 20 will directly use the pre-specified image analysis method to convert the image to be analyzed into image features and proceed with the subsequent process, which will not be elaborated here.

[0073] The image analysis method described above can automatically achieve parallel analysis based on the number of images to be analyzed simultaneously to ensure the immediacy of detection. Furthermore, the monitoring images and the images to be analyzed are further provided with merged sending processing. The message publishing and receiving adopts Kafka technology, and the message storage uses the Postgres database and related technologies.

[0074] Next, the operation method of the present invention will be described below, and please refer to the following: Figure 6 As shown, Figure 6 The diagram illustrates the method flowchart for automatic monitoring of electronic dashboards and real-time response to production anomalies according to the present invention.

[0075] The automatic monitoring and real-time response method for electronic dashboards disclosed in this invention includes the following steps:

[0076] First, the electronic dashboard displays the monitoring information that has been set to be monitored (step 101); then, the monitoring device establishes a connection with the electronic dashboard (step 102); next, the monitoring device captures the monitoring image from the monitoring information (step 103); next, the monitoring device selects at least one analysis area in the monitoring image and captures it as at least one image to be analyzed (step 104); next, the monitoring device analyzes the image style of the at least one image to be analyzed and sets at least one image analysis method according to the analyzed image style (step 105); next, the monitoring device converts the at least one image to be analyzed into corresponding image features according to the set at least one image analysis method (step 106); next, when the image features are found to be abnormal compared with preset features, the monitoring device generates an alarm message (step 107); finally, when the monitoring device generates an alarm message, the monitoring device records the alarm message, sends the alarm message, and / or issues an alarm (step 108).

[0077] In summary, the difference between the present invention and the prior art lies in that the monitoring device analyzes the image pattern of the monitored image to identify at least one analysis area and extracts it as the corresponding image to be analyzed. Based on the image pattern of the image to be analyzed, at least one image analysis method is set to convert the image to be analyzed into corresponding image features. When the image features are found to be abnormal compared with preset features, the monitoring device generates, records and sends warning messages and / or issues an alarm.

[0078] This technology can solve the problems of misjudgment and oversight that can easily occur when manually monitoring electronic dashboards, thereby achieving the technical effect of providing automatic monitoring of electronic dashboards and real-time response to production anomalies.

[0079] While the embodiments disclosed in this invention are as described above, the content is not intended to directly limit the scope of patent protection of this invention. Anyone skilled in the art to which this invention pertains may make minor variations in form and detail without departing from the spirit and scope of this invention. The scope of patent protection of this invention shall still be defined by the appended claims.

Claims

1. An electronic dashboard automatic monitoring and real-time production anomaly response system, suitable for displaying monitoring information on an electronic dashboard, characterized in that, Include: A monitoring device, wherein the monitoring device is connected to the electronic signboard, and the monitoring device further comprises: An image capturing module captures a monitoring image from the monitoring information; A selected extraction module analyzes the image pattern of the monitored image to extract at least one analysis region corresponding to the image pattern and extract it as at least one image to be analyzed. An analysis method setting module sets at least one image analysis method according to the image style of the at least one image to be analyzed; An image analysis module converts the at least one image to be analyzed into a corresponding image feature according to the at least one preset image analysis method; A generation module generates a warning message when the image features are found to be abnormal compared to preset features; and An alert module, when the generation module generates the alert message, records the alert message, sends the alert message, and / or issues an alarm; The image analysis module converts the at least one image to be analyzed into a grayscale image and calculates the ratio of the average grayscale value of the at least one image to the normal grayscale value or the abnormal grayscale value as the image feature of the at least one image to be analyzed; or, the image analysis module sequentially compares the similarity of the at least one image to be analyzed with a pre-established abnormal template and uses the mean square error to calculate the image feature of the at least one image to be analyzed; or, the image analysis module converts the at least one image to be analyzed into a grayscale image and then calculates the image feature of the at least one image to be analyzed converted into a grayscale image using the histogram of oriented gradients.

2. The electronic Kanban automatic monitoring and real-time production anomaly response system according to claim 1, characterized in that, The generation module further includes converting multiple sample images according to pre-established categories into grayscale images, then using directional gradient histograms to calculate image features for each sample image of each category after conversion into grayscale images, clustering image features of the same category, with the cluster center of image features of the same category being the sample image prototype of each category, and then calculating the distance between the image features of the at least one image to be analyzed and the sample image prototype. When the distance is less than or equal to a preset threshold value, the image features of the at least one image to be analyzed belong to the category corresponding to the sample image prototype.

3. A method for automatic monitoring and real-time response to production anomalies using electronic dashboards, characterized in that, An electronic dashboard for displaying monitoring information includes the following steps: A monitoring device is connected to the electronic signboard; The monitoring device captures a monitoring image from the monitoring information; The monitoring device analyzes the image pattern of the monitored image to identify at least one analysis region corresponding to the image pattern and extract it as at least one image to be analyzed. The monitoring device sets at least one image analysis method according to the image style of the at least one image to be analyzed; The monitoring device converts the at least one image to be analyzed into a corresponding image feature according to the at least one preset image analysis method; When the image features are found to be abnormal compared to preset features, the monitoring device generates an alert message; and When the monitoring device generates the warning message, the monitoring device records the warning message, sends the warning message, and / or issues an alarm; The step of the monitoring device converting the at least one image to be analyzed into the image features according to the preset image analysis method involves converting the at least one image to be analyzed into a grayscale image and calculating the ratio of the average grayscale value of the at least one image to the normal grayscale value or the abnormal grayscale value as the image features of the at least one image to be analyzed; or, the step of the monitoring device converting the at least one image to be analyzed into the image features according to the preset image analysis method involves sequentially comparing the similarity between the at least one image to be analyzed and a pre-established abnormal template, and using the mean square error to calculate the image features of the at least one image to be analyzed; or, the step of the monitoring device converting the at least one image to be analyzed into the image features according to the preset image analysis method involves converting the at least one image to be analyzed into a grayscale image, and then calculating the image features of the at least one image to be analyzed converted into a grayscale image using the histogram of oriented gradients.

4. The method for automatic monitoring and real-time response to production anomalies using electronic dashboards according to claim 3, characterized in that, Multiple sample images of pre-established categories are converted into grayscale images. Then, the directional gradient histogram is used to calculate the image features of each sample image of each category after conversion to grayscale images. Image features of the same category are clustered together, and the cluster center of image features of the same category is the sample image prototype of each category. Then, the distance between the image features of the at least one image to be analyzed and the sample image prototype is calculated. When the distance is less than or equal to a preset threshold value, the image features of the at least one image to be analyzed belong to the category corresponding to the sample image prototype.

Images