Image processing device

The image processing device aids in analyzing visualized data by classifying and highlighting critical regions within input images, addressing the challenge of inexperienced personnel analyzing complex data sets, thereby enhancing data interpretation and reducing analysis errors.

WO2024241868A1PCT designated stage expired Publication Date: 2024-11-28PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017002
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-07
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In workplaces with a shortage of human resources, inexperienced personnel struggle to analyze visualized data effectively, leading to wasted time and potentially inaccurate conclusions due to insufficient knowledge, making it difficult to derive meaningful information from complex data sets.

Method used

An image processing device that acquires input images, classifies them into normal or abnormal states, determines the partial regions influencing the classification, and generates output images by highlighting these regions, aiding analysis by personnel with insufficient experience or knowledge.

Benefits of technology

The device supports the analysis of visualized data by highlighting critical areas, reducing the burden on analysts and preventing errors, allowing for easier extraction of meaningful information from complex graphs and improving data interpretation skills.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017002_28112024_PF_FP_ABST
    Figure JP2024017002_28112024_PF_FP_ABST
Patent Text Reader

Abstract

An image input unit (21) acquires an input image including visualization data. A determination unit (22) acquires a classification result obtained by classifying the input image into one of a plurality of classes. A determination unit (23) determines a partial region of the input image serving as a basis for determining the classification result. An image generation unit (24) generates an output image by highlighting the partial region in the input image. A display device (16) outputs the output image.
Need to check novelty before this filing date? Find Prior Art

Description

Image Processing Device

[0001] The present disclosure relates to an image processing device, an image processing method, and a program.

[0002] In recent years, digital transformation and the use of big data have become a hot topic, creating a demand for data visualization, which visualizes various data in various formats, such as graphs, charts, timelines, or diagrams.

[0003] For example, Patent Documents 1 and 2 disclose display devices using heat maps.

[0004] International Publication No. 2021 / 009819 Patent No. 6868981

[0005] For business improvement purposes, there is a need to analyze and understand visualized or numerical data. However, many workplaces are facing a labor shortage, making it difficult to find personnel specialized in analysis. Even if inexperienced personnel are assigned to analyze data, they may not know what to focus on in the data and end up wasting time. Furthermore, without specialized knowledge of the subject represented by the data, it is difficult to extract meaningful information from the data. Inaccurate conclusions may be drawn due to insufficient experience or knowledge. Therefore, there is a need to support personnel with insufficient experience or knowledge in analyzing visualized or numerical data.

[0006] The present disclosure provides an image processing device, an image processing method, and a program that can assist personnel with insufficient experience or knowledge in analyzing visualized data or numerical data.

[0007] According to one aspect of the present disclosure, an image processing device includes an image input unit that acquires an input image including visualization data; a first determination unit that acquires a classification result that classifies the input image into one of a plurality of classes; a second determination unit that determines a partial region of the input image that serves as a basis for determining the classification result; an image generation unit that generates an output image by highlighting the partial region in the input image; and an image output unit that outputs the output image.

[0008] According to one aspect of the present disclosure, analysis of visualized data or numerical data can be supported by personnel with insufficient experience or knowledge.

[0009] 13 is a block diagram showing the configuration of a system including an image processing device 1 according to a first embodiment. It is a functional block diagram for explaining the operation of the image processing device 1 of FIG. 1. It is a diagram showing the configuration of a determination unit 22 of FIG. 2. It is a schematic diagram for explaining the operation of the determination unit 22 of FIG. 2. It is a flowchart showing image generation processing executed by the processor 11 of FIG. 1. It is a diagram showing a first exemplary input image 51 processed by the image processing device 1 of FIG. 1. It is a diagram showing reference data 52 acquired by the determination unit 23 when processing the input image 51 of FIG. 6. It is a diagram showing a heat map 53 generated based on the reference data 52 of FIG. 7. It is a diagram showing a heat map 54 generated based on the heat map 53 of FIG. 8, the heat map 54 having a size changed to fit the input image 51. It is a diagram showing an output image 55 generated by superimposing the heat map 54 of FIG. 9 on the input image 51 of FIG. 1. It is a diagram showing a second exemplary input image 61 processed by the image processing device 1 of FIG. 1. It is a diagram showing an exemplary output image 62 corresponding to the input image 61 of FIG. 11. It is a diagram showing a third exemplary input image 71 processed by the image processing device 1 of FIG. 13. It is a diagram showing an exemplary output image 72 corresponding to the input image 61 of FIG. 13. 21 is a flowchart showing an image generation process executed by the processor 11B of FIG. 19; a flowchart showing a subroutine of step S12 (bottleneck detection process) of FIG. 21; a chart showing the operation of an exemplary manufacturing site analyzed by the image processing device according to the third embodiment; a diagram showing an exemplary image generated by the image processing device according to the third embodiment; a block diagram showing the configuration of a system including an image processing device 1C according to a fourth embodiment;26 is a flowchart showing a manufacturing equipment monitoring process executed by the processor 11C of FIG. 25.

[0010] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, in some cases, more detailed explanations than necessary will be omitted. For example, detailed explanations of already well-known matters and duplicate explanations of substantially identical configurations will be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art. Note that components with the same reference numerals have the same functions in each embodiment.

[0011] It should be noted that the present disclosure provides the accompanying drawings and the following description for the understanding of those skilled in the art, and is not intended to limit the subject matter described in the claims.

[0012] [First Embodiment] [Configuration of First Embodiment] Fig. 1 is a block diagram showing the configuration of a system including an image processing device 1 according to a first embodiment. The system in Fig. 1 includes an image processing device 1, a communication line 2, a manufacturing device 3, and a sensor 4. The image processing device 1 is connected to one or more sensors 4 provided in the manufacturing device 3 via the communication line 2. The manufacturing device 3 manufactures some kind of product. The sensor 4 acquires measured values ​​indicating the operating state of the manufacturing device 3, for example, whether the manufacturing device 3 is in a pause, normal operation, changeover, preparation, or emergency stop state. The image processing device 1 generates an image including visualization data indicating changes over time in the operating state of the manufacturing device 3 based on the measured values ​​acquired by the sensor 4. The image processing device 1 further determines whether the manufacturing device 3 is in a normal state or an abnormal state based on the generated image.

[0013] The image processing device 1 includes a bus 10, a processor 11, a memory 12, a storage device 13, a communication device 14, an input device 15, and a display device 16. The processor 11 controls the overall operation of the image processing device 1. The memory 12 temporarily stores programs and data necessary for the operation of the image processing device 1. The storage device 13 is a non-volatile storage medium that stores programs necessary for the operation of the image processing device 1. The communication device 14 is communicatively connected to the sensor 4 via the communication line 2. The input device 15 receives user input that controls the operation of the image processing device 1. The input device 15 includes, for example, a keyboard and a pointing device. The display device 16 displays information related to the operating status of the manufacturing apparatus 3. The processor 11, the memory 12, the storage device 13, the communication device 14, the input device 15, and the display device 16 are connected to each other via the bus 10.

[0014] Fig. 2 is a functional block diagram for explaining the operation of the image processing device 1 in Fig. 1. The processor 11 operates as an image input unit 21, a determination unit 22, a determination unit 23, and an image generation unit 24 by executing a program stored in the storage device 13.

[0015] The image input unit 21 acquires an image including visualization data. Here, visualization data includes data that visually represents numerical information, and includes, for example, a graph, a chart, a timeline, or a diagram. In the example of FIG. 2 , the image input unit 21 acquires measurement values ​​from the sensor 4 and generates an image including visualization data based on the measurement values. In this example, the visualization data is a graph showing changes over time in the operating state of the manufacturing apparatus 3. The image input unit 21 may also read an image including visualization data from the storage device 13 or from a storage device external to the image processing device 1. The image input unit 21 may also generate an image including visualization data based on data stored in the storage device 13 or based on data stored in a storage device external to the image processing device 1. In this specification, the image acquired by the image input unit 21 is referred to as an "input image."

[0016] The determination unit 22 obtains a classification result in which the input image is classified into one of a plurality of classes. The plurality of classes may include a normal state and an abnormal state of the manufacturing apparatus 3. The plurality of classes may also include a normal state and a plurality of abnormal states having different levels. The determination unit 22 may be trained in advance by machine learning to classify the input image into a plurality of classes.

[0017] The determination unit 23 acquires reference data from the determination unit 22 and, based on the reference data, determines a partial region of the input image that serves as the basis for determining the classification result. This partial region is an area that includes important features in the input image that significantly influenced the determination of the classification result. The determination unit 23 operates as explainable AI (artificial intelligence), allowing the user to understand the basis for the classification made by the determination unit 22.

[0018] The image generation unit 24 generates an output image by highlighting a partial region in the input image. The image generation unit 24 may, for example, generate a heat map indicating the degree to which each partial region of the input image influenced the determination of the classification result, and highlight the partial region by superimposing the heat map on the input image. Furthermore, the image generation unit 24 may superimpose the classification result of the input image on the input image.

[0019] The output image is displayed on the display device 16. The output image may also be output to a device external to the image processing device 1 via the communication device 14.

[0020] FIG. 3 is a diagram showing the configuration of the determination unit 22 in FIG. 2. The determination unit 22 may be configured as a neural network. The neural network in FIG. 3 includes nodes 31-1-1 to 31-1-M1 in an input layer 32-1, nodes 31-2-1 to 31-2-M2 in an intermediate layer 32-2, nodes 31-3-1 to 31-3-M3, ... in an intermediate layer 32-3, and nodes 31-N-1 to 31-N-M in an output layer 32-N. The nodes 31-N-1 to 31-N in the output layer 32-N correspond to multiple classes into which the input image is classified. The neural network is machine-trained in advance so that when pixel values ​​of an input image are input to the input layer 32-1, only the node in the output layer 32-N corresponding to the class of the input image becomes "1" and the other nodes become "0." Each node in the intermediate layer 32-2 is set with a weighting coefficient that has been machine-learned in advance so as to output "1" or "0" for the input photographed image.

[0021] FIG. 4 is a schematic diagram illustrating the operation of the determination unit 22 in FIG. 2 . The determination unit 22 may be configured to operate as a convolutional neural network as shown in FIG. 4 . In this case, the determination unit 22 includes a first convolutional layer 41, a first activation layer 42, a first pooling layer 43, a second convolutional layer 44, a second activation layer 45, a second pooling layer 46, a fully connected layer 47, and an output layer 48. An input image is input to the first convolutional layer 41, and a classification result is output from the output layer 48. The first convolutional layer 41 extracts features from the input image. The first activation layer 42 is provided after the first convolutional layer 41 and serves as a threshold function that removes noise from the features extracted by the first convolutional layer 41, leaving only strong values. The first pooling layer 43 is provided after the first activation layer 42 and performs downsampling to reduce the spatial resolution of the feature map by one level. The second convolutional layer 44 is provided after the first pooling layer 43 and extracts features over a wider range based on the features aggregated for each local region by the first pooling layer 43. The second activation layer 45 is provided after the second convolutional layer 44 and acts as a threshold function that removes noise by retaining only the strongest values ​​among the features extracted by the second convolutional layer 44. The second pooling layer 46 is provided after the second activation layer 45 and performs downsampling, which reduces the spatial resolution of the feature map by one level. The fully connected layer 47 is provided after the second pooling layer 46 and serves to further consolidate the features aggregated by the second pooling layer. The convolutional layer farthest from the first convolutional layer 41, the second convolutional layer 44 in this example, determines a map indicating where in the image each local pattern learned by the kernel is strongly present. The data of the second convolutional layer 44 is acquired as reference data by the determination unit 23. The reference data indicates the degree to which each pixel influenced the determination of the classification result, and is used to determine a partial region of the input image that serves as the basis for determining the classification result.

[0022] The determination unit 22 may operate as, for example, a Gradient-weighted Class Activation Mapping (Grad-CAM) convolutional neural network.

[0023] 4 illustrates a case where the determination unit 22 includes two sets of convolutional layers, activation layers, and pooling layers. However, the determination unit 22 may include three or more sets of convolutional layers, activation layers, and pooling layers.

[0024] The determination unit 22 is implemented by predetermined hardware, software, or a combination thereof so as to operate as the convolutional neural network of FIG.

[0025] [Operation of the First Embodiment] FIG. 5 is a flowchart showing an image generation process executed by the processor 11 of FIG.

[0026] In step S1, the processor 11 acquires an input image showing the change in the operating state of the manufacturing device 3 over time.

[0027] In step S2, the processor 11 processes the input image using a convolutional neural network to determine whether an abnormal state has occurred in the manufacturing equipment 3.

[0028] In step S3, the processor 11 outputs the abnormal state determination result to the display device 16.

[0029] In step S4, the processor 11 acquires reference data from the final convolutional layer of the convolutional neural network. The reference data includes a plurality of pixels arranged in a rectangle, and each pixel includes a numerical value indicating the degree to which it influenced the determination of the judgment result.

[0030] In step S5, the processor 11 generates a heat map by converting the numerical value of each pixel of the reference data into a predetermined color or pattern.

[0031] In step S6, the processor 11 changes the size of the heat map to match the input image.

[0032] In step S7, the processor 11 superimposes the heat map on the input image and outputs it.

[0033] When the determination unit 22 operates as a Grad-CAM type convolutional neural network, the processor 11 may operate as follows.

[0034] (1) Obtain a classification result that classifies the input image into one of multiple classes. (2) Calculate the loss function value in this case. (3) Calculate the gradient from the last convolutional layer to the loss function value. (4) Calculate the weighting coefficient by calculating the average of the gradients of the last convolutional layer, remove noise, and generate a heat map. (5) Superimpose the heat map on the input image.

[0035] The processor 11 may display the input image, the determination result, and the heat map together on the display device 16, as will be described later with reference to FIGS.

[0036] 6 is a diagram showing a first exemplary input image 51 processed by the image processing device 1 of FIG. 1. The input image 51 includes a graph showing temporal changes in the operating state of the manufacturing equipment 3, including pauses, normal operation, changeover, preparation, and emergency stop of the manufacturing equipment 3. In the example of FIG. 6, the sensor 4 includes a human presence sensor, and the input image 51 further indicates whether or not a worker is present near the manufacturing equipment 3.

[0037] When processing an input image 51 including a graph showing the change over time in the operating state of the manufacturing equipment 3, as shown in Figure 6, the judgment unit 22 may be machine-learned in advance by a person with sufficient knowledge and experience regarding the operating state of the manufacturing equipment 3 according to one or more of the following criteria.

[0038] - If an "emergency shutdown" occurs in the manufacturing equipment 3, it is an abnormal state, and if no "emergency shutdown" occurs in the manufacturing equipment 3, it is a normal state. - If the time taken to "switch" the manufacturing equipment 3 is equal to or greater than the threshold, it is an abnormal state, and if the time taken to "switch" the manufacturing equipment 3 is shorter than the threshold, it is a normal state. - If "switching" of the manufacturing equipment 3 occurs at an unscheduled time, it is an abnormal state, and if "switching" of the manufacturing equipment 3 does not occur at an unscheduled time, it is a normal state. - If the time taken to "prepare" the manufacturing equipment 3 is equal to or greater than the threshold, it is an abnormal state, and if the time taken to "prepare" the manufacturing equipment 3 is shorter than the threshold, it is a normal state. - If "preparation" of the manufacturing equipment 3 occurs at an unscheduled time, it is an abnormal state, and if "preparation" of the manufacturing equipment 3 does not occur at an unscheduled time, it is a normal state.

[0039] 7 is a diagram showing reference data 52 acquired by the determination unit 23 when processing the input image 51 of FIG. 6. The reference data 52 includes a plurality of pixels arranged in a rectangle, and each pixel includes a numerical value indicating the degree to which it influenced the determination of the determination result. The larger the numerical value of a pixel, the greater the influence it had on the determination of the determination result.

[0040] Fig. 8 is a diagram showing a heat map 53 generated based on the reference data 52 of Fig. 7. The heat map 53 is generated by converting the numerical value of each pixel of the reference data 52 into a predetermined color or pattern.

[0041] 9 is a diagram showing a heatmap 54 generated based on the heatmap 53 of FIG. 8, the heatmap 54 having a size changed to fit the input image 51. To reshape the heatmap 53, for example, the CV2.resize function of OpenCV-Python may be used.

[0042] Fig. 10 is a diagram showing an output image 55 generated by superimposing the heat map 54 of Fig. 9 on the input image 51 of Fig. 1. Fig. 10 shows a case where it is determined that the manufacturing equipment 3 is in an abnormal state due to an "emergency shutdown" of the manufacturing equipment 3, and a partial region indicating the "emergency shutdown" in the input image 51 is highlighted. In the heat map 54, the vicinity of the partial region indicating the "emergency shutdown" is brighter than other parts and is highlighted. The output image 55 may include an alert 56 indicating the determination result.

[0043] The partial region of the input image that is the basis for determining the classification result may be highlighted in any format, such as a box or an arrow, instead of a heat map.

[0044] According to this embodiment, the image processing device 1 outputs a determination result indicating whether the manufacturing device 3 is in a normal state or an abnormal state, and also highlights a partial area of ​​the input image that is the basis for determining the determination result, thereby supporting analysis of graphs by personnel with insufficient experience or knowledge. Use of the image processing device 1 according to this embodiment makes it possible to reduce the occurrence of errors in analysis.

[0045] According to this embodiment, the user can easily understand which parts of the graph are important and require attention. Since it is clear at a glance what parts of the graph to look at, the burden and man-hours on the analyst can be significantly reduced.

[0046] An analyst with sufficient knowledge and experience can extract meaningful information from a complex graph, such as "the worker's movements are suspicious during this time period" or "there is a tendency for problems to occur with a certain manufacturing machine." By having the determination unit 22 learn the know-how of such analysts through machine learning in advance, it is possible to prevent analysis from becoming dependent on one individual, and it is expected that this will lead to the development of human resources who can analyze complex graphs.

[0047] [Modification of First Embodiment] The image processing device 1 is not limited to processing the input image 51 shown in FIG. 6, and may be configured to process an input image including visualization data in another format.

[0048] 11 is a diagram showing a second exemplary input image 61 processed by the image processing device 1 of FIG. 1. The input image 61 includes, for example, a graph showing temporal changes in the daily production amounts of products A to C. In this case, the image processing device 1 is connected to a plurality of sensors 4, and the sensors 4 detect the production amounts of products A to C, respectively. The determination unit 22 has been trained in advance by machine learning to classify whether the production amounts are in a normal state or an abnormal state.

[0049] Figure 12 is a diagram showing an exemplary output image 62 corresponding to the input image 61 of Figure 11. The output image 62 includes highlights 63 and 64 indicating the occurrence of a sudden decrease in production volume as an abnormal condition. The output image 62 may also include an alert 65 indicating the determination result.

[0050] 13 is a diagram showing a third exemplary input image 71 processed by the image processing device 1 of FIG. 1. The input image 71 includes a map showing changes in the position of a person or object over time. In this case, the image processing device 1 is connected to one or more sensors 4, and the sensors 4 detect the position of the person or object. The determination unit 22 has been trained in advance by machine learning to classify the position of the person or object as being in a normal state or an abnormal state.

[0051] Figure 14 is a diagram showing an exemplary output image 72 corresponding to the input image 61 of Figure 13. The output image 72 includes a highlight 73 indicating that the person or object has meandered as an abnormal condition. The output image 72 may also include an alert 74 indicating the determination result.

[0052] 15 is a diagram showing a fourth exemplary input image 81 processed by the image processing device 1 of FIG. 1. The input image 81 includes a map showing the stay time of a person or object at each location. The diameter of the circle indicates the length of the stay time. In this case, the image processing device 1 is connected to one or more sensors 4, and the sensors 4 detect the location and stay time of the person or object. The determination unit 22 has been trained in advance by machine learning to classify the stay time of the person or object at each location as being in a normal state or an abnormal state.

[0053] Fig. 16 is a diagram showing an exemplary output image 82 corresponding to the input image 61 of Fig. 15. The output image 82 includes highlighting 83, 84 indicating that a person or object has remained at a certain position for a period longer than a threshold, which is an abnormal state. The output image 82 may also include an alert 85 indicating the determination result.

[0054] By processing images such as those shown in FIGS. 6 to 16 , it is possible to appropriately grasp the status of a site, including a manufacturing site or a logistics site. By processing input image 51 including visualization data showing temporal changes in the operating status of manufacturing equipment 3, and by processing input image 61 including visualization data showing temporal changes in production volume, it becomes easier to grasp the status of a site related to manufacturing equipment 3. Furthermore, by processing input image 71 including visualization data showing temporal changes in the location of people, such as workers, and by processing input image 81 including a graph showing the time spent by people at each location, it becomes easier to grasp the status of a site related to people. Furthermore, by processing input image 71 including visualization data showing temporal changes in the location of objects, such as manufactured products, and by processing input image 81 including visualization data showing the time spent by objects at each location, it becomes easier to grasp the status of a site related to objects.

[0055] The image processing device 1 may be configured to process any other input image, not limited to the images shown in FIGS. 6 to 16 . The input image may include any other type of graph, such as a bar graph or a pie chart. The input image may include, for example, a graph showing temporal changes in stock prices, frequency, or power usage. The input image may include any type of chart, an image in which data is superimposed on a map, or a timeline in any type of timeline. The input image may also be an image in which information such as numerical values ​​or statistical data is converted into a visual representation. In this case, the determination unit 22 has been trained in advance by machine learning to classify the input image into multiple classes depending on the content of the input image.

[0056] [Effects of the first embodiment, etc.] The image processing device 1 according to the first embodiment includes an image input unit 21 that acquires an input image including visualization data, a first determination unit 22 that acquires a classification result that classifies the input image into one of a plurality of classes, a second determination unit 23 that determines a partial region of the input image that serves as the basis for determining the classification result, an image generation unit 24 that generates an output image by highlighting the partial region in the input image, and an image output unit that outputs the output image.

[0057] This can assist personnel with insufficient experience or knowledge in analyzing visualized data.

[0058] According to the image processing device 1 of the first embodiment, the first determination unit 22 may be a convolutional neural network including a plurality of convolutional layers, and may include a convolutional neural network that has been machine-learned to classify an input image into a plurality of classes. The second determination unit 23 may determine a partial region based on data of the last convolutional layer of the convolutional neural network.

[0059] This makes it possible to classify the input image and present the basis for the classification.

[0060] According to the image processing device 1 of the first embodiment, the visualized data may include data that visually represents information based on numbers.

[0061] This allows you to process visualization data that includes numerical information.

[0062] According to the image processing device 1 according to the first embodiment, the visualization data may include information indicating the state of a site, such as a manufacturing site or a logistics site.

[0063] This allows the status of manufacturing or logistics sites to be represented using visualized data, making it possible to visualize issues at the sites.

[0064] According to the image processing device 1 of the first embodiment, the input image may include a graph representing one of the following: the change in the operating status of a machine over time, the change in the production volume of a product over time, the change in the position of a person or object over time, and the stay time of a person or object at each position.

[0065] This allows the status of manufacturing or logistics sites to be represented using visualized data, making it possible to visualize issues at the sites.

[0066] According to the image processing device 1 of the first embodiment, the multiple classes may include a normal state and an abnormal state, or may include a normal state and multiple abnormal states having different levels.

[0067] This allows the on-site conditions of manufacturing, logistics, etc. to be properly understood.

[0068] According to the image processing device 1 of the first embodiment, the image input unit 21 may acquire measurement values ​​from the sensor 4 and generate an input image including visualization data based on the measurement values.

[0069] This makes it easier to properly grasp the status of manufacturing or logistics sites.

[0070] According to the image processing device 1 according to the first embodiment, the image output unit may output the classification result.

[0071] This allows the user to understand the classification results and the basis for determining the classification results.

[0072] An image processing method according to a first embodiment processes an input image including visualization data by a computer, the method including: acquiring the input image; acquiring a classification result that classifies the input image into one of a plurality of classes; determining a subregion of the input image that serves as a basis for determining the classification result; generating an output image by highlighting the subregion in the input image; and outputting the output image.

[0073] This can assist personnel with insufficient experience or knowledge in analyzing the graphs.

[0074] A program according to a first embodiment includes instructions executed by a processor implemented in a computer that processes an input image including visualization data, the instructions causing the processor to acquire the input image, acquire a classification result that classifies the input image into one of a plurality of classes, determine a subregion of the input image that serves as a basis for determining the classification result, generate an output image by highlighting the subregion in the input image, and output the output image.

[0075] This can assist personnel with insufficient experience or knowledge in analyzing visualized data.

[0076] Second Embodiment Fig. 17 is a block diagram showing the configuration of a system including an image processing device 1A according to a second embodiment. The system in Fig. 17 includes the image processing device 1A, a communication line 2, a manufacturing device 3, a sensor 4, and a server device 5. For convenience of explanation, the image processing device 1A includes a processor 11A instead of the processor 11 in Fig. 1. The image processing device 1A is connected to the server device 5 via the communication line 2. The server device 5 includes a convolutional neural network that has been machine-trained to classify input images into multiple classes. The image processing device 1A does not internally generate classification results and reference data, but instead obtains them from the server device 5.

[0077] FIG. 18 is a functional block diagram for explaining the operation of the image processing device 1A in FIG.

[0078] The processor 11A executes a program stored in the storage device 13 to operate as an image input unit 21, a determination unit 22A, a determination unit 23, and an image generation unit 24. The input image acquired by the image input unit 21 is sent to the server device 5.

[0079] The server device 5 includes a determination unit 91 configured similarly to the determination unit 22 in Fig. 2. The determination unit 91 has been trained in advance by machine learning to classify input images into a plurality of classes, similar to the determination unit 22 in Fig. 2.

[0080] The determination unit 22A acquires a classification result in which the input image is classified into one of a plurality of classes from the determination unit 91. The determination unit 23 acquires reference data from the determination unit 91 and determines a partial region of the input image that will be the basis for determining the classification result, based on the reference data.

[0081] The image generating unit 24 in FIG. 18 generates an output image by highlighting a partial region in an input image, similar to the image generating unit 24 in FIG.

[0082] By having the server device 5 classify the input image using machine learning, the image processing device 1A can reduce the processing load, power consumption, and program size of the image processing device 1A compared to the image processing device 1 of the first embodiment.

[0083] [Effects of the Second Embodiment] The image processing device 1A according to the second embodiment may further include a communication device 16 that communicates with an external device. The external device includes a server device 5 that includes a convolutional neural network including multiple convolutional layers, the convolutional neural network having been machine-learned to classify an input image into multiple classes. In this case, the first determination unit 22 sends the input image to the server device 5 and obtains the classification result from the server device 5. Furthermore, the second determination unit 23 obtains data of the last convolutional layer of the convolutional neural network from the server device 5 and determines a partial region based on the data of the last convolutional layer of the convolutional neural network.

[0084] This allows the processing load, power consumption, and program size of the image processing device 1A to be reduced compared to the image processing device 1 of the first embodiment.

[0085] In the first and second embodiments, a case has been described in which it is determined whether a manufacturing device is in a normal state or an abnormal state based on an image including visualization data showing a change over time in the operating state of the manufacturing device. In the third embodiment, a case will be described in which a bottleneck in a manufacturing site including a plurality of processes is identified based on numerical data showing a change over time in the state of the manufacturing site.

[0086] [Configuration of the Third Embodiment] Fig. 19 is a functional block diagram for explaining the operation of an image processing device according to the third embodiment. The image processing device according to the third embodiment has the same configuration as the image processing device 1 in Fig. 1. For convenience of explanation, the image processing device according to the third embodiment includes a processor 11B instead of the processor 11 in Fig. 1. The processor 11B executes programs stored in the storage device 13 to operate as a data input unit 21B, a determination unit 21B, a determination unit 23B, and an image generation unit 24B.

[0087] The data input unit 21B acquires input data including numerical data. In the example of FIG. 19 , the data input unit 21B acquires measurement values ​​from the sensor 4 and generates input data including numerical data in a predetermined format based on the measurement values. In this example, the input data may include the start and end times of work for each process in a manufacturing site including multiple processes. The data input unit 21B may also read the input data including numerical data from the storage device 13 or from a storage device external to the image processing device.

[0088] The determination unit 22B obtains a classification result in which the input data is classified into one of a plurality of classes. The plurality of classes may include whether each process at the manufacturing site is in a bottleneck state or not in a bottleneck state. The determination unit 22B may determine whether a first process is in a bottleneck state or not based on, for example, the start and end times of a first process among the plurality of processes, a second process preceding the first process, and a third process following the first process.

[0089] The determination unit 23B acquires the reference data from the determination unit 22B and determines, based on the reference data, a subelement of the input data that serves as the basis for determining the classification result. This subelement is an area that includes an important feature in the input data and a feature that significantly influenced the determination of the classification result.

[0090] The image generation unit 24B converts the input data into visualized data and generates an output image including the visualized data, the classification results, and the subelements.

[0091] The output image is displayed on the display device 16. The output image may also be output to a device external to the image processing device 1 via the communication device 14.

[0092] 20 is a diagram showing an exemplary manufacturing site analyzed by an image processing device according to the third embodiment. As described above, the manufacturing site includes multiple processes, i.e., process 1, process 2, process 3, .... Each process may include a separate manufacturing device. Between each process, buffers 1, 2, 3, ... are provided to temporarily hold articles completed in the previous process before being sent to the next process.

[0093] FIG. 21 is a flowchart showing the image generation process executed by the processor 11B of FIG.

[0094] In step S11, the processor 11B acquires input data including the operating status of the manufacturing site, that is, the start and end times of work for each process.

[0095] In step S12, processor 11B executes a bottleneck detection process to obtain classification results that classify each process as being in a bottleneck state or not, and also determines the partial elements of the input data that serve as the basis for determining the classification results.

[0096] In step S13, the processor 11B generates an image showing the operating status of the manufacturing site.

[0097] In step S14, the processor 11B outputs the bottleneck position and the reason for the bottleneck by superimposing them on the image.

[0098] FIG. 22 is a flowchart showing a subroutine of step S12 (bottleneck detection processing) in FIG.

[0099] In step S21, the processor 11B selects one of a plurality of processes at the manufacturing site, for example, process n.

[0100] In step S22, the processor 11B calculates a bottleneck score Sbn(n) for the current process n based on the start and end times of work on multiple items in the selected process n and the processes n-1 and n+1 before and after it. To calculate the bottleneck score Sbn(n), for example, the processor 11B first calculates the residence rates bst(n-1,i) and bst(n,i) of items in buffer n-1 and buffer n before and after process n, respectively, using the following equation:

[0101] bst(n-1,i)=(pe(n-1,i)-ps(n,i)) / T0 bst(n,i)=(pe(n,i)-ps(n+1,i)) / T0

[0102] Here, i represents an identification number that uniquely identifies an individual item processed in each process. pe(n-1,i) represents the time when work on item i in process n-1 is completed, and ps(n,i) represents the time when work on item i in process n begins. T0 represents the total length of time of interest. Also, pe(n,i) represents the time when work on item i in process n is completed, and ps(n+1,i) represents the time when work on item i in process n+1 begins.

[0103] Processor 11B then calculates the delay index d(n,i) for the work on item i in process n using the following formula:

[0104] d(n,i)=bst(n-1,i)-bst(n,i)

[0105] The delay index d(n, i) indicates the amount of articles accumulated upstream of process n. Therefore, the larger the delay index d(n, i), the more serious the bottleneck that is considered to be.

[0106] Next, the processor 11B calculates the bottleneck score Sbn(n) using the following formula:

[0107] Sbn(n) = Σ i (α×d(n,i))

[0108] Here, α is a predetermined coefficient. The larger the coefficient α, the more likely it is that a delay in work will be determined to be a bottleneck, and the smaller the coefficient α, the less likely it is that a delay in work will be determined to be a bottleneck.

[0109] In step S23, the processor 11B determines whether the bottleneck score Sbn(n) is higher than a predetermined threshold value th, and if YES, proceeds to step S24, and if NO, proceeds to step S38.

[0110] In step S24, processor 11B selects one of the multiple articles to be processed in step n, for example, article i.

[0111] In step S25, the processor 11B calculates parameters bP(n, i), aP(n, i), and pD(n, i) based on the start and end times of work on item i in the selected process n and the processes n-1 and n+1 before and after it. bP(n, i) indicates whether work on item i in process n-1, which is immediately before the current process n, has been ongoing for a long time (i.e., beyond a predetermined threshold time). aP(n, i) indicates whether the start of work on item i in process n+1, which is immediately after the current process n, has been pending for a long time (i.e., beyond a predetermined threshold time). pD(n, i) indicates whether work on item i in the current process n has been ongoing for a long time (i.e., beyond a predetermined threshold time).

[0112] In step S26, the processor 11B determines whether bP(n, i)=TRUE and aP(n, i)=TRUE, and if YES, proceeds to step S27, and if NO, proceeds to step S30.

[0113] In step S27, the processor 11B determines whether pD(n, i)=TRUE, and if YES, proceeds to step S28, and if NO, proceeds to step S29.

[0114] In step S28, processor 11B determines that the work on item i in process n is taking an excessively long time, and that the current process n is a bottleneck that should be checked with high priority (i.e., is likely to be serious). In step S29, processor 11B determines that a discrepancy has occurred in the cycle time (i.e., the length of time from the start to the completion of work on item i) between the current process n and the previous and subsequent processes n-1 and n+1, and that the current process n is a bottleneck that should be checked with high priority.

[0115] In step S30, the processor 11B determines whether bP(n, i)=TRUE or aP(n, i)=TRUE, and if YES, proceeds to step S31, and if NO, proceeds to step S34.

[0116] In step S31, the processor 11B determines whether pD(n, i)=TRUE, and if YES, proceeds to step S32, and if NO, proceeds to step S33.

[0117] In step S32, the processor 11B determines that there may be some kind of trouble occurring in connection with the work on item i in process n, and that the current process n is a bottleneck that should be checked with medium priority. Examples of trouble include the absence of a worker, or the worker performing work other than the work for process n. In step S33, the processor 11B determines that a mismatch in cycle time has occurred between the current process n and the preceding or following processes n-1 or n+1, and that the current process n is a bottleneck that should be checked with medium priority.

[0118] In step S34, the processor 11B determines that the current process n is a bottleneck that should be checked with low priority.

[0119] In step S35, the processor 11B extracts the reason corresponding to the detected bottleneck. The reason for the bottleneck includes, for example, a delay indicator d(n, i) of the work for the item i in the process n.

[0120] In step S36, processor 11B determines whether or not the presence or absence of a bottleneck has been determined for all items processed in process n, and if YES, proceeds to step S38, and if NO, proceeds to step S37.

[0121] In step S37, processor 11B selects another one of the multiple articles to be processed in process n, and repeats steps S25 to S36.

[0122] In step S38, the processor 11B determines whether or not the presence or absence of a bottleneck has been determined for all processes, and if YES, proceeds to step S13 in FIG. 21, and if NO, proceeds to step S39 in FIG.

[0123] In step S39, the processor 11B selects another one of the multiple processes at the manufacturing site and repeats steps S22 to S38.

[0124] The bottleneck detection process in Figure 22 makes it possible to detect in which process at the manufacturing site a bottleneck occurred and when work was being performed on which item, and also to present the basis for determining that a bottleneck occurred.

[0125] Fig. 23 is a chart showing an exemplary operation of a manufacturing site analyzed by an image processing device according to the third embodiment. In step S13 of Fig. 21, the processor 11B may generate a chart such as that shown in Fig. 23 as an image showing the operating status of the manufacturing site. Section 101 of each process 1 to 14 indicates the duration from the start to the end of work on one item in that process.

[0126] FIG. 24 is a diagram showing an exemplary image generated by an image processing device according to the third embodiment. A threshold time indicating the upper limit of the time required to perform work in that process on one item is set for each process. A section 101 having a time length exceeding the threshold time may be represented by a different color or pattern from a section 101 having a time length equal to or less than the threshold time. In the example of FIG. 24, such a section is indicated by reference numeral 102. Reference numeral 103 indicates a bottleneck detected in steps S28, S29, S32, and S33 of FIG. 22. Display of a bottleneck detected in step S34 of FIG. 22 may be omitted. Reference numeral 104 indicates a delay indicator corresponding to the bottleneck. Changes in the magnitude of the delay indicator may be represented by different colors. Reference numeral 105 indicates room for improvement in work in the process.

[0127] The delay indicator 104 is presented as a subelement of input data that serves as a basis for determining whether each process is in a bottleneck state or not. By displaying the delay indicator 104 near the bottleneck 103, the user can understand the basis for detecting the bottleneck 103.

[0128] The improvement margin IM(n) of the process n is calculated using, for example, the following formula:

[0129] IM(n)=(ti(n)+to(n)) / (p2(n)-p1(n))×100

[0130] Here, ti(n) indicates the total length of time that items are waiting to be worked on. to(n) indicates the total amount of time that the duration of work for each item exceeds the threshold time. p1(n) indicates the start time of all operations in process n. p2(n) indicates the end time of all operations in process n.

[0131] According to this embodiment, the image processing device outputs a determination result indicating whether each process in a manufacturing site is in a bottleneck state or not, and also presents partial elements of the numerical data that are the basis for determining the determination result, thereby supporting analysis of numerical data by personnel with insufficient experience or knowledge. Use of the image processing device according to this embodiment makes it possible to reduce the occurrence of errors in analysis.

[0132] [Effects of the Third Embodiment, etc.] The image processing device according to the third embodiment includes a data input unit 21B that acquires input data including numerical data, a first determination unit 22B that acquires a classification result in which the input data is classified into one of a plurality of classes, a second determination unit 23B that determines partial elements of the input data that serve as the basis for determining the classification result, an image generation unit 24B that converts the input data into visualized data and generates an output image including the visualized data, the classification result, and the partial elements, and an image output unit that outputs the output image.

[0133] This can assist personnel with insufficient experience or knowledge in analyzing the numerical data.

[0134] According to the image processing device of the third embodiment, the numerical data may include information indicating the status of a manufacturing site including a plurality of processes, and the plurality of classes may include information indicating whether each process is in a bottleneck state or not.

[0135] This makes it possible to identify bottlenecks at the manufacturing site based on numerical data indicating the status of the manufacturing site.

[0136] According to the image processing device of the third embodiment, the information indicating the status of the manufacturing site may include the start time and end time of work for each process. The first determination unit 22B may determine whether the first process is in a bottleneck state or not based on the start time and end time of a first process among the multiple processes, a second process preceding the first process, and a third process following the first process.

[0137] This makes it possible to identify bottlenecks at the manufacturing site based on numerical data indicating the status of the manufacturing site.

[0138] Fourth Embodiment In a fourth embodiment, a case will be described in which a manufacturing device 3 is monitored using the image processing device according to the first to third embodiments.

[0139] [Configuration of Fourth Embodiment] Fig. 25 is a block diagram showing the configuration of a system including an image processing device 1C according to a fourth embodiment. The system in Fig. 25 includes an image processing device 1C, a communication line 2, a manufacturing device 3, a sensor 4, and a terminal device 6. For convenience of explanation, the image processing device 1C includes a processor 11C instead of the processor 11 in Fig. 1 (or the processor 11A in Fig. 17 or the processor 11B in Fig. 19). The image processing device 1C is connected to the terminal device 6 via the communication line 2. The terminal device 6 is, for example, a personal computer or a mobile phone used by an administrator of the manufacturing device 3. The terminal device 6 includes an output device such as a display or a speaker.

[0140] [Operation of Fourth Embodiment] FIG. 26 is a flowchart showing a manufacturing equipment monitoring process executed by the processor 11C of FIG.

[0141] In step S41, the processor 11C executes the image generation process of FIG. 5 or FIG.

[0142] When the processor 11C executes the image generation process of Fig. 5, the processor 11C determines whether an abnormal state has occurred in the manufacturing equipment 3, as described in relation to step S2 of Fig. 5. Furthermore, when the processor 11C executes the image generation process of Fig. 21, the processor 11C calculates the bottleneck score Sbn(n) as described in relation to step S22 of Fig. 22, and also calculates the room for improvement IM(n) as described in relation to Fig. 24. When the bottleneck score Sbn(n) is higher than a predetermined threshold value and / or when the room for improvement IM(n) is higher than a predetermined threshold value, the processor 11C may determine that an abnormal state has occurred in the manufacturing equipment 3.

[0143] In step S42, the processor 11C determines whether or not an abnormal state has occurred in the manufacturing device 3, and if YES, proceeds to step S43, and if NO, ends the process.

[0144] In step S43, the processor 11C generates an alert signal.

[0145] In step S44, the processor 11C transmits an alert signal to the terminal device 6.

[0146] The terminal device 6 that receives the alert signal notifies the manager that an abnormal state has occurred in the manufacturing equipment 3. This allows the manager to grasp and analyze the occurrence of the abnormal state and implement business improvements without having to directly monitor the manufacturing equipment 3 in the vicinity.

[0147] [Advantages of the Fourth Embodiment] The image processing device 1C according to the fourth embodiment may further include a communication device 16 that communicates with the terminal device 6. In this case, the first determination unit generates an alert signal based on the classification result. The communication device 16 transmits the alert signal to the terminal device 6.

[0148] With this configuration, the manager can grasp and analyze the occurrence of an abnormal state and implement operational improvements without having to directly monitor the manufacturing equipment 3 in its vicinity.

[0149] [Other Embodiments] As described above, the input image may be classified into multiple classes, and the multiple classes may include, for example, a normal state and multiple abnormal states having different levels. For example, if the input image includes visualization data showing temporal changes in the operating state of the manufacturing device 3, when the time required for "switching" is equal to or greater than a first threshold, for example, several minutes, it may be determined that the state is in a "small-scale abnormal state." When the time required for "switching" is equal to or greater than a second threshold, for example, several hours, it may be determined that the state is in a "large-scale abnormal state." Abnormal states of different levels may be highlighted using different colors or patterns. Depending on the different levels of abnormal states, the image processing device 1 may generate messages such as "Watch," "Caution Required," and "Alert."

[0150] When the image processing device obtains the classification result of the input image, the image processing device may enlarge and display the vicinity of the partial region of the input image that is the basis for determining the classification result on the display device 16. This allows the user to more clearly understand the basis for determining the classification result.

[0151] When processing the input image 51 in Figure 6, the image processing device may determine whether the manufacturing equipment 3 is in a normal state or an abnormal state by taking into consideration the presence of an operator in addition to the operating state of the manufacturing equipment 3. For example, even if the time required for "switching" is shorter than a threshold value, if there is no operator near the manufacturing equipment 3, it may be determined that the manufacturing equipment 3 is in an abnormal state. Also, even if the time required for "preparation" is equal to or longer than a threshold value, if there is an operator near the manufacturing equipment 3, it may be determined that the manufacturing equipment 3 is in a normal state.

[0152] The image processing apparatuses according to the first and second embodiments can be applied to cases where a single input image includes multiple graphs, charts, and / or diagrams. For example, the image processing apparatuses according to the first and second embodiments can be applied to cases where the input image includes multiple graphs each showing the operating status of multiple manufacturing devices included in a single factory. Even when the input image includes multiple graphs, the determination unit 22 performs machine learning in advance, as in the case of an input image including a single graph.

[0153] Instead of the final convolutional layer of the convolutional neural network, the determination unit 23 may obtain reference data from another layer that indicates the degree to which each pixel influenced the determination of the classification result and includes data that can be used to determine a partial region of the input image that serves as the basis for determining the classification result.

[0154] During operation of the image processing device 1, the determination unit 22 may additionally perform machine learning based on the input image and user input.

[0155] [Summary of the embodiment] According to a first aspect of the present disclosure, an image processing device includes: an image input unit that acquires an input image including visualization data; a first determination unit that acquires a classification result that classifies the input image into one of a plurality of classes; a second determination unit that determines a partial region of the input image that serves as a basis for determining the classification result; an image generation unit that generates an output image by highlighting the partial region in the input image; and an image output unit that outputs the output image.

[0156] According to a second aspect of the present disclosure, in the first aspect, the first determination unit is a convolutional neural network including a plurality of convolutional layers, the convolutional neural network being machine-learned to classify the input image into the plurality of classes, and the second determination unit determines the partial region based on data of a final convolutional layer of the convolutional neural network.

[0157] According to a third aspect of the present disclosure, in the first aspect, the image processing device further includes a communication unit that communicates with a first external device, the first external device including a server device having a convolutional neural network including a plurality of convolutional layers, the convolutional neural network having been machine-learned to classify the input image into the plurality of classes, the first determination unit sending the input image to the server device and obtaining the classification result from the server device, and the second determination unit obtaining data of a final convolutional layer of the convolutional neural network from the server device and determining the partial region based on the data of the final convolutional layer of the convolutional neural network.

[0158] According to a fourth aspect of the present disclosure, in one of the first to third aspects, the visualized data includes data that visually represents numerical information.

[0159] According to a fifth aspect of the present disclosure, in one of the first to fourth aspects, the visualization data includes information indicating the status of a site, including a manufacturing site or a logistics site.

[0160] According to a sixth aspect of the present disclosure, in one of the first to fifth aspects, the input image includes a graph representing a change over time in the operating state of the machine.

[0161] According to a seventh aspect of the present disclosure, in one of the first to sixth aspects, the input image includes a graph representing a change in production volume of a product over time.

[0162] According to an eighth aspect of the present disclosure, in one of the first to seventh aspects, the input image includes a graph representing a change in the position of a person or an object over time.

[0163] According to a ninth aspect of the present disclosure, in one of the first to eighth aspects, the input image includes a graph representing the stay time of a person or object at each location.

[0164] According to a tenth aspect of the present disclosure, in one of the first to ninth aspects, the plurality of classes include a normal state and an abnormal state, or include a normal state and a plurality of abnormal states having different levels from each other.

[0165] According to an eleventh aspect of the present disclosure, in one of the first to tenth aspects, the image input unit acquires measurement values ​​from a sensor, and generates an input image including the graph based on the measurement values.

[0166] According to a twelfth aspect of the present disclosure, in one of the first to eleventh aspects, the image output unit outputs the classification result.

[0167] According to a thirteenth aspect of the present disclosure, in one of the first to twelfth aspects, the image processing device further includes a communication unit that communicates with a second external device, the first determination unit generates an alert signal based on the classification result, and the communication unit transmits the alert signal to the second external device.

[0168] According to a fourteenth aspect of the present disclosure, an image processing device includes: a data input unit that acquires input data including numerical data; a first determination unit that acquires a classification result that classifies the input data into one of a plurality of classes; a second determination unit that determines a partial element of the input data that serves as a basis for determining the classification result; an image generation unit that converts the input data into visualized data and generates an output image that includes the visualized data, the classification result, and the partial element; and an image output unit that outputs the output image.

[0169] According to the fifteenth aspect of the present disclosure, in the fourteenth aspect, the visualization data includes information indicating the state of a manufacturing site including a plurality of processes, and the plurality of classes include whether each of the processes is in a bottleneck state or not.

[0170] According to a sixteenth aspect of the present disclosure, in the fifteenth aspect, the information indicating the status of the manufacturing site includes a start time and an end time of work for each of the processes, and the first judgment unit determines whether the first process is in a bottleneck state or not based on the start time and end time of a first process among the plurality of processes, a second process preceding the first process, and a third process following the first process.

[0171] According to a 17th aspect of the present disclosure, in one of the 14th to 16th aspects, the image processing device further includes a communication unit that communicates with an external device, the first determination unit generates an alert signal based on the classification result, and the communication unit transmits the alert signal to the external device.

[0172] According to an eighteenth aspect of the present disclosure, there is provided an image processing method for processing an input image including a graph by a computer, comprising: acquiring the input image; acquiring a classification result that classifies the input image into one of a plurality of classes; determining a subregion of the input image that serves as a basis for determining the classification result; generating an output image by highlighting the subregion in the input image; and outputting the output image.

[0173] According to a nineteenth aspect of the present disclosure, there is provided a program including instructions executed by a processor implemented in a computer that processes an input image including visualization data, the instructions causing the processor to: acquire the input image; acquire a classification result that classifies the input image into one of a plurality of classes; determine a subregion of the input image that serves as a basis for determining the classification result; generate an output image by highlighting the subregion in the input image; and output the output image.

[0174] The image processing device according to the present disclosure is applicable to a business intelligence tool that analyzes visualized data for the purpose of business improvement, etc.

[0175] DESCRIPTION OF SYMBOLS 1, 1A, 1C Image processing device 2 Communication line 3 Manufacturing device 4 Sensor 5 Server device 6 Terminal device 10 Bus 11, 11A, 11B, 11C Processor 12 Memory 13 Storage device 14 Communication device 15 Input device 16 Display device 21 Image input unit 22, 22A Determination unit (class) 23 Determination unit (basis of classification) 24 Image generation unit 21B Data input unit 22B Determination unit (bottleneck) 23B Determination unit (basis of classification) 24B Image generation unit 31-1-1 to 31-N-M Nodes 32-1 to 32-N Layer 41 First convolution layer 42 First activation layer 43 First pooling layer 44 Second convolution layer 45 Second activation layer 46 Second pooling layer 47 Fully connected layer 48 Output layer 51 Input image 52 Reference data 53 Heat map 54 Deformed heat map 55 Output image 56 Alert 61 Input image 62 Output image 63, 64 Highlighted display 65 Alert 71 Input image 72 Output image 73 Highlighted display 74 Alert 81 Input image 82 Output image 83, 84 Highlighted display 85 Alert 91 Judgment unit (class)

Claims

1. An image processing device comprising: an image input unit that acquires an input image including visualization data; a first determination unit that acquires a classification result that classifies the input image into one of a plurality of classes; a second determination unit that determines a partial region of the input image that serves as a basis for determining the classification result; an image generation unit that generates an output image by highlighting the partial region in the input image; and an image output unit that outputs the output image.

2. The image processing device according to claim 1, wherein the first determination unit is a convolutional neural network including a plurality of convolutional layers and is machine-trained to classify the input image into the plurality of classes, and the second determination unit determines the partial region based on data of a final convolutional layer of the convolutional neural network.

3. The image processing device according to claim 1, further comprising a communication unit for communicating with a first external device, the first external device including a server device having a convolutional neural network including a plurality of convolutional layers, the convolutional neural network being machine-learned to classify the input image into the plurality of classes, the first determination unit sending the input image to the server device and obtaining the classification result from the server device, and the second determination unit obtaining data of a final convolutional layer of the convolutional neural network from the server device and determining the partial region based on the data of the final convolutional layer of the convolutional neural network.

4. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes data that visually represents numerical information.

5. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes information indicating the state of a site, including a manufacturing site or a logistics site.

6. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes a graph showing a change over time in the operating state of a machine.

7. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes a graph showing a change in production volume of a product over time.

8. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes a graph showing the change over time in the position of a person or object.

9. The image processing device according to any one of claims 1 to 3, wherein the visualization data includes a graph showing one of the dwell times of people or objects for each location.

10. The image processing device according to any one of claims 1 to 3, wherein the plurality of classes include a normal state and an abnormal state, or include a normal state and a plurality of abnormal states having mutually different levels.

11. The image processing device according to any one of claims 1 to 3, wherein the image input unit acquires measurement values ​​from a sensor, and generates an input image including the visualization data based on the measurement values.

12. The image processing device according to any one of claims 1 to 3, wherein the image output unit outputs the classification result.

13. The image processing device according to any one of claims 1 to 3, further comprising a communication unit for communicating with a second external device, the first determination unit generating an alert signal based on the classification result, and the communication unit transmitting the alert signal to the second external device.

14. An image processing device comprising: a data input unit that acquires input data including numerical data; a first determination unit that acquires a classification result that classifies the input data into one of a plurality of classes; a second determination unit that determines a partial element of the input data that serves as a basis for determining the classification result; an image generation unit that converts the input data into visualized data and generates an output image including the visualized data, the classification result, and the partial element; and an image output unit that outputs the output image.

15. The image processing device according to claim 14, wherein the numerical data includes information indicating the state of a manufacturing site including a plurality of processes, and the plurality of classes include whether each of the processes is in a bottleneck state or not.

16. An image processing device as described in claim 15, wherein the information indicating the status of the manufacturing site includes a start time and an end time of work for each of the processes, and the first judgment unit determines whether the first process is in a bottleneck state or not based on the start time and end time of a first process among the multiple processes, a second process preceding the first process, and a third process following the first process.

17. The image processing device according to any one of claims 14 to 16, further comprising a communication unit for communicating with an external device, the first determination unit generating an alert signal based on the classification result, and the communication unit transmitting the alert signal to the external device.

18. An image processing method for processing an input image including visualization data by a computer, comprising: acquiring the input image; acquiring a classification result that classifies the input image into one of a plurality of classes; determining a partial region of the input image that serves as a basis for determining the classification result; generating an output image by highlighting the partial region in the input image; and outputting the output image.

19. A program including instructions executed by a processor implemented in a computer that processes an input image including visualization data, the instructions causing the processor to: acquire the input image; acquire a classification result that classifies the input image into one of a plurality of classes; determine a partial region of the input image that serves as a basis for determining the classification result; generate an output image by highlighting the partial region in the input image; and output the output image.

Citation Information

Patent Citations

  • Heat map display device and heat map display method

    JP6868981B2

  • Display control device, display control method, and display control program

    WO2021009819A1

  • Time-series data analyzer

    JP2021096541A

  • Abnormal area detection method and system

    JP2023030355A

  • Production line monitoring device, production line monitoring program, and production line monitoring method

    WO2016051464A1