Information processing system, information processing method, and program

Abstract

An information processing system is provided in accordance with one aspect of the present disclosure. Provided is an information processing system. The information processing system includes at least one control unit. The control unit receives a captured image of a product from an imaging apparatus. The imaging apparatus uses a laser to acquire the captured image of the product. The control unit inputs the captured image to a trained model. The trained model has been trained using, as input data, the captured image of the product captured by the imaging apparatus, and using, as output data, data indicating the presence or absence of a defect in the product. The control unit outputs, to an inspector who inspects the product on the basis of the output data, information indicating that the product requires inspection by the inspector.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

[0002] Patent Document 1 discloses an inspection apparatus that inspects the appearance of a mold using a learned model constructed by machine learning.

[0003] Japanese Patent Application Laid-Open No. 2024-111218

[0004] In an actual field where product inspection is being carried out, multiple inspectors may manually determine whether there are defects (such as scratches, dents, cracks, etc.) in the product. Due to reasons such as the recent soaring labor costs, it may be required to maintain the product quality while reducing the number of inspectors.

[0005] According to one aspect of the present disclosure, an information processing system is provided. The information processing system has at least one or more control units. The control unit receives a captured image of a product from an imaging device. The imaging device is a device that acquires a captured image of the product using a laser. The control unit inputs the captured image into a learned model. The learned model is a learned model that has been learned with the captured image of the product captured by the imaging device as input data and the data indicating whether there are defects in the product as output data. The control unit outputs information indicating that the product needs to be inspected by an inspector who conducts an inspection of the product based on the output data.

[0006] FIG. 1 is a diagram showing an example of the system configuration of the information processing system. FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device. FIG. 3 is a diagram for explaining the concept of the processing of the information processing system. FIG. 4 is a flowchart showing an example of the information processing of the information processing device. FIG. 5 is a diagram showing an example of a captured image, heatmap data, and display data. FIG. 6 is a diagram for explaining the processing of Modification 2. FIG. 7 is a diagram showing an example of the display data of Modification 4.

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Various characteristic matters shown in the following embodiments can be combined with each other.

[0008] <Embodiment 1> 1. System Configuration Diagram Figure 1 is a diagram showing an example of the system configuration of the information processing system 1000. As shown in Figure 1, the information processing system 1000 includes, as a system configuration, an information processing device 100, a robot arm 140, a camera 110, a rotating roller 130, and a warning light 160. The information processing device 100, the robot arm 140, the camera 110, the rotating roller 130, and the warning light 160 are communicated together via a network 150. The network 150 consists of a WAN (Wide Area Network), a LAN (Local Area Network), or the Internet, or any combination thereof.

[0009] The robot arm 140 picks up the product 120 on the conveyor belt and moves it to a predetermined position based on the control of the information processing device 100. In the illustrated example, the predetermined position is on the rotating roller 130. The product 120 placed on the rotating roller 130 is rotated in accordance with the rotation of the rotating roller 130. The product 120 is a tangible object processed or assembled from raw materials and / or parts. The material of the product 120 is not particularly limited and may be resin, metal, glass, or rubber.

[0010] Camera 110 is a device that acquires images of product 120. In the illustrated example, camera 110 is a device that, based on the control of information processing device 100, shines a laser onto product 120 on a rotating roller 130 to acquire a three-dimensional image of the surface shape of product 120. To explain further, camera 110 is a device that irradiates the target object, product 120, with laser light and acquires an image showing the shape of the target object. Although two cameras 110 are shown in Figure 1, there may be one camera 110 or two or more cameras 110. By using a laser-based camera 110, the surface state of product 120 with an uneven shape can be imaged. In addition, by using a laser-based camera 110, product 120 can be imaged stably under any lighting conditions and brightness. Camera 110 generates images using a simple length measurement function, so it can handle product shapes with a deep depth of field and uneven surfaces. Since the laser itself is not natural light, it is resistant to external noise.

[0011] The warning light 160 emits a light of a predetermined color, based on the control of the information processing device 100, to indicate that a skilled inspection (or detailed inspection) by an inspector is required. This warning light 160 can be understood as outputting information to users such as inspectors that the product 120 in question does not meet a predetermined standard, i.e., is a defective product.

[0012] The information processing device 100 is a PC (Personal Computer) or a mini-PC, which controls the camera 110, rotating roller 130, robot arm 140, warning light 160, etc., and performs the main processing described in the specification below. A mini-PC is a personal computer that is much smaller and lighter than a typical desktop computer. The size of a mini-PC may be small enough to fit in the palm of your hand. An example of an OS for the information processing device 100 is Linux®.

[0013] Here, the information processing system described in the claim may consist of multiple devices or of a single device. If the information processing system described in the claim consists of a single device, an example of such device is the information processing device 100. If the information processing system described in the claim consists of multiple devices, an example of multiple devices is the information processing device 100 and a camera 110, or a cloud server composed of multiple server devices that provide the functions of the information processing device 100.

[0014] 2. Hardware Configuration (1) Hardware Configuration of Information Processing Device 100 Figure 2 shows an example of the hardware configuration of the information processing device 100. As shown in Figure 2, the information processing device 100 includes a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240 as its hardware configuration. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0015] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire information processing device 100.

[0016] The storage unit 220 is one of the following: HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid Sate Drive), or any combination thereof, and stores programs, data used by the control unit 210 when executing processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used by the control unit 210 when executing processing based on the programs include images captured by the camera 110 and trained models.

[0017] In this specification, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the information processing device 100. In other words, the data may be stored in the storage unit of any device as long as the control unit 210 can access or retrieve it. The control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functions of the information processing device 100 and the processing of the flowchart shown in Figure 4, which will be described later.

[0018] The communications unit 230 connects the information processing device 100 to the network 150 and manages communication with other devices.

[0019] Note that the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the information processing device 100. The same applies to the client device shown below.

[0020] 3. Information Processing The information processing of Embodiment 1 will be described below.

[0021] (1) Overview of the process The control unit 210 inputs the captured image taken by the camera 110 into a trained model (hereinafter, the trained model is also referred to as AI (Artificial Intelligence)). The trained model is a trained model that has been trained using the captured image of the product taken by the imaging device as input data and data indicating whether or not the product has defects as output data. Based on the output data, the control unit 210 outputs information to the inspector indicating that the product requires inspection by the inspector. In other words, the control unit 210 outputs information to the user, such as the inspector, indicating that the product in question is defective.

[0022] This configuration allows for two-stage product inspection. To further explain, with this configuration, rough inspection of products is performed using a trained model, and products that the trained model estimates may have defects can then be subjected to detailed inspection by an inspector. In other words, it is possible to combine inspection using a trained model with inspection by an inspector. In actual product inspection sites, attempts to introduce systems using trained models built with machine learning may be met with opposition from inspectors due to psychological resistance such as the fear of losing their jobs, making implementation difficult. Instead of eliminating inspectors, a two-stage inspection system consisting of AI-based inspection and inspection by an inspector reduces psychological resistance from inspectors, allowing for the rapid introduction of the system into product inspection sites.

[0023] Figure 3 is a diagram illustrating the processing concept of the information processing system 1000. As shown in Figure 3, the information processing system 1000 first performs a rough inspection 310 on the product using AI. The rough inspection 310 is performed to pick out products that may have defects. In other words, the rough inspection 310 is an inspection that selects products that need to be reinspected. Products that are judged to be NG, i.e., potentially defective, in the rough inspection 310 are sent for a detailed inspection 320 by an inspector. The detailed inspection 320 is a more detailed inspection than the rough inspection 310, performed manually by an experienced inspector to determine whether the product is truly defective. By performing the processing shown in Figure 3, it becomes unnecessary for an inspector to inspect all products. Furthermore, by having the inspector spend more time inspecting products that are estimated to have a higher probability of defects, it becomes possible to perform inspections with higher accuracy.

[0024] (2) Detailed Processing Figure 4 is a flowchart showing an example of information processing by the information processing device 100. In step S410, the control unit 210 receives an image captured by the camera 110. The image captured is an image of the product 120 captured by the camera 110. In step S420, the control unit 210 inputs the received image captured by the camera 110 into the trained model. The trained model is a trained model that has been trained using the image of the product captured by the camera 110 as input data and data including data indicating whether or not the product has defects as output data. More preferably, the trained model is a trained model that has been trained on good products.

[0025] The advantages of learning from good products include the fact that it eliminates the need to collect images of defective products (hereinafter also referred to as defective product data), and a learning model can be created using only images of non-defective products (hereinafter also referred to as good product data), thus significantly reducing the cost and effort of data collection. Also, because the preparation of defective product data is unnecessary, the time to system implementation can be significantly shortened compared to supervised learning. Furthermore, a model learned from good product data may be able to handle unknown defect patterns to some extent. Therefore, it becomes easier to detect abnormal patterns. In addition, because it is learned only from good product data, a highly accurate model can be created even with a small amount of data. Moreover, because it is easy to add and / or update good product data, it can flexibly respond to changes in the manufacturing line and / or changes in product specifications.

[0026] Preferably, the output data from the trained model includes data indicating the presence or absence of defects in the product and an image showing the location of the defect in the product. More preferably, the output data from the trained model includes data indicating the presence or absence of defects in the product and an image showing the location and size of the defect in the product. An example of an image showing the location and size of a defect is heatmap data showing the location and size of a defect in the product. Here, heatmap data showing the location and size of a defect is visualization data that represents the location and size of a defect in the product using color. To illustrate with an example of a defect, a scratch on the product is used to explain the size of the scratch, which is represented by the size of the circle on the heatmap data. The depth of the scratch is represented by the intensity of the color on the heatmap data. The control unit 210 determines whether a detailed inspection is necessary based on this output data. For example, if the output data from the trained model includes data indicating that the product has a defect, the control unit 210 determines that a detailed inspection is necessary. If the output data from the trained model includes data indicating that the product does not have a defect, the control unit 210 determines that a detailed inspection is not necessary.

[0027] If the control unit 210 determines that a detailed inspection is necessary, it proceeds to step S430; if it determines that a detailed inspection is not necessary, it returns to step S410.

[0028] In step S430, the control unit 210 outputs information to the inspector performing the inspection of the product 120 indicating that the product 120 requires inspection by the inspector (detailed inspection). For example, the control unit 210 transmits a control signal instructing the warning light 160 to illuminate a red lamp. The control signal instructing the warning light 160 to illuminate a red lamp is an example of information indicating to the inspector performing the inspection of the product 120 that the product 120 requires inspection by the inspector. Alternatively, for example, the control unit 210 transmits a message containing text indicating that a detailed inspection is required to the inspector's device (e.g., the inspector's PC and / or the inspector's mobile terminal device). The message containing text indicating that a detailed inspection is required is an example of information indicating to the inspector performing the inspection of the product 120 that the product 120 requires inspection by the inspector.

[0029] The control unit 210 may place products 120 that it has determined require inspection by an inspector onto a predetermined conveyor belt. The products 120 placed on the predetermined conveyor belt are transported to a location where the inspector performs a detailed inspection. As another example, the control unit 210 may place products 120 that it has determined require inspection by an inspector into a predetermined basket or the like. The products 120 placed in the predetermined basket or the like are transported by a person in charge to a location where the inspector performs a detailed inspection. The control unit 210 can be considered as distinguishing between products 120 that it has determined require further inspection and products 120 that do not require further inspection. Therefore, instead of outputting information that identifies products 120 that it has determined require further inspection as described above, it may also output information that identifies products 120 that do not require further inspection. Similarly, it may also be configured to transport products 120 that do not require further inspection to a specific location.

[0030] In step S440, the control unit 210 generates display data for the product 120 that requires detailed inspection. More specifically, the control unit 210 generates display data by superimposing the captured image taken by the camera 110 with the output data (heatmap data showing the location and size of defects in the product) output from the trained model.

[0031] Figure 5 shows an example of an image, heatmap data, and display data. The image 510 is an image of product 120 captured by camera 110. The heatmap data 520 is visualization data that represents the location and size of defects in product 120 using color, as described above. The display data 530 is data for display to inspectors, created by superimposing the heatmap data 520 onto the image 510. For clarity, the display data 530 in Figure 5 highlights the appearance of product 120 with dashed lines. By generating and displaying the display data 530, inspectors can confirm where and how large defects the AI ​​estimated to be on product 120. For example, inspectors can compare the display data 530 with the actual product 120 to confirm in detail whether product 120 actually has defects.

[0032] In step S450, the control unit 210 transmits the generated display data to the inspector who will perform a detailed inspection of the product. The processing in steps S430 and S450 is an example of a process that outputs information indicating that the product requires inspection by the inspector, along with display data, to the inspector who will perform the product inspection.

[0033] In step S460, the control unit 210 receives the inspection results of the relevant product from the inspector's device. The inspection results are the results of the detailed inspection performed by the inspector and contain information on whether or not there were defects. As another example, if there were defects, the detailed inspection results may also include an image showing which part of the captured image of the relevant product 120 is defective. The process in step S460 is an example of a process for obtaining the results of an inspection performed by an inspector.

[0034] In step S470, the control unit 210 retrains the trained model based on the received detailed inspection results. For example, if the detailed inspection results indicate that the product is free of defects, the control unit 210 uses the corresponding captured image as good product data and retrains the trained model. The process in step S470 is an example of a process that retrains the trained model based on the search results.

[0035] In step S480, the control unit 210 determines whether or not to terminate the process. If the control unit 210 determines to terminate the process, it terminates the process shown in the flowchart in Figure 4. If the control unit 210 determines not to terminate the process, it returns the process to step S410.

[0036] As described above, according to Embodiment 1, rough inspection of products is performed using a trained model, and products that the trained model estimates to have potential defects can be subjected to detailed inspection by an inspector. In other words, it is possible to combine inspection using a trained model with inspection by an inspector. As a result, the system can be quickly introduced to places where products are inspected. Furthermore, a system can be constructed that can handle complex inspection specifications and a wide variety of products (120) with a two-stage configuration of rough inspection by a good product learning AI and detailed inspection by an inspector.

[0037] (Modification 1) Hereinafter, Modification 1 of Embodiment 1 will be described. Modification 1 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 1 are the same as in Embodiment 1.

[0038] In the embodiment 1 described above, the trained model was explained as a trained model trained on good products. However, the information processing device 100 may use a trained model that has trained on both good and defective products. Alternatively, the information processing device 100 may perform a rough inspection of the product 120 by combining a trained model that has trained on both good and defective products with a trained model that has trained on good products only.

[0039] (Modification 2) Modification 2 of Embodiment 1 is described below. Modification 2 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 2 are the same as in Embodiment 1.

[0040] In Modification 2, for example, when generating a trained model, the system may focus on the corners and / or edges of the product in accordance with the crack generation principle to train the system to recognize good products. That is, the system may use images of the corners and / or edges of defective products to train the trained model to recognize good products. Furthermore, in Modification 2, when performing a rough inspection of the product 120, the control unit 210 may also input images of the corners and / or edges of the product 120 from the captured image of the product 120 into the trained model and receive data indicating the presence or absence of defects in the product as output data from the trained model. To further explain, the system may also be configured to input only images containing the corners and / or edges of the product 120 into the trained model.

[0041] In other words, a trained model may be generated using images from a partial range of the captured image of product 120, rather than the entire range.

[0042] Figure 6 is a diagram illustrating the processing of Modified Example 2. The good product data 610 is the captured image used for good product training in Modified Example 2. The good product data 610 is data only from a predetermined area (shown by a dashed line in the figure), such as a predetermined distance from the edge 615 of the captured image of product 120. In Modified Example 2, the data input to the trained model is also data (captured image) only from a predetermined area, such as a predetermined distance from the edge 625 of the captured image of product 120. The defective product data 620 is an example of data input to the trained model in Modified Example 2, which indicates that the product has a defect in the output data. The circle 621 indicates that product 120 has a defect. The information processing device 100 may perform rough inspection by arbitrarily combining trained models trained using the entire range of the captured image of the product, trained models trained using a portion of the captured image of the product, trained models trained using captured images of the product with different resolutions, etc. By combining multiple trained models in this way, it is possible to prevent missing defective products 120. As shown in Modification 2, by processing only a portion of the image of product 120, the processing load on the control unit 210 is reduced. In addition, since only a portion of the inspector needs to check, the check becomes easier and the speed of the check is also improved.

[0043] (Modification 3) Modification 3 of Embodiment 1 is described below. Modification 3 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 3 are the same as in Embodiment 1.

[0044] In FIG. 1 of Embodiment 1, it was described that the robot arm 140 picks up the product 120 on the belt conveyor, moves it to a predetermined position, and images it with the camera 110. However, the camera 110 may be configured to image the product 120 on the belt conveyor. Then, the information processing device 100 performs a rough inspection based on the captured image or the like. And the control unit 210 of Modification 3 picks up the product 120 estimated to have a defect in the rough inspection from the belt conveyor, and places it on a predetermined belt conveyor that transports the product 120 to the inspector, or the robot arm 140 may be controlled to be placed in a predetermined basket that is carried by the person in charge to the inspector.

[0045] (Modification 4) Hereinafter, Modification 4 of Embodiment 1 will be described. Modification 4 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. The configurations and processes not described in Modification 4 are the same as those in Embodiment 1.

[0046] FIG. 7 is a diagram showing an example of the display data 700 of Modification 4. The control unit 210 of Modification 4 may attach an image (mark images 710 and 720 shown as circles in the example of FIG. 7) indicating the position and size of the defect in the product 120 to the display data 700 or the like. Note that the mark images 710 and 720 etc. may be included in the output data output from the learned model, or the control unit 210 may perform image analysis on heatmap data etc. and attach it based on the result of the image analysis. By attaching an image indicating the position and size of the defect in the product 120, the inspector can immediately grasp the place to be inspected in more detail. Note that the image indicating the position and size of the defect may be in a mode using other figures such as arrows, or may be in a mode that makes it easier to visually recognize by blinking or the like. Also, the display mode may be as follows. - Display the position or area in coordinates - Display the dimensions of the defect

[0047] (Modification 5) Hereinafter, Modification 5 of Embodiment 1 will be described. Modification 5 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. The configurations and processes not described in Modification 5 are the same as those in Embodiment 1.

[0048] The control unit 210 of the fifth modification example may switch the display mode (information output mode) of the warning lamp 160 according to the probability of being a defective product. For example, red: high probability of being a defective product, yellow: medium probability of being a defective product, blue: low probability of being a defective product (non-defective product), etc. By switching the display mode (output mode), for example, for those with a high probability of being a defective product, the inspection work can be made more efficient by simplifying or omitting the inspection by the inspector.

[0049] <Notes> The system may be provided in the following embodiments. (Note 1) An information processing system having at least one control unit, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the image is input to a trained model, the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data, and based on the output data, outputs information to an inspector that the product requires inspection by the inspector. (Note 2) An information processing system according to Note 1, wherein the trained model is a trained model that has learned to use an image of a product free of defects captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data. (Note 3) An information processing system according to Note 1, wherein the output data is an image showing the location of a defect in the product. (Note 4) An information processing system as described in Note 3, wherein the output data is an image showing the location and size of a defect in the product. (Note 5) An information processing system as described in Note 4, wherein the output data is heatmap data showing the location and size of a defect in the product. (Note 6) An information processing system as described in Note 1, which generates display data by superimposing the captured image and the output data. (Note 7) An information processing system as described in Note 6, wherein the control unit outputs information to an inspector performing an inspection of the product indicating that the product requires inspection by the inspector, and the display data. (Note 8) An information processing system as described in Note 1, wherein the control unit acquires the results of the inspection by the inspector and retrains the trained model based on the inspection results.(Note 9) An information processing method to be performed by an information processing system, comprising: receiving an image of a product from an imaging device, the imaging device being a device that acquires an image of the product using a laser, inputting the image of the product into a trained model, the trained model being a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating whether or not the product has defects as output data, and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector. (Note 10) A program to cause a computer to function as an information processing system described in any one of Notes 1 to 8.

[0050] While various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. Novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. Embodiments and variations thereof are included in the scope and spirit of the invention, as well as in the claims and their equivalents. Modifications may be implemented in any combination.

[0051] 100: Information processing device 110: Camera 120: Product 150: Network 210: Control unit 220: Memory unit 230: Communication unit 240: Internal bus 1000: Information processing system

Claims

1. An information processing system comprising at least one control unit, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the image is input to a trained model, the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data, and based on the output data, outputs information to an inspector indicating that the product requires inspection by the inspector.

2. An information processing system according to claim 1, wherein the trained model is a trained model that has been trained using an image of a product free of defects, captured by the imaging device, as input data and data indicating the presence or absence of defects in the product as output data.

3. An information processing system according to claim 1, wherein the output data is an image showing the location of a defect in the product.

4. An information processing system according to claim 3, wherein the output data is an image showing the location and size of a defect in the product.

5. An information processing system according to claim 4, wherein the output data is heatmap data indicating the location and size of defects in the product.

6. An information processing system according to claim 1, which generates display data by superimposing the captured image and the output data.

7. An information processing system according to claim 6, wherein the control unit outputs to an inspector performing an inspection of the product information indicating that the product requires inspection by the inspector and the display data.

8. An information processing system according to claim 1, wherein the control unit acquires the results of an inspection by the inspector and retrains the trained model based on the results of the inspection.

9. An information processing method to be performed by an information processing system, comprising: receiving an image of a product from an imaging device, the imaging device being a device that acquires an image of the product using a laser, inputting the image of the product into a trained model, the trained model being a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data, and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector.

10. A program that causes a computer to function as an information processing system according to any one of claims 1 to 8.

Images