Information processing system, information processing method, and program

Abstract

One aspect of the present disclosure provides an information processing system. The information processing system includes at least one control unit. The control unit receives a captured image of a product from an imaging device. The imaging device uses a laser to acquire a captured image of a product. The control unit inputs the captured image to a trained model. The trained model has been trained using, as input data, a captured image of a product captured by the imaging device, and, as output data, data indicating the presence or absence of a product defect. The control unit outputs, to an inspector who inspects the product on the basis of the output data, information indicating that the product requires inspection by the inspector.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] Patent Document 1 discloses an inspection apparatus that inspects the appearance of a mold using a learned model constructed by machine learning.

[0003] Japanese Patent Application Laid-Open No. 2024-111218

[0004] In an actual field where product inspection is being carried out, multiple inspectors may manually determine whether there are defects (such as scratches, dents, cracks, etc.) in the product. Due to reasons such as the recent increase in labor costs, it may be required to maintain the quality of the product while reducing the number of inspectors.

[0005] According to one aspect of the present disclosure, an information processing system is provided. The information processing system includes at least one or more control units. The control unit receives a captured image of a product from an imaging device. The imaging device is a device that acquires a captured image of the product using a laser. The control unit inputs the captured image into a learned model. The learned model is a learned model that has been learned with the captured image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data. The control unit outputs information indicating that the product needs to be inspected by an inspector who performs product inspection based on the output data.

[0006] Figure 1 is a diagram showing an example of the system configuration of the information processing system of Embodiment 1. Figure 2 is a diagram showing an example of the hardware configuration of the information processing device of Embodiment 1. Figure 3 is a diagram for explaining the concept of processing in the information processing system of Embodiment 1. Figure 4 is a flowchart showing an example of information processing in the information processing device of Embodiment 1. Figure 5 is a diagram showing an example of an captured image and display data. Figure 6 is a diagram for explaining the processing of Modification 2. Figure 7 is a diagram showing an example of display data for Modification 4. Figure 8 is a diagram showing an example of the system configuration of the information processing system of Embodiment 2. Figure 9 is a diagram showing an example of the hardware configuration of the information processing device of Embodiment 2. Figure 10 is a functional block diagram showing an example of the functional configuration of the information processing device of Embodiment 2. Figure 11 is a flowchart showing the basic operation of information processing in the information processing device of Embodiment 2. Figure 12 is a flowchart showing an example of the operation of inspection processing in the information processing device of Embodiment 2. Figure 13 is a flowchart showing another example of the operation of inspection processing in the information processing device of Embodiment 2. Figure 14 is a flowchart showing another example of the operation of inspection processing in the information processing device of Embodiment 2. Figure 15 is a flowchart showing another example of the operation of inspection processing in the information processing device of Embodiment 2. Figure 16 is a flowchart showing another example of the operation of inspection processing in the information processing device of Embodiment 2. Figure 17 is a flowchart showing an example of another operation of the inspection process in the information processing apparatus of Embodiment 2. Figure 18 is a schematic diagram showing an example of the captured image segmentation process in the information processing apparatus of Embodiment 2. Figure 19 is a schematic diagram showing an example of the image list output process for each segmented area in the information processing apparatus of Embodiment 2. Figure 20 is a schematic diagram showing an example of the defective product identification information output process in the information processing apparatus of Embodiment 2.

[0007] Embodiments of this disclosure will be described below with reference to the drawings. The various features shown in the embodiments below are interchangeable.

[0008] <Embodiment 1> 1. System Configuration Diagram Figure 1 is a diagram showing an example of the system configuration of the information processing system 1000. As shown in Figure 1, the information processing system 1000 includes, as a system configuration, an information processing device 100, a robot arm 140, a camera 110, a rotating roller 130, and a warning light 160. The information processing device 100, the robot arm 140, the camera 110, the rotating roller 130, and the warning light 160 are communicated together via a network 150. The network 150 is composed of a WAN (Wide Area Network), a LAN (Local Area Network), or the Internet, or any combination thereof.

[0009] The robot arm 140 picks up the product 120 on the conveyor belt and moves it to a predetermined position based on the control of the information processing device 100. In the illustrated example, the predetermined position is on the rotating roller 130. The product 120 placed on the rotating roller 130 is rotated in accordance with the rotation of the rotating roller 130. The product 120 is a tangible object processed or assembled from raw materials and / or parts. The material of the product 120 is not particularly limited and may be resin, metal, glass, or rubber.

[0010] Camera 110 is a device that acquires images of product 120. In the illustrated example, camera 110 is a device that, based on the control of information processing device 100, shines a laser onto product 120 on a rotating roller 130 to acquire a three-dimensional image of the surface shape of product 120. To explain further, camera 110 is a device that irradiates the target object, product 120, with laser light and acquires an image showing the shape of the target object. Although two cameras 110 are shown in Figure 1, there may be one camera 110 or two or more cameras 110. By using a laser-based camera 110, the surface state of product 120 with an uneven shape can be imaged. In addition, by using a laser-based camera 110, product 120 can be imaged stably under any lighting conditions and brightness. Camera 110 generates images using a simple length measurement function, so it can handle product shapes with a deep depth of field and uneven surfaces. Since the laser itself is not natural light, it is resistant to external noise.

[0011] The warning light 160 emits a light of a predetermined color, based on the control of the information processing device 100, to indicate that a skilled inspection (or detailed inspection) by an inspector is required. This warning light 160 can be understood as outputting information to users such as inspectors that the product 120 in question does not meet a predetermined standard, i.e., is a defective product.

[0012] The information processing device 100 is a PC (Personal Computer) or a mini-PC, which controls the camera 110, rotating roller 130, robot arm 140, warning light 160, etc., and performs the main processing described below in the specification. A mini-PC is a personal computer that is much smaller and lighter than a typical desktop computer. The size of a mini-PC may be small enough to fit in the palm of your hand. An example of an OS for the information processing device 100 is Linux®.

[0013] Here, the information processing system described in the claim may consist of multiple devices or of a single device. If the information processing system described in the claim consists of a single device, an example of such device is the information processing device 100. If the information processing system described in the claim consists of multiple devices, an example of multiple devices is the information processing device 100 and a camera 110, or a cloud server composed of multiple server devices that provide the functions of the information processing device 100.

[0014] 2. Hardware Configuration (1) Hardware Configuration of Information Processing Device 100 Figure 2 shows an example of the hardware configuration of the information processing device 100. As shown in Figure 2, the information processing device 100 includes a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240 as its hardware configuration. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0015] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire information processing device 100.

[0016] The storage unit 220 is one of the following: HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), or any combination thereof, and stores programs, data used by the control unit 210 when executing processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used by the control unit 210 when executing processing based on the programs include images captured by the camera 110 and trained models.

[0017] In this embodiment, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the information processing device 100. In other words, the data may be stored in the storage unit of any device as long as the control unit 210 can access or retrieve it. The control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functions of the information processing device 100 and the processing of the flowchart shown in Figure 4, which will be described later.

[0018] The communications unit 230 connects the information processing device 100 to the network 150 and manages communication with other devices.

[0019] Furthermore, the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the information processing device 100. The same applies to the inspector's device shown below.

[0020] 3. Information Processing The information processing of Embodiment 1 will be described below.

[0021] (1) Overview of the process The control unit 210 inputs the captured image taken by the camera 110 into a trained model (hereinafter, the trained model is also referred to as AI (Artificial Intelligence)). The trained model is a trained model that has been trained using the captured image of the product taken by the imaging device as input data and data indicating whether or not the product has defects as output data. Based on the output data, the control unit 210 outputs information to the inspector indicating that the product requires inspection by the inspector. In other words, the control unit 210 outputs information to the user, such as the inspector, indicating that the product in question is defective.

[0022] This configuration allows for two-stage product inspection. To further explain, with this configuration, rough inspection of products is performed using a trained model, and products that the trained model estimates may have defects can then be subjected to detailed inspection by an inspector. In other words, it is possible to combine inspection using a trained model with inspection by an inspector. In actual product inspection sites, attempts to introduce systems using trained models built with machine learning may be met with opposition from inspectors due to psychological resistance such as the fear of losing their jobs, making implementation difficult. Instead of eliminating inspectors, by implementing a two-stage inspection system consisting of AI-based inspection and inspection by an inspector, the psychological resistance from inspectors can be reduced, and the system can be quickly introduced to product inspection sites.

[0023] Figure 3 is a diagram illustrating the processing concept of the information processing system 1000. As shown in Figure 3, the information processing system 1000 first performs a rough inspection 310 on the product using AI. The rough inspection 310 is performed to pick out products that may have defects. In other words, the rough inspection 310 is an inspection that selects products that need to be reinspected. Products that are judged to be NG, i.e., potentially defective, in the rough inspection 310 are sent for a detailed inspection 320 by an inspector. The detailed inspection 320 is a more detailed inspection than the rough inspection 310, performed manually by an experienced inspector to determine whether the product is truly defective. By performing the processing shown in Figure 3, it becomes unnecessary for an inspector to inspect all products. Furthermore, by having the inspector spend more time inspecting products that are estimated to have a higher probability of defects, it becomes possible to perform inspections with higher accuracy.

[0024] (2) Detailed Processing Figure 4 is a flowchart showing an example of information processing by the information processing device 100. In step S410, the control unit 210 receives an image captured by the camera 110. The image captured is an image of the product 120 captured by the camera 110. In step S420, the control unit 210 inputs the received image captured by the camera 110 into the trained model. The trained model is a trained model that has been trained using the image of the product captured by the camera 110 as input data and data including data indicating whether or not the product has defects as output data. More preferably, the trained model is a trained model that has been trained on good products.

[0025] The advantages of learning from good products include the fact that it eliminates the need to collect images of defective products (hereinafter also referred to as defective product data), and a learning model can be created using only images of non-defective products (hereinafter also referred to as good product data), thus significantly reducing the cost and effort of data collection. Also, because the preparation of defective product data is unnecessary, the time to system implementation can be significantly shortened compared to supervised learning. Furthermore, a model learned from good product data may be able to handle unknown defect patterns to some extent. Therefore, it becomes easier to detect abnormal patterns. In addition, because it is learned only from good product data, a highly accurate model can be created even with a small amount of data. Moreover, because it is easy to add and / or update good product data, it can flexibly respond to changes in the manufacturing line and / or changes in product specifications.

[0026] Preferably, the output data from the trained model includes data indicating the presence or absence of defects in the product and an image showing the location of the defect in the product. More preferably, the output data from the trained model includes data indicating the presence or absence of defects in the product and an image showing the location and size of the defect in the product. An example of an image showing the location and size of a defect is heatmap data showing the location and size of a defect in the product. Here, heatmap data showing the location and size of a defect is visualization data that represents the location and size of a defect in the product using color. To illustrate with an example of a defect, a scratch on the product is used to explain the size of the scratch, which is represented by the size of the circle on the heatmap data. The depth of the scratch is represented by the intensity of the color on the heatmap data. The control unit 210 determines whether a detailed inspection is necessary based on this output data. For example, if the output data from the trained model includes data indicating that the product has a defect, the control unit 210 determines that a detailed inspection is necessary. If the output data from the trained model includes data indicating that the product does not have a defect, the control unit 210 determines that a detailed inspection is not necessary.

[0027] If the control unit 210 determines that a detailed inspection is necessary, it proceeds to step S430; if it determines that a detailed inspection is not necessary, it returns to step S410.

[0028] In step S430, the control unit 210 outputs information to the inspector performing the inspection of the product 120 indicating that the product 120 requires inspection by the inspector (detailed inspection). For example, the control unit 210 transmits a control signal instructing the warning light 160 to illuminate a red lamp. The control signal instructing the warning light 160 to illuminate a red lamp is an example of information indicating to the inspector performing the inspection of the product 120 that the product 120 requires inspection by the inspector. Alternatively, for example, the control unit 210 transmits a message containing text indicating that a detailed inspection is required to the inspector's device (e.g., the inspector's PC and / or the inspector's mobile terminal device). The message containing text indicating that a detailed inspection is required is an example of information indicating to the inspector performing the inspection of the product 120 that the product 120 requires inspection by the inspector.

[0029] The control unit 210 may place products 120 that it has determined require inspection by an inspector onto a predetermined conveyor belt. The products 120 placed on the predetermined conveyor belt are transported to a location where the inspector performs a detailed inspection. As another example, the control unit 210 may place products 120 that it has determined require inspection by an inspector into a predetermined basket or the like. The products 120 placed in the predetermined basket or the like are transported by a person in charge to a location where the inspector performs a detailed inspection. The control unit 210 can be considered as distinguishing between products 120 that it has determined require further inspection and products 120 that do not require further inspection. Therefore, instead of outputting information that identifies products 120 that it has determined require further inspection as described above, it may also output information that identifies products 120 that do not require further inspection. Similarly, it may also be configured to transport products 120 that do not require further inspection to a specific location.

[0030] In step S440, the control unit 210 generates display data for the product 120 that requires detailed inspection. More specifically, the control unit 210 generates display data by superimposing the captured image taken by the camera 110 with the output data (heatmap data showing the location and size of defects in the product) output from the trained model.

[0031] Figure 5 shows an example of an image, heatmap data, and display data. The image 510 is an image of product 120 captured by camera 110. The heatmap data 520 is visualization data that represents the location and size of defects in product 120 using color, as described above. The display data 530 is data for display to inspectors, created by superimposing the heatmap data 520 onto the image 510. For clarity, the display data 530 in Figure 5 highlights the appearance of product 120 with dashed lines. By generating and displaying the display data 530, inspectors can confirm where and how large defects the AI ​​estimated to be on product 120. For example, inspectors can compare the display data 530 with the actual product 120 to confirm in detail whether product 120 actually has defects.

[0032] In step S450, the control unit 210 transmits the generated display data to the inspector who will perform a detailed inspection of the product. The processing in steps S430 and S450 is an example of a process that outputs information indicating that the product requires inspection by the inspector, along with display data, to the inspector who will perform the product inspection.

[0033] In step S460, the control unit 210 receives the inspection results of the relevant product from the inspector's device. The inspection results are the results of the detailed inspection performed by the inspector and contain information on whether or not there were defects. As another example, if there were defects, the detailed inspection results may also include an image showing which part of the captured image of the relevant product 120 is defective. The process in step S460 is an example of a process for obtaining the results of an inspection performed by an inspector.

[0034] In step S470, the control unit 210 retrains the trained model based on the received detailed inspection results. For example, if the detailed inspection results indicate that the product is free of defects, the control unit 210 uses the corresponding captured image as good product data and retrains the trained model. The process in step S470 is an example of a process that retrains the trained model based on the search results.

[0035] In step S480, the control unit 210 determines whether or not to terminate the process. If the control unit 210 determines to terminate the process, it terminates the process shown in the flowchart in Figure 4. If the control unit 210 determines not to terminate the process, it returns the process to step S410.

[0036] As described above, according to Embodiment 1, rough inspection of products is performed using a trained model, and products that the trained model estimates to have potential defects can be subjected to detailed inspection by an inspector. In other words, it is possible to combine inspection using a trained model with inspection by an inspector. As a result, the system can be quickly introduced to places where products are inspected. Furthermore, a system can be constructed that can handle complex inspection specifications and a wide variety of products (120) with a two-stage configuration of rough inspection by a good product learning AI and detailed inspection by an inspector.

[0037] (Modification 1) Hereinafter, Modification 1 of Embodiment 1 will be described. Modification 1 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 1 are the same as in Embodiment 1.

[0038] In the embodiment 1 described above, the trained model was explained as a trained model trained on good products. However, the information processing device 100 may use a trained model that has trained on both good and defective products. Alternatively, the information processing device 100 may perform a rough inspection of the product 120 by combining a trained model that has trained on both good and defective products with a trained model that has trained on good products only.

[0039] (Modification 2) Modification 2 of Embodiment 1 is described below. Modification 2 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 2 are the same as in Embodiment 1.

[0040] In Modification 2, for example, when generating a trained model, the system may focus on the corners and / or edges of the product in accordance with the crack generation principle to train the system to recognize good products. That is, the system may use images of the corners and / or edges of defective products to train the trained model to recognize good products. Furthermore, in Modification 2, when performing a rough inspection of the product 120, the control unit 210 may also input images of the corners and / or edges of the product 120 from the captured image of the product 120 into the trained model and receive data indicating the presence or absence of defects in the product as output data from the trained model. To further explain, the system may also be configured to input only images containing the corners and / or edges of the product 120 into the trained model.

[0041] In other words, a trained model may be generated using images from a partial range of the captured image of product 120, rather than the entire range.

[0042] Figure 6 is a diagram illustrating the processing of Modified Example 2. The good product data 610 is the captured image used for good product training in Modified Example 2. The good product data 610 is data only from a predetermined area (shown by a dashed line in the figure), such as a predetermined distance from the edge 615 of the captured image of product 120. In Modified Example 2, the data input to the trained model is also data (captured image) only from a predetermined area, such as a predetermined distance from the edge 625 of the captured image of product 120. The defective product data 620 is an example of data input to the trained model in Modified Example 2, which indicates that the product has a defect in the output data. The circle 621 indicates that product 120 has a defect. The information processing device 100 may perform rough inspection by arbitrarily combining trained models trained using the entire range of the captured image of the product, trained models trained using a portion of the captured image of the product, trained models trained using captured images of the product with different resolutions, etc. By combining multiple trained models in this way, it is possible to prevent missing defective products 120. As shown in Modification 2, by processing only a portion of the image of product 120, the processing load on the control unit 210 is reduced. In addition, since only a portion of the inspector needs to check, the check becomes easier and the speed of the check is also improved.

[0043] (Modification 3) Modification 3 of Embodiment 1 is described below. Modification 3 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 3 are the same as in Embodiment 1.

[0044] In FIG. 1 of Embodiment 1, it was described that the robot arm 140 picks up the product 120 on the belt conveyor, moves it to a predetermined position, and images it with the camera 110. However, the camera 110 may be configured to image the product 120 on the belt conveyor. Then, the information processing device 100 performs a rough inspection based on the captured image or the like. And the control unit 210 of the third modification picks up the product 120 estimated to have a defect in the rough inspection from the belt conveyor, and places it on a predetermined belt conveyor that transports the product 120 to the inspector, or the robot arm 140 may be controlled to be placed in a predetermined basket that is carried by the person in charge to the inspector.

[0045] (Modification 4) Hereinafter, Modification 4 of Embodiment 1 will be described. Modification 4 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. The configurations and processes not described in Modification 4 are the same as those in Embodiment 1.

[0046] FIG. 7 is a diagram showing an example of the display data 700 of Modification 4. The control unit 210 of Modification 4 may attach an image (mark images 710 and 720 shown as circles in the example of FIG. 7) indicating the position and size of the defect in the product 120 to the display data 700 or the like. Note that the mark images 710 and 720 etc. may be included in the output data output from the learned model, or the control unit 210 may perform image analysis on heat map data etc. and attach it based on the result of the image analysis. By attaching an image indicating the position and size of the defect in the product 120, the inspector can immediately grasp the place etc. to be inspected in more detail. Note that the image indicating the position and size of the defect may be in a mode using other figures such as arrows, or may be in a mode that makes it easier to visually recognize by blinking etc. Also, the display mode may be as follows. - Display the position or area in coordinates - Display the dimensions of the defect

[0047] (Modification 5) Hereinafter, Modification 5 of Embodiment 1 will be described. Modification 5 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. The configurations and processes not described in Modification 5 are the same as those in Embodiment 1.

[0048] The control unit 210 of the fifth modification may switch the display mode (information output mode) of the warning lamp 160 according to the probability that the product is defective. For example, red indicates a high probability that the product is defective, yellow indicates a medium probability that the product is defective, and blue indicates a low probability that the product is defective (good product). By switching the display mode (output mode), for example, for products with a high probability of being defective, the inspection work can be made more efficient by simplifying or omitting the inspection by the inspector.

[0049] <Embodiment 2> Next, Embodiment 2 will be described. In Embodiment 2 described below, the description will focus on the parts different from Embodiment 1 described above. Therefore, the same components as those in Embodiment 1 are denoted by the same reference numerals and their description is omitted, and it is assumed that they are the same unless otherwise specified.

[0050] 1. System Configuration Diagram FIG. 8 is a diagram showing an example of the system configuration of the information processing system 2000 according to Embodiment 2. As shown in FIG. 8, the information processing system 2000 includes at least a robot arm 140, a camera 110 as an imaging device, a buffer processing device 170, and an information processing device 2100 as system components. Further, although not shown, the information processing system 2000 includes a rotating roller 130 and a warning lamp 160, similar to the information processing system 1000 of Embodiment 1 described above. The information processing device 2100, the robot arm 140, the camera 110, the rotating roller 130, the warning lamp 160, and the buffer processing device 170 are communicably connected via a network 150.

[0051] Then, the information processing system 2000 according to Embodiment 2 temporarily stores the captured images of a plurality of different products 120 acquired by the imaging device, and temporarily stores the plurality of products 120 after imaging by the imaging device in a pallet 171 (an example of a storage unit) of the buffer processing device 170 capable of storing the products 120. By continuously processing the captured images of the plurality of products 120 stored in the pallet 171 using a learned model, output data indicating the presence or absence of defects in the plurality of products 120 is output. This will be described in detail below.

[0052] Based on the control of the information processing device 2100, the robot arm 140 picks up the product 120 on the conveyor belt and moves it to a predetermined position. In the illustrated example, the predetermined position is on the rotating roller 130 as well as on the buffer processing device 170. That is, the robot arm 140 moves the product 120 onto the rotating roller 130 to acquire a three-dimensional image, and after imaging, moves the product 120 onto the pallet 171, which is a container provided by the buffer processing device 170, and stores it in predetermined positions in order.

[0053] The buffer processing unit 170 is designed to store a predetermined quantity (for example, 64 units) of product 120 as one batch. The pallet 171 is provided with multiple recesses for stably accommodating each product 120 individually. The products 120, after being imaged by the camera 110, are transported one by one to the pallet 171 of the buffer processing unit 170 by the robot arm 140. In the buffer processing unit 170 of this embodiment, for example, once one batch (a predetermined quantity) of product 120, which is placed on the left side, is accommodated in the pallet 171, it rotates, and another pallet 171 that was placed on the right side moves to the left side. In the illustrated example of the buffer processing unit 170, two pallets 171 are provided to rotate, but the number of pallets 171 is not limited.

[0054] Furthermore, the products 120 contained in the pallet 171 are then separated into OK products (no defects) and NG products (defective products) based on the inspection results and placed in their respective discharge containers (not shown). After the imaging images are acquired, the products 120 are arranged in order on the left pallet 171 shown in Figure 8, as indicated by the Z-shaped arrows. Once a batch of products 120 has been contained, the pallet 171 is inverted, and the products are removed in order on the right pallet 171 shown in Figure 8, as indicated by the Z-shaped arrows. At this time, each product 120 is sorted into OK or NG products based on the inspection results and placed in its respective discharge container.

[0055] 2. Hardware Configuration (1) Hardware Configuration of Information Processing Device 2100 Figure 9 shows an example of the hardware configuration of the information processing device 2100. As shown in Figure 9, the information processing device 2100, like the information processing device 100 in Embodiment 1 described above, includes a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240 as its hardware configuration. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0056] (2) The functional configuration diagram 10 of the information processing device 2100 is a block diagram illustrating an example of the configuration of the information processing device 2100 that functions as a server in this embodiment.

[0057] As shown in Figure 10, the information processing device (server) 2100 includes an acquisition unit 211, a holding unit 212, a splitting unit 213, an input unit 214, an inspection unit 215, and an output unit 216. The information processing device 2100 also includes an inspection model information DB 221 and an inspection result DB 222.

[0058] The acquisition unit 211 is a means that has the function of receiving and acquiring captured images of the product 120 from the imaging device. That is, the acquisition unit 211 receives the captured images of the product 120 acquired by the camera 110 using a laser.

[0059] The storage unit 212 is a means that has the function of temporarily storing multiple captured images of the product 120 acquired by the imaging device. In other words, the storage unit 212 is a so-called buffer area that temporarily stores multiple captured images together as one batch. In this case, the number of captured images that the storage unit 212 temporarily stores is not particularly limited, but for example, it is preferable to store a number that is a multiple of 2 or a multiple of 8, or preferably a power of 2, which is easier for a computer to process.

[0060] In the information processing system 2000 of this embodiment, the acquisition unit 211 does not immediately input the captured images of the product 120 into the trained model, but instead the holding unit 212 temporarily stores multiple captured images (up to a predetermined number).

[0061] The division unit 213 is a means that has the function of dividing the temporarily held captured image into multiple regions. That is, the division unit 213 divides the captured image of the product 120 stored in the holding unit 212 into multiple regions (for example, six divisions, see Figure 18) (hereinafter sometimes referred to as "tiles"). By dividing the captured image into multiple regions in this way, defect inspection can be performed with increased resolution on the captured image.

[0062] The input unit 214 is a means that has the function of inputting captured images into a trained model in batches. That is, the input unit 214 inputs multiple captured images temporarily stored by the holding unit 212 into the trained model in batches. Therefore, the trained model processes multiple captured images as one batch. In this embodiment, for example, one captured image is provided for one product 120. However, the number of captured images is not limited to one per product 120. For example, if the product is a polyhedron, there may be captured images taken from different viewpoints of the product 120.

[0063] The inspection unit 215 is a means that has the function of processing each divided region together using a pre-trained model. In the information processing system 2000 of Embodiment 2, a pre-trained model is provided for each divided region. That is, in the information processing system 2000 of Embodiment 2, pre-trained models are provided in advance according to the number of divisions of the region. The inspection unit 215 then inspects the captured images that the division unit 213 has divided into multiple divided regions, grouping together multiple captured images for each region at the same position. Specifically, for example, if the holding unit 212 is set to hold 64 captured images, the inspection unit 215 processes each region at the same position that has been divided in the 64 captured images consecutively. In other words, for example, if the holding unit 212 is set to hold 64 captured images, the 64 images are inspected together as one batch.

[0064] This allows for accurate and coordinated inspection of each divided region using the same defect detection model. Furthermore, when performing inspections with multiple defect detection models, the lead time can be shortened by switching to the next defect detection model and performing the same inspection on all regions one by one after the inspection of all regions of one model has been completed.

[0065] In other words, if a defect detection model (for example, a pre-trained model described later) is set up, the processing time is almost the same whether inspecting one captured image or inspecting 64 captured images together. Therefore, when inspecting each captured image individually, the total inspection time has been calculated by multiplying the inspection time for one region (tile) by the number of divided regions, and then by 64, the number of captured images. However, with the system 2000 of Embodiment 2, although it takes time to collect 64 captured images, the total inspection time is calculated by multiplying the inspection time for one region by the number of divided regions.

[0066] Specifically, if the number of regions into which the captured image is divided is six, and the inspection time is approximately one second, then previously, a total inspection time of approximately six seconds was required for one captured image, and for 64 captured images, the total inspection time would be 384 seconds (approximately six minutes). However, with the system 2000 of Embodiment 2, only approximately 6.6 seconds of inspection time is required, resulting in a significantly reduced total inspection time and improved efficiency.

[0067] In this system 2000, a defect detection model is prepared for each of the multiple divided regions (tiles) in the captured image. When inspecting multiple divided regions that make up a single captured image in sequence, it is necessary to switch the defect detection model each time the divided region to be inspected is changed. A certain amount of time is required when switching these defect inspection models. In contrast, with the system 2000 of Embodiment 2, for example, 64 captured images can be inspected together for each divided region, so the defect detection model only needs to be switched once for each of the six divided regions, for example, and the total inspection time can be significantly reduced.

[0068] Furthermore, the inspection unit 215 also has a function that, if a defect is found in the captured image in one of the divided regions, does not perform inspection in the other regions. In other words, the inspection unit 215 efficiently processes multiple captured images in batches, and also divides a single captured image into multiple regions and processes each region with high precision. Therefore, it is possible to change subsequent inspections to improve efficiency according to the results of the batch processing inspection.

[0069] As an example of modifying the inspection process based on the inspection results, if the inspection unit 215 finds a defect in one of the divided regions of the captured image, it may choose not to perform inspections using the trained model in the other regions. In other words, if a defect is found in the captured image, it is assumed that there is a high probability of finding defects in other regions as well, and therefore, inspections using the trained model in other regions are not performed for one batch of captured images.

[0070] This allows for early detection and resolution of issues, preventing wasted inspection time and improving inspection efficiency. Furthermore, by outputting information indicating the need for re-inspection by inspectors, the quality of product 120 can be enhanced through more accurate re-inspections.

[0071] Furthermore, for example, if defects are found in multiple consecutive images from the first captured image, re-inspection of the previous batch can be considered, or if defects are found in multiple consecutive images up to the last captured image, re-inspection of the subsequent batch can be considered, thereby enabling highly accurate inspection of a series of products 120 on a batch basis.

[0072] Furthermore, the inspection unit 215 also has a function that, if no defect is found in one of the divided regions, will not perform inspections in the other regions, or will reduce the accuracy of the inspection that determines a defect. In other words, if the inspection unit 215 is unable to find a defect in any one of the divided regions, for example, a region prone to defects, in a series of consecutive captured images, it can determine that the risk of defects occurring in other regions is low, and will either not perform inspections in the other regions for one batch of captured images, or will reduce the accuracy (level) of the inspection that determines a defect. This can improve the efficiency of the inspection. In this case, the system 2000 can accept a specification of which of the divided regions should be inspected first. In other words, the system 2000 can accept a specification of the inspection order of the divided regions (tiles).

[0073] Furthermore, the inspection unit 215 also has a function that performs inspection regardless of the number of captured images if the preset image retention time is exceeded, depending on the number of captured images to be temporarily held. In other words, the inspection unit 215 takes into account the timing difference in the transport of the product 120, and assuming there is no problem with the transport timing of the product 120, if the preset image retention time is exceeded, assuming that a preset batch of captured images will be temporarily stored in the holding unit 212, it performs inspection regardless of the number of captured images to be temporarily stored, without waiting for a preset batch of captured images to be temporarily stored in the holding unit 212.

[0074] In other words, the manufactured products 120 are placed on a conveyor belt or the like and transported one after another, but the transport interval of the products 120 is not always constant, and the transport interval may widen significantly due to circumstances upstream in the manufacturing process. In that case, the downstream inspection process will also be affected, but by performing inspections based on a pre-set image retention time for one batch, inspections can be performed as appropriate according to the manufacturing status of the products 120, thereby improving the efficiency of inspection time.

[0075] The output unit 216 is a means that outputs information to inspectors performing inspections of products 120, indicating that the products 120 require inspection by the inspector, based on the output data of a trained model that has learned the captured image of the product 120 captured by the imaging device (camera 110) as input data and data indicating the presence or absence of defects in the product 120 as output data. In other words, the output unit 216 outputs information indicating that, as a result of the inspection performed by the trained model, a re-inspection by the inspector is necessary, in a way that makes it clear which products 120 require re-inspection.

[0076] Furthermore, the output unit 216 also has a function to output information indicating that inspection by an inspector is necessary if a defect is found in the captured image in one of the divided regions and the inspection unit 215 does not perform inspection in the other regions. In other words, if the inspection unit 215 finds a defect in any one region in multiple consecutive captured images, the output unit 216 outputs information to the inspector prompting them to inspect the other batches along with the current batch, as there is a high possibility that defects will be found in other batches as well.

[0077] Furthermore, the output unit 216 also has a function to output information regarding stopping the manufacturing line if defects are found in the captured images in any one of the divided regions. In other words, if the inspection unit 215 finds defects in the captured images in any one region, the output unit 216 determines that it is not merely a coincidence but that there is some cause in the manufacturing equipment (machinery) or raw materials, and outputs alert information to stop the manufacturing line and conduct an inspection. This allows for early detection and action to prevent wasted manufacturing time and materials, enabling a review of the manufacturing line and improving the quality of the product 120.

[0078] Furthermore, the output unit 216 also has the function of outputting a list of output data for each divided region. That is, after the inspection by the inspection unit 215 is completed, the output unit 216 outputs a list of images for each divided region in one batch as output data. In this embodiment, the output unit 216 outputs multiple images grouped by each divided region as output data.

[0079] This not only outputs information indicating that a re-examination by an inspector is necessary, but also allows for efficient and highly accurate re-examination by sequentially reviewing one batch of images for each area that appears to have a problem, rather than reviewing all of the captured images during the re-examination.

[0080] Furthermore, the output unit 216 also has the function of outputting information that identifies the product 120 in which a defect was found during inspection. The defective product 120 can be output as location information or visual information that identifies its location on an image showing the container containing the product 120 after inspection has been completed. That is, the output unit 216 outputs information that identifies the product 120 in which a defect was found during inspection by the inspection unit 215, such as location information such as coordinates that identify its location on an image showing the container (pallet 171) containing the product 120, or visual information such as a map with markers to identify its location.

[0081] This not only outputs information indicating that re-inspection by an inspector is necessary, but also allows for easy identification of which product 120 needs to be re-inspected based on location and visual information, enabling efficient and highly accurate re-inspections.

[0082] The inspection model information DB221 is a means for storing trained models for inspecting defects. Specifically, the defects to be detected in the inspection include, for example, shell cracks, material cracks, chips, surface chips, processing defects, steps, black scale, uncoated processing, protrusions, peeling, AC chips, etc., and the trained model information that has been trained to determine the presence or absence of each defect is stored.

[0083] The inspection result DB 222 is a means for storing the inspection results of product 120. The inspection result DB 222 stores the image of product 120 acquired by the acquisition unit 211 and information regarding the presence or absence of defects in product 120, as performed by the inspection unit 215. In particular, in this embodiment, the inspection result DB 222 stores the image of product 120 and the result information regarding the presence or absence of defects on a batch basis.

[0084] Furthermore, the information processing system 2000 of Embodiment 2 performs so-called batch processing, which involves the continuous inspection of a certain number (e.g., 64) of products 120 together. In this manufacturing line of Embodiment 2, one batch is a unit related to quality assurance. Here, it can be assumed that the multiple products 120 treated as one batch have the same manufacturing conditions throughout the series of manufacturing processes. Therefore, in the inspection process, one product 120 in one batch, and furthermore, one divided area of ​​one product 120, can be considered representative of that batch.

[0085] Furthermore, the information processing system 2000 of Embodiment 2 manages statistical information for each batch. Examples of statistical information include the average and standard deviation of quantitative inspection results regarding defects in multiple products 120 in a batch. The inspection results DB 222 also stores statistical information regarding defects in products 120 for each batch. Based on the quantitative inspection results of multiple batches, the entire production line of products 120 can be managed. In this way, the information processing system 2000 of Embodiment 2 is capable of managing so-called group-based management of products 120.

[0086] As described above, the information processing device 2100 according to this embodiment functions as a device including an acquisition unit 211, a holding unit 212, a splitting unit 213, an input unit 214, an inspection unit 215, and an output unit 216, when various programs (OS, applications, etc.) stored in the auxiliary storage device are loaded into the main storage device and executed by the control unit 210.

[0087] In this embodiment, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the information processing device 2100. In other words, the data may be stored in the storage unit of any device as long as the control unit 210 can access or retrieve it. Furthermore, the control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functions of the information processing device 2100 and the processing of the flowcharts shown in Figures 10 to 17, which will be described later.

[0088] Note that the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the information processing device 2100. Also, the hardware configuration of the inspector device 180 in this embodiment is the same as that of the information processing device 2100 described above.

[0089] 3. Information Processing The information processing of Embodiment 2 will be described below.

[0090] (1) Overview of the process The control unit 210 temporarily holds multiple images captured by the camera 110 and inputs the multiple images together as one batch into the trained model. The trained model is a trained model that has been trained using images of the product 120 captured by the imaging device as input data and data indicating the presence or absence of defects in the product 120 as output data. The control unit 210 also outputs information to the inspector who will inspect the product 120 based on the output data, indicating that the product 120 requires inspection by the inspector. In other words, the control unit 210 outputs information to the user, such as the inspector, indicating that the product 120 in question is defective. The control unit 210 also divides the captured image into multiple regions and inspects multiple captured images (one batch) from each region at the same location using the trained model.

[0091] With this configuration, the information processing system 2000 of Embodiment 2 can also perform inspection of product 120 in two stages: a rough inspection using a trained model, and a detailed inspection performed by an inspector on product 120 that the trained model has estimated to have potential defects.

[0092] Furthermore, according to the information processing system 2000 of Embodiment 2, by providing a buffer area in the system, inspections using a single defect detection model can be performed efficiently all at once. That is, when inspecting a single product 120 with multiple defect detection models, if the inspection of the next product 120 is performed only after the inspection of all defect detection models has been completed, it will take time to switch between defect detection models. However, by providing a buffer area in the system, multiple products 120 can be inspected together using the same defect detection model. As a result, the lead time can be shortened.

[0093] (2) Details of the process Next, the flow of information processing in the information processing system 2000 of Embodiment 2 will be described. Figure 11 is a flowchart showing the basic operation of information processing in the information processing device 2100 of Embodiment 2.

[0094] In step S810, the control unit 210 receives the captured image from the camera 110, similar to step S410 in the information processing device 100 of Embodiment 1 described above. That is, the acquisition unit 211 performs the process of acquiring the captured image of the product 120 acquired by the camera 110 using a laser. In step S820, the control unit 210 stores the captured image in a buffer. That is, the holding unit 212 performs the process of storing a plurality (a predetermined number) of captured images of the product 120 acquired by the acquisition unit 211.

[0095] In step S830, the control unit 210 determines whether the number of captured images of the product 120 acquired is a predetermined quantity. That is, the holding unit 212 performs a process to determine whether the number of captured images of the product 120 to be temporarily stored is a predetermined quantity. If the holding unit 212 determines that the number of captured images is a predetermined quantity (YES in S830), the process proceeds to step S840. On the other hand, if it determines that the number of captured images is not a predetermined quantity (NO in S830), the process returns to step S810 and is repeated.

[0096] In step S840, the control unit 210 divides all acquired captured images. That is, the division unit 213 divides the captured images of the product 120, which are temporarily stored in the holding unit 212, into multiple regions. In step S850, the control unit 210 inputs the received captured images into the trained model. That is, the input unit 214 inputs the multiple captured images temporarily stored in the holding unit 212 into the trained model, grouping them into regions divided by the division unit 213.

[0097] Then, in step S860, the control unit 210 determines whether a detailed inspection is necessary based on the output data output from the trained model, similar to step S420 in the information processing device 100 of the embodiment 1 described above. That is, the inspection unit 215 inspects multiple images divided by the division unit 213 for each region at the same location, and performs a process to determine whether a detailed inspection by an inspector is necessary.

[0098] While there are no specific restrictions on the order of inspection, it is advisable to start with areas prone to defects. This allows for efficiency improvements by modifying subsequent inspections, for example, by not performing inspections using pre-trained models in other areas of an image batch if defects are found in that area.

[0099] Subsequently, as shown in Figure 11, the processing from step S870 to step S920 is the same as the processing from step S430 to step S480 in the information processing device 100 of Embodiment 1 described above. That is, the output unit 216 outputs information to the inspector performing the inspection of the product 120 indicating that the product 120 requires inspection by the inspector (detailed inspection) (step S870), generates display data for the product 120 that requires detailed inspection (step S880), and outputs (transmits) the generated display data to the inspector (inspector device 180) performing the detailed inspection of the product 120 (step S890). The acquisition unit 211 receives the inspection result of the corresponding product 120 from the inspector device 180 (step S900), and the input unit 214 performs the processing of retraining the learned model based on the received detailed inspection result (step S910).

[0100] The acquisition unit 211 then performs a process to determine whether or not it has received an instruction to terminate the process (step S920). If the acquisition unit 211 has received an instruction to terminate the process (YES in S920), the operation of the information processing device 2100 in Embodiment 2 ends. On the other hand, if the acquisition unit 211 has not received an instruction to terminate the process (NO in S920), the process returns to step S810 and is repeated.

[0101] Next, the flow of the inspection process in the information processing system 2000 of Embodiment 2 will be described. Figure 12 is a flowchart showing an example of the operation of the inspection process in the information processing device 2100 of Embodiment 2.

[0102] As shown in Figure 12, the processing from step S1010 to step S1050 is the same as the processing from step S810 to step S850 described above, so the explanation is omitted. In step S1060, the control unit 210 determines whether or not a defect has been found in one of the divided regions. That is, the inspection unit 215 performs the process of determining whether or not a defect has been found in the captured image in any of the divided regions (tiles). If the inspection unit 215 finds a defect in the captured image in one of the divided regions (YES in S1060), the process proceeds to step S1070. On the other hand, if the inspection unit 215 does not find a defect in the captured image (NO in S1060), the same processing as the processing from step S860 onwards described above is performed, as shown in Appendix III in Figure 11, so the explanation is omitted.

[0103] In step S1070, the control unit 210 terminates the inspection without performing inspections in other divided regions. That is, the inspection unit 215 determines that there is a high probability of finding defects in other regions and performs a process of not performing inspections using the trained model in other regions for one batch of captured images. In step S1080, the control unit 210 performs a process in which the output unit 216 outputs information to the inspector performing the inspection of the product 120 indicating that the product 120 requires inspection by the inspector (detailed inspection). After that, as shown in Figure 11 with appendix I, the same process as in step S880 and later described above is performed, so the explanation is omitted.

[0104] Next, the flow of other inspection processes in the information processing system 2000 of Embodiment 2 will be described. Figure 13 is a flowchart showing an example of other operations of the inspection process in the information processing device 2100 of Embodiment 2.

[0105] As shown in Figure 13, the processing from step S1110 to step S1150 is the same as the processing from step S810 to step S850 described above, so the explanation is omitted. In step S1160, the control unit 210 determines whether or not defects have been found in the captured image in a single divided region. That is, the inspection unit 215 performs the process of determining whether or not defects have been found in the captured image in any one of the divided regions. If the inspection unit 215 finds defects in the captured image in a continuous manner (YES in S1160), the process proceeds to step S1170. On the other hand, if the inspection unit 215 does not find defects in the captured image in a continuous manner (NO in S1160), the same processing as the processing from step S860 onwards described above is performed, as shown in Appendix III in Figure 11, so the explanation is omitted.

[0106] In step S1170, the control unit 210 terminates the inspection without performing inspections in other divided regions. That is, if the inspection unit 215 finds defects in consecutively captured images, it determines that the defect is not merely accidental, but is caused by some factor in the manufacturing equipment (machinery), operator (personnel), or material, and performs a process of not performing inspections using the trained model in other regions for one batch of captured images. In step S1180, the control unit 210 outputs manufacturing line stop information. That is, the output unit 216 outputs alert information to stop the manufacturing line and perform an inspection. After that, as shown in Appendix II in Figure 11, the same process as described above from step S870 onwards is performed, so the explanation is omitted.

[0107] Next, we will describe the flow of other inspection processes in the information processing system 2000 of Embodiment 2. Figure 14 is a flowchart showing an example of other operations in the inspection process of the information processing device 2100 of Embodiment 2.

[0108] As shown in Figure 14, the processing from step S1210 to step S1250 is the same as the processing from step S810 to step S850 described above, so the explanation is omitted. In step S1260, the control unit 210 determines whether or not there are any defects in all the captured images in one divided region. That is, the inspection unit 215 performs a process to determine whether or not it was not able to find any defects in all the captured images in any of the divided regions (areas). The divided region that the inspection unit 215 inspects first is preferably an area where defects are likely to occur. If the inspection unit 215 does not find any defects in all the captured images (YES in S1260), the process proceeds to step S1270. On the other hand, if the inspection unit 215 finds a defect in any of the captured images (NO in S1260), the same processing as the processing from step S860 onwards described above is performed, as shown in Appendix III in Figure 11, so the explanation is omitted.

[0109] In step S1270, the control unit 210 either terminates the inspection without performing inspections in other divided regions, or performs inspections with reduced inspection accuracy in other divided regions. That is, the inspection unit 215 determines that there is a low risk of defects occurring in other areas and either does not perform inspections in other regions for one batch of captured images, or reduces the accuracy (level) of the inspection that determines defects. After that, as shown in Appendix III in Figure 11, the same processing as described above from step S860 onwards is performed, so the explanation is omitted.

[0110] Next, we will describe the flow of other inspection processes in the information processing system 2000 of Embodiment 2. Figure 15 is a flowchart showing an example of other operations of the inspection process in the information processing device 2100 of Embodiment 2.

[0111] As shown in Figure 15, the processing from step S1310 to step S1370 is the same as the processing from step S810 to step S870 described above, so the explanation is omitted. In step S1380, the control unit 210 outputs a list of images of the divided regions. That is, after the inspection by the inspection unit 215 is completed, the output unit 216 outputs the captured images for each divided region in a list format as output data, one batch at a time. This allows for efficient and highly accurate reinspection by continuously checking one batch of captured images for each region that indicates a specific region that seems to have a problem. After that, as shown in Figure 11 with appendix I, the processing is the same as the processing from step S880 onwards described above, so the explanation is omitted.

[0112] Next, we will describe the flow of other inspection processes in the information processing system 2000 of Embodiment 2. Figure 16 is a flowchart showing an example of other operations of the inspection process in the information processing device 2100 of Embodiment 2.

[0113] As shown in Figure 16, the processing from step S1410 to step S1470 is the same as the processing from step S810 to step S870 described above, so the explanation is omitted. In step S1480, the control unit 210 outputs information to identify the defective product 120. That is, the output unit 216 not only outputs information indicating that re-inspection by an inspector is necessary after the inspection by the inspection unit 215 is completed, but also performs processing to output information to identify the product 120 in which a defect was found during the inspection by the inspection unit 215, so that it is easier to identify which product 120 should be re-inspected during the re-inspection using location information and visual information. As information to identify the product 120, for example, location information such as coordinates that identify the location on an image showing the container that contains the product 120, or visual information such as a map with marks to identify the location, can be output. After that, as shown in Figure 11 with appendix I, the processing is the same as the processing from step S880 onwards described above, so the explanation is omitted.

[0114] Next, we will describe the flow of other inspection processes in the information processing system 2000 of Embodiment 2. Figure 17 is a flowchart showing an example of other operations of the inspection process in the information processing device 2100 of Embodiment 2.

[0115] As shown in Figure 17, the processing in steps S1510 and S1520 is the same as the processing in steps S810 and S820 described above, so the explanation is omitted. In step S1530, the holding unit 212 performs a process to determine whether the number of captured images of the product 120 to be temporarily stored has reached a predetermined quantity. If the holding unit 212 determines that the number of captured images has reached a predetermined quantity (YES in S1530), the process proceeds to step S1550. On the other hand, if the holding unit 212 determines that the number of captured images has not reached a predetermined quantity (NO in S1530), the process proceeds to step S1540.

[0116] In step S1540, the control unit 210 determines whether the time for temporarily holding the captured image has elapsed. That is, the holding unit 212 performs a process to determine whether the image retention time, which has been set in advance as the time for one batch of captured images to be received by the acquisition unit 211, has elapsed. If the holding unit 212 determines that the image retention time has elapsed (YES in S1540), the process proceeds to step S1550. On the other hand, if the holding unit 212 determines that the image retention time has not elapsed (NO in S1540), the process returns to step S1510 and is repeated. In step S1550, the control unit 210 divides all the acquired image images, similar to step S840 described above. That is, the dividing unit 213 divides the image of the product 120 temporarily stored by the holding unit 212 into multiple regions. After that, as shown in Figure 11, the same process as the process from step S850 onwards described above is performed, so the explanation is omitted.

[0117] Next, the process of dividing an captured image into multiple regions (tiles) in the information processing system 2000 of Embodiment 2 will be described. Figure 18 is a schematic diagram showing an example of the captured image division process in the information processing device 2100 of Embodiment 2.

[0118] In Figure 18, 64 captured images 2200, such as the captured image 2200-1 of the first product 120, the captured image 2200-2 of the second product 120, ... the captured image 2200-k of the kth product 120, ... the captured image 2200-64 of the 64th product 120, are shown to be held as one batch. Therefore, in the information processing system 2000 of Embodiment 2, the captured images 2200 held as one batch (for example, 64 images) are input into the trained model and inspected in parallel.

[0119] Furthermore, the process of outputting a list of divided region (tile) images in the information processing system 2000 of Embodiment 2 will be described. Figure 19 is a schematic diagram showing an example of the image list output process for each divided region in the information processing device 2100 of Embodiment 2.

[0120] In Figure 19, all 64 captured images 2200 are shown to be divided into two horizontally and three vertically, resulting in six regions: upper left region 220011, middle left region 220012, lower left region 220013, upper right region 220021, middle right region 220022, and lower right region 220023. It is also shown that a list image of 64 upper left regions 220011 is output, such as upper left region 220011-1 of the first product 120's captured image 2200-1, upper left region 220011-2 of the second product 120's captured image 2200-2, ... upper left region 220011-k of the kth product 120's captured image 2200-k, ... upper left region 220011-64 of the 64th product 120's captured image 2200-64. Therefore, in the information processing system 2000 of Embodiment 2, by continuously processing one region (tile) of the captured image for one batch, efficient and highly accurate re-inspection can be performed.

[0121] Furthermore, the process for outputting information identifying the defective product 120 in the information processing system 2000 of Embodiment 2 will be described. Figure 20 is a schematic diagram showing an example of the defective product identification information output process in the information processing device 2100 of Embodiment 2.

[0122] In Figure 20, the buffer processing device 170 is shown to output defective product identification information consisting of a map image 2300 with colored circles marking the locations of the first defective product 120 (NG product location 230054) and the second defective product 120 (NG product location 230086) on an image simulating the pallet 171 of the buffer processing device 170. The defective product identification information may be output as coordinate information where the storage positions of the products 120 on the pallet 171 of the buffer processing device 170 are assigned numbers from 1 to 8 from left to right, and numbers from 1 to 8 from top to bottom, for example, first defective product (5,4), second defective product (8,6).

[0123] Therefore, in the information processing system 2000 of Embodiment 2, it is possible to easily identify which product 120 needs to be reinspected based on location information and visual information, enabling efficient and highly accurate reinspection. In particular, as in this embodiment, if the product 120 is relatively small in size, it is easy to move it from the pallet 171 to a predetermined discharge container and sort it according to the presence or absence of defects. On the other hand, in the case of a relatively large product 120, for example, by identifying the presence or absence of defects based on the location information and visual information of the product 120, it is not necessary to move the product 120 itself according to the presence or absence of defects.

[0124] As described above, according to Embodiment 2, by performing rough inspections of product images using a trained model in batches, inspections can be carried out efficiently. Furthermore, by dividing the product images into predetermined parts, it is possible to arbitrarily change and adjust the inspection frequency and inspection accuracy, such as omitting subsequent inspections or performing detailed inspections by inspectors based on the combined inspection results for the divided parts (regions).

[0125] <Notes> The system may be provided in the following embodiments. (Note 1) An information processing system having at least one control unit, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the image is input to a trained model, the trained model is a trained model that has been trained to use the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data, and the system outputs information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector. (Note 2) An information processing system according to Note 1, wherein the trained model is a trained model that has been trained to use an image of a product free of defects captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data. (Note 3) An information processing system according to Note 1, wherein the output data is an image indicating the location of a defect in the product. (Note 4) An information processing system as described in Note 3, wherein the output data is an image showing the location and size of a defect in the product. (Note 5) An information processing system as described in Note 4, wherein the output data is heatmap data showing the location and size of a defect in the product. (Note 6) An information processing system as described in Note 1, which generates display data by superimposing the captured image and the output data. (Note 7) An information processing system as described in Note 6, wherein the control unit outputs information to an inspector performing an inspection of the product indicating that the product requires inspection by the inspector, and the display data. (Note 8) An information processing system as described in Note 1, wherein the control unit acquires the results of the inspection by the inspector and retrains the trained model based on the inspection results.(Note 9) An information processing method to be performed by an information processing system, comprising: receiving an image of a product from an imaging device; the imaging device is a device that acquires an image of the product using a laser; inputting the image of the product into a trained model; the trained model is a trained model that has been trained using the image of the product captured by the imaging device as input data and data indicating whether or not the product has defects as output data; and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector. (Note 10) An information processing system comprising at least one control unit and a storage unit capable of storing a plurality of products, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the control unit inputs the image of the product into a trained model, the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating whether or not the product has defects as output data, stores an array of images of the product acquired by the imaging device, temporarily stores the array of products after imaging by the imaging device in the storage unit, and outputs output data indicating whether or not the array of products has defects by processing the images of the array of products stored in the storage unit in succession using the trained model, and outputs information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector. (Note 11) An information processing system according to Note 10, wherein the control unit divides the captured image into a plurality of regions, and processes the plurality of captured images sequentially for each corresponding region using the trained model provided for each divided region. (Note 12) An information processing system according to Note 11, wherein if a defect is detected in one of the plurality of divided regions, the control unit outputs information indicating that inspection by the inspector is required without performing the processing in the other regions.(Note 13) An information processing system as described in Note 11, wherein the control unit outputs information regarding the suspension of production of the product when defects are detected in the captured images in one of the divided regions. (Note 14) An information processing system as described in Note 11, wherein the control unit does not perform the processing in the other regions or reduces the accuracy of the processing that determines a defect when no defects are detected in one of the divided regions. (Note 15) An information processing system as described in Note 11, wherein the output data is a plurality of captured images grouped together for each divided region. (Note 16) An information processing system as described in Note 11, wherein the control unit determines the timing for performing the processing according to the time taken when a plurality of the products are stored in the storage unit and a time set in advance according to the number of captured images. (Note 17) An information processing method to be executed by an information processing system, comprising: receiving an image of a product from an imaging device; the imaging device is a device that acquires an image of the product using a laser; inputting the image of the product into a trained model; the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating whether or not the product has defects as output data; storing a plurality of images of the product acquired by the imaging device; temporarily storing the plurality of products after imaging by the imaging device in a storage unit capable of storing a plurality of products; outputting output data indicating whether or not the plurality of products have defects by processing the image of the plurality of products stored in the storage unit in succession using the trained model; and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector. (Note 18) A program to cause a computer to function as an information processing system as described in any one of Notes 1 to 8 and Notes 10 to 16.

[0126] While various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. Novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. Embodiments and variations thereof are included in the scope and spirit of the invention, as well as in the claims and their equivalents. Furthermore, each embodiment and variation may be implemented in any combination.

[0127] 100, 2100: Information processing device 110: Camera 120: Product 150: Network 170: Buffer processing device 210: Control unit 220: Storage unit 230: Communication unit 240: Internal bus 1000, 2000: Information processing system

Claims

1. An information processing system comprising at least one control unit, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the image is input to a trained model, the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating whether or not the product has defects as output data, and based on the output data, outputs information to an inspector indicating that the product requires inspection by the inspector.

2. An information processing system according to claim 1, wherein the trained model is a trained model that has been trained using an image of a product free of defects, captured by the imaging device, as input data and data indicating the presence or absence of defects in the product as output data.

3. An information processing system according to claim 1, wherein the output data is an image showing the location of a defect in the product.

4. An information processing system according to claim 3, wherein the output data is an image showing the location and size of a defect in the product.

5. An information processing system according to claim 4, wherein the output data is heatmap data indicating the location and size of defects in the product.

6. An information processing system according to claim 1, which generates display data by superimposing the captured image and the output data.

7. An information processing system according to claim 6, wherein the control unit outputs to an inspector performing an inspection of the product information indicating that the product requires inspection by the inspector and the display data.

8. An information processing system according to claim 1, wherein the control unit acquires the results of an inspection by the inspector and retrains the trained model based on the results of the inspection.

9. An information processing method to be performed by an information processing system, comprising: receiving an image of a product from an imaging device; the imaging device is a device that acquires an image of the product using a laser; inputting the image of the product into a trained model; the trained model is a trained model that has been trained using the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data; and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector.

10. An information processing system comprising at least one control unit and a storage unit capable of storing a plurality of products, wherein the control unit receives an image of a product from an imaging device, the imaging device is a device that acquires an image of the product using a laser, the control unit inputs the image of the product into a trained model, the trained model is a trained model that has learned to use the image of the product captured by the imaging device as input data and data indicating the presence or absence of defects in the product as output data, stores an array of images of the product acquired by the imaging device, temporarily stores the array of products after imaging by the imaging device in the storage unit, and outputs output data indicating the presence or absence of defects in the array of products by processing the images of the array of products stored in the storage unit in succession using the trained model, and outputs information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector.

11. An information processing system according to claim 10, wherein the control unit divides each of the captured images into a plurality of regions, and processes the plurality of captured images sequentially for each corresponding region using the learned model provided for each of the divided regions.

12. An information processing system according to claim 11, wherein the control unit outputs information indicating that an inspection by the inspector is required when a defect is detected in one of the divided regions, without performing the processing in the other regions.

13. An information processing system according to claim 11, wherein the control unit outputs information regarding the suspension of production of the product when defects are continuously detected in the captured image in one of the divided regions.

14. An information processing system according to claim 11, wherein the control unit, if no defect is detected in one of the divided regions, does not perform the processing in the other regions, or reduces the accuracy of the processing that determines a defect.

15. An information processing system according to claim 11, wherein the output data is a plurality of captured images grouped together for each of the divided regions.

16. An information processing system according to claim 11, wherein the control unit determines the timing for performing the processing according to the time taken when a plurality of the products are stored in the storage unit and a time set in advance according to the number of captured images.

17. An information processing method to be executed by an information processing system, comprising: receiving captured images of a product from an imaging device; the imaging device is a device that acquires captured images of the product using a laser; inputting the captured images together into a trained model; the trained model is a trained model that has been trained using captured images of the product taken by the imaging device as input data and data indicating the presence or absence of defects in the product as output data; storing multiple captured images of the product acquired by the imaging device; temporarily storing the multiple products after imaging by the imaging device in a storage unit capable of storing multiple products; outputting output data indicating the presence or absence of defects in multiple products by continuously processing the captured images of the multiple products stored in the storage unit using the trained model; and outputting information to an inspector who inspects the product based on the output data indicating that the product requires inspection by the inspector.

18. A program for causing a computer to function as an information processing system according to any one of claims 1 to 8 and 10 to 16.

Images