Information processing system and control method

Abstract

To improve a technology for imaging an object transported by transportation means using a camera to determine quality of the object.SOLUTION: An information processing system comprising a first image generating section 11 for imaging a first area A1 to generate an invisible light image, a second image generating section 12 for imaging a second area A2 to generate a visible light image, a storage section 14, and a control section 15 inputs the invisible light image generated by the first image generating section 11 to a first learning model; on the basis of a first output signal of the first learning model, determines whether or not an object 30 is included in the invisible light image; when the object 30 is included in the invisible light image, identifies timing at which the object 30 is positioned in the second area A2; controls the second image generating section 12 so as to image the second area A2 when the identified timing comes; inputs the generated the visible light image to a second learning model; and determines quality of the object 30 on the basis of a second output signal of the second learning model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and a control method. [Background technology]

[0002] A conventional technique is known in which an object conveyed by a conveying means such as a belt conveyor is imaged with a camera to determine the quality of the object. For example, Patent Document 1 discloses that, in order to obtain a still image when an object to be inspected or sorted reaches an appropriate imaging position of a solid-state imaging camera, position detection sensors such as proximity switches and infrared sensors are disposed above the solid-state imaging camera in the direction of movement of the object and at the side or above the conveying means such as a belt conveyor that conveys the object, and these position detection sensors detect the passage of the object and issue triggers, which are sent to the solid-state imaging camera with a delay time that is preset according to the speed of the conveying means, the size of the object, the position to be inspected, etc., and when this trigger arrives, the electric charge accumulated across the entire surface of the light-receiving element of the solid-state imaging device is read out as an image signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4210844 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the method using a proximity switch or infrared sensor as a position detection sensor for an object, if there are individual differences in the shape or posture of each object conveyed sequentially by a conveying means, the timing at which the position detection sensor detects the object may differ for each object, and as a result, the object may not be captured by a camera. Therefore, there is room for improvement in the technology for capturing images of objects conveyed by a conveying means with a camera to determine the quality of the object.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for capturing an image of an object conveyed by a conveying means using a camera and determining the quality of the object. [Means for solving the problem]

[0006] An information processing system according to an embodiment of the present disclosure includes: An information processing system comprising: a first image generating unit that captures an image of a first area through which an object conveyed by a conveying device passes and generates a non-visible light image; a second image generating unit that captures an image of a second area through which the object passes after the first area and generates a visible light image; a storage unit that stores a first learning model and a second learning model; and a control unit, The control unit inputting a first input signal including the non-visible light image generated by the first image generation unit into the first learning model; determining whether the object is included in the non-visible light image based on a first output signal of the first learning model in response to the first input signal; If it is determined that the object is included in the non-visible light image, identifying the timing at which the object is located within the second region; controlling the second image generation unit to capture an image of the second region and generate a visible light image when the specified timing arrives; inputting a second input signal including the generated visible light image into the second learning model; The quality of the object is determined based on a second output signal of the second learning model for the second input signal.

[0007] A control method according to an embodiment of the present disclosure includes: A control method for an information processing system including: a first image generation unit that captures an image of a first area through which an object conveyed by a conveying device passes and generates an invisible light image; a second image generation unit that captures an image of a second area through which the object passes after the first area and generates a visible light image; a storage unit that stores a first learning model and a second learning model; and a control unit, The control unit inputting a first input signal including the non-visible light image generated by the first image generation unit into the first learning model; determining whether the object is included in the non-visible light image based on a first output signal of the first learning model for the first input signal; When it is determined that the object is included in the non-visible light image, identifying the timing at which the object passes through the second area; controlling the second image generating unit to capture an image of the second region and generate a visible light image when the specified timing arrives; inputting a second input signal including the generated visible light image into the second learning model; and determining the quality of the object based on a second output signal of the second learning model for the second input signal. [Effects of the Invention]

[0008] According to one embodiment of the present disclosure, there is provided an improved technique for capturing an image of an object conveyed by a conveying means with a camera to determine the quality of the object. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a schematic configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a diagram illustrating how an object is transported. [Figure 3] 10 is a flowchart showing the operation of the information processing system. [Figure 4] FIG. 10 is a diagram showing an example of a non-visible light image including a target object. [Figure 5] FIG. 10 is a diagram illustrating an illumination unit included in an information processing system according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described.

[0011] (Outline of the embodiment) An overview of an information processing system 10 according to an embodiment of the present disclosure will be described with reference to Figures 1 and 2. As shown in Figure 1, the information processing system 10 includes a first image generation unit 11, a second image generation unit 12, a communication unit 13, a storage unit 14, and a control unit 15. Details of each component of the information processing system 10 will be described later.

[0012] Typically, as shown in FIG. 2, the information processing system 10 is used in a production factory or the like for determining the quality of each object 30 conveyed by a conveying device 20 such as a belt conveyor. The object 30 is, for example, a structure or food, but is not limited to these and may be any product. The shape of the object 30 and the posture of the object 30 when conveyed by the conveying device 20 may vary from object to object. As shown in FIG. 2, each object 30 is conveyed sequentially by the conveying device 20 to the right in the drawing, passing through a first area A1 and a second area A2.

[0013] First, an overview of this embodiment will be described, and details will be described later. The first image generation unit 11 is capable of capturing an image of a first region A1 through which an object 30 conveyed by a conveying device 20 passes, and generating a non-visible light image (e.g., a distance image). The second image generation unit 12 is capable of capturing an image of a second region A2 through which the object 30 passes after the first region A1, and generating a visible light image. The storage unit 14 shown in FIG. 1 stores a first learning model and a second learning model. The control unit 15 inputs a first input signal including the non-visible light image generated by the first image generation unit 11 to the first learning model. The control unit 15 determines whether the object 30 is included in the non-visible light image based on a first output signal of the first learning model in response to the first input signal. If it is determined that the object 30 is included in the non-visible light image, the control unit 15 determines the timing at which the object 30 is located within the second region A2. The control unit 15 controls the second image generation unit 12 to capture an image of the second region A2 and generate a visible light image when the specified timing arrives. The control unit 15 inputs a second input signal including the generated visible light image to the second learning model. The control unit 15 then determines the quality of the object 30 based on a second output signal of the second learning model in response to the second input signal.

[0014] According to this embodiment, a determination is made using a first learning model as to whether or not the object 30 is included in the non-visible light image captured of the first region A1 (in other words, a determination is made as to the timing at which the object 30 is located within the first region A1). Furthermore, the determination that the object 30 is included in the non-visible light image triggers the determination of the timing at which the object 30 is located within the second region A2. Therefore, by using an appropriately trained first learning model, even if there are individual differences in the shape or posture of the object 30, for example, it is possible to accurately determine the timing at which the object 30 is located within the first region A1, and as a result, it is possible to accurately determine the timing at which the object 30 is located within the second region A2. Therefore, according to this embodiment, the probability that the object 30 will be included in the visible light image generated by the second image generation unit 12 is increased, thereby improving the technique for determining the quality of an object by capturing an image of an object transported by a transport means with a camera.

[0015] Next, each component of the information processing system 10 will be described in detail with reference to FIG.

[0016] (Configuration of information processing system) The information processing system 10 according to this embodiment includes a first image generating unit 11, a second image generating unit 12, a communication unit 13, a storage unit 14, and a control unit 15.

[0017] The first image generating unit 11 is provided to capture an image of a first area A1 through which an object 30 conveyed by the conveying device 20 passes, thereby generating an invisible light image. In the example shown in FIG. 2, the first image generating unit 11 is provided to capture an image of the first area A1 from above, but the installation manner of the first image generating unit 11 is not limited to this example. The first image generating unit 11 according to this embodiment includes, for example, a TOF (Time of Flight) sensor and is capable of generating a distance image as an invisible light image. A "distance image" is an image that indicates the distance from the first image generating unit 11 to the subject. However, the first image generating unit 11 is not limited to a TOF sensor and may include any distance image sensor. Alternatively, the first image generating unit 11 may include an interface for connecting an external distance image sensor.

[0018] The second image generating unit 12 is provided so as to be able to capture an image of a second region A2 through which the object 30 conveyed by the conveying device 20 passes after the first region A1, and generate a visible light image. In the example shown in FIG. 2, the second image generating unit 12 is provided so as to be able to capture an image of the second region A2 from above, but the installation manner of the second image generating unit 12 is not limited to this example. The second image generating unit 12 according to this embodiment includes, for example, a visible light camera and is able to generate a visible light image. Alternatively, the second image generating unit 12 may include an interface for connecting an external visible light camera.

[0019] The information processing system 10 may include a plurality of second image generating units 12. In one embodiment, the plurality of second image generating units 12 may be provided so as to be able to image a common second region A2 from different angles. In another embodiment, the plurality of second image generating units 12 may be provided so as to be able to image different second regions A2. In such a case, the conveying device 20 may include a mechanism (e.g., a robot arm) that changes the posture of the object 30 between one second region A2 and the next second region A2. Alternatively, a process of performing any operation on the object 30 may be included between one second region A2 and the next second region A2.

[0020] The communication unit 13 includes an interface for communicating with an external device such as a computer or a conveyance device 20. The interface may be compatible with any wired or wireless communication standard. The information processing system 10 can communicate with the external device via the communication unit 13.

[0021] The storage unit 14 includes one or more memories. In this embodiment, the "memory" may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these. Each memory included in the storage unit 14 may function as, for example, a main memory device, an auxiliary memory device, or a cache memory. The storage unit 14 stores any information used in the operation of the information processing system 10. For example, the storage unit 14 may store a system program, an application program, embedded software, and the like.

[0022] In this embodiment, the storage unit 14 stores a first learning model and a second learning model.

[0023] The first learning model is a machine learning model trained to output a first output signal indicating an inference result as to whether or not the non-visible light image contains the object 30 in response to a first input signal including the non-visible light image. The first learning model is trained, for example, by supervised learning, but is not limited to this example and may be trained by any machine learning algorithm, such as deep learning. The first output signal may be, for example, a binary signal indicating "1" when it is inferred that the non-visible light image contains the object 30 and indicating "0" when it is inferred that the non-visible light image does not contain the object 30. Typically, the first learning model is trained to infer that the non-visible light image contains the object 30 when the non-visible light image contains the entire object 30. Note that, in cases where there is a possibility of individual differences in the shape and posture of the object 30, the first learning model may be trained to infer that the non-visible light image contains the object 30 when the non-visible light image contains the entire object 30, regardless of individual differences in the object 30. Alternatively, the first output signal may be a signal that indicates, by a numerical value between 0 and 1, the likelihood (probability) that the object 30 is included in the non-visible light image.

[0024] The second learning model is a machine learning model trained to output a second output signal indicating an inference result of the quality of the object 30 included in the visible light image in response to a second input signal including the visible light image. The second learning model is trained, for example, by supervised learning, but is not limited to this example and may be trained by any machine learning algorithm, such as deep learning. The second output signal may be, for example, a binary signal indicating "1" if the object 30 is inferred to be a good product and "0" if the object 30 is inferred to be a defective product. Typically, the second learning model is trained to infer that the object 30 is a good product when the object 30 included in the visible light image does not have any defects such as scratches or damage. Note that, if there is a possibility of individual differences in the shape and orientation of the object 30, the second learning model may be trained to infer that the object 30 is a good product when the object 30 included in the visible light image does not have any such defects, regardless of individual differences in the object 30. Alternatively, the second output signal may be a signal that indicates, by a numerical value between 0 and 1, the likelihood (probability) that the object 30 is a non-defective product.

[0025] The control unit 15 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process, but is not limited to these. The programmable circuit may be, for example, a FPGA (Field-Programmable Gate Array), but is not limited to these. The dedicated circuit may be, for example, an ASIC (Application Specific Integrated Circuit), but is not limited to these. Hereinafter, the processor, programmable circuit, and dedicated circuit will be referred to as "processor, etc." without any particular distinction being made. The control unit 15 controls the operation of the entire information processing system 10.

[0026] For ease of explanation, the present embodiment will be described assuming that the above-described components of the information processing system 10 are arranged in one device. However, the components of the information processing system 10 may be distributed across multiple devices that can communicate with each other. When the above-described components are distributed across multiple devices, the control unit 15 includes multiple processors, etc., and at least one processor, etc. is arranged in each of the multiple devices. In one example, at least one of the first image generation unit 11 and the second image generation unit 12 may be arranged in one independent device (e.g., an edge device).

[0027] (Operation flow of information processing system) The operation flow of the information processing system 10 according to this embodiment will be described with reference to FIG.

[0028] Step S100: The control unit 15 of the information processing system 10 acquires the non-visible light image generated by the first image generation unit 11.

[0029] Specifically, the control unit 15 controls the first image generation unit 11 to repeatedly capture images of the first area A1 at a predetermined frame rate, for example, to generate a non-visible light image. The control unit 15 then obtains the latest non-visible light image from the first image generation unit 11 at the time step S100 is executed.

[0030] Step S101: The control unit 15 inputs the first input signal including the non-visible light image acquired in step S100 to the first learning model.

[0031] Step S102: The control unit 15 determines whether or not the object 30 is included in the non-visible light image based on the first output signal of the first learning model for the first input signal. If it is determined that the object 30 is included in the non-visible light image (step S102—Yes), the process proceeds to step S103. On the other hand, if it is determined that the object 30 is not included in the non-visible light image (step S102—No), the process returns to step S100.

[0032] Any method can be used to determine whether the object 30 is included in the non-visible light image. For example, as described above, a case will be described in which the first output signal is a binary signal that indicates "1" when it is inferred that the object 30 is included in the non-visible light image and indicates "0" when it is inferred that the object 30 is not included in the non-visible light image. The control unit 15 may determine that the object 30 is included in the non-visible light image when the first output signal indicates "1," and may determine that the object 30 is not included in the non-visible light image when the first output signal indicates "0."

[0033] Further, for example, a case will be described in which the first output signal is a signal indicating the likelihood (probability) that the object 30 is included in the invisible light image as a numerical value between 0 and 1, as described above. The control unit 15 may determine that the object 30 is included in the invisible light image if the numerical value indicated in the first output signal is equal to or greater than a predetermined first reference value, and may determine that the object 30 is not included in the invisible light image if the numerical value is less than the first reference value. Here, the first reference value may be correctable by an operator or may be correctable automatically by the control unit 15.

[0034] Regardless of which method is adopted, by using an appropriately trained first learning model, it is possible to accurately determine whether or not the object 30 is included in the invisible light image. For example, three patterns of invisible light images P1 to P3 shown in Fig. 4 each show an object 30 with a different posture. As described above, the first learning model is trained to infer that the invisible light image contains the object 30 when the entire object 30 is captured in the invisible light image, regardless of individual differences in the posture of the object 30. For example, the first learning model can infer that the invisible light image contains the object 30 or estimate that there is a high likelihood (probability) that the object 30 is included in the invisible light image for a first input signal including any of the invisible light images P1 to P3.

[0035] It should be noted that the processing time for steps S100 to S102 is sufficiently short, and therefore, "determining that the object 30 is included in the invisible light image" is practically equivalent to "determining that the object 30 is located within the first area A1." In other words, in step S102, the control unit 15 determines whether or not the object 30 is located within the first area A1 based on the first output signal.

[0036] Step S103: If it is determined in step S102 that the object 30 is included in the non-visible light image (S102-Yes), the control unit 15 identifies the timing at which the object 30 is located within the second area A2.

[0037] Any method can be used to determine the timing. For example, the control unit 15 calculates the transport time required for the object 30 to be transported from the first region A1 to the second region A2 based on the transport speed of the transport device 20 and the distance from the first region A1 to the second region A2. Here, information indicating the transport speed may be stored in advance in the storage unit 14 or may be acquired from the transport device 20 via the communication unit 13. Furthermore, information indicating the distance from the first region A1 to the second region A2 may be stored in advance in the storage unit 14. Then, the control unit 15 may determine, based on the first time when the invisible light image determined in step S102 to include the object 30 was captured (i.e., the first time when it was determined that the object 30 was located in the first region A1), the time when the transport time has elapsed since the first time. In addition, the control unit 15 may acquire information indicating the conveying speed from the conveying device 20 in real time until that timing arrives at the latest, and sequentially correct the time required for the object 30 to be conveyed from the first area A1 to the second area A2 (and the timing at which the object 30 is located within the second area A2).

[0038] Step S104: The control unit 15 controls the second image generating unit 12 so as to capture an image of the second area A2 and generate a visible light image when the timing specified in S103 arrives.

[0039] Specifically, the control unit 15 outputs a control command including information indicating the identified timing to the second image generation unit 12. In accordance with the control command, the second image generation unit 12 captures an image of the second area A2 when the timing arrives to generate a visible light image.

[0040] When controlling second image generation unit 12, control unit 15 may power on second image generation unit 12 or cancel the sleep mode of second image generation unit 12, and when the specified timing arrives, have second image generation unit 12 capture the image of second area A2 and generate a visible light image, and then power off second image generation unit 12 or transition second image generation unit 12 to the sleep mode. With this configuration, there is no need to keep second image generation unit 12 operating all the time, and therefore power consumption of information processing system 10 can be reduced.

[0041] Step S105: The control unit 15 acquires the visible light image generated in S104, and inputs a second input signal including the visible light image to the second learning model.

[0042] Step S106: Based on the second output signal of the second learning model for the second input signal, the control unit 15 determines the quality of the object 30. After that, the process returns to step S100.

[0043] Any method can be used to determine the quality of the object 30. For example, as described above, the second output signal is a binary signal that indicates "1" if the object 30 is inferred to be a good product and indicates "0" if the object 30 is inferred to be a defective product. The control unit 15 may determine that the object 30 is a good product when the second output signal indicates "1" and determine that the object 30 is a defective product when the second output signal indicates "0." Alternatively, the control unit 15 may store a score indicating the high quality of the object 30 in the storage unit 14 in advance, and may add a predetermined value to the score indicating the high quality of the object 30 when the second output signal indicates "1," and subtract a predetermined value from the score when the second output signal indicates "0." This method using a score may be used in an embodiment in which the information processing system 10 includes multiple second image generation units 12.

[0044] Further, for example, a case will be described in which the second output signal is a signal indicating the likelihood (probability) that the object 30 is a good product using a numerical value between 0 and 1, as described above. The control unit 15 may determine that the object 30 is a good product if the numerical value indicated by the second output signal is equal to or greater than a predetermined second reference value, and may determine that the object 30 is a defective product if the numerical value is less than the second reference value. Here, the second reference value may be correctable by an operator or may be correctable automatically by the control unit 15. Alternatively, the control unit 15 may store a score indicating the high quality of the object 30 in the storage unit 14 in advance and increase or decrease the score based on the numerical value indicated by the second output signal. In one example, when the numerical value indicated by the second output signal is equal to or greater than the second reference value, the control unit 15 may increase the score more significantly as the numerical value increases. On the other hand, when the numerical value indicated by the second output signal is less than the second reference value, the control unit 15 may decrease the score more significantly as the numerical value decreases. This approach using scores may be adopted in an embodiment in which the information processing system 10 includes multiple second image generators 12.

[0045] As described above, the information processing system 10 according to this embodiment includes the first image generation unit 11 that captures an image of a first region A1 through which the object 30 conveyed by the conveying device 20 passes and generates an invisible light image; the second image generation unit 12 that captures an image of a second region A2 through which the object 30 passes after the first region A1 and generates a visible light image; a storage unit 14 that stores a first learning model and a second learning model; and a control unit 15. The control unit 15 inputs a first input signal including the invisible light image generated by the first image generation unit 11 to the first learning model. The control unit 15 determines whether the object 30 is included in the invisible light image based on a first output signal of the first learning model in response to the first input signal. If it is determined that the object 30 is included in the invisible light image, the control unit 15 identifies the timing at which the object 30 is located within the second region A2. The control unit 15 controls the second image generation unit 12 to capture an image of the second region A2 and generate a visible light image at the identified timing. The control unit 15 inputs a second input signal including the generated visible light image to the second learning model, and determines the quality of the object 30 based on a second output signal of the second learning model in response to the second input signal.

[0046] According to this configuration, the first learning model is used to determine whether the object 30 is included in the invisible light image captured of the first region A1 (in other words, to determine the timing at which the object 30 is located within the first region A1). Furthermore, the determination that the object 30 is included in the invisible light image triggers the identification of the timing at which the object 30 is located within the second region A2. Therefore, by using an appropriately trained first learning model, even if there are individual differences in the shape or posture of the object 30, the timing at which the object 30 is located within the first region A1 can be accurately determined, and as a result, the timing at which the object 30 is located within the second region A2 can be accurately identified. Therefore, according to this embodiment, the probability that the object 30 will be included in the visible light image generated by the second image generation unit 12 is increased, thereby improving the technology for determining the quality of an object by capturing an image of an object transported by a transport means with a camera.

[0047] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0048] In the above-described embodiment, an example has been described in which first image generation unit 11 generates a distance image as the invisible light image. In other embodiments, first image generation unit 11 may include a thermal image sensor and be capable of generating a thermal image as the invisible light image. Such a configuration may be employed when the product, which is object 30, is a product that heats up during transportation, such as a casting or heated food.

[0049] Furthermore, in the above-described embodiment, the resolution of the non-visible light image generated by the first image generation unit 11 may be lower than the resolution of the visible light image generated by the second image generation unit 12. Alternatively, the number of pixels constituting the non-visible light image generated by the first image generation unit 11 may be smaller than the number of pixels constituting the visible light image generated by the second image generation unit 12. In the field of image processing, generally, a process for detecting the presence or absence of the object 30 itself from an image requires a lower image resolution or fewer pixels than a process for detecting the presence or absence of defects, such as scratches or damage, on the object 30 from an image. Generally, an image with a higher resolution or a larger number of pixels imposes a greater image processing load. Therefore, according to the above-described configuration in which the resolution of the non-visible light image generated by the first image generation unit 11 is relatively low or the number of pixels constituting the non-visible light image is relatively small, the processing load of the information processing system 10 can be reduced compared to, for example, an embodiment in which a non-visible light image having a resolution or number of pixels similar to that of the visible light image generated by the second image generation unit 12 is used.

[0050] Furthermore, in the above-described embodiment, the object 30 may be affixed with a marker including, for example, a one-dimensional code or a two-dimensional code. In such a case, the control unit 15 may acquire identification information of the object 30 based on the marker included in the visible light image generated by the second image generation unit 12. The control unit 15 may also store information indicating the quality assessment result of the object 30 (e.g., information indicating whether the object 30 is a good product or a defective product) in association with the acquired identification information in the storage unit 14. With this configuration in which information indicating the quality assessment result is stored in association with the identification information of the object 30, information about the object 30 including the quality assessment result can be easily referenced later, for example, by searching using the identification information as a query, thereby improving the traceability of the object 30.

[0051] In the above-described embodiment, the control unit 15 may count the number of objects 30 determined to be good and the number of objects 30 determined to be defective, for example, at predetermined time intervals or for each lot of objects 30. The control unit 15 may also calculate, for example, at predetermined time intervals or for each lot of objects 30, the ratio of the number of objects 30 determined to be good or the number of objects 30 determined to be defective to the total number of objects 30 whose quality has been determined. Furthermore, if the number or ratio of objects 30 determined to be defective increases over time, the control unit 15 may notify a predetermined terminal that there may be an abnormality in a process upstream of the transport process of the objects 30. This allows a transport process manager or the like using the predetermined terminal to recognize that there may be an abnormality in a process upstream of the transport process of the objects 30.

[0052] Furthermore, in the above-described embodiment, the conveying device 20 may have a defective product moving mechanism that moves the object 30 determined to be a defective product by the control unit 15 to a device related to a different process from the conveying device 20. In this case, when the control unit 15 determines that the object 30 is a defective product, the control unit 15 may control the defective product moving mechanism to move the object 30 determined to be a defective product to a device related to a different process from the conveying device 20. This allows the objects 30 determined to be non-defective to be efficiently separated from the objects 30 determined to be defective in the conveying process.

[0053] In the above-described embodiment, the information processing system 10 may further include an illumination unit 16 capable of irradiating the object 30 located within the second area A2 with illumination light in multiple illumination patterns, each of which differs in at least one of brightness, color, and illumination direction. For example, the illumination unit 16 shown in FIG. 5 includes a first illumination unit 16A attached to the second image generation unit 12 and a ring-shaped second illumination unit 16B located above the second area A2. Each of the first illumination unit 16A and the second illumination unit 16B includes one or more light sources. Each of the first illumination unit 16A and the second illumination unit 16B may include multiple light sources provided at different positions. However, the specific shape and arrangement of the illumination unit 16 are not limited to the example shown in FIG. 5. The control unit 15 can individually control the on / off, brightness adjustment, color adjustment, etc. of each light source included in the illumination unit 16.

[0054] Furthermore, in the above embodiment in which the information processing system 10 includes the illumination unit 16, the first learning model may be a machine learning model trained to output, in response to a first input signal including an invisible light image, a first output signal including an inference result as to whether the invisible light image includes the object 30 and an inference result of the posture of the object 30. The first output signal may include, for example, a binary signal indicating "1" when it is inferred that the invisible light image includes the object 30 and "0" when it is inferred that the object 30 is not included, and a multi-value signal indicating a numerical value corresponding to the inferred posture of the object 30. Typically, the first learning model is trained to infer that the invisible light image includes the object 30 when the entire object 30 is captured in the invisible light image, and to infer a numerical value corresponding to the posture of the object 30. In one example, the first learning model may be trained to output, in response to a first input signal including the invisible light image P1 shown in FIG. 4, a first output signal including a multi-value signal indicating "1." Similarly, the first learning model may be trained to output a first output signal including a multi-value signal indicating "2" in response to a first input signal including non-visible light image P2. Similarly, the first learning model may be trained to output a first output signal including a multi-value signal indicating "3" in response to a first input signal including non-visible light image P3. Note that the first learning model may be trained to output a first output signal including a multi-value signal indicating "0" when it infers that the non-visible light image does not include the object 30. The possible postures of the object 30 are not limited to the three types shown in FIG. 4 and may include any number of types.

[0055] Here, the appropriate illumination pattern used by the second image generation unit 12 when capturing the second region A2 may vary depending on the posture of the object 30. Typically, for an object 30 in a certain posture, one illumination pattern that best improves the accuracy of the second learning model's estimation of the quality of the object 30 can be identified through experimentation or simulation. The storage unit 14 may pre-store information indicating one of the multiple illumination patterns of the illumination unit 16 (e.g., one illumination pattern that best improves the accuracy of the second learning model's estimation of the quality of the object 30) in association with each possible posture of the object 30. The control unit 15 may detect the posture of the object 30 included in the invisible light image based on a first output signal (multi-value signal) of the first learning model in response to the first input signal. The control unit 15 may then control the illumination unit 16 to irradiate the object 30 with one of the multiple illumination patterns of the illumination unit 16 that corresponds to the detected posture when the time arrives for the object 30 to be located within the second region A2. According to this embodiment, when the second image generation unit 12 generates a visible light image, illumination light is emitted in an appropriate illumination pattern depending on the posture of the object 30, thereby improving the accuracy of inferring the quality of the object 30 by the second learning model.

[0056] Also, an embodiment is possible in which, for example, a general-purpose computer functions as the information processing system 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing system 10 according to the above-described embodiment is stored in the memory of the computer, and the program is read and executed by a processor or the like of the computer. Therefore, the present disclosure can also be realized as a program executable by a processor or the like, or a non-transitory computer-readable medium storing the program. [Explanation of symbols]

[0057] 10 Information Processing Systems 11 First image generation unit 12 Second image generation unit 13 Communications Department 14 Storage section 15 Control Unit 16,16A,16B Lighting section 20. Conveyor 30 Objects

Claims

1. An information processing system including: a first image generating unit that captures an image of a first area through which an object conveyed by a conveying device passes and generates a non-visible light image; a second image generating unit that captures an image of a second area through which the object passes after the first area and generates a visible light image; a storage unit that stores a first learning model and a second learning model; and a control unit, The control unit inputting a first input signal including the non-visible light image generated by the first image generation unit into the first learning model; determining whether the object is included in the non-visible light image based on a first output signal of the first learning model in response to the first input signal; If it is determined that the object is included in the non-visible light image, identifying the timing at which the object is located within the second region; controlling the second image generation unit to capture an image of the second area and generate a visible light image when the specified timing arrives; inputting a second input signal including the generated visible light image into the second learning model; determining a quality of the object based on a second output signal of the second learning model in response to the second input signal; Information processing system.

2. 2. The information processing system according to claim 1, An information processing system, wherein the non-visible light image generated by the first image generation unit is a distance image or a thermal image.

3. 2. The information processing system according to claim 1, An information processing system, wherein a resolution of the non-visible light image generated by the first image generation unit is lower than a resolution of the visible light image generated by the second image generation unit.

4. 2. The information processing system according to claim 1, The control unit identifies the timing at which the object is located within the second area based on the time at which the non-visible light image determined to include the object was captured.

5. 2. The information processing system according to claim 1, When controlling the second image generation unit, the control unit turns on the power of the second image generation unit or cancels a sleep mode of the second image generation unit, and when the specified timing arrives, causes the second image generation unit to capture an image of the second area and generate a visible light image, and then turns off the power of the second image generation unit or transitions the second image generation unit to the sleep mode.

6. 2. The information processing system according to claim 1, a marker including a one-dimensional code or a two-dimensional code is attached to the object; The control unit acquiring identification information of the object based on the marker included in the generated visible light image; The information processing system stores information indicating the judgment result of the quality of the object in the storage unit in association with the acquired identification information.

7. 2. The information processing system according to claim 1, further comprising an illumination unit capable of irradiating an object located within the second region with illumination light in a plurality of illumination patterns; The control unit detecting a posture of the object included in the non-visible light image based on the first output signal of the first learning model in response to the first input signal; and controlling the illumination unit so that illumination light is emitted with one of the plurality of illumination patterns that corresponds to the detected posture when the specified timing arrives.

8. A control method for an information processing system including: a first image generating unit that captures an image of a first area through which an object conveyed by a conveying device passes and generates an invisible light image; a second image generating unit that captures an image of a second area through which the object passes after the first area and generates a visible light image; a storage unit that stores a first learning model and a second learning model; and a control unit, The control unit inputting a first input signal including the non-visible light image generated by the first image generation unit into the first learning model; determining whether the object is included in the non-visible light image based on a first output signal of the first learning model for the first input signal; When it is determined that the object is included in the non-visible light image, identifying a timing at which the object passes through the second area; controlling the second image generating unit to capture an image of the second area and generate a visible light image when the specified timing arrives; inputting a second input signal including the generated visible light image into the second learning model; determining a quality of the object based on a second output signal of the second learning model for the second input signal; A control method comprising:

Images