Information processing system, information processing device, and information generation method

Abstract

An information processing system according to one embodiment of the present disclosure comprises: a first imaging unit that acquires an image by imaging an inspection object; and a processing unit that generates information which includes the image obtained by the first imaging unit, imaging-conditions information from when the inspection object was captured, identification information of an inspector who has inspected the inspection object on the basis of the image, and inspection-results information of the inspection object.

Description

Information processing system, information processing device, and information generation method

[0001] The present disclosure relates to an information processing system, an information processing device, and an information generation method.

[0002] In appearance inspections in manufacturing processes, for example, an inspection method has been proposed that aims to improve the accuracy and efficiency of inspections by using a robot to control the position and orientation of the camera and illuminator to capture images of the workpiece from all angles, learning image data that encompasses the shooting conditions, and evaluating the effectiveness of the shooting conditions.In appearance inspections using a camera, important shooting conditions include the position and angle from which the light should be irradiated onto the object to be inspected, and the position and angle at which the camera should be positioned to maximize the contrast of the reflected light (for example, when detecting scratches, the difference in brightness between the scratched area and other areas).

[0003] Patent Document 1 proposes dividing an area of ​​an entire inspection object photographed with a wide-angle camera into multiple locations, photographing inspection images of each area with a high-speed camera under all of the various photographing conditions, and determining by learning the photographing conditions under which predetermined conditions are met from the inspection results. This proposal is an effective technique for automating all or part of visual inspection by a robot, but it is premised on the fact that numerical criteria for defect judgment are set in advance.

[0004] International Publication No. 2020 / 194541

[0005] However, in reality, inspections for defects and other defects are sensory inspections, and judgment criteria, i.e., inspection standards, are generally established initially through human visual judgment, including the shape of defects that cannot be fully expressed by simple numerical criteria. Inspection standards are also necessary to verify the effectiveness of photography conditions, and in order to establish such inspection standards, various information is required, such as a reasonable number of defective samples and the inspection results for those samples. However, it is difficult to obtain appropriate information for creating inspection standards, making it difficult to efficiently create inspection standards for visual inspections and other purposes.

[0006] Therefore, the present disclosure proposes an information processing system, an information processing device, and an information generation method that are capable of efficiently creating inspection standards.

[0007] An information processing system according to one embodiment of the present disclosure includes a first imaging unit that photographs an object to be inspected to obtain an image, and a processing unit that generates information including the image obtained by the first imaging unit, information about the photographing conditions when the object to be inspected was photographed, identification information of the inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

[0008] An information processing device according to one embodiment of the present disclosure includes a processing unit that generates information including an image obtained by a first imaging unit that photographs an object to be inspected, information about the imaging conditions when the object to be inspected was photographed, identification information of the inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

[0009] An information generation method according to one embodiment of the present disclosure generates information including an image obtained by a first imaging unit that photographs an object to be inspected, information about the imaging conditions when the object to be inspected was photographed, identification information of the inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

[0010] 1 is a diagram illustrating an example configuration of a visual inspection system according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example configuration of a visual inspection system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating surface normal information for each partial region of an inspection object according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating position information and posture information of a camera and a lighting unit, and position information and surface normal information for each partial region of an inspection object according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example configuration of metadata of shooting conditions according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example configuration of metadata of an image file according to an embodiment of the present disclosure. FIG. 6 is a flowchart illustrating the flow of a shooting condition learning process according to an embodiment of the present disclosure. FIG. 7 is a flowchart illustrating the flow of an expert certification process according to an embodiment of the present disclosure. FIG. 8 is a flowchart illustrating the flow of a judgment skill evaluation process according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example analysis of metadata according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example index of proficiency according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example index of accuracy and stability according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example index of variance according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example measurement of durability and quick decision-making ability according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example measurement of persistence according to an embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example systematization of skill levels according to an embodiment of the present disclosure. FIG. 16 is a diagram illustrating an example configuration of hardware.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the embodiments do not limit the systems, devices, methods, etc. according to the present disclosure. Furthermore, in the following embodiments, essentially the same components are designated by the same reference numerals, and redundant explanations will be omitted. The following embodiments also include examples and modified examples.

[0012] One or more of the following embodiments can be implemented independently. However, at least a portion of the following embodiments may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, each embodiment may contribute to solving different purposes or problems, and may provide different effects. The effects of each embodiment are merely examples and are not intended to be limiting, and other effects may also be provided.

[0013] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration example of visual inspection system 1-2. Operation example of robot system 1-3. Operation example of working device and storage device 1-4. Processing example of visual inspection system 1-5. Analysis example of metadata 1-6. Systematization example of skill level 1-7. Modified example of visual inspection system 1-8. Actions and effects 2. Other embodiments 3. Configuration example of hardware 4. Supplementary notes

[0014] <1. Embodiment> <1-1. Configuration Example of Visual Inspection System> An example of the configuration of a visual inspection system 1 according to this embodiment will be described with reference to Fig. 1 and Fig. 2. Each of Fig. 1 and Fig. 2 is a diagram showing an example of the configuration of the visual inspection system 1 according to this embodiment. The visual inspection system 1 according to this embodiment is an example of an information processing system.

[0015] 1 and 2 , the visual inspection system 1 according to this embodiment includes a robot system 10, an operating device 20, a storage device 30, and a learning device 40. The robot system 10, the operating device 20, the storage device 30, and the learning device 40 are configured to be able to communicate (transmit and receive) various types of information via a network 50.

[0016] (Robot System) The robot system 10 includes a robot device 11 and a robot control device 12. The robot system 10 is a system that photographs an inspection object A1 (see FIG. 2) for inspection. In the example of FIG. 2, the inspection object A1 is placed on an inspection table 13. The inspection object A1 is, for example, a circular metal part, but may be made of various materials and have various shapes.

[0017] The robot device 11 includes a polarization camera 111, a camera 112, an illumination unit 113, and a robot arm 114. The robot arm 114 includes a pair of arms 114a and 114b (see FIG. 2).

[0018] The polarization camera 111 is a camera that photographs (captures) the inspection object A1 to obtain a polarization image. The polarization camera 111 is installed above the inspection table 13 (see FIG. 2 ) and captures an overhead image of the entire inspection object A1 placed on the placement surface, which is the upper surface of the inspection table 13. The captured polarization image contains information on the degree of polarization (DOP) and polarization direction (surface normal) of light, and it is possible to visualize the information on the degree of polarization and polarization direction on a pixel-by-pixel basis. The polarization camera 111 functions as a second imaging unit.

[0019] The camera 112 is an inspection image acquisition camera that photographs (takes an image of) the inspection object A1 to obtain an inspection image. The camera 112 is attached to the tip of one arm 114a of the robot arm 114 (see FIG. 2). The camera 112 may be a normal RGB camera (color camera), or may be a non-visible light (UV, near-infrared, thermal, etc.) camera or a polarization camera depending on the inspection purpose. The camera 112 functions as a first imaging unit.

[0020] The illumination unit 113 is an illuminator that irradiates the inspection target A1 with light. The illumination unit 113 is attached to the tip of the other arm 114b of the robot arm 114 (see FIG. 2). The illumination unit 113 may include a light source such as an LED light bulb, a fluorescent lamp, or an incandescent light bulb, or may include a light source that irradiates invisible light (UV, near-infrared, thermal, etc.) depending on the inspection purpose.

[0021] The robot control device 12 includes a control unit 121 and an arm control unit 122. The robot control device 12 functions as an information processing device.

[0022] The control unit 121 controls, for example, the polarization camera 111, the camera 112, the lighting unit 113, etc. For example, the control unit 121 may be any of a desktop PC (personal computer), a notebook PC, a tablet device, a smartphone, etc. The control unit 121 can access the operation device 20, the storage device 30, the learning device 40, etc. via the network 50.

[0023] For example, the control unit 121 controls the polarization camera 111, the camera 112, the illumination unit 113, etc. based on the shooting conditions (image capturing conditions), etc. The polarization camera 111 acquires a polarization image of the inspection object A1 in accordance with the control of the control unit 121. The camera 112 acquires an inspection image of the inspection object A1 in accordance with the control of the control unit 121. The polarization image and the inspection image are taken into the control unit 121 together with metadata such as shooting condition information, and are sent to the storage device 30 via the network 50. For example, when a series of shooting operations is completed, the multiple inspection images obtained by the series of shooting operations are sent to the storage device 30 together with the corresponding metadata.

[0024] The arm control unit 122 controls the robot arm 114, i.e., the arm 114a and the arm 114b (see FIG. 2). The arm control unit 122 controls the arm 114a and the arm 114b within the range of motion of both the arm 114a and the arm 114b, and can freely adjust the positions and attitudes of the camera 112 and the lighting unit 113 independently.

[0025] For example, the arm control unit 122 sends control signals to the arms 114a and 114b based on the imaging conditions and the like, and controls the positions and angles of the joints of the arms 114a and 114b. The arm control unit 122 controls the trajectories of the arms 114a and 114b in accordance with an operation program registered in the arm control unit 122. The operation program includes an imaging program that determines the position and orientation on the trajectory at which the camera 112 will capture an inspection image, and the position and orientation on the trajectory at which the illumination unit 113 will irradiate the inspection object A1 with light during the imaging. For example, it is possible to automatically capture 100 images (obtain 100 sets of imaging conditions) by executing the operation program once.

[0026] (Operation Device) The operation device 20 has a display unit 21 and an operation unit 22. The operation device 20 may be, for example, a desktop PC (personal computer), a notebook PC, a tablet device, a smartphone, or the like. The operation device 20 can access the robot system 10, the storage device 30, the learning device 40, and the like via a network 50. The operation device 20 functions as an information processing device.

[0027] The display unit 21 displays various images. For example, the display unit 21 displays an inspection image obtained by the camera 112 of the robot device 11 on a display screen. The display unit 21 may be, for example, a liquid crystal display, an organic electroluminescence (EL) display, a head-mounted display, or the like. A plurality of display units 21 may be provided depending on the application.

[0028] The operation unit 22 accepts input operations from the assessor A2 who is the operator. For example, the operation unit 22 may be any of a keyboard, a mouse, operation keys, a touch panel, etc. When a touch panel is used, the assessor A2 performs various operations by touching the touch panel with a finger or a stylus. The operation unit 22 may also be an audio input device (for example, a microphone) that accepts input operations by voice from the assessor A2.

[0029] The judge A2 is an inspector who inspects the inspection object A1 by operating the operation unit 22 while viewing the inspection image displayed on the display unit 21. For example, the judge A2 determines whether or not there are scratches on the surface of the inspection object A1, and inputs the judgment result (inspection result) by operating the operation unit 22. The input judgment result information (inspection result information) is sent to the storage device 30, for example, via the network 50, as metadata for the inspection image.

[0030] Furthermore, the assessor A2 can operate the operation unit 22 as necessary to remotely control the robot device 11. For example, the assessor A2 can operate the operation unit 22 to change the positions and postures of the camera 112 and the lighting unit 113, the zoom of the polarization camera 111 and the camera 112, the brightness of the lighting unit 113, and other imaging conditions.

[0031] (Storage Device) The storage device 30 includes a processing unit 31 and a storage unit 32. For example, the storage device 30 may be a cloud server or a local PC. The local PC may be, for example, a desktop PC (personal computer), a notebook PC, a tablet device, a smartphone, or the like. The storage device 30 functions as an information processing device.

[0032] The processing unit 31 records various information such as the inspection image in the storage unit 32 and provides it to the operating device 20. For example, the processing unit 31 adds metadata (e.g., shooting condition information, determination result information, etc.) to the inspection image to generate information including the metadata and the inspection image. The storage unit 32 stores various information such as the generated information.

[0033] Furthermore, the processing unit 31 certifies that the assessor A2 is an expert and assesses the skill level of the assessor A2, for example, based on metadata (e.g., judgment result information) added to the inspection image. An expert is an assessor A2 who has ample experience in judgment and has accurate judgment criteria, for example, an assessor A2 whose skill level exceeds the evaluation criteria.

[0034] The inspection images and metadata (e.g., information on the imaging conditions) acquired by photographing are transmitted from the robot system 10 to the storage device 30 via the network 50, and metadata (e.g., information on the determination results) for the inspected inspection images is transmitted from the operating device 20 via the network 50. These pieces of metadata are added to the inspection images as appropriate and stored in the storage unit 32 together with the inspection images.

[0035] (Learning Device) The learning device 40 has a processing unit 41. The processing unit 41 performs machine learning such as a neural network. For example, the processing unit 41 generates a learning model using the inspection images in an inspection image folder recorded in the storage device 30 as training data. As the neural network, for example, deep learning such as a deep neural network (DNN) can be used. The learning device 40 functions as an information processing device.

[0036] As described above, metadata (e.g., imaging condition information, determination result information, etc.) is attached to the inspection image. Therefore, the learning device 40 can generate learning models for various purposes (e.g., learning models for inspection) by focusing on which information in the metadata to extract data and what to use as input values ​​for the neural network. The learning device 40 can also learn the relationships between pieces of information in the metadata.

[0037] (Network) The network 50 may be wireless and / or wired, or may be a dedicated line and / or public line. The network 50 may be any network as long as it allows communication between the robot system 10, the operating device 20, the storage device 30, and the learning device 40.

[0038] For example, the network 50 may be a communication network such as a local area network (LAN), a wide area network (WAN), a cellular network, a fixed telephone network, a regional Internet Protocol (IP) network, or the Internet. The network 50 may include a wired network or a wireless network. The network 50 may also include a core network. The core network may be, for example, an Evolved Packet Core (EPC) or a 5G Core network (5GC). The network 50 may also include a data network other than the core network. For example, the data network may be a service network of a telecommunications carrier, such as an IP Multimedia Subsystem (IMS) network. The data network may also be a private network such as an in-house network. The network 50 may also be configured as a software defined network (SDN).

[0039] 1 and 2, there is one operating device 20, but there may be multiple operating devices 20, for example, one for each assessor A2. Also, in the examples of FIGS. 1 and 2, there is one robot device 11 and one robot control device 12, but there may be multiple operating devices. Also, there is one storage device 30 and one learning device 40, but there may be multiple operating devices.

[0040] Furthermore, the processing unit 31 of the storage device 30 may be provided in a device other than the storage device 30, such as the operation device 20, the learning device 40, the robot control device 12, or another new device. In other words, some or all of the various processes, such as the process of adding metadata to an inspection image, the process of certifying an expert, and the process of determining the skill level of the assessor A2, may be executed by the operation device 20, the learning device 40, the robot control device 12, or another new device other than the storage device 30. In this case, the storage device 30 is dedicated to recording and managing various types of information.

[0041] <1-2. Operation Example of Robot System> An operation example of the robot system 10 according to this embodiment will be described with reference to Figs. 3 to 5. Fig. 3 is a diagram for explaining surface normal information for each partial region A1a of the inspection target A1 according to this embodiment. Fig. 4 is a diagram for explaining position information and orientation information of the camera 112 and the illumination unit 113 according to this embodiment, and position information and surface normal information for each partial region A1a of the inspection target A1. Fig. 5 is a diagram showing an example of the configuration of metadata (metadata table) of imaging conditions according to this embodiment.

[0042] As shown in FIG. 3 , the polarization camera 111 of the robot device 11 captures an image of the inspection object A1 from above the inspection object A1, obtaining a polarization image of the inspection object A1. The control unit 121 of the robot control device 12 divides the polarization image of the inspection object A1 into regions (partial regions A1a) of appropriate size. For example, the control unit 121 divides the polarization image into a plurality of partial regions A1a by cutting the polarization image into a grid pattern. The polarization image contains at least polarization direction information, i.e., surface normal information. The shape of the partial regions A1a may be square, or may be other shapes such as rectangular or diamond.

[0043] In the example of FIG. 3 , the surface normal information is represented by (x, y, z, θ, φ, λ). (x, y, z) is coordinate information (position information) in three-dimensional space, and (θ, φ, λ) is orientation information (position information) in three-dimensional space. The surface normal information is used, for example, to calculate the surface shape of the inspection object A1. For example, the control unit 121 can calculate the surface shape of the inspection object A1 from the surface normal information.

[0044] As shown in FIG. 4 , the arm control unit 122 operates the arm 114a holding the camera 112 so that the desired partial area A1a becomes the imaging range of the camera 112 (the imaging range for the inspection image). For example, the arm control unit 122 determines the distance between the camera 112 and the inspection object A1 based on the FOV (field of view) of the lens of the camera 112, and operates the arm 114a so that the distance between the camera 112 and the inspection object A1 becomes the determined distance. The arm control unit 122 also operates the arm 114b holding the illumination unit 113 so that light is irradiated onto the desired partial area A1a. For example, when detecting a scratch on the surface of the inspection object A1, the arm control unit 122 operates the arm 114b holding the illumination unit 113 so that the difference in brightness between the scratched area and the remaining areas is maximized.

[0045] In the example of FIG. 4 , the position information and orientation information of the camera 112 and the lighting unit 113 when the desired partial area A1a is within the shooting range of the camera 112 are indicated by (x, y, z, θ, φ, λ). The position information of the partial area A1a is indicated by (xs, ys, xe, ye). (xs, ys, xe, ye) are position information indicating the positions of the four corners of the desired partial area A1a. "xs" is the start position of the x coordinate of the partial area A1a, "ys" is the start position of the y coordinate of the partial area A1a, "xe" is the end position of the x coordinate of the partial area A1a, and "ye" is the end position of the y coordinate of the partial area A1a. The surface normal information is indicated by (x, y, z, θ, φ, λ), as in FIG. 3 described above.

[0046] Here, the polarized image is a bird's-eye view image of the inspection object A1 from above, while the inspection image is an image captured from various angles. Therefore, the image capture area of ​​the camera 112 (e.g., each partial area A1a in the inspection image) is not rectangular but distorted. Therefore, it is desirable to perform, for example, coordinate transformation to match the coordinates on both the polarized image and the inspection image. The coordinate transformation can be performed based on information such as the sensor information (e.g., optical size and number of pixels) and lens information (e.g., focal length and distortion) of each of the polarization cameras 111 and 112, the distance between the polarization camera 111 and a reference point on the inspection table 13, the elevation angle between the polarization camera 111 and the optical axis, the distance between the camera 112 and a reference point on the inspection table 13, and the elevation angle between the camera 112 and the optical axis. A conversion formula or a conversion table consisting of constants and variables for these parameters is stored, for example, in the control unit 121. The control unit 121 performs the coordinate transformation based on, for example, the conversion formula or the conversion table.

[0047] The camera 112 captures images at a position and orientation specified by the operation program on the trajectory of the arm 114a controlled by the arm control unit 122 according to the operation program. At this time, the illumination unit 113 emits light at a position and orientation specified by the operation program on the trajectory of the arm 114b controlled by the arm control unit 122 according to the operation program. The inspection image obtained by capturing the images is taken into the control unit 121 and transmitted to the storage device 30 together with metadata, which is various information related to the inspection image (e.g., imaging condition information).

[0048] In the example of Fig. 5, various information related to the inspection image includes an image ID (identification), a shooting condition ID, and a timestamp. The image ID is an example of identification information for the inspection image. The shooting condition ID is an example of identification information for the shooting conditions. The timestamp is an example of date and time information indicating the date and time when the inspection image was captured.

[0049] The photographing condition information corresponding to the photographing condition ID includes, for example, area information, camera information, lighting information, and work information. This information is information about the photographing conditions when the partial area A1a was photographed, and is obtained for each partial area A1a. In the example of FIG. 5, area information, camera information, lighting information, and work information are shown for each area ID (e.g., 1, 2, 3, ...) of the partial area A1a. For example, the photographing condition for the partial area A1a, which is the hatched area in the example of FIG. 4, is the photographing condition for which the area ID in the example of FIG. 5 is 6 (see the dotted area), and the photographing conditions are recorded for each partial area A1a.

[0050] As with Figure 4 described above, the area information (xs, ys, xe, ye) is position information indicating the four corners of the captured partial area A1a. The camera information (#, x, y, z, θ, φ, λ) is position information and orientation information of the camera 112 when the partial area A1a was captured. The "#" in the camera information is the camera ID (e.g., a number) of the camera 112. The lighting information (#, x, y, z, θ, φ, λ) is position information and orientation information of the lighting unit 113 when the partial area A1a was captured. The "#" in the lighting information is the lighting ID (e.g., a number) of the lighting unit 113. The work information (#, x, y, z, θ, φ, λ) is surface normal information of the captured partial area A1a. The "#" in the work information is the work ID (e.g., a number) of the inspection object A1.

[0051] That is, the camera information includes identification information (#) of the camera 112, position information (x, y, z) of the camera 112 at the time of capturing an image, and orientation information (θ, φ, λ) of the camera 112 at the time of capturing an image. The lighting information includes identification information (#) of the lighting unit 113, position information (x, y, z) of the lighting unit 113 at the time of emitting light, and orientation information (θ, φ, λ) of the lighting unit 113 at the time of emitting light. The workpiece information includes identification information (#) of the inspection target A1 and surface normal information (x, y, z, θ, φ, λ) of the partial region A1a of the inspection target A1.

[0052] 1-3. Operational Example of Operation Apparatus and Storage Device> An operational example of the operation apparatus 20 and storage device 30 according to this embodiment will be described with reference to Fig. 2 and Fig. 6. Fig. 6 is a diagram showing an example of the configuration of metadata of an image file according to this embodiment.

[0053] As shown in FIG. 2 , the maintenance apparatus 20 can execute a visual inspection application and display an image related to the visual inspection application on the display unit 21. For example, the maintenance apparatus 20 can execute the visual inspection application and display a desired inspection image stored in the storage device 30 on the screen of the display unit 21 together with an image related to the visual inspection application. The image related to the visual inspection application may include a UI (user interface) image. The visual inspection application may be stored locally on the maintenance apparatus 20 or may be stored on a server. If the visual inspection application is stored on a server, the maintenance apparatus 20 may access the visual inspection application stored on the server and display an image related to the visual inspection application via a browser.

[0054] The judge A2 operates the operation unit 22 to select a desired inspection image and performs the inspection while viewing the inspection image displayed on the screen of the display unit 21. This inspection is identified by an inspection item ID, and the inspection target A1 is identified by a workpiece ID. The inspection item ID is an example of identification information for an inspection item. The judge A2 operates the operation unit 22 to input the inspection item ID and the workpiece ID. For example, when the inspection item ID and the workpiece ID are input, an inspection image folder may be formed in the storage device 30 for each combination of the inspection item ID and the workpiece ID, and a series of inspection images may be stored in each inspection image folder.

[0055] For example, the operating device 20 executes an appearance inspection application and sequentially presents all inspection images in a desired inspection image folder to the judge A2 on the display unit 21. The judge A2 determines whether a flaw is detected in the inspection image (NG) or not (OK) and operates the operation unit 22 to register the result in a radio button included in the image related to the appearance inspection application. This registers the judgment result (inspection result). If a flaw is detected, the judge A2 designates the flaw area on the inspection image by clicking or dragging the mouse on the screen.

[0056] When the aforementioned judgment result is registered, the storage device 30 adds the judge's ID (inspector ID), judgment result, designated coordinates (if a scratch is detected), and timestamp to the inspection image as metadata, and records the inspection image including the metadata. The judge's ID is an example of identification information for the judge A2. The judgment result is judgment result information indicating OK / NG. The designated coordinates are designated coordinate information indicating the position of the scratch. The timestamp is date and time information indicating the date and time of the inspection. Note that the designated coordinate information may include, for example, area information indicating the area of ​​the scratch.

[0057] As shown in Figure 6, an image file related to an inspection image is assigned a set J1 of an inspection item ID, a workpiece ID, a camera ID, and an illumination ID. A set J2 of an image ID, an imaging condition ID, and a timestamp is assigned as metadata to each of the multiple inspection images included in the image file. Set J2 is assigned each time a partial region A1a of the inspection target A1 is imaged. In addition, a set J3 of an evaluator ID, an evaluation result, designated coordinates, and a timestamp is assigned to the inspection image for each set J2 as metadata. Each time evaluator A2 evaluates an inspection image corresponding to the image ID of set J2, set J3 is assigned in association with set J2 corresponding to the evaluated inspection image.

[0058] For example, if multiple judges A2 judge the same test image, the set J3 (judgment results) for each judge A2 is assigned in association with the same test image, i.e., the same image ID. Also, if the judge A2 judges the same test image multiple times, the set J3 (judgment results) for each judgment is assigned in association with the same test image, i.e., the same image ID.

[0059] In addition, various data (e.g., inspection item information, work information, camera information, lighting information, image information, shooting condition information, judge information, etc.) corresponding to various IDs (e.g., inspection item ID, work ID, camera ID, lighting ID, image ID, shooting condition ID, judge ID, etc.) are recorded by the storage device 30.

[0060] After the registration of the inspection results is completed, the operating device 20 presents the next inspection image in the desired inspection image folder to the judge A2 in accordance with the appearance inspection application. The judge A2 then performs an OK / NG judgment on all the inspection images in the inspection image folder. When the judge A2 has completed the OK / NG judgment on all the inspection images, the inspection is completed.

[0061] Here, judge A2 can repeatedly judge the same inspection image at intervals, and in this case, new metadata such as the judger ID, the judgment result, the specified coordinates (if a flaw is detected), and the timestamp at the time of completion of the inspection are added to the inspection image.

[0062] Furthermore, the number of judges A2 is not limited to one. Multiple judges A2 can operate the appearance inspection application to access the storage device 30 at different times and re-evaluate the inspection images in the folder of inspected inspection images. For example, judges A2 with different skill levels, such as an expert in appearance inspection, an experienced inspector, an inspector training subject, and a beginner inspector, can separately evaluate the same inspection image. In this case, too, the judger ID, the judgment result, the specified coordinates (if a flaw is detected), and the timestamp at the end of the inspection are newly added to the inspection image as metadata, and the inspection image is recorded together with the metadata in the storage device 30.

[0063] As described above, the inspection image includes metadata J1 (a set of inspection item ID, workpiece ID, camera ID, and lighting ID), J2 (a set of image ID, shooting condition ID, and timestamp (photography date and time information)), and J3 (a set of evaluator ID, judgment result (judgment result information), designated coordinates (designated coordinate information), and timestamp (inspection date and time information). Therefore, the learning device 40 can focus on a specific ID (e.g., image ID, shooting condition ID, evaluator ID, etc.) and extract data corresponding to the ID (e.g., image information, shooting condition information, evaluator information, judgment result information corresponding to the evaluator information, etc.) to generate learning models for various purposes (e.g., learning models for inspections) depending on what is used as input values ​​for the neural network. Various data corresponding to each ID is stored in the storage device 30 and is extracted and used, for example, according to the ID of interest.

[0064] <1-4. Processing Examples of Visual Inspection System> Various processing examples of the visual inspection system 1 according to this embodiment will be described with reference to Figs. 7 to 9. Fig. 7 is a flowchart showing the flow of a photographing condition learning process according to this embodiment. Fig. 8 is a flowchart showing the flow of an expert certification process according to this embodiment. Fig. 9 is a flowchart showing the flow of a judgment skill evaluation process according to this embodiment.

[0065] 7 is a process for estimating effective photographing conditions for visual inspection through machine learning. In this process, an expert judge A2 looks at a plurality of inspection images of the inspection target A1 taken from various angles using a camera 112 and an illumination unit 113 attached to a robot arm 114, inspects each image for scratches, and uses the judged inspection images, for which an OK / NG judgment has been made, as training data to estimate effective photographing conditions for visual inspection through machine learning.

[0066] 7 , in step S11, the assessor A2 operates the operation unit 22 of the operating device 20 to input profile information for the equipment and the workpiece (inspection target A1) (initial setting). In step S12, the processing unit 31 of the storage device 30 stores the input profile information in the storage unit 32.

[0067] In step S13, the robot device 11 uses the polarization camera 111 to capture an overhead image of the workpiece (inspection target A1) on the inspection table 13. In step S14, the control unit 121 of the robot control device 12 divides the captured image into multiple regions (partial regions A1a) and measures the surface normal of each region. In step S12, the processing unit 31 of the storage device 30 stores information such as the captured image (polarized image) and the surface normal in the storage unit 32 in association with the profile information.

[0068] In step S15, the control unit 121 of the robot control device 12 determines a desired photographing area (e.g., a photographing area including multiple partial areas A1a) in response to an input operation (e.g., an input operation specifying a desired photographing area) by the assessor A2 on the operation unit 22 of the working device 20, and converts the determined desired photographing area into coordinates. For example, the control unit 121 converts each of the partial areas A1a into coordinates (xs, ys, xe, ye).

[0069] In step S16, the assessor A2 performs robot teaching (motion planning) by operating the operation unit 22 of the working device 20. In step S17, the robot device 11 performs a photographing operation using the camera 112 and the lighting unit 113.

[0070] In step S18, the control unit 121 of the robot control device 12 adds a captured image ID and metadata to the captured image. In step S12, the processing unit 31 of the storage device 30 stores information such as the captured image (inspection image) and metadata in association with the profile information in the storage unit 32. The metadata includes an image ID, a shooting condition ID, a timestamp, etc.

[0071] In step S19, the control unit 121 of the robot control device 12 determines whether or not the imaging has been completed. If the control unit 121 determines that the imaging has not been completed (No in step S19), the process returns to step S17, and the imaging operation is repeated. If the control unit 121 determines that the imaging has been completed (Yes in step S19), the imaging mode is shifted to the inspection mode in step S20.

[0072] In step S21, the judge A2 operates the operation unit 22 of the operation device 20 to register the judge A2, i.e., the judge ID. At this time, the judge A2 operates the operation unit 22 of the operation device 20 to select an inspection image folder to be inspected from the storage device 30. In step S22, the processing unit 31 of the storage device 30 adds the judge ID to each of the multiple inspection images in the selected inspection image folder. As a result, a judge ID is assigned to each inspection image.

[0073] In step S23, the processing unit 31 of the storage device 30 sequentially sends the plurality of inspection images in the selected inspection image folder to the maintenance device 20. In step S24, the display unit 21 of the maintenance device 20 displays the sent inspection images.

[0074] In step S25, the processing unit 31 of the storage device 30 determines whether the test has been completed in response to an input operation (e.g., an input operation instructing the end of the test) by the assessor A2 on the operation unit 22 of the working device 20. If the processing unit 31 determines that the test has not been completed (No in step S25), the process proceeds to step S26. On the other hand, if the processing unit 31 determines that the test has been completed (Yes in step S25), the process proceeds to step S31.

[0075] In step S26, the processing unit 31 of the storage device 30 determines whether or not there is a scratch in the inspection image in response to an input operation by the judge A2 on the operation unit 22 (for example, an input operation indicating that there is a scratch in the inspection image). If the processing unit 31 determines that there is no scratch in the inspection image (No in step S26), the processing returns to step S23, and in step S27, a scratch-free flag is linked to the image ID of the inspection image. This assigns a determination result (no scratch) to the inspection image.

[0076] The judge A2 looks at the inspection image displayed on the display unit 21 to determine whether or not there is a scratch, and, depending on the determination result, performs input operations on the operation unit 22. For example, if the judge A2 determines that there is no scratch, he / she performs input operations on the operation unit 22 to notify the processing unit 31 that there is no scratch, or if he / she determines that there is a scratch, he / she performs input operations on the operation unit 22 to notify the processing unit 31 that there is a scratch, the position of the scratch, the area of ​​the scratch, etc.

[0077] On the other hand, if the processing unit 31 determines in step S26 that a scratch is present in the inspection image in response to the input operation of the judge A2 on the operation unit 22 (Yes in step S26), in step S28, the processing unit 31 specifies a scratch area on the display screen in response to the input operation of the judge A2 on the operation unit 22 (e.g., an input operation to specify a scratch area), and returns the processing to step S23. In step S29, the processing unit 31 associates the presence of a scratch and scratch coordinate information with the image ID of the inspection image. This assigns a judgment result (e.g., judgment information on the presence or absence of a scratch, and specified coordinate information of the scratch) to the inspection image. In step S30, the processing unit 31 stores information such as a learning image (inspected inspection image) including judgment result information in the memory unit 32, in association with profile information.

[0078] In step S31, the processing unit 41 of the learning device 40 performs ID filtering on the learning images. ID filtering involves using the judge's ID to select learning images (inspected test images) for which the judge A2 is an expert (expert master). Because it is possible to determine whether the judge A2 is an expert from the judge's ID, the judge's ID can be used to select learning images for which the judge A2 is an expert.

[0079] In step S32, the processing unit 41 of the learning device 40 performs supervised machine learning using the selected learning images to generate a learning model such as an appearance inspection module. In step S33, the processing unit 31 of the storage device 30 stores the learning model such as the appearance inspection module in the memory unit 32. The appearance inspection module may be used for robot teaching in the above-mentioned step S16.

[0080] 8 is a process for certifying experts who can obtain highly reliable training data when learning effective photographing conditions. In this process, samples (limit samples) with different levels of difficulty in visually inspecting the actual objects are photographed intensively to prepare a group of training images, and the training images are evaluated using a visual inspection application. Those whose OK / NG evaluation results meet the certification criteria are registered as experts.

[0081] 8, in step S51, a person who certifies an expert or the like selects a limit sample (an object to be inspected A1). This limit sample is placed on the inspection table 13.

[0082] In step S52, the robot device 11 uses the polarization camera 111 to capture an overhead image of the workpiece (limited sample = inspection target A1) on the inspection table 13. In step S53, the control unit 121 of the robot control device 12 divides the captured image into multiple regions (partial regions A1a) and measures the surface normal of each region. In step S54, the processing unit 31 of the storage device 30 stores information such as the captured image (polarized image) and the surface normal in the storage unit 32.

[0083] In step S55, the control unit 121 of the robot control device 12 determines a desired photographing area (for example, a photographing area including multiple partial areas A1a) in response to an input operation by the assessor A2 on the operation unit 22 of the working device 20, and converts the determined desired photographing area into coordinates. For example, the control unit 121 converts each of the partial areas A1a into coordinates (xs, ys, xe, ye).

[0084] In step S56, the assessor A2 performs robot teaching (motion planning) by operating the operation unit 22 of the working device 20. In step S57, the robot device 11 performs a photographing operation using the camera 112 and the lighting unit 113.

[0085] In step S58, the control unit 121 of the robot control device 12 adds a captured image ID and metadata to the captured image. In step S54, the processing unit 31 of the storage device 30 stores information such as the captured image (inspection image) and metadata in the storage unit 32. The metadata includes an image ID, a shooting condition ID, a timestamp, etc.

[0086] In step S59, the control unit 121 of the robot control device 12 determines whether or not the imaging has been completed. If the control unit 121 determines that the imaging has not been completed (No in step S59), the process returns to step S57, and the imaging operation is repeated. If the control unit 121 determines that the imaging has been completed (Yes in step S59), the imaging mode is shifted to the inspection mode in step S60.

[0087] In step S61, judge A2 operates the operation unit 22 of the operation device 20 to register judge A2, i.e., an judge ID. In step S62, the processing unit 31 of the storage device 30 selects an inspection skill assessment image set from the storage unit 32 in response to an input operation by judge A2 on the operation unit 22 of the operation device 20 (for example, an input operation specifying an inspection skill assessment image set). In step S63, the processing unit 31 adds an judge ID to each of the multiple inspection images in the selected inspection skill assessment image set. This assigns an judge ID to each inspection image.

[0088] In step S64, the processing unit 31 of the storage device 30 sequentially sends the plurality of inspection images in the selected inspection skill determination image set to the operation device 20. In step S65, the display unit 21 of the operation device 20 displays the sent inspection images.

[0089] In step S66, the processing unit 31 of the storage device 30 determines whether the examination has been completed in response to the input operation of the assessor A2 on the operation unit 22 of the working device 20. If the processing unit 31 determines that the examination has not been completed (No in step S66), the process proceeds to step S67, whereas if the processing unit 31 determines that the examination has been completed (Yes in step S66), the process proceeds to step S72.

[0090] In step S67, the processing unit 31 of the storage device 30 determines whether or not there is a scratch in the inspection image in response to an input operation by the judge A2 on the operation unit 22 of the working device 20. If the processing unit 31 determines that there is no scratch in the inspection image (No in step S67), the processing returns to step S64, and in step S68, a scratch-free flag is linked to the image ID of the inspection image. This assigns a determination result (no scratch) to the inspection image.

[0091] On the other hand, if the processing unit 31 determines in step S67 that there is a scratch in the inspection image (Yes in step S67), then in step S69, the processing unit 31 specifies the scratch area on the display screen in response to an input operation by the judge A2 on the operation unit 22, and returns the process to step S64. Also, in step S70, the presence of a scratch and information about the scratch's coordinates are linked to the image ID of the inspection image. This assigns the inspection result (e.g., information about the presence or absence of a scratch, and information about the designated coordinates of the scratch) to the inspection image. In step S71, the processing unit 31 stores in the memory unit 32 a group of inspection skill judgment images, which are multiple inspection images (inspected inspection images) including the judgment result information.

[0092] In step S72, the processing unit 31 of the storage device 30 confirms whether or not the judgment match is true. In confirming whether or not the judgment match is true, the processing unit 31 confirms, for example, the degree of match between the judgment result of the expert (expert) on the limited samples (each of the test images included in the aforementioned group of images for testing skill judgment) and the judgment result of the evaluator A2 on the limited samples. For example, the unit of the degree of match is %, and a degree of match of 100% indicates that all questions are correct.

[0093] In step S73, the processing unit 31 of the storage device 30 determines whether all the questions (all the judgments) are correct. If the processing unit 31 determines that all the questions are not correct (No in step S73), the processing returns to step S64 and the inspection is repeated. On the other hand, if the processing unit 31 determines that all the questions are correct (Yes in step S73), the processing unit 31 certifies the assessor A2 as an expert in step S74. In step S75, the processing unit 31 registers the assessor A2, who has been certified as an expert, in the storage unit 32 as an expert.

[0094] 9 is a process for evaluating and registering the judgment skill (skill level) of the judge A2. In this process, the accuracy of the OK / NG judgment results obtained by using a visual inspection application to judge a group of inspection images, to which the judgment results of an expert are attached as metadata, is determined as training images, and the accuracy of the results is determined to be in agreement with the judgment results of an expert master. The time required for the judgment and the continuity measured as the inspection is continued are also comprehensively scored and registered as evaluation indices.

[0095] 9 , in step S101, judge A2 operates the operation unit 22 of the operation device 20 to register judge A2, i.e., an judge ID. In step S102, the processing unit 31 of the storage device 30 selects an inspection skill assessment image set from the inspection skill assessment image group stored in the storage unit 32 in response to an input operation by judge A2 on the operation unit 22 of the operation device 20. In step S103, the processing unit 31 assigns an judge ID to each of the multiple inspection images in the selected inspection skill assessment image set. This assigns an judge ID to each inspection image.

[0096] In step S104, the processing unit 31 of the storage device 30 sequentially sends the images in the selected inspection skill determination image set to the operation device 20. In step S105, the display unit 21 of the operation device 20 displays the sent inspection images.

[0097] In step S106, the processing unit 31 of the storage device 30 determines whether the examination has been completed in response to an input operation by the assessor A2 on the operation unit 22 of the working device 20. If the processing unit 31 determines that the examination has not been completed (No in step S106), the process proceeds to step S107, whereas if the processing unit 31 determines that the examination has been completed (Yes in step S106), the process proceeds to step S112.

[0098] In step S107, the processing unit 31 of the storage device 30 determines whether or not there is a scratch in the inspection image in response to an input operation by the judge A2 via the operation unit 22 of the working device 20. If the processing unit 31 determines that there is no scratch in the inspection image (No in step S107), the processing returns to step S104, and in step S108, a scratch-free flag is linked to the image ID. This assigns a determination result (no scratch) to the inspection image.

[0099] On the other hand, if the processing unit 31 determines in step S107 that there is a scratch in the inspection image (Yes in step S107), then in step S109, the processing unit 31 specifies the scratch area on the display screen in response to an input operation by the judge A2 via the operation unit 22, and returns the process to step S104. In step S110, the processing unit 31 associates the presence of a scratch and scratch coordinate information with the image ID. This assigns the inspection image a judgment result (e.g., judgment information on the presence or absence of a scratch, and designated coordinate information of the scratch). In step S111, the processing unit 31 records, in the storage unit 32, a group of inspection skill judgment images, which are multiple inspection images (inspected inspection images) including judgment result information.

[0100] In step S112, the processing unit 31 of the storage device 30 checks the degree of agreement with the expert judgment. In checking the degree of agreement with the expert judgment, the processing unit 31, for example, determines the degree of agreement between the judgment result of the judge A2 for each of the test images included in the aforementioned group of images for assessment of inspection skills and the judgment result of the expert for each of the test images included in the aforementioned group of images for assessment of inspection skills. For example, the unit of the degree of agreement is %.

[0101] In step S113, the processing unit 31 of the storage device 30 determines whether the skill evaluation has been completed in response to an input operation by the assessor A2 on the operation unit 22 (for example, an input operation instructing the end of the skill evaluation). If the processing unit 31 determines that the skill evaluation has not been completed (No in step S113), the processing returns to step S104 and the inspection is repeated. If the processing unit 31 determines that the skill evaluation has been completed (Yes in step S113), the processing unit 31 determines the assessor skill in step S114. In step S115, the processing unit 31 registers the assessor A2, whose skill (skill level) has been determined, in the storage unit 32 together with the skill level.

[0102] In the judge's skill assessment, the processing unit 31 determines the skill level of the judge A2 based on, for example, the degree of agreement between the judge's assessment results and those of the expert for each inspection image included in the aforementioned inspection skill assessment image group. For example, the processing unit 31 determines the skill level of the judge A2 to be higher the higher the degree of agreement. For example, if the skill level is divided into five levels, when the degree of agreement is 0 to 20%, the skill level is 1; when the degree of agreement is 20 to 40%, the skill level is 2; when the degree of agreement is 40 to 60%, the skill level is 3; when the degree of agreement is 60 to 80%, the skill level is 4; and when the degree of agreement is 80 to 100%, the skill level is 5. These level divisions are merely examples.

[0103] <1-5. Metadata Analysis Example> An example of metadata analysis according to this embodiment will be described with reference to FIGS. 10 to 15. FIG. 10 is a diagram for explaining an example of metadata analysis according to this embodiment. FIG. 11 is a diagram for explaining an example of a proficiency index according to this embodiment. FIG. 12 is a diagram for explaining an example of accuracy and stability index according to this embodiment. FIG. 13 is a diagram for explaining an example of a variance index according to this embodiment. FIG. 14 is a diagram for explaining an example of measurement of durability and quick decision-making ability according to this embodiment. FIG. 15 is a diagram for explaining an example of measurement of sustainability according to this embodiment.

[0104] (A) Training Data As shown in FIG. 10, the results of judgments made by four judges (four judge IDs #1 to #4) on four image IDs #1 to #4, with OK / NG judgments, are shown. In the example of FIG. 10, judge ID #1 is an expert, judge ID #2 is Mr. A, judge ID #3 is Mr. B, and judge ID #4 is Mr. C. Each of the four image IDs #1 to #4 is an inspection image. Metadata related to this inspection image includes, for example, an inspection item ID, a workpiece ID, a camera ID, an illumination ID, and a photographing condition ID. The inspection item ID is #1 (scratch), the workpiece ID is #1 (sample No.), the camera ID is #1 (model name + lens), the illumination ID is #1 (model name), and the photographing condition ID is #1 (position / posture between devices).

[0105] In such a judgment result, a group of inspection images extracted with a focus on the judge's ID, for example, a group of inspection images extracted with a focus on the expert judge's ID #1, can be used as highly reliable training data to judge the effectiveness of the shooting conditions. For example, assuming that an inspection image judged by an expert (judge's ID #1) as having a flaw detected (NG) was captured under appropriate shooting conditions in terms of being able to detect flaws, a learning model of effective shooting conditions can be generated by machine learning the group of images combined with other inspection images that were photographed in the same area but were judged as having no flaw detected (OK) by a non-expert judge A2, using the inspection image judged as having a flaw detected by the expert as training data.

[0106] (B) Difficulty of Judgment As shown in FIG. 10 , for example, a group of test images extracted with a focus on image IDs is effective for measuring the skill level of judge A2. Image ID #2 was judged as NG by all four judges IDs #1 to #4, making it teacher data with a low judgment difficulty. On the other hand, image ID #4 was judged as NG only by the skilled judge ID #1, but was judged as OK by the other three judges IDs #2 to #4. In other words, the other three judges IDs #2 to #4 were unable to detect it as NG and judged it as OK. Therefore, image ID #4 is teacher data with a high judgment difficulty.

[0107] For example, in the example of Figure 10, for four test images, the judgment results of the expert judge ID #1 are used as the standard, and if there are three non-expert judges A2, the judgment results are tallied for each image ID, with the same judgment result (OK → OK / NG → NG) = 0, the underkill judgment result (NG → OK) = -1, and the overkill judgment result (OK → NG) = +1.

[0108] As a result, the four test images can be classified as follows: -3: Image with a strong tendency toward underkill and high difficulty in judgment -2: ... -1: ... 0: Image with low difficulty in judgment +1: ... +2: ... +3: Image with a strong tendency toward overkill and high difficulty in judgment.

[0109] That is, image ID #1 has a value of 0 + 0 + (-1) = -1, image ID #2 has a value of 0 + 0 + 0 = 0, image ID #3 has a value of 0 + 0 + (-1) + (-1) = -2, and image ID #4 has a value of (-1) + (-1) + (-1) = -3. Images classified from such a perspective can be effectively used for training judge A2.

[0110] (C) Evaluation Index For example, for four test images, the judgment results of an expert judge ID #1 are used as the standard, and if there are three non-expert judges A2, the judgment results are tallied for each judge ID, with the same judgment result (OK → OK / NG → NG) = 0, the underkill judgment result (NG → OK) = -1, and the overkill judgment result (OK → NG) = +1. In other words, by accumulating the judgment results for each judge ID rather than for each image ID, and performing statistical analysis, an evaluation index for the skill of judge A2 can be obtained.

[0111] As a result, the three judges A2 can be classified as follows: -4: Judges who tend to judge underkill -3: ... -2: ... -1: ... 0: Judges who are certain of their judgments +1: ... +2: ... +3: ... +4: Judges who tend to judge overkill.

[0112] That is, for evaluator ID #1, 0 + 0 + 0 + (-1) = -1, for evaluator ID #2, 0 + 0 + (-1) + (-1) = -2, and for evaluator ID #3, (-1) + 0 + (-1) + (-1) = -3. Evaluation indices classified from such perspectives can be effectively used to certify the skill level of evaluator A2.

[0113] In such cases, it is desirable to add weighting that takes into account the judgment difficulty of each image. For example, it is not appropriate to treat a case in which an image with a judgment difficulty of -3 is judged as -1 and a case in which an image with a judgment difficulty of -1 is judged as -1 equivalently. When the judgment result has the same sign as the judgment difficulty, it is desirable to multiply the judgment result by the absolute value of the judgment difficulty and a coefficient of the inverse slope (for example, x1 if the judgment difficulty is -3, and x3 if the judgment difficulty is -1). Furthermore, when the judgment result and the judgment difficulty have opposite signs, it is desirable to multiply the judgment result by the absolute value of the judgment difficulty and a larger coefficient of the quasi-slope (for example, x4 to x6).

[0114] (D) Accuracy, proficiency, stability / variability If assessor A2 performs multiple visual inspections of the same image folder at different times, the following analysis can be performed in relation to assessor A2's skills from the group of inspection images extracted based on the assessor ID.

[0115] Accuracy (certainty) is expressed as the probability of a count of 0. Probability of a count of 0 = (count of 0) / (total number of tests)

[0116] In the example of Figure 10, the reliability of assessor ID #2 is 3 / 4 the first time, 4 / 4 the second time, and 4 / 4 the third time. The reliability of assessor ID #3 is 2 / 4 the first time, 2 / 4 the second time, and 2 / 4 the third time. The reliability of assessor ID #4 is 1 / 4 the first time, 3 / 4 the second time, and 2 / 4 the third time.

[0117] The level of proficiency (speed of proficiency) is shown by the change in the count value of 0 over time. The count value of 0 is considered to be the score (count value of 0 = score). Change in count value of 0 over time = (latest score - previous score) / (100 - (latest score + previous score) / 2)

[0118] It should be noted that if a count value of 0 is considered to be the score, the higher the score, the more weighted the count value becomes. Also, the higher the rate of change in the count value, the higher the level of proficiency (the faster the speed of proficiency).

[0119] 11, when judgments are made on 100 inspection images, the target measurement value (score) of 0 is 100. From graph B1, the change in the count value of 0 over time is (70-40) / (100-55)=0.67 for the second time, and (85-70) / (100-77.5)=0.67 for the third time.

[0120] The stability is expressed by the standard deviation of the change in the count value of 0 over time. This standard deviation is calculated by the following formula (1).

[0121] In the example of Fig. 12, similar to Fig. 11 described above, when judgments are made on 100 inspection images, the target measurement value (score) of 0 becomes 100. The standard deviation corresponding to graph B2a is 10.4, and the standard deviation corresponding to graph B2b is 12.6. Note that graphs B2a and B2b each represent the results of a different judge A2.

[0122] Variation (unevenness) is indicated by the change over time in the frequency of occurrence of -1 / +1 (for example, amplitude tendency, bias tendency). Amplitude tendency = (count value of +1) weighted in (C) above + (count value of -1) Bias tendency = (count value of +1) weighted in (C) above - (count value of -1)

[0123] In the example of FIG. 13 , in graph B3a, the frequency of +1 is 1 the first time, 1 the second time, and 3 the third time. In graph B3b, the frequency of -1 is 3 the first time, 1 the second time, and 1 the third time. The amplitude trend is 1 + 3 = 4 the first time, 1 + 1 = 2 the second time, and 3 + 4 the third time. Furthermore, the bias trend is 1 - 3 = -2 the first time, 1 - 1 = 0 the second time, and 3 - 2 = 2 the third time. Therefore, the amplitude trend (change) is 4 → 2 → 4, and the bias trend (change) is -2 → 0 → 2. Note that graphs B3a and B3b each represent the results of the same evaluator A2.

[0124] (E) Durability, Decisiveness, and Persistence A timestamp indicating the date and time (e.g., time) when the judgment was made is recorded in the test image. By analyzing the timestamp and the judgment result over time, the following analysis related to the skills of the evaluator A2 is possible.

[0125] Durability (duration) is indicated by the duration (continuity) of the test, which is the difference between the timestamps of the start of the test (t1) and the end of the test (tn), as shown in FIG. 14. Duration = tn - t1

[0126] Decision making ability (time required) is shown in Figure 14 (see graph B4) as the time from the end of inspection of the previous image to the end of inspection of the next image. The average value and standard deviation of this time can be calculated using the following equations (2) and (3).

[0127] Continuity (concentration) is indicated by the correlation (change over time) between the timestamp (t) of the image and the accuracy (%), as shown in Figure 15 (see graph B5). This correlation is calculated using the following equation (4):

[0128] (F) Training Menu It is possible to design an individual training menu for judge A2 according to the skill level of judge A2 using a group of images extracted with a focus on the image IDs described above.

[0129] For example, for judge A2 whose judgment results as defined in (D) above tend to lean towards the positive side (tend to overkill), it may be possible to adjust the images (+1 to +3) that tend to overkill in (B) above by focusing on +3 or mixing +1 to +3 evenly, depending on the level of the worker.

[0130] <1-6. Example of Systematization of Skill Levels> An example of systematization of skill levels according to this embodiment will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining an example of systematization of the skill level of assessor A2. In the example of Fig. 16, indexing of judgment skills is shown as the skill level of assessor A2, and a spider chart (radar chart) C1 is shown as a specific example of evaluation axes and systematization.

[0131] As shown in FIG. 16 , the judgment skills of assessor A2 are shown by a spider chart C1. The judgment skills include accuracy (certainty), persistence (concentration), stability, proficiency, endurance (duration), and quick decision-making (time required). These judgment skills are obtained as shown in (D) and (E) above. This spider chart C1 is displayed, for example, on the display unit 21 of the working device 20. This allows assessor A2 to understand his or her own judgment skills.

[0132] The spider chart C1 may be displayed on another display device as needed. For example, other experts or managers may check the judgment skill of the assessor A2 by looking at the spider chart C1 displayed on the display device. The judgment skill of each assessor A2 may be recorded in the storage device 30 or another storage device. Furthermore, the judgment skill of the assessor A2 may be displayed not only by the spider chart C1 but also by a graph such as a pie chart or a bar graph, or by another diagram.

[0133] <1-7. Modified Example of Visual Inspection System> A configuration example of a visual inspection system 1 according to a modified example of the present embodiment will be described with reference to Fig. 17. Fig. 17 is a diagram showing a configuration example of a visual inspection system 1 according to a modified example of the present embodiment. The visual inspection system 1 according to the modified example of the present embodiment is an example of an information processing system.

[0134] The judgment skill according to this embodiment is not a specialized skill specialized for visual inspection, but can be generalized as a skill for visually checking an image and extracting and judging a context according to the intended use, and can be applied to other applications. In a modification according to this embodiment, as a specific example, an application example to a remote operation task of a harvesting robot deployed in a farm that harvests vegetables such as tomatoes is shown. The visual inspection system 1 functions as an agricultural harvesting system.

[0135] As shown in FIG. 17, the appearance inspection system 1 according to the modified example includes a robot device 60 in addition to the above-described working device 20, storage device 30, and learning device 40.

[0136] The robot device 60 has a body 61, wheels 62, a camera 63, and a gripping unit 64. The robot device 60 functions as a harvesting robot. Note that the robot device 60 may have a lighting unit 113, as in the above-described embodiment.

[0137] The machine body 61 is the main body of the robot device 60. The wheels 62 are rotatably mounted on the underside of the machine body 61, allowing the machine body 61 to move. The camera 63 corresponds to the camera 112 described above. This camera 63 captures an image of a desired area including, for example, the harvest object A3. The harvest object A3 may be, for example, a vegetable such as a tomato or a cucumber, or may be a fruit other than a vegetable. The harvest object A3 corresponds to the inspection object A1 described above. The camera 63 functions as a first imaging unit.

[0138] The gripping unit 64 has a hand 641 and a robot arm 642. The hand 641 is attached to the tip of the robot arm 642 and is configured to be able to grip the harvest target A3. The robot arm 642 is configured to be able to move the hand 641 located at its tip to a desired position.

[0139] The aforementioned machine body 61 incorporates a control device 611, an image sensor 612, a unit drive unit 613, and a machine body drive unit 614. The control device 611 corresponds to the aforementioned robot control device 12. For example, the control device 611 controls the unit drive unit 613, the machine body drive unit 614, etc. The image sensor 612 is part of the camera 63, and is, for example, an image sensor. The unit drive unit 613 drives the grip unit 64. The machine body drive unit 614 drives the machine body 61, i.e., the wheels 62 provided on the machine body 61.

[0140] The working device 20 includes a computing device 23 in addition to the display unit 21 and operation unit 22 described above. The display unit 21 and operation unit 22 are connected to the operation unit 23. The assessor A2 can, for example, operate the operation unit 22 as needed to remotely control the robot device 60. For example, the assessor A2 can operate the operation unit 22 to change the imaging conditions, such as the zoom of the camera 63, or to operate the gripping unit 64 (remote control).

[0141] In this visual inspection system 1, the robot device 60 travels, for example, between the rows and stops in front of each plant. The camera 63 takes, for example, a photograph of the entire plant. The captured image is transmitted to the computing device 23 via the network 50 and displayed on the display unit 21. In this way, the inspection image is provided to the assessor A2, who is an operator.

[0142] The assessor A2 identifies a mature harvest target A3 (e.g., a tomato) to be harvested from the inspection image displayed on the display unit 21, and uses the operation unit 22 to specify the mature harvest target A3 on the image and issue a harvest instruction to the robot device 60. The harvest instruction is transmitted to the control device 611 of the robot device 60 via the network 50. The control device 611 controls the unit drive unit 613 to drive the gripping unit 64, i.e., the robot arm 642 and hand 641, to harvest the harvest target A3.

[0143] If there are other mature harvest targets A3 in the same inspection image, the assessor A2 repeats the harvest instruction using the operation unit 22. When there are no more harvest targets A3 to be harvested, the assessor A2 inputs an end instruction using the operation unit 22. The end instruction is also transmitted to the control device 611 of the robot device 60 via the network 50. The control device 611 controls the machine body drive unit 614 to drive the wheels 62 and move the robot device 60 to the next plant.

[0144] The captured inspection image is assigned, for example, a unique ID, a timestamp of the date and time of the image capture, and if the assessor A2 determines that the fruit is ripe, the coordinate information of the harvest target A3 for which harvesting is instructed, a timestamp of the harvest date and time (inspection date and time), and the assessor ID are assigned as metadata. For example, the inspection image with the metadata assigned is recorded in the storage device 30. The learning device 40 reads and analyzes these inspection images and records the learning results (learning model) in the storage device 30.

[0145] In this workflow, the speed at which evaluator A2 makes decisions affects the overall work time, and his or her stamina affects the amount of work (harvest yield). Accuracy and persistence affect the quality of the harvested product. In other words, if evaluator A2 scores highly on these skill evaluation indices in visual inspection, it is possible to recommend the agricultural harvesting robot remote operation task.

[0146] Here, inspection standards are usually necessary to verify the effectiveness of the imaging conditions, and to establish such inspection standards, for example, a considerable number of defect samples is required. Furthermore, a process is required to efficiently create the inspection standards using the knowledge of inspectors (experts) who have empirically accurate judgment standards. Furthermore, as the number of experienced inspectors is on the decline, it is important to effectively pass on the judgment standards for sensory inspections to train and expand the number of inspectors who can engage in such inspections. Furthermore, parameters that define the imaging conditions include not only the individual positions and orientations of the camera 112 and the illumination unit 113, but also, for example, the surface shape of the object A1 to be inspected.

[0147] According to the visual inspection system 1 of the above-described embodiment or modification, the following metadata are attached to the inspection image: a set J1 of an inspection item ID (inspection item information), a workpiece ID (workpiece information), a camera ID (camera information), and an illumination ID (illumination information); a set J2 of an image ID (image information), a shooting condition ID (shooting condition information), and a timestamp (shooting date and time information); and a set J3 of an evaluator ID (evaluator information), a judgment result (judgment result information), designated coordinates (designated coordinate information), and a timestamp (judgment date and time information) (see FIG. 6 ). That is, the inspection image is recorded together with the metadata by the storage device 30. The recording of the inspection image and metadata is performed for each shooting timing, i.e., for each inspection image. This allows information including, for example, shooting condition information, identification information of the evaluator A2, and judgment result information to be stored for each inspection image. Therefore, appropriate information for creating the inspection standard can be efficiently obtained, thereby enabling the inspection standard to be efficiently created.

[0148] In addition, the imaging condition information for the imaging condition ID also includes work information, i.e., surface normal information (see Figure 5), so not only the individual position and posture information of the camera 112 and the lighting unit 113 but also the surface shape of the object to be inspected A1 can be obtained.

[0149] Furthermore, the photographing of the inspection object A1 by the robot device 11 and the image inspection by the assessor A2 can be separated in time, allowing the assessor A2 to perform the inspection work remotely regardless of location. Furthermore, by learning the relationship between the assessment result information by the skilled assessor A2 and the metadata, the effectiveness of the imaging conditions can be evaluated based on highly reliable training data. Furthermore, by having an inexperienced assessor A2 make assessments on the same group of inspection images as the skilled assessor A2 and evaluating the difference between the assessment results of the assessor A2 and the skilled assessor A2, it is possible to specifically recognize discrepancies in assessment skills.

[0150] Furthermore, by indexing the judgment skills in accordance with specific rules, it is possible to objectively recognize the competence of the judge A2, manage the index together with the judge ID of the judge A2, and train the judge A2. Furthermore, by allowing an inexperienced judge A2 and an experienced judge A2 to simultaneously access the same inspection image from different locations via the network 50 and share each other's judgment results in real time, a more effective training tool can be provided. For example, if the judgment skills of the inexperienced judge A2 improve under the guidance of the experienced judge A2, the results can be indexed as the training and instruction skills of the experienced judge A2.

[0151] Furthermore, if the overall shape information or texture information of the object to be inspected A1 is confidential information, even during remote assessment work, the information can be kept confidential by changing the order of the partial images presented to the assessor A2 or by providing only the information necessary for the inspection purpose from the information of the polarization camera 111 (e.g., degree of polarization, polarization direction, reflection components, etc.).

[0152] <1-8. Actions and Effects> As described above, the visual inspection system 1 according to this embodiment includes a first imaging unit (e.g., camera 112 or camera 63) that photographs the inspection object A1 to obtain an image, and a processing unit 31 that generates information including the image (e.g., an inspection image) obtained by the first imaging unit, information about the photographing conditions when the inspection object A1 was photographed (e.g., position information and posture information of camera 112 or camera 63), identification information of the inspector who inspected the inspection object A1 based on the image (e.g., the judge ID of judge A2), and inspection result information of the inspection object A1 (e.g., judgment result information on the presence or absence of scratches). This generates information including the photographing condition information, the inspector's identification information, and the inspection result information for each image. Therefore, it is possible to obtain appropriate information for creating inspection criteria, thereby enabling the inspection criteria to be created efficiently.

[0153] The processing unit 31 may also add information about the imaging conditions to the image, and then add the examiner's identification information and the inspection result information to the image to which the imaging condition information has been added. This makes it possible to reliably obtain appropriate information for creating inspection standards.

[0154] The visual inspection system 1 may further include a second imaging unit (e.g., a polarization camera 111) that captures an image of the inspection target A1 to obtain surface normal information, and the imaging condition information may include the surface normal information, thereby obtaining the surface shape of the inspection target A1.

[0155] The first imaging unit may capture an image of a partial region A1a of the inspection target A1, and the imaging condition information may include position information of the partial region A1a and surface normal information of the partial region A1a, thereby obtaining the position information of the partial region A1a of the inspection target A1 as well as the surface shape of the partial region A1a.

[0156] The imaging condition information may also include position information and orientation information of the first imaging unit when imaging the inspection target A1. This generates various types of information, thereby providing appropriate information for creating inspection standards.

[0157] The visual inspection system 1 may further include an illumination unit 113 that irradiates the inspection target A1 with light, and the photographing condition information may include position information and posture information of the illumination unit 113 when photographing the inspection target A1. This generates such information, thereby obtaining appropriate information for creating an inspection standard.

[0158] The information may also include information on the imaging conditions, the identification information of the inspector, and the inspection result information for each of the multiple images. This allows the generation of such information, thereby obtaining appropriate information for creating inspection standards.

[0159] The information may also include the identification information and test result information of each of the multiple testers, thereby generating the information appropriate for creating the test standard.

[0160] The inspection result information may also include information on the presence or absence of scratches. This generates the information on the presence or absence of scratches, so that appropriate information for creating an inspection standard can be obtained.

[0161] The inspection result information may also include designated coordinate information indicating the location of the flaw. This generates the designated coordinate information, making it possible to obtain appropriate information for creating an inspection standard.

[0162] Furthermore, the processing unit 31 may compare the inspector's inspection result information for the image with the expert's inspection result information for the image, and certify that the inspector is an expert based on the comparison result. This allows for the implementation of an expert certification process for certifying an expert.

[0163] The processing unit 31 may also determine the skill level of the inspector using the inspection result information of the inspector for the image, thereby realizing a skill evaluation process for evaluating the skill level.

[0164] Furthermore, the processing unit 31 may determine the skill level of the inspector using information on the inspection result of an expert on the image in addition to information on the inspection result of the inspector on the image, thereby enabling accurate evaluation of the skill level.

[0165] The processing unit 31 may also determine the skill level based on one or more of the inspector's proficiency in the inspection, accuracy, stability, variability, durability, and quick decision-making ability, thereby enabling accurate evaluation of the skill level.

[0166] Furthermore, the processing unit 31 may systematize the skill levels based on any or all of proficiency, accuracy, stability, variability, durability, and quick decision-making ability, thereby enabling the skill levels to be accurately systematized.

[0167] The visual inspection system 1 may further include a learning device 40 that performs machine learning using a plurality of pieces of information described above. This allows machine learning based on the various pieces of information described above.

[0168] Furthermore, the learning device 40 may generate a learning model for testing using, as training information, information indicating that the inspector is an expert, among the multiple pieces of information described above. This allows a learning model for testing based on information indicating that the inspector is an expert to be obtained.

[0169] The learning device 40 may also generate a learning model for inspection using multiple images, imaging condition information for each image, inspector identification information for each image, and / or inspection result information, thereby obtaining a learning model for inspection based on various types of information.

[0170] 2. Other Embodiments The configurations and processes according to the above-described embodiments (examples and modifications) may be implemented in various different forms other than the above-described embodiments. For example, the configurations and processes are not limited to the above-described examples and may be implemented in various forms. Furthermore, for example, the configurations, processing procedures, specific names, or information including various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified.

[0171] Furthermore, the components and processes according to the above-described embodiments (examples and modifications) do not necessarily have to be physically configured as shown in the drawings. In other words, the specific form of distribution and integration of the components and processes is not limited to that shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0172] Furthermore, the configurations and processes of the above-described embodiments (examples and modifications) may be combined as appropriate. For example, at least a part of an embodiment may be combined as appropriate with at least a part of another embodiment. Furthermore, the effects of the embodiments are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0173] Furthermore, in the above-described embodiments (examples and variations), a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network 50, and a single device housed in a single housing and containing multiple modules, are both systems.

[0174] Furthermore, in the above-described embodiments (examples and variations), a cloud computing configuration can be adopted in which one function is shared and processed jointly by multiple devices via a network 50. Furthermore, each step described in the above-described process flow (e.g., a flowchart) can be executed by one device or shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or shared and executed by multiple devices.

[0175] 3. Example of Hardware Configuration The information processing devices such as the robot control device 12, the working device 20, the storage device 30, the learning device 40, and the control device 611 described above may be realized by, for example, a computer 1000 configured as shown in Fig. 18. Fig. 18 is a diagram showing an example of the hardware configuration of the computer 1000.

[0176] 18 , the computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0177] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0178] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0179] The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records program data 1450.

[0180] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0181] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0182] For example, when the computer 1000 functions as an information processing device (e.g., the robot control device 12, the working device 20, the storage device 30, the learning device 40, and the control device 611), the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of each unit of the information processing device. The information processing program and various data are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0183] <4. Supplementary Note> The present technology may also be configured as follows. (1) An information processing system comprising: a first imaging unit that photographs an inspection object to obtain an image; and a processing unit that generates information including an image obtained by the first imaging unit, information about imaging conditions when the inspection object was photographed, identification information of an inspector who inspected the inspection object based on the image, and inspection result information of the inspection object. (2) The information processing system described in (1), wherein the processing unit adds the imaging condition information to the image, and adds the identification information of the inspector and the inspection result information to the image with the imaging condition information added. (3) The information processing system described in (1) or (2), further comprising: a second imaging unit that photographs the inspection object to obtain surface normal information, wherein the imaging condition information includes the surface normal information. (4) The information processing system described in (3), wherein the first imaging unit photographs a partial region of the inspection object, and the imaging condition information includes position information of the partial region and the surface normal information of the partial region. (5) The information processing system according to any one of (1) to (4), wherein the imaging condition information includes position information and orientation information of the first imaging unit when the inspection object is imaged. (6) The information processing system according to any one of (1) to (5), further comprising an illumination unit that irradiates the inspection object with light, wherein the imaging condition information includes position information and orientation information of the illumination unit when the inspection object is imaged. (7) The information processing system according to any one of (1) to (6), wherein the information includes the imaging condition information, the identification information of the inspector, and the inspection result information for each of the multiple images. (8) The information processing system according to any one of (1) to (7), wherein the information includes the identification information of the inspector and the inspection result information for each of the multiple inspectors. (9) The information processing system according to any one of (1) to (8), wherein the inspection result information includes determination information on the presence or absence of a scratch. (10) The information processing system according to (9), wherein the inspection result information includes specified coordinate information indicating the position of the scratch.(11) The information processing system according to any one of (1) to (10), wherein the processing unit compares the inspection result information of the inspector for the image with the inspection result information of an expert for the image and certifies that the inspector is an expert based on the comparison result. (12) The information processing system according to any one of (1) to (11), wherein the processing unit determines a skill level of the inspector using the inspection result information of the inspector for the image. (13) The information processing system according to (12), wherein the processing unit determines a skill level of the inspector using the inspection result information of the expert for the image in addition to the inspection result information of the inspector for the image. (14) The information processing system according to (12) or (13), wherein the processing unit determines the skill level based on one or more of the inspector's inspection proficiency, accuracy, stability, variability, durability, and decision-making ability. (15) The information processing system according to (14), wherein the processing unit systematizes the skill level based on any one or all of the proficiency, the accuracy, the stability, the variance, the durability, and the ability to make quick decisions. (16) The information processing system according to any one of (1) to (15), further comprising a learning device that performs machine learning using a plurality of the information. (17) The information processing system according to (16), wherein the learning device generates a learning model for inspection using, as teacher information, information among the plurality of pieces of information that indicates that the inspector is an expert. (18) The information processing system according to (16) or (17), wherein the learning device generates a learning model for inspection using any or all of the plurality of images, the shooting condition information for each image, the identification information of the inspector for each image, and the inspection result information. (19) An information processing device comprising: a processing unit that generates information including an image obtained by a first imaging unit that images an object to be inspected, imaging condition information when the object to be inspected was imaged, identification information of an inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.(20) An information generation method that generates information including an image obtained by a first imaging unit that images an object to be inspected, imaging condition information when the object to be inspected was imaged, identification information of an inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected. (21) An information processing device that has any or all of the multiple components included in the information processing system described in any one of (1) to (18). (22) An information generation method that uses any or all of the multiple components included in the information processing system described in any one of (1) to (18).

[0184] 1 Appearance inspection system 10 Robot system 11 Robot device 12 Robot control device 13 Inspection table 20 Work device 21 Display unit 22 Operation unit 23 Arithmetic device 30 Storage device 31 Processing unit 32 Storage unit 40 Learning device 41 Processing unit 50 Network 60 Robot device 61 Machine body 62 Wheels 63 Camera 64 Grip unit 111 Polarized camera 112 Camera 113 Lighting unit 114 Robot arm 114a Arm 114b Arm 121 Control unit 122 Arm control unit 611 Control device 612 Image sensor 613 Unit drive unit 614 Machine body drive unit 641 Hand 642 Robot arm A1 Inspection object A1a Partial area A2 Judge A3 Harvested object B1 Graph B2a Graph B2b Graph B3a Graph B3b Graph B4 Graph B5 Graph C1 Spider Chart J1 Set J2 Set J3 Set

Claims

1. An information processing system comprising: a first imaging unit that photographs an object to be inspected to obtain an image; and a processing unit that generates information including the image obtained by the first imaging unit, information about the imaging conditions when the object to be inspected was photographed, identification information of an inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

2. The information processing system of claim 1, wherein the processing unit adds the shooting condition information to the image, and adds the identification information of the examiner and the test result information to the image with the shooting condition information added.

3. The information processing system according to claim 1, further comprising a second imaging unit that images the object to be inspected and obtains surface normal information, wherein the imaging condition information includes the surface normal information.

4. The information processing system according to claim 3, wherein the first imaging unit images a partial area of ​​the object to be inspected, and the imaging condition information includes position information of the partial area and surface normal information of the partial area.

5. An information processing system according to claim 1, wherein the imaging condition information includes position information and orientation information of the first imaging unit when the object to be inspected is imaged.

6. An information processing system as described in claim 1, further comprising an illumination unit that shines light on the object to be inspected, and wherein the shooting condition information includes position information and orientation information of the illumination unit when the object to be inspected is photographed.

7. The information processing system according to claim 1, wherein the information includes the shooting condition information, the identification information of the examiner, and the examination result information for each of the multiple images.

8. The information processing system according to claim 1, wherein the information includes the identification information of the inspector and the inspection result information for each of a plurality of inspectors.

9. The information processing system according to claim 1, wherein the inspection result information includes information on the presence or absence of scratches.

10. The information processing system according to claim 9, wherein the inspection result information includes specified coordinate information indicating the position of the scratch.

11. The information processing system of claim 1, wherein the processing unit compares the inspection result information of the inspector for the image with the inspection result information of an expert for the image, and certifies that the inspector is an expert based on the comparison result.

12. The information processing system according to claim 1, wherein the processing unit determines a skill level of the inspector using the inspection result information of the inspector for the image.

13. The information processing system according to claim 12, wherein the processing unit determines the skill level of the inspector by using the inspection result information of an expert for the image in addition to the inspection result information of the inspector for the image.

14. The information processing system according to claim 12, wherein the processing unit determines the skill level based on one or more of the inspector's proficiency in inspection, accuracy, stability, variation, durability, and decision-making ability.

15. The information processing system according to claim 14, wherein the processing unit organizes the skill levels based on any or all of the proficiency, the accuracy, the stability, the variability, the durability, and the quick decision-making ability.

16. The information processing system according to claim 1, further comprising a learning device that performs machine learning using a plurality of said information.

17. The information processing system according to claim 16, wherein the learning device generates a learning model for testing using, as teacher information, the information among the plurality of pieces of information for which the inspector is an expert.

18. The information processing system of claim 16, wherein the learning device generates a learning model for inspection using any or all of a plurality of images, the shooting condition information for each image, the identification information of the inspector for each image, and the inspection result information.

19. An information processing device comprising a processing unit that generates information including an image obtained by a first imaging unit that photographs an object to be inspected, information about the imaging conditions when the object to be inspected was photographed, identification information of an inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

20. An information generation method, which generates information including an image obtained by a first imaging unit that images an object to be inspected, information about the imaging conditions when the object to be inspected was imaged, identification information of an inspector who inspected the object to be inspected based on the image, and inspection result information of the object to be inspected.

Images