PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING APPARATUS

Abstract

To provide a program or the like capable of extracting an appropriate matching area from a matching object. [Solution] In one aspect, the program acquires a part image of a part that includes multiple feature point objects, recognizes the object within the acquired part image, extracts multiple area images of a predetermined size at a predetermined offset position from the recognized object, acquires part-specific information corresponding to each extracted area image, determines the number of matches of the part-specific information corresponding to each area image in each of the area images, and causes a computer to execute a process to authenticate the part based on the number of matches of the part-specific information and a predetermined threshold value.

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] In recent years, there has been an increasing need to verify and recognize verification objects such as products or parts (components) assembled from products for quality control and distribution management. Patent Document 1 discloses a verification system that verifies a verification object and a registered object based on statistics of pixel brightness values ​​of a registered image corresponding to pixel positions estimated to cause regular reflection in a verification image of a predetermined verification area of ​​the verification object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 136900 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 may not be able to automatically extract the matching area (area image) of the matching object.

[0005] In one aspect, an object of the present invention is to provide a program or the like capable of extracting an appropriate matching region from a matching object. [Means for solving the problem]

[0006] In one aspect, the program includes acquiring a part image of a part including a plurality of feature point objects, Each Recognized and recognized the object EachThe computer extracts multiple area images of a predetermined size at a predetermined offset position from the object, obtains part-specific information corresponding to each of the extracted area images, determines the number of matches of the part-specific information corresponding to each area image in each of the area images, and executes a process of authenticating the part based on the number of matches of the part-specific information and a predetermined threshold value. Effect of the Invention

[0007] In one aspect, it becomes possible to extract an appropriate matching region for a matching object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of a traceability system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of an image processing server. [Diagram 3] FIG. 2 is a block diagram showing a configuration example of a parts server; [Figure 4] FIG. 11 is an explanatory diagram illustrating an example of a record layout of a parts specific information DB. [Diagram 5] FIG. 2 is a block diagram showing a configuration example of a DB server. [Figure 6] FIG. 2 is an explanatory diagram illustrating an example of a record layout of a product management DB. [Figure 7] FIG. 13 is an explanatory diagram illustrating an example of a record layout of a parts detail DB. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of a factory server. [Figure 9] FIG. 11 is an explanatory diagram illustrating an example of a record layout of a product DB. [Figure 10] FIG. 11 is an explanatory diagram for explaining an operation of outputting part specific information. [Figure 11] FIG. 2 is an explanatory diagram for explaining an object recognition model. [Figure 12] 13 is a flowchart showing a processing procedure for outputting part specific information. [Figure 13]13 is a flowchart showing a processing procedure for outputting part specific information. [Figure 14] FIG. 13 is an explanatory diagram for explaining an operation of outputting a product ID in association with a part ID. [Figure 15] 13 is a flowchart showing a processing procedure for outputting a product ID in association with a part ID. [Figure 16] 13 is a flowchart showing a processing procedure for outputting a product ID in association with a part ID. [Figure 17] FIG. 11 is an explanatory diagram showing an overview of a traceability system according to a third embodiment. [Figure 18] 11 is a block diagram showing a configuration example of a deletion server. [Figure 19] FIG. 2 is an explanatory diagram illustrating an example of a record layout of a parts management DB. [Figure 20] FIG. 2 is an explanatory diagram illustrating an example of a record layout of a production plan DB. [Figure 21] 13 is a flowchart showing a processing procedure for deleting a registered part. [Figure 22] FIG. 1 is an explanatory diagram for improving object recognition. [Figure 23] FIG. 13 is an explanatory diagram showing how part images are captured by two photographing devices. [Figure 24] 10 is a flowchart showing a processing procedure for acquiring a part image. [Diagram 25] FIG. 11 is an explanatory diagram for explaining correction of an offset position. [Figure 26] 10 is a flowchart showing a processing procedure for correcting a part image in which area distortion has occurred. [Figure 27] FIG. 23 is a block diagram showing an example of the configuration of an image processing server according to the seventh embodiment. [Figure 28] FIG. 23 is a block diagram showing an example of the configuration of a DB server according to the seventh embodiment. [Figure 29] FIG. 2 is an explanatory diagram showing an example of a record layout of a parts damage information DB. [Diagram 30]10 is a flowchart showing a processing procedure for outputting both part-specific information and information regarding damage to a part. [Diagram 31] FIG. 23 is an explanatory diagram showing an example of a record layout of a parts specific information DB according to the eighth embodiment. [Diagram 32] FIG. 23 is an explanatory diagram showing an example of a record layout of a parts detail DB according to the eighth embodiment. [Diagram 33] 13 is a flowchart showing a processing procedure for outputting part storage information in association with a part ID. [Diagram 34] 13 is a flowchart showing a processing procedure for detecting a retained part. [Diagram 35] FIG. 13 is a block diagram showing an example of the configuration of an image processing server according to a ninth embodiment. [Diagram 36] FIG. 13 is a block diagram showing an example of the configuration of a DB server according to a ninth embodiment. [Figure 37] FIG. 13 is an explanatory diagram illustrating an example of a record layout of a matching result DB. [Figure 38] FIG. 11 is an explanatory diagram for explaining a wear identification model. [Figure 39] 11 is a flowchart showing a processing procedure when checking parts. [Diagram 40] 11 is a flowchart showing a processing procedure when performing a part registration process. [Diagram 41] 10 is a flowchart showing a processing procedure when performing a part matching process. [Diagram 42] 1 is an explanatory diagram showing an example of a cylindrical object having a plurality of objects distributed on an end surface of the cylindrical object; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present invention will be described in detail with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) The first embodiment relates to a form in which an object in a part image is recognized, an area image is extracted at a predetermined offset position from the recognized object, and part-specific information corresponding to the extracted area image is output. The part is a part (part or component) for constituting the entire product such as an automobile, a personal computer, a home appliance, a smartphone, or a machine tool. For example, if the product is an automobile, the part is a door panel, a mirror, a tire, or the like. The object is a feature point on the part, such as a press punched shape, a drill hole, an edge shape, a paint, a seal, a forced dent hole, a laser marking, a three-dimensional character, or the like.

[0011] The part-specific information includes the Fingerprint of the part, manufacturing information, material information, or mounting information. The Fingerprint of the thing is a fine pattern (e.g., unevenness or a pattern) on the surface of a part (object). The Fingerprint of the thing is generated naturally during the manufacturing stage, and the Fingerprint of the thing itself does not change even if it is scratched later, so that each part can be distinguished by using the Fingerprint of the thing recognition technology (image recognition technology).

[0012] 1 is an explanatory diagram showing an overview of a traceability system. The system of this embodiment includes an information processing device 1, an image capturing device 2, an information processing device 3, a database device 4, and an information processing device 5, and each device transmits and receives information via a network N such as the Internet.

[0013] The information processing device 1 is an information processing device that processes, stores, and transmits / receives various types of information. The information processing device 1 is, for example, a server device, an edge computer, a personal computer, or a general-purpose tablet PC (personal computer). In this embodiment, the information processing device 1 is a server device, and hereinafter, for simplicity, it will be read as an image processing server 1.

[0014] The photographing device 2 photographs parts including an object to generate an image. The photographing device 2 of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits and receives photographed images to and from the image processing server 1 and the like via the network N. Note that instead of the photographing device 2, an information processing terminal capable of photographing may be used. The information processing terminal is, for example, an information processing device such as a smartphone, a mobile phone, a tablet, or a personal computer terminal.

[0015] The information processing device 3 is an information processing device that performs registration, authentication, etc. of parts. The information processing device 3 is, for example, a server device, a personal computer, a general-purpose tablet PC, etc. In this embodiment, the information processing device 3 is a server device, and will be referred to as a parts server 3 below for simplicity.

[0016] The database (DB) device 4 is a storage device that stores part-specific information and information about products using the parts. In the present embodiment, the database device 4 will be read as a DB server 4 for the sake of simplicity.

[0017] The information processing device 5 is an information processing device that manages manufacturing information and material information of each part for each product in a factory or the like. The information processing device 5 is, for example, a server device, a personal computer, a general-purpose tablet PC, etc. In this embodiment, the information processing device 5 is a server device, and hereinafter will be referred to as a factory server 5 for simplicity.

[0018] In the present embodiment, an example including one information processing device 1 and one photographing device 2 has been described, but the present invention is not limited to this, and may include a plurality of information processing devices 1 and a plurality of photographing devices 2. For example, when a combination of one information processing device 1 (for registration) and one photographing device 2 (for registration) and a combination of one information processing device 1 (for authentication) and one photographing device 2 (for authentication) are used, the information processing device 1 (for registration) performs a process of registering a part image photographed by the photographing device 2 (for registration) in the parts server 3. The information processing device 1 (for authentication) performs a process of authenticating the part image photographed by the photographing device 2 (for authentication) by collating it with the part image already registered in the parts server 3.

[0019] The image processing server 1 according to this embodiment acquires a part image including an object that is a feature point from the photographing device 2, and recognizes the object in the acquired part image. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object, and acquires part unique information corresponding to the extracted area image. The image processing server 1 assigns a part ID to the acquired part unique information, and outputs the assigned part ID and part unique information to the part server 3 and DB server 4.

[0020] 2 is a block diagram showing an example of the configuration of the image processing server 1. The image processing server 1 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and a large-capacity storage unit 17. Each component is connected by a bus B.

[0021] The control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and performs various information processing, control processing, and the like related to the image processing server 1 by reading and executing a control program 1P stored in the storage unit 12. Note that, although the control unit 11 is described in Fig. 2 as being a single processor, it may be a multiprocessor.

[0022] The storage unit 12 includes memory elements such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and stores the control program 1P or data required for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information between the imaging device 2, the parts server 3, the DB server 4, and the factory server 5 via the network N.

[0023] The input unit 14 is an input device such as a mouse, a keyboard, a touch panel, or a button, and outputs received operation information to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11.

[0024] The reading unit 16 reads the portable storage medium 1a including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the mass storage unit 17. The control unit 11 may also download the control program 1P from another computer via a network N or the like and store it in the mass storage unit 17. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

[0025] The mass storage unit 17 includes a recording medium such as a hard disk drive (HDD) or a solid state drive (SSD). The mass storage unit 17 includes an object recognition model 171. The object recognition model 171 is a recognizer that recognizes an object in a part image based on the part image, and is a trained model generated by machine learning.

[0026] In this embodiment, the storage unit 12 and the large-capacity storage unit 17 may be configured as an integrated storage device. The large-capacity storage unit 17 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 17 may be an external storage device connected to the image processing server 1.

[0027] In this embodiment, the image processing server 1 is described as being one information processing device, but the image processing server 1 may be configured as a virtual machine, or may be configured as a distributed processing device.

[0028] 3 is a block diagram showing a configuration example of the parts server 3. The parts server 3 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, a display unit 35, a reading unit 36, and a large-capacity storage unit 37. Each component is connected by a bus B.

[0029] The control unit 31 includes an arithmetic processing unit such as a CPU, an MPU, or a GPU, and performs various information processing, control processing, and the like related to the parts server 3 by reading and executing a control program 3P stored in the storage unit 32. Note that, although the control unit 31 is described as being a single processor in FIG. 3, it may be a multiprocessor.

[0030] The storage unit 32 includes memory elements such as RAM and ROM, and stores the control program 3P or data required for the control unit 31 to execute processing. The storage unit 32 also temporarily stores data required for the control unit 31 to execute arithmetic processing. The communication unit 33 is a communication module for performing processing related to communication, and transmits and receives information to and from the image processing server 1 via the network N.

[0031] The input unit 34 is an input device such as a mouse, a keyboard, a touch panel, or a button, and outputs received operation information to the control unit 31. The display unit 35 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 31.

[0032] The reading unit 36 ​​reads the portable storage medium 3a including a CD-ROM or a DVD-ROM. The control unit 31 may read the control program 3P from the portable storage medium 3a via the reading unit 36 ​​and store it in the mass storage unit 37. The control unit 31 may also download the control program 3P from another computer via the network N or the like and store it in the mass storage unit 37. Furthermore, the control unit 31 may read the control program 3P from the semiconductor memory 3b.

[0033] The mass storage unit 37 includes a recording medium such as an HDD, an SSD, etc. The mass storage unit 37 includes a part unique information DB 371. The part unique information DB 371 stores part unique information in association with a part ID.

[0034] In this embodiment, the storage unit 32 and the large-capacity storage unit 37 may be configured as an integrated storage device. The large-capacity storage unit 37 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 37 may be an external storage device connected to the parts server 3.

[0035] In this embodiment, the parts server 3 is described as being one information processing device, but the parts server 3 may be configured as a virtual machine, or may be configured as a distributed information processing device.

[0036] 4 is an explanatory diagram showing an example of a record layout of the parts unique information DB371. The parts unique information DB371 includes a parts ID column, a parts image column, and a registration date and time column. The parts ID column stores a uniquely specified part ID in order to identify each part. The parts image column stores an area image of the whole or a part of the part image. The registration date and time column stores date and time information when the part was registered. Note that manufacturing information, material information, attachment information, etc. of a part belonging to the parts unique information may also be stored in the parts unique information DB371.

[0037] 5 is a block diagram showing an example of the configuration of the DB server 4. The DB server 4 includes a control unit 41, a storage unit 42, a communication unit 43, a reading unit 44, and a large-capacity storage unit 45. Each component is connected by a bus B.

[0038] The control unit 41 includes an arithmetic processing unit such as a CPU, an MPU, or a GPU, and performs data reading, data storage, etc. by reading and executing a control program 4P stored in the storage unit 42. Note that, although the control unit 41 is described in Fig. 5 as being a single processor, it may be a multiprocessor.

[0039] The storage unit 42 includes memory elements such as RAM and ROM, and stores the control program 4P or data required for the control unit 41 to execute processing. The storage unit 42 also temporarily stores data required for the control unit 41 to execute arithmetic processing. The communication unit 43 is a communication module for performing processing related to communication, and transmits and receives information to and from the image processing server 1 via the network N.

[0040] The reading unit 44 reads the portable storage medium 4a including a CD-ROM or a DVD-ROM. The control unit 41 may read the control program 4P from the portable storage medium 4a via the reading unit 44 and store it in the mass storage unit 45. The control unit 41 may also download the control program 4P from another computer via the network N or the like and store it in the mass storage unit 45. Furthermore, the control unit 41 may read the control program 4P from the semiconductor memory 4b.

[0041] The mass storage unit 45 includes a recording medium such as an HDD or SSD. The mass storage unit 45 includes a product management DB 451 and a parts detail DB 452. The product management DB 451 stores information such as a correspondence between a product ID and the ID of a part that is assembled from the product. The parts detail DB 452 stores detailed information of parts in association with the part ID.

[0042] In this embodiment, the storage unit 42 and the large-capacity storage unit 45 may be configured as an integrated storage device. The large-capacity storage unit 45 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 45 may be an external storage device connected to the DB server 4.

[0043] In this embodiment, the DB server 4 is described as being one database device, but the processing may be distributed among a plurality of devices, or the DB server 4 may be configured as a virtual machine.

[0044] 6 is an explanatory diagram showing an example of a record layout of the product management DB 451. The product management DB 451 includes a product ID column, a type column, a product name column, and a part ID column. The product ID column stores a unique product ID to identify each product. The type column stores product type information. The product name column stores the name of the product. The part ID column stores part IDs that identify the parts assembled into the product.

[0045] 7 is an explanatory diagram showing an example of a record layout of the part details DB 452. The part details DB 452 includes a part ID column, a manufacturing information column, a material information column, and an attachment information column. The part ID column stores a unique part ID to identify each part.

[0046] The manufacturing information column includes a processing machine ID column, a processing machine manufacturer name column, a location address column, and a processing date and time column. The processing machine ID column stores a uniquely specified ID of the processing machine to identify each processing machine. The processing machine manufacturer name column stores the name of the processing machine manufacturer. The location address column stores the address or name of the production line where the part was processed. The processing date and time column stores information on the date and time when the part was processed.

[0047] The material information column includes a material ID column, a material manufacturer ID column, a lot number column, and a delivery date and time column. The material ID column stores a unique material ID to identify each material. The material manufacturer ID column stores a unique material manufacturer ID to identify each material manufacturer. The lot number column stores material lot numbers assigned to each production unit when the material is manufactured. The delivery date and time column stores information on the date and time when the material was delivered.

[0048] The installation information column includes an installation robot ID column, an installation manufacturer name column, and a location address column. The installation robot ID column stores a unique ID of the installation robot to identify each installation robot. The installation manufacturer name column stores the name of the manufacturer of the installation robot used when installing the part on the product. The location address column stores the address or name of the production line where the part was installed.

[0049] 8 is a block diagram showing an example of the configuration of the factory server 5. The factory server 5 includes a control unit 51, a storage unit 52, a communication unit 53, an input unit 54, a display unit 55, a reading unit 56, and a large-capacity storage unit 57. Each component is connected by a bus B.

[0050] The control unit 51 includes an arithmetic processing unit such as a CPU, an MPU, or a GPU, and performs various information processing, control processing, and the like related to the factory server 5 by reading and executing a control program 5P stored in the storage unit 52. Note that, although the control unit 51 is described in FIG. 8 as being a single processor, it may be a multiprocessor.

[0051] The storage unit 52 includes memory elements such as RAM and ROM, and stores the control program 5P or data required for the control unit 51 to execute processing. The storage unit 52 also temporarily stores data required for the control unit 51 to execute arithmetic processing. The communication unit 53 is a communication module for performing processing related to communication, and transmits and receives information to and from the image processing server 1 via the network N.

[0052] The input unit 54 is an input device such as a mouse, a keyboard, a touch panel, or a button, and outputs received operation information to the control unit 51. The display unit 55 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 51.

[0053] The reading unit 56 reads the portable storage medium 5a including a CD-ROM or a DVD-ROM. The control unit 51 may read the control program 5P from the portable storage medium 5a via the reading unit 56 and store it in the mass storage unit 57. The control unit 51 may also download the control program 5P from another computer via the network N or the like and store it in the mass storage unit 57. Furthermore, the control unit 51 may read the control program 5P from the semiconductor memory 5b.

[0054] The mass storage unit 57 includes a recording medium such as an HDD, an SSD, etc. The mass storage unit 57 includes a product DB 571. The product DB 571 stores information about products and information about parts that assemble the products.

[0055] In this embodiment, the storage unit 52 and the large-capacity storage unit 57 may be configured as an integrated storage device. The large-capacity storage unit 57 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 57 may be an external storage device connected to the factory server 5.

[0056] In this embodiment, the factory server 5 is described as being one information processing device, but the processing may be distributed among a plurality of devices, or may be configured as a virtual machine.

[0057] 9 is an explanatory diagram showing an example of a record layout of the product DB 571. The product DB 571 includes a product ID column, a product type column, a part type column, a part feature column, a manufacturing information column, a material information column, and an attachment information column.

[0058] The product ID column stores a unique product ID to identify each product. The product type column includes an ID column and a name column. The ID column stores the ID of the product type. The name column stores the name of the product type. The part type column includes an ID2 column and a name2 column. The ID2 column stores the ID of the part type. The name2 column stores the name of the part type. Note that the layout of the product type column or part type column is not limited to the above-mentioned. The product type column or part type column may store other information in addition to the ID and name.

[0059] The part feature sequence includes an offset position sequence, an area image size sequence, and an area image sequence. The offset position sequence stores distance information from an object. The area image size sequence stores the size of an area image for acquiring part-specific information. The area image sequence stores an image (a part of a part image) showing the fingerprint (feature) of a part.

[0060] The manufacturing information column, material information column, and attachment information column are similar to the manufacturing information column, material information column, and attachment information column of the parts detail DB 452, and therefore the explanation thereof will be omitted.

[0061] The above-mentioned storage formats of each DB are merely examples, and other storage formats may be used as long as the relationships between the data are maintained.

[0062] 10 is an explanatory diagram for explaining the operation of outputting part-specific information. The control unit 11 of the image processing server 1 acquires a part image including an object that is a feature point from the photographing device 2 via the communication unit 13. Note that, in this embodiment, an example in which the part image is transmitted from the photographing device 2 has been described, but this is not limiting. For example, the part image photographed by the photographing device 2 may be transferred to the image processing server 1 via a memory card.

[0063] The control unit 11 inputs the acquired part image to the trained object recognition model 171, which outputs a recognition result of recognizing an object in the part image when the acquired part image is input, and outputs the recognition result of recognizing an object in the part image. Note that the object recognition process will be described later.

[0064] When the control unit 11 recognizes an object based on the recognition result, it extracts an area image of a predetermined size at a predetermined offset position from the recognized object. The offset position is a distance away from the object, for example, 20 pixels from the right of the object. The predetermined size is the size of the area image extracted from the part image in pixel (picture element) units, for example, 120×120 pixels. The predetermined offset position and predetermined size are stored in advance in the storage unit 12 or the large-capacity storage unit 17 for each type of object.

[0065] The control unit 11 acquires part unique information corresponding to the extracted area image from the factory server 5. Specifically, the control unit 11 transmits the part type ID and the offset position by the object together with the extracted area image to the factory server 5 via the communication unit 13. The control unit 51 of the factory server 5 receives the part type ID, offset position, and area image transmitted from the image processing server 1 via the communication unit 53. The control unit 51 acquires the corresponding part unique information from the product DB 571 of the mass storage unit 57 based on the received part type ID, offset position, and area image. The part unique information includes manufacturing information, material information, and attachment information of the part to the product.

[0066] Specifically, the control unit 51 acquires the size of the area image from the received area image. The control unit 51 extracts an area image stored in the area image string of the product DB 571 based on the size of the acquired area image and the received part type ID and offset position. The control unit 51 performs a matching process between the received area image and the area image extracted from the product DB 571. When the control unit 51 determines that the degree of match is equal to or greater than a predetermined threshold, it acquires the corresponding part specific information from the product DB 571. The control unit 51 may compare the directly captured area image with the area image stored in the product DB 571.

[0067] The control unit 51 transmits the acquired part-specific information, including manufacturing information, material information, and attachment information for the part to the product, to the image processing server 1 via the communication unit 53. The control unit 11 of the image processing server 1 receives the part-specific information transmitted from the factory server 5 via the communication unit 13, and assigns a part ID to the received part-specific information. Note that, in the present embodiment, an example has been described in which the control unit 11 assigns a part ID, but this is not limiting. For example, the control unit 11 may obtain and use a part ID for management purposes from the factory server 5.

[0068] The control unit 11 registers the part corresponding to the assigned part ID in the parts server 3. Specifically, the control unit 11 transmits (outputs) the extracted area image to the parts server 3 in association with the assigned part ID via the communication unit 13. The control unit 31 of the parts server 3 receives the part ID and area image transmitted from the image processing server 1 via the communication unit 33. The control unit 31 stores the area image in association with the received part ID in the parts unique information DB 371 of the mass storage unit 37. Specifically, the control unit 31 stores the part ID, the area image, and the registration date and time in the parts unique information DB 371 as one record.

[0069] The control unit 11 outputs the part unique information (part detailed information) corresponding to the assigned part ID to the DB server 4. Specifically, the control unit 11 transmits (outputs) the received part unique information to the DB server 4 via the communication unit 13 in association with the assigned part ID. The control unit 41 of the DB server 4 receives the part ID and the part unique information transmitted from the image processing server 1 via the communication unit 43. The control unit 41 stores the part unique information in the part detail DB 452 of the mass storage unit 45 in association with the received part ID. Specifically, the control unit 41 stores the part ID, manufacturing information (processing machine ID, processing machine manufacturer name, location address, processing date and time), material information (material ID, material manufacturer ID, lot number, delivery date and time), and installation information (installation robot ID, installation manufacturer name, location address) as one record in the part detail DB 452.

[0070] Next, an object recognition process using the object recognition model 171 will be described. Fig. 11 is an explanatory diagram for explaining the object recognition model 171. The object recognition model 171 is used as a program module that is a part of artificial intelligence software. The object recognition model 171 is a recognizer in which a neural network has been constructed, which receives a part image as input and outputs a recognition result including the position and type of an object in the part image.

[0071] The object recognition model 171 of the present embodiment performs estimation using RCNN (Regions with Convolutional Neural Network) as an example. The object recognition model 171 includes a region candidate extraction unit 71a and a classification unit 71b. The classification unit 71b includes a neural network (not shown). The neural network includes a convolution layer, a pooling layer, and a fully connected layer.

[0072] The control unit 11 of the image processing server 1 performs learning using a combination of part images and training data in which the positions (coordinate ranges of object areas) and types (e.g., triangular paint marks, star-shaped stickers, etc.) of objects in the part images are associated with each other. The training data is data in which objects in the part images are labeled for each part image.

[0073] A part image is input to the trained object recognition model 171. The region candidate extraction unit 71a extracts region candidates of various sizes from the part image. The classification unit 71b calculates feature amounts of the extracted region candidates. Based on the calculated feature amounts, the classification unit 71b classifies whether or not a subject shown in the region candidate is an object, and if the subject is an object, the type of object. The object recognition model 171 repeats the extraction and classification of region candidates.

[0074] The object recognition model 171 outputs the probability that an object is captured, the position (coordinate range of the area), and the type of the object for the area candidate classified as having an object captured with a probability higher than a predetermined threshold (e.g., 85%). In the example shown in Fig. 11, an area (object position) where an object is captured with a probability of 90% and an object type are detected.

[0075] 11, an example in which a single object in a part image is recognized has been described, but the present invention is not limited to this. When there are multiple objects in a part image, control unit 11 can recognize the multiple objects by using object recognition model 171.

[0076] In this case, the classification unit 71b calculates the feature amounts of the area candidates extracted by the area candidate extraction unit 71a, and classifies whether or not each subject shown in the area candidate is an object based on the calculated feature amount, and classifies the type of the object if each subject is an object. The object recognition model 171 repeats the extraction and classification of area candidates, and outputs the recognition result including the position and type of each object.

[0077] Instead of RCNN, any object detection algorithm may be used, such as CNN (Convolution Neural Network), Fast RCNN, Faster RCNN, Mask RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), SVM (Support Vector Machine), Bayesian network, or regression tree.

[0078] In this embodiment, the object recognition model 171 may be constructed by the image processing server 1, or may be constructed by an external information processing device. When the object recognition model 171 is constructed by an external information processing device, the image processing server 1 may download the trained object recognition model 171 from the external information processing device and install it.

[0079] The object recognition process is not limited to the above-mentioned machine learning recognition process. For example, objects in part images may be recognized using local feature extraction methods such as A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), SURF (Speed-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and HOG (Histograms of Oriented Gradients).

[0080] 12 and 13 are flowcharts showing a processing procedure for outputting unique part information. The control unit 11 of the image processing server 1 acquires a part image from the photographing device 2 via the communication unit 13 (step S101). The control unit 11 inputs the acquired part image to a trained object recognition model 171 that outputs a recognition result of recognizing an object in the part image when the acquired part image is input, and outputs the recognition result of recognizing the object in the part image (step S102).

[0081] The control unit 11 judges whether or not an object has been recognized based on the recognition result (step S103). If the control unit 11 judges that an object has not been recognized (NO in step S103), the process returns to step S101. If the control unit 11 judges that an object has been recognized (YES in step S103), the control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S104).

[0082] Specifically, the control unit 11 identifies the type of object (for example, a star sticker, a round forced dent hole, etc.) based on the recognition result. The control unit 11 reads out the offset position and the size of the area image corresponding to the identified type of object from the storage unit 12 or the mass storage unit 17. For example, the offset position corresponding to the recognized object is 20 pixels to the right from the center of the bounding box of the object in the horizontal direction. The control unit 11 extracts an area image of the size of the read area image (for example, 120 x 120 pixels) at the offset position read out from the center of the bounding box of the recognized object. Note that the extraction process of the area image is not limited to the center of the bounding box of the object described above. For example, the control unit 11 may extract an area image at an offset position starting from the coordinates of a specific corner of the object.

[0083] The control unit 11 transmits the part type ID, the offset position by the object, and the extracted area image to the factory server 5 via the communication unit 13 (step S105). The control unit 51 of the factory server 5 receives the part type ID, the offset position, and the area image transmitted from the image processing server 1 via the communication unit 53 (step S501).

[0084] The control unit 51 acquires part-specific information including manufacturing information, material information, and attachment information of the part to the product from the product DB 571 of the mass storage unit 57 based on the received part type ID, offset position, and area image (step S502). The control unit 51 transmits the acquired part-specific information to the image processing server 1 via the communication unit 53 (step S503). The control unit 11 of the image processing server 1 receives the part-specific information transmitted from the factory server 5 via the communication unit 13 (step S106), and assigns a part ID to the received part-specific information (step S107).

[0085] The control unit 11 transmits the extracted area image to the parts server 3 by the communication unit 13 in association with the assigned part ID (step S108). The control unit 31 of the parts server 3 receives the part ID and area image transmitted from the image processing server 1 via the communication unit 33 (step S301). The control unit 31 stores the area image in the part specific information DB 371 of the mass storage unit 37 in association with the received part ID (step S302).

[0086] The control unit 11 transmits the received part unique information to the DB server 4 via the communication unit 13 in association with the assigned part ID (step S109). The control unit 41 of the DB server 4 receives the part ID and part unique information transmitted from the image processing server 1 via the communication unit 43 (step S401). The control unit 41 stores the part unique information in the part detail DB 452 of the mass storage unit 45 in association with the received part ID (step S402).

[0087] According to this embodiment, it is possible to obtain part specific information corresponding to a region image extracted at a predetermined offset position from an object in a part image.

[0088] According to this embodiment, it is possible to recognize an object in a part image by using an object recognition model 171 based on the part image.

[0089] According to this embodiment, by aligning with the position of the object, it is possible to achieve highly accurate alignment without using a position correction system that uses a laser or the like.

[0090] (Embodiment 2) The second embodiment relates to a form in which the product ID of a product using a part is output to the DB server 4 in association with the part ID of the part. Note that a description of the contents that overlap with the first embodiment will be omitted.

[0091] 14 is an explanatory diagram explaining the operation of outputting a product ID in association with a part ID. The control unit 11 of the image processing server 1 acquires a part image from the photographing device 2 via the communication unit 13. The control unit 11 inputs the acquired part image to the object recognition model 171, and outputs a recognition result of recognizing an object in the part image. When the control unit 11 recognizes an object based on the recognition result, it extracts an area image of a predetermined size at a predetermined offset position from the recognized object.

[0092] Next, a process of acquiring a part ID from the parts server 3 will be described. The control unit 11 transmits the extracted area image to the parts server 3 via the communication unit 13. The control unit 31 of the parts server 3 receives the area image transmitted from the image processing server 1 via the communication unit 33, and extracts the feature amount of the received area image. The control unit 31 compares the feature amount of the extracted image area with the feature amount of the part image stored in the part specific information DB 371 of the large-capacity storage unit 37.

[0093] When the control unit 31 determines that the feature amounts of both match, it acquires the corresponding part ID from the part specific information DB 371. Note that the comparison process of the feature amounts is not limited to a perfect match, and for example, if the degree of match is equal to or greater than a predetermined threshold, it may output a result that the feature amounts of both match. Also, the comparison process of the feature amounts may be performed using a trained image recognition model generated by machine learning. The control unit 31 transmits the acquired part ID to the image processing server 1 via the communication unit 33. The control unit 11 of the image processing server 1 receives the part ID transmitted from the part server 3 via the communication unit 13.

[0094] Next, a process of acquiring the product ID from the factory server 5 will be described. The control unit 11 transmits the part type ID and the offset position by the object together with the extracted area image to the factory server 5 via the communication unit 13. The control unit 51 of the factory server 5 receives the part type ID, the offset position, and the area image transmitted from the image processing server 1 via the communication unit 53.

[0095] The control unit 51 acquires a corresponding product ID from the product DB 571 in the mass storage unit 57 based on the received part type ID, offset position, and area image. The control unit 51 transmits the acquired product ID to the image processing server 1 via the communication unit 53. The control unit 11 of the image processing server 1 receives the product ID transmitted from the factory server 5 via the communication unit 13, and transmits (outputs) the received product ID to the DB server 4 in association with the part ID.

[0096] The control unit 41 of the DB server 4 receives the product ID and part ID transmitted from the image processing server 1 via the communication unit 43. The control unit 41 associates the received product ID with the part ID and stores them in the product management DB 451 of the mass storage unit 45. Specifically, the control unit 41 stores the product ID, product type, product name, and part ID in the product management DB 451 as one record.

[0097] Figures 15 and 16 are flowcharts showing the processing procedure when outputting the product ID in association with the part ID. Note that the same reference numerals are used for the same contents as in Figures 12 and 13, and the description thereof will be omitted. The control unit 11 of the image processing server 1 executes steps S101 to S104. The control unit 11 transmits the extracted area image to the part server 3 via the communication unit 13 (step S121).

[0098] The control unit 31 of the parts server 3 receives the area image transmitted from the image processing server 1 through the communication unit 33 (step S321). The control unit 31 acquires a corresponding part ID from the part specific information DB 371 of the mass storage unit 37 based on the received area image (step S322). The control unit 31 transmits the acquired part ID to the image processing server 1 through the communication unit 33 (step S323). The control unit 11 of the image processing server 1 receives the part ID transmitted from the parts server 3 through the communication unit 13 (step S122).

[0099] The control unit 11 executes step S105. The control unit 51 of the factory server 5 executes step S501. The control unit 51 acquires a corresponding product ID from the product DB 571 of the mass storage unit 57 based on the received part type ID, the offset position by the object, and the area image (step S521). The control unit 51 transmits the acquired product ID to the image processing server 1 via the communication unit 53 (step S522).

[0100] The control unit 11 of the image processing server 1 receives the product ID transmitted from the factory server 5 via the communication unit 13 (step S123), and transmits the received product ID to the DB server 4 in association with the part ID (step S124). The control unit 41 of the DB server 4 receives the product ID and part ID transmitted from the image processing server 1 via the communication unit 43 (step S421). The control unit 41 associates the received product ID with the part ID and stores it in the product management DB 451 of the mass storage unit 45 (step S422).

[0101] According to this embodiment, by storing and managing information that associates a part with a product that uses the part, it becomes possible to ensure traceability in the process after shipment.

[0102] (Embodiment 3) The third embodiment relates to a mode in which a registered part is deleted. Note that a description of the contents overlapping with the first and second embodiments will be omitted. When the registration period of a second part associated with a registered first part based on a production plan for a product has passed a predetermined period, an alert or command to delete the registered first part can be output.

[0103] 17 is an explanatory diagram showing an overview of a traceability system according to a third embodiment. Note that the same reference numerals are used to designate the same contents as those in FIG. 1, and the description thereof will be omitted. The system according to this embodiment includes an information processing device 6.

[0104] The information processing device 6 is an information processing device that detects expired parts, transmits an instruction to delete registered parts associated with the expired parts, etc. The information processing device 6 is, for example, a server device, a personal computer, a general-purpose tablet PC, etc. In this embodiment, the information processing device 6 is a server device, and hereinafter will be referred to as a deletion server 6 for the sake of simplicity.

[0105] 18 is a block diagram showing a configuration example of the deletion server 6. The deletion server 6 includes a control unit 61, a storage unit 62, a communication unit 63, an input unit 64, a display unit 65, a reading unit 66, a large-capacity storage unit 67, and a clock unit 68. Each component is connected by a bus B.

[0106] The control unit 61 includes an arithmetic processing unit such as a CPU, an MPU, or a GPU, and performs various information processing, control processing, and the like related to the deletion server 6 by reading and executing a control program 6P stored in the storage unit 62. Note that, although the control unit 61 is described in FIG. 18 as being a single processor, it may be a multiprocessor.

[0107] The storage unit 62 includes memory elements such as RAM and ROM, and stores the control program 6P or data required for the control unit 61 to execute processing. The storage unit 62 also temporarily stores data required for the control unit 61 to execute arithmetic processing. The communication unit 63 is a communication module for performing processing related to communication, and transmits and receives information between the parts server 3, the DB server 4, etc. via the network N.

[0108] The input unit 64 is an input device such as a mouse, a keyboard, a touch panel, or a button, and outputs received operation information to the control unit 61. The display unit 65 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 61.

[0109] The reading unit 66 reads the portable storage medium 6a including a CD-ROM or a DVD-ROM. The control unit 61 may read the control program 6P from the portable storage medium 6a via the reading unit 66 and store it in the mass storage unit 67. The control unit 61 may also download the control program 6P from another computer via the network N or the like and store it in the mass storage unit 67. Furthermore, the control unit 61 may read the control program 6P from the semiconductor memory 6b.

[0110] The mass storage unit 67 includes a recording medium such as an HDD or SSD. The mass storage unit 67 includes a parts management DB 671 and a production plan DB 672. The parts management DB 671 stores management information for managing registration deadlines and the like of parts based on a production plan for the product. The production plan DB 672 stores a production plan for the product. The production plan is information about the product, the parts to assemble the product, the quantity of the parts, and information about the production plan such as the registration (delivery) date and time.

[0111] In this embodiment, the storage unit 62 and the large-capacity storage unit 67 may be configured as an integrated storage device. The large-capacity storage unit 67 may be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 67 may be an external storage device connected to the deletion server 6.

[0112] In this embodiment, the removal server 6 is described as a single information processing device, but it may be distributed to multiple devices for processing, or may be configured as a virtual machine. Furthermore, the removal server 6 may be integrated with the factory server 5 or another server.

[0113] FIG. 19 is an explanatory diagram showing an example of a record layout of the parts management DB 671. The parts management DB 671 includes a management ID column, a production plan ID column, a parts ID column, a registration status column, and a registration date and time column. The management ID column stores a uniquely specified ID of management data to identify each management data. The production plan ID column stores a production plan ID that identifies a production plan. The parts ID column stores a part ID that identifies a part. The registration status column stores the registration status of the part in the parts server 3. The registration status column may store, for example, "registered", "not registered", etc. The registration date and time column stores date and time information when the part was registered in the parts server 3.

[0114] FIG. 20 is an explanatory diagram showing an example of a record layout of the production plan DB 672. The production plan DB 672 includes a production plan ID column, a product ID column, a parts quantity column, a parts ID column, and a registration deadline column. The production plan ID column stores a uniquely specified production plan ID to identify each production plan. The product ID column stores a product ID that identifies the product. The parts quantity column stores the quantity of parts to assemble the product. The parts ID column stores part IDs that identify the parts. The registration deadline column stores deadline information for registering parts in the parts server 3.

[0115] The control unit 61 of the deletion server 6 extracts part IDs (multiple parts are possible) whose registration status is "unregistered" and production plan IDs corresponding to the part IDs from the parts management DB 671 of the mass storage unit 67. The control unit 61 acquires the registration deadline of the part corresponding to the part ID from the production plan DB 672 of the mass storage unit 67 based on the extracted part ID and production plan ID.

[0116] The control unit 61 obtains the current date and time from the clock unit 68, and compares the obtained current date and time with the registration expiration date of the part. If the control unit 61 determines that the registration expiration date of the part has passed, it transmits (outputs) an alert to the parts server 3 or the DB server 4 in association with the part ID to delete the part. The alert may be, for example, a message including a message that the registration expiration date of the part has passed. The control unit 61 may also create a report for the expired part and output the created report.

[0117] Furthermore, the control unit 61 can transmit (output) a command to delete the part in association with the expired part ID to the parts server 3 or the DB server 4. When the parts server 3 or the DB server 4 receives a command to delete the expired part from the deletion server 6, it deletes the corresponding registered part that has been stored.

[0118] 21 is a flow chart showing the processing procedure when deleting a registered part. The control unit 61 of the deletion server 6 extracts part IDs whose registration status is "unregistered" and production plan IDs corresponding to the part IDs from the parts management DB 671 of the mass storage unit 67 (step S631). The extracted part ID may be a single part ID or multiple part IDs. An example of multiple part IDs will be described below.

[0119] Based on each extracted part ID and the production plan ID corresponding to the part ID, the control unit 61 acquires the registration deadline corresponding to each part ID from the production plan DB 672 of the mass storage unit 67 (step S632). The control unit 61 acquires the current date and time from the clock unit 68, compares the acquired current date and time with the registration deadline of the part, and extracts expired part IDs (step S633).

[0120] The control unit 61 generates a report of the expired parts based on the extracted part ID (step S634). The report includes, for example, the part ID and the registration expiration date of the part. The control unit 61 transmits the generated report and a deletion instruction to delete the expired parts to the parts server 3 and the DB server 4 via the communication unit 63 (step S635). An alert including a notice to delete the expired parts may also be transmitted.

[0121] The control unit 31 of the parts server 3 receives the report and the deletion instruction transmitted from the deletion server 6 via the communication unit 33 (step S331). The control unit 31 reads the part ID from the received report, and deletes the corresponding part from the part specific information DB 371 of the mass storage unit 37 (step S332).

[0122] The control unit 41 of the DB server 4 receives the report and the deletion instruction transmitted from the deletion server 6 via the communication unit 43 (step S431). The control unit 41 reads the part ID from the received report, and deletes the corresponding part from the part detail DB 452 of the mass storage unit 45 (step S432).

[0123] However, the process is not limited to the above. For example, the deletion server 6 may transmit only the generated report to the parts server 3 or the DB server 4. In this case, the parts server 3 or the DB server 4 receives and displays the report transmitted from the deletion server 6. After the person in charge checks the contents of the report, the person in charge may input a deletion instruction for the parts to be deleted to the parts server 3 or the DB server 4.

[0124] According to this embodiment, if an error occurs when registering (authenticating) a part, an alert or deletion instruction is output to delete a registered part associated with the part, which can be useful for process and quality improvement.

[0125] (Embodiment 4) The fourth embodiment relates to an embodiment for improving the recognition of objects. Note that a description of the contents that overlap with the first to third embodiments will be omitted.

[0126] To improve recognition of objects in the part image, a virtual camera lens center line or a lighting device can be used. The virtual camera lens center line will be described later. When a lighting device is used, the photographing device 2 photographs the part image after the lighting that illuminates the object is turned on. The control unit 11 of the image processing server 1 acquires the photographed part image from the photographing device 2. Thereafter, similar to the processing of the first embodiment, the control unit 11 performs processing for recognizing objects in the part image, processing for extracting area images, and processing for acquiring part-specific information corresponding to the area images.

[0127] The lighting device is a device equipped with a light source that emits illumination light used to illuminate an object, and has optical functions such as light distribution control from the light source and light color adjustment. The light source device may be an LED (light emitting diode) or a projector. Other light source devices, for example, an electrodeless light source, may also be used. In this embodiment, the lighting device is separate from the image capture device 2, but this is not limited to this. For example, the lighting device may be an integrated type built into the image capture device 2.

[0128] Fig. 22 is an explanatory diagram relating to improved object recognition. Fig. 22A is an image diagram showing a part image without illumination. 10a is an object that is a feature point. 10b is a region image extracted in a predetermined size from the part image at a predetermined offset position from the object.

[0129] FIG. 22B is an image diagram showing a part image captured using the virtual camera lens center line. 10c is a virtual camera lens center line that is an assumed camera lens center line. The virtual camera lens center line is an assumed straight line that passes through the center of the camera lens and is horizontal to the lens surface. For example, the virtual camera lens center line may be set by projection from a projector. When the shutter timing for object recognition is near the center of the camera (virtual camera lens center line), the camera takes multiple images in this vicinity. The image processing server 1 determines the image quality conditions such as the focus state, selects an image that is in good condition for Fingerprint of Object authentication, and performs Fingerprint of Object registration or authentication.

[0130] FIG. 22C is an image showing a part image captured using striped illumination. FIG. 10d is an area image of a part image illuminated using striped illumination. A striped pattern is a pattern consisting of multiple parallel or intersecting lines using two or more different colors or shades of the same color. By using an illumination device to project a striped pattern onto the surface of a part including an object and photographing it, the object becomes easier to recognize due to the change in brightness that occurs on the part surface. Note that the illumination is not limited to striped illumination, and color LED illumination (red, yellow, green, blue, etc.) may also be used.

[0131] FIG. 22D is an image showing a part image captured using pinpoint lighting. FIG. 10e is a region image of a part image illuminated using pinpoint lighting. Pinpoint lighting is a type of lighting that focuses light on one point. By illuminating only the object with pinpoint lighting, the object becomes easier to recognize.

[0132] In the above-mentioned process, an example of recognizing an object using various kinds of illumination has been described, but the present invention is not limited to this. The same can be applied to the Fingerprint of Things of a part surface (region image). For example, a striped pattern is projected onto the surface of a part having fine irregularities such as a highly glossy surface such as metal, and an image is captured, and a feature amount is extracted from the brightness change occurring on the part surface, and the Fingerprint of Things of the part surface is recognized based on the extracted feature amount.

[0133] According to this embodiment, it is possible to capture part images using virtual camera lens centerlines, stripes or pinpoint lighting.

[0134] According to this embodiment, by using a part image captured using a virtual camera lens center line, a striped pattern, or pinpoint lighting, it becomes easier to recognize objects in the part image.

[0135] (Embodiment 5) The fifth embodiment relates to a form in which part images are acquired by two photographing devices. Note that a description of the contents that overlap with the first to fourth embodiments will be omitted. The system of the present embodiment includes a photographing device 2a and a photographing device 2b instead of the photographing device 2.

[0136] The photographing device 2a photographs the entire part. For example, when the photographing target range is the front door of the car, the photographing device 2a photographs the front door to generate an image. For example, the photographing device 2a may be provided with a wide-angle lens with a deep depth of field.

[0137] The photographing device 2b has a higher magnification ratio than the photographing device 2a, and photographs a part of the part image photographed by the photographing device 2a in an enlarged manner. The photographing device 2b may be equipped with a telephoto lens for taking a close-up image of a subject. The photographing device 2b is installed in a position where it can photograph an enlarged image of a part including an object when the photographing device 2a recognizes a target part image.

[0138] The photographing device 2a or the photographing device 2b of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits and receives photographed images to and from the image processing server 1, etc., via the network N. Note that the photographing device 2a or the photographing device 2b may be replaced by an information processing terminal capable of photographing (for example, a smartphone, a mobile phone, a tablet, a personal computer terminal, etc.).

[0139] Fig. 23 is an explanatory diagram of capturing part images using two photographing devices. 11a is an object that is a feature point. 11b is an area image extracted with a predetermined size at a predetermined offset position from the object. The control unit 11 of the image processing server 1 acquires a first photographed image of the part being transported photographed by the photographing device 2a via the communication unit 13.

[0140] The control unit 11 extracts the feature amount of the acquired first captured image, and judges whether or not a part image is present in the first captured image based on the extracted feature amount. For example, when the shooting target range is a front door, the control unit 11 may make a judgment by comparing the feature amount of the acquired first captured image with the feature amount of the front door stored in advance. Regarding the extraction process of the feature amount, the feature amount may be extracted using a local feature extraction method such as A-KAZE or SIFT. Note that, regarding the judgment process of the part image, the judgment may be automatically made using a model constructed by machine learning.

[0141] When the control unit 11 determines that a part image is present in the first captured image, it acquires a second captured image (an enlarged image of a part including an object) captured by the photographing device 2b via the communication unit 13. The control unit 11 inputs the acquired second captured image to the object recognition model 171, and outputs a recognition result of recognizing the object in the second captured image. Note that the object recognition process is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0142] When the control unit 11 determines that no part image is present in the first captured image, it transmits an adjustment instruction for the capturing angle, etc. to the capturing device 2a. The capturing device 2a adjusts the capturing control parameters in response to the adjustment instruction transmitted from the image processing server 1.

[0143] FIG. 24 is a flowchart showing a processing procedure for acquiring a part image. Note that the description will be given assuming that the parts in the part image have been registered in advance in the part server 3. The photographing device 2a photographs a first photographed image of the part being transported, and transmits the photographed first photographed image to the image processing server 1 (step S241). The control unit 11 of the image processing server 1 acquires the first photographed image photographed by the photographing device 2a via the communication unit 13 (step S641). The control unit 11 extracts a feature amount of the acquired first photographed image (step S642). The control unit 11 determines whether or not a part image is present in the first photographed image based on the extracted feature amount (step S643).

[0144] When the control unit 11 determines that a part image exists in the first captured image (YES in S643), it transmits a capture instruction for the second captured image to the photographing device 2b via the communication unit 13 (step S644). The photographing device 2b captures the second captured image in response to the capture instruction for the second captured image transmitted from the image processing server 1 (step S245). The photographing device 2b transmits the captured second captured image to the image processing server 1 (step S246). The control unit 11 of the image processing server 1 receives the second captured image transmitted from the photographing device 2b via the communication unit 13 (step S645). The control unit 11 inputs the received second captured image to the object recognition model 171, and outputs a recognition result for recognizing the object in the second captured image (step S646).

[0145] When the control unit 11 determines that no part image exists in the first photographed image (NO in step S643), it transmits an adjustment instruction such as a photographing angle to the photographing device 2a via the communication unit 13 (step S647). The photographing device 2a receives the adjustment instruction transmitted from the image processing server 1 (step S242). The photographing device 2a adjusts the photographing control parameters (e.g., angle, etc.) in response to the received adjustment instruction (step S243). The photographing device 2a photographs the first photographed image again (step S244) and returns to step S241. Note that if the photographing device 2a is not adjusted, after the processing of NO in step S643, it returns to step S641 and waits until the next part is transported.

[0146] Since the photographing device 2a photographs the entire part, in addition to the object recognition process described above, damage such as scratches, dirt, cracks, wrinkles, and deformations of the part can be detected from the part image. The image processing server 1 may input the first photographed image of the part photographed by the photographing device 2a to a trained damage recognition model generated by machine learning, for example, and output a recognition result that recognizes damage to the part. The image processing server 1 may also extract features of the first photographed image using a local feature extraction method such as A-KAZE, and detect damage to the part based on the extracted features. The process of detecting damage to the part will be described later in the seventh embodiment.

[0147] According to this embodiment, by using two image capturing devices to capture images of parts, it is possible to improve the recognition rate of objects.

[0148] (Embodiment 6) The sixth embodiment relates to a form in which, when a distortion or tilt is detected in a part image, the offset position or the part image is corrected in accordance with the distortion or tilt. Note that a description of the contents that overlap with the first to fifth embodiments will be omitted.

[0149] When a distortion or tilt is detected in a captured part image, it is necessary to correct the offset position or the part image. Fig. 25 is an explanatory diagram for explaining the correction of the offset position. Fig. 25A is an explanatory diagram for explaining a part image without distortion or tilt. 12a is an object in the part image. 12b is a region image extracted with a predetermined size at a predetermined offset position from the object.

[0150] 25B is an explanatory diagram for explaining a part image in which left-right distortion (deformation) has occurred. In this example, the vertical width of an object in a part image is equal to the reference (normal) vertical width, and the horizontal width is shorter than the reference horizontal width, resulting in left-right distortion. In this case, if a region image is extracted at a specified offset position from the object, a positional shift occurs with respect to the region image to be extracted, and therefore the offset position needs to be corrected.

[0151] Specifically, the control unit 11 of the image processing server 1 calculates the aspect ratio after transformation. The control unit 11 corrects the offset position based on the calculated aspect ratio and the reference aspect ratio. For example, the offset position is moved 20 pixels further to the right from the original offset position. In addition to directly correcting the offset position, the offset position may be corrected by correcting the entire part image. Specifically, the control unit 11 performs an aspect ratio correction process based on the reference aspect ratio so as to approach the reference part image, and extracts a region image at a predetermined offset position from the corrected part image.

[0152] Fig. 25C is an explanatory diagram for explaining a part image in which vertical distortion has occurred. In this example, the horizontal width of an object in a part image is equal to the reference horizontal width, and the vertical width is shorter than the reference vertical width, resulting in vertical distortion. The correction process for the offset position is the same as that in Fig. 25B, and therefore will not be described.

[0153] FIG. 25D is an explanatory diagram illustrating a part image in which area reduction distortion has occurred. Differences arise in the area of ​​each area image depending on the perspective of the area image. If the photographing device 2 is installed at a distance far from the part to be photographed, the photographed part image will look small. In this case, the offset position can be corrected by performing area (size) correction on the part image. Specifically, the control unit 11 performs enlargement processing on the area image by increasing the number of pixels that make up the area image of the part image based on the area of ​​the reference part image.

[0154] FIG. 25E is an explanatory diagram illustrating a part image in which area enlargement distortion has occurred. When the photographing device 2 is installed at a close distance from the part to be photographed, the photographed part image appears large. In this case, the offset position can be corrected by performing area correction on the part image. Specifically, the control unit 11 performs a reduction process on the area image by reducing the number of pixels constituting the area image of the reference part image based on the area of ​​the part image.

[0155] For example, nearest neighbor interpolation may be used for the image scaling process. Nearest neighbor interpolation is a linear interpolation method in which the pixel after scaling is divided by the scaling ratio and the resulting value is rounded off to use the pixel at the coordinate obtained as is. In addition, bilinear interpolation, bicubic interpolation, Lanczos interpolation, and other image scaling techniques may be used.

[0156] FIG. 25F is an explanatory diagram for explaining a tilted part image. When there is a deviation in the shooting angle, it is necessary to perform tilt correction on the captured part image. Specifically, the control unit 11 calculates the tilt angle of the tilted part image based on the reference part image. The control unit 11 performs a rotation process according to the calculated tilt angle to correct the tilted part image.

[0157] In addition, with regard to the above-mentioned image processing, the entire part image may be corrected, or area images of the object and offset position may be extracted from the part image, and the extracted area images may be corrected.

[0158] FIG. 26 is a flowchart showing a processing procedure for correcting a part image in which area distortion has occurred. In the following, an example in which area reduction distortion or area enlargement distortion has occurred is shown as an example. The control unit 11 of the image processing server 1 acquires the part image transmitted from the photographing device 2 via the communication unit 13 (step S151). The control unit 11 judges whether or not area distortion has occurred in the acquired part image (step S152). When the control unit 11 judges that area distortion has not occurred (NO in step S152), it ends the processing. When the control unit 11 judges that area distortion has occurred (YES in step S152), it judges whether or not area reduction has occurred (step S153).

[0159] When control unit 11 determines that the area is reduced (YES in step S153), it performs an enlargement process on the area image by increasing the number of pixels that make up the area image of the part image based on the area of ​​the reference part image (step S154). Note that the reference part image is stored in advance in storage unit 12 or mass storage unit 17. When control unit 11 determines that the area is not reduced (area enlargement) (NO in step S153), it performs a reduction process on the area image by reducing the number of pixels that make up the area image of the part image based on the area of ​​the reference part image (step S155).

[0160] According to this embodiment, by performing offset position correction processing on a distorted or tilted part image, it is possible to extract a highly accurate area image.

[0161] (Embodiment 7) The seventh embodiment relates to a form in which both part-specific information (such as a fingerprint of an object) and information on damage to the part are output. Note that a description of the contents that overlap with the first to sixth embodiments will be omitted. Damage to the part includes scratches, stains, cracks, wrinkles, deformations, etc. of the part.

[0162] Fig. 27 is a block diagram showing a configuration example of the image processing server 1 of the seventh embodiment. The same reference numerals are used for the contents overlapping with Fig. 2, and the description thereof will be omitted. The large-capacity storage unit 17 stores a damage recognition model 172. The damage recognition model 172 is a recognizer that recognizes damage to a part in a part image based on the part image, and is a trained model generated by machine learning.

[0163] Fig. 28 is a block diagram showing an example of the configuration of the DB server 4 of the seventh embodiment. The same reference numerals are used for the contents that overlap with Fig. 5, and the description thereof will be omitted. The mass storage unit 45 stores a part damage information DB 453. The part damage information DB 453 stores information on damage to a part detected based on a part image. The information on damage to a part includes the position and type of damage, etc.

[0164] 29 is an explanatory diagram showing an example of a record layout of the part damage information DB453. The part damage information DB453 includes a part ID column, a damage type column, a position column, and a detection date and time column. The part ID column stores a part ID that identifies a part. The damage type column stores the type of damage to the part. The position column stores the position of the damage (e.g., coordinates, etc.). The detection date and time column stores date and time information when damage to the part was detected.

[0165] The image processing server 1 can acquire and output both the part unique information and information regarding damage to the part from the same part image captured by the photographing device 2. Note that the process of acquiring the part unique information is similar to the process of acquiring the part unique information in the first embodiment, and therefore a description thereof will be omitted. The process of acquiring information regarding damage to the part will be described in detail below.

[0166] The control unit 11 of the image processing server 1 acquires a part image from the photographing device 2 via the communication unit 13. The control unit 11 inputs the acquired part image to a trained damage recognition model 172 that outputs a recognition result that recognizes damage to a part in the part image when the acquired part image is input, and outputs the recognition result that recognizes damage in the part image.

[0167] The damage recognition model 172 is used as a program module that is a part of artificial intelligence software. The damage recognition model 172 is a recognizer in which a neural network has been constructed, which receives a part image as input and outputs a recognition result including the position and type of damage to the part in the part image.

[0168] The damage recognition model 172 of the present embodiment performs estimation using RCNN as an example. The damage recognition model 172 includes a region candidate extraction unit and a classification unit. The classification unit includes a neural network.

[0169] The control unit 11 of the image processing server 1 performs learning using a combination of part images and training data in which types of damage (e.g., scratches, stains, cracks, wrinkles, deformations, etc.) in the part images are associated with each other. The training data is data in which damage in the part images is labeled for each part image.

[0170] When a part image is input to trained damage recognition model 172, the region candidate extraction unit extracts region candidates of various sizes from the part image. The classification unit calculates feature amounts of the extracted region candidates. Based on the calculated feature amounts, the classification unit classifies whether the subject shown in the region candidate is damage to a part, and if the subject is damage, the type of damage. Damage recognition model 172 repeats the extraction and classification of region candidates.

[0171] For candidate areas classified as containing damage with a probability higher than a predetermined threshold (e.g., 85%), the damage recognition model 172 outputs the probability that the area contains damage to the part, the location (coordinate range of the area) and the type of damage.

[0172] In addition, the process is not limited to the above, and for example, a dedicated recognition model that has been trained for each type of damage to a part may be used. For example, the control unit 11 performs training using a combination of part images and teacher data in which scratches in the part images are associated with each other, and constructs a scratch recognition model. Using the constructed scratch recognition model, the control unit 11 inputs the part image and outputs a recognition result that recognizes scratches in the part image.

[0173] Note that any object detection algorithm may be used instead of RCNN, such as CNN, Fast RCNN, Faster RCNN, Mask RCNN, SSD, YOLO, SVM, Bayesian network, or regression tree.

[0174] The process of recognizing damage to a part is not limited to the above-mentioned machine learning process. For example, damage in a part image may be recognized using a local feature extraction method such as A-KAZE or SIFT.

[0175] The control unit 11 assigns a part ID to the acquired part unique information and information on part damage. The control unit 11 transmits the part unique information and information on part damage (recognition result) to the DB server 4 via the communication unit 13 in association with the assigned part ID. The control unit 41 of the DB server 4 receives the part ID, part unique information, and information on part damage transmitted from the image processing server 1 via the communication unit 43. The control unit 41 stores the received part ID, part unique information, and information on part damage in the part detail DB 452 and part damage information DB 453 of the mass storage unit 45.

[0176] Specifically, the control unit 41 associates the damage type, location, and detection date and time of the part with the part ID, and stores them as one record in the part damage information DB 453. Note that the part unique information storage process is similar to the part unique information storage process in the first embodiment, and therefore a description thereof will be omitted.

[0177] 30 is a flowchart showing a processing procedure when outputting both part-specific information and information relating to damage to a part. The control unit 11 of the image processing server 1 acquires a part image from the photographing device 2 via the communication unit 13 (step S161). The control unit 11 acquires part-specific information based on the acquired part image (step S162). The control unit 11 inputs the acquired part image to the trained damage recognition model 172, and acquires information relating to damage to the part in the part image (damage recognition result) (step S163).

[0178] The control unit 11 assigns a part ID to the acquired part unique information and information on part damage (step S164). The control unit 11 transmits the part unique information and information on part damage to the DB server 4 via the communication unit 13 in association with the assigned part ID (step S165). The control unit 41 of the DB server 4 receives the part ID, part unique information, and information on part damage transmitted from the image processing server 1 via the communication unit 43 (step S461). The control unit 41 stores the received part ID, part unique information, and information on part damage in the part detail DB 452 and part damage information DB 453 of the mass storage unit 45 (step S462).

[0179] According to this embodiment, by outputting both part-specific information and information regarding damage to the part, it is possible to simultaneously realize a traceability function using part-specific information (such as a fingerprint of a thing) and an appearance inspection function.

[0180] (Embodiment 8) The eighth embodiment relates to a mode of detecting registered parts that have been stored in a storage location for a certain period of time or more. Note that a description of the contents that overlap with the first to seventh embodiments will be omitted.

[0181] FIG. 31 is an explanatory diagram showing an example of a record layout of the parts unique information DB371 of the eighth embodiment. The parts unique information DB371 is stored in the mass storage unit 37 of the parts server 3. Note that description of the contents that overlap with FIG. 4 will be omitted. The parts unique information DB371 includes a storage information column. The storage information column includes a storage location column, a location details column, a storage start date and time column, and a storage period column. The storage location column stores the storage location until the part is attached to the product. The location details column stores position information such as the latitude, longitude, or shelf number of the storage location. The storage start date and time column stores date and time information when storage of the part started. The storage period column stores the storage period until the part is attached to the product (for example, six months, ten months, or one year).

[0182] FIG. 32 is an explanatory diagram showing an example of a record layout of the part details DB 452 of the eighth embodiment. The part details DB 452 is stored in the mass storage unit 45 of the DB server 4. Note that explanations of contents that overlap with FIG. 7 will be omitted. The part details DB 452 includes a storage information column and an installation information column. Note that explanations of the storage information column will be omitted as it is similar to the storage information column of the part specific information DB 371. The installation information column includes an installation robot ID column, an installation manufacturer name column, a location address column, and an installation date and time column. Explanations of the installation robot ID column, the installation manufacturer name column, and the location address column will be omitted as they are similar to FIG. 7. The installation date and time column stores information on the date and time when the part was installed on the product.

[0183] First, parts storage information is registered in advance in the parts server 3. The storage information includes the storage location (e.g., location name, longitude, latitude, or shelf information, etc.), storage start date and time, and storage period until the part is attached to the product. Specifically, the parts server 3 acquires the parts storage information. For example, the parts server 3 may receive the parts storage information input by a person in charge at the parts production factory through the input unit 34. The parts server 3 stores the part storage location, detailed location information (longitude, latitude, or shelf number, etc.), storage start date and time, and storage period included in the acquired parts storage information in the parts specific information DB 371 in association with the part ID.

[0184] Next, a process of outputting part storage information to the DB server 4 in association with the part ID will be described. The image processing server 1 acquires a part image including an object that is a feature point from the photographing device 2. As in the first embodiment, the image processing server 1 recognizes an object in the acquired part image using the trained object recognition model 171. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object. The image processing server 1 transmits the extracted area image to the part server 3.

[0185] The parts server 3 receives the area image transmitted from the image processing server 1 and extracts the feature amount of the received area image. The parts server 3 compares the feature amount of the extracted area image with the feature amount of the part image stored in the parts unique information DB 371. Note that the feature amount comparison process is the same as in the first embodiment, so a description thereof will be omitted. If the parts server 3 determines that the feature amounts of both match, it acquires the corresponding part ID and the storage information of the part corresponding to the part ID from the parts unique information DB 371.

[0186] The parts server 3 transmits the acquired part ID and part storage information to the image processing server 1. The image processing server 1 receives the part ID and part storage information transmitted from the parts server 3, and transmits the received part ID and part storage information to the DB server 4. The DB server 4 receives the part ID and part storage information transmitted from the image processing server 1. The DB server 4 stores the part storage location, detailed location information (longitude, latitude, shelf number, etc.), storage start date and time, and storage period in the parts detail DB 452 in association with the received part ID.

[0187] Next, a process for detecting registered parts that have been retained in a storage location for a certain period of time or more will be described. The image processing server 1 determines whether or not a registered part has been retained in a storage location for a certain period of time or more, for example, at predetermined intervals by batch processing. Specifically, the image processing server 1 acquires a registered part ID and storage information of a part corresponding to the part ID from the parts server 3. The image processing server 1 determines whether or not the part has been retained for a certain period of time or more (for example, six months) based on the storage start date and time and storage period included in the acquired storage information of the part. If the image processing server 1 determines that the part has been retained for a certain period of time or more, it acquires the part ID of the part.

[0188] The image processing server 1 determines whether or not the part has been attached to the product based on the acquired part ID. Specifically, the image processing server 1 determines whether or not attachment information of the part exists in the part detail DB 452 of the DB server 4 based on the part ID. The part attachment information includes the ID of the attachment robot, the manufacturer name of the attachment robot used when attaching the part to the product, the address or name of the production line where the part was attached, or the date and time when the part was attached to the product.

[0189] When the image processing server 1 determines that the attachment information of the part exists in the part details DB 452, it determines that the part is not a retained part because it has already been attached to the product. When the image processing server 1 determines that the attachment information of the part does not exist in the part details DB 452, it identifies the part as a retained target.

[0190] 33 is a flowchart showing a processing procedure when outputting part storage information in association with a part ID. The control unit 11 of the image processing server 1 acquires a part image including an object from the photographing device 2 via the communication unit 13 (step S171). The control unit 11 recognizes an object in the acquired part image using the learned object recognition model 171 (step S172). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S173). The control unit 11 transmits the extracted area image to the part server 3 via the communication unit 13 (step S174).

[0191] The control unit 31 of the parts server 3 receives the area image transmitted from the image processing server 1 through the communication unit 33 (step S371) and extracts the feature amount of the received area image (step S372). Based on the feature amount of the extracted area image, the control unit 31 acquires the corresponding part ID and the storage information of the part corresponding to the part ID from the part specific information DB 371 (step S373). The control unit 31 transmits the acquired part ID and the storage information of the part to the image processing server 1 through the communication unit 33 (step S374).

[0192] The control unit 11 of the image processing server 1 receives the part ID and part storage information transmitted from the parts server 3 via the communication unit 13 (step S175), and transmits the received part ID and part storage information to the DB server 4 (step S176). The control unit 41 of the DB server 4 receives the part ID and part storage information transmitted from the image processing server 1 via the communication unit 43 (step S471). The control unit 41 stores the part storage location, detailed location information (longitude, latitude, shelf number, etc.), storage start date and time, and storage period in the part detail DB 452 of the mass storage unit 45 in association with the received part ID (step S472), and ends the process.

[0193] 34 is a flow chart showing a processing procedure for detecting a retained part. The control unit 11 of the image processing server 1 acquires a plurality of registered part IDs and storage information of parts corresponding to each part ID from the part specific information DB 371 of the parts server 3 via the communication unit 13 (step S181). The control unit 11 acquires one part ID from the acquired plurality of part IDs (step S182). The control unit 11 determines whether the storage period (e.g., six months) has passed based on the storage start date and time included in the storage information of the part corresponding to the acquired part ID (step S183).

[0194] When the control unit 11 determines that the storage period of the part has not passed (NO in step S183), it returns to the process of step S182 and acquires the next part ID. When the control unit 11 determines that the storage period of the part has passed (YES in step S183), it determines whether or not the part has been attached to the product (step S184). Specifically, the control unit 11 transmits the part ID to the DB server 4 via the communication unit 13 and inquires whether or not the attachment information of the part already exists in the part details DB 452. The attachment information includes the ID of the attachment robot, the name of the manufacturer of the attachment robot, the location address, or the attachment date and time. When the attachment information of the part exists in the part details DB 452, the control unit 11 determines that the part has already been attached to the product. When the attachment information of the part does not exist in the part details DB 452, the control unit 11 determines that the part has not been attached to the product.

[0195] If the control unit 11 determines that the part has already been attached to the product (YES in step S184), it returns to the process of step S182 and acquires the next part ID. If the control unit 11 determines that the part has not been attached to the product (NO in step S184), it specifies the part as a retention target (step S185). The control unit 11 determines whether the part ID is the last part ID among the multiple part IDs (step S186).

[0196] If the control unit 11 determines that the part ID is not the last one (NO in step S186), it returns to the process in step S182 and acquires the next part ID. If the control unit 11 determines that the part ID is the last one (YES in step S186), it generates a report for the part to be retained (step S187) and ends the process. The report may be, for example, a PDF (Portable Document Format) file that describes the part ID to be retained.

[0197] However, the processing is not limited to the above. For example, the parts server 3 extracts parts that have been stored in a storage location for a certain period of time from the parts specific information DB 371 at predetermined intervals and transmits the parts to the image processing server 1. When the image processing server 1 determines that a part transmitted from the parts server 3 is not attached to a product, the image processing server 1 may specify the part as a retained part.

[0198] According to this embodiment, it is possible to output to the DB server 4 the storage location and storage period until the part is attached to the product in association with the part ID.

[0199] According to this embodiment, it is possible to strengthen management of parts by detecting registered parts that have been stored in a storage location for a certain period of time or more.

[0200] (Embodiment 9) The ninth embodiment relates to a form in which the part to be checked is checked against registered parts to determine whether or not a part has been replaced or the part state. Note that a description of the contents that overlap with the first to eighth embodiments will be omitted.

[0201] The part state refers to the degree of wear (deterioration) of the part as it is used. In this embodiment, an example is described in which parts are classified into three wear levels according to their wear, but other classification methods are similarly applicable. For example, wear level 1 is a normal state in which the part is not worn out. Wear level 2 is a level in which the part is not worn out very much and the part can be used continuously. Wear level 3 is a level in which the part is worn out and maintenance or replacement is recommended.

[0202] Fig. 35 is a block diagram showing an example of the configuration of the image processing server 1 of the ninth embodiment. Note that the same reference numerals are used for the contents that overlap with Fig. 2, and the description thereof will be omitted. A wear identification model 173 is stored in the large-capacity storage unit 17. The wear identification model 173 is a classifier that identifies the wear level of a part based on a part image, and is a trained model generated by machine learning.

[0203] Fig. 36 is a block diagram showing a configuration example of the DB server 4 of the ninth embodiment. The same reference numerals are used for the contents that overlap with Fig. 5, and the description thereof will be omitted. A matching result DB 454 is stored in the large-capacity storage unit 45. The matching result DB 454 stores the matching results obtained by matching the part to be matched with the registered parts.

[0204] FIG. 37 is an explanatory diagram showing an example of a record layout of the matching result DB 454. The matching result DB 454 includes a part ID column, a part image column, a replaced / not replaced column, a wear level column, and a matching date / time column. The part ID column stores a part ID for identifying a part. The part image column stores an area image of the whole or a part of the part image. The replaced / not replaced column stores information on whether or not a part has been replaced. The wear level column stores the wear level of the part. The matching date / time column stores date / time information on when the part to be matched is matched with a registered part.

[0205] First, the image processing server 1 compares the part to be compared with the registered parts. Specifically, the image processing server 1 acquires a part image of the part to be compared from the photographing device 2. As in the first embodiment, the image processing server 1 recognizes an object in the acquired part image by using the trained object recognition model 171. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object.

[0206] The image processing server 1 transmits the extracted area image to the factory server 5. The factory server 5 receives the area image transmitted from the image processing server 1. The factory server 5 performs a matching process between the received area image and an area image stored in the product DB 571. The factory server 5 transmits a matching result including a degree of match to the image processing server 1. The image processing server 1 receives the matching result transmitted from the factory server 5.

[0207] Based on the received matching result, the image processing server 1 determines whether or not the part unique information of the part exists in the product DB 571. For example, if the degree of match is equal to or greater than a predetermined threshold (e.g., 95%), the image processing server 1 determines that the part unique information exists in the product DB 571. If the degree of match is less than the predetermined threshold, the image processing server 1 determines that the part unique information does not exist in the product DB 571.

[0208] When the image processing server 1 determines that the part unique information does not exist in the product DB 571, it determines that the part has been replaced with a part from another company or a part with a different model number. When the image processing server 1 determines that the part unique information exists in the product DB 571, it determines the wear level of the part. Specifically, the image processing server 1 inputs a part image to a trained wear identification model 173 that outputs an identification result that identifies the wear level corresponding to the part image when a part image is input, and outputs an identification result that identifies the wear level corresponding to the part image. The wear level identification process will be described later.

[0209] The image processing server 1 transmits the comparison result obtained by comparing the part to be compared with the registered parts to the DB server 4. The comparison result includes the part ID of the part to be compared, the part image, the information on whether or not it has been replaced, the wear level of the part, or the comparison date and time, etc. The DB server 4 stores the comparison result transmitted from the image processing server 1 in the comparison result DB 454. Specifically, the DB server 4 associates the part ID with the part image, the information on whether or not it has been replaced, the wear level, and the comparison date and time, and stores them as one record in the comparison result DB 454. Note that if the wear level has not been determined, a blank or "-" may be stored in the wear level column.

[0210] Next, a process for identifying the wear level of a part using wear identification model 173 will be described. Fig. 38 is an explanatory diagram for explaining wear identification model 173. Wear identification model 173 is used as a program module that is part of artificial intelligence software. Wear identification model 173 is a classifier in which a neural network has been constructed that receives an input of an entire or partial area image of a part image and outputs the result of identifying the wear level corresponding to the input image.

[0211] The image processing server 1 constructs (generates) the wear identification model 173 by performing deep learning to learn the feature amounts of wear locations within an entire or partial area image of a part image. For example, the wear identification model 173 is a CNN, and has an input layer that accepts an input of a part image, an output layer that outputs a result of identifying a wear level corresponding to the part image, and an intermediate layer that has been trained by backpropagation.

[0212] The input layer has a plurality of neurons that accept input of pixel values ​​of each pixel included in the part image, and passes the input pixel values ​​to the intermediate layer. The intermediate layer has a plurality of neurons that extract features of the part image, and passes the extracted image features to the output layer. The intermediate layer is configured by alternately connecting a convolution layer that convolves the pixel values ​​of each pixel input from the input layer, and a pooling layer that maps the pixel values ​​convolved in the convolution layer, thereby compressing the pixel information of the part image while finally extracting the image features. The intermediate layer then uses a fully connected layer whose parameters have been learned by backpropagation to identify (predict) the wear level corresponding to the part image. The identification result is output to an output layer having a plurality of neurons.

[0213] The image processing server 1 acquires a plurality of combinations of training data in which part images are associated with wear levels corresponding to the part images. The training data is data in which the part images are labeled with wear levels corresponding to the part images. The image processing server 1 performs learning using the acquired training data.

[0214] The image processing server 1 inputs part images, which are training data, into an input layer, undergoes arithmetic processing in an intermediate layer, and obtains a classification result that identifies the wear level corresponding to the part image from an output layer. As shown in the figure, the classification result output from the output layer is a continuous probability value (e.g., a value ranging from "0" to "1") indicating the wear level.

[0215] The image processing server 1 compares the classification result output from the output layer with the information labeled for the part image in the training data, i.e., the correct answer value, and optimizes the parameters used in the calculation process in the intermediate layer so that the output value from the output layer approaches the correct answer value. The parameters are, for example, the weights (coupling coefficients) between neurons, the coefficients of the activation functions used in each neuron, etc. The method for optimizing the parameters is not particularly limited, but for example, the image processing server 1 optimizes various parameters using the backpropagation method.

[0216] The image processing server 1 performs the above-mentioned processing on each part image included in the training data to generate the wear identification model 173. As a result, for example, the image processing server 1 can construct a model capable of identifying the wear level corresponding to the part image by learning the wear identification model 173 using the training data.

[0217] When the image processing server 1 acquires a part image, it inputs the acquired part image to the wear identification model 173. The image processing server 1 performs a calculation process to extract features of the part image in the middle layer of the wear identification model 173. The image processing server 1 inputs the extracted features to the output layer of the wear identification model 173, and acquires as an output a recognition result that identifies the wear level corresponding to the part image.

[0218] As shown in the figure, the classification results are output for the parts image with the probability values ​​of "wear level 1 (no wear)", "wear level 2", and "wear level 3" being "0.10", "0.84", and "0.06", respectively. A predetermined threshold value may also be used to output the classification result. For example, when the image processing server 1 determines that the probability value (0.84) of "wear level 2" is equal to or greater than a predetermined threshold value (e.g., 0.80), the classification result is output with the wear level corresponding to the parts image being "wear level 2". It is also possible to output the wear level corresponding to the highest probability value from among the probability values ​​of each wear level classified by the wear classification model 173 as the classification result without using the above-mentioned threshold value.

[0219] The wear level identification process is not limited to the machine learning process described above. For example, the wear level may be identified using a local feature extraction method such as A-KAZE, SIFT, SURF, ORB, or HOG.

[0220] 39 is a flowchart showing a processing procedure for matching parts. The control unit 11 of the image processing server 1 acquires a part image of a part to be matched from the photographing device 2 via the communication unit 13 (step S191). The control unit 11 recognizes an object in the acquired part image using the trained object recognition model 171 (step S192). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S193).

[0221] The control unit 11 transmits the extracted area image to the factory server 5 via the communication unit 13 (step S194). The control unit 51 of the factory server 5 receives the area image transmitted from the image processing server 1 via the communication unit 53 (step S591). The control unit 51 performs a matching process between the received area image and an area image stored in the product DB 571 of the mass storage unit 57 (step S592). The control unit 51 transmits a matching result including a degree of match to the image processing server 1 via the communication unit 53 (step S593).

[0222] The control unit 11 of the image processing server 1 receives the matching result transmitted from the factory server 5 through the communication unit 13 (step S195). Based on the received matching result, the control unit 11 determines whether or not the part-specific information of the part exists in the product DB 571 (step S196). For example, when the degree of matching included in the matching result is equal to or greater than a predetermined threshold (e.g., 95%), the control unit 11 determines that the part-specific information exists in the product DB 571.

[0223] When control unit 11 determines that the part-specific information does not exist in product DB 571 (NO in step S196), it transitions to processing in step S198, which will be described later. When control unit 11 determines that the part-specific information exists in product DB 571 (YES in step S196), it determines the wear level of the part using trained wear identification model 173 (step S197). Specifically, control unit 11 inputs a part image to trained wear identification model 173, which outputs an identification result that identifies the wear level corresponding to a part image when a part image is input, and outputs an identification result that identifies the wear level corresponding to the part image.

[0224] The control unit 11 transmits the matching result, including the part ID, part image, replacement information, wear level, or matching date and time, to the DB server 4 via the communication unit 13 (step S198). The control unit 41 of the DB server 4 receives the matching result transmitted from the image processing server 1 via the communication unit 43 (step S491). The control unit 41 stores the matching result in the matching result DB 454 in association with the part ID (step S492), and ends the process.

[0225] According to this embodiment, by comparing the part to be compared with registered parts, it is possible to determine whether or not a part has been replaced and the part state.

[0226] According to this embodiment, by determining the part status, it is possible to utilize the determined status as prediction data for the timing of part replacement or maintenance, etc.

[0227] (Embodiment 10) The tenth embodiment relates to a form in which a registration or matching process is performed for a part in which a plurality of objects are distributed. Note that a description of the contents that overlap with the first to ninth embodiments will be omitted.

[0228] First, the process of registering a part in which a plurality of objects are distributed will be described. Fig. 40 is a flow chart showing the processing procedure for registering a part. A plurality of objects are distributed on the part in Fig. 40.

[0229] The control unit 11 of the image processing server 1 acquires a part image including at least two or more objects from the photographing device 2 via the communication unit 13 (step S01). The control unit 11 recognizes each object in the acquired part image by using the trained object recognition model 171 (step S02). Note that the object recognition process is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0230] The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from each recognized object (step S03). The control unit 11 acquires part unique information corresponding to any one of the extracted multiple area images from the product DB 571 of the factory server 5 via the communication unit 13 (step S04). The control unit 11 assigns a part ID to the acquired part unique information (step S05). The control unit 11 transmits the assigned part ID and multiple area images to the parts server 3 via the communication unit 13 (step S06). The control unit 11 transmits the assigned part ID and part unique information to the DB server 4 via the communication unit 13 (step S07).

[0231] The control unit 31 of the parts server 3 receives the part ID and the multiple area images transmitted from the image processing server 1 through the communication unit 33 (step S08). The control unit 31 stores the multiple area images in the part unique information DB 371 of the mass storage unit 37 in association with the received part ID (step S09), and ends the process. The control unit 41 of the DB server 4 receives the part ID and the part unique information transmitted from the image processing server 1 through the communication unit 43 (step S10). The control unit 41 stores the part unique information in the part detail DB 452 of the mass storage unit 45 in association with the received part ID (step S11), and ends the process.

[0232] Next, a process of matching a part in which a plurality of objects are distributed will be described. Fig. 41 is a flow chart showing a processing procedure for performing a process of matching a part. A plurality of objects are distributed on the part in Fig. 41.

[0233] The control unit 11 of the image processing server 1 acquires a part image including one or a plurality of objects from the photographing device 2 via the communication unit 13 (step S15). The control unit 11 recognizes each object in the acquired part image by using the trained object recognition model 171 (step S16). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from each recognized object (step S17).

[0234] The control unit 11 determines whether or not the part unique information can be acquired from the product DB 571 of the factory server 5 based on the extracted multiple area images via the communication unit 13 (step S18). Specifically, the control unit 11 determines that the part unique information can be acquired when the part unique information corresponding to any of the multiple area images is acquired. The control unit 11 determines that the part unique information cannot be acquired when the control unit 11 determines that the part unique information corresponding to all area images cannot be acquired.

[0235] When the control unit 11 determines that the part unique information cannot be acquired (NO in step S18), it determines that the authentication of the part has failed (step S22) and ends the process. When the control unit 11 determines that the part unique information can be acquired (YES in step S18), it performs a process of verifying the part unique information (step S19).

[0236] Specifically, the control unit 11 compares the part unique information corresponding to each area image in each area image. For example, when the control unit 11 extracts area image A, area image B, and area image C from the part image, the control unit 11 acquires the part unique information corresponding to the acquired area image A, area image B, and area image C from the product DB 571. The control unit 11 compares the part unique information corresponding to area image A with the part unique information corresponding to area image B. The control unit 11 compares the part unique information corresponding to area image A with the part unique information corresponding to area image C. The control unit 11 compares the part unique information corresponding to area image B with the part unique information corresponding to area image C.

[0237] The control unit 11 determines whether the number of matches is equal to or greater than a predetermined threshold (e.g., 2) (step S20). If the control unit 11 determines that the number of matches is equal to or greater than the predetermined threshold (YES in step S20), it determines that the authentication of the part has been successful (step S21) and ends the process. If the control unit 11 determines that the number of matches is less than the predetermined threshold (NO in step S20), it transitions to the process of step S22.

[0238] According to this embodiment, it is possible to perform registration or matching processing for a part in which a plurality of objects are distributed.

[0239] According to this embodiment, by registering a plurality of area images, it is possible to improve the accuracy of part authentication.

[0240] <Variation 1> The registration or matching process for a cylindrical part will now be described. Fig. 42 is an explanatory diagram showing an example of a cylindrical object having a plurality of objects distributed on its end face.

[0241] 42A is an explanatory diagram showing an example of a captured image of a cylinder end surface for registration of a cylindrical object. As shown in the figure, three objects including 13a, 13b, and 13c are arranged at 120 degrees on the cylinder end surface. The image processing server 1 acquires a captured image of the cylinder end surface including the objects 13a, 13b, and 13c from the photographing device 2. The image processing server 1 recognizes the objects 13a, 13b, and 13c in the acquired captured image of the cylinder end surface.

[0242] The image processing server 1 extracts area images of a predetermined size at a predetermined offset position from each object, similar to the above-mentioned process. The image processing server 1 acquires part unique information corresponding to any one of the extracted area images from the product DB 571 of the factory server 5. The image processing server 1 assigns a part ID to the acquired part unique information. The image processing server 1 registers the part in the part server 3 and the DB server 4 in association with the assigned part ID. Note that the registration process is similar to the above-mentioned process, and therefore a description thereof will be omitted.

[0243] FIG. 42B is an explanatory diagram showing an example of a captured image of a tube end surface for matching of a cylindrical object. Depending on the capturing position or the shape of the part, it may not be possible to capture the same part every time. For example, in the case of a cylindrical object, if the cylindrical object rotates, it may not be possible to identify the position and capture the same part every time. Also, when the entire tube end surface of a cylindrical object is captured, the area of ​​the effective edge portion becomes small. Furthermore, when a large portion of the edge portion is captured, there is an area that is partially hidden.

[0244] As shown in the figure, when the edge of the cylinder end face is photographed in a large size, the photographed image of the cylinder end face includes the objects 13a and 13b, but does not include the object 13c. The image processing server 1 acquires the photographed image of the cylinder end face including the objects 13a and 13b from the photographing device 2. The image processing server 1 recognizes the objects 13a and 13b in the acquired photographed image of the cylinder end face. The image processing server 1 determines whether or not it is possible to acquire part specific information corresponding to the area image extracted based on the recognized objects 13a and 13b from the product DB 571 of the factory server 5, in the same manner as in the above-mentioned process.

[0245] When the image processing server 1 determines that it cannot acquire both the part unique information (first part unique information) corresponding to the area image extracted based on the 13a object and the part unique information (second part unique information) corresponding to the area image extracted based on the 13b object, it determines that the authentication of the part has failed because the part does not exist in the product DB 571. When the image processing server 1 acquires only either the first part unique information or the second part unique information, it determines that the authentication of the part has failed.

[0246] When the image processing server 1 acquires both the first part unique information and the second part unique information, the image processing server 1 compares the first part unique information with the second part unique information. When the image processing server 1 determines that the first part unique information and the second part unique information match, the image processing server 1 determines that the authentication of the part has been successful.

[0247] According to this modified example, for a cylindrical object for which it is difficult to photograph the same location each time, matching is performed based on a plurality of objects, thereby making it possible to improve authentication accuracy.

[0248] (Embodiment 11) The eleventh embodiment relates to a mode in which a part is photographed based on the parameters of the photographing conditions when the reference image was photographed. Note that a description of the contents that overlap with the first to tenth embodiments will be omitted.

[0249] A reference image including an object that is a feature point is captured by, for example, the photographing device 2. The photographing device that captures the reference image is not limited to the photographing device 2, and may be, for example, an external photographing device. A predetermined allowable range set according to parameters of the photographing conditions under which the reference image was captured is stored in advance in the storage unit 12 of the image processing server 1. The parameters of the photographing conditions include a similarity score, a frequency component distribution score, a variability score, a luminance score, or a hue score. The allowable range is a range within which the value of each parameter can be adjusted.

[0250] The similarity score is a numerical value that indicates the degree of similarity between the reference image and the image to be photographed. The higher the similarity score, the higher the similarity between the reference image and the image to be photographed. The frequency component distribution score is a numerical value that indicates the frequency components obtained by transforming the image data, for example, by a fast Fourier transform (FFT). The frequency component distribution includes the distribution (proportion) of low-frequency components, mid-frequency components, and high-frequency components. The variability score is a standard deviation that indicates the magnitude of variability. By using the frequency component distribution score and the variability score, the focus of the image can be confirmed. The luminance score is a numerical value that indicates the brightness of each color component R (red), G (green), and B (blue), and includes a MAX luminance, MIN luminance, or average luminance score. The hue score is a numerical value that indicates the color tone.

[0251] The image processing server 1 acquires a predetermined allowable range of the parameter from the storage unit 12. When the photographing device 2 photographs a part to be registered or matched, it transmits an image of the part being photographed to the image processing server 1. The image processing server 1 receives the part image transmitted from the photographing device 2 and determines whether or not the parameters of the received part image are within a predetermined allowable range. If the image processing server 1 determines that the parameters are within the predetermined allowable range, it transmits an instruction to photograph the part to the photographing device 2.

[0252] The photographing device 2 receives the photographing instruction transmitted from the image processing server 1, and photographs the parts in accordance with the received photographing instruction. The photographing device 2 transmits the photographed part images to the image processing server 1. Thereafter, the image processing server 1 performs part registration or matching processing using the part images transmitted from the photographing device 2, as in the first to tenth embodiments.

[0253] In the present embodiment, an example has been described in which the process of determining whether the parameters of a part image are within a predetermined allowable range is performed on the image processing server 1 side, but the present invention is not limited to this, and the process may be performed on the photographing device 2 side. Specifically, the photographing device 2 acquires a predetermined allowable range set according to the parameters of the photographing conditions from a storage unit or the image processing server 1. When the photographing device 2 determines that the parameters of the part image being photographed are within the predetermined allowable range, it photographs the part. The photographing device 2 transmits the photographed part image to the image processing server 1.

[0254] According to this embodiment, an optimal part image can be acquired by photographing a part based on the parameters of the photographing conditions when the reference image was photographed.

[0255] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0256] 1. Information processing device (image processing server) 11 Control section 12 Storage section 13. Communications Department 14 Input section 15 Display section 16 Reading unit 17 Mass storage 171 Object Recognition Model 172 Damage Recognition Model 173 Attrition Identification Model 71a Region candidate extraction part 71b Classification section 1a Removable storage media 1b Semiconductor memory 1P control program 2. Imaging Device 2a Imaging device 2b Imaging device 3. Information processing equipment (parts server) 31 Control Unit 32 Storage section 33 Communications Department 34 Input section 35 Display section 36 Reading unit 37 Mass storage 371 Parts specific information DB 3a Removable storage media 3b Semiconductor memory 3P Control Program 4 Database device (DB server) 41 Control section 42 Storage section 43 Communications Department 44 Reading unit 45 Mass storage 451 Product management DB 452 Parts Details DB 453 Parts Damage Information DB 454 Matching result DB 4a Removable storage media 4b Semiconductor memory 4P control program 5. Information processing equipment (factory server) 51 Control section 52 Storage section 53 Communications Department 54 Input section 55 Display section 56 Reading unit 57 Mass storage 571 Product DB 5a Portable storage media 5b Semiconductor memory 5P Control Program 6. Information processing device (deletion server) 61 Control section 62 Storage section 63 Communications Department 64 Input section 65 Display section 66 Reading unit 67 Mass storage 671 Parts Management DB 672 Production planning database 68 Clock section 6a Removable storage media 6b Semiconductor memory 6P Control Program

Claims

1. Acquire a part image of a part including a plurality of feature point objects; Recognizing each object in the acquired part image; Extracting a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects; Acquire part-specific information corresponding to each of the extracted area images; determining a number of matches of part specific information corresponding to each of the area images; Authenticating the part based on the number of matches of the part-specific information and a predetermined threshold. A program that causes a computer to carry out processing.

2. inputting the acquired part image into an image recognition model that has been trained to identify the position and type of an object in the part image when the part image is input, and outputting the position and type of the object in the part image; Refer to a storage unit in which offset positions are stored for each type of object, read out the offset position corresponding to the output type, A region image corresponding to the offset position read from the output position of the object is extracted. The program according to claim 1, which causes a process to be executed.

3. Obtaining a part ID corresponding to the part unique information; The acquired part ID and the part specific information are output.

3. The program according to claim 1 or 2, which causes a process to be executed.

4. Allocating a part ID to the part unique information; Output the assigned part ID and the part specific information.

3. The program according to claim 1 or 2, which causes a process to be executed.

5. Acquire a part ID corresponding to part unique information of the region image; Acquire a product ID of a product using a part corresponding to the acquired part ID; The acquired product ID is output to a database device in association with the part ID.

5. A program for executing a process according to claim 1.

6. A part image of the part is acquired after a light for illuminating the part is turned on.

6. A program for executing a process according to claim 1.

7. When a distortion or tilt is detected in the part image, the offset position or the part image is corrected according to the distortion or tilt.

7. A program for executing a process according to claim 1.

8. Acquire a first photographed image photographed by a first photographing device; determining whether the part image is present in the acquired first captured image; When it is determined that the part image exists, a second photographing device having a higher magnification than the first photographing device is used to obtain a second photographing image that is a part of the part image; The acquired second captured image is input into the image recognition model. The program according to claim 2, which causes the program to execute processing.

9. The storage location and storage period until the part is attached to the product are output to a database device in association with the part ID of the part.

9. A program for executing a process according to claim 1.

10. By comparing the part to be compared with the registered parts, the presence or absence of replacement of the part to be compared and the part status are determined.

10. A program for executing a process according to claim 1.

11. The parts are distributed with multiple objects, acquiring a part image including at least two objects among the plurality of objects; Recognizing each object in the acquired part image; Extracting an area image of a predetermined size at a predetermined offset position from each recognized object; Acquire part-specific information corresponding to any one of the extracted multiple area images; Allocating a part ID to the acquired part unique information; Output the assigned part ID and the part specific information.

11. The program according to claim 1, which executes a process.

12. the part is a cylindrical object, and a plurality of objects are distributed on an end surface of the part; Obtaining a captured image of the end face of the tube including one or more objects 12. A program for executing a process according to claim 1.

13. Extracting a plurality of area images of a predetermined size at a predetermined offset position from an object in the photographed image of the tube end surface; The part is authenticated based on part-specific information corresponding to each of the extracted area images. The program according to claim 12, which causes processing to be executed.

14. A predetermined allowable range is obtained according to parameters of the photographing conditions when the reference image including the object as the feature point was photographed; When it is determined that the parameters of the part image to be photographed including the object are within the predetermined allowable range, an instruction to photograph the part is output.

14. A program for executing a process according to any one of claims 1 to 13.

15. Acquire a part image of a part including a plurality of feature point objects; Recognizing each object in the acquired part image; Extracting a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects; Acquire part-specific information corresponding to each of the extracted area images; determining a number of matches of part specific information corresponding to each of the area images; Authenticating the part based on the number of matches of the part-specific information and a predetermined threshold. An information processing method for causing a computer to execute processing.

16. a first acquisition unit that acquires a part image of a part including objects that are feature points; a recognition unit that recognizes each object in the acquired part image; an extracting unit that extracts a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects; a second acquisition unit that acquires part specific information corresponding to each of the extracted area images; a determination unit for determining a number of matches of part unique information corresponding to each of the area images; an authentication unit that authenticates the part based on the number of matches of the part unique information and a predetermined threshold value; An information processing device comprising:

17. a first acquisition unit that acquires a part image of a part including objects that are feature points; a recognition unit that recognizes each object in the acquired part image; an extracting unit that extracts a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects; a second acquisition unit that acquires part specific information corresponding to each of the extracted area images; a determination unit for determining a number of matches of part unique information corresponding to each of the area images; an authentication unit that authenticates the part based on the number of matches of the part unique information and a predetermined threshold value; an output unit that outputs the acquired part specific information; The recognition unit recognizes damage to a part in the part image acquired by the first acquisition unit, The output unit outputs information regarding the damage to the part recognized by the recognition unit.

23. An information processing apparatus comprising:

Images