Information processing apparatus, information processing system, information processing method, and program

Abstract

To accurately identify the type of an object.SOLUTION: An information processing apparatus has: first identification means that inputs a first image obtained by picking up an object according to a first photographing condition to a first model to identify a photographing condition suitable for identification of the type of the object; and second identification means that inputs a second image obtained by picking up the object according to a second photographing condition identified by the first identification means to a second model to identify the type. The second identification means uses, of a plurality of models, a model linked to the second photographing condition as the second model.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to object identification. [Background technology]

[0002] Japan's fish catch has been declining year by year. One of the reasons for this is the decline in fishery resources. To halt the decline in fish catch, the government is trying to manage catches and strengthen fishery resource management. To grasp catch volumes, the government requires fishermen to report detailed catch records, but fishermen currently do not have the time to devote to surveying catch volumes. This has led to a demand for an image analysis system for monitoring catches. To realize an image analysis system for monitoring catches, a model for classifying fish species is required. Patent Document 1 discloses a method for identifying fish species using deep learning. Furthermore, Patent Document 2 discloses a method for recognition by applying an appropriate recognition method depending on the shooting conditions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-135986 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-239871 [Non-patent literature]

[0004] [Non-Patent Document 1] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Renra.NIPS2015 [Non-patent document 2] Yann LeCun, Leon Bottou, Yoshua Bengio, Partick Haffner. “Gradient-Based Learning Applied to Document Recognition”. In Proceedings of the IEEE, 86, pp. 2278-2324, 1998. Summary of the Invention [Problem to be solved by the invention]

[0005] However, because the brightness and color of fish bodies vary depending on the fish species, it can be difficult to identify fish species from images taken under fixed shooting conditions. For example, blue fish such as horse mackerel and saury tend to reflect light, and if the fish is illuminated by a light source from the front, it will shine and be impossible to identify correctly. Also, if the color of the stand on which the fish is placed is similar to the color of the fish itself, it will be difficult to distinguish the fish from the image, making it difficult to identify the fish species.

[0006] The present invention has been made in view of the above problem, and has as its object to identify the type of an object with high accuracy. [Means for solving the problem]

[0007] The information processing device of the present invention comprises a first identification means for identifying shooting conditions suitable for identifying the type of object by inputting a first image of the object taken under first shooting conditions into a first model, and a second identification means for identifying the type by inputting a second image of the object taken under second shooting conditions identified by the first identification means into a second model, wherein the second identification means uses a model linked to the second shooting conditions from among a plurality of models as the second model. [Effects of the Invention]

[0008] According to the present invention, the type of an object can be identified with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an information processing system. [Figure 3] FIG. 2 is a diagram showing the flow of data used in the information processing system. [Figure 4] 10 is a flowchart showing a discrimination process. [Figure 5] FIG. 10 is a diagram illustrating an example of imaging conditions. [Figure 6] 10 is a flowchart showing a learning process. [Figure 7] FIG. 10 is a diagram illustrating data generated by a learning process. [Figure 8] FIG. 1 illustrates an example of a learning system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0011] FIG. 1 shows an example of the overall configuration of an information processing system according to this embodiment. The information processing system includes a belt conveyor 104 that transports objects 105 and 106, imaging devices 101 and 102 that capture images of the objects 105 and 106, an information processing device 100 that performs identification, and an automatic sorting machine 107 that sorts the objects 105 and 106. The imaging devices 101 and 102 capture images of the objects 105 and 106 on the belt surface of the belt conveyor 104 and transmit the images to the information processing device 100. The imaging device 102 is disposed downstream of the imaging device 101 in the conveying direction of the belt conveyor 104. A lighting device 103 is also provided as lighting equipment for the imaging device 102. The lighting device 103 includes a light source that illuminates objects on the belt surface from the front at the imaging position of the imaging device 102, and a backlight incorporated in the belt conveyor 104 to illuminate objects on the belt surface from the back. The belt surface is translucent. The automatic sorting machine 107 is disposed downstream of the imaging device 102 in the conveying direction of the belt conveyor 104, and sorts objects 105 and 106 traveling on the belt conveyor 104 into one of a plurality of sorting boxes 108 to 110 according to the identification result. The information processing device 100, the imaging devices 101 and 102, and the lighting device 103 are connected to each other via a network 211 (FIG. 2) so as to be able to communicate with each other. The information processing device 100 outputs the identification result to the automatic sorting machine 107 via an output I / F 206 (FIG. 2). In the following description, the objects are described as being fish, but fish is merely an example of an object as the object.

[0012] The information processing device 100 receives from the imaging device 101 an image captured under predetermined imaging conditions at the imaging position of the imaging device 101 and detects a target fish area from the received image. Then, the image of the fish area is input into a first model that has been trained in advance, thereby identifying imaging conditions suitable for identifying the type of target fish. The information processing device 100 also identifies a second model associated with the identified imaging conditions. The information processing device 100 controls the camera settings of the imaging device 102 and the lighting settings of the lighting device 103 to be changed according to the identified imaging conditions. The information processing device 100 then receives from the imaging device 102 an image captured when the target fish travels on the belt conveyor 104 and arrives at the imaging position of the imaging device 102. The information processing device 100 predicts, based on the transport speed of the belt conveyor 104 and the distance between the imaging devices 101 and 102, when the same fish as the one detected in the image of the imaging device 101 will arrive at the imaging position of the imaging device 102. The information processing device 100 may receive video from the imaging device 102 and acquire frame images at predicted times from the received video. The information processing device 100 detects a fish area from the image received from the imaging device 102 and inputs the image of the fish area into a pre-trained second model to estimate the species of the target fish. The information processing device 100 controls the automatic sorting machine 107 to sort the target fish into sorting boxes 108-110 according to the estimated fish species.

[0013] Here, the fish serving as the target object in this embodiment have different characteristics, such as surface gloss, surface color, surface smoothness, shape, and size, depending on the fish species. Therefore, it is difficult to accurately identify all fish species from images captured under fixed imaging conditions. For example, blue fish such as horse mackerel and saury tend to reflect light, so when photographed from the front with a frontal light source, the fish body will shine. In such cases, adjusting the installation angle of the lighting device and imaging device so that the reflection of the illumination light does not directly enter the imaging device can make it easier to identify the fish species. Furthermore, if the color of the fish body surface is similar to the color of the conveyor belt, it is difficult to distinguish the fish body from the original. In such cases, using a conveyor belt backlight and changing the backlight color to match the color of the fish body surface can make it easier to identify the fish species. Furthermore, for fish with an uneven surface, such as a rockfish, it is easier to identify the fish species by shining light from an angle to create a shadow, or for flat fish, shining light from the front to illuminate the entire fish body. As described above, it is believed that the ability to identify fish species can be improved by using images captured under shooting conditions suitable for identifying fish species. Therefore, in this embodiment, the first model is used to identify shooting conditions suitable for identifying the target fish species.

[0014] Next, an overview of the photographing conditions for the object to be identified using the first model will be described with reference to FIG. 5. In this embodiment, as shown in FIG. 5, various photographing parameter settings are defined for each of a plurality of photographing conditions. Note that while FIG. 5 shows photographing conditions A to D, photographing conditions A to D are merely examples, and there may be additional photographing conditions. The photographing parameter settings include camera settings for the image capture device 102 and lighting settings for the lighting device 103. The camera settings include, for example, settings related to the photographing position, such as whether to photograph from the front or from a 30-degree angle, and settings related to camera parameters such as shutter speed. The lighting settings include, for example, settings related to the lighting position, such as whether to illuminate from the front or from a 45-degree angle, settings related to light intensity, and settings related to the backlight (ON / OFF, light intensity, color). Note that the photographing parameter settings are not limited to these. For example, settings such as the type of light source and color temperature may also be included. For example, settings such as aperture and focus may also be included. Furthermore, settings such as the conveying speed of the belt conveyor 104 may also be included. Furthermore, there may be unused combinations of settings between the photographing parameters. For example, a combination of low light intensity and a fast shutter speed is not used because it results in insufficient exposure. The camera settings also include zoom magnification settings, such as zooming in on the target fish if the fish area in the image does not meet a predetermined size. Learning is performed to determine which of the multiple shooting conditions A to D shown in Figure 5 is suitable for identifying fish species. The learning method will be described later.

[0015] In this embodiment, classification is performed in two stages as described below. In the first stage, the photographing conditions suitable for identifying fish species are identified, and in the second stage, the fish species are identified using images captured under the photographing conditions identified in the first stage. In the first stage, a first model, which is a classification model for performing the first stage, is used to identify photographing conditions suitable for identifying the type of fish appearing in an image (first image) captured under fixed photographing conditions (first photographing conditions). The second model is a classification model for performing the second stage, which is trained for each photographing condition and stored in association with the photographing condition. In this embodiment, since there are multiple photographing conditions, there are also multiple second models. In the second stage, the second model, which is associated with the photographing conditions identified in the first stage, is used to identify the type of fish appearing in an image (second image) captured under the photographing conditions (second photographing conditions) determined in the first stage. As described above, in the first stage of classification, classification is performed under shooting conditions suitable for classifying fish species, and in the second stage of classification, fish species are classified using images captured under shooting conditions suitable for each fish species. This improves the accuracy of fish species classification. Furthermore, in the first stage of classification, fish species groups are classified under shooting conditions suitable for classifying fish species, and in the second stage of classification, classification is performed within groups of fish species with the same suitable shooting conditions. This makes it possible to reduce the number of classes to be classified, and shorten the time required to classify fish species. The learning methods for the first model and the second model will be described later.

[0016] Next, the hardware configuration of the information processing system according to this embodiment will be described. Fig. 2 shows an example of the hardware configuration of the information processing system according to this embodiment. The information processing device 100 is composed of a CPU 201, a ROM 202, a RAM 203, an external storage device 204, an input I / F 205, an output I / F 206, and a communication I / F 207. These components are connected to each other via a system bus 208 so as to be able to communicate with each other.

[0017] A CPU (Central Processing Unit) 201 controls the entire information processing device 100. A ROM (Read Only Memory) 202 stores programs and parameters that do not require change. A RAM (Random Access Memory) 203 temporarily stores data supplied from an external device or the like. An external storage device 204 is a storage device such as a hard disk or memory card that is fixedly installed in the information processing device 100. The external storage device 204 may include an optical disk such as a flexible disk (FD) or a compact disk (CD), a magnetic or optical card, an IC card, a memory card, or the like that is detachable from the information processing device 100. The functions and processes of the information processing device 100, which will be described later, are realized by the CPU 201 reading and executing programs stored in the ROM 202 or the like.

[0018] The input I / F 205 is an interface with an input device 209 such as a pointing device or keyboard that receives user operations and inputs data. The output device I / F 206 is an interface with a monitor 210 that displays data held by the information processing device 100 and data supplied thereto, and with an automatic sorting machine 107 that automatically sorts fish based on the identification results. The communication I / F 207 is connected to a network 211. The CPU 201 transmits and receives data to and from the imaging devices 101 and 102 and the lighting device 103 via the network 211. The imaging devices 101 and 102 and the lighting device 103 have a communication function and are connected to a network 211. The belt conveyor 104 may also be connected to the network 211. In this case, the information processing device 100 may control the start / end of conveyance and the conveyance speed of the belt conveyor 104.

[0019] Fig. 3 shows an example of the flow of data used in the information processing system according to this embodiment. In Fig. 3, the information processing device 100 functions as a detection unit 301 that detects fish regions from an image and a classification unit 302 that performs classification using an image of the fish region. The external storage device 204 of the information processing device 100 stores a first model and a second model used by the classification unit 302. The second model is prepared for each of a plurality of shooting conditions. The following will explain the flow of data.

[0020] First, the information processing device 100 acquires the captured image 303 from the image capturing devices 101 and 102 via the network 211 and the communication I / F 207 . The detection unit 301 analyzes the acquired captured image 303 and detects the area of ​​fish in the image. The detection unit 301 uses, for example, an object detection method using CNN (Convolutional Neural Networks), such as Faster R-CNN described in Non-Patent Document 1. Faster R-CNN can detect multiple classes of objects, such as people and cars. When using Faster R-CNN, learning is performed using a large number of fish images collected in advance so that it can detect fish.

[0021] The identification unit 302 identifies the fish region extracted by the detection unit 301. When the target captured image 303 is acquired from the imaging device 101, the identification unit 302 reads out a first model 304 from the external storage device 204 and identifies the photographing conditions using the first model 304. Furthermore, the identification unit 302 identifies a second model 307 associated with the identified photographing conditions 306 from among a plurality of second models stored in the external storage device 204. Thereafter, the information processing device 100 transmits the identified photographing conditions 306 to the imaging device 102 and the lighting device 103 via the communication I / F 207. The imaging device 102 and the lighting device 103 adjust various settings based on the received photographing conditions 306 and then capture an image.

[0022] On the other hand, if the target captured image 303 has been acquired from the imaging device 102, the classification unit 302 reads the second model 307 identified above from the external storage device 204, classifies the fish species using the second model 307, and acquires a classification result 305. Thereafter, the information processing device 100 notifies the automatic sorting machine 107 of the classification result 305 via the output I / F 206. The automatic sorting machine 107 performs sorting in accordance with the classification result 305. Note that the classification unit 302 may be configured to acquire the shooting conditions used when capturing the image together with the captured image 303 from the imaging device 102, and use the second model associated with the acquired shooting conditions.

[0023] Next, the classification process executed by the information processing device 100 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the classification process. The flowchart in Fig. 4 is implemented by the CPU 201 loading a program stored in the ROM 202 or the like into the RAM 203 and executing it. The flowchart in Fig. 4 starts when the input device 209 is operated to operate the belt conveyor 104 and a fish is placed on the belt conveyor 104. Each process (step) in the flowchart will be described below with an S (step) added to the beginning of each reference number.

[0024] First, in step S401 , the CPU 201 acquires a first image from the imaging device 101 via the communication I / F 207 . Next, in S402, the CPU 201 detects a fish region from the first image acquired in S401. Note that the flow when one fish region is detected will be described here. If multiple fish regions are detected, the subsequent processes from S403 to S409 can be repeated, and the description thereof will be omitted here. Next, in S403, the CPU 201 inputs the image of the fish region detected in S402 into the first model 304, and identifies the shooting conditions 306 and second model 307 that are suitable for identifying the fish species. The first model 304 is a model that identifies which shooting conditions are suitable for identifying the type of fish detected in S402. The CPU 201 uses the first model 304 to identify a class that corresponds to the shooting conditions that are suitable for identifying the fish species in the input image. Specifically, class classification is performed for shooting conditions A to D, etc., shown in FIG. 5. The second model 307 is a model that identifies the fish species identified under each shooting condition, and one second model is linked to each shooting condition.

[0025] Next, in S404, CPU 201 compares the shooting conditions 306 identified in S403 with the shooting conditions used when the first image was captured, and determines whether they are the same. If CPU 201 determines that the shooting conditions are the same, the first image can be used as the second image as is, so S405 to S407 are skipped and processing proceeds to S408. On the other hand, if CPU 201 determines that the shooting conditions are different, processing proceeds to S405. In S405, the CPU 201 transmits the shooting conditions 306 identified in S403 to the image capturing device 102 and the lighting device 103 via the communication I / F 207. The image capturing device 102 and the lighting device 103 change their settings in accordance with the received shooting conditions 306. Next, in S406, the CPU 201 acquires a second image captured under the shooting conditions 306 from the imaging device 102 via the communication I / F 207 at the timing when the same fish as the fish detected in S402 arrives at the shooting position of the imaging device 102. Next, in S407, the CPU 201 detects a fish region from the second image acquired in S406. Next, in S408, the CPU 201 inputs the image of the fish region detected in S407 into the second model 307 specified in S403 to identify the fish species. Note that if the photographing conditions 306 specified in S403 are the same as the photographing conditions used when capturing the first image, the image of the fish region detected in S402 is input. Next, in S409, the CPU 201 notifies the automatic sorter 107 of the identified fish species via the output I / F 206. The automatic sorter 107 sorts the fish detected in S402 according to the notified fish species. After that, the series of identification processes shown in the flowchart ends.

[0026] According to the classification process of this embodiment, images captured under suitable photographing conditions for each fish species are used, thereby improving the accuracy of fish species classification. Furthermore, because suitable photographing conditions are used to classify fish within the same fish species group, the number of classes to be classified can be reduced. Therefore, the time required for fish species classification can be shortened.

[0027] Next, the learning process in the learning phase of the information processing device 100 according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the learning process. The flowchart in Fig. 6 is implemented by the CPU 201 loading a program stored in the ROM 202 or the like into the RAM 203 and executing it. Fig. 7 shows data generated by the learning process in Fig. 6. First, as training data for learning, a plurality of captured images are prepared by capturing images of a plurality of types of fish under different photographing conditions using the imaging devices 101 and 102 according to this embodiment. Each captured image is labeled with the photographing conditions and fish species used when capturing the image as ground truth data. First, in S601, the CPU 201 uses a DNN (deep neural network) to train the second model. Details of the training method will be described later. This training is performed for each of the shooting conditions in FIG. 5. The CPU 201 then trains each fish species to create a separate class, and determines the discrimination accuracy for each fish species. Once training for all shooting conditions is complete, the CPU 201 tallies the discrimination accuracy for each shooting condition for each fish species. The CPU 201 then determines the shooting conditions suitable for discrimination for each fish species based on the tallying results.

[0028] FIG. 7(a) shows an example of the results of tallying the classification accuracy for each shooting condition by fish species. FIG. 7(a) shows the classification accuracy for shooting conditions A to D for horse mackerel, mackerel, squid, thresher, red sea bream, and rockfish, respectively. Note that shooting conditions A to D in FIGS. 7(a) to 7(c) are the same as shooting conditions A to D shown in FIG. 5. Here, the CPU 201 determines the shooting condition most suitable for classification as the shooting condition with the highest classification accuracy. Note that this does not necessarily mean that the classification accuracy is higher than the others. The star marks (★) in FIG. 7(a) indicate the shooting condition with the highest classification accuracy. For example, in the case of horse mackerel, the classification accuracy for shooting conditions A to D is 32%, 47%, 66%, and 95%, so shooting condition D is determined. Similarly, in the case of mackerel, the classification accuracy for shooting conditions A to D is 18%, 33%, 52%, and 94%, so shooting condition D is determined. For example, in the case of squid, the identification accuracy rates for imaging conditions A to D are 99%, 75%, 83%, and 67%, respectively, so imaging condition A is selected. Similarly, imaging conditions suitable for identification are determined for thresher, red sea bream, and rockfish based on the identification accuracy rates for each imaging condition. FIG. 7(b) shows an example of a list of fish species suitable for identification under each imaging condition A to D. As shown in FIG. 7(b), fish species can be grouped according to the imaging conditions suitable for their identification. As a result, in S403 of FIG. 4, fish can be classified into the same number of groups as the number of imaging conditions. In the example of FIG. 7(b), horse mackerel, mackerel, and saury are classified into the same group as fish species suitable for identification under imaging condition D.

[0029] Next, in S602, the CPU 201 uses DNN to train a first model. This training is performed using training data for one of the multiple shooting conditions (for example, shooting condition A). This training is performed so that fish species in groups with the same shooting condition are classified into the same class, and fish species in groups with different shooting conditions are classified into different classes. The model trained in this step is used as the first model in the identification process of FIG. 4. Furthermore, the shooting condition of the first image acquired in S401 is the shooting condition used in the training in this step (here, shooting condition A). In this way, a first model that identifies shooting conditions suitable for identifying fish species is generated from the image captured under shooting condition A. Next, in S603, the CPU 201 re-learns the second model learned in S601. The learning in S601 was performed to identify all fish species. In this step, the learning is performed for each shooting condition to identify fish species that belong to the same shooting condition. For example, using the learning data for shooting condition A, learning is performed so that fish determined to belong to shooting condition A in S601 are classified into separate classes for each fish species. If shooting condition A is identified in S403 in the classification process of FIG. 4, the model learned in this step is used as the second model associated with shooting condition A. The second model is similarly re-learned for shooting conditions B to D. The fish species to be classified by the second model differ depending on the associated shooting condition. In this way, a second model for each shooting condition is generated.

[0030] In this embodiment, the shooting condition for the first image acquired in S401 is shooting condition A. Here, the CPU 201 may perform learning to determine which shooting condition is best for capturing the first image. For example, learning data for shooting conditions A to D is used to perform learning for each shooting condition in S602. Then, the classification accuracy is calculated for each shooting condition. Once learning for all shooting conditions is complete, the CPU 201 determines the shooting condition suitable for classification based on the classification accuracy for each shooting condition. FIG. 7C shows an example of classification accuracy for each shooting condition. Here, the CPU 201 determines the shooting condition suitable for classification as the shooting condition with the highest classification accuracy. Note that this is not limited to the case where the classification accuracy is higher than the others. The star (★) in FIG. 7C indicates the shooting condition with the highest classification accuracy, and in the example of FIG. 7C, it is shooting condition C. This shows that it is optimal to train the first model using images captured under shooting condition C. In this case, in the identification process of FIG. 4, a first image captured under shooting condition C is acquired in S401, and a first model trained using the image captured under shooting condition C is used in S403.

[0031] Fig. 8 shows an example of the configuration of a learning system used for learning. Below, we will explain a learning method when the learning system is configured with a DNN (deep neural network). In this embodiment, for example, LeNet described in Non-Patent Document 2 is used as the neural network.

[0032] In FIG. 8, an image 81 is input to a DNN 82, which outputs a vector 83 whose number of dimensions is the number of fish species to be identified in the cases of S601 and S603, or the number of photographing conditions in the case of S602. Each element of vector 83 is a probability between 0 and 1, and the total value is 1. A loss function 84 is then calculated using the correct vector of image 81. The correct vector is a vector in which only the correct element is 1, and the other elements are 0. The cross-entropy error H(p, q) in the following formula (1) is used as the loss function 84.

[0033]

number

[0034] In the above formula (1), p is the correct vector of the image 81, and q is the vector 83 output from the DNN 82. x is the number of dimensions, that is, the number of fish species or the number of shooting conditions. Learning is performed by updating the weights of the DNN 82 using the calculated value of the loss function 84 through error propagation. The first model and the second model are learned using the above learning system.

[0035] According to the information processing system of this embodiment as described above, the first stage of classification determines the photographing conditions suitable for identifying fish species, and the second stage of classification uses images captured under those photographing conditions to identify the fish species. In the second stage of classification, classification is performed using images captured under photographing conditions suitable for each fish species, thereby improving the accuracy of fish species classification compared to when images captured under fixed photographing conditions are used. Furthermore, in the second stage of classification, the suitable photographing conditions are used to identify fish within the same fish species group, so the number of classes to be classified can be reduced. Therefore, the time required for fish species classification can be shortened.

[0036] Although the present embodiment has been described as being applied to the identification of fish species, it can also be applied to the identification of various other objects, such as the identification of the types of parts on a conveyor belt in a factory assembly line. Parts have different characteristics, such as glossiness, texture, shape, and size. In the first stage of identification, appropriate shooting conditions are identified according to the above-mentioned part characteristics, and in the second stage of identification, the type of part is identified using images captured under the identified shooting conditions. This makes it possible to identify the type of part with high accuracy.

[0037] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0038] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0039] Modifications of the above embodiment will now be described. As a first modification, instead of transporting the objects 105 and 106 on the belt conveyor 104 and taking an image for identification of the first stage and then taking an image for identification of the second stage at different points, the images for identification may be taken at the same point under different imaging conditions. In this case, the belt conveyor 104 does not need to be used. As a second variant, images may be taken at multiple points on the belt conveyor 104 for the number of shooting conditions, and then an image taken under shooting conditions suitable for classification may be selected from those images to perform the second stage of classification.

[0040] As a third modification, objects 105 and 106 may be arranged in an aligned state on belt conveyor 104 so that the number of objects appearing in the first image and the second image is limited. As a fourth modification, the learning process shown in Fig. 6 may be executed by another device that performs learning. In this case, the first model and the second model used in the classification process may be acquired from the other device. [Explanation of symbols]

[0041] 100: information processing device, 101, 102: imaging device, 103: lighting device, 104: belt conveyor, 105, 106: object, 107: automatic sorting machine

Claims

1. a first identification means for inputting a first image of an object captured under a first imaging condition into a first model, and thereby identifying an imaging condition suitable for identifying the type of the object; a second identification means for identifying the type of the object by inputting a second image of the object captured under the second photographing condition identified by the first identification means into a second model; and The information processing device is characterized in that the second identification means uses, as the second model, a model associated with the second shooting conditions from among a plurality of models.

2. The information processing apparatus according to claim 1 , wherein the plurality of models differ from each other in the type of object to be identified.

3. 3. The information processing device according to claim 1, wherein the photographing conditions suitable for classifying the type are determined for each type based on a result of aggregating the classification accuracy of the second model for each of a plurality of photographing conditions.

4. An information processing device according to any one of claims 1 to 3, further comprising a first learning means for learning the first model so that images of the object captured under the first shooting conditions are classified into classes according to the types that have the same shooting conditions suitable for identifying the type.

5. 5. The information processing device according to claim 1, further comprising a second learning means for learning the second model so that images of the object captured under shooting conditions suitable for identifying the type are classified into classes according to the type.

6. The information processing device according to claim 4, characterized in that the first learning means determines the shooting conditions to be used as the first shooting conditions based on the results of aggregating the identification accuracy of the first model for each of a plurality of shooting conditions.

7. 7. The information processing device according to claim 1, wherein the second identification means inputs the first image into the second model when the second shooting conditions are the same as the first shooting conditions.

8. 8. The information processing apparatus according to claim 1, further comprising: a detection unit for detecting an area of ​​the object from at least one of the first image and the second image.

9. 9. The information processing apparatus according to claim 1, wherein the photographing conditions include at least one setting of a photographing position and a setting related to camera parameters.

10. 10. The information processing apparatus according to claim 1, wherein the photographing conditions include at least one setting of a lighting position, a lighting amount, and a lighting color.

11. The imaging device further includes a control unit for controlling the imaging device for capturing an image of the object. the control means controls the imaging device to change the settings to the second imaging conditions, 11. The information processing apparatus according to claim 1, wherein the second identification means acquires the second image from the imaging device.

12. 12. The information processing apparatus according to claim 11, wherein the control means further controls an illumination device of the imaging device, and controls the illumination device so as to change the settings of the illumination device to the second photographing conditions.

13. the object is transported on a conveyor belt; the control means controls a first imaging device and a second imaging device provided downstream of the first imaging device in a conveying direction of the belt conveyor; The first identification means acquires the first image from the first imaging device, The information processing device described in claim 11 or 12, characterized in that the second identification means acquires the second image from the second imaging device at a timing when it is predicted that the object identical to the object imaged by the first imaging device will have reached the imaging position of the second imaging device, based on the distance between the first imaging device and the second imaging device and the conveying speed of the belt conveyor.

14. 14. The information processing apparatus according to claim 1, wherein the second identification means selects an image captured under the second photographing condition from among images captured under a plurality of photographing conditions.

15. 15. The information processing apparatus according to claim 1, wherein the object is a fish.

16. 15. The information processing apparatus according to claim 1, wherein the object is a part.

17. 17. The information processing apparatus according to claim 1, wherein at least one of the first model and the second model is configured as a neural network.

18. An information processing system including an imaging device that captures an image of an object and an information processing device that identifies the object, The information processing device includes: a first acquisition means for acquiring a first image captured under a first imaging condition from the imaging device; a first identification means for inputting the first image into a first model to identify a photographing condition suitable for identifying the type of the object; a control means for controlling the imaging device to change the settings to a second imaging condition; a second acquisition means for acquiring a second image captured under the second imaging conditions from the imaging device; a second identification means for identifying the type by inputting the second image into a second model; and The information processing system is characterized in that the second identification means uses, as the second model, a model associated with the second shooting conditions from among a plurality of models.

19. a first identification step of inputting a first image of an object captured under a first imaging condition into a first model, thereby identifying an imaging condition suitable for identifying the type of the object; a second classification step of classifying the type by inputting a second image of the object captured under the second photographing condition identified in the first classification step into a second model; Including, An information processing method, characterized in that in the second identification step, a model associated with the second shooting conditions is used as the second model from among a plurality of models.

20. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 17.

Images