Information processing device, item recognition device, item recognition system, data processing method, and program

Abstract

The present invention provides an information processing device, an item recognition device, an item recognition system, a data processing method, and a program that can reduce the possibility of an item being judged as unreliable in the reliability determination of recognition results by an item recognition device, even when the item has been correctly recognized. [Solution] The information processing device 100 performs at least one of the following processes: a first process in which, if the arrangement of the items in the recognition result of the item recognition device 110 for the input image matches the arrangement of the items shown in the input image, the recognition result of the item recognition device 110 for the converted image obtained by converting the input image matches; and a second process in which, if the arrangement of the items in the recognition result of the item recognition device 110 matches the arrangement of the items shown in the input image, the recognition result of the item recognition device 110 for the input image matches the recognition result of the item recognition device 110 for the converted image obtained by converting the input image. The device includes an image creation unit 103.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an article recognition apparatus, an article recognition system, a data processing method, and a program.

Background Art

[0002] In the background of the labor shortage due to the declining birthrate and aging population, the automation of warehouse systems using robots has been promoted. In particular, in the automation of warehouse systems, there is a demand for material handling equipment that can recognize various articles and handle them. However, in an article recognition device for recognizing articles, an improvement in versatility to handle various articles and an improvement in reliability that the recognition result is accurate generally have a trade-off relationship. For example, regarding the latest article recognition device, although it is equipped with advanced functions and can expand the automation range, due to its先进性, its functions are not sufficiently mature, and thus it is generally less reliable than mature functions.

[0003] As a means to compensate for the low reliability of an article recognition device equipped with advanced functions, recovery can be performed through execution of an alternative algorithm, assistance by a remote operator, assistance by another material handling device, etc. At this time, in order to determine the necessity of recovery, it is necessary to evaluate whether the recognition result of the article recognition device can be trusted.

[0004] It should be noted that the term "先进性" in the original text seems a bit inappropriate in this context. It might be a misspelling or an unclear expression. If it can be corrected to a more appropriate term, the translation will be more accurate.One example of a technique for evaluating the reliability of the recognition results of an object recognition device is the technique described in Patent Document 1. This publication states that "the processor 10 compares the recognition result (R1) and the recognition result (R2) using the determination unit 22 (S7). Based on the comparison result 140 of the determination unit 22, the processor 10 determines that the CNN model 20 has recognized the object correctly if the recognition results (R1, R2) are the same (YES in S8) (S9). On the other hand, based on the comparison result 140 of the determination unit 22, the processor 10 determines that if the recognition results (R1, R2) are different (NO in S8), an event has occurred in which the CNN model 20 misrecognizes due to the influence of AdE (S10). In other words, the processor 10 determines that the reliability of the recognition process of the CNN model 20 is low." [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-154386 [Overview of the project] [Problems that the invention aims to solve]

[0006] The technology described in Patent Document 1 applies a perturbation that changes the recognition result, regardless of whether the original recognition result was correct or not. Therefore, there is a risk that even correct recognition results may be judged as having low reliability.

[0007] This invention has been made in view of these circumstances, and aims to provide an information processing device, an item recognition device, an item recognition system, a data processing method, and a program that can reduce the possibility of an item being judged as unreliable in the reliability determination of the recognition result by an item recognition device, even though it has been correctly recognized. [Means for solving the problem]

[0008] To solve the above problems, the present invention provides an information processing device comprising: an input unit for inputting an input image; an image creation unit for creating a converted image by converting the input image; a transmission unit for transmitting the converted image and the input image to an object recognition device; a receiving unit for receiving the recognition results of the object recognition device for the converted image and the input image; an image comparison unit for comparing the recognition results of the object recognition device for the input image and the recognition results of the object recognition device for the converted image received by the receiving unit; and an output unit for outputting the comparison results of the image comparison unit, wherein the image creation unit performs the following in relation to the recognition results of the object recognition device for the input image The method is characterized by performing at least one of the following processes: a first process that, if the arrangement of the items matches the arrangement of the items shown in the input image, creates a converted image such that the recognition result of the item recognition device matches the recognition result of the item recognition device on the converted image obtained by converting the input image; and a second process that, if the arrangement of the items in the recognition result of the item recognition device does not match the arrangement of the items shown in the input image, creates a converted image such that the recognition result of the item recognition device on the input image differs from the recognition result of the item recognition device on the converted image obtained by converting the input image. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information processing device, an item recognition device, an item recognition system, a data processing method, and a program that can reduce the possibility of an item being judged as unreliable in the reliability determination of the recognition result by an item recognition device, even if it has been correctly recognized. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration of an information processing device according to the first embodiment of the present invention, and the relationship between the information processing device, the object recognition device, and the robot. [Figure 2] This is a block diagram showing an example of the hardware configuration of an information processing device according to the first embodiment of the present invention. [Figure 3] This diagram illustrates the operation of the image creation unit during training in the comparative example. [Figure 4] This figure shows the operation of the image creation unit of the information processing apparatus according to the first embodiment of the present invention during learning. [Figure 5] This figure shows a table illustrating the relationship between the recognition result and the loss for both the correct and incorrect recognition results of the information processing device according to the first embodiment of the present invention. [Figure 6] This figure illustrates the construction of the image creation unit of an information processing apparatus according to the first embodiment of the present invention. [Figure 7] This figure shows a table illustrating the selection criteria for maintaining / not maintaining the recognition result in the image creation unit of the information processing apparatus according to the first embodiment of the present invention. [Figure 8] This figure shows an example of a shooting system for capturing an image of an item to be recognized, provided in the information processing device according to the first embodiment of the present invention. [Figure 9] This is an explanatory diagram showing an example of an article to be recognized by the article recognition device of the information processing device according to the first embodiment of the present invention. [Figure 10] This figure shows the application of median blurring, an image processing technique, to an item to be recognized and the entire image containing the item to be recognized, according to a second embodiment of the present invention. [Figure 11] This is an explanatory diagram showing an example of an item to be recognized by the item recognition device of an information processing device according to a second embodiment of the present invention. [Figure 12] This figure shows the recognition target article and the entire image containing the recognition target article enhanced in contrast to an information processing device according to a second embodiment of the present invention. [Figure 13] This is a flowchart of the decision-making process for a rule-based image conversion method implemented in an information processing device according to a second embodiment of the present invention. [Figure 14] This figure illustrates a training image and actual arrangement information of articles within an information processing device according to a second embodiment of the present invention. [Figure 15] This figure illustrates the recognition result of the item recognizer 110 on a training image of an information processing device according to a second embodiment of the present invention. [Figure 16]This is a diagram for explaining a comparison result obtained by comparing arrangement information and recognition results of an information processing apparatus according to a second embodiment of the present invention. [Figure 17] This is a diagram showing an example of the behavior of an image creation unit during learning of an information processing apparatus according to a second embodiment of the present invention. [Figure 18] This is a diagram showing an example of the behavior of an image creation unit and an image comparison unit during inference of an information processing apparatus according to a second embodiment of the present invention. [Figure 19] This is a diagram showing a case where recognition results are integrated and recognized as one article by performing image conversion using an image creation unit of an information processing apparatus according to a third embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. (First Embodiment) FIG. 1 is a diagram showing a configuration example of an information processing apparatus 100 according to a first embodiment of the present invention and the relationship among the information processing apparatus 100, an article recognizer 110, and a robot 120. In each figure, components with the same reference numerals indicate the same constituent elements or constituent elements having similar functions. As shown in FIG. 1, an article recognition system 1 includes an information processing apparatus 100 and an article recognizer 110 that outputs recognition results for a conversion image and an input image to the information processing apparatus 100. The information processing apparatus 100 includes a storage unit 101, an input unit 102, an image creation unit 103, a transmission unit 104, a reception unit 105, an image comparison unit 106, an output unit 107, and a control unit 108. An article recognizer 110 and a robot 120 are connected to the information processing apparatus 100. Further, the information processing apparatus 100 is built in the article recognition system 1 and has a function as an article recognition apparatus that performs article recognition while evaluating the reliability of recognition results.

[0012] The memory unit 101 stores, for example, a group of images, an input image conversion method, etc. The group of images consists of images of objects with annotations added. The images include not only 2D images but also 3D images, etc. Annotation is information about the arrangement of objects in the image. Examples of annotations for 2D images include bounding boxes, segmentation, polygon annotation, line annotation, etc. Furthermore, there are also composite annotations that combine these annotation methods. Examples of annotations for 3D images include 3D bounding boxes, volume segmentation, polygon mesh annotation, and surface annotation. In addition, there are composite annotations that combine these methods.

[0013] The input image transformation method is a method of selecting a specific area of ​​an image and performing image processing. Methods for selecting a specific area include, for example, randomly selecting pixels. Alternatively, a specific area can be selected, and methods for selecting an area include selecting a fixed area at an arbitrary position, the center of the image, or a corner of the image, or selecting based on a ratio according to the image size. Furthermore, the image can be divided into a grid, and specific areas can be randomly selected. Multiple areas can be selected, not just one, and the entire image can be selected as an area. Possible image processing methods include geometric transformation, color space transformation, feature transformation, edge enhancement, image quality modification, morphological transformation, spatial filtering, frequency filtering, and fill. The input image transformation method may also have parameters. The parameter values ​​are determined by the image creation unit 103, which will be described later.

[0014] The input unit 102 receives an input image. Examples of input image formats include binary images, grayscale images, and RGB images. Examples of input image types include point cloud images, distance images, infrared images, thermal images, and hyperspectral images.

[0015] The image creation unit 103 performs at least one of the following processes: a first process in which, if the arrangement of the items in the recognition result of the item recognition unit 110 for the input image matches the arrangement of the items shown in the input image, the recognition result of the item recognition unit 110 for the input image matches the recognition result of the item recognition unit 110 for the converted image obtained by converting the input image; and a second process in which, if the arrangement of the items in the recognition result of the item recognition unit 110 does not match the arrangement of the items shown in the input image, the recognition result of the item recognition unit 110 for the input image matches the recognition result of the item recognition unit 110 for the converted image obtained by converting the input image.

[0016] The image creation unit 103 creates a converted image based on the input image. To create the converted image, an input image conversion method may be used from the storage unit 101 based on the input image. The method for determining the input image conversion method follows the process described later. By following the process described later, the image creation unit 103 determines an input image conversion method that changes the recognition result of the item recognition unit 110 on the converted image only if the actual arrangement of the items does not match the recognition result of the item recognition unit 110 on the input image. That is, if the arrangement of the items in the recognition result of the item recognition unit does not match the arrangement of the items shown in the input image, a second process is performed to create a converted image such that the recognition result of the item recognition unit on the input image and the recognition result of the item recognition unit on the converted image obtained by converting the input image are different.

[0017] Here, the recognition result of the object recognition device 110 is an image in which the object recognition device 110 predicts the position of the object in the image and adds annotation information to the image of the object input to the object recognition device 110. An example of the recognition result is specifying annotations such as bounding boxes and segmentation corresponding to the position of the object in the image. Multiple bounding boxes may be specified for the position of the object.

[0018] The image creation unit 103 may determine one or more input image conversion methods. If multiple methods are determined, the image creation unit 103 also determines the order in which the input image conversion methods are applied. Furthermore, if the input image conversion method has parameters, the image creation unit 103 determines the values ​​of the parameters.

[0019] The image creation unit 103 may create more than one converted image. Furthermore, the image size of the converted image does not necessarily have to match that of the input image. Additionally, the image creation unit 103 may adjust the image size as needed, for example, to match the size of the input image. To change the image size, methods such as scaling, padding, and clipping can be used.

[0020] The transmitting unit 104 transmits the input image and the converted image to the object recognition unit 110. In other words, the input and / or output of the transmitting unit 104 is the input image.

[0021] The receiving unit 105 receives the recognition results of the object recognition unit 110 for the input image and the recognition results of the object recognition unit 110 for the converted image.

[0022] The image comparison unit 106 compares the recognition result of the object recognition unit 110 for the input image and the recognition result of the object recognition unit 110 for the converted image, as received by the receiving unit 105, and outputs the comparison result. The comparison result may be a numerical value or a set of multiple numerical values. For example, if the object recognition result of the object recognition device 110 is to represent the position of an object in an image with a bounding box, IoU (Intersection over Union) can be used as an evaluation metric to check whether the recognition result of the object recognition device 110 for the input image matches the recognition result of the object recognition device 110 for the converted image. IoU is an evaluation metric that quantifies the overlap of bounding boxes on a scale from 0 to 1, where a value closer to 1 indicates a large overlap and a value closer to 0 indicates a small overlap. Specifically, a perfect match is 1, and no overlap at all is 0.

[0023] Evaluation metrics such as precision, recall, and Dice (Dice Similarity Coefficient: DSC) can also be used. Similarly, when the recognition result represents the position of objects in an image using segmentation, evaluation metrics such as IoU and Dice can be used to evaluate whether the recognition result of the object recognizer 110 for the input image matches the recognition result of the object recognizer 110 for the converted image.

[0024] Here, it should be noted that the image creation unit 103 determines an input image conversion method that changes the recognition result of the item recognition unit 110 for the converted image only if the actual arrangement of the items does not match the recognition result of the item recognition unit 110 for the input image. For example, if the comparison result of the image comparison unit 106 does not match, it can be evaluated that the recognition result for the input image does not match the actual arrangement. Conversely, if the comparison result of the image comparison unit 106 matches, it can be evaluated that the recognition result for the input image matches the actual arrangement.

[0025] The output unit 107 outputs output information including the comparison results from the image comparison unit 106. The output information includes, for example, the recognition result of the object recognition unit 110 for the input image or the recognition result of the object recognition unit 110 for the converted image, the name of the evaluation index used for the comparison, and the comparison result.

[0026] The control unit 108 controls the operation of the robot 120 with respect to the objects shown in the input image based on the output information output by the output unit 107.

[0027] [Hardware configuration of information processing equipment] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 100. The information processing device 100 consists of a computer comprising a processor 201, memory 202, auxiliary storage device 203, input device 204, output device 205, and communication interface 206, all connected by internal communication lines 207 such as a bus.

[0028] The processor 201 executes the program stored in memory 202. Memory 202 includes non-volatile memory elements such as ROM (Read Only Memory) and volatile memory elements such as RAM (Random Access Memory). ROM stores immutable programs (e.g., BIOS (Basic Input / Output System)). RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory) and temporarily stores the program executed by the processor 201 and the data used during program execution.

[0029] The auxiliary storage device 203 is a high-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive) or flash memory (SSD: Solid State Drive), and stores the program executed by the processor 201 and the data used when the program is executed. The program is read from the auxiliary storage device 203, loaded into memory 202, and executed by the processor 201.

[0030] Input device 204 is a device that receives input from the operator, such as a keyboard or mouse. Output device 205 is a device that outputs the program execution results in a format that the operator can see, such as a display device or printer. Communication IF 206 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0031] The program executed by the processor 201 is provided to the information processing device 100 via removable media (such as a CD-ROM or flash memory) or a network, and stored in the non-volatile auxiliary storage device 203, which is a non-temporary storage medium. For this reason, the information processing device 100 should have an interface for reading data from the removable media.

[0032] The information processing device 100 is a computer system that is physically located on a single computer or on multiple computers configured logically or physically. It may operate in separate threads on the same computer, or it may operate on a virtual computer built on multiple physical computer resources. For example, the information processing device 100 may not be a single computer, but may be divided into a teaching object registration device, which is a computer that registers teaching objects and recognition methods for recognizing objects, and a determination device, which is a computer that determines whether or not an object is a teaching object using the set recognition method.

[0033] The processor 201 includes, for example, the aforementioned functional units: the input unit 102, the image creation unit 103, the transmission unit 104, the reception unit 105, the image comparison unit 106, the output unit 107, and the control unit 108.

[0034] The processor 201 functions as an input unit 102 by operating according to the image acquisition program loaded into memory 202, and functions as an output unit 107 by operating according to the image display program loaded into memory 202. The relationship between the program and the function unit is similar for the other function units included in the processor 201.

[0035] Furthermore, some or all of the functions provided by the functional units included in the processor 201 may be implemented by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0036] The auxiliary storage device 203 holds the aforementioned image group and input image conversion method, etc. Some or all of the information stored in the auxiliary storage device 203 may be stored in the memory 202, or in an external database connected to the information processing device 100.

[0037] In this embodiment, the information used by the information processing device 100 is independent of the data structure and may be represented in any data structure. In this embodiment, the information is represented in a table format, but for example, a data structure appropriately selected from a list, database, or queue can store the information.

[0038] The operation of the information processing device configured as described above will be explained below. (Explanation of the principle) First, I will explain the basic concept of this invention. Figures 3 and 4 illustrate the operation of the image creation unit during learning. Figure 3 shows the operation of the comparative example device during learning, and Figure 4 shows the operation of the image creation unit 103 of the information processing device 100 in Figure 1 during learning.

[0039] <Comparative Example> In the comparative example apparatus shown in Figure 3, input image 1 is input to image creation unit 2, which creates converted image 3 based on input image 1. Converted image 3 is input to object recognition unit 4, which predicts the position of the object in the input image and outputs an image with annotation information as recognition result 5 (5a, 5b). Recognition result 5 is compared with Ground Truth 6 (white arrow a in Figure 3).

[0040] The learning process is as follows: During training, the comparative example device calculates the degree of agreement between the recognition result 5 and the Ground Truth 6, outputs the degree of agreement calculation result to the image creation unit 2 (white arrow b in Figure 3), and also outputs it to the object recognition unit 4 (white arrow c in Figure 3).

[0041] Image creation unit 2 creates a transformed image during training so that the degree of agreement between the recognition result 5 and the Ground Truth 6 decreases. Specifically, image creation unit 2 creates an image so that the recognition result changes due to the image transformation (for example, by adding noise to the image).

[0042] During training, the object recognition unit 4 creates a transformed image in such a way that the degree of agreement between the recognition result 5 and the Ground Truth 6 is increased. Specifically, the object recognition unit 4 is trained to correctly recognize images even after they have been transformed (for example, correctly recognize images with noise added).

[0043] The comparative example device in Figure 3 is evaluated as having high reliability in its recognition results for the input image if the recognition results for the converted image remain unchanged.

[0044] However, the comparative example device in Figure 3 applies a transformation that changes the recognition result regardless of whether the recognition result is correct or not, which may lead to the correct recognition result being judged as having low reliability.

[0045] (Embodiment) In this embodiment, the information processing device 100 correctly determines reliability by performing image conversion so that the recognition result is maintained if the recognition result is correct. This will be explained in detail below.

[0046] In the information processing device 100 shown in Figure 4, input image 1 is input to the image creation unit 103 (Figure 1), and the image creation unit 103 creates a converted image 3 based on input image 1. The converted image 3 is input to the object recognition device 110 (Figure 1), and the object recognition device 110 predicts the position of the object in the image based on the input image of the object, and outputs an image with annotation information as the recognition result 111 (111a, 111b).

[0047] On the other hand, input image 1 is also directly input to the object recognition device 110 (arrow d in Figure 4). The object recognition device 110, without going through the image creation unit 103, directly inputs input image 1, predicts the position of the object in the image, and outputs an image with annotation information as the recognition result 112 (112a, 112b). Of the recognition results 112, recognition result 112a is the correct recognition result, and recognition result 112b is the incorrect recognition result.

[0048] The information processing device 100 (Figure 4) calculates the degree of agreement between the recognition result 112 (112a, 112b) of the object recognition device 110 and Ground Truth 6, based on the input image 1 that is directly input (white arrow e in Figure 4).

[0049] The operation of the information processing device 100 (Figure 4) is compared with that of the comparative example device in Figure 3. The information processing device 100 (Figure 4) calculates the degree of agreement between the recognition result 112 (112a, 112b) of the object recognition device 110, based on the directly input image 1, and Ground Truth 6. In contrast, in the comparison example in Figure 3, the object recognition device 4 calculates the degree of agreement between the recognition result 5 obtained by image recognition of the converted image 3 created by the image creation unit 2 and Ground Truth 6 (white arrow a in Figure 3).

[0050] In other words, when calculating the degree of agreement with Ground Truth 6, the comparison target in the comparison example in Figure 3 is the recognition result 5 obtained by image recognition of the converted image 3 created by the image creation unit 2, whereas in the information processing device 100 (Figure 4), it is the recognition result 112 for the directly inputted input image 1. Therefore, the loss calculation performed based on the degree of agreement also differs between the device in the comparative example in Figure 3 and the information processing device 100. The loss calculation L performed by the information processing device 100 will be described later.

[0051] Returning to Figure 4, the information processing device 100 (Figure 4) calculates a loss L based on equation (1) (described later) between the recognition result 111 ("recognition result for the converted image") obtained by image recognition of the converted image 3 created by the image creation unit 103 and the recognition result 112 ("recognition result for the input image") of the item recognizer 110 based on the directly inputted input image 1 (the bidirectional arrow f in Figure 4). Details of the loss calculation L will be described later.

[0052] Figure 5 is a table showing the relationship between the recognition result and the loss for both the correct recognition result 112a and the incorrect recognition result 112b. In the case of a correct recognition result 112a, if the recognition result matches, the loss is small; if the recognition result does not match, the loss is large. Similarly, in the case of an incorrect recognition result 112b, if the recognition result matches, the loss is large; if the recognition result does not match, the loss is small. The table in Figure 5 is the basis for deriving the loss calculation L.

[0053] As shown by arrow d in Figure 4, the image creation unit 103 of the information processing device 100 creates an image such that only the incorrect recognition result changes according to the loss calculation L (arrow g in Figure 4). In other words, if the recognition result is correct, the image creation unit 103 creates an image such that the recognition result is maintained.

[0054] Figure 6 is a diagram illustrating the construction of the image creation unit 103. Figure 7 is a table showing the selection criteria for maintaining / not maintaining the recognition results in the image creation unit 103 of Figure 6. The image creation unit 103 creates images according to the table shown in Figure 7, so that only the incorrect recognition results are changed (arrow g in Figure 7). Specifically, as shown in Figure 7, if the recognition result of the correct recognition result 112a matches, the recognition result is not changed (recognition result is maintained). If the recognition result of the correct recognition result 112a does not match, the recognition result is changed. If the recognition result of the incorrect recognition result 112b matches, the recognition result is changed. If the recognition result of the incorrect recognition result 112b does not match, the recognition result is not changed (recognition result is maintained). When the image creation unit 103 changes the recognition result, it selects one of the image conversion methods 1, ..., image conversion method K, or image conversion method N, for example, image conversion method K, as shown by arrow g in Figure 7.

[0055] In this way, the information processing device 100 can correctly determine reliability by performing image transformation so that the recognition result is maintained if the recognition result is correct.

[0056] [Examples of application] Figure 8 shows an example of a shooting system that captures an image of an item to be recognized, which is provided to the information processing device 100. The imaging system (object recognition system) shown in Figure 8 includes a camera 303, a table 306, and a terminal 301. The camera 303 captures images of the object to be recognized, including the object to be recognized 305. A robotic arm 302 is attached to the camera 303, and the movement of the robotic arm 302 allows the camera 303 to capture images from various positions and angles.

[0057] The objects to be recognized, including the object to be recognized 305, are placed on the table 306. Terminal 301 is a computer connected to camera 303. Terminal 301 controls the shooting by camera 303 and the movement of robot arm 302. Terminal 301 also acquires images of the objects captured by camera 303.

[0058] Terminal 301 is connected to the information processing device 100 (Figure 1) and transmits an image of the recognized items, including the acquired recognized item 305, to the information processing device 100.

[0059] The terminal 301 may control the camera 303 and arm 302 according to instructions from the information processing device 100. Furthermore, the information processing device 100 and the terminal 301 may be integrated into a single unit.

[0060] Furthermore, the camera 303 and the robot arm 302 do not have to be integrated. For example, the camera 303 may be independent of the arm and fixed in a position where it can capture images of the recognition target items, including the recognition target item 305. In addition, a hand may be attached to the tip of the robot arm 302 instead of the camera 303. By attaching a hand to the tip of the robot arm 302, for example, the recognition target items, including the recognition target item 305, can be grasped.

[0061] <Example 1: An example of a box with a slit> Figure 9 is an explanatory diagram showing Example 1 of an object to be recognized by the object recognition device 110. In Example 1 of Figure 9, the object to be recognized 305 and the object to be recognized 308 are boxes with lids, and an image of one side of the box is shown. The box has its lid closed, and the slit 307 is visible in the image. On the other hand, the boundary line 309 of the object to be recognized 305 and the object to be recognized 308 is also visible.

[0062] In the situation shown in Figure 9, consider an item recognition device 110 that recognizes the arrangement of boxes. If the item recognition device 110 does not include recognition processing for boxes that show the slit 307, it may mistakenly recognize the slit 307 as a boundary line and mistakenly identify the boxes that show the slit 307 as separate items, even though they should be detected as a single item.

[0063] On the other hand, if the item recognition device 110 includes recognition processing for the box in which the slit 307 is visible, it detects the box in which the slit 307 is visible as a single item.

[0064] <Evaluation of the recognition result of the box with the slit in Example 1> Thus, the recognition result for a box showing the slit 307 changes depending on whether or not the object recognition device 110 includes recognition processing for the box showing the slit 307. Therefore, if it is unclear whether a box showing the slit is being considered, it is difficult to determine whether the recognition result for the input image of the object recognition device 110 matches the actual arrangement of the object based solely on the recognition result for the input image of the object recognition device 110.

[0065] <Solution for Example 1> Figure 10 shows the result of applying a median blur image processing technique to the entire image containing the recognition target objects 305 and 308, as explained in Figure 9. Median blur is an image processing technique that transforms a pixel to the median value (the midpoint of the density of surrounding pixels). As shown in Figure 6, the image creation unit 103 (Figure 6) of the information processing device 100 (Figure 1) selects one of the image conversion methods 1, ..., K, or N, for example, image conversion method K, based on the loss calculation L. Here, as an example of an image conversion method, an image processing technique called median blur is applied.

[0066] As shown in Figure 10, the slit 307 can be removed by applying median blurring. When Figure 10 is used as the input image for the item recognition device 110 (Figures 1 and 6), the recognition result is unaffected by the slit 307 and the box is detected as a single item.

[0067] Based on the above, if the item recognition device 110 (Figures 1 and 6) does not include recognition processing for the box with the slit 307 (Figure 9) reflected in it, the recognition result will differ between Figure 9 and Figure 10. On the other hand, if the item recognition device 110 includes recognition processing for the box with the slit 307 reflected in it, the recognition result will not differ between Figure 9 and Figure 10. Therefore, if the recognition result differs between Figure 9 and Figure 10, it can be said that the item recognition device 110 does not include recognition processing for the box with the slit 307 reflected in it, and there is a possibility of misrecognition.

[0068] <Example 2: An example of taping a box> Figure 11 is an explanatory diagram showing example 2 of the objects to be recognized by the object recognition device. In example 2 of Figure 11, the objects to be recognized 501 and 502 are boxes sealed with tape 503, and an image of one side of the box is shown. The boundary line 504 between objects to be recognized 501 and 502 is visible.

[0069] In the situation shown in Figure 11, for example, if the item recognizer 110 does not include recognition processing for boxes with tape attached, such as the tape 503, it may mistakenly recognize the tape-attached box as a separate item, even though it should be detected as a single item, by recognizing the color change between the tape 503 and the box as a boundary line.

[0070] On the other hand, if the item recognition device 110 includes recognition processing for the box on which the tape 503 is displayed, the box on which the tape 503 is displayed can be detected as a single item.

[0071] <Evaluation of the recognition results for the box showing the tape in Example 2> Thus, the recognition result for the box on which the tape 503 is visible changes depending on whether or not the item recognition device 110 includes recognition processing for the box on which the tape 503 is visible. Therefore, if it is unclear whether the box on which the tape 503 is visible is being considered, it is difficult to determine whether the recognition result for the input image of the item recognition device 110 matches the actual arrangement of the items based solely on the recognition result for the input image of the item recognition device 110.

[0072] <Solution for Example 2> Figure 12 shows the entire image containing the recognition target objects 501 and 502, as explained in Figure 11, with the contrast enhanced. Contrast is the difference between the bright and dark parts of an image. When the contrast is enhanced, this difference becomes stronger, making the bright parts even brighter and the dark parts even darker. For example, in the case of tape, median blurring (Figure 10) makes it difficult to erase the tape boundary, but as shown in Figure 12, reducing the contrast can make tape 503 less visible.

[0073] As shown in Figure 6, the image creation unit 103 (Figure 6) of the information processing device 100 (Figure 1) selects one of the image conversion methods 1, ..., K, or N, for example, image conversion method K, based on the loss calculation L. Here, as an example of an image conversion method, an image processing that reduces contrast is applied.

[0074] When Figure 12 is input as an input image to the item recognition device 110 (Figures 1 and 6), the information processing device 100 (Figure 1) can detect the box as a single item without being affected by the tape 503. Based on the above, if the item recognition device 110 does not include recognition processing for the box containing the tape 503, the recognition results will differ between Figure 11 and Figure 12. On the other hand, if the item recognition device 110 does not include recognition processing for the box containing the tape 503, the recognition results will not differ between Figure 11 and Figure 12. Therefore, if the recognition results differ between Figure 11 and Figure 12, it can be said that the item recognition device 110 does not include recognition processing for the box containing the tape 503, and there is a possibility of misrecognition.

[0075] <Summary of solutions for Example 1 and Example 2> This embodiment is an example where a predetermined rule base is in place. In this embodiment, by utilizing the fact that different results can be obtained by converting the input image to an appropriate image, it is possible to evaluate whether the recognition result for the input image is an error or not. The following explains how to determine the appropriate image transformation based on the input image.

[0076] [flowchart] Figure 13 is a flowchart of the process for determining the rule-based image transformation method. First, in step S11, the information processing device 100 acquires information about the item to be recognized. As a means of obtaining information about the item to be recognized, it uses information such as SKU (Stock Keeping Unit), FKU (Face Keeping Unit), part number, lot number, EAN, UPC, and JAN used in logistics. In addition to this information, by registering the fact that the item to be recognized has a slit or tape on it, it is possible to obtain information about the item to be recognized.

[0077] Next, in step S12, the information processing device 100 determines from the acquired information on the item to be recognized whether or not the item to be recognized has a slit. If the item to be recognized has a slit (S11: Yes), select median blur in step S13 to terminate the processing of this flow.

[0078] If the item to be recognized does not have a slit (S11: No), in step S12 the information processing device 100 determines from the acquired information on the item to be recognized whether or not tape is attached to the item. If tape is attached to the item to be recognized (S14: Yes), select "Contrast Reduction" in step S15 to terminate the processing of this flow. If no tape is applied (S14: No), the system determines that appropriate image conversion processing is not available and terminates the processing of this flow.

[0079] Based on the above, by determining the appropriate image conversion, it is possible to appropriately determine whether or not an image contains misrecognitions, regardless of the object being recognized.

[0080] (Second embodiment) The first embodiment is an example of a case that can be solved using a rule-based approach, as described in the flow chart of Figure 13. The second embodiment is an example of solving a problem that cannot be solved using a rule-based approach. The configuration example of the information processing device 100 according to the second embodiment of the present invention, and the relationship between the information processing device 100, the item recognition device 110, and the robot 120, are the same as in Figure 1.

[0081] <Challenges that cannot be solved by rule-based systems> This section explains challenges that cannot be solved using rule-based approaches. When various factors can influence boundary recognition, it is necessary to consider different image processing methods for each factor. To address this challenge, this paper describes an approach that uses machine learning to create images appropriate to the factors influencing boundary recognition.

[0082] In other words, this approach transforms the image so that, if the recognition result of the object recognition device 110 for the training image matches the arrangement information of the training image, the recognition result of the object recognition device 110 for the transformed image obtained by transforming the training image matches. Conversely, if the recognition result of the object recognition device 110 for the training image does not match the arrangement information of the training image, the image transformation method is learned so that the recognition result of the object recognition device 110 for the training image and the recognition result of the object recognition device 110 for the transformed image obtained by transforming the training image differ.

[0083] First, let's explain the dataset used in the information processing device 100 (Figure 1). A dataset consists of a large number of pairs of "problem data" and "model answer data." The model is the output of training this dataset using Deep Learning. In other words, the model is the parameters of the neural network used in the final extraction process.

[0084] The dataset used for training the information processing device 100 (Figure 1) includes the following three types: (1) Training images and information on the actual placement of objects in the images (2) Recognition results of the object recognition device 110 on the training images (3) Comparison results comparing the arrangement information in (1) above with the recognition results in (2)

[0085] Figure 14 is a diagram illustrating (1) the training image and the actual placement information of the objects within the image. For training images (Figure 14, left), the actual placement information of objects is represented using, for example, bounding boxes or segmentation (thick solid lines in Figure 14, right). This placement information is added, for example, by a human.

[0086] Figure 15 is a diagram illustrating the recognition results of the object recognition device 110 for the training images (2). A training image (left image in Figure 15) is input to the object recognition device 110, and the output from the object recognition device 110 is, for example, a bounding box (right image in Figure 15). For example, if the object recognition device 110 does not include recognition processing for boxes with slits, even though the boxes with slits should be detected as one item, it may recognize the slits as boundaries and produce an output result that misrecognizes them as separate items, as shown in the two thick solid line frames in the lower part of the right image in Figure 15.

[0087] Figure 16 is a diagram illustrating the comparison results between the arrangement information (1) and the recognition results (2). Figure 16 shows an example of a comparison where the actual placement information of items is compared with the placement of items as recognized by the item recognition device 110, and a comparison result is calculated indicating whether they match or not. As a criterion for determining whether or not they match, one can, for example, check whether the IoU of the recognition result before image conversion and the annotation are above a threshold. Alternatively, the comparison can be performed manually by a human.

[0088] In this embodiment, the image creation unit 103 performs at least one of the following processes: a first process in which, if the arrangement of the items in the recognition result of the item recognition unit 110 for the input image matches the arrangement of the items shown in the input image, the recognition result of the item recognition unit 110 for the converted image obtained by converting the input image matches; and a second process in which, if the arrangement of the items in the recognition result of the item recognition unit 110 matches the arrangement of the items shown in the input image, the recognition result of the item recognition unit 110 for the input image matches the recognition result of the item recognition unit 110 for the converted image obtained by converting the input image.

[0089] In this embodiment, the object recognition result for the image before image conversion and the object recognition result for the image after image conversion are compared. If the recognition results match, it is determined that the recognition result is correct ("first process"). If the recognition results differ, it is determined that there is a possibility of false detection ("second process"). Therefore, when the annotation and the recognition result before image conversion match, i.e., when the recognition result is correct, the loss is low when the recognition result for the image after image conversion also matches. Conversely, when the annotation and the recognition result before image conversion do not match, i.e., when the recognition result is incorrect, the loss function is designed such that the loss is low when the recognition result for the image after image conversion also does not match. This makes it possible to train the image creation unit 103 to select an image transformation that changes the recognition result only for false detections. As an example of a loss function L, for example, a loss function L like the following equation (1) can be designed.

[0090]

number

[0091] Here, M is the number of training images, and MN is the number of pairs (rn,rn') in image m, where rn is the object recognition result for the image before image transformation and rn' is the object recognition result for the image after image transformation. This pair can be determined by calculating the pair that maximizes the IoU (Individual IoU) between the object recognition result for the image before image transformation and the object recognition result after image transformation for the training image m. Furthermore, I(rn=p) represents the case where rn is the correct recognition result, I(rn=n) represents the case where ri is the incorrect recognition result, and U(rn,rn') represents the IoU of rn and rn'.

[0092] Figure 17 shows an example of the behavior of the image creation unit 103 during training. As shown in Figure 17, the image creation unit 103 determines an input image transformation method from a predetermined set of image transformation methods based on the training image. The training image is input to the determined input image transformation method to obtain a transformed image. Then, the transformed image is input to the object recognition unit 110 to obtain a recognition result for the transformed image. The loss function L shown in equation (1) is used to calculate the value of the loss function for the selected image transformation method, and the image creation unit 103 is trained according to that value. For example, the image creator can be realized by using a CNN (Convolutional Neural Network) to calculate image features representing the training image, and then inputting the calculated features into an SVM (Support Vector Machine) or Random Forest to output the class and parameters of the image transformation method.

[0093] In Figure 17, the image creation unit 103 is shown with a training image as its input and an input image conversion method as its output, but the input and output examples are not limited to these. For example, the recognition result before image conversion may be used as the input. Also, the image creation unit 103 may output the converted image itself directly, rather than an image conversion method. For this reason, the image creation unit 103 could be, for example, a GAN (generative adversarial network), a Conditional VAE (variational autoencoder), or a normalized flow model.

[0094] Figure 18 shows an example of the behavior of the image creation unit 103 and the image comparison unit 106 during inference. The reasoning process consists of the following three steps. (1) An input image containing the item to be recognized is input to the item recognizer 110, and the recognition result is obtained (flow in the upper part of Figure 18). (2) The input image containing the item to be recognized is input to the image creation unit 103, and the input image is converted based on the input image conversion method obtained. Then the converted image is input to the item recognizer 110 to obtain the recognition result (flowchart in the lower part of Figure 18). (3) The recognition results obtained in (1) and (2) above are input to the image comparison results unit. If the recognition results match, it is assumed that the arrangement of the items in the recognition results of the item recognition device 110 obtained in (1) above matches the actual arrangement of the items. If the recognition results do not match, it is assumed that the arrangement of the items in the recognition results of the item recognition device 110 obtained in (1) above may not match the actual arrangement of the items.

[0095] If the arrangement of items in the recognition result matches the actual arrangement of items, the first process is performed; otherwise, the second process is performed. Incidentally, the information processing device 100 may have the function to perform only the first process, the function to perform only the second process, or the function to perform both the first and second processes.

[0096] Therefore, even when various factors can influence boundary recognition, machine learning can be used to select the most appropriate image processing method.

[0097] (Third embodiment) The second embodiment can determine whether the recognition result of the item recognition device 110 is a false detection or not. However, it cannot determine how to recover if there is a possibility of a false detection. In the third embodiment, an example is described in which the control unit 108 changes the operation of the robot 120 based on the image result of the image comparison unit 106. The configuration example of the information processing device 100 according to the third embodiment of the present invention, and the relationship between the information processing device 100, the item recognition device 110, and the robot 120, are the same as in Figure 1.

[0098] Figure 19 shows the case where the recognition results are integrated and recognized as a single item after image conversion is performed using the image creation unit 103. The control unit 108 (Figure 1) of the information processing device 100 (Figure 1) changes the operation of the robot 120 based on the image results from the image comparison unit 106. The upper part of Figure 19 shows the recognition image before the change in the robot 120's operation, and the lower part of Figure 19 shows the recognition image after the change in the robot 120's operation.

[0099] The information processing device 100 (Figure 1) determines that the recognition result may be a false detection because the object recognition result for the image before image conversion and the object recognition result for the image after image conversion are different. In this case, recovery can be performed by shifting the object, for example, vertically, relative to the integrated boundary line. That is, if there are multiple objects adjacent to each other, shifting the objects will separate them, making the boundary line clearer, and by inputting the information back into the object recognition device 110, it can be recognized that there are multiple objects (recognition result in the upper part of Figure 19). Also, if there is only one object, the objects will move together by the amount that their positions are shifted, so it can be seen that there is only one object (recognition result in the lower part of Figure 19).

[0100] In this embodiment, we have shown a case where the recognition results are integrated and recognized as a single item by performing image conversion using the image creation unit 103 (Figure 1). Conversely, even if the recognition results are divided and recognized as multiple items, recovery can be performed using the same approach.

[0101] Furthermore, even if the aforementioned shifting operation cannot be performed, a recovery method can be considered in which the camera 303 mounted on the robot's end-effector is moved closer to the boundary line where the recognition result has changed, thereby changing the position of the camera 303 to an appropriate position and performing recognition again, thereby improving the accuracy of object recognition.

[0102] Based on the above, the recognition result of the object recognition device 110 (Figure 1) can be identified as either a false detection or not, and even if a false detection is determined, recovery can be achieved by operating the robot 120 based on the comparison result.

[0103] (Fourth embodiment) Even if recovery through the robot 120's actions is not effective, notifying the user of this fact allows the system to inform the user that user support is needed. In the fourth embodiment, a display unit is newly provided to notify the user of the possibility of false detection. One means of notifying the user of the possibility of false detection is a GUI (Graphical User Interface). The content displayed on the GUI may include, for example, the input image, the recognition result, the recognition result of the object recognition device 110 on the converted image, and highlighting the changed parts.

[0104] Furthermore, to avoid requiring users to constantly monitor the GUI, notifications of false positives may be provided not only through visual information from the GUI, but also through vibrations or sounds via the user's mobile device.

[0105] Therefore, in the event of a false positive, notifying the user of this fact allows us to inform the user that they require support.

[0106] (Fifth embodiment) Knowing the range of items that the item recognition device 110 (Figure 1) can recognize is important when dealing with the item recognition device 110, which is a black box in that only its input and output are known. In the fifth embodiment, an item recognition feasibility determination unit is provided, which receives a plurality of evaluation images for determining whether an item can be recognized and item information for the items contained in the evaluation images, and determines whether an item linked to the item information can be recognized based on the output result of the output unit 107 (Figure 1) for the evaluation images.

[0107] First, the input unit 102 (Figure 1) receives multiple evaluation images to determine whether a specific item can be recognized. Each evaluation image contains item information for the item included in the evaluation image. For example, each evaluation image contains SKU information for the item being identified.

[0108] By inputting evaluation images into the item recognition unit 110, the recognition result of the item recognition unit 110 (Figure 1) for the evaluation images is obtained. Similarly, by inputting the evaluation images converted by the image creation unit 103 (Figure 1) into the item recognition unit 110, the recognition result for the converted evaluation images is obtained. Based on these recognition results, the output unit 107 calculates the number of evaluation images that may be falsely detected in the comparison results output. If a predetermined number or more cannot be recognized, the target item is determined to be an item that is difficult to recognize.

[0109] Furthermore, a single evaluation image may contain multiple SKUs as part of its item information. In this case, by receiving placement information that contains item information corresponding to the area of ​​each item, items that are difficult to recognize can be classified in the same manner as described above.

[0110] Therefore, even with an item recognition device 110 (Figure 1), which is a black box and only provides input and output information, it is possible to determine the range of items that can be recognized.

[0111] (Sixth embodiment) For the object recognition device 110 (Figure 1), which is a black box whose input and output are the only known parameters, it is difficult to improve the success rate of recognition by the object recognition device 110 itself, as its internal parameters cannot be adjusted. However, if a converted image can be created so that the recognition result of the object recognition device 110 matches the actual placement information, then inputting this converted image into the object recognition device 110 is expected to improve the success rate of recognition. In the sixth embodiment, the above objective is achieved by learning a second image conversion method that converts an image so that the recognition result of the object recognition device 110 (Figure 1) for the converted image obtained by converting the training image matches the arrangement information of the training image. Hereafter, the conversion methods up to the fifth embodiment will be referred to as the first conversion method to distinguish them.

[0112] By using the recognition results and placement information for the training images, the success or failure of each recognition result can be determined. In the sixth embodiment, a loss function for general object recognition is used to learn a second transformation method so that the recognition result matches the placement information. For example, the loss function L shown in equation (1) is designed and learned so that the actual placement and the recognition result overlap.

[0113] During inference, the image comparison unit 106 (Figure 1) can determine that the recognition result is correct if the recognition result of the item recognition unit 110 (Figure 1) for an image created using the second conversion method matches the recognition result of the item recognition unit 110 for an image created using the first conversion method, and determine that there is a possibility of false detection if they do not match.

[0114] Based on the above, even with the black-box object recognition device 110 (Figure 1), it is possible to determine whether or not a detection is false while increasing the success rate of recognition.

[0115] The present invention is not limited to the embodiments described above, and includes other modifications and applications, as long as they do not depart from the spirit of the invention as described in the claims.

[0116] The above embodiments are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those comprising all the described configurations. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0117] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software by having the processor 201 (Figure 2) interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory 202 (Figure 2), a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. [Explanation of Symbols]

[0118] 1. Item Recognition System 100 Information Processing Device (Item Recognition Device) 101 Storage section 102 Input section 103 Image Creation Department 104 Transmitter 105 Receiving Unit 106 Image Comparison Section 107 Output section 108 Control Unit 110 Article recognition device 120 robots 201 Processor 202 memory 203 Auxiliary storage device 204 Input device 205 Output device 206 Communication IF 207 Internal communication lines 301 terminals 302 Robot Arm 303 Camera 305,308 Items to be identified (boxes with lids) 306 Tables 307 Slit 309 Boundary of the item to be recognized

Claims

1. An input section for inputting an image, An image creation unit that creates a converted image by converting the aforementioned input image, A transmission unit that transmits the converted image and the input image to an object recognition device, A receiving unit that receives the recognition result of the item recognition device for the converted image and the input image, The receiving unit receives an image comparison unit which compares the recognition result of the object recognition device for the input image with the recognition result of the object recognition device for the converted image. The system includes an output unit that outputs the comparison results of the image comparison unit, The aforementioned image creation unit, If the arrangement of the items in the recognition result of the item recognition device for the input image matches the arrangement of the items shown in the input image, then a first process is performed to create a converted image such that the recognition result of the item recognition device matches the recognition result of the item recognition device for the converted image obtained by converting the input image. If the arrangement of the items in the recognition result of the item recognition device does not match the arrangement of the items shown in the input image, then at least one of the following processes is executed: a second process that creates a converted image such that the recognition result of the item recognition device for the input image and the recognition result of the item recognition device for the converted image obtained by converting the input image are different. An information processing device characterized by the following:

2. It has a memory unit that stores multiple image conversion methods, which are methods for converting images. The input unit receives item information as a rule base, The image creation unit selects one or more image conversion methods from among a plurality of image conversion methods based on the item information to create the converted image. The information processing apparatus according to feature 1.

3. The aforementioned article information is the SKU and / or FKU of the article. The information processing apparatus according to feature 2.

4. It includes a storage unit that stores training images showing objects and arrangement information indicating the arrangement of objects within the training images. The image creation unit learns an image conversion method such that, if the recognition result of the object recognition device for the training image matches the arrangement information of the training image, it converts the image so that the recognition result of the object recognition device for the training image matches the recognition result of the object recognition device for the converted image obtained by converting the training image, and if the recognition result of the object recognition device for the training image does not match the arrangement information of the training image, it converts the image so that the recognition result of the object recognition device for the training image and the recognition result of the object recognition device for the converted image obtained by converting the training image are different. The information processing apparatus according to feature 1.

5. The image creation unit takes at least one of the input image and the item recognition result for the input image as input. The information processing apparatus according to claim 3.

6. Equipped with a control unit that controls the robot's movements, The control unit changes the robot's operation method based on the comparison result. The information processing apparatus according to feature 1.

7. Equipped with a display unit, The display unit displays recognition results that may be false detections. The information processing apparatus according to feature 1.

8. Equipped with an item recognition feasibility determination unit, The input unit receives a plurality of evaluation images for determining whether an item can be recognized and item information for the items contained in the evaluation images. The item recognition feasibility determination unit determines whether an item associated with the item information can be recognized based on the output result of the output unit for the evaluation image. The information processing apparatus according to feature 1.

9. The image creation unit creates a second transformed image by transforming the image so that the recognition result of the object recognition device for the transformed image obtained by transforming the training image matches the arrangement information of the training image. The output unit determines that the recognition result is correct if the recognition result of the item recognition device for the second converted image matches the recognition result of the item recognition device for the converted image, and determines that there is a possibility of false detection if the recognition result of the item recognition device for the second converted image differs from the recognition result of the item recognition device for the converted image. The information processing apparatus according to feature 1.

10. The information processing device is provided according to any one of claims 1 to 9, and performs item recognition while evaluating the reliability of the recognition result. An article recognition device characterized by the following features.

11. The information processing apparatus according to claim 1, An item recognition system comprising an item recognizer that outputs the recognition results for the converted image and the input image to the information processing device.

12. An information processing method for an information processing device, The aforementioned information processing device is The steps include: inputting an image, A step of creating a converted image by converting the input image, and if the arrangement of the items in the recognition result of the item recognition device for the input image matches the arrangement of the items shown in the input image, a first process is performed to create the converted image such that the recognition result of the item recognition device matches the recognition result of the item recognition device for the converted image obtained by converting the input image; and if the arrangement of the items in the recognition result of the item recognition device does not match the arrangement of the items shown in the input image, a second process is performed to create the converted image such that the recognition result of the item recognition device for the input image matches the recognition result of the item recognition device for the converted image obtained by converting the input image; and at least one of these processes is performed. The steps include transmitting the converted image and the input image to the object recognition device, The steps include receiving the recognition result of the object recognition device for the converted image and the input image, A step of comparing the recognition result of the object recognition device for the received input image with the recognition result of the object recognition device for the converted image, The steps to output the comparison results and to execute An information processing method characterized by the following:

13. A program for causing a computer to function as an information processing device as described in claim 1.

Images