Identification support system, identification support method, and program

Abstract

To provide an identification support system that improves matching accuracy while suppressing a decrease in the speed of the matching process. [Solution] The identification support system according to this disclosure includes: an extraction means for extracting an item image, which is the region of an item, from an input image of an item to be inspected; an acquisition means for acquiring image characteristic information relating to the image quality of the item image and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched; and a determination means for determining, based on the image characteristic information and the similar item information, whether to use the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching with a pre-registered image of the item to be matched.

Description

Technical Field

[0004] , , , , , , , , , ,

[0006] , , , ,

[0005] , , , , ,

[0007] , , ,

[0001] This disclosure relates to an identification support device and the like.

Background Art

[0002] In the process of shipping distribution goods handled by logistics operators such as warehousing and wholesaling industries, before shipping the goods, there is an operation to confirm whether the goods to be shipped are the goods described in the shipping document. This operation is called inspection work. There is a subject identification system that photographs the goods to be shipped and identifies the goods by comparing them with a master image of the goods registered in advance.

[0003] It is expected that the accuracy of collation will be improved by increasing the resolution of the image. Patent Document 1 discloses a technique for increasing the resolution of a candidate region including a specific object from an image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0008] An identification support method in one aspect of the present disclosure extracts an item image, which is the region of an item, from an input image of an item to be inspected; obtains image characteristic information relating to the image quality of the item image, and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched; and determines, based on the image characteristic information and the similar item information, whether to use the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching with a pre-registered image of the item to be matched.

[0009] A program in one aspect of this disclosure extracts an item image, which is the region of an item, from an input image of an item to be inspected, obtains image characteristic information relating to the image quality of the item image, and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched, and causes a computer to perform a process to determine, based on the image characteristic information and the similar item information, whether to use the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching the item to be matched with a pre-registered registered image of the item to be matched. The program may be stored on a non-temporary recording medium that is readable by a computer. [Effects of the Invention]

[0010] One example of the effects of this disclosure is that it improves matching accuracy while suppressing a decrease in the speed of the matching process. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example configuration of an identification support device. [Figure 2] This figure shows an example of an input image. [Figure 3] This figure shows an example of an item image extracted from an input image. [Figure 4] This flowchart shows an example of the operation of the identification support device. [Figure 5] This is a block diagram showing an example configuration of an identification support device. [Figure 6] This table shows an example of registration data for items to be matched. [Figure 7] This figure shows an example of a screen displayed on a display. [Figure 8] This flowchart shows an example of the operation of the identification support device. [Figure 9] This flowchart shows an example of the operation of the identification support device. [Figure 10] This is a block diagram showing an example configuration of an identification support device. [Figure 11] This flowchart shows an example of the operation of the identification support device. [Figure 12] This is a block diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0012] [Embodiment 1] An example of the configuration of the identification support device 10 in this disclosure will be explained using Figure 1. The identification support device 10 comprises an extraction unit 101, an acquisition unit 102, and a determination unit 103. The extraction unit 101, the acquisition unit 102, and the determination unit 103 are embodiments of the extraction means, acquisition means, and determination means, respectively.

[0013] The extraction unit 101 extracts an object image, which is the area of the object, from the input image obtained by photographing the object to be inspected. The object to be inspected refers to the object that is the target of the inspection work. As an example of a specific inspection work, there may be a case where the object is placed in front of the camera and it is verified whether a correct object has been prepared. In this case, the image captured by the camera is input into the identification support device 10 as the input image. The extraction unit 101 extracts, by image recognition, the part in which a single object that is the target of inspection appears as the object image. When an input image that captures a single object without including the background is used, the extraction unit 101 may treat the input image as the object image.

[0014] The input image is an image used for comparison with the image of the comparison target object. The comparison target object is an object that is registered in advance in the database. The object to be inspected is verified whether a correct object has been prepared by being compared with the comparison target object. As the input image, there may be a case where a low-resolution image captured by a low-resolution camera is used. Also, as the input image, there may be a case where an image obtained by reducing the image captured by the camera is used. Reducing an image means changing it to a low-resolution image with a size smaller than the original image. For example, an image of 1000×1000 pixels is reduced to 500×500 pixels. For reducing the image, interpolation methods such as bilinear and bicubic are used to calculate the pixel values in the new low-resolution image. Thus, it is expected that the speed of the comparison process will be improved by using a lower-resolution image.

[0015] FIG. 2 is a diagram showing an example of the input image. In FIG. 2, a grid pattern is drawn at the location where the object to be inspected is placed. This pattern can be recognized as a known background. FIG. 3 is a diagram showing an example of the object image extracted from the input image. The extraction unit 101 extracts the object image by image processing such as background difference method using the known background pattern. Specifically, the part different from the background is detected, and a rectangle surrounding that part is extracted. The process of extracting the object image in this way is also called a cutting-out process.

[0016] Note that the method for extracting the item image is not limited to the above. Other image processing techniques such as edge detection, segmentation for recognition by region, object detection using a machine learning model, etc. may be used.

[0017] The acquisition unit 102 acquires image characteristic information regarding the image quality of the item image and similar item information indicating the number of items registered in advance as similar items similar to the collation target item. The image characteristic information regarding the image quality includes the blurring degree of the item image or the size of the item image. The similar item refers to an item whose similarity score with the collation target item is higher than the threshold.

[0018] The blurring degree of the item image is, for example, a value obtained by calculating the variance of the item image. Blurring may occur due to shutter speed, exposure, focal length, movement of the item or the camera, lens quality, etc.

[0019] The size of the item image is, for example, a value obtained by counting the number of pixels of the item image. Since the size varies depending on the type of item, the size of the item image may be different. Also, when multiple items overlap, the item image may become smaller. Also, the shooting distance changes depending on the position of the item with respect to the camera, and the size of the item image may be different.

[0020] The similar item information can be calculated by the acquisition unit 102 from the similarity scores registered in the database in advance. The similarity score is a numerical value indicating the similarity with other collation target items. In the database, for each collation target item, there may be registered the similarity scores with a predetermined number of collation target items, such as the top 30 items in descending order of the similarity score. Alternatively, there may be registered the similarity scores with all other collation target items having a similarity score exceeding the threshold. The acquisition unit 102 calculates the number of similar items whose similarity score exceeds the threshold using these similarity scores, and treats the number of similar items as similar item information. For example, the number of items exceeding the threshold among the top 30 items or the number of items registered due to exceeding the threshold is used as the similar item information.

[0021] The determination unit 103 determines, based on image characteristic information and similar item information, whether to use an item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching the item to be matched with a pre-registered registered image. The high-resolution image shows the item at a resolution high enough to be suitable for matching. For matching, the captured image before it is reduced as an input image may be used as the high-resolution image. If the high-resolution image that is the source of the input image is stored, it will be used. Alternatively, super-resolution technology may be applied. A machine learning-based super-resolution algorithm is applied to a low-resolution item image to generate a high-resolution image. The determination unit 103 makes a determination considering the balance between processing speed and matching accuracy, based on both the image quality of the item image and the number of similar items.

[0022] Generally, the determination unit 103 tends to determine that a higher resolution image should be used when there are many similar items, based on the similar item information. This is because it is preferable to use more detailed features for matching when there are many similar items. Also, the determination unit 103 tends to determine that a higher resolution image should be used when the image is more blurry, based on the image characteristic information. This is because fine details are likely to be lost in blurry images. Furthermore, the determination unit 103 tends to determine that a higher resolution image should be used when the item image is small, because fine details may be lacking in small images.

[0023] In one example, the determination unit 103 may perform the determination using a determination model that has learned the relationship between image characteristic information, similar item information, and whether to use a high-resolution image. To create the determination model, for example, a group of item images for which matching with a high-resolution image is desirable and a group of item images for which a high-resolution image is not necessary are prepared. For each item image, image characteristic information and similar item information of the photographed item are acquired. Then, the image characteristic information and similar item information are input to the determination model. The determination model determines whether or not to use a high-resolution image and outputs the result. Learning is performed using a machine learning algorithm with pre-prepared correct labels. The determination unit 103 inputs the acquired image characteristic information and similar item information for a new item image into the trained determination model. Based on the input information, the determination model determines whether or not to use a high-resolution image. The determination unit 103 performs the determination based on the output result of the model.

[0024] The decision model may use information on the speed and accuracy of the matching process as input. If the processing is too slow, the model is retrained to determine that high-resolution images are not being used. If the matching accuracy is too low, the model is retrained to determine that high-resolution images are being used.

[0025] An example of the operation of the identification support device 10 will be explained using the flowchart in Figure 4. The process in Figure 4 is started, for example, when an item to be inspected is photographed.

[0026] In step S1, the extraction unit 101 extracts an item image, which is the region of the item, from the input image of the item to be inspected. In step S2, the acquisition unit 102 acquires image characteristic information regarding the image quality of the item image, and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched. In step S3, the determination unit 103 determines, based on the image characteristic information and the similar item information, whether to use the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching the item to be matched with a pre-registered image of the item to be matched.

[0027] The determination result from the determination unit 103 is output to another device, such as an item identification device, which identifies the item. In the item identification device, item identification is performed using either an item image or a high-resolution image, using features point comparison or optical character recognition. When a high-resolution image is used, the item identification device extracts the area of ​​the corresponding item from the captured image before it is reduced as an input image and uses it for matching. Alternatively, the item identification device applies a super-resolution algorithm to a low-resolution item image to generate a high-resolution image. The item identification device may also enhance the resolution of a portion of the item. The identification result from the item identification device is fed back to the determination unit 103 and may be used to improve the determination model. The determination model is updated considering the balance between processing time and accuracy.

[0028] According to Embodiment 1, it is possible to improve matching accuracy while suppressing a decrease in matching processing speed. This is because the determination unit 103 makes a judgment based on image characteristic information regarding the image quality of the item image and similar item information, so that high-resolution images are used for matching only when necessary. This avoids unnecessary high-resolution processing and matching using high-resolution images. The determination unit 103 can improve matching accuracy by using high-resolution images in situations where more detailed identification is required, such as when there are many similar items or when the image quality is low. On the other hand, it maintains processing speed by using low-resolution images when this is not the case. In this way, the identification support device 10 can support the identification of items by an item identification device, etc.

[0029] The above describes the case in Embodiment 1 where the determination unit 103 uses image characteristic information regarding the image quality of the item image. The determination unit 103 may further make a determination using image characteristic information regarding the shape of the item image. For example, the aspect ratio of a rectangle may be considered. The determination unit 103 tends to determine that the image should be re-resolution when it is narrow and elongated. In addition, it tends to determine that the image should be re-resolution when the item image has a distinctive shape. Furthermore, the determination unit 103 may also make a determination using the number of item images included in the input image. Generally, the more items there are, the higher the possibility that the worker will place the wrong item, so the determination unit tends to determine that the image should be re-resolution.

[0030] The determination unit 103 may dynamically adjust the threshold for determining whether to use high-resolution images depending on the system usage and time of day. For example, the threshold may be changed to prioritize processing speed when there are many matching requests during a given time period, and to prioritize accuracy when the system is quiet.

[0031] [Embodiment 2] An example of the configuration of the identification support device 12 in this disclosure will be explained using Figure 5. The configuration of the identification support device 12 that is the same as that of the identification support device 10 will not be explained. The identification support device 12 differs from the identification support device 10 of Embodiment 1 in that it includes a matching unit 104, an output unit 105, and a resolution processing unit 106. The identification support device 12 is communicatively connected to the image input device 20, the database 30, and the display 40.

[0032] The image input device 20 is a camera equipped with an imaging unit and an image output unit that transmits the captured image data to the identification support device 12. When the camera is fixed above the inspection workbench, the worker can photograph the items in the basket below the camera by placing the basket containing the items to be inspected on the workbench. The camera is fixed in such a way that it can photograph multiple items simultaneously, even if the basket contains multiple items. Alternatively, the image input device 20 may be a handheld camera, smartphone, or tablet device used by the worker.

[0033] Database 30 stores information about the items to be matched. Specifically, it includes registered images, item information, feature data, similar item information, and registered weight. High-resolution photographic images may be registered as registered images. Item information is basic information such as product name, product code, and price. Feature data is features extracted from the registered images. Features are used for calculating similarity scores and for matching. Similar item information is a list of other items similar to each item and their similarity scores. Registered weight is the weight measured at the time of registration or the standard weight of each item. Database 30 provides this information upon request from the identification support device 12.

[0034] Database 30 may store a list of items to be checked during the inspection process. Each list will record a unique ID (Identifier) ​​to identify the list, the date and time of creation, and information about the items it contains (product code, quantity, etc.). Database 30 may also record information indicating that the inspection of the items on the list has been completed.

[0035] Multiple databases 30 may be provided for each type of data to be stored.

[0036] The display 40 is an output device for displaying the processing results of the identification support device 12. The display 40 has a touch panel function, allowing the operator to directly request additional information, confirm and correct the identification results.

[0037] Figure 6 illustrates an example of registration data registered in database 30. Figure 6 shows an example of registration data for matching items that are compared with items to be inspected. The items recorded are those handled by logistics companies. For each item, the product name, master image, feature quantities, similarity score, and registered weight are registered. The master image is a registered image taken of each matching item.

[0038] The extraction unit 101 may accept the operator's specification of an area of ​​an item after performing the extraction process using image recognition. This is useful in situations where automatic extraction is difficult or when more accurate area specification is required. Based on the operator's specification, the extraction unit 101 deletes incorrectly extracted areas, adds areas that were not extracted, or corrects inappropriate areas.

[0039] The acquisition unit 102 acquires similar item information for the matching items included in the target item list. If the target item list includes multiple items, the acquisition unit 102 acquires similar item information indicating the number of similar items for each item. Alternatively, the acquisition unit 102 may acquire similar item information indicating the number of similar items for the matching item with the largest number of similar items among the items included in the target item list.

[0040] Similar to Embodiment 1, the determination unit 103 makes a determination based on image characteristic information and similar article information. For example, based on the similar article information, the determination unit 103 determines to use a high-resolution image if the list of target articles includes a matching target article for which the number of similar articles exceeds a certain standard.

[0041] In Embodiment 2, if the determination unit 103 determines that a high-resolution image should be used, it determines that the item image should be processed to increase its resolution. For example, the determination unit 103 requests the resolution processing unit 106 to process the item image to increase its resolution. If gradual resolution improvements (e.g., doubling, quadrupling, or octupling the number of pixels) are possible, the determination unit 103 may determine which resolution level to adopt. An appropriate resolution level is adopted based on image characteristic information and similar item information.

[0042] The matching unit 104 can select only items included in the target item list for matching. This eliminates the need for unnecessary matching processes.

[0043] The matching unit 104 performs matching using either an item image or a high-resolution image based on the determination result of the judgment unit 103. First, the matching unit 104 extracts feature quantities from the item image or high-resolution image. Using the extracted feature quantities, it performs matching with registered images in the database 30. It calculates a similarity score with each registered image. The registered image with the highest similarity score is identified as the item to be matched. The matching unit 104 also calculates the confidence level of the matching result. The confidence level is determined, for example, by the similarity score. If the confidence level falls below a certain threshold, it notifies the output unit 105 and prompts the operator to confirm.

[0044] The output unit 105 outputs the matching results from the matching unit 104. The output unit 105 outputs the matching results to, for example, the display 40. The matching results may include the name and quantity of the identified items, the matching confidence level of each item, the location information of the items in the input image, and a notification that there are unidentified items. The output unit 105 may also output whether the input image matches the target item list, which indicates the items to be inspected that should be included in the input image. For example, in the list of identified items, the output unit 105 may highlight correctly identified items in green and extra items not in the target item list in red. Items that are in the target item list but not in the list of identified items may be highlighted in yellow and displayed as missing items.

[0045] The output unit 105 may output information regarding the resolution of the image used for matching, along with the matching result. The information regarding resolution may include, for example, whether a high-resolution image was used for matching by the matching unit 104, and the determination result by the determination unit 103 of whether a high-resolution image should be used. The information regarding resolution may include the resolution of the image used, information about the item to which the high-resolution processing was applied, and the reason for applying the high-resolution processing.

[0046] The resolution processing unit 106 performs a process to increase the resolution of the item image. The resolution processing unit 106 may increase the resolution of parts of the item image that contain the item's logo or text. The resolution processing unit 106 uses a trained resolution enhancement processing model to enhance the resolution. Resolution enhancement processing refers to the process of generating a higher resolution image from a lower resolution image. This process increases the number of pixels in the image, creating an image that contains more detailed information. The resolution enhancement processing model uses a deep learning-based algorithm such as VDSR (Very-Deep Super-Resolution). The resolution processing unit 106 inputs the item image into the model and transmits the resulting high-resolution image to the matching unit 104. Note that the method of resolution enhancement processing is not limited to the above, and other existing techniques such as edge enhancement can be applied.

[0047] An example of the operation of the identification support device 12 will be explained using the flowchart in Figure 8. The process in Figure 8 is initiated, for example, when an operator performs a shooting operation on the image input device 20.

[0048] In step S11, the extraction unit 101 extracts an item image, which is the region of the item, from the input image of the item to be inspected. In step S12, the acquisition unit 102 acquires image characteristic information regarding the image quality of the item image, and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched. In step S13, the determination unit 103 determines, based on the image characteristic information and the similar item information, whether to use the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching the item to be matched with a pre-registered registered image of the item to be matched.

[0049] If it is determined in step S13 to use an item image, in step S14 the matching unit 104 matches the item image with the registered image. If it is determined in step S13 to use a high-resolution image, in step S16 the resolution processing unit 106 processes the item image to increase its resolution. In step S17 the matching unit 104 matches the high-resolution processed item image with the registered image. Then the output unit 105 outputs the matching result. With this, the identification support device 12 completes the process shown in Figure 8.

[0050] The processes from step S11 to step S17 are repeatedly performed for each extracted item image. After processing of all item images is complete, the output unit 105 outputs an overall matching result. This overall result includes a list of all identified items, their respective matching confidence levels, and the status of the high-resolution processing application.

[0051] Figure 7 shows an example of the matching results displayed on the display 40 by the output unit 105. In the upper left, a table representing the contents of the shipping slip is displayed. The shipping slip is an example of a list of items to be inspected, which should be included in the input image. "Shipping Slip 2" contains one item A and two items C. In the lower left, the captured image is displayed as the input image. In the captured image, the extraction results of the item images of the four products are shown by dashed rectangles. The inspection results are displayed in the center of the bottom. The inspection results are an example of the matching results. The inspection results show the products identified from the image and their quantities (two items A and two items C). The inspection results also include a display of errors (one extra item A).

[0052] A message requesting confirmation of the product quantity is displayed in the upper right corner of the screen. The incorrect information may be highlighted on the shipping slip. In the shipping slip table, the inspection quantity "1" for product A is underlined for emphasis.

[0053] The screen may also display the reliability of the matching process. In the example in Figure 7, it is shown that the reliability of the matching process is high. The screen may also display whether an item image or a high-resolution image was used for matching. In the example in Figure 7, it is shown that a low-resolution image was used as an example of an item image. Furthermore, the processing time for matching and other processes may be displayed. Based on this information, the operator can change various settings.

[0054] After reviewing the results in Figure 7, the operator removes one of the excess items A. This action ensures that the actual quantity of goods matches the contents of the shipping slip. The operator presses the "Re-execute" button in the lower left corner of the screen. This causes the image input device 20 to take another picture, and the process in Figure 8 is executed again. Based on the newly captured image, the items are re-identified. If the re-inspection confirms that the inspected items match those listed on the shipping slip and there are no abnormalities, the operator presses the "Next Slip" button. The identification support device 12 confirms and records the inspection results. Furthermore, the output unit 105 displays a list of the next items to be inspected, such as "Shipping Slip 3". The operator places the new items to be inspected in front of the camera.

[0055] According to Embodiment 2, in addition to the effects of Embodiment 1, it is possible to identify items with high accuracy while suppressing a decrease in processing speed. This is because the matching unit 104 selectively uses either an item image or a high-resolution image for matching based on the judgment of the determination unit 103. In this way, matching using a high-resolution image is performed only when necessary. Furthermore, since the output unit 105 outputs the matching results to the display 40, the operator can quickly and accurately confirm the results. In addition, since the resolution processing unit 106 increases the resolution of the item image, the accuracy of identifying similar items is improved. Therefore, the identification support device 12 can support the operator's inspection work.

[0056] <Variation> Embodiment 2 can be modified as follows.

[0057] In Embodiment 2, the resolution processing unit 106 may be provided as needed. If the identification support device 12 does not have a resolution processing unit 106, the resolution enhancement process may be performed in another device connected to the identification support device 12. Alternatively, the matching unit 104 may acquire the high-resolution image before reduction and use it for matching.

[0058] Furthermore, in Embodiment 2, the matching process can be modified as follows. In one example, the matching unit 104 performs a first matching process and a second matching process. The first matching process is a process of matching an item image taken of the item to be inspected with a pre-registered registered image of the item to be matched. At this stage, a lower resolution image is used for all items to perform a high-speed matching. The second matching process is performed after the first matching process. The second matching process matches the registered image with either the item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image. In this example, the determination unit 103 determines whether to use the item image or the high-resolution image for the second matching process.

[0059] The second matching process is performed, for example, when the reliability of the first matching process is determined to be low. Alternatively, the second matching process may be performed when there are similar items with a high degree of similarity to the item identified in the first matching process. The matching unit 104 may use different algorithms for the first and second matching processes. The second matching process performs a more detailed comparison. For example, it performs matching by focusing on finer local features of the items. Based on the determination result of the determination unit 103, the matching unit 104 performs a second matching using high-resolution images for some of the items. Using high-resolution images allows for more accurate analysis of local features. However, performing the second matching process using high-resolution images for all images may reduce processing speed. Based on the image characteristic information and similar item information, if the determination unit 103 determines that the resolution of the item image is sufficient for performing the second matching process, it decides to use the item image for the second matching process as well.

[0060] Figure 9 illustrates another example of operation of the identification support device 12 in this disclosure.

[0061] In step S21, the target item list and the similarity information file corresponding to the target item list are loaded. The target item list is a list of items that should be checked during the inspection process. The target item list includes information such as the name, quantity, and code of each item. The similarity information file is a file that contains detailed information of the matching items registered in the database 30. The similarity information file includes the feature quantities of each matching item, the similarity score between each item, and the registered weight of each item.

[0062] In step S22, the extraction unit 101 extracts an item image, which represents the area of ​​the item, from the input image of the item to be inspected. In step S23, the matching unit 104 performs a first matching process that compares the item image with the registered image. At this time, the matching process is performed efficiently by utilizing the feature quantities and similarity scores contained in the read similarity information file.

[0063] In step S24, the determination unit 103 determines whether the reliability of the result of the matching process performed by the matching unit 104 is higher than the standard. If the reliability of the matching result is high (step S24: Yes), the output unit 105 outputs the matching result from the first matching process in step S28. Then, the identification support device 12 terminates the process shown in Figure 9.

[0064] If the reliability of the matching result of the first matching process is low (step S24: No), the acquisition unit 102 executes step S25. In step S25, the acquisition unit 102 acquires image characteristic information regarding the image quality of the item image, and similar item information indicating the number of similar items that have been pre-registered as similar items to the item to be matched. For example, it acquires similar item information from the similarity information file read in step S21. The acquisition unit 102 may also acquire registered similar item information for matching items that matched the item to be inspected in the first matching process. A matching matching item is a matching item that the matching unit 104 determined to match the item to be inspected because the similarity is above a threshold when comparing the item image and the registered image.

[0065] In step S26, the determination unit 103 determines, based on image characteristic information and similar item information, whether to use an item image or a high-resolution image that shows the item to be inspected at a higher resolution than the item image for matching the item to be matched with a pre-registered registered image. In step S27, the matching unit 104 performs a second matching process that matches either the item image or the high-resolution image with the registered image. Then, in step S28, the output unit 105 outputs the matching result of the second matching process. With this, the identification support device 12 completes the process shown in Figure 9.

[0066] Steps S23 to S28 are repeated for each extracted item image. The output unit 105 aggregates the matching results for all items included in the input image and displays them on the display 40.

[0067] The operation shown in Figure 9 is illustrative, and other variations in the operation of the identification support device 12 are conceivable. For example, step S22 may be executed after the first matching process. In this case, the first matching process in step S23 may be replaced with a process that matches the input image with the registered image. Then, if the confidence level of the matching result is low, step S22 is executed after step S24.

[0068] According to a modified version of Embodiment 2, if the reliability of the first matching process is high, the second matching process can be omitted, thereby preventing a decrease in processing speed. Furthermore, in the second matching process, a decrease in processing speed can be prevented by performing matching using high-resolution images only when it is deemed necessary.

[0069] [Embodiment 3] An example of the configuration of the identification support device 13 will be explained using Figure 10. The configuration of the identification support device 13 in Embodiment 3 that is the same as that of the identification support device 10 in Embodiment 1 and the identification support device 12 in Embodiment 2 will not be explained. The identification support device 13 differs from the identification support device 10 in Embodiment 1 in that it includes a matching unit 104, an output unit 105, and a character recognition processing unit 107. The character recognition processing unit 107 may be provided as needed. Embodiment 3 can be appropriately combined with Embodiments 1 and 2 and their modified versions. For example, the identification support device 13 may include a resolution processing unit 106.

[0070] The identification support device 13 is connected to the image input device 20, the database 30, and the display 40 in a communicative manner. Furthermore, the identification support device 13 is connected to the weight measuring device 21 and the illuminance measuring device 22 in a communicative manner.

[0071] The weight measuring device 21 includes a weight measuring unit that measures the weight of the item to be inspected after it has been photographed, and a weight information output unit that outputs weight information indicating the measured weight. The weight is measured during the inspection process and input to the identification support device 13. For example, if the image input device 20 is fixed above the inspection workbench, the weight measuring device 21 can be installed on the workbench. The worker can start taking images and measuring the weight by placing a basket containing the items to be inspected on the workbench. The weight measuring device 21 calculates the weight of the items in the basket by subtracting the weight of the basket from the total weight of the items and the basket.

[0072] The illuminance measuring device 22 includes an illuminance measuring unit that measures the illuminance of the shooting environment of the item to be inspected, and an illuminance information output unit that outputs illuminance information indicating the measured illuminance.

[0073] The weight measuring device 21 and the illuminance measuring device 22 may be provided as needed. If these devices are not provided, weight information and illuminance information will not be used. In addition, other devices that measure the conditions of the shooting environment of the items to be inspected may be connected to the identification support device 13 in a communication manner.

[0074] The character recognition processing unit 107 performs the process of recognizing characters contained in the image of an object using optical character recognition. A known OCR (Optical Character Recognition) engine is used for character recognition. The OCR engine may output a confidence level for character recognition. The confidence level is output for each character. Factors that are thought to affect the confidence level include the size of the characters, the presence of characters that are difficult to distinguish, the contrast of the image, and the tilt of the object. If the characters are blurry, it can be determined that the image as a whole is blurry.

[0075] The acquisition unit 102 acquires identification-related information in addition to image characteristic information and similar item information. The identification-related information is used to determine the need for more accurate identification of the item or high-resolution processing. The identification-related information includes, for example, weight information, environmental information, or the results of optical character recognition. Weight information represents data related to the weight of the item. Environmental information is data that shows the surrounding conditions when the item was photographed and represents factors that affect image quality, such as lighting conditions. The results of optical character recognition include characters extracted from the item image and the confidence level of character recognition. Each piece of information may be used individually or in combination.

[0076] In one example, the acquisition unit 102 acquires weight information relating to at least one of the measured weight of the item to be inspected that has been photographed, or the registered weight that has been pre-registered for the item to be matched. The weight information may also include a relative value of the weight difference between the measured weight and the registered weight. First, the acquisition unit 102 calculates the difference between the total weight of the measured item to be inspected and the total registered weight of the items included in the item list. The acquisition unit 102 obtains the value obtained by dividing this difference by the total registered weight as the relative value of the weight difference. This value represents the ratio of the actual weight error to the expected total weight. If the error is large, it may indicate that the wrong item has been placed or that there is an error in the identification result.

[0077] Alternatively, the acquisition unit 102 acquires weight information from at least one of the measured weight of the photographed item to be inspected, or the registered weight pre-registered for the item to be matched. When the items to be matched include lightweight items, matching by weight is difficult, so it is necessary to improve the accuracy of matching by image.

[0078] In one example, the acquisition unit 102 acquires environmental information of the environment in which the item was photographed. The environmental information includes illuminance information indicating the illuminance measured by the illuminance measuring device 22. Information other than illuminance information may also be used as environmental information, such as the presence and direction of shadows, the complexity of the background, the shooting angle, and the type and settings of the camera. For example, the acquisition unit 102 detects the presence and direction of shadows by analyzing the luminance distribution of the input image. The acquisition unit 102 acquires sensor data of the shooting angle and the settings and type of the camera from the camera metadata. Alternatively, the acquisition unit 102 may acquire environmental information that has been pre-registered for each inspection work location.

[0079] Furthermore, in one example, the acquisition unit 102 also acquires the results of optical character recognition of the item image. The results of optical character recognition include, for example, the recognized string, the recognition confidence level of each character, the position information of the characters, and the size of the characters.

[0080] The determination unit 103 uses image characteristic information, similar item information, and identification-related information to determine whether or not to use a high-resolution image. The determination unit 103 can make a comprehensive judgment by combining various types of information, but the methods of utilizing each type of information will be explained individually below.

[0081] In one example, the determination unit 103 determines whether to use a high-resolution image based on weight information. If the difference between the measured weight and the registered weight is large, there is a high possibility of an error in the matching, so it is determined to use a high-resolution image. For example, a high-resolution image is used when the assumed value of the weight difference exceeds a threshold (e.g., 10%). If the item to be matched is lightweight, it is difficult to match by weight, so it is determined to use a high-resolution image to improve the accuracy of image matching.

[0082] In one example, the determination unit 103 determines whether to use a high-resolution image based on environmental information. If the illumination is low, for example, if the illumination is below a certain value, the determination unit 103 determines to use a high-resolution image. In addition, if a strong shadow is detected in the image, if the background is complex, or if the shooting angle deviates significantly from the vertical, the determination unit 103 also determines to use a high-resolution image.

[0083] In one example, the determination unit 103 determines whether to use a high-resolution image based on the results of optical character recognition. For example, the determination unit 103 uses the confidence level of character recognition to make the determination. Based on the confidence level of character recognition, the state of the image can be determined. If the percentage of characters output with a low confidence level exceeds 50%, the determination unit 103 determines that the image is in poor condition. If the percentage of characters output with a low confidence level is between 30% and 50%, the determination unit 103 determines that the image is in normal condition. If the percentage of characters output with a low confidence level is between 0% and 30%, the determination unit 103 determines that the image is in good condition. If the image is in poor condition, the determination unit 103 determines to use a high-resolution image. In addition, if the average confidence level of character recognition is below a threshold, the determination unit 103 may determine to use a high-resolution image. Furthermore, if the average size of the characters is below a certain value, or if the recognized string differs significantly from the expected string registered in the database, the determination unit 103 determines to use a high-resolution image.

[0084] When the determination unit 103 uses the determination model, it modifies the determination model of Embodiment 1 to accept weight information, environmental information, and the results of optical character recognition as inputs, in addition to image characteristic information and similar item information. The determination model comprehensively evaluates the input information and outputs whether or not high-resolution images are necessary.

[0085] The matching unit 104 may compare the item to be inspected with the item to be matched registered in the database using weight information. The matching unit 104 may perform matching using weight information when the determination unit 103 determines that a high-resolution image should be used. Furthermore, the matching unit 104 may perform matching using weight information when the second matching process described in the modified embodiment 2 is executed. In other words, matching using weight information is not performed in the first matching process.

[0086] The matching unit 104 may adjust the parameters of the matching algorithm based on environmental information. For example, the matching unit 104 may lower the feature point threshold in low-light environments. The matching unit 104 also uses the OCR results obtained from the character recognition processing unit 107 as auxiliary information for matching. The matching unit 104 compares the recognized string with the character information of the registered image to improve the accuracy of the matching.

[0087] The output unit 105 outputs the verification result from the verification unit 104. The output unit 105 may also display the inspection result, including weight information, on the display 40, as shown in Figure 7. Figure 7 shows that the weight information includes the measured total weight, the target total weight based on the target item list, and the total weight error (500 gram deficit) between the measured total weight and the target total weight.

[0088] An example of the operation of the identification support device 13 will be explained using the flowchart in Figure 11.

[0089] In step S31, the extraction unit 101 extracts an item image, which is the region of the item, from the input image of the item to be inspected. In step S32, the acquisition unit 102 acquires image characteristic information, identification-related information, and similar item information. In step S33, the determination unit 103 determines, based on the image characteristic information, identification-related information, and similar item information, whether to use the item image or the high-resolution image for matching with a pre-registered registered image of the item to be matched.

[0090] In step S34, the matching unit 104 compares the registered image with either the item image or the high-resolution image. In step S35, the output unit 105 outputs the matching result. With this, the identification support device 13 completes the process shown in Figure 11.

[0091] According to Embodiment 3, the determination unit 103 uses identification-related information to determine whether to use a high-resolution image, enabling more situation-appropriate judgments. For example, by considering weight information and environmental information, an appropriate processing method can be selected according to the characteristics of the item and the shooting conditions. In addition, since the matching unit 104 uses identification-related information for matching, the identification of similar items and the accuracy of identification when shooting conditions are poor are improved.

[0092] The above describes a method using relative weight differences, but the acquisition unit 102 may also acquire a simple weight difference as weight information. For example, the acquisition unit 102 acquires the difference between the measured weight of the photographed item and the total registered weight. The difference is not divided by the total registered weight. If the absolute value of the difference is large, the determination unit 103 determines that a high-resolution image should be used. Also, if the weight of each item is measured, the acquisition unit 102 may acquire the weight difference of each individual item as weight information. The acquisition unit 102 acquires the difference between the measured weight of one item to be inspected and the registered weight of one item to be compared.

[0093] [Examples of application] In the above embodiment, an example of using an identification support device to assist a logistics company with inspection was described. By applying the identification support device of this disclosure to other applications, efficiency can be improved in each respective field. For example, when the item identification device is applied to drug picking in a medical setting, the work of collecting drugs by placing them in baskets based on prescriptions can be made more efficient. Also, when applied to counting work parts in the manual assembly phase of a factory, the work of collecting work parts can be made more efficient by placing them in baskets and performing verification. Furthermore, when applied to sorting agricultural products, agricultural products harvested on a farm can be collected in baskets and sorted according to quality and type. In this way, in each field, inspection becomes faster and more accurate compared to visual inspection work. Moreover, by further applying the identification support device of this disclosure, high-resolution images can be used as needed, thereby suppressing a decrease in processing speed while improving the accuracy of item identification.

[0094] [Hardware configuration] In each of the embodiments described above, the components of the identification support devices 10, 12, and 13 represent functional blocks. Some or all of the components of the identification support devices 10, 12, and 13 may be implemented by any combination of the computer 500 and a program.

[0095] Figure 12 is a block diagram showing an example of the hardware configuration of computer 500. Referring to Figure 12, computer 500 includes, for example, a processor 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0096] The processor 501 controls the entire computer 500. An example of a processor 501 is a CPU (Central Processing Unit). The number of processors 501 is not particularly limited; there may be one or more processors 501.

[0097] Program 504 includes instructions for implementing the functions of identification support devices 10, 12, and 13. Program 504 is pre-stored in ROM 502, RAM 503, and storage device 505. The processor 501 implements the functions of identification support devices 10, 12, and 13 by executing the instructions contained in program 504. RAM 503 may also store data processed by the functions of identification support devices 10, 12, and 13.

[0098] The drive device 507 reads and writes to the recording medium 506. The communication interface 508 provides an interface with the communication network. The input device 509 is, for example, a mouse, keyboard, touch panel, etc., and receives information input from the operator, etc. The output device 510 is, for example, a display, and outputs (displays) information to the operator, etc. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects each of these hardware components. The program 504 may be supplied to the processor 501 via the communication network, or it may be stored in the recording medium 506 beforehand, read by the drive device 507, and supplied to the processor 501.

[0099] Note that the hardware configuration shown in Figure 12 is an example, and other components may be added, or some components may be omitted.

[0100] There are various modifications to the implementation of the identification support devices 10, 12, and 13. For example, the identification support devices 10, 12, and 13 may be implemented by any combination of different computers and programs for each component. Alternatively, the multiple components of the identification support devices 10, 12, and 13 may be implemented by any combination of a single computer and program.

[0101] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with one another, as long as they do not depart from the scope of the present disclosure.

[0102] Some or all of the above embodiments may be described as follows, but are not limited to the following:

[0103] [Note 1] An extraction means for extracting an item image, which represents the area of ​​an item, from an input image of an item to be inspected, An acquisition means for acquiring image characteristic information relating to the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched, A determination means that determines, based on the image characteristic information and the similar article information, whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. An identification support system equipped with the following features.

[0104] [Note 2] The system further comprises a matching means that performs a first matching process for matching the article image with the registered image, and a second matching process performed after the first matching process for matching either the article image or the high-resolution image with the registered image. The determination means determines whether to use the article image or the high-resolution image for the second matching process. The identification support system described in Appendix 1.

[0105] [Note 3] The acquisition means acquires the similar item information registered for the item to be matched as an item matching the item to be inspected in the first matching process. The identification support system described in Appendix 2.

[0106] [Note 4] The system includes output means that outputs whether the input image matches the list of target items that should be included in the input image based on the matching result. The identification support system described in Appendix 2 or 3.

[0107] [Note 5] If the determination means determines that the high-resolution image should be used, it determines that the image of the item should be processed to increase its resolution. The system further comprises resolution processing means for increasing the resolution of the aforementioned article image. An identification support system as described in any one of the appendices 1 to 4.

[0108] [Note 6] The acquisition means further acquires weight information relating to at least one of the measured weight of the photographed inspection target item or the registered weight that has been previously registered for the matching target item. The determination means determines whether to use the high-resolution image based on the weight information. An identification support system as described in any one of the appendices 1 to 5.

[0109] [Note 7] The acquisition means further acquires environmental information of the environment in which the item was photographed. The determination means determines whether to use the high-resolution image based on the environmental information. An identification support system as described in any one of the appendices 1 to 6.

[0110] [Note 8] The acquisition means further acquires the results of optical character recognition of the image of the article, The determination means determines whether to use the high-resolution image based on the optically recognized character string or the size of the characters. An identification support system as described in any one of the appendices 1 to 7.

[0111] [Note 9] From the input image of the item to be inspected, the item image, which represents the area of ​​the item, is extracted. Image characteristic information regarding the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched are obtained. Based on the image characteristic information and the similar article information, it is determined whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. Identification support method.

[0112] [Note 10] From the input image of the item to be inspected, the item image, which represents the area of ​​the item, is extracted. Image characteristic information regarding the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched are obtained. Based on the image characteristic information and the similar article information, it is determined whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. A program that instructs a computer to perform a process.

[0113] Some or all of the configurations described in Appendices 2-8, which are dependent on Appendice 1 above, may also be dependent on Appendices 9-10 in the same manner as in Appendices 2-8. Not limited to Appendices 1 and 9-10, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording devices or systems for recording software, without departing from the embodiments described above. [Explanation of symbols]

[0114] 10, 12, 13 Identification support device 101 Extraction part 102 Acquisition Department 103 Judgment section 104 Verification Unit 105 Output section 106 Resolution Processing Unit 107 Character Recognition Processing Unit 20 Image input device 21 Weight measuring device 22 Illuminance measuring device 30 databases 40 displays

Claims

1. An extraction means for extracting an item image, which represents the area of ​​an item, from an input image of an item to be inspected, An acquisition means for acquiring image characteristic information relating to the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched, A determination means that determines, based on the image characteristic information and the similar article information, whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. An identification support system equipped with the following features.

2. The system further comprises a matching means that performs a first matching process for matching the article image with the registered image, and a second matching process performed after the first matching process for matching either the article image or the high-resolution image with the registered image. The determination means determines whether to use the article image or the high-resolution image for the second matching process. The identification support system according to claim 1.

3. The acquisition means acquires the similar item information registered for the item to be matched as an item matching the item to be inspected in the first matching process. The identification support system according to claim 2.

4. The system includes output means that outputs whether the input image matches the list of target items that should be included in the input image based on the matching result. The identification support system according to claim 2.

5. If the determination means determines that the high-resolution image should be used, it determines that the image of the item should be processed to increase its resolution. The system further comprises resolution processing means for increasing the resolution of the aforementioned article image. The identification support system according to any one of claims 1 to 3.

6. The acquisition means further acquires weight information relating to at least one of the measured weight of the photographed inspection target item or the registered weight that has been previously registered for the matching target item. The determination means determines whether to use the high-resolution image based on the weight information. The identification support system according to any one of claims 1 to 4.

7. The acquisition means further acquires environmental information of the environment in which the item was photographed. The determination means determines whether to use the high-resolution image based on the environmental information. The identification support system according to any one of claims 1 to 4.

8. The acquisition means further acquires the results of optical character recognition of the image of the article, The determination means determines whether to use the high-resolution image based on the optically recognized character string or the size of the characters. The identification support system according to any one of claims 1 to 4.

9. From the input image of the item to be inspected, the item image, which represents the area of ​​the item, is extracted. Image characteristic information regarding the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched are obtained. Based on the image characteristic information and the similar article information, it is determined whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. Identification support method.

10. From the input image of the item to be inspected, the item image, which represents the area of ​​the item, is extracted. Image characteristic information regarding the image quality of the aforementioned article image, and similar article information indicating the number of similar articles that have been pre-registered as similar articles to the article to be matched are obtained. Based on the image characteristic information and the similar article information, it is determined whether to use the article image or a high-resolution image that shows the article to be inspected at a higher resolution than the article image for matching the article to be matched with a pre-registered registered image of the article to be matched. A program that instructs a computer to perform a process.

Images