Information processing system and its control method, program

Abstract

To enable accurate recognition of an object.SOLUTION: Provided are storing means that stores registered images regarding objects and associated with the kinds of the objects, specifying means that specifies, using information expressing the size of a region related to the object extracted from a captured image and a predetermined condition related to the size, the registered image that satisfies the predetermined condition among the registered images, and kind specifying means that specifies the kind of the object related to the region on the basis of the registered image specified by the specifying means. The specifying means uses different conditions when the shape of the object related to the region is a first shape and when the shape is other than the first shape.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a technique for recognizing an object to be recognized included in an image using the image.

Background Art

[0002] Conventionally, a technique is known in which a learned model is generated by machine learning using an image (training image, teacher data) including an object to be identified, and the object included in the image is recognized by inputting the image into the generated learned model.

[0003] Patent Document 1 discloses recognizing an object using a machine learning model, automating the determination of the edge detection direction and the determination of the region where edge processing should be performed, and measuring the shape of the object by high-precision edge extraction such as in subpixel units.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] When outputting the recognition result of an object, if the appearances of the objects are similar, it is necessary to distinguish them by size. However, depending on the placement state of the object, an appropriate size may not be detected. For example, if the object is placed obliquely, it will affect the detection of the size.

[0006] On the other hand, depending on the shape of the object, an appropriate size may be detected regardless of the placement state. Therefore, it is necessary to change the criteria for detecting the size according to the shape of the object.

[0007] The patent documents do not disclose how to change the criteria for detecting size depending on the shape of an object.

[0008] Therefore, the present invention aims to provide a mechanism that enables more accurate recognition of objects. [Means for solving the problem]

[0009] To solve the above problems, the present invention provides A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The aforementioned identification means is characterized by using different conditions depending on whether the shape of the object relating to the region is a first shape or a shape other than the first shape. [Effects of the Invention]

[0010] According to the present invention, objects can be recognized with greater accuracy. [Brief explanation of the drawing]

[0011] [Figure 1] This figure illustrates a system to which the information processing device according to this embodiment can be applied. [Figure 2] This figure shows an example of the hardware configuration of various devices. [Figure 3] This flowchart shows an example of AI inference. [Figure 4] This is a diagram illustrating an example of tableware detection. [Figure 5] This is a diagram illustrating an example of a tableware shape pattern. [Figure 6]This is a diagram for explaining an example of normalizing the orientation of tableware. [Figure 7] This is a diagram for explaining an example when the input image of tableware is incomplete. [Figure 8] This is a diagram for explaining an example of comparing the long sides of tableware images. [Figure 9] This is a diagram for explaining an example of classification results based on similarity.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] First, referring to FIG. 1, an example of the configuration of an information processing system in an embodiment of the present invention will be described.

[0014] In the information processing system of the present invention, a cafeteria settlement lane 102 composed of a camera 103, a display 104, and a cash register 105 is communicably connected to a client terminal 101 via a network 107 (for example, Ethernet) from a predetermined controller 106 (for example, a PoE hub). Note that a plurality of cafeteria settlement lanes 102 may be connected to the client terminal 101.

[0015] The camera 103 is installed at a position where it can photograph the entire tray on the cash register 105.

[0016] On the cash register 105, a tray with tableware after a meal is placed for accounting. Note that the tray with tableware may also be in the state before the meal.

[0017] The client terminal 101 is, for example, a personal computer (hereinafter referred to as a PC), which identifies tableware from the image captured by the camera 103 and performs processes such as settlement. The client terminal 101 uses the technology of deep metric learning to identify the type of tableware placed on the cash register 105.

[0018] Deep distance learning is a method that extracts only the features of an image, calculates an image feature vector from the extracted features using an algorithm, and measures the distance between these vectors to determine which product is the closest match. Sample images are prepared in advance, and feature vectors are extracted from each image. For the input image, the distance between each sample image and its feature vector is measured, and the sample with the closest distance is determined to be of the same type. In this example, deep distance learning is used for explanation, but other methods such as Deep Learning Classification may also be used.

[0019] The display 104 displays the payment information processed by the client terminal 101 and instructs the person who paid for the meal to settle the bill. The display 104 may also display the video feed from the camera 103.

[0020] Next, with reference to Figure 2, an example configuration of a client terminal 101, which is an example of a device to which the present invention can be applied, is shown.

[0021] In Figure 2, the CPU 201, memory 202, non-volatile memory 203, image processing unit 204, display 205, operation unit 206, recording medium interface 207, external interface 209, and communication interface 210 are connected to the internal bus 250. Each component connected to the internal bus 250 is configured to exchange data with each other via the internal bus 250.

[0022] Memory 202 consists of, for example, RAM (volatile memory using semiconductor elements). The CPU 201 controls various parts of the client terminal 101 using memory 202 as work memory, according to a program stored in, for example, non-volatile memory 203. Non-volatile memory 203 stores image data, audio data, other data, and various programs for the operation of the CPU 201. Non-volatile memory 203 consists of, for example, a hard disk (HD) or ROM.

[0023] The image processing unit 204 performs various image processing operations on image data stored in the non-volatile memory 203 and recording medium 208, video signals acquired via the external I / F 209, image data acquired via the communication I / F 210, and captured images, based on the control of the CPU 201. The image processing operations performed by the image processing unit 204 include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 204 may be composed of dedicated circuit blocks for performing specific image processing operations. Depending on the type of image processing, the CPU 201 may also perform image processing according to a program without using the image processing unit 204. The process of recognizing the object to be recognized (tableware) from the image is performed by the CPU 201 in cooperation with the image processing unit 204.

[0024] The display 205 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 201. The CPU 201 generates display control signals according to the program and controls various parts of the client terminal 101 to generate and output video signals for display on the display 205. The display 205 displays images based on the output video signals. The client terminal 101 itself is configured to have an interface for outputting video signals for display on the display 205, and the display 205 may be an external monitor (such as a television).

[0025] The operation unit 206 is an input device for receiving user input, including text input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel is configured planarly by being superimposed on the display 205, and is an input device that outputs coordinate information corresponding to the position of contact.

[0026] The recording medium interface (I / F207) allows for the insertion of recording media 208 such as memory cards, CDs, and DVDs. Based on the control of the CPU 201, it reads data from the inserted recording media 208 and writes data to the recording media 208. The external interface (I / F209) is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting video and audio signals. The communication interface (I / F210) is an interface for communicating with external devices and the internet 211 to send and receive various data such as files and commands.

[0027] The camera unit 212 is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal.

[0028] Next, referring to Figure 3, an example of the tableware recognition process in this embodiment is shown. This process is the inference phase using a trained model and is performed when a customer in the cafeteria uses the cafeteria payment lane 102. Note that each step is executed by the CPU 201.

[0029] In S301, the CPU 201 uses camera 103 to photograph the area around the payment counter 105. When photographing the payment counter with camera 103, it may either continuously photograph, or it may start photographing only when it detects any moving object within the photographic range.

[0030] In S302, CPU201 performs a tray placement determination process to determine whether a tray is placed within a predetermined range from the captured image. If it is determined in S303 that a tray is placed, the dish position detection process in S304 is performed; if it is determined that no tray is placed, the tray placement determination in S302 is performed again.

[0031] In S304, CPU201 corrects distortion in the captured image. Distortion in the image prevents the proper size of tableware from being detected, so correction is performed. For example, even if the pattern and shape of tableware are the same, their sizes may differ, so distortion correction is performed so that they can be detected at the correct size. Distortion correction is a process that corrects tilt, curvature, and perspective in the vertical and horizontal directions.

[0032] In S305, CPU201 extracts and acquires the region that includes the entire tray from the captured image, as shown in Figure 4.

[0033] In S306, CPU201 detects the position and shape of the dishes on the tray. Figure 5 shows an example of the detected shapes. Detection in this step is possible by training the AI ​​model with rectangular coordinate information and shape tags during the training of the dishes. Rectangular coordinate information is information that is pre-trained to set circumscribing rectangles 404a to 404d around the dishes. By training the AI ​​model with rectangular coordinate information along with the dishes, it becomes possible to extract and acquire images of each dish by circumscribing the rectangle, as described below.

[0034] First, let's explain the detection of the position of the tableware. The CPU 201 extracts an image of each piece of tableware from the image containing the tableware captured by the camera 103 using a bounding rectangle. Specifically, it detects the region of the tableware from the image captured by the camera 103. This detection process is different from the detection of the type of tableware, which will be described later. The type is unknown, but it is a detection process that can determine that there is tableware (or an object that is not a tray). For the region of the tableware detected in this detection process, a rectangle (hereinafter referred to as the bounding rectangle) is set so as to be tangent to the outer shape of the tableware. A partial image of the region of the set bounding rectangle (i.e., an image containing a single piece of tableware) is extracted from the original image and obtained. In the following, the extracted images of each piece of tableware will be described as input images.

[0035] Figure 4 shows an example of an image taken by camera 103. Image 401 shows a tray 402 and dishes 403a to 403d placed on the checkout counter 105. The position of the dishes on the tray is detected, and the circumscribing rectangles 404a to 404d are calculated for each dish. Note that Figure 4 is an example of an image of dishes without leftover food, but if there is leftover food, the leftover food will be visible inside each dish.

[0036] Next, we will explain how to detect the shape of tableware. The shape of tableware is classified using tags such as circle 501 (an example of the first shape in this invention), ellipse 502, square 503, and undefined 504. As a classification criterion for tagging, if the shape is circular and each side of the image or tableware is approximately the same length, it is tagged as circle 501. Even if it is a polygon, if each side can be considered to be of the same length, such as an octagon, it may also be tagged as circle 501. Among the circular shapes, if it does not fit the description of a circle, it is tagged as ellipse 502. Also, for square 503, it does not matter whether it is a square or a rectangle. Tableware with special shapes other than circle 501, ellipse 502, and square 503 is tagged as undefined 504. By training the system with tableware tagged according to these classification criteria, it becomes possible to detect the shape of tableware.

[0037] When generating a pre-trained model or performing inference using the generated pre-trained model, it is common practice to enlarge or reduce the training images and the images to be inferred to unify their image sizes before training and inference. However, if the image sizes are unified, it becomes difficult to detect the shape by looking at the side lengths of the images. Therefore, to maintain the aspect ratio of the tableware images, black bars are added to the top and bottom or left and right sides of the images during training. In other words, the aspect ratio of the original image is not changed, and the image size to be input to the AI ​​is adjusted by adding black bars to the top and bottom or left and right sides of the image. This allows training to be performed without distorting the aspect ratio, making it possible to detect the shape from the side lengths of the tableware.

[0038] In S307, CPU201 determines whether there has been any change in the dishes on the tray. That is, it determines whether the tray or dishes have been moved while this flowchart is being executed. If there has been a change in the dishes, it returns to S301 and takes another picture; otherwise, it proceeds to S308.

[0039] In S308, CPU201 excludes detected objects outside the tray within the imaging range. Specifically, the S306 process may sometimes detect objects outside the tray. Therefore, it excludes these erroneously detected objects outside the tray. For example, it excludes hands or mobile devices such as smartphones placed near the tray.

[0040] In S309, CPU201 retrieves one dish from those detected in S306 that has not yet undergone processing in S309-S320. Processing in S309-S320 is then performed on the retrieved dish.

[0041] In S310, CPU201 determines whether the shape of the tableware obtained in S309 is a perfect circle or not. If it is a perfect circle, proceed to S311; otherwise, proceed to S313.

[0042] In S311, CPU201 extracts and acquires only the input image of the tableware to be processed from the image acquired in S305.

[0043] In S312, CPU201 extracts features from the acquired input image to be used for comparison with the registered image.

[0044] Meanwhile, in S313, CPU201 performs a process to normalize the orientation of the tableware. This will be explained using Figure 6. In the case of tableware that is not perfectly round, as shown in the pre-normalization image 601, when cropped with a bounding rectangle, the difference in size between the tableware and the bounding rectangle becomes large (the margin becomes large). When these differences become large, it becomes difficult to identify the tableware using size information, which leads to a decrease in accuracy. Therefore, it is necessary to normalize the tableware to reduce the difference in size between the tableware and the bounding rectangle.

[0045] Multiple images are obtained from the input image with slightly different rotation angles to generate a grid image 602. From the grid image, the dish image with the smallest area of ​​the bounding rectangle is extracted, that is, the dish image where the rectangle is almost the same size as the dish and the margin is smallest. The extracted image is obtained as normalized image 603. Since the input image of the dish cropped by the bounding rectangle is used for inference, if the difference in size between the dish and the rectangle is large, dishes of different sizes will be detected. Therefore, by normalizing the orientation of the dish, the difference in size between the dish and the rectangle can be reduced, and more accurate detection results can be output.

[0046] In S314, CPU201 obtains both the pre-normalization image and the normalized image of the tableware to be processed from the images acquired in S305.

[0047] In S315, CPU201 extracts features from both the pre-normalization image 601 and the normalized image 603, and calculates their average. If only the normalized image 603 is used, there is a possibility that features may not be extracted properly due to factors such as lighting conditions, so features are extracted from both the pre-normalization and normalized images, and their average value is calculated.

[0048] In S316, CPU201 compares the features of the registered images of the tableware with the features of the input images extracted in S312 or S315 to calculate the similarity. Registered images refer to images used to train the AI ​​model. Multiple registered images are required for each piece of tableware. The AI ​​model is used beforehand to extract the features of the registered images, and the registered images and their features are associated and stored in memory. The similarity is calculated by comparing the features of the input images extracted in S312 or S315 with the features of the registered images.

[0049] In S317, CPU201 excludes registered images from the candidates if the calculated similarity score is below a threshold. For example, if the threshold is set to 0.85, registered images with a similarity score lower than that will be excluded as candidates.

[0050] In S318, CPU201 compares information indicating the size of the registered image and the input image (in this embodiment, "length of the longest side" is used as an example) and removes registered images that do not meet the criteria (do not satisfy the conditions) from the candidates.

[0051] Here, we will explain the effect of using the longer side as information to indicate size. As shown in Figure 7, depending on how the tray is placed when the image is taken, the image may be cut off. As shown in Figure 7(a), if the shorter side is used as the comparison target, when the input image of a perfectly round dish is cut off, the cut-off part will be recognized as the shorter side and used as the comparison target because the length of each side of a perfect circle is uniform. In that case, the shorter side 701 of the input image will be compared with the shorter side 702 of the registered image, and even though they are actually the same size, they will be judged as being of different sizes.

[0052] In contrast, if the longer side is used as the comparison point, even if the input image of a perfectly round dish is cut off, the longer side of the uncut portion will be used. That is, as shown in Figure 7(b), the upper longer side 703 of the input image that is not cut off will be used as the comparison point. By using the longer side 703 of the input image in this way, even if the image is cut off, the difference when compared with the longer side 704 of the registered image becomes smaller, making it possible to determine the size with greater accuracy.

[0053] The process in S318 will be explained in detail using Figure 8. The value used as the basis for comparison is changed depending on whether the shape is a perfect circle or not. First, let's explain the case where the target dish 801 is a perfect circle. From the input image, the longest side 803 of the bounding rectangle 802 of the dish 801 is calculated. Similarly, from the registered image, the longest side 806 of the bounding rectangle 805 of the dish 804 is calculated. Registered images whose difference from these values ​​falls outside a predetermined standard value (for example, a range of 0.9 to 1.1 times the length of the longest side of the input image) are excluded from the candidates. For example, if the longest side 803 of the input image is 20 cm, registered images that fall outside the range of 18 cm to 22 cm are excluded from the candidates.

[0054] Next, we will explain the case where the target dish 811 is not a perfect circle. From the input image, the longer side 813 of the circumscribing rectangle 812 of the dish 811 is calculated. Similarly, from the registered image, the longer side 816 of the circumscribing rectangle 815 of the dish 814 is calculated. Registered images whose difference falls outside a predetermined standard value (for example, a range of 0.6 to 1.4 times the length of the longer side of the input image) are excluded from the candidates. For example, if the longer side 813 of the input image is 20 cm, registered images that fall outside the range of 12 cm to 28 cm are excluded from the candidates.

[0055] Note that the reference value is chosen to achieve higher accuracy during inference, so the value presented here is merely an example. While multiples are used to set the reference value, pixel values ​​could also be used. Registered images whose pixel value difference from the input image exceeds a predetermined reference value are excluded from the candidates. For example, registered images that are 50 pixels or more away from the input image are excluded. Once the pixel value is calculated, it can be converted to millimeters, making it possible to detect the size of tableware. In this case, as with the case using multiples, the reference value is changed according to the shape.

[0056] Here, we will explain why different reference values ​​are used for the case of a perfect circle and for non-circular shapes (the first shape and shapes other than the first shape) (the reference value for non-circular shapes is made looser). In the case of a perfect circle, regardless of the orientation of the dish, the length of each side of the circumscribed rectangle is approximately equal to the diameter of the dish. Therefore, the error between the circumscribed rectangle in the input image and the circumscribed rectangle in the registered image is small. Also, as mentioned above, even if the image is cut off, the error does not become large by using the length of the longer side as the comparison point. In other words, even if the range (condition) of the reference value is set narrowly, candidate registered images can be appropriately identified. On the other hand, in the case of non-circular shapes, if the dish is placed at an angle, the circumscribed rectangle will be set to be larger than the dish. Therefore, when identifying candidate registered images, it is necessary to set the range of the reference value so that registered images that are a certain length shorter than the length of the longer side of the input image can also be identified. Note that although normalization processing is performed in S313, it is often only possible to normalize up to the point where the dish is placed at some angle, so it is necessary to set the reference value wide to avoid omissions from the candidates. Furthermore, if the longer side is cut off as shown in Figure 7(c), the length of the longer side of the input image will be shorter than its original length. Therefore, it is necessary to set the reference value range wider so that registered images longer than the input image can be identified.

[0057] Thus, in order to properly identify candidate registered images and contribute to improving the accuracy of the final inference results, it is necessary to use different ranges for reference values ​​for perfect circles and non-perfect circles.

[0058] In this embodiment, we have used the comparison of the lengths of the longer sides as an example, but we are not limited to this. Any method of comparing information about the size of tableware, such as comparing the lengths of corresponding sides of the input image and the registered image, comparing the lengths of the diagonals, or comparing the areas, may be used. The results calculated using a different method may be used to narrow down the candidate registered images based on criteria according to the shape.

[0059] In S319, CPU201 determines whether there are any remaining candidates as a result of processing in S317 and S318. If there are candidates remaining, proceed to S320; otherwise, proceed to S321.

[0060] In S320, CPU201 determines the class with the highest number of occurrences from the top-ranked similarity items as the classification result. The tableware candidates are sorted in descending order of similarity calculated in S316, and the tableware with the highest similarity (for example, the top 10 similarity items) is extracted. The tableware with the highest number of detections from the extracted tableware is determined as the classification result. Now, let's explain the number of detections. As mentioned above, multiple images are registered for each tableware item. For example, for the registered image of a soba zaru (bamboo basket), similarity is calculated for each of the multiple registered images, so there may be multiple soba zaru remaining as tableware candidates. The number of detections refers to the number of tableware items among the remaining candidates whose name or identification number (such as ID) matches.

[0061] Let's illustrate an example using Figure 9. In Figure 9, the top 10 tableware candidates 901 with a similarity score of 902 are extracted. The large plate 903 has the highest similarity score, but only 2 were detected. In contrast, the soba noodle basket 904 was detected 8 times. Therefore, the classification result is determined to be a soba noodle basket.

[0062] If the number of detected tableware items is the same, the one with the higher similarity score of 902 will be determined as the classification result. Furthermore, the criteria for extracting tableware candidates based on similarity are adjustable, so it may be set to prevent the number of detected items from being the same. Also, even if the remaining tableware candidates do not meet the criteria after processing in S317 and S318, a classification result will still be determined from among the tableware candidates. For example, if only one candidate remains, that tableware will be determined as the classification result.

[0063] Even if a dish has the highest similarity score, it's possible that only that dish is randomly assigned a high similarity score, which increases the likelihood of a false positive. Therefore, by classifying the dish with the highest number of detections, we aim to provide more reliable detection results.

[0064] On the other hand, in S321, CPU201 determines that the detected tableware is unregistered tableware (unregistered item). In this case, the unregistered item is processed so as not to be included in the accounting. For example, if towels or other items besides tableware are placed on the tray, they are recognized as unregistered items and not included in the accounting. In addition, a notification such as "Unknown" may be sent to the item so that it can be identified as an unregistered item.

[0065] In S322, CPU201 determines whether all detected dishes have been processed. If they have been processed, this flowchart ends; otherwise, it returns to S309.

[0066] Once the type of tableware is identified in S416 and S417, the CPU 201 refers to the menu information for the day and obtains the dishes and prices corresponding to the identified tableware. After obtaining the dishes and prices corresponding to all the tableware in a single tray image, it controls the display 104 to display the detection results, including the name of each dish, its price, and the total amount. Subsequently, in response to the user's payment request, the system settles the payment using the displayed total amount.

[0067] The above is an explanation of Figure 4.

[0068] As explained above, according to this embodiment, objects can be recognized with greater accuracy.

[0069] The embodiments described above used tableware as an example of an object and explained an example of recognizing the type of tableware. However, the present invention is not limited to recognizing the type of tableware. For example, it can be applied to generating or inferring a trained model that can determine the type of pot itself from an image, regardless of the food or dish inside. It can also be applied to recognizing (detecting) containers used in experiments such as beakers and petri dishes, containers that can hold pharmaceuticals, cosmetics, food, and beverages (such as bottles and cups), and containers for luggage (such as wooden boxes, cardboard boxes, and plastic containers). In all cases, it contributes to accurately recognizing (detecting) the container itself, regardless of its contents. Furthermore, it may be used to recognize the type of food or dish itself, such as bread, from an image, regardless of the container itself.

[0070] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0071] Furthermore, the various controls described above, which are performed by CPU201, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0072] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0073] Furthermore, although the above-described embodiments used the application of the present invention to a PC as an example, this is not limited to this example, and the invention can be applied to any device capable of calculating the size of an article according to a standard corresponding to the shape of the article. In other words, the present invention can be applied to PDAs, mobile phone terminals (smartphones), tablet terminals, and the like.

[0074] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0075] 101 Client terminals 107 Network

Claims

1. A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The predetermined conditions used when the object relating to the said region is of the first shape are narrower in scope than the predetermined conditions used when the object relating to the said region is of a shape other than the first shape. An information processing system characterized by the following.

2. A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The identification means identifies a plurality of registered images using predetermined conditions that differ depending on whether the shape of the object relating to the region is a first shape or a shape other than the first shape. The aforementioned type identification means identifies the type with the highest number of identified images among the types associated with the plurality of registered images as the type relating to the object. An information processing system characterized by the following.

3. If the aforementioned number of specific images is the same, the type associated with the registered image with the highest degree of similarity shall be identified as the type relating to the object. The information processing system according to claim 2, characterized by the following:

4. A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of a rectangular region extracted from a captured image as a region relating to an object, and predetermined conditions relating to the size. A type identification means for identifying the type of object relating to the rectangular area based on the registered image identified by the identification means, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the rectangular region is the first shape or a shape other than the first shape. An information processing system characterized by the following.

5. A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the region is a first shape or a shape other than the first shape, and the conditions indicate whether there is a difference of a predetermined value or more between the information indicating the size of the region relating to the object extracted from the captured image and the information indicating the size of the region relating to the object extracted from the registered image. An information processing system characterized by the following.

6. A storage means for storing registered images relating to an object, to which the type of the object is associated; If the shape of the object extracted from the captured image is a shape other than the first shape, the acquisition means acquires the rotated image from among multiple rotated images obtained by rotating the captured image, the rotated image having the smallest area of ​​the region related to the object extracted from the captured image. A means for identifying a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the rotated image in which the area of ​​the region acquired by the acquisition means is smallest, and predetermined conditions relating to the size. A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the region is the first shape or a shape other than the first shape. An information processing system characterized by the following.

7. A storage means for storing registered images relating to an object, to which the type of the object is associated; A means for identifying registered images that satisfy the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, A type identification means for identifying the type of object relating to the area based on the registered image identified by the identification means, Equipped with, The aforementioned predetermined conditions are different depending on whether the shape of the object relating to the region is the first shape or a shape other than the first shape, and are determined by a magnification of the size information. An information processing system characterized by the following.

8. The first shape is a perfect circle. An information processing system according to any one of claims 1 to 7, characterized by the above.

9. The system further comprises shape identification means for identifying the shape of an object, The shape-identifying means identifies shapes other than the first shape as one of the following shapes: ellipse, quadrilateral, or polygon. An information processing system according to any one of claims 1 to 7, characterized by the above.

10. The information indicating the size is one of the following: the length of the longer side of the region, the length of the shorter side of the region, the length of the diagonal of the region, or the size of the area of ​​the region. An information processing system according to any one of claims 1 to 7, characterized by the above.

11. The identification means identifies a registered image that satisfies the predetermined conditions from among registered images that include an object similar to the object extracted from the captured image. An information processing system according to any one of claims 1 to 7, characterized by the above.

12. The aforementioned object is tableware. An information processing system according to any one of claims 1 to 7, characterized by the above.

13. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The identification means of the information processing system includes an identification step in which it identifies a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the region based on the registered image identified by the identification step, Equipped with, The predetermined conditions used when the object relating to the said region is of the first shape are narrower in scope than the predetermined conditions used when the object relating to the said region is of a shape other than the first shape. A control method for an information processing system characterized by the following.

14. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The identification means of the information processing system includes an identification step in which it identifies a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the region based on the registered image identified by the identification step, Equipped with, The aforementioned identification step involves identifying a plurality of registered images using predetermined conditions that differ depending on whether the shape of the object relating to the region is a first shape or a shape other than the first shape. The aforementioned type identification step involves identifying the type with the highest number of identified images among the types associated with the multiple registered images as the type relating to the object. A control method for an information processing system characterized by the following.

15. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The identification means of the information processing system includes an identification step in which it identifies a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of a rectangular region extracted from a captured image as a region relating to an object, and predetermined conditions relating to the size. The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the rectangular area based on the registered image identified by the identification step, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the rectangular region is the first shape or a shape other than the first shape. A control method for an information processing system characterized by the following.

16. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The identification means of the information processing system includes an identification step in which it identifies a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the region based on the registered image identified by the identification step, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the region is a first shape or a shape other than the first shape, and the conditions indicate whether there is a difference of a predetermined value or more between the information indicating the size of the region relating to the object extracted from the captured image and the information indicating the size of the region relating to the object extracted from the registered image. A control method for an information processing system characterized by the following.

17. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The acquisition means of the information processing system includes an acquisition step in which, if the shape of the object extracted from the captured image is a shape other than the first shape, it acquires a rotated image from among a plurality of rotated images obtained by rotating the captured image, the rotated image having the smallest area of ​​the region relating to the object extracted from the captured image. The identification means of the information processing system includes an identification step in which the identification means identifies a registered image from among the registered images that satisfies the predetermined conditions, using information indicating the size of the region relating to the object extracted from the rotated image which has the smallest area of ​​the region acquired in the acquisition step, and predetermined conditions relating to the size. The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the region based on the registered image identified by the identification step, Equipped with, The predetermined conditions are different depending on whether the shape of the object relating to the region is the first shape or a shape other than the first shape. A control method for an information processing system characterized by the following.

18. A method for controlling an information processing system, The storage means of the information processing system includes a storage step of storing registered images relating to an object, to which the type of the object is associated; The identification means of the information processing system includes an identification step in which it identifies a registered image that satisfies the predetermined conditions among the registered images, using information indicating the size of the region relating to the object extracted from the captured image and predetermined conditions relating to the size, The type identification means of the information processing system includes a type identification step that identifies the type of object relating to the region based on the registered image identified by the identification step, Equipped with, The aforementioned predetermined conditions are different depending on whether the shape of the object relating to the region is the first shape or a shape other than the first shape, and are determined by a magnification of the size information. A control method for an information processing system characterized by the following.

19. A program for causing at least one computer to function as one of the means of the information processing system described in any one of claims 1 to 7.

Images