Information processing system and method for controlling the same, and program

Abstract

To provide a mechanism that appropriately detects a detection object that should be detected as a set.SOLUTION: A method specifies the types of articles detected from an image, when articles of a type forming an article group registered by combining a plurality of types of articles are detected from the image, determines whether the articles of the type forming the article group are detected as a set, and when determining that the articles of the type forming the article group are not detected as a set, outputs a result obtained by correcting a result of the specification.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for using an image to recognize an object to be recognized that is contained in the image. [Background technology]

[0002] Patent Document 1 discloses a technique relating to a rental management device for renting tableware sets, each of which is a combination of one or more types of tableware, to users. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-160556 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0004] When detecting objects from an image and reporting the detection results, the system may report incorrect results. For example, when paying the bill after a meal at a restaurant, one possible use case is to recognize tableware from an image and process the bill according to the recognized tableware. When recognizing tableware from an image, if there is tableware that looks similar to the tableware being detected, the system may report the incorrect result.

[0005] For a certain piece of tableware A to be detected, tableware B, which should be detected as a set with tableware A, may not be detected from the image and may be detected as tableware A alone. For example, if a soba choko (tableware A) appears as a detection result, it would normally be detected as a set with a soba sieve (tableware B), but if only the soba choko (tableware A) is detected, there is a high possibility of erroneous judgment. For this reason, when there are multiple sets of tableware, it is desirable to be able to output detection results that detect the complete set. Patent Document 1 does not consider a method for determining whether the tableware included in a registered tableware list are complete.

[0006] Therefore, an object of the present invention is to provide a mechanism capable of appropriately detecting detection targets that should be detected as a set. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides An identification means for identifying a type of an item detected from an image; a determination means for determining whether or not all of the items of the type constituting the item group, which is registered by combining multiple types of items, have been detected when the item of the type constituting the item group is detected from the image; an output control means for controlling the identification means to output a corrected result of the identification result when the determination means determines that all of the items of the type constituting the item group have not been detected; The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present invention, it is possible to appropriately detect detection targets that are to be detected in a set. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a system to which the information processing system according to the present embodiment can be applied. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of various devices. [Diagram 3] 13 is a flowchart showing an example of registering a tableware set. [Figure 4] 1 is a flowchart showing an example of AI inference. [Diagram 5] FIG. 13 is a diagram showing an example of a combination list registration screen. [Figure 6] FIG. 11 is a diagram illustrating an example of a combination candidate list. [Figure 7] FIG. 11 is a diagram illustrating an example of tableware detection. [Figure 8]FIG. 13 is a diagram showing an example in which tableware that should be detected as a set is detected individually. [Figure 9] FIG. 13 is a diagram showing an example in which all tableware to be detected as a set is present. [Figure 10] 13 is a diagram showing an example in which tableware not registered in the tableware set is detected. FIG. [Figure 11] FIG. 11 is a diagram showing an example of a determination result list. [Figure 12] FIG. 11 is a diagram showing an example of a determination result list. [Figure 13] FIG. 13 is a diagram showing an example of a list of possible combinations. [Figure 14] FIG. 11 is a diagram showing an example of a determination result list. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] First, with reference to FIG. 1, an example of the configuration of an information processing system according to an embodiment of the present invention will be described.

[0012] The information processing system of the present invention is configured such that a restaurant checkout lane 102, which is made up of a camera 103, a display 104, and a checkout counter 105, is communicatively connected to a client terminal 101 from a predetermined controller 106 (e.g., a PoE hub) via a network 107 (e.g., Ethernet). Note that multiple restaurant checkout lanes 102 may be connected to the client terminal 101.

[0013] The camera 103 is installed in a position where it can capture an image of the entire tray on the checkout counter 105 .

[0014] A tray with tableware after eating is placed on the payment counter 105 for payment. Note that the tray with tableware before eating may be placed on the payment counter 105 for payment.

[0015] The client terminal 101 is, for example, a personal computer (hereinafter, PC), which identifies tableware from images captured by a camera 103 and performs processes such as payment. The client terminal 101 identifies the type of tableware placed on the payment counter 105 using deep metric learning technology.

[0016] Deep distance learning is a method of extracting only the features of an image, calculating the feature vector of the image from the extracted features using an algorithm, and measuring the distance to determine which product it is closest to. Sample images are prepared in advance, and feature vectors are extracted from each image. For an input image, the distance between each sample image and the feature vector is measured, and it is determined that the input image is of the same type as the sample that is closest. In this embodiment, deep distance learning is used for explanation, but other methods such as Deep Learning Classification may also be used.

[0017] The display 104 displays the payment information processed by the client terminal 101, and instructs the payer who has eaten to settle the bill. The display 104 may also display an image captured by the camera 103.

[0018] Next, referring to FIG. 2, an example of the configuration of a client terminal 101 as an example of an apparatus to which the present invention can be applied is shown.

[0019] 2, a CPU 201, a memory 202, a non-volatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. The units connected to the internal bus 250 are configured to be able to exchange data with each other via the internal bus 250.

[0020] The memory 202 is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 201 uses the memory 202 as a work memory in accordance with a program stored in, for example, the non-volatile memory 203 to control each part of the client terminal 101. The non-volatile memory 203 stores image data, audio data, other data, various programs for the operation of the CPU 201, etc. The non-volatile memory 203 is, for example, a hard disk (HD) or a ROM.

[0021] The image processing unit 204 performs various image processing on image data stored in the non-volatile memory 203 or the recording medium 208, video signals acquired via the external I / F 209, image data acquired via the communication I / F 210, captured images, etc., based on the control of the CPU 201. The image processing performed by the image processing unit 204 includes A / D conversion processing, D / A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, etc. The image processing unit 204 may be configured with a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 201 can also perform image processing according to a program without using the image processing unit 204. The processing for recognizing an object (tableware) to be recognized from an image is performed by the CPU 201 in cooperation with the image processing unit 204.

[0022] The display 205 displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of the CPU 201. The CPU 201 generates a display control signal according to a program, and controls each unit of the client terminal 101 to generate a video signal for display on the display 205 and output it to the display 205. The display 205 displays video based on the output video signal. Note that the configuration of the client terminal 101 itself is limited to an interface for outputting a video signal for display on the display 205, and the display 205 may be configured as an external monitor (such as a television).

[0023] The operation unit 206 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, a touch pad, etc. The touch panel is an input device that is configured as a plane overlaid on the display 205 and outputs coordinate information according to the touched position.

[0024] The recording medium I / F 207 allows a recording medium 208 such as a memory card, CD, or DVD to be mounted, and reads data from the mounted recording medium 208 and writes data to the recording medium 208 under the control of the CPU 201. The external I / F 209 is an interface for connecting to an external device via a wired cable or wirelessly, and inputting and outputting video signals and audio signals. The communication I / F 210 is an interface for communicating with an external device, the Internet 211, etc., and transmitting and receiving various data such as files and commands.

[0025] The camera unit 212 is a camera unit including an image sensor (image sensor) including a CCD or CMOS element that converts an optical image into an electrical signal.

[0026] Next, the basic process of registering a tableware combination list in an embodiment of the present invention will be described with reference to the flowchart in Fig. 3. The process of each step is executed by the CPU 201 of the client terminal 101. The process in Fig. 3 is started, for example, when a combination list creation button is pressed on a menu screen (not shown).

[0027] In S301, the CPU 201 places the tableware to be registered in the combination list 603 on the payment counter 105 and photographs it with the camera 103. The tableware to be registered may be placed on the payment counter 105 one by one, or multiple tableware to be registered may be placed on the payment counter 105.

[0028] In S302, the CPU 201 acquires an image of each piece of tableware by cutting out the image including the tableware captured by the camera 103 using a circumscribing rectangle.

[0029] In S303, the CPU 201 creates a folder for each type of tableware and stores the folders in the recording medium 208. An example of a table in which tableware is registered by type is shown in Fig. 6(a). In the table, identification information 601 given to the tableware and tableware name 602 are registered.

[0030] In S304, the CPU 201 creates a combination list 603 of tableware selected by the user. This will be specifically described with reference to FIG. 5. The image 501 and name 502 of the tableware acquired in S302 are displayed on the screen. After the pressing of the tableware selection button 503 is accepted, the selection of tableware to be registered as the combination list 603 is accepted. For example, when the selection of the tableware image 501 and the tableware name 502 is accepted, a colored frame 504 is displayed to indicate that the tableware has been selected. Other methods may be used, such as surrounding the tableware with a thick frame or adding a symbol, as long as it is clear that the tableware has been selected. After the selection of multiple tableware is accepted, when the pressing of the registration button 505 is accepted, the combination list 603 of the selected tableware is registered. The registration button 505 is grayed out so that the tableware cannot be registered when only one tableware is selected, for example, until multiple tableware is selected.

[0031] The registered combination list 603 is registered by linking the identification information 601 of the tableware as shown in Fig. 6(b). For example, a soba strainer (ID: 0005), a soba cup (ID: 0006), and a condiment dish (ID: 0007) can be registered as a combination list 603 by storing linking information 604 that sets them as one set in the recording medium 208. The registered combination list 603 may be confirmed by accepting the pressing of a button for viewing the combination list from a menu screen (not shown). Editing and new registration may also be accepted on the screen where the combination list 603 can be viewed.

[0032] 3 is started when the combination list creation button is pressed on a menu screen (not shown), the processes in S301 to S303 may be skipped. That is, when a selection is received to skip the process of photographing and registering the tableware, the process may proceed to S304, and a combination list may be created from the tableware that has been saved on the combination list registration screen in FIG. 5.

[0033] In this way, by creating the combination list 603, it becomes possible to judge whether the detection result is an unnatural combination during the inference process of Fig. 4. Furthermore, if the combination is unnatural, it becomes possible to correct the combination of tableware by referring to the combination list 603 so that it becomes a correct combination of tableware.

[0034] Hereinafter, in this embodiment, the set of soba strainer, soba cup, and condiment dish will be referred to as a "soba set," and each piece of tableware that makes up the combination (set) will be referred to as a "set-constituting tableware." Next, with reference to Figs. 8 to 10, the tableware combinations assumed as the problem of this embodiment and the correct tableware combinations will be described. In Fig. 8, the tableware on the tray are detected as a large plate 801, a soba cup 802, and a small plate 803. At this time, it is natural that the soba cup 802 is placed on the tray as a set with a soba basket and a condiment plate, and it is unlikely that the soba cup 802 will be detected by itself. Therefore, in this embodiment, the detection result is suspected to be erroneous by the process of Fig. 4 described later, and the detection result is re-evaluated using the combination list 603 so that the tableware combination is correct.

[0035] 9, the detection result shows that the tableware on the tray is a soba strainer 901, a soba choko 802, and a condiment dish 902. Whether or not the combination of tableware is correct is determined by referring to combination list 603. That is, soba strainer 901, soba choko 802, and condiment dish 902 are registered in combination list 603, and the detection result shows that the tableware combination is correct if the registered tableware group is present (if the set is complete).

[0036] 10 shows an example in which none of the tableware registered in the combination list 603 is detected. As a result of the detection, the tableware detected on the tray are a large plate 801, a soup bowl 1010, and a small plate 803. In this case, since none of the tableware is registered in the combination list 603, there is no need to be aware of the tableware combination, and the detection result is output without modification.

[0037] Next, an example of tableware recognition processing in this embodiment will be explained using the flowchart of Figure 4, but first an overview of the processing will be explained using Figures 11 and 14 (the step numbers of the corresponding processing in the flowchart of Figure 4 are indicated in parentheses).

[0038] First, the case where a soba noodle set is completed will be described with reference to FIG.

[0039] Among the tableware candidates related to each cutout image, the tableware with the highest recognition score (in the example of FIG. 11, it is "large plate", "soba choko", and "small plate". These tableware are registered in the detection result candidate list first.) is examined, and the tableware "soba choko" related to cutout image B is included in the tableware that constitutes the soba set (see FIG. 6) (S414: YES). In other words, the soba choko is not used alone, but is used in a set with other tableware. Here, it is confirmed whether the soba set is completed with the tableware with the highest recognition score (S415, S416). In order to complete the soba set, a soba strainer and a condiment plate are necessary, but looking at the tableware in the current detection result candidate list, the tableware related to cutout image A is a "large plate" and the tableware related to cutout image C is a "small plate", so the soba set is not completed (both are processed as S416: NO → S417). Therefore, it is confirmed whether the soba set is completed with the tableware candidates with the second or subsequent recognition scores for cutout images A and C (S420). In the example of Fig. 11, the tableware with the second highest recognition score for cropped image A is a "buckwheat strainer," and the tableware with the second highest recognition score for cropped image C is a "condiment dish," so these two items and the soba choko complete a soba set (S420: YSE -> S424 YES). Since the set is complete, the tableware in the detection result candidate list is corrected to "buckwheat strainer," "buckwheat choko," and "condiment dish" (S425). Then, "buckwheat strainer," "buckwheat choko," and "condiment dish" are output as the detection results as the tableware corresponding to cropped images A to C (S430).

[0040] Next, a case where the soba set is not completed will be described with reference to FIG.

[0041] Among the tableware candidates related to each cutout image, the tableware with the highest recognition score (in the example of FIG. 14, it is "large plate", "soba choko", and "small plate". These tableware are registered in the detection result candidate list first.) is examined, and the tableware "soba choko" related to cutout image B is included in the tableware that constitutes the soba set (see FIG. 6) (S414: YES). In other words, the soba choko is not used alone, but is used in a set with other tableware. Here, it is confirmed whether the soba set is completed with the tableware with the highest recognition score (S415, S416). In order to complete the soba set, a soba strainer and a condiment plate are necessary, but looking at the tableware in the current detection result candidate list, the tableware related to cutout image A is a "large plate" and the tableware related to cutout image C is a "small plate", so the soba set is not completed (both are processed as S416: NO → S417). Therefore, it is confirmed whether the soba set is completed with the tableware candidates with the second or subsequent recognition scores for cutout images A and C (S420). In the example of Figure 14, the dish with the second highest recognition score for cut-out image A is a "medium plate," and the dish with the second highest recognition score for cut-out image C is a "condiment plate," so even if the second dish is included, there is no combination that completes the soba set S (S420: NO).

[0042] Therefore, processing is performed assuming that the tableware related to cropped image B is a "soup bowl" which is the second candidate tableware with the recognition score (S426). Since the "soup bowl" is not included in the set list of Fig. 6, that is, it is not a tableware in the set, "large plate", "soup bowl" and "small plate" are output as tableware corresponding to cropped images A to C (S428: YES → S429: YES, S414 is repeated for cropped images A to C, S428 YES, S429: NO, S430).

[0043] Hereinafter, an example of tableware recognition processing in this embodiment will be described with reference to FIG. This process is an inference phase process using a trained model, and is performed when a customer of the restaurant uses the restaurant payment lane 102. Note that the process of each step is executed by the CPU 201 of the client terminal 101.

[0044] In S401, the CPU 201 captures an image of the area of ​​the checkout counter 105 with the camera 103. When capturing an image of the checkout counter with the camera 103, the image may be captured continuously, or the image may be captured only when a moving object is detected within the capture range.

[0045] In S402, the CPU 201 executes a tray placement determination process to determine whether a tray is placed within a predetermined range from the captured image. If it is determined in S403 that a tray is placed, the CPU 201 executes a dish position detection process in S404, and if it is determined that a tray is not placed, the CPU 201 executes the tray placement determination process in S402 again.

[0046] In S404, the CPU 201 captures images using the camera 103, and similarly to S301, cuts out images of each piece of tableware from the captured image using circumscribing rectangles and acquires them (the cut-out images are referred to as cut-out images). Fig. 7 shows an example of an image captured by the camera 103. A captured image 701 shows a tray 702 and tableware 703a-703d placed on the payment counter 105. The positions of the tableware on the tray are detected, and circumscribing rectangles 704a-704d are calculated for each piece of tableware. In S405, the CPU 201 executes tableware type discrimination using AI. Specifically, the cut-out image is input to a trained model (a trained model stored in the recording medium 208) and inference processing is performed. If multiple cut-out images were acquired in S404, the processing of S405 to S413 is performed for each of them. As a result of the inference processing, a score for each of multiple tableware types (likelihood of the corresponding tableware type) is output for each cut-out image. The CPU 201 extracts those with scores exceeding a predetermined threshold value and sets them as tableware candidates corresponding to the cut-out image. The number of types extracted as tableware candidates may be 0, 1, or multiple.

[0047] In S406, it is determined whether or not a tableware candidate has been extracted as a result of the inference process in S405. If one or more tableware candidates have been extracted, the process proceeds to S407, and if not, that is, if there are zero tableware candidates (if there is no type whose score exceeds the threshold), an error or the like is output and the process proceeds to the next cropped image.

[0048] The processes of S407 to S411 are performed for each of the tableware candidates. Below, as an example, a case will be described in which three tableware candidates, a soba choko (small cup for noodles), a large plate, and a small plate, are extracted from one cut-out image in S405. In this case, the processes of S407 to S411 are performed for each of the soba choko (small cup for noodles), the large plate, and the small plate.

[0049] In S407, the CPU 201 acquires a sample image corresponding to the tableware candidate extracted in S405 and to be processed in S407. The sample image is an image included in the correct answer data (teacher data) of tableware that may be a detection result, and is an image that was recorded in advance in the recording medium 208 at the learning process stage.

[0050] In S408, the CPU 201 executes a process of comparing the aspect ratio of the cut-out image (circumscribing rectangle) that is the recognition target image from which the tableware candidates are obtained, with the aspect ratio of the sample image obtained in S407.

[0051] In S409, CPU 201 judges whether the difference in aspect ratio is within the allowable range as a result of the comparison in S408. If it is within the allowable range, the process proceeds to S410, and if it is outside the allowable range, the process proceeds to S413. For example, in a sample image of a large plate, the aspect ratio of the circumscribing rectangle of the dish is horizontally long at 2:3. In contrast, if the aspect ratio of the cut-out image (circumscribing rectangle) that is the recognition target image from which the large plate, which is a dish candidate, was obtained, is 1:1, the aspect ratio of the large plate is outside the allowable range, so this step is judged as No, and the large plate is excluded from the dish candidates.

[0052] In S410, the CPU 201 executes a process of comparing the size of the cut-out image (circumscribing rectangle) that is the recognition target image from which the tableware candidates were obtained with the size of the sample image obtained in S407. Specifically, the areas (number of pixels) are compared. The CPU 201 executes a process of comparing the area (number of pixels) of the circumscribing rectangle detected in the tableware position detection in S404 with the area (number of pixels) of the candidates in the sample image group narrowed down in S409.

[0053] In S411, the CPU 201 judges whether the difference in size is within the allowable range as a result of the comparison in S410. If it is within the allowable range, the process proceeds to S412, and if it is outside the allowable range, the process proceeds to S413. For example, the size of the sample image of the soba choko is assumed to be size 2, which is larger than size 1 of the sample image of the small plate. In contrast, if the size of the cut-out image (circumscribed rectangle) that is the recognition target image from which the soba choko, which is a candidate tableware, is obtained is size 1, and the difference between size 1 and size 2 exceeds the allowable range, this step is judged as No, and the soba choko is excluded from the tableware candidates. In this way, since the sizes of tableware may differ even if the tableware has a similar shape, a process is performed to compare the sizes of the tableware and exclude tableware of different sizes from the candidates. For example, there are various sizes of soba choko, from large to small, and in order to identify them, the candidates can be narrowed down by comparing the areas of the tableware images.

[0054] In S412, the CPU 201 determines whether or not all of the tableware candidates have been processed. If all have been processed, the process proceeds to S414, and if not, the process proceeds to S407 to process the next tableware candidate.

[0055] In S413, the CPU 201 excludes the tableware candidate to be processed from the candidates, i.e., the type is not determined as a recognition result.

[0056] By the above process, candidates for tableware (tableware candidates) corresponding to each cut-out image are identified. Specifically, as shown in 1102 to 1104 in FIG. 11. In the example of FIG. 11, a "large plate" and a "buckwheat strainer" are identified as tableware candidates corresponding to tableware A (cut-out image A), a "buckwheat sake cup" and a "soup bowl" are identified as tableware candidates corresponding to tableware B (cut-out image B), and a "small plate" and a "condiment plate" are identified as tableware candidates corresponding to tableware C (cut-out image C). In addition, as shown in FIG. 11, the recognition scores of each tableware candidate are also recorded.

[0057] In S414, CPU 201 determines whether the tableware candidate with the highest recognition score among the tableware candidates in the first cropped image (assumed to be cropped image A) is included in the combination list (FIG. 9(b)) created in FIG. 3. If it is included, proceed to S415; if not, proceed to S428. For example, the tableware candidate with the highest recognition score among the tableware candidates in cropped image A is "large plate," but "large plate" is not registered in the combination list, so this step is determined as NO and the process transitions to step S428.

[0058] In S415, the CPU 201 acquires images of the tableware (tableware constituting a set) to be combined with the tableware candidate determined to be included in the combination list in S414. For example, if the tableware candidate is a soba choko, images of a soba basket and a condiment dish, which are tableware constituting a soba set that includes the soba choko, are acquired.

[0059] In S416, the CPU 201 determines whether or not the Nth set constituent tableware among the set constituent tableware acquired in S415 is included in the detection result candidate list 1105. If it is included, the process proceeds to S418, and if not, the process proceeds to S417.

[0060] For example, when a "buckwheat strainer" is selected as the processing target from among the tableware constituting the set, since the "buckwheat strainer" is not included in the detection result candidate list 1105, a NO determination is made in S416.

[0061] Here, the detection result candidate list 1105 will be described. The detection result candidate list 1105 is a list in which information indicating which tableware was finally recognized (tableware output in S430) is registered. First, the tableware with the highest recognition score among the tableware related to each cut-out image is registered, and is corrected by the processing from S414 onwards.

[0062] That is, in the detection result candidate list 1105, the tableware candidate with the highest recognition score is registered in the first processing, but in the processing after the determination of YES in S429, the tableware candidate with the second or subsequent recognition scores is registered.

[0063] In the example of Figure 11, the tableware candidate with the highest recognition score for cut-out image A is a "large plate," and similarly, cut-out image B is a "soba choko" and cut-out image C is a "small plate," so "large plate," "soba choko," and "small plate" are registered in the detection result candidate list 1105.

[0064] In S417, the CPU 201 records that the Nth set constituent tableware is not included in the detection result candidate list 1105. The recording method may be, for example, setting a flag and checking later whether the flag is set or not.

[0065] In S418, the CPU 201 determines whether or not all of the N pieces of tableware constituting the set acquired in S415 have been confirmed. If they have been confirmed, the process proceeds to S419. If not, the process proceeds to S416 and subsequent steps for the tableware constituting the set that has not been confirmed.

[0066] In S419, the CPU 201 determines whether or not there is any tableware that constitutes the set and that was recorded in S417 as not being in the detection result candidate list 1105. If there is any tableware, the process proceeds to S420, and if not, the process proceeds to S428.

[0067] That is, if all the tableware that should be detected in combination (set) with the tableware corresponding to the extracted image to be processed are present, the result is NO, and if even one of them is not present, the result is YES.

[0068] In S420, the CPU 201 judges whether or not there is a set of tableware among the tableware candidates corresponding to the cutout image other than the cutout image to be processed and having a recognition score equal to or greater than a threshold value. For example, if the cutout image to be processed is cutout image B and the threshold value is 0.90 or greater, tableware candidates in cutout image A with a recognition score equal to or greater than 0.90 include a "large plate" and a "buckwheat basket". Similarly, tableware candidates in cutout image C with a recognition score equal to or greater than 0.90 include a "small plate" and a "condiment plate". It is judged whether or not there are "buckwheat basket" and "condiment plate" among these four tableware candidates, which are tableware that constitute a "buckwheat set", which is a combination including a tableware candidate "buckwheat choko" corresponding to cutout image B. As a result, since "buckwheat basket" and "condiment plate" are included among the four tableware candidates, the process in S420 is judged as YES in this case.

[0069] Next, a case where it is determined that the target image is not included in the determination candidate list will be specifically described with reference to FIG. 12. For example, if the cutout image to be processed is cutout image B and the threshold value is 0.90 or more, among the tableware candidates in cutout image A, there is a "large plate" as a tableware candidate with a recognition score of 0.90 or more. Similarly, among the tableware candidates in cutout image C, there is a "rice bowl" as a tableware candidate with a recognition score of 0.90 or more. It is determined whether or not these two tableware candidates include a "buckwheat basket" and a "condiment plate" which are tableware that constitute a "buckwheat set", which is a combination including a "buckwheat sake cup" which is a tableware candidate corresponding to cutout image B. As a result, since neither a "buckwheat basket" nor a "condiment plate" is included in the two tableware candidates, the process of S420 is determined as NO in this case.

[0070] If the answer to S420 is YES, the process proceeds to S421, and the processes of S421 to S423 are performed on the first piece of tableware that constitutes the set. If the answer to S420 is NO, the process proceeds to S426.

[0071] In S421, the CPU 201 determines whether the tableware candidate corresponding to the extracted image to be processed matches the tableware constituting the set to be processed. If they match, the process proceeds to S423, and if not, the process proceeds to S422.

[0072] In S422, the CPU 201 adds to the combination candidate list (FIG. 13) tableware candidates that have the second or higher recognition score and that constitute a combination among the tableware candidates corresponding to the cut-out image other than the cut-out image to be processed. Specifically, when the cut-out image to be processed is cut-out image B and the tableware constituting the set to be processed is a "buckwheat basket," "buckwheat choko" and "buckwheat basket" do not match, so "buckwheat basket" is added to the combination candidate list 1301. In S423, the CPU 201 determines whether or not all the tableware constituting the set has been checked. If checked, the process proceeds to S424, and if not, the process returns to S421.

[0073] In S424, the CPU 201 determines whether the combination is satisfied (whether a set is completed) by the tableware candidate corresponding to the cutout image to be processed and the tableware candidates corresponding to the other cutout images. That is, in S424, it is determined whether a set is completed by the tableware candidate with the highest recognition score among the tableware candidates corresponding to the cutout image to be processed and the tableware candidates with the second or lower recognition scores among the tableware candidates corresponding to the other cutout images. If the combination is satisfied, the process proceeds to S425, and if not, the process proceeds to S426.

[0074] In addition, in the case of processing after the judgment of YES in S429 (second or subsequent processing), whether a set is complete is determined based on the tableware with the second highest recognition score among the tableware candidates corresponding to the cut-out image being processed and the tableware with the second or subsequent recognition scores among the tableware candidates corresponding to the other cut-out images.

[0075] In S425, the CPU 201 modifies the tableware in the detection result candidate list 1105 to tableware constituting the set. Specifically, the detection result candidate list 1105 has registered a "large plate" as the tableware corresponding to the cut-out image A, a "soba choko" as the tableware corresponding to the cut-out image B, and a "small plate" as the tableware corresponding to the cut-out image C, but in S424 it was determined that the set would be completed with a soba basket, a soba choko, and a condiment dish. Therefore, the tableware corresponding to the cut-out image A is modified to a "soba basket". Similarly, the tableware corresponding to the cut-out image C is modified to a "condiment dish".

[0076] In S426, the CPU 201 modifies the tableware corresponding to the cut-out image to be processed in the detection result candidate list 1105 to the tableware with the second highest recognition score. Specifically, when the cut-out image to be processed is cut-out image A, the "large plate" registered as the tableware corresponding to cut-out image A in the detection result candidate list 1105 is modified to a "buckwheat strainer."

[0077] In S427, the CPU 201 sets a correction flag indicating that the tableware in the detection result candidate list 1105 has been corrected.

[0078] On the other hand, in S428, the CPU 201 determines whether or not confirmation has been completed for all the tableware for the Mth tableware in the detection result candidate list 1105. If so, the process proceeds to S429, and if not, the process returns to S414.

[0079] In S429, the CPU 201 judges whether or not the correction flag is set. If the correction flag is set, the correction flag is lowered and the process returns to S414 to check whether or not the tableware of the detection candidate after correction is valid. If the correction flag is not set, the process proceeds to S430.

[0080] In S430, the CPU 201 outputs the detection result candidate list 1105 at that time as the determination result.

[0081] This completes the explanation of FIG.

[0082] As described above, according to this embodiment, it is possible to output a result in which a detection target that should be detected in a plurality of sets is not detected singly.

[0083] In this embodiment, a tableware set has been described as an example, but the present invention is not limited to tableware. For example, a medical product set, a textbook set, or other items with a predetermined combination may be used. When such a set is available, it is determined whether the detection result of the set image is appropriate.

[0084] It is also possible to create an editing screen for the combination list 603, and to set some of the tableware constituting the set to "optional". For example, a check box or the like is provided for each item of the soba strainer, soba choko, and spice plate to select whether or not to register the tableware as "optional" tableware constituting the set. If the tableware constituting the set set to "optional" is not detected, the determination result of the other tableware constituting the set may not be corrected, but a re-photograph may be prompted or it may be output as a determination result that there is no problem even if it is not detected. In other words, the determination of the tableware constituting the set set to "optional" may be adjusted to be looser. This allows, for example, a combination to be established with only the soba strainer and soba choko if a person does not need a spice plate.

[0085] The present invention may be embodied, for example, as a system, an apparatus, a method, a program, or a recording medium. Specifically, the present invention can be applied to a system that is composed of multiple devices. The present invention may be applied to a single device or to an apparatus consisting of a single device.

[0086] The various controls described above as being performed by the CPU 201 can be implemented by a single piece of hardware. Alternatively, multiple hardware (e.g., multiple processors or circuits) may perform the processing separately. The entire device may be controlled by the controller.

[0087] Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments. The present invention is not limited to the embodiments, and various forms within the scope of the gist of the present invention are also possible. Furthermore, the above-described embodiments are merely examples of the present invention. It is also possible to combine the respective embodiments as appropriate.

[0088] In the above embodiment, the present invention is applied to a PC. However, this is not limited to this example, and can be applied to any device that can generate black-filled images. In other words, the present invention can be applied to PDAs, mobile phone terminals (smartphones), tablet terminals, etc. Possible

[0089] (Other embodiments) The present invention can also be realized by executing the following process. Software (programs) that realize the functions of the above are provided via networks or various storage media. The system or device is supplied with the computer (or CPU or M) of the system or device. In this case, the program code is read and executed by the PU. The program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0090] 101 Client terminals 102 Cafeteria Checkout Lane 103 Camera 104 Display 105 Payment desk 106 Controller 107 Network

Claims

1. An identification means for identifying a type of an item detected from an image; a determination means for determining whether or not all of the items of the type constituting the item group, which is registered by combining multiple types of items, have been detected when the item of the type constituting the item group is detected from the image; an output control means for controlling the identification means to output a corrected result of the identification result when the determination means determines that all of the items of the type constituting the item group have not been detected; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the determining means determines whether or not the group of items is made up of a specific item and an item other than the specific item detected from the image.

3. The information processing system according to claim 2, characterized in that, when the group of items is not established, the output control means controls so that items other than the specific item are modified to other items that satisfy predetermined conditions and are output.

4. 3. The information processing system according to claim 2, wherein the output control means controls so that, if the group of items is not established, the specific item is modified to another item that satisfies a predetermined condition and output.

5. 5. The information processing system according to claim 3, wherein the predetermined condition is a condition for modifying the recognition rate to an item having a second best recognition rate compared to the recognition rate of the specific item or an item other than the specific item.

6. The information processing system according to claim 1, characterized in that the output control means controls the output of the judgment result without making any corrections if any of the items detected from the image are not registered in the item group.

7. 2. The information processing system according to claim 1, wherein the article is tableware.

8. An identification step of identifying a type of item detected from the image; a determination step of determining whether or not all of the items of the type constituting the item group, which is registered by combining multiple types of items, have been detected when the item of the type constituting the item group is detected from the image; an output control step of controlling to output a result of correcting the identification result by the identification step when it is determined by the determination step that all of the items of the type constituting the item group have not been detected; A method for controlling an information processing system comprising:

9. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 4 and 7.

Images