Information processing system and its control method, program

Abstract

To provide a mechanism that enables more accurate recognition of a recognition target that may have an object other than the recognition target on the inside.SOLUTION: An information processing system disclosed herein is configured to acquire a container area including a container of an image, perform specific processing on a partial area inside the container area so as to reduce the difference from other images, and provide control to perform learning processing using the processed image.The specific processing is performed using a randomly determined method.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a technique for recognizing an object to be recognized included in an image using the image.

Background Art

[0002] Conventionally, a technique is known in which a learned model is generated by machine learning using an image (training image, teacher data) including an object to be identified, and the object included in the image is recognized by inputting the image into the generated learned model.

[0003] In Prior Art Document 1, it is proposed to improve the detection accuracy of an unknown object (such as an object not included in the training image) by generating a mask image in which an image is separated into an object region and a background region. When generating the mask image, it is disclosed that for each pixel of the frame image, the pixel is associated with "1" (a value corresponding to white) or "0" (a value corresponding to black), and classified into an object region (white range) and a background region (black range).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] In the case of post-meal accounting in a cafeteria, use cases such as recognizing tableware from an image and performing accounting processing according to the recognized tableware can be considered. Thus, when tableware is the target of image recognition, if there is food residue in the tableware, the recognition accuracy will decrease. That is, if there is something different from the object to be recognized inside the object to be recognized, it may be a factor causing a decrease in recognition accuracy. In Prior Art Document 1, the possibility that there is something different from the object to be recognized inside the object to be recognized is not considered.

[0006] Furthermore, as in Prior Art Document 1, if a mask is applied with a specific color during the generation of the mask image, there is a possibility of misidentification as an item of that specific color. In other words, if the inside of a dish is masked with black, the dish being recognized may be misidentified as a black dish even if it is not actually black.

[0007] Therefore, the present invention aims to provide a mechanism that enables more accurate recognition of an object to be recognized, even if it may contain something different from the object to be recognized inside. [Means for solving the problem]

[0008] An acquisition means for acquiring the container region including the container from the image, Processing means for applying specific processing to a portion of the inner area of ​​the container region to reduce differences from other images, Control means for controlling the processing of the image processed by the processing means to perform learning processing, Equipped with, The processing means is characterized by performing the specific processing in a randomly determined manner. [Effects of the Invention]

[0009] According to the present invention, it is possible to recognize objects that may contain something different from the object to be recognized inside with greater accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This figure illustrates a system to which the information processing device according to this embodiment can be applied. [Figure 2] This figure shows an example of the hardware configuration of various devices. [Figure 3] This flowchart shows an example of AI learning. [Figure 4] This flowchart shows an example of AI inference. [Figure 5]This is a diagram for explaining an example of dish detection. [Figure 6] This is a diagram for explaining an example of a dish mask processing method. **Embodiments for Carrying Out the Invention**

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] First, referring to FIG. 1, an example of the configuration of an information processing system in an embodiment of the present invention will be described.

[0014] In the information processing system of the present invention, a cafeteria settlement lane 102 composed of a camera 103, a display 104, and a cash register 105 is communicably connected to a client terminal 101 via a network 107 (e.g., Ethernet) from a predetermined controller 106 (e.g., a PoE hub). Note that a plurality of cafeteria settlement lanes 102 may be connected to the client terminal 101.

[0015] The camera 103 is installed at a position where it can photograph the entire tray on the cash register 105.

[0016] On the cash register 105, a tray with dishes after a meal is placed for accounting. Note that the tray with dishes may also be in the state before the meal.

[0017] The client terminal 101 is, for example, a personal computer (hereinafter, referred to as a PC), which identifies dishes from the image captured by the camera 103 and performs processes such as settlement. The client terminal 101 uses deep metric learning technology to identify the types of dishes placed on the cash register 105.

[0018] Deep distance learning is a method that extracts only the feature amounts of images, calculates the feature amount vectors of the images by an algorithm from the extracted feature amounts, and measures the distance to determine which product is the closest. Prepare sample images in advance and extract the feature amount vectors from each image. For the input image, measure the distance between each sample image and the feature amount vector, and determine that it is of the same type as the sample with the closest distance. In this embodiment, the explanation is made using deep distance learning, but other methods such as Deep Learning Classification may also be used.

[0019] The display 104 displays the settlement information processed by the client terminal 101 and instructs the payer who had the meal to settle. Note that the video of the camera 103 may be displayed on the display 104.

[0020] Next, referring to FIG. 2, an example of the configuration of the client terminal 101 as an example of an apparatus to which the present invention is applicable is shown.

[0021] In FIG. 2, a CPU 201, a memory 202, a nonvolatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. Each unit connected to the internal bus 250 is configured to be able to exchange data with each other via the internal bus 250.

[0022] The memory 202 is composed of, for example, a RAM (a volatile memory using semiconductor elements, etc.). The CPU 201 controls each unit of the client terminal 101 using the memory 202 as a work memory according to a program stored in the nonvolatile memory 203, for example. The nonvolatile memory 203 stores image data, audio data, other data, various programs for the CPU 201 to operate, and the like. The nonvolatile memory 203 is composed of, for example, a hard disk (HD) or a ROM.

[0023] The image processing unit 204 performs various image processing operations on image data stored in the non-volatile memory 203 and recording medium 208, video signals acquired via the external I / F 209, image data acquired via the communication I / F 210, and captured images, based on the control of the CPU 201. The image processing operations performed by the image processing unit 204 include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 204 may be composed of dedicated circuit blocks for performing specific image processing operations. Depending on the type of image processing, the CPU 201 may also perform image processing according to a program without using the image processing unit 204. The process of recognizing the object to be recognized (tableware) from the image is performed by the CPU 201 in cooperation with the image processing unit 204.

[0024] The display 205 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 201. The CPU 201 generates display control signals according to the program and controls various parts of the client terminal 101 to generate and output video signals for display on the display 205. The display 205 displays images based on the output video signals. The client terminal 101 itself is configured to have an interface for outputting video signals for display on the display 205, and the display 205 may be an external monitor (such as a television).

[0025] The operation unit 206 is an input device for receiving user input, including text input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel is a flat input device superimposed on the display 205, and outputs coordinate information corresponding to the position of contact.

[0026] The recording medium interface (I / F207) allows for the insertion of recording media 208 such as memory cards, CDs, and DVDs. Based on the control of the CPU 201, it reads data from the inserted recording media 208 and writes data to the recording media 208. The external interface (I / F209) is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting video and audio signals. The communication interface (I / F210) is an interface for communicating with external devices and the internet 211 to send and receive various data such as files and commands.

[0027] The camera unit 212 is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal.

[0028] Next, referring to Figure 3, the basic processing of the learning process (learning process using AI: Artificial Intelligence) related to tableware recognition in an embodiment of the present invention will be described. Note that each step of the processing is executed by the CPU 201 of each device. As a process before a customer in the cafeteria uses the cafeteria payment lane 102, the processing shown in Figure 3 begins when the client terminal 101 learns images.

[0029] In S301, the CPU 201 extracts an image of each dish from the image including the dishes captured by the camera 103 using a bounding rectangle and saves it to the recording medium 208. Specifically, it detects the area of ​​the dishes from the image captured by the camera 103. This detection process is different from the detection of the type of dishes described later; it is a detection process that can determine that there are dishes (or objects other than trays), although the type is unknown. For the area of ​​the dishes detected in this detection process, a rectangle (hereinafter referred to as a bounding rectangle) is set so as to be tangent to the outer shape of the dishes. A partial image of the area of ​​the set bounding rectangle (i.e., an image containing a single dish) is extracted from the original image and saved to the recording medium 208.

[0030] In S302, CPU201 randomly sets the percentage of the area inside the cropped image acquired and saved in S301 that is filled in. Figure 6 shows an example of how to fill in the tableware. The cropped images 601a to 601c, which are cropped using the bounding rectangle (container region), show tableware 602a to 602c, leftover food 603a to 603c, and dirt, respectively. Since leftover food and dirt affect the accuracy of tableware identification, processing is performed to fill in parts that are unnecessary for image recognition, such as leftover food and dirt.

[0031] The following is an example of a process that randomly sets the percentage of the area to be filled. (1) The percentage of the image width to be filled is randomly determined from the specified range [c*0.5, c]. *c: A decimal number representing the upper limit of the fill percentage (0.0~1.0) (2) The percentage of the image's height that should be filled is randomly determined from the specified range [c*0.5, c]. *c: A decimal number representing the upper limit of the fill percentage (0.0~1.0) (3) The horizontal axis is (the value determined in (1) × the width of the image), the vertical axis is (the value determined in (2) × the height of the image), and the area to be filled is an ellipse with the center of the image as the origin.

[0032] Figure 6 shows an example where the values ​​substituted for c on the horizontal and vertical axes are 0.6. That is, (1) the specified range for the fill ratio relative to the image width is 0.3 to 0.6, and (2) the specified range for the fill ratio relative to the image height is 0.3 to 0.6. For the cropped image 601a cut out with a bounding rectangle, an ellipse is filled from the center at a ratio of 30% horizontally and 60% vertically to generate region 604a. Similarly, for cropped image 601b, an ellipse is filled from the center at a ratio of 60% horizontally and 40% vertically, and for cropped image 601c, an ellipse is filled at a ratio of 50% horizontally and 50% vertically to generate regions 604b and 604c. Note that when filling at a ratio of 50% horizontally and 50% vertically from the center, a perfect circle is created if the bounding rectangle is a square. For horizontally elongated tableware 602b and rectangular tableware 602c, if the circumscribing rectangle (cropped images 601b, 601c) is filled in with 50% horizontally and 50% vertically, respectively, horizontally elongated elliptical regions 604b and 604c are generated as shown in the illustration.

[0033] In this way, by randomly changing the area to be filled, an effect of artificially augmenting the image is expected. In other words, when training the AI, even if the number of input images is small, an improvement in recognition accuracy can be expected. Note that in the method of randomly setting the percentage of the area to be filled, the value substituted for c may be arbitrarily selected by the user or randomly selected by CPU201. Also, the percentage to be filled may be set arbitrarily, but setting it to 100% will fill the entire image, so it should be set to less than 100%.

[0034] In S303, CPU201 randomly sets a color to fill a portion of the inside of the cropped image acquired and saved by S301.

[0035] An example of the processing in this step is shown below. (1) The base color is listed as a candidate color that is common in food. From this, CPU201 randomly selects a candidate color to fill, and the base color is corrected by the process in (2).

[0036] (1) Select a base color from the following options. 1. Mayonnaise → White: RGB=(232,232,189) 2.Egg → Yellow: RGB=(202, 173, 67) 3. Ketchup → Red: RGB=(181,75,70) 4. Salad / Vegetables → Green: RGB=(79, 86, 27) 5. Soy sauce / sauce → Brown: RGB=(106,96,82) 6. Sesame → Black: RGB=(39,34,24 (2) Add random correction values ​​[-α, α] to each RGB component of the base color. *α: An integer representing the range of the correction value. *If the pixel value is 0 or less, it will be set to 0; if it is 255 or more, it will be set to 255.

[0037] Figure 6 shows examples of filling in parts of the inside of the tableware with different colors in the cropped images 601a to 604c. For example, area 604a is filled with a black-based color, area 604b with a yellow-based color, and area 604c with a white-based color.

[0038] In this way, recognition accuracy is improved by filling a portion of the inside of the tableware with a randomly selected color, rather than a specific color. That is, if it is predetermined to fill the area with black, it is conceivable that the system may misidentify tableware that is not black as black after learning multiple images of tableware with black interiors. Therefore, by not fixing the color to be filled and instead determining it randomly, it is expected that the occurrence of such misidentification will be reduced.

[0039] Note that while (1) shows an example where candidate colors are randomly selected from the base colors, the user may also arbitrarily select candidate colors from the base colors. Also, although six types of base colors are listed in this embodiment, there are no limitations. Furthermore, the order of processing S302 and S303 may be reversed, or they may be performed simultaneously.

[0040] In S304, the CPU 201 fills a portion of the inside of each of the cropped images (regions including tableware) acquired and saved in S301 with the color set in S303, based on the ratio set in S302, and saves it to the recording medium 208. Note that in the processing of S302 and S303, which are part of the training data creation process, it is also acceptable to use images of only tableware, without food or leftovers. Even with images of only tableware, by training with images where the inside is filled in, it is possible to train the model so that in the inference phase described later, the inside of the tableware, which may contain images of leftovers rather than the tableware itself, is not considered a feature. In other words, the processing in S302 and S303 is a process that reduces the difference between a portion of the inside of the container region including tableware and other images.

[0041] In this way, by coloring in unnecessary areas such as leftover food or stains, it is possible to prevent the false detection of leftover food, etc., and improve recognition accuracy. Furthermore, the area to be colored in changes according to the shape of the area where tableware is recognized (detected) (shape of cropped images 601a to 601c) (whether it is a perfect circle, a horizontally elongated ellipse, etc.). In other words, by coloring in areas that are likely to contain something other than the detected object such as food, according to the shape of the tableware, it becomes possible to detect the type of tableware that is the target object with greater accuracy. Note that unnecessary areas are not limited to leftover food or stains; any area that does not have distinctive features (unnecessary area) when detecting a certain object is acceptable.

[0042] In S305, CPU201 adds random noise to the filled area of ​​the filled image. Specifically, for random pixels within the filled area, a random value [-β,β] is added to the RGB values ​​of those pixels. β is assigned an integer representing the noise range. If the pixel value is 0 or less, it is set to 0; if it is 255 or greater, it is set to 255. By adding noise in this way, it becomes possible to fill the image with natural colors. For example, if part of an image appears white, it is often not actually pure white, but rather contains some other color. Therefore, adding noise can achieve natural colors. Alternatively, the noise may be smoothed before adding [-β,β] to the filled image. This step may also be omitted.

[0043] In S306, the CPU 201 performs a training process using both the original image cropped by the bounding rectangle (the cropped image before filling acquired in S301, the training image) and the filled image saved in S305 (the training image), along with label information indicating the type of tableware in each image, to create a trained model. The created trained model is recorded on the recording medium 208. Note that the original image may not be used, and only the filled image may be used as the training image. In that case, it is preferable to use only the filled image for the inference phase as well.

[0044] The above is an explanation of Figure 3.

[0045] Next, with reference to Figure 4, an example of the tableware recognition process in this embodiment is shown. This process is the inference phase using the trained model generated in the learning process of Figure 3, and is performed when a customer in the cafeteria uses the cafeteria payment lane 102. Note that the CPU 201 executes each step of the process. Although an example is described in which the processes in Figure 3 and Figure 4 are performed by the same client terminal 101, if the trained model generated in the process of Figure 3 is used, the process in Figure 4 may be executed by an information processing device (e.g., a PC) that is separate from the client terminal 101 that performs the process in Figure 3.

[0046] In S401, the CPU201 uses camera103 to photograph the area around the payment counter 105. When photographing the payment counter with camera103, it may either continuously photograph, or it may start photographing only when it detects any moving object within the photographic range.

[0047] In S402, the CPU201 performs a tray placement determination process to determine from the captured image whether a tray is placed within a predetermined range. If it is determined in S403 that a tray is placed, the dish position detection process in S404 is performed; if it is determined that no tray is placed, the tray placement determination in S402 is performed again.

[0048] In S404, the CPU 201 takes an image using the camera 103 and, similar to S301, extracts an image of each dish using a bounding rectangle from the captured image. Figure 5 shows an example of an image taken by the camera 103. Captured image 501 shows a tray 502 and dishes 503a to 503d placed on the checkout counter 105. The position of the dishes on the tray is detected, and bounding rectangles 504a to 504d are calculated for each dish. Note that Figure 5 is an example of an image of dishes without leftover food, but if there is leftover food, the leftover food will be captured inside each dish.

[0049] In S405, CPU201 fills the interior of the partial images of each dish obtained in S404 with random colors at random proportions. The processing in this step is the same as the processing in S302 to S305 of the learning flowchart in Figure 3.

[0050] In S406, CPU201 performs AI-based tableware type discrimination. Specifically, it inputs the processed cropped images created in S406 into the trained model created in S304 (the trained model stored in the recording medium 208) and performs inference processing. If multiple cropped images were acquired in S404, inference processing is performed for all of them. As a result of the inference processing, a score (the likelihood of the corresponding tableware type) for each of the multiple tableware types is output for each cropped image. CPU201 extracts those whose scores exceed a predetermined threshold and uses them as candidate types for the discrimination result. The number of types extracted as candidate types can be 0, 1, or multiple.

[0051] In S407, it is determined whether or not candidate types were extracted as a result of the inference process in S406. If one or more candidate types were extracted, the process proceeds to S408; otherwise, i.e., if there were 0 candidate types (no types whose scores exceeded the threshold), the process proceeds to S414.

[0052] The processing in S408 to S412 is performed for each candidate type. As an example, let's explain a case where, for one cropped image, three candidate types—soup bowl, rice bowl, and grilled fish plate—are extracted in S406. In this case, the processing in S408 to S412 is performed for each of the soup bowl, rice bowl, and grilled fish plate.

[0053] In S408, CPU201 acquires a sample image corresponding to the candidate type extracted in S406 and the candidate type to be processed in S408. The sample image is an image included in the correct data (training data) of tableware that is a possible detection result, and is an image that was previously recorded on the recording medium 208 in S301.

[0054] In S409, CPU201 performs a process that compares the aspect ratio of the cropped image (circumscribed rectangle), which is the recognition target image from which the candidate type was obtained, with the aspect ratio of the sample image obtained in S408.

[0055] In S410, CPU201 determines whether the difference in aspect ratio is within an acceptable range based on the comparison in S409. If it is within an acceptable range, the process proceeds to S411; otherwise, it proceeds to S414. For example, in the sample image of a grilled fish plate, the aspect ratio of the bounding rectangle of the dish is a widescreen 2:3. In contrast, if the aspect ratio of the cropped image (bounding rectangle), which is the source image for the candidate type grilled fish plate, is 1:1, then the aspect ratio of the grilled fish plate is outside the acceptable range, and the result is determined as No in this step, and the grilled fish plate is excluded from the candidate types.

[0056] In S411, CPU201 performs a process to compare the size of the cropped image (circumscribed rectangle), which is the recognition target image from which the candidate type was obtained, with the size of the sample image obtained in S408. Specifically, it compares the area (number of pixels). It performs a process to compare the area (number of pixels) of the circumscribed rectangle detected in the dish position detection in S404 with the area (number of pixels) of the candidate sample image group narrowed down in S410.

[0057] In S412, CPU201 determines whether the size difference from the comparison in S411 is within an acceptable range. If it is within an acceptable range, the process proceeds to S413; otherwise, it proceeds to S414. For example, suppose the size of the sample image of a rice bowl is size 2, which is larger than the size 1 of the sample image of a soup bowl. In contrast, the size of the cropped image (circumscribing rectangle), which is the recognition target image from which the candidate type of rice bowl was obtained, is size 1. If the difference between size 1 and size 2 exceeds the acceptable range, then in this step, it is determined to be No, and the rice bowl is excluded from the candidate types. In this way, even tableware with similar shapes may differ in size, so the size of the tableware is compared, and tableware of different sizes is excluded from the candidates. For example, even among rice bowls, there are various sizes from large to small, and in order to distinguish these, the candidates can be narrowed down by comparing the area of ​​the tableware images.

[0058] In S413, CPU201 determines whether all candidate types have been processed. If all have been processed, proceed to S415; otherwise, proceed to S408 to process the next candidate type.

[0059] In S414, CPU201 excludes the candidate type of object to be processed from the list of candidates. In other words, the type is not determined as a recognition result.

[0060] In S415, CPU201 determines whether there are any candidate types extracted in S406 that were not excluded from the candidates during processing S408 to S414. If there are, proceed to S416; otherwise (all types have been excluded), proceed to S417.

[0061] In S416, CPU201 identifies the one type of tableware with the highest score from among the candidate types extracted in S406 that were not excluded from the candidates in processing S408 to S412, and confirms it as the recognition result. In other words, one type of tableware is identified for each container area.

[0062] On the other hand, in S417, CPU201 determines that the detected tableware is unregistered tableware (unregistered item). In this case, the unregistered item is not included in the accounting. For example, if towels or other items besides tableware are placed on the tray, they are recognized as unregistered items and not included in the accounting. In addition, a notification such as "Unknown" may be given to the item in question to identify it as an unregistered item.

[0063] Once the type of tableware is identified in S416 and S417, the CPU 201 refers to the menu information for the day and obtains the dishes and prices corresponding to the identified tableware. After obtaining the dishes and prices corresponding to all the tableware in a single tray image, it controls the display 104 to display the detection results, including the name of each dish, its price, and the total amount. Subsequently, in response to the user's payment request, the system settles the payment using the displayed total amount.

[0064] The above is an explanation of Figure 4.

[0065] As explained above, according to this embodiment, images in which a portion of the inside of a dish that is likely to contain something other than the dish to be recognized (food) is filled in are trained and used for inference. In this way, the filled-in area becomes an image with the same features regardless of the type of dish. Therefore, in the training phase, the filled-in area does not become an area from which data showing features (differences) effective for determining the type of dish is obtained, and a trained model is generated in which the degree to which the filled-in area is used as the basis for determining the type of dish is low. In the trained model thus generated, even if there is any foreign object in the area of ​​the dish corresponding to the area that was filled in during training, the impact on the determination of the type of dish is low. In other words, the possibility of making a wrong judgment due to leftover food is reduced, and it becomes possible to determine the type of dish with higher accuracy. In the trained model thus generated, even if an image with leftover food in its original state is input as the target image for detecting the type of dish for inference, the area corresponding to the area that was filled in during training is used in a low degree to which the basis for determining the type of dish is low. In other words, the reduction in recognition accuracy due to the image of the leftover food portion is negligible or limited. Therefore, even without performing the fill-in process during inference, as shown in Figure 3, it is possible to improve recognition accuracy simply by using an image with a portion of the inside of the tableware image as a training image during training. For this reason, the fill-in process in S405 of Figure 4 does not need to be performed. This reduces the processing time and processing load during inference, and allows for the notification of inference results with high responsiveness. Of course, higher accuracy can be expected if the S405 process is performed.

[0066] Furthermore, according to this embodiment, images are trained and used for inference by filling a portion of the inside of a dish that is likely to contain something other than the dish being recognized (food) with a randomly selected color instead of a specific color. If the model is trained with a specific color, that specific color will also be recognized as part of the features, making it easier for false detections of dishes to occur. Therefore, by training with a randomly selected color, an improvement in recognition accuracy can be expected.

[0067] In the above embodiment, an example was described in which the percentage of the interior of the tableware is filled in for an image cropped with a bounding rectangle around a region of the tableware, but this is not the only example. Alternatively, the AI ​​could be used to extract only the portion of the tableware where food remains, and only that region could be filled in.

[0068] Alternatively, the process may be carried out as follows: After processing in S404, S405 is omitted, and the dish type discrimination in S406 is performed on the image of the container region of the unfilled recognition target image detected in S404. Then, the difference is extracted between multiple sample images corresponding to each of the multiple candidate types (images of dishes that do not have foreign objects such as leftover food, which are stored in advance) and the image of the container region of the unfilled recognition target image detected in S404. Then, images are created for each of the multiple candidate types of dishes by filling in the difference region (i.e., the area that is not a dish and is presumed to be leftover food, etc.) from the image of the container region of the unfilled recognition target image detected in S404, and the dish type discrimination in S406 is performed on these again. The single dish type with the highest score obtained as a result may be confirmed as the detected dish type. In other words, it is also possible to identify the leftover portion from the difference between the detection target image (recognition target image) and the candidate images, fill in that portion, and perform inference processing.

[0069] Alternatively, the AI ​​may be used to extract only the areas with leftover food on the detected dishes, and these areas may be filled in with the color of the dish. Furthermore, the leftover areas may be reconstructed as dishes.

[0070] In the above embodiment, an example was described in which the tableware is filled in an oval shape, as shown in Figure 6, but this is not the only example. The shape of the filling can be changed to match the shape of the tableware. For example, if the shape of the tableware is a rectangle 606, the shape of the filling can be made into a rectangle to match.

[0071] Furthermore, while we have described an example of processing the image by filling in a portion of the inside of a dish that is likely to show something other than the dish being recognized (food), this is not the only processing method that reduces the likelihood of the dish being identified and reduces the image's unique features (differences). For example, it could be a process that replaces the image with a simple pattern image, such as alternating black and gray every few pixels. In the embodiment described above, the colors are set randomly, but a pattern image may also be included as one of the random patterns. In other words, the image of the dish is processed in a way that is randomly selected from colors, pattern images, etc., on a portion of the inside of the dish.

[0072] Furthermore, the area described as being filled may be treated as a blank area without an image. Alternatively, an image of the dish with a hole in the center, excluding the area described as being filled, which is included in the bounding rectangle of the dish, may be used for training and inference. Note that if the shape of the dish differs between the image of the dish used during training and the image of the dish used during inference, there is a possibility that dishes that should be of the same type may be judged as different types. For example, if images of dishes with a hole in the center are used for training, there is a higher probability that images without a hole in the center will be judged as different dishes during inference. In this respect, if the process of filling with a single color or pattern is used during training, the shape itself does not change, so the degree of freedom of the image used during inference increases (effective inference can be performed even if the original, unfilled image is used). If a blank area is chosen instead of being filled, the exact type of dish included in the image to be detected is not known before inference, so it is difficult to cut out a blank area with the same shape as during training during inference. In other words, it is assumed that replacing with a featureless pattern such as a fill is more effective than treating it as a blank area.

[0073] Although the above-described embodiment illustrates the recognition of tableware types, it is applicable not only to the recognition of tableware types but also to any situation where something different from the detection target may exist inside the detection target. For example, it can be applied when generating or inferring a trained model that can determine the type of pot itself from an image, regardless of the food or dishes inside the pot. It can also be applied to the recognition (detection) of experimental containers such as beakers and petri dishes, containers that hold pharmaceuticals, cosmetics, food, and beverages (such as bottles and cups), and packaging containers (such as wooden boxes, cardboard boxes, and plastic containers). In all cases, it contributes to accurately recognizing (detecting) the container itself, regardless of its contents.

[0074] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0075] Furthermore, the various controls described above, which are performed by CPU201, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0076] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0077] Furthermore, although the above-described embodiments used the application of the present invention to a PC as an example, this is not limited to this example, and the invention can be applied to any device capable of generating filled images. In other words, the present invention can be applied to PDAs, mobile phone terminals (smartphones), tablet terminals, and the like.

[0078] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0079] 101 Client terminals 107 Network

Claims

1. An acquisition means for acquiring the container region including the container from the image, A determination means for selectively determining the color to be used to color the container region acquired by the acquisition means, A processing means for coloring the inner region of the container, excluding at least a portion of the outer edge of the container, with a color selectively determined by the determination means, Control means for controlling the learning process to be performed using an image that has been processed to be colored with the color selectively determined by the aforementioned determination means. An information processing system characterized by comprising the following features.

2. The information processing system according to claim 1, characterized in that the learning process is a process of generating a trained model for identifying the type of container from an image.

3. An acquisition means for acquiring the container region including the container from the image, A determination means for selectively determining the color to be used to color the container region acquired by the acquisition means, A processing means for coloring the inner region of the container, excluding at least a portion of the outer edge of the container, with a color selectively determined by the determination means, Control means for controlling the inference process to be performed using an image that has been processed to be colored with the color selectively determined by the determination means. An information processing system characterized by comprising the following features.

4. The information processing system according to claim 1 or 3, characterized in that the region is a region corresponding to the object, based on the difference between an image in which an object other than the container is inside the container region and an image in which no object other than the container is inside the container region.

5. The information processing system according to claim 1 or 3, characterized in that the determination means randomly selects a color from among a plurality of colors.

6. The information processing system according to claim 1 or 3, characterized in that the color is a color that represents something different from the container inside the container area.

7. The information processing system according to claim 1 or 3, characterized in that the aforementioned color is selected from at least one of the following: white, yellow, red, green, brown, and black.

8. The information processing system according to claim 1 or 3, characterized in that the processing means randomly determines the range to which the processing is performed.

9. The information processing system according to claim 8, characterized in that the area to be processed is determined from a ratio randomly determined with respect to the width of the image and a ratio randomly determined with respect to the height of the image.

10. The information processing system according to claim 8, characterized in that the processing means applies noise to the area to which the processing is performed.

11. The information processing system according to claim 1 or 3, characterized in that the processing means performs the processing on a portion of the container area at a predetermined ratio from the center of the container area.

12. The information processing system according to claim 1 or 3, characterized in that the processing means performs a process to color the area in a shape corresponding to the shape of the container area.

13. The information processing system according to claim 1 or 3, characterized in that the processing means applies a coloring process to the container so that it is not recognized as a container of a specific color, by coloring it with a color selectively determined by the determination means.

14. A method for controlling an information processing system, The acquisition means of the information processing system includes an acquisition step of acquiring a container region including the container from the image, The determination means of the information processing system includes a determination step of selectively determining the color to be used to color the container region acquired in the acquisition step, The processing means of the information processing system includes a processing step of coloring the inner region of the container, excluding at least a portion of the outer edge of the container, with the color selectively determined in the determination step, A control step which controls the control means of the information processing system to perform a learning process using an image that has been processed to be colored with the color selectively determined in the decision step. A control method for an information processing system, characterized by comprising the following:

15. A method for controlling an information processing system, The acquisition means of the information processing system includes an acquisition step of acquiring a container region including the container from the image, The determination means of the information processing system includes a determination step of selectively determining the color to be used to color the container region acquired in the acquisition step, The processing means of the information processing system includes a processing step of coloring the inner region of the container, excluding at least a portion of the outer edge of the container, with the color selectively determined in the determination step, A control step which controls the control means of the information processing system to perform inference processing using an image that has been processed to be colored with the color selectively determined in the decision step. A control method for an information processing system, characterized by comprising the following:

16. A program for causing at least one computer to function as each of the means of the information processing system described in claim 1 or 3.

Images