Information processing system and its control method, program

Abstract

To provide a mechanism that enables accurate recognition even when an image size is changed for learning and inference.SOLUTION: An information processing system disclosed herein is configured to acquire an image including a recognition target object, perform processing so as to add an area of a size that is inversely proportional to a size of the recognition target object to the image including the recognition target object, and provide control to perform learning processing using the processed image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0004]

[0001] The present invention relates to a technique for recognizing an object to be recognized included in an image using the image.

Background Art

[0002] Conventionally, a technique is known in which a learned model is generated by machine learning using an image (training image, teacher data) including an object to be identified, and the object included in the image is recognized by inputting the image into the generated learned model.

[0003] Patent Document 1 describes a method for generating a teacher image using a process of spatially inverting, a process of changing color tone, a process of enlarging, a process of reducing, a process of translating, a process of distorting, and a process of synthesizing a teacher image with another image in order to solve the problem that it is difficult to generate a high-quality teacher image when the number of teacher images is insufficient.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] When generating a learned model or performing inference using the generated learned model, it is common to enlarge or reduce the training image or the image to be inferred to unify the image size and then perform learning or inference. By changing the image size in this way, information related to the size of the recognition target object is lost, and there is a problem that the recognition accuracy decreases for objects with features other than size, such as shape and color, that are similar.

[0006] The technology disclosed in Patent Document 1 does not disclose a method for generating training images that takes into account the problem of decreased recognition accuracy due to changes in image size.

[0007] Therefore, the present invention aims to provide a mechanism that enables accurate recognition even when learning and inference are performed with changed image sizes. [Means for solving the problem]

[0008] An acquisition means for acquiring an image containing the object to be recognized, Processing means for adding a region whose size is inversely proportional to the size of the object to be recognized to an image containing the object to be recognized, Control means for controlling the processing of the image processed by the processing means to perform learning processing, It is characterized by having the following features. [Effects of the Invention]

[0009] According to the present invention, even when changing the image size during training and inference, accurate recognition becomes possible. [Brief explanation of the drawing]

[0010] [Figure 1] This figure illustrates a system to which the information processing device according to this embodiment can be applied. [Figure 2] This figure shows an example of the hardware configuration of various devices. [Figure 3] This flowchart shows an example of AI learning. [Figure 4] This flowchart shows an example of AI inference. [Figure 5] This is an example of a figure used to illustrate the problem of the present invention. [Figure 6] This diagram illustrates an example of expanding the width of the outer edge of an image. [Figure 7] This diagram illustrates an example of coloring the width of the outer edges of an image. [Figure 8] This is a diagram illustrating an example of tableware detection.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, as a specific example of the application target, a case of identifying the types of tableware used in a cafeteria will be used for explanation. However, the application target is not limited to this, and it is also applicable to processes for distinguishing and recognizing objects with the same design but different sizes, such as clothes, beverage containers, prepared foods, toys (for example, a toy car and the actual object), etc.

[0012] First, referring to FIG. 1, an example of the configuration of an information processing system in an embodiment of the present invention will be described.

[0013] In the information processing system according to the present invention, a cafeteria settlement lane 102 composed of a camera  103, a display 104, and a cash register 105 is communicably connected to a client terminal 101 via a network 107 (for example, Ethernet) from a predetermined controller 106 (for example, a PoE hub). Note that a plurality of cafeteria settlement lanes 102 may be connected to the client terminal 101.

[0014] The camera 103 is installed at a position where it can photograph the entire tray on the cash register 105.

[0015] On the cash register 105, a tray with tableware after a meal is placed for accounting. Note that the tray with tableware may be in the state before the meal.

[0016] The client terminal 101 is, for example, a personal computer (hereinafter referred to as a PC), which identifies tableware from an image captured by the camera 103 and performs processes such as settlement. The client terminal 101 uses the technology of deep metric learning to identify the types of tableware placed on the cash register 105.

[0017] Deep distance learning is a method that extracts only the feature amounts of images, calculates the feature amount vectors of the images by an algorithm from the extracted feature amounts, and measures the distance to determine which product is the closest. Prepare sample images in advance and extract the feature amount vectors from each image. For the input image, measure the distance between each sample image and the feature amount vector, and determine that it is the same type as the sample with the closest distance. In this embodiment, the explanation is made using deep distance learning, but other methods such as Deep Learning Classification may also be used.

[0018] The display 104 displays the settlement information processed by the client terminal 101 and instructs the payer who had the meal to settle. Note that the video of the camera 103 may be displayed on the display 104.

[0019] Next, referring to FIG. 2, an example of the configuration of the client terminal 101 as an example of an apparatus to which the present invention can be applied is shown.

[0020] In FIG. 2, a CPU 201, a memory 202, a non-volatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. Each unit connected to the internal bus 250 is configured to be able to exchange data with each other via the internal bus 250.

[0021] The memory 202 is composed of, for example, a RAM (a volatile memory using semiconductor elements, etc.). The CPU 201 controls each part of the client terminal 101 using the memory 202 as a work memory according to a program stored in the non-volatile memory 203, for example. In the non-volatile memory 203, image data, audio data, other data, various programs for the CPU 201 to operate, etc. are stored. The non-volatile memory 203 is composed of, for example, a hard disk (HD) or a ROM.

[0022] The image processing unit 204 performs various image processing operations on image data stored in the non-volatile memory 203 and recording medium 208, video signals acquired via the external I / F 209, image data acquired via the communication I / F 210, and captured images, based on the control of the CPU 201. The image processing operations performed by the image processing unit 204 include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 204 may be composed of dedicated circuit blocks for performing specific image processing operations. Depending on the type of image processing, the CPU 201 may also perform image processing according to a program without using the image processing unit 204. The process of recognizing the object to be recognized (tableware) from the image is performed by the CPU 201 in cooperation with the image processing unit 204.

[0023] The display 205 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 201. The CPU 201 generates display control signals according to the program and controls various parts of the client terminal 101 to generate and output video signals for display on the display 205. The display 205 displays images based on the output video signals. The client terminal 101 itself is configured to have an interface for outputting video signals for display on the display 205, and the display 205 may be an external monitor (such as a television).

[0024] The operation unit 206 is an input device for receiving user input, including text input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel is a flat input device superimposed on the display 205, and outputs coordinate information corresponding to the position of contact.

[0025] The recording medium interface (I / F207) allows for the insertion of recording media 208 such as memory cards, CDs, and DVDs. Based on the control of the CPU 201, it reads data from the inserted recording media 208 and writes data to the recording media 208. The external interface (I / F209) is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting video and audio signals. The communication interface (I / F210) is an interface for communicating with external devices and the internet 211 to send and receive various data such as files and commands.

[0026] The camera unit 212 is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal.

[0027] Next, referring to Figure 3, the basic processing of the learning process (learning process using AI: Artificial Intelligence) related to tableware recognition in an embodiment of the present invention will be described. Note that each step of the processing is executed by the CPU 201 of each device. As a process before a customer in the cafeteria uses the cafeteria payment lane 102, the processing shown in Figure 3 begins when the client terminal 101 learns images.

[0028] In S301, the CPU 201 extracts an image of each dish from the image including the dishes captured by the camera 103 using a bounding rectangle and saves it to the recording medium 208. Specifically, it detects the area of ​​the dishes from the image captured by the camera 103. This detection process is different from the detection of the type of dishes described later; it is a detection process that can determine that there are dishes (or objects other than trays), although the type is unknown. For the area of ​​the dishes detected in this detection process, a rectangle (hereinafter referred to as a bounding rectangle) is set so as to be tangent to the outer shape of the dishes. A partial image of the area of ​​the set bounding rectangle (i.e., an image containing a single dish) is extracted from the original image and saved to the recording medium 208.

[0029] In S302, CPU201 expands the width of the outer edge of the cropped image acquired and saved in S301 in inverse proportion to the image size. The inverse proportion described in this embodiment includes a relationship in which, as one thing increases, the other decreases.

[0030] This step will be explained in detail using Figures 5 and 6. Figure 5(a) shows cropped images of multiple dishes that are similar in color and shape but different in size (large dish image 501, small dish image 502). However, in typical AI models for image classification, it is necessary to standardize the size of the data input to the AI ​​model. Therefore, when images are used to train the AI, they are resized to the same size, and size information is lost. That is, as shown in Figure 5(b), the images are resized to the same size when training the AI, as seen in the large dish image 503 and the small dish image 504. As a result, even if the original image sizes are different, they are recognized as the same dish, leading to a decrease in recognition accuracy.

[0031] Therefore, in this step, as shown in Figure 6(a), an interpolation width of 601 is set (added) to the outer edge of the cropped image. The interpolation width 601 is set to be inversely proportional to the original image size. For example, the interpolation width 601 for the large dish image 501 is set to be narrow, and the interpolation width 601 for the small dish image 502 is set to be wide. An example of how to calculate the specific interpolation width 601 is shown below. (1) Expand the horizontal axis (e-width of image) * d / 2 d: Expansion coefficient 0.0~1.0 e: Maximum image size to expand *If the image size is larger than e, the frame will be removed. (2) Expand the vertical axis (e - vertical width of image) * d / 2 d: Expansion coefficient 0.0~1.0 e: Maximum image size to expand *If the image size is larger than e, the frame will be removed. (3) Add an interpolation width of 601 to expand the left and right sides of the image by the value obtained in (1) (width of expansion on the horizontal axis) and the top and bottom sides by the value obtained in (2) (width of expansion on the vertical axis).

[0032] As described above, after setting an interpolation width of 601 to the cropped image, the AI ​​is trained. Figure 6(b) shows an example of the image size used for training the AI. The large tableware image 503 and the small tableware image 504 for training are resized to the same size when training the AI. At this time, an interpolation width of 601 is set for the outer edges of the image. As a result, even if the image is resized during AI training, the size of the tableware can be determined by the interpolation width of 601 set for the image. In other words, since the image reflects the size information of the tableware, it is possible to improve the recognition accuracy based on the difference in size, even for tableware that is similar in color and shape.

[0033] The interpolation width of 601 can be changed using a formula such as the following. In the following formula, the width is adjusted by an arbitrary coefficient (here, 0.5) relative to the value being expanded. Vertical axis: (Upper limit of image size to expand - Image size) / 2 × 0.5 Horizontal axis: (Upper limit of image size to expand - Image size) / 2 × 0.5

[0034] Figure 6(c) shows an example of adjusting the size of the interpolation width 601. Figure 6(c) is the image after it has been resized for training the AI, and the size of the interpolation width 601 has been adjusted compared to Figure 6(b). For images with a small size before resizing, setting the interpolation width to 601 will result in a reduced image, which may lead to loss of information and a decrease in recognition accuracy. However, by adjusting the scale of the interpolation width 601, it is possible to suppress the reduction rate and prevent a decrease in recognition accuracy.

[0035] In S303, CPU201 sets the color information (RGB) for the added interpolation width of 601. Specifically, by setting the value obtained based on the size of the cropped image as the color information, the added interpolation width is made to match the color of the original image size. An example of a formula for calculating the value to be set as color information is shown below. Note that the following formula uses height and width for calculation, but any value based on image size, such as image area, aspect ratio, or externally obtained position information and distance information, may be used for the calculation. (1) Represents the width of the image (Image width - f) / g*h f: Lower limit of size g: Upper limit of size h: Maximum pixel value of the image *A value inversely proportional to the size is also acceptable. *When the size is less than or equal to f, the value representing the width is 0; when the size is greater than or equal to g, the value representing the width is shown. (2) Represents the vertical width of the image (Image height - f) / g*h f: Lower limit of size g: Upper limit of size h: Maximum pixel value of the image *A value inversely proportional to the size is also acceptable. *When the size is less than or equal to f, the value representing the width is 0; when the size is greater than or equal to g, the value representing the width is shown. (3) Fill the R component of the extended area with the value from (1) and the B component with the value from (2).

[0036] Figure 7(a) shows an example where the value calculated using the above formula is input to the interpolation width 601. The colored interpolation width 701 of the large dish image 501 is filled with the value RGB=(200,0,200). This is visually represented as a light purple. Similarly, the colored interpolation width 702 of the small dish image 502 is filled with the value RGB=(20,0,20), and is visually represented as a dark purple. In other words, the size of the image is determined by the intensity of the color. In the above example, the large dish image 501 is shown in a light color and the small dish image 502 in a dark color, but the intensity of the colors can be reversed. Figure 7(b) is an example after resizing for AI learning. By coloring the interpolation width 601 of the outer edge in this way, the AI ​​can determine the size of the image by extracting not only the size of the interpolation width but also the color as a material, leading to improved recognition accuracy.

[0037] In S304, the CPU 201 performs a training process on both the large dish image 703 and the small dish image 704, which have values ​​input for the interpolation width generated in S303, using label information indicating the type of dish in each image, and creates a trained model. The created trained model is recorded on the recording medium 208.

[0038] The above is an explanation of Figure 3.

[0039] Next, with reference to Figure 4, an example of the tableware recognition process in this embodiment is shown. This process is the inference phase using the trained model generated in the learning process of Figure 3, and is performed when a customer in the cafeteria uses the cafeteria payment lane 102. Note that the CPU 201 executes each step of the process. Although an example is described in which the processes in Figure 3 and Figure 4 are performed by the same client terminal 101, if the trained model generated in the process of Figure 3 is used, the process in Figure 4 may be executed by an information processing device (e.g., a PC) that is separate from the client terminal 101 that performs the process in Figure 3.

[0040] In S401, the CPU201 uses camera103 to photograph the area around the payment counter 105. When photographing the payment counter with camera103, it may either continuously photograph, or it may start photographing only when it detects any moving object within the photographic range.

[0041] In S402, the CPU201 performs a tray placement determination process to determine from the captured image whether a tray is placed within a predetermined range. If it is determined in S403 that a tray is placed, the dish position detection process in S404 is performed; if it is determined that no tray is placed, the tray placement determination in S402 is performed again.

[0042] In S404, the CPU 201 takes an image using the camera 103 and, similar to S301, extracts an image of each dish using a bounding rectangle from the captured image. Figure 8 shows an example of an image taken by the camera 103. Captured image 801 shows a tray 802 and dishes 803a to 803d placed on the checkout counter 105. The position of the dishes on the tray is detected, and bounding rectangles 804a to 804d are calculated for each dish. Note that Figure 8 is an example of an image of dishes without leftover food, but if there is leftover food, the leftover food will be captured inside each dish.

[0043] In S405, CPU201 expands the width of the outer perimeter of the detected tableware rectangle inversely proportional to the image size. This expansion process is the same as the process in S302 during training.

[0044] In S406, CPU201 inputs a value proportional to the expanded portion of the bounding rectangle of the detected tableware. The processing in this step is the same as the processing in S303 during training.

[0045] In S407, CPU201 performs AI-based tableware type discrimination. Specifically, it inputs the processed image created in S406 into the trained model created in S304 (the trained model stored in the recording medium 208) and performs inference processing. If multiple cropped images were obtained in S404, inference processing is performed on all of them. As a result of the inference processing, a score (the likelihood of the corresponding tableware type) for each of the multiple tableware types is output for each processed image. CPU201 extracts those whose scores exceed a predetermined threshold and uses them as candidate types for the discrimination result. The number of types extracted as candidate types can be 0, 1, or multiple. At this time, the processed image created in S406 is used only when extracting candidate types. That is, in subsequent processing, inference processing is performed based on a comparison between the unprocessed cropped image and the sample image.

[0046] In S408, it is determined whether or not candidate types were extracted as a result of the inference process in S407. If one or more candidate types were extracted, the process proceeds to S409; otherwise, i.e., if there were 0 candidate types (no types whose scores exceeded the threshold), the process proceeds to S415.

[0047] The processing in S409 to S413 is performed for each candidate type. As an example, let's explain a case where, for one cropped image, three candidate types—soup bowl, rice bowl, and grilled fish plate—are extracted in S407. In this case, the processing in S409 to S413 is performed for each of the soup bowl, rice bowl, and grilled fish plate.

[0048] In S409, CPU201 acquires a sample image corresponding to the candidate type extracted in S407 and the candidate type to be processed in S409. The sample image is an image included in the correct data (training data) of tableware that is a possible detection result, and is an image that was previously recorded on the recording medium 208 in S301.

[0049] In S410, CPU201 performs a process that compares the aspect ratio of the cropped image (circumscribed rectangle), which is the recognition target image from which the candidate type was obtained, with the aspect ratio of the sample image obtained in S409.

[0050] In S411, CPU201 determines whether the difference in aspect ratio is within an acceptable range based on the comparison in S410. If it is within an acceptable range, the process proceeds to S412; otherwise, it proceeds to S415. For example, in the sample image of a grilled fish plate, the aspect ratio of the bounding rectangle of the dish is a widescreen 2:3. In contrast, if the aspect ratio of the cropped image (bounding rectangle), which is the source image for the candidate type grilled fish plate, is 1:1, then the aspect ratio of the grilled fish plate is outside the acceptable range, and the result is determined as No in this step, and the grilled fish plate is excluded from the candidate types.

[0051] In S412, CPU201 performs a process to compare the size of the cropped image (circumscribed rectangle), which is the recognition target image from which the candidate type was obtained, with the size of the sample image obtained in S409. Specifically, it compares the area (number of pixels). It performs a process to compare the area (number of pixels) of the circumscribed rectangle detected in the tableware position detection in S404 with the area (number of pixels) of the candidate sample image group narrowed down in S411.

[0052] In S413, CPU201 determines whether the size difference, based on the comparison in S412, is within an acceptable range. If it is within an acceptable range, the process proceeds to S414; otherwise, it proceeds to S415. For example, suppose the size of the sample image of a rice bowl is size 2, which is larger than the size 1 of the sample image of a soup bowl. In contrast, the size of the cropped image (circumscribing rectangle), which is the recognition target image from which the candidate type of rice bowl was obtained, is size 1. If the difference between size 1 and size 2 exceeds the acceptable range, then in this step, it is determined to be No, and the rice bowl is excluded from the candidate types. In this way, even dishes with similar shapes may differ in size, so the size of the dishes is compared, and dishes of different sizes are excluded from the candidates. For example, even among rice bowls, there are various sizes from large to small, and in order to distinguish these, the candidates can be narrowed down by comparing the area of ​​the images of the dishes.

[0053] In S414, CPU201 determines whether all candidate types have been processed. If all have been processed, proceed to S416; otherwise, proceed to S409 to process the next candidate type.

[0054] In S415, CPU201 excludes the candidate type of object to be processed from the list of candidates. In other words, that type is not determined as a recognition result.

[0055] In S416, CPU201 determines whether there are any candidate types extracted in S407 that were not excluded from the candidates during processing S409 to S415. If there are, the process proceeds to S417; otherwise, the process proceeds to S418.

[0056] In S417, CPU201 identifies the one type of tableware with the highest score from among the candidate types extracted in S407 that were not excluded from the candidates in processing S409 to S413, and confirms it as the recognition result. In other words, one type of tableware is identified for each container area.

[0057] On the other hand, in S418, CPU201 determines that the detected tableware is unregistered tableware (unregistered item). In this case, the unregistered item is not included in the accounting. For example, if towels or other items besides tableware are placed on the tray, they are recognized as unregistered items and not included in the accounting. In addition, a notification such as "Unknown" may be sent to the item to identify it as an unregistered item.

[0058] Once the type of tableware is identified in S417 and S418, the CPU 201 refers to the menu information for the day and obtains the dishes and prices corresponding to the identified tableware. After obtaining the dishes and prices corresponding to all the tableware in a single tray image, it controls the display 104 to display the detection results, including the name of each dish, its price, and the total amount. Subsequently, in response to the user's payment request, the system settles the payment using the displayed total amount.

[0059] The above is an explanation of Figure 4.

[0060] As explained above, according to this embodiment, even in a mechanism where size information is lost during learning and inference, learning and inference using size information becomes possible, so that even tableware with similar colors and shapes can be recognized more accurately.

[0061] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0062] Furthermore, the various controls described above, which are performed by CPU201, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0063] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0064] Furthermore, although the above-described embodiments used the application of the present invention to a PC as an example, this is not limited to this example, and the invention can be applied to any device capable of generating black-out images. In other words, the present invention can be applied to PDAs, mobile phone terminals (smartphones), tablet terminals, and the like.

[0065] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0066] 101 Client terminals 107 Network

Claims

1. Processing means for adding to an image containing an object to be recognized a region whose size is inversely proportional to the size of the image containing the object to be recognized, and whose color is set according to the size of the image containing the object to be recognized, A control means that controls the processing to be performed using an image including the region processed by the processing means, An information processing system characterized by comprising the following features.

2. The information processing system according to claim 1, characterized in that the control means controls the processed image to be resized before the learning process is performed.

3. The information processing system according to claim 1, characterized in that the learning process is a process for generating a trained model for identifying the type of object to be recognized from an image.

4. Processing means for adding to an image containing an object to be recognized a region whose size is inversely proportional to the size of the image containing the object to be recognized, and whose color is set according to the size of the image containing the object to be recognized, A control means that controls the inference process to be performed using an image including the region processed by the processing means, An information processing system characterized by comprising the following features.

5. The information processing system according to claim 1 or 4, characterized in that it identifies the size of an image using the information of the region.

6. The information processing system according to claim 1 or 4, characterized in that it identifies the size of an image using the aforementioned color information.

7. The information processing system according to claim 1 or 4, characterized in that the aforementioned area accepts size adjustment.

8. The information processing system according to claim 7, characterized in that the size adjustment accepts adjustments to the vertical width and horizontal width of the area.

9. The information processing system according to claim 8, characterized in that the vertical width of the region is a width based on the vertical width of the image, and the horizontal width of the region is a width based on the horizontal width of the image.

10. The information processing system according to any one of claims 1 or 4, characterized in that the aforementioned area is a frame.

11. The information processing system according to claim 1 or 4, characterized in that the object to be recognized includes at least one of tableware, clothing, beverage containers, prepared foods, and toys.

12. A method for controlling an information processing system, The processing means of the information processing system includes a processing step of adding to the image containing the object to be recognized a region whose size is inversely proportional to the size of the image containing the object to be recognized, and which has a color set according to the size of the image containing the object to be recognized, A control step in which the control means of the information processing system controls the system to perform learning processing using an image including the region processed in the processing step, A control method for an information processing system, characterized by comprising the following:

13. A method for controlling an information processing system, The processing means of the information processing system includes a processing step of adding to the image containing the object to be recognized a region whose size is inversely proportional to the size of the image containing the object to be recognized, and which has a color set according to the size of the image containing the object to be recognized, A control step in which the control means of the information processing system controls the system to perform inference processing using an image that includes the region processed in the processing step, A control method for an information processing system, characterized by comprising the following:

14. A program for causing at least one computer to function as each of the means of the information processing system described in claim 1 or 4.

Images