Information processing system, method of controlling the same, and program

Abstract

To provide an information processing system, a method of controlling the system, and a program which properly detect detection targets which are overlapped with each other.SOLUTION: A method according to the present invention includes the steps of: acquiring a tableware region including a first tableware (large plate 503) from an image 501; determining whether or not a second tableware (teacup 504) different from the first tableware is detected; and, if the second tableware is detected, controlling to notify that the tableware included in the image has an overlapped region.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a technique for recognizing an object to be recognized included in an image using the image.

Background Art

[0002] Patent Document 1 discloses a technique for checking whether there is an unconfirmed area caused by a product overlapping in a captured image captured by a camera 4, and checking whether there is an unconfirmed product in the captured image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0004] When recognizing a detection object from an image and notifying a detection result, an incorrect result may be notified. For example, a use case such as recognizing tableware from an image and performing accounting processing according to the recognized tableware during post-meal accounting in a cafeteria can be considered. If the tableware overlaps, there is a possibility of making an incorrect determination. Patent Document 1 does not consider a solution to the problem caused by the overlap of tableware.

[0005] Therefore, an object of the present invention is to provide a mechanism for appropriately detecting overlapping detection objects.

Means for Solving the Problems

[0006] To solve the above problems, the present invention an acquisition means for acquiring a tableware area including a first tableware from an image, a determination means for determining whether a second tableware different from the first tableware is detected from the tableware area, When a second tableware is detected, control means for controlling to notify that an overlapping area is included for the tableware included in the image; It is characterized by comprising.

Effect of the Invention

[0007] According to the present invention, it becomes possible to appropriately detect overlapping detection objects.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] First, with reference to FIG. 1, an example of the configuration of the information processing system in the embodiment of the present invention will be described.

[0011] The information processing system in the present invention includes a cafeteria settlement lane 102 composed of a camera 103, a display 104, and a cash register 105, which is communicably connected to a client terminal 101 via a network 107 (for example, Ethernet) from a predetermined controller 106 (for example, a PoE hub). Note that a plurality of cafeteria settlement lanes 102 may be connected to the client terminal 101.

[0012] The camera 103 is installed at a position where it can photograph the entire tray on the cash register 105.

[0013] On the cash register 105, a tray with dishes after a meal is placed for accounting. Note that the tray with dishes may also be in the state before the meal.

[0014] The client terminal 101 is, for example, a personal computer (hereinafter referred to as a PC), which identifies dishes from the image captured by the camera 103 and performs processes such as settlement. The client terminal 101 uses the technology of deep metric learning to identify the types of dishes placed on the cash register 105.

[0015] Deep metric learning is a method of extracting only the feature amounts of an image, calculating a feature amount vector of the image by an algorithm from the extracted feature amounts, and measuring the distance to determine which product is the closest. Prepare sample images in advance and extract feature amount vectors from each image. For the input image, measure the distance between each sample image and the feature amount vector, and determine that it is the same type as the sample with the closest distance. In this embodiment, the explanation is made using deep metric learning, but other methods such as Deep Learning Classification may also be used.

[0016] The display 104 displays the settlement information processed by the client terminal 101 and instructs the payer who had the meal to settle. Note that the video of the camera 103 may be displayed on the display 104.

[0017] Next, with reference to FIG. 2, an example of the configuration of the client terminal 101 as an example of an apparatus to which the present invention is applicable is shown.

[0018] In FIG. 2, a CPU 201, a memory 202, a non-volatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I / F 207, an external I / F 209, and a communication I / F 210 are connected to an internal bus 250. Each unit connected to the internal bus 250 is configured to be able to exchange data with each other via the internal bus 250.

[0019] The memory 202 is composed of, for example, a RAM (such as a volatile memory using semiconductor elements). The CPU 201 controls each part of the client terminal 101 by using the memory 202 as a work memory according to a program stored in the non-volatile memory 203, for example. The non-volatile memory 203 stores image data, audio data, other data, various programs for the operation of the CPU 201, and the like. The non-volatile memory 203 is composed of, for example, a hard disk (HD) or a ROM.

[0020] The image processing unit 204 performs various image processes on image data stored in the non-volatile memory 203 or the recording medium 208, a video signal acquired via the external I / F 209, image data acquired via the communication I / F 210, a captured image, etc., based on the control of the CPU 201. The image processes performed by the image processing unit 204 include A / D conversion processing, D / A conversion processing, encoding processing of image data, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, and the like. The image processing unit 204 may be composed of dedicated circuit blocks for performing specific image processes. Also, depending on the type of image process, it is also possible for the CPU 201 to perform image processing according to a program without using the image processing unit 204. The process of recognizing an object (tableware) to be recognized from an image is performed by the CPU 201 in cooperation with the image processing unit 204.

[0021] Based on the control of the CPU 201, the display 205 displays images, GUI screens that make up a GUI (Graphical User Interface), and the like. The CPU 201 generates a display control signal according to a program, generates a video signal for display on the display 205, and controls each part of the client terminal 101 to output it to the display 205. The display 205 displays a video based on the output video signal. Note that the configuration of the client terminal 101 itself includes up to an interface for outputting a video signal for display on the display 205, and the display 205 may be configured with an external monitor (such as a TV).

[0022] The operation unit 206 is an input device for receiving user operations, including character information input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, touch pads, and the like. Note that the touch panel is an input device that is configured flatly by overlapping it with the display 205 and outputs coordinate information according to the touched position.

[0023] The recording medium I / F 207 is configured such that a recording medium 208 such as a memory card, CD, or DVD can be attached, and based on the control of the CPU 201, reads data from the attached recording medium 208 and writes data to the recording medium 208. The external I / F 209 is an interface for connecting to an external device by a wired cable or wirelessly and performing input / output of video signals and audio signals. The communication I / F 210 is an interface for communicating with an external device, the Internet 211, etc., and performing transmission and reception of various data such as files and commands.

[0024] The camera unit 212 is a camera unit composed of an imaging element (imaging sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal.

[0025] Hereinafter, an example of the recognition process of tableware in the present embodiment will be described with reference to FIG. 3.

[0026] This process is the inference phase process using the learned model, and it is the process performed when a customer in the cafeteria uses the cafeteria settlement lane 102. Note that the process of each step is executed by the CPU 201 of the client terminal 101.

[0027] In S301, the CPU 201 takes a picture of the range of the settlement counter 105 with the camera 103. When taking a picture of the settlement counter with the camera 103, it is okay to keep taking pictures all the time, or when detecting any moving object within the shooting range, the shooting may be started.

[0028] In S302, the CPU 201 executes a tray placement determination process to determine whether a tray is placed in a predetermined range from the captured image. If it is determined in S303 that a tray is placed, the tableware position detection process of S304 is performed, and if it is determined that no tray is placed, the tray placement determination of S302 is executed again.

[0029] In S304, the CPU 201 takes a picture with the camera 103, and from the captured image, similar to S301, the image of each tableware is cut out and acquired with a bounding rectangle (the cut-out image is referred to as a cut-out image). Fig. 4 shows an example of the image captured by the camera 103. In the captured image 401, a tray 402 and tablewares 403a to 403c placed on the settlement counter 105 are shown. Detect the positions of the tablewares on the tray, and calculate bounding rectangles 404a to 404c for each tableware.

[0030] In S305, the CPU 201 performs discrimination of the type of tableware by AI. Specifically, it inputs the cut-out image into the learned model (the learned model stored in the recording medium 208) and performs inference processing. If a plurality of cut-out images were acquired in S304, the processes of S305 to S313 are performed for each of them. As a result of the inference processing, for each cut-out image, scores (likelihoods for the corresponding tableware types) for a plurality of tableware types are output. The CPU 201 extracts those with scores exceeding a predetermined threshold among these and uses them as tableware candidates corresponding to the cut-out images. The number of types extracted as tableware candidates can be any of 0, 1, or a plurality.

[0031] In S306, it is determined whether tableware candidates have been extracted as a result of the inference processing in S305. If one or more tableware candidates are extracted, the process proceeds to S307. Otherwise, that is, if the number of tableware candidates is 0 (there is no type with a score exceeding the threshold), an error or the like is output and the process proceeds to the processing of the next cut-out image.

[0032] The processes of S307 to S311 are performed for each of the tableware candidates one by one. Hereinafter, as an example, an example in which three tableware candidates, a large plate, a small plate, and a bowl, are extracted in S305 for one cut-out image will be described. In this case, the processes of S307 to S311 are performed for each of the large plate, the small plate, and the bowl.

[0033] In S307, the CPU 201 acquires a sample image corresponding to the tableware candidate that was extracted in S305 and is the processing target in S307. The sample image is an image included in the correct answer data (teacher data) of the tableware that can be a detection result and was pre-recorded in the recording medium 208 at the learning processing stage.

[0034] In S308, the CPU 201 executes a process of comparing the aspect ratio of the cut-out image (outer circumscribed rectangle), which is the recognition target image from which the tableware candidate was acquired, with the aspect ratio of the sample image acquired in S307.

[0035] In S309, the CPU 201 determines whether the difference in aspect ratio is within the allowable range as a result of the comparison in S308. If it is within the allowable range, the process proceeds to S310; if it is outside the allowable range, the process proceeds to S313. For example, in the sample image of a large plate, assume that the aspect ratio of the circumscribed rectangle of the tableware is 2:3 in landscape orientation. In contrast, if the aspect ratio of the cut-out image (circumscribed rectangle), which is the recognition target image from which the large plate as a tableware candidate was obtained, is 1:1, then the large plate has an aspect ratio outside the allowable range, so it is determined as No in this step and the large plate is excluded from the tableware candidates.

[0036] In S310, the CPU 201 executes a process of comparing the size of the cut-out image (circumscribed rectangle), which is the recognition target image from which the tableware candidate was obtained, with the size of the sample image acquired in S307. Specifically, the areas (number of pixels) are compared. A process of comparing the area (number of pixels) of the circumscribed rectangle detected in the tableware position detection in S304 with the area (number of pixels) of the candidates in the sample image group narrowed down in S309 is executed.

[0037] In S311, the CPU 201 determines whether the difference in size is within the allowable range as a result of the comparison in S310. If it is within the allowable range, the process proceeds to S312; if it is outside the allowable range, the process proceeds to S313. For example, assume that the size of the sample image of a bowl is size 2, which is larger than size 1 of the sample image of a small plate. In contrast, if the size of the cut-out image (circumscribed rectangle), which is the recognition target image from which the bowl as a tableware candidate was obtained, is size 1 and the difference between size 1 and size 2 exceeds the allowable range, then it is determined as No in this step and the bowl is excluded from the tableware candidates. Thus, even for tableware of the same shape, there may be differences in size, so a process of comparing the sizes of the tableware and excluding tableware of different sizes from the candidates is performed. For example, among bowls, the sizes vary from large to small, and in order to distinguish them, the candidates can be narrowed down by comparing the areas of the tableware images.

[0038] In S312, the CPU 201 determines whether all of the tableware candidates have been processed. If all have been processed, it proceeds to S314; otherwise, it proceeds to S307 to process the next tableware candidate.

[0039] In S313, the CPU 201 excludes the tableware candidate to be processed from the candidates. That is, its type is not determined as the recognition result.

[0040] Through the above processing, tableware candidates corresponding to each cut-out image (tableware candidates) are specified. Here, it is assumed that large plates, small plates, and bowls are specified as tableware candidates.

[0041] In S314, the CPU 201 selects one cut-out image of tableware for which the overlapping determination has not been performed. The overlapping of tableware refers to, for example, a state where a tea bowl overlaps on a large plate within the circumscribed rectangle 404c. If the tableware is in an overlapping state, there is a possibility that the tableware has not been accurately detected in the processing of S305 to S312. In many cases, only the larger tableware is detected, and the other tableware is not detected, and the tableware has not been correctly detected. Therefore, it is determined whether there is any overlapping of the detected tableware for all the detected tableware.

[0042] In S315, the CPU 201 trims the upper, lower, left, and right edges of the cut-out image. As shown in Fig. 5(a), the upper, lower, left, and right edges of the cut-out image 501 are trimmed to generate a trimmed cut-out image 502. The trimming process is performed to make it difficult to detect the same tableware in the subsequent processing. In the processing of S305 to S312, although the large plate 503 has been detected, it is highly likely that the tea bowl 504 has not been detected. Therefore, trimming is performed to make it difficult to recognize the large plate 503 so that the large plate 503 is not detected again in the subsequent processing.

[0043] Note that the trimming width is determined according to the parameter file stored in the non-volatile memory 203 or the recording medium 208. The parameter can be set arbitrarily by the user or determined randomly.

[0044] In S316, the CPU 201 performs a tableware detection process on the trimmed and cut-out image 502. That is, the processes of S305 to S312 are performed again on the trimmed and cut-out image 502.

[0045] In S317, the CPU 201 determines whether a tableware different from the tableware already detected by the process of S316 has been detected. In FIG. 5(b), a detection frame 505 is displayed on the teacup 504, indicating that a tableware different from the already detected large plate 503 has been detected. If a tableware different from the already detected tableware is detected, the process proceeds to S317; otherwise, the process proceeds to S319.

[0046] In S318, the CPU 201 sets a superimposition flag indicating that there is a superimposition of tableware in the detection result. The superimposition flag may be set for the entire detection result or for each tableware.

[0047] In S319, the CPU 201 determines whether a superimposition determination has been made for all the detected tableware. If it has been made for all the tableware, the process proceeds to S320; otherwise, the process returns to S314 and a superimposition determination is made for the tableware for which the superimposition determination has not been performed.

[0048] In S320, the CPU 201 determines whether the superimposition flag is set. If it is set, the process proceeds to S321; otherwise, the process ends.

[0049] In S321, the CPU 201 notifies that the tableware is stacked. Specifically, when the stacking flag is set, since the tableware is overlapping on the display 104, the CPU 201 of the client terminal 101 controls to display a warning screen including a message prompting the user to resolve the overlap. If the flag is set for each piece of tableware, specific information such as the name and position of the overlapping tableware may be displayed on the display 104. In addition to displaying on the display 104, the notification may be made in a way that the user can recognize the overlap, such as by a voice message.

[0050] As described above, according to this embodiment, it becomes possible to appropriately detect the overlapping detection target objects.

[0051] In this embodiment, an example of notifying the overlap of tableware has been described, but the correct detection result may be displayed on the display 104. Specifically, if it is specified as a large plate, a small plate, or a bowl in the processes of S307 to S311, and it is recognized that a tea bowl is overlapping on the large plate by the overlap determination after S314, the large plate, the small plate, the bowl, and the tea bowl may be displayed on the display 104 as the detection result. Thereby, the user can save the trouble of resolving the overlap of the tableware and can smoothly proceed with the settlement process.

[0052] The present invention can take an embodiment as, for example, a system, an apparatus, a method, a program, or a recording medium, etc. Specifically, it may be applied to a system composed of a plurality of devices, or it may also be applied to an apparatus composed of a single device.

[0053] Note that the various controls described above as being performed by the CPU 201 may be performed by one piece of hardware, or the overall control of the apparatus may be performed by a plurality of pieces of hardware (for example, a plurality of processors or circuits) sharing the processing.

[0054] ​Also, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine each embodiment.

[0055] In addition, in the above-described embodiments, the case where the present invention is applied to a PC has been described as an example, but this is not limited to this example, and it is applicable to any device that can notify overlapping detection objects. That is, the present invention can be applied to a PDA, a mobile phone terminal (smartphone), a tablet terminal, etc. possible

[0056] (Other embodiments) The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program, and the storage medium storing the program constitute the present invention. and the storage medium storing the program constitute the present invention. and the storage medium storing the program constitute the present invention.

Explanation of reference numerals

[0057] 101 Client terminal 102 Canteen settlement lane 103 Camera 104 Display 105 Settlement desk 106 Controller 107 Network

Claims

1. An acquisition means for acquiring a tableware area including a first tableware from an image; A determination means for determining whether or not a second tableware different from the first tableware is detected from the tableware area; A control means for controlling to notify that a superimposed area is included for the tableware included in the image when the second tableware is detected; An information processing system characterized by comprising the above.

2. An acquisition means for acquiring a tableware area including a first tableware from an image; A determination means for determining whether or not the tableware area includes a second tableware different from the first tableware; A control means for controlling to notify the detection result including the second tableware; An information processing system characterized by comprising the above.

3. Further comprising a generation means for generating an image obtained by performing trimming processing on a part of the tableware area including the first tableware, The information processing system according to claim 1 or 2, wherein the determination means determines whether or not a second tableware is detected in the image obtained by the trimming processing.

4. The information processing system according to claim 3, wherein the trimming process is a process of trimming each side according to a preset value for an image including a part of the tableware area including the first tableware.

5. The information processing system according to claim 1 or 2, wherein the determination means determines whether or not a second tableware different from the first tableware is detected for all tableware areas included in the image acquired by the acquisition means.

6. The information processing system according to claim 1, wherein the control means controls to notify that the first tableware and the second tableware overlap when the second tableware is included.

7. The information processing system according to claim 1, wherein the control means controls to notify by a voice message when the second tableware is included.

8. An acquisition step of acquiring a tableware area including a first tableware from an image; A determination step of determining whether or not a second tableware different from the first tableware is detected from the tableware area; A control step of controlling to notify that a superimposed area is included for the tableware included in the image when the second tableware is detected; A control method for an information processing system characterized by comprising the above.

9. An acquisition step of acquiring a tableware area including a first tableware from an image; A determination step of determining whether or not the tableware area includes a second tableware different from the first tableware; A control step of controlling to notify a detection result including the second tableware; A control method for an information processing system, comprising:

10. A program for causing at least one computer to function as each means of the information processing system according to Claim 1 or 2.

Images