Information processing system, information processing device, and information processing method

Abstract

To track a commodity until the commodity is put into a storage bag or the like from a container or the like including a shopping basket without manually scanning the commodity by a buyer and determine whether the commodity has been appropriately recognized.SOLUTION: An information processing system includes a first imaging unit 1 and a control unit 2 that performs processing related to an image captured by the first imaging unit 1. The control unit 2 performs processing based on an image captured in each of a first stage in which an article is put into a container 11 and a second stage in which the article is taken out of the container 11.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, and an information processing method.

Background Art

[0002] In the settlement of purchased goods in the retail industry, for example, Patent Document 1 discloses an invention that tracks the process from when a product is taken out of a shopping basket until it is put into a storage bag, and determines whether the product has been properly scanned.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention according to this disclosure, it is necessary for the purchaser to manually scan the product for identification of the product, and there is room for improvement in this regard.

[0005] Therefore, an object of the present disclosure made in view of the above points is to track the process from when a product is put into a storage bag or the like from a container including a shopping basket without the purchaser manually scanning the product, and to determine whether the product has been properly recognized.

Means for Solving the Problems

[0006] To achieve the above-described object, an information processing system according to one aspect of the present disclosure includes a first imaging unit and a control unit that performs processing on an image captured by the first imaging unit, and the control unit performs processing based on images captured in each of a first stage of putting an object into a container and a second stage of taking the object out of the container.

[0007] Also, an information processing apparatus according to an aspect of the present disclosure includes a control unit that performs processing related to an image captured by a first imaging unit, and the control unit performs processing based on images captured in each of a first stage of putting an object into a container and a second stage of taking the object out of the container.

[0008] Also, an information processing method according to an aspect of the present disclosure includes the first imaging unit performing imaging in a first stage of putting an object into a container, the first imaging unit performing imaging in a second stage of taking the object out of the container, and performing processing based on images captured by the first imaging unit in each of the first stage and the second stage.

Advantages of the Invention

[0009] According to the present disclosure configured as described above, it becomes possible to track the process from when a purchaser manually scans a product until the product is placed in a storage bag or the like from a shopping basket without the purchaser manually scanning the product, and to determine whether the product has been appropriately recognized.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an information processing system according to the present disclosure will be described with reference to the drawings.

[0012] As shown in FIG. 1, the information processing system according to the present disclosure includes a first imaging unit 1 and a control unit 2. Further, the information processing system may include a notification unit 3 and a sensor 4.

[0013] In the following figures, solid arrows connecting each functional block indicate the flow of control signals or communicated data. The communication indicated by the arrows may be either wired / wireless, or a combination of both.

[0014] The first imaging unit 1 captures an image of an object that is put into and taken out of a container 11 that temporarily stores the object.

[0015] In order to capture an image of an object and generate an image, the first imaging unit 1 includes an imaging optical system and an imaging element. The imaging optical system includes, for example, one or more optical members such as lenses and diaphragms. The lens may be of any type regardless of the focal length, and may be, for example, a general lens, a wide-angle lens including a fisheye lens, or a zoom lens with a variable focal length. The imaging optical system forms an object image on the light receiving surface of the imaging element. The imaging element includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The imaging element captures the object image formed on the light receiving surface and generates an image.

[0016] Note that the first imaging unit 1 may continuously capture still images in any frame, or may discontinuously capture still images at specific timings. Further, the first imaging unit 1 may capture a moving image.

[0017] Also, the first imaging unit 1 may be provided in any form as long as it can image an object that enters and exits a container 11 including a shopping basket or a shopping cart. For example, it may be provided in the container 11, or may be provided in the form of a portable electronic terminal including a smartphone or a business-use terminal, or may be provided on a merchandise shelf, ceiling, cash register, etc. inside a retail store. In particular, when provided in the container 11, it is easy to image an object that enters and exits the container 11. Further, when the first imaging unit 1 is provided in a portable electronic terminal, the container 11 may include a holder that holds the portable electronic terminal. In the following description, it is assumed to be provided in the container 11 (also shown as being provided in the container 11 in FIG. 1).

[0018] Also, the image captured by the first imaging unit 1 is transmitted to the control unit 2 via a communication unit 12 provided together with the first imaging unit 1.

[0019] The communication units 12 (hereinafter, the communication unit 12 of the container), 22 (the communication unit 22 of the control unit), 31 (the communication unit 31 of the notification unit), and 41 (the communication unit 41 of the sensor), including those provided in the container 11, the control unit 2, the notification unit 3, and the sensor 4 described later, are configured to include either or both of a wired / wireless module. The communication units 12, 22, 31, and 41 do not necessarily have the same configuration and may have different configurations. The communication units 12, 22, 31, and 41 transmit and receive control signals or data.

[0020] The control unit 2 performs processing and the like based on the image captured by the first imaging unit 1.

[0021] The control unit 2 is configured to include, as internal modules, an arithmetic unit 21, a communication unit 22 of the control unit, and a storage unit 23. Physically, the control unit 2 includes a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array), and in addition, as the storage unit 23, it is configured to include any one or more of, for example, a semiconductor memory, a magnetic memory, and an optical memory. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape. The optical memory may be configured to include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray (registered trademark) Disc).

[0022] The arithmetic unit 21 includes, as processing based on the image captured by the first imaging unit 1, for example, control related to the imaging of the first imaging unit 1, object detection / recognition processing related to the image captured by the first imaging unit 1, data processing related to the recognition result, and control related to the notification of the notification unit 3 related to the data processing result. The data processing related to the recognition result includes, for example, status management of the recognition result. The status management includes, for example, adding or deleting the recognition result of the object to / from a list regarding the addition or deletion of an object in the container 11.

[0023] The arithmetic unit 21 may output, as the recognition result and the attribute information of the recognition result, for example, the object name (product name) of the recognized object, the price, the moving direction of the object, and the reliability. The moving direction of the object includes, for example, binary information indicating whether the object has been put into the container 11 or taken out of the container 11. For example, the reliability is a quantification of how reliable the recognition result is, and it can be said that the higher the reliability, the higher the accuracy of the recognition result.

[0024] The communication unit 22 of the control unit performs transmission and reception of, for example, the following control signals or data. 1. Receives the image captured by the first imaging unit 1 from the communication unit 12 of the container. 2. Receives the signal acquired by the sensor 4 from the communication unit 41 of the sensor. 3. Transmits and receives various control signals or data with the arithmetic unit 21. 4. Transmits the control signal of the first imaging unit 1 to the communication unit 12 of the container. 5. Transmits the control signal of the notification unit 3 to the communication unit 31 of the notification unit.

[0025] The storage unit 23 stores at least one or more of the image captured by the first imaging unit 1, the recognition result obtained by the object detection / recognition process performed by the arithmetic unit 21 based on the image captured by the first imaging unit 1, the processing result of the process performed based on the recognition result, the signal acquired from the sensor 4, or the program related to the processing in the information processing system.

[0026] Note that the control unit 2 does not necessarily have to be on-premises and may perform processing in the cloud on a web service. Also, the control unit 2 may be functionally divided such that processing is performed on-premises or in the cloud in module units. For example, the arithmetic unit 21 may perform processing in the cloud. Also, the storage unit 23 may perform processing on-premises.

[0027] The notification unit 3 performs notification based on a control signal or the like from the control unit 2 via the communication unit 31 of the notification unit. The content to be notified may include, for example, the recognition result obtained by the control unit 2 or the result obtained by performing processing based on the recognition result.

[0028] As means for performing the above-described notification, the notification unit 3 includes, for example, those appealing to vision, hearing, or touch. Therefore, the notification unit 3 may be configured to include, for example, a display, a lamp, a speaker, a motor, etc. Also, the notification unit 3 may be provided with an input unit for, for example, correcting or making a confirmation input regarding the notified recognition result or the like. The input unit may include, for example, a mouse, a keyboard, a microphone, a touch sensor, and the touch sensor may be provided in the display as a touch display.

[0029] The sensor 4 monitors the movement of the purchaser or the object, and transmits the detected data obtained by the monitoring to the control unit 2.

[0030] The sensor 4 includes, for example, a weight sensor, an infrared sensor, and a second imaging unit configured to include an imaging optical system and an imaging device. Also, regarding the arithmetic processing related to these control or detection data, the control unit 2 may perform it, or the sensor 4 may be separately provided with a component corresponding to the control unit 2 and perform this.

[0031] When the sensor 4 is provided in the container 11, for example, it may be provided as a weight sensor at the bottom of the container 11. When the weight detected by the weight sensor changes, assuming that an object is added or removed from the container 11, based on the detection result of the weight sensor, the control unit 2 or the weight sensor may generate a trigger signal for the first imaging unit 1 to perform imaging.

[0032] Also, when the sensor 4 is provided outside the container 11, for example, it may be provided on the merchandise shelf or ceiling inside the retail store. For example, when a weight sensor is provided on the merchandise shelf as the sensor 4 and the weight detected by the weight sensor changes, assuming that an object is added or removed from the container 11, the weight sensor may generate a trigger signal. Similarly, for example, when a second imaging unit is provided on the ceiling as the sensor 4 and a specific movement (such as an operation where the purchaser takes out an object from the merchandise shelf, the trajectory of the object moving from the merchandise shelf, etc.) to add or remove an object from the container 11 is detected in the image captured by the second imaging unit, assuming that an object is added or removed from the container 11, based on the image captured by the second imaging unit, the control unit 2 or the second imaging unit may generate a trigger signal.

[0033] Next, the imaging method and the like in the first stage and the second stage will be briefly described with reference to FIGS. 2 and 3.

[0034] Figure 2 is an image diagram related to an imaging method and the like in the first stage of putting an object into container 11. The first stage here includes the stage of a purchaser selecting a product (object) to be purchased. That is to say, the first stage can be paraphrased as the pre-recognition determination stage before a purchaser determines the purchased product (determines the recognition result of the product) in a specific area including the cash register for accounting processing.

[0035] According to Figure 2, when object A (the object) is put into container 11, the first imaging unit 1 transmits the image of the imaged object A to the control unit 2.

[0036] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains the recognition result.

[0037] Figure 3 is an image diagram related to an imaging method and the like in the second stage of taking an object out of container 11. The second stage here includes the stage of a purchaser performing accounting processing on the product to be purchased. That is to say, the second stage can be paraphrased as the recognition determination stage for determining the purchased product in a specific area.

[0038] According to Figure 3, first, in order to shift from the first stage to the second stage, for example, the first imaging unit 1 images a marker provided in a specific area and transmits the imaged image to the control unit 2. Then, the control unit 2 shifts from the first stage to the second stage. That is to say, it can be paraphrased that the stage before the control unit 2 recognizes the marker provided in the specific area is the first stage, and the stage after recognizing the marker is the second stage.

[0039] Note that in order to determine the shift from the first stage to the second stage, other means may be used. For example, a weight sensor may be provided in a specific area, and based on the weight sensor detecting a change in weight when container 11 is placed on the weight sensor, or a separate sensor may be provided in container 11, and based on the separate sensor detecting a specific signal or object including the marker provided in the specific area.

[0040] After the object A is taken out of the container 11 after shifting to the second stage, the first imaging unit 1 transmits the image of the imaged object A to the control unit 2.

[0041] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains a recognition result.

[0042] Thereafter, the control unit 2 performs final determination processing of the recognition result based on the images captured in each of the first stage of putting the object into the container 11 and the second stage of taking the object out of the container 11.

[0043] At this time, the control unit 2 may transmit a control signal so as to cause the notification unit 3 to notify the recognition result obtained by performing the final determination processing. Further, the control unit 2 may transmit a control signal so as to cause the notification unit 3 to notify the recognition result of the object imaged by the first imaging unit 1 in the second stage before the final determination processing of the recognition result. For example, according to FIG. 3, when each recognition result obtained based on the images of the object A captured in the first stage and the second stage matches "apple", the notification unit 3 displays "apple".

[0044] Next, an information processing method of the information processing system according to the first embodiment of the present disclosure will be described. In the information processing method of the information processing system according to the first embodiment of the present disclosure, the first imaging unit 1 continuously performs imaging at an arbitrary frame rate, and the control unit 2 performs object detection and object recognition based on the image captured by the first imaging unit 1. Then, as a method for status management of the recognition result, the first to fourth lists are used, and based on these lists, the control unit 2 performs final determination processing of the recognition result.

[0045] Then, in the first stage, the control unit 2 adds, as a purchase candidate product, an object determined to be an object put into the container 11 to a first list that stores the recognition result of the purchase candidate product.

[0046] Also, in the second stage, the control unit 2 adds, as a purchased product, an object determined to be an object taken out of the container 11 to a second list that stores the recognition result of the purchased product.

[0047] Then, it is verified whether the first list and the second list are exactly the same. If they are exactly the same, the control unit 2 directly adds the recognition result of the object included in the second list as the finally determined purchased product to the third list that stores the recognition result of the finally determined purchased product, and determines the recognition result of the object. Also, if the first list and the second list are not exactly the same, the recognition result of the matching object is added to the third list, and the recognition result of the non-matching object, that is, the recognition result of the finally recognized failed product, is added to the fourth list that stores the recognition result of the finally recognized failed product, respectively.

[0048] The control unit 2 may perform the settlement process based on the recognition result stored in the third list, and may also display, in list form, the recognition result of the object for which the recognition has failed and the object of the recognition candidate based on the recognition result stored in the fourth list on the notification unit 3, and may suggest that the customer correct it using the input unit provided in the notification unit 3. The process based on the third list or the fourth list may be the same even in the second embodiment and subsequent embodiments.

[0049] Hereinafter, the processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG. 4.

[0050] <s001>The first imaging unit 1 starts continuous imaging at an arbitrary frame rate. After that, <s002>Proceed to

[0051] <s002>The control unit 2 determines that it is the first stage. After that, <s003>Proceed to.

[0052] <s003>The control unit 2 determines whether to shift from the first stage to the second stage. As a determination method, for example, the method described in paragraphs

[0037] to

[0039] may be used. When the control unit 2 determines that it should shift from the first stage to the second stage, the control unit 2 shifts from the first stage to the second stage. After that, <s008>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s004>Proceed to

[0053] <s004>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. When the control unit 2 detects an object, <s005>Proceed. When the control unit 2 does not detect an object, <s003>Return to.

[0054] Note that, as a method for object detection, a publicly known object detection method may be used. The publicly known object detection method refers to, for example, a method using segmentation, semantic segmentation and / or a bounding box, or performing machine learning including pattern matching or deep learning in combination with object recognition described later.

[0055] <s005>The control unit 2 performs object recognition based on the object detection result. Thereafter, <s006>Proceed to

[0056] As the object recognition method, a publicly known object recognition method may be used. The publicly known object recognition method includes, for example, machine learning including pattern matching or deep learning.

[0057] <s006>The control unit 2 determines whether the recognition result should be added to the first list as a purchase candidate product. As criteria for determining that it should be added, for example, the reliability of the recognition result is equal to or higher than a predetermined threshold value, and it is determined that the movement of the object is to be put into the container 11. When the control unit 2 determines that the recognition result should be added to the first list as a purchase candidate product, <s007>Proceed. If the control unit 2 determines that the recognition result should not be added to the first list as a purchase candidate product, the recognition result is discarded and then <s003>Return.

[0058] <s007>The control unit 2 adds the recognition result to the first list. After that, <s003>Return to.

[0059] <s008>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. When the control unit 2 detects an object, <s009>Proceed to. When the control unit 2 does not detect an object, <s011>Proceed to.

[0060] <s009>The control unit 2 performs object recognition based on the object detection result. Then, <s010>Proceed to

[0061] <s010>The control unit 2 adds the recognition result to the second list. After that, <s011>Proceed to

[0062] <s011>The control unit 2 determines whether to end the second stage. As a determination method, for example, the case where an object is not detected for a certain period of time can be cited. When the control unit 2 determines to end the second stage, <s012>Proceed. When the control unit 2 does not determine that the second stage has ended, <s008>Return to.

[0063] <s012>The control unit 2 determines whether the recognition results included in the first list and the second list exactly match. If the recognition results included in the first list and the second list exactly match, <s013>Proceed to. If the recognition results included in the first list and the second list do not exactly match, <s014>Proceed to.

[0064] <s013>The control unit 2 copies the recognition results included in the second list and adds them to the third list. Thereafter, the information processing system ends the process.

[0065] <s014>When the control unit 2 finds a matching recognition result of an object between the first list and the second list, it copies the recognition result included in the second list and adds it to the third list. For the recognition result of an object that does not match between the first list and the second list, the control unit 2 adds the non-matching recognition result of the object to the fourth list. After that, the control unit 2 performs processing based on the third list and the fourth list, and then ends the overall processing.

[0066] Next, an information processing method of the information processing system according to the second embodiment of the present disclosure will be described. In the information processing method of the information processing system according to the first embodiment of the present disclosure, continuous imaging is performed, but there is a concern that the amount of data handled by the control unit 2 will increase. Therefore, in the information processing method of the information processing system according to the second embodiment of the present disclosure, the imaging timing of the first imaging unit 1 is estimated by predicting the timing when the purchaser puts an object into the container 11. Then, based on the estimated imaging timing, the first imaging unit 1 performs imaging. By this method, it is expected that the first imaging unit 1 does not need to perform continuous imaging, and the amount of data handled by the control unit 2 is reduced.

[0067] FIG. 5 is an image diagram regarding an imaging method and the like in the first stage of the second embodiment. FIG. 5 shows that the sensor 4 is added as a new component compared to FIG. 2 and the first embodiment. Here, the sensor 4 is provided on the ceiling inside the retail store, but in reality, it is not limited to this pattern.

[0068] The sensor 4 monitors the container 11 and / or the customer holding the container 11, and transmits the acquired detection data to the control unit 2.

[0069] The control unit 2 determines whether the object A is put into the container 11 based on the detection data acquired by the sensor 4. As a determination method, for example, the method described in paragraphs

[0029] to

[0032] may be used.

[0070] When the control unit 2 determines that the object A has been put into the container 11 based on the detection data acquired by the sensor 4, the control unit 2 transmits a trigger signal for imaging to the first imaging unit 1.

[0071] When the first imaging unit 1 receives the trigger signal from the control unit 2, the first imaging unit 1 performs imaging. After that, the first imaging unit 1 transmits the captured image to the control unit 2.

[0072] Although FIG. 5 is an explanation of the imaging method in the first stage, the same imaging method may be used in the second stage.

[0073] Hereinafter, based on FIG. 6, the processing flow of the information processing system according to the second embodiment of the present disclosure will be described.

[0074] <s101>This process is of the first embodiment <s002>corresponds to. After the execution of this process, <s102>Proceed to

[0075] <s102>Based on the detection data acquired by the sensor 4, the control unit 2 predicts the timing at which the purchaser puts an object into the container 11. Thereafter, <s103>Proceed to

[0076] <s103>The control unit 2 determines whether it is the timing at which the first imaging unit 1 performs imaging based on the prediction result of the timing when the purchaser puts the object into the container 11. When it is determined that it is the timing at which the first imaging unit 1 performs imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. Thereafter, <s104>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s102>Return to.

[0077] <s104>The first imaging unit 1 captures an image of an object based on a trigger signal from the control unit 2. Thereafter, <s105>Proceed to

[0078] <s105>This process is for the first embodiment <s005>corresponds to. After the execution of this process, <s106>Proceed to

[0079] <s106>This process is for the first embodiment <s006>corresponds thereto. When the control unit 2 determines that the recognition result should be added to the first list as a purchase candidate product, <s107>Proceed. If the control unit 2 determines that the recognition result should not be added to the first list as a purchase candidate product, after discarding the recognition result <s102>Return to

[0080] <s107>This process is of the first embodiment <s007>corresponds to. After the execution of this process, <s108>Proceed to

[0081] <s108>This process is of the first embodiment <s003>corresponds to. When the control unit 2 determines that it should shift from the first stage to the second stage, the control unit 2 shifts from the first stage to the second stage. After that, <s109>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s102>Return.

[0082] <s109>Based on the detection data acquired by the sensor 4, the control unit 2 predicts the timing at which the purchaser takes out the object from the container 11. Thereafter, <s110>Proceed to

[0083] <s110>This process is <s103>corresponds to it. When it is determined that it is the timing for the first imaging unit 1 to perform imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. After that, <s111>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s109>Return to.

[0084] <s111>even <s113>These processes are respectively <s104> 、 <s105>and <s010>Corresponds to. <s111>After the processing of <s112>to <s112>After the processing of <s113>to <s113>After the processing of <s114>Proceed to

[0085] <s114>This process is of the first embodiment <s011>corresponds to. When the control unit 2 determines that the second stage has ended, <s115>Proceed. When the control unit 2 does not determine that the second stage has ended, <s109>Return to.

[0086] <s115>This process is of the first embodiment <s012>corresponds to it. When the recognition results included in the first list and the second list exactly match, <s116>Proceed to. If the recognition results included in the first list and the second list do not exactly match, <s117>Proceed to

[0087] <s116>and <s117>This process is respectively <s013>and <s014>It corresponds to this. The control unit 2 executes each process and performs processes based on the third list and the fourth list, and then ends the overall process.

[0088] Next, an information processing method of the information processing system according to the third embodiment of the present disclosure will be described. In the information processing methods of the information processing systems according to the first and second embodiments of the present disclosure, a determination process is always performed as to whether the recognition results based on the images captured in the first stage and the second stage match each other. However, for example, when a high recognition rate is obtained in the recognition result in the first stage, it is also assumed that processing based on two captures and the captured images is not always necessary. Therefore, in the information processing method of the information processing system according to the third embodiment of the present disclosure, when the recognition rate in object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition (for example, the recognition rate is 90% or more, any threshold value), it is directly added to the third list. When the control unit 2 confirms that all objects have been added to the third list, object recognition based on the image captured in the second stage is not performed. By this method, it is expected that the data processing amount of the control unit 2 will be reduced.

[0089] Hereinafter, the processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG. 7.

[0090] <s201>even to <s205>These processes are respectively those of the second embodiment <s101>even <s105>corresponds to. <s201>After the processing of <s202>to <s202>After the processing of <s203>to <s203>After the processing of <s204>to <s204>After the processing of <s205>to <s205>After the processing of <s206>Proceed to

[0091] <s206>The control unit 2 determines whether the recognition rate obtained by object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition. When it is determined that the recognition result satisfies the predetermined condition, <s207>Proceed. If it is not determined that the recognition result satisfies a predetermined condition, <s208>Proceed to

[0092] <s207>The control unit 2 adds the recognition result to the third list. Thereafter, <s209>Proceed to

[0093] <s208>The control unit 2 adds the recognition result to the first list. Then, <s209>Return.

[0094] <s209>This process is of the second embodiment <s108>corresponds to. When the control unit 2 determines that it should transition from the first stage to the second stage, the control unit 2 transitions from the first stage to the second stage. After that, <s210>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s202>Return.

[0095] <s210>The control unit 2 determines whether the first list is a null value, that is, whether all recognized objects satisfy a predetermined condition. If it is determined that the first list is a null value, the information processing system simply ends the process. If it is not determined that the first list is a null value, <s211>Proceed to

[0096] <s211>This process is of the second embodiment <s109>corresponds to. After the execution of this process, <s212>Proceed to

[0097] <s212>This process is of the second embodiment <s110>corresponds to. When it is determined that it is the timing for the first imaging unit 1 to perform imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. After that, <s213>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s211>Return.

[0098] <s213>even <s215>These processes are respectively those of the second embodiment <s111>even to <s113>corresponds to. <s213>After the processing of <s214>to, <s214>After the processing of <s215>to, <s215>After the processing of <s216>Proceed to

[0099] <s216>This process is for the second embodiment <s114>corresponds to. When the control unit 2 determines that the second stage has ended, <s217>Proceed to. When the control unit 2 does not determine that the second stage has ended, <s211>Return.

[0100] <s217>The control unit 2 determines whether the recognition result included in the second list exactly matches the combination of the recognition results included in the first list and the third list. If the combination of the recognition results included in the first list and the third list exactly matches, <s218>Proceed. If the combination of the recognition results included in the first list and the third list does not exactly match, <s219>Proceed to

[0101] <s218>and <s219>These processes are respectively those of the second embodiment <s116>and <s117>It corresponds to. By executing each process, the information processing system ends the process.

[0102] The above is the description of the information processing method of the information processing system according to the first to third embodiments.

[0103] In the above description, it has been mentioned up to the point where the control unit 2 adds the recognition result of the object to the third list or the fourth list. However, as described in, for example, paragraphs

[0027] and

[0028] , after the control unit 2 adds the recognition result of the object to the third list or the fourth list, the control unit 2 may transmit a control signal to the notification unit 3 based on the third list or the fourth list.

[0104] For example, the notification unit 3 may display the recognition result of the object for which recognition has failed and the candidate objects for recognition in list form based on the fourth list, and suggest that the customer use the input unit provided in the notification unit 3 for correction.

[0105] Also, when the recognition result of the object is included in the fourth list, the notification unit 3 may notify the customer or an employee of the retail store, etc. of an error warning.

[0106] Also, the notification unit 3 may display the total amount based on the third list and the fourth list reflecting the above-mentioned correction by the customer, and suggest that the customer perform accounting processing.

[0107] Note that up to this point, regarding the information processing system, the information processing apparatus, and the information processing method, it should be noted that it is easy to make corrections to the present disclosure in the various figures.

[0108] For example, in the information processing method of the information processing system according to the first to third embodiments, in the first stage, based on the moving direction of the object included in the recognition result of the first imaging unit 1, when the control unit 2 detects that an object already added to the first list is taken out of the container 11, the control unit 2 may delete the recognition result of the object taken out of the container 11 from the first list.

[0109] Also, in the information processing method of the information processing system according to the first to third embodiments, although the collation with the first list is performed after the generation of the second list is completed, for example, at the same time as adding the recognition result to the second list, it may be collated whether the added recognition result is included in the first list.

[0110] Also, in the information processing method of the information processing system according to the first to third embodiments, although the control unit 2 performs processing based on the first to fourth lists, as shown in FIG. 8, for example, without using the second to fourth lists, as attributes of the recognition results included in the first list, "Recognition in the first imaging" represented by two values of Yes / No (hereinafter Y / N) corresponding to the first list and "Recognition in the second imaging" corresponding to the second list may be newly used (hereinafter, parentheses are omitted). In this case, Y may be given to "Recognition in the first imaging" for the recognition result added in the first stage, and Y may be given to "Recognition in the second imaging" for the recognition result added in the second stage. That is, Y is given to both "Recognition in the first imaging" and "Recognition in the second imaging" for the recognition results of the objects recognized in both the first stage and the second stage, and for the recognition results of the objects recognized in the first stage but not recognized in the second stage, Y is given to "Recognition in the first imaging" and N is given to "Recognition in the second imaging", and for the recognition results of the objects not recognized in the first stage but recognized in the second stage, N is given to "Recognition in the first imaging" and Y is given to "Recognition in the second imaging".

[0111] For example, referring to FIG. 8, for the apple with ID number 1, Y is given to both "Recognition in the first imaging" and "Recognition in the second imaging", and the control unit 2 may process the apple as the final confirmed purchase item corresponding to the third list. On the other hand, for the nothing and mandarin with ID numbers 2 and 3, N is given to either "Recognition in the first imaging" or "Recognition in the second imaging", and the control unit 2 may process the nothing and mandarin as the final recognition failure items corresponding to the fourth list.

[0112] Also, in the information processing method of the information processing system according to the third embodiment, the control unit 2 causes the storage unit 23 to temporarily store, in the first stage, those recognition results of objects that do not satisfy a predetermined condition, and in the second stage, the control unit 2 may prompt the customer via the notification unit 3 to cause only those with a low recognition rate to be imaged by the first imaging unit 1. That is, for those whose recognition rate of object recognition satisfies a predetermined condition in the first stage, the control unit 2 may perform a process of finally determining the recognition result as it is, and for those whose recognition rate of object recognition does not satisfy a predetermined condition in the first stage, the control unit 2 may perform a process based on the image captured by the first imaging unit 1 in the second stage.

[0113] Also, in the information processing method of the information processing system according to the first to third embodiments, although the customer is supposed to use the container 11, it is not necessarily the case that the customer uses the container 11. For example, it is also assumed that the customer holds it by their own hand. In that case, the information processing system may, for example, instead of providing the first imaging unit 1 in the container 11, provide it on a merchandise shelf or ceiling in the store of the retail store. For example, the first imaging unit 1 may constantly track the customer and constantly associate the first list with the customer.

[0114] Also, in the information processing method of the information processing system according to the first to third embodiments, the first imaging unit 1 that captures an object in the first stage and the second stage is the same, but it is not necessarily so. For example, the first imaging unit 1 in the first stage may be the one provided in the container 11, and the first imaging unit 1 in the second stage may be the one provided in the cash register terminal, and different ones may be used respectively.

[0115] Furthermore, as described above, the solution means of the present disclosure has been described as an information processing system, an information processing apparatus, and an information processing method. However, the present disclosure can also be realized as an aspect including these, and can also be realized as a program substantially corresponding to these and a storage medium storing the program. It should be understood that these are also included in the scope of the present disclosure.

Explanation of Reference Numerals

[0116] 1 First imaging unit 2 Control unit 3 Notification unit 4 Sensor 11 Container 12 Communication unit of the container 21 Calculation unit 22 Communication unit of the control unit 23 Storage unit 31 Communication unit of the notification unit 41 Communication unit of the sensor < / s104>

Claims

1. A mounting table for mounting an object, the mounting table including a specific area provided with a marker; an imaging unit that generates a captured image including an image of the object; a control unit that recognizes the object included in the captured image, The control unit recognizes an object included in the captured image after recognizing the marker, and causes a notification unit to notify the recognition result. Information processing system.

2. When the control unit recognizes an object, the control unit adds the recognition result to a list. The information processing system according to claim 1 .

3. A mounting table for mounting an object, the mounting table including a specific area provided with a marker; An imaging unit that captures an image; A control unit that recognizes the object captured by the imaging unit, The control unit causes a display to display a recognition result of the object recognized after the marker included in the captured image is recognized after the imaging unit starts imaging. Information processing system.

4. A mounting table for mounting an object, the mounting table including a specific area provided with a marker; An imaging unit that captures an image; A control unit that recognizes the object captured by the imaging unit, The control unit, after starting imaging by the imaging unit, causes a display to display a recognition result of the object recognized before the marker included in the captured image is recognized, after the marker is recognized. Information processing system.

5. A mounting table for mounting an object, the mounting table including a specific area provided with a marker; An imaging unit that captures an image; A control unit that recognizes the object captured by the imaging unit, the control unit, after starting imaging by the imaging unit, determines a recognition result of the object recognized before the marker included in the captured image is recognized as a recognition result after the marker is recognized, and displays the result on a display. Information processing system.

6. The recognition result includes at least an object name of the object.

6. The information processing system according to claim 3.

Images