Information processing system, information processing apparatus, information processing method

Abstract

To track movement of a commodity from a container or the like including a shopping basket to a storage bag or the like without a purchaser scanning the commodity by hand, and to determine whether or not the commodity has been properly recognized.SOLUTION: Provided is an information processing system including: a first imaging unit 1; and a controller 2 configured to perform processing relating to an image captured by the first imaging unit 1. The controller 2 performs the processing based on an image captured at a first stage at which an object is put into a container 11 and an image captured at a second stage at which the object is taken out of the container 11.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, and an information processing method.

Background Art

[0002] In the settlement of purchased goods in the retail industry, for example, Patent Document 1 discloses an invention that tracks the process from when a product is taken out of a shopping basket until it is placed in a storage bag and determines whether the product has been properly scanned.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention according to this disclosure, it is necessary for the purchaser to manually scan the product for identification of the product, and there is room for improvement in this regard.

[0005] Therefore, an object of the present disclosure made in view of the above points is to track the process from when a product is placed in a storage bag or the like from a container including a shopping basket without the purchaser manually scanning the product, and to determine whether the product has been properly recognized.

Means for Solving the Problems

[0006] To achieve the above-described object, an information processing system according to one aspect of the present disclosure includes a first imaging unit and a control unit that performs processing on an image captured by the first imaging unit, and the control unit performs processing based on images captured in each of a first stage of putting an object into a container and a second stage of taking the object out of the container.

[0007] Also, an information processing apparatus according to an aspect of the present disclosure includes a control unit that performs processing related to an image captured by a first imaging unit, and the control unit performs processing based on images captured in each of a first stage of putting an object into a container and a second stage of taking the object out of the container.

[0008] Also, an information processing method according to an aspect of the present disclosure includes the first imaging unit performing imaging in a first stage of putting an object into a container, the first imaging unit performing imaging in a second stage of taking the object out of the container, and performing processing based on images captured by the first imaging unit in each of the first stage and the second stage.

Advantages of the Invention

[0009] According to the present disclosure configured as described above, it is possible to track the process from when a purchaser manually scans a product until the product is placed in a storage bag or the like from a shopping basket without manually scanning the product, and to determine whether the product is properly recognized.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of an information processing system according to the present disclosure will be described with reference to the drawings.

[0012] As shown in FIG. 1, the information processing system according to the present disclosure includes a first imaging unit 1 and a control unit 2. Further, the information processing system may include a notification unit 3 and a sensor 4.

[0013] In the following figures, a solid arrow connecting each functional block indicates the flow of a control signal or communicated data. The communication indicated by the arrow may be either wired / wireless or a combination of both.

[0014] The first imaging unit 1 images an object that is put into and taken out of a container 11 that temporarily stores the object.

[0015] In order to image an object and generate an image, the first imaging unit 1 includes an imaging optical system and an imaging element. The imaging optical system includes, for example, one or more optical members such as lenses and diaphragms. The lens may be any type regardless of the focal length, and may be, for example, a general lens, a wide-angle lens including a fish-eye lens, or a zoom lens with a variable focal length. The imaging optical system forms a subject image on the light receiving surface of the imaging element. The imaging element includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The imaging element images the subject image formed on the light receiving surface to generate an image.

[0016] Note that the first imaging unit 1 may continuously capture still images in any frame, or may discontinuously capture still images at specific timings. Further, the first imaging unit 1 may capture a moving image.

[0017] Also, the first imaging unit 1 may be provided in any form as long as it can image an object that enters and exits a container 11 including a shopping basket or a shopping cart. For example, it may be provided in the container 11, or may be provided in the form of a portable electronic terminal including a smartphone or a business-use terminal, etc., or may be provided on a merchandise shelf, ceiling, cash register, etc. inside a retail store. In particular, when provided in the container 11, it is easy to image an object that enters and exits the container 11. Also, when the first imaging unit 1 is provided in a portable electronic terminal, the container 11 may be provided with a holder for holding the portable electronic terminal. Note that in the following description, it is assumed to be provided in the container 11 (also shown as being provided in the container 11 in FIG. 1).

[0018] Also, the image captured by the first imaging unit 1 is transmitted to the control unit 2 via the communication unit 12 provided together with the first imaging unit 1.

[0019] The communication units 12 (hereinafter, the communication unit 12 of the container), 22 (the communication unit 22 of the control unit), 31 (the communication unit 31 of the notification unit), and 41 (the communication unit 41 of the sensor), including those provided in the container 11, the control unit 2, the notification unit 3, and the sensor 4 described later, are configured to include either or both of a wired / wireless module. The communication units 12, 22, 31, and 41 do not necessarily have the same configuration and may have different configurations respectively. The communication units 12, 22, 31, and 41 transmit and receive control signals or data.

[0020] The control unit 2 performs processing etc. based on the image captured by the first imaging unit 1.

[0021] The control unit 2 is configured to include, as internal modules, an arithmetic unit 21, a communication unit 22 of the control unit, and a storage unit 23. Physically, the control unit 2 includes a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array), and in addition, as the storage unit 23, it is configured to include any one or more of, for example, a semiconductor memory, a magnetic memory, and an optical memory. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape. The optical memory may be configured to include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray (registered trademark) Disc).

[0022] The arithmetic unit 21 includes, as processing based on the image captured by the first imaging unit 1, for example, control related to the imaging of the first imaging unit 1, object detection / recognition processing related to the image captured by the first imaging unit 1, data processing related to the recognition result, and control related to the notification of the notification unit 3 related to the data processing result. The data processing related to the recognition result includes, for example, status management of the recognition result. The status management includes, for example, adding or deleting the recognition result of the object to / from a list regarding the addition or deletion of an object in the container 11.

[0023] The arithmetic unit 21 may output, as the recognition result and the attribute information of the recognition result, for example, the object name (product name) of the recognized object, the price, the moving direction of the object, and the reliability. The moving direction of the object includes, for example, binary information indicating whether the object has been put into the container 11 or taken out of the container 11. For example, the reliability is a quantification of how reliable the recognition result is, and it can be said that the higher the reliability, the higher the accuracy of the recognition result.

[0024] The communication unit 22 of the control unit performs, for example, the transmission and reception of the following control signals or data. 1. Receives the image captured by the first imaging unit 1 from the communication unit 12 of the container. 2. Receives the signal acquired by the sensor 4 from the communication unit 41 of the sensor. 3. Transmits and receives various control signals or data with the arithmetic unit 21. 4. Transmits the control signal of the first imaging unit 1 to the communication unit 12 of the container. 5. Transmits the control signal of the notification unit 3 to the communication unit 31 of the notification unit.

[0025] The storage unit 23 stores at least one or more of the image captured by the first imaging unit 1, the recognition result obtained by the object detection / recognition process performed by the arithmetic unit 21 based on the image captured by the first imaging unit 1, the processing result of the process performed based on the recognition result, the signal acquired from the sensor 4, or the program related to the processing in the information processing system. including storing at least any one or more of the above.

[0026] Note that the control unit 2 does not necessarily have to be on-premises, and the processing may be performed in the cloud on a web service. Also, the control unit 2 may be functionally divided such that the processing is performed on-premises or in the cloud in module units. For example, the arithmetic unit 21 may perform the processing in the cloud. Also, the storage unit 23 may perform the processing on-premises.

[0027] The notification unit 3 performs notification based on the control signal and the like from the control unit 2 via the communication unit 31 of the notification unit. The content of the notification may include, for example, the recognition result obtained by the control unit 2 or the result obtained by performing processing based on the recognition result.

[0028] As means for performing the above-described notification, the notification unit 3 includes, for example, those appealing to vision, hearing, or touch. Therefore, the notification unit 3 may be configured to include, for example, a display, a lamp, a speaker, a motor, etc. Also, the notification unit 3 may be provided with an input unit for, for example, correcting or performing a confirmed input regarding the notified recognition result or the like. The input unit may include, for example, a mouse, a keyboard, a microphone, a touch sensor, and the touch sensor may be provided in the display as a touch display.

[0029] The sensor 4 monitors the movement of the purchaser or the object, and transmits the detected data obtained by the monitoring to the control unit 2.

[0030] The sensor 4 includes, for example, a weight sensor, an infrared sensor, and a second imaging unit configured to include an imaging optical system and an imaging element. Also, regarding the arithmetic processing related to these control or detection data, the control unit 2 may perform it, or the sensor 4 may be separately provided with a component corresponding to the control unit 2 and perform this.

[0031] When the sensor 4 is provided in the container 11, for example, it may be provided as a weight sensor at the bottom of the container 11. When the weight detected by the weight sensor changes, assuming that an object is added or removed from the container 11, the control unit 2 or the weight sensor may generate a trigger signal for the first imaging unit 1 to perform imaging based on the detection result of the weight sensor.

[0032] Also, when the sensor 4 is provided outside the container 11, for example, it may be provided on a merchandise shelf or the ceiling inside a retail store. For example, when a weight sensor is provided on the merchandise shelf as the sensor 4 and the weight detected by the weight sensor changes, assuming that an object is added or removed from the container 11, the weight sensor may generate a trigger signal. Similarly, for example, when a second imaging unit is provided on the ceiling as the sensor 4 and a specific movement (such as an action of a purchaser taking an object out of the merchandise shelf, a trajectory of an object moving from the merchandise shelf, etc.) attempting to add or remove an object from the container 11 is detected in the image captured by the second imaging unit, assuming that an object is added or removed from the container 11, the control unit 2 or the second imaging unit may generate a trigger signal based on the image captured by the second imaging unit.

[0033] Next, with reference to FIGS. 2 and 3, the imaging method and the like in the first stage and the second stage will be briefly described.

[0034] Figure 2 is an image diagram regarding an imaging method and the like in the first stage of putting an object into container 11. The first stage here includes the stage of a purchaser selecting a product (object) to be purchased. That is to say, the first stage can be paraphrased as the pre - recognition - determination stage before a purchaser determines the purchased product (determines the recognition result of the product) in a specific area including the cash register for accounting processing.

[0035] According to Figure 2, when object A (the object) is put into container 11, the first imaging unit 1 transmits the image of the imaged object A to the control unit 2.

[0036] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains the recognition result.

[0037] Figure 3 is an image diagram regarding an imaging method and the like in the second stage of taking an object out of container 11. The second stage here includes the stage of a purchaser performing accounting processing on the product to be purchased. That is to say, the second stage can be paraphrased as the recognition - determination stage for determining the purchased product in a specific area.

[0038] According to Figure 3, first, in order to transition from the first stage to the second stage, for example, the first imaging unit 1 images a marker provided in a specific area and transmits the imaged image to the control unit 2. Then, the control unit 2 transitions from the first stage to the second stage. That is to say, it can be paraphrased that the stage before the control unit 2 recognizes the marker provided in the specific area is the first stage, and the stage after recognizing the marker is the second stage.

[0039] Note that in order to determine the transition from the first stage to the second stage, other means can also be used. For example, a weight sensor can be provided in a specific area, and based on the weight sensor detecting a change in weight when container 11 is placed on the weight sensor, or a separate sensor can be provided in container 11, and based on the separate sensor detecting a specific signal or object including the marker provided in the specific area.

[0040] After moving to the second stage, when the object A is taken out of the container 11, the first imaging unit 1 transmits the captured image of the object A to the control unit 2.

[0041] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains the recognition result.

[0042] Thereafter, the control unit 2 performs final determination processing of the recognition result based on the images captured in each of the first stage of putting the object into the container 11 and the second stage of taking the object out of the container 11.

[0043] At this time, the control unit 2 may transmit a control signal so that the notification unit 3 is notified of the recognition result on which the final determination processing has been performed. Further, the control unit 2 may transmit a control signal so that the notification unit 3 is notified of the recognition result of the object imaged by the first imaging unit 1 in the second stage before the final determination processing of the recognition result. For example, according to FIG. 3, when each recognition result obtained based on the images of the object A captured in the first stage and the second stage matches "apple", the notification unit 3 displays "apple".

[0044] Next, an information processing method of the information processing system according to the first embodiment of the present disclosure will be described. In the information processing method of the information processing system according to the first embodiment of the present disclosure, the first imaging unit 1 continuously captures images at an arbitrary frame rate, and the control unit 2 performs object detection and object recognition based on the images captured by the first imaging unit 1. Then, as a method for managing the status of the recognition result, the first to fourth lists are used, and based on these lists, the control unit 2 performs final determination processing of the recognition result.

[0045] In the first stage, the control unit 2 adds, as a purchase candidate product, an object determined to be the object put into the container 11 to the first list that stores the recognition result of the purchase candidate product.

[0046] Also, in the second stage, the control unit 2 determines that it is the one taken out of the container 11 The recognized item is used as the purchased item and added to a second list that stores the recognition result of the purchased item.

[0047] Then, it is verified whether the first list and the second list exactly match. If they exactly match, the control unit 2 directly uses the recognition result of the object included in the second list as the finally determined purchased item and adds it to a third list that stores the recognition result of the finally determined purchased item, thereby determining the recognition result of the object. If the first list and the second list do not exactly match, the recognition result of the matching object is added to the third list, and the recognition result of the non-matching object, that is, the recognition result of the finally recognized failed item, is added to a fourth list that stores it.

[0048] The control unit 2 may perform settlement processing based on the recognition result stored in the third list, and may also display, in list form, at the notification unit 3 the recognition result of the object for which recognition has failed and the object of the recognition candidate based on the recognition result stored in the fourth list, and may suggest that the customer use the input unit provided in the notification unit 3 to make corrections. The processing based on the third list or the fourth list may be the same even in the second embodiment and subsequent embodiments.

[0049] Hereinafter, the processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG. 4.

[0050] <s001>The first imaging unit 1 starts continuous imaging at an arbitrary frame rate. Thereafter, <s002>Proceed to

[0051] <s002>The control unit 2 determines that it is the first stage. After that, <s003>Proceed to

[0052] <s003>The control unit 2 determines whether to shift from the first stage to the second stage. As a determination method, for example, the method described in paragraphs

[0037] to

[0039] may be used. When the control unit 2 determines that it should shift from the first stage to the second stage, the control unit 2 shifts from the first stage to the second stage. After that, <s008>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s004>Proceed to

[0053] <s004>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. When the control unit 2 detects an object, <s005>Proceed. When the control unit 2 does not detect an object, <s003>Return to

[0054] Note that, as a method for object detection, a publicly known object detection method may be used. The publicly known object detection method includes, for example, a method using segmentation, semantic segmentation, and / or a bounding box, or performing machine learning including pattern matching or deep learning in combination with object recognition described later.

[0055] <s005>The control unit 2 performs object recognition based on the object detection result. Then, <s006>Proceed to

[0056] Note that, as an object recognition method, a publicly known object recognition method may be used. The publicly known object recognition method includes, for example, machine learning including pattern matching or deep learning.

[0057] <s006>The control unit 2 determines whether the recognition result should be added to the first list as a purchase candidate product. As criteria for addition, for example, the reliability of the recognition result being equal to or higher than a predetermined threshold, and the movement of the object being determined to be one that is put into the container 11 can be cited. When the control unit 2 determines that the recognition result should be added to the first list as a purchase candidate product, <s007>Proceed. When the control unit 2 determines that the recognition result should not be added to the first list as a purchase candidate product, after discarding the recognition result <s003>Return to

[0058] <s007>The control unit 2 adds the recognition result to the first list. Then, <s003>Return.

[0059] <s008>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. When the control unit 2 detects an object, <s009>Proceed. When the control unit 2 does not detect an object, <s011>Proceed to

[0060] <s009>The control unit 2 performs object recognition based on the object detection result. Then, <s010>Proceed to

[0061] <s010>The control unit 2 adds the recognition result to the second list. After that, <s011>Proceed to

[0062] <s011>The control unit 2 determines whether to end the second stage. As a determination method, for example, the case where an object is not detected for a certain period of time can be cited. When the control unit 2 determines to end the second stage, <s012>Proceed. When the control unit 2 does not determine that the second stage has ended, <s008>Return.

[0063] <s012>The control unit 2 determines whether the recognition results included in the first list and the second list exactly match. When the recognition results included in the first list and the second list exactly match, <s013>Proceed to. If the recognition results included in the first list and the second list do not exactly match, <s014>Proceed to

[0064] <s013>The control unit 2 copies the recognition results included in the second list and adds them to the third list. Thereafter, the information processing system terminates the process.

[0065] <s014>For the recognition results of the objects that match between the first list and the second list, the control unit 2 copies the recognition results included in the second list and adds them to the third list. Then, for the recognition results of the objects that do not match between the first list and the second list, the control unit 2 adds the recognition results of the unmatched objects to the fourth list. After that, the control unit 2 performs processing based on the third list and the fourth list, and then ends the overall processing.

[0066] Next, the information processing method of the information processing system according to the second embodiment of the present disclosure will be described. In the information processing method of the information processing system according to the first embodiment of the present disclosure, continuous imaging is performed, but there is a concern that the amount of data handled by the control unit 2 will increase. Therefore, in the information processing method of the information processing system according to the second embodiment of the present disclosure, the imaging timing of the first imaging unit 1 is estimated by predicting the timing when the purchaser puts an object into the container 11. Then, based on the estimated imaging timing, the first imaging unit 1 performs imaging. By this method, it is expected that the first imaging unit 1 does not need to perform continuous imaging, and the amount of data handled by the control unit 2 is reduced.

[0067] FIG. 5 is an image diagram regarding the imaging method and the like in the first stage of the second embodiment. FIG. 5 shows that the sensor 4 is added as a new component compared to FIG. 2 and the first embodiment. Here, the sensor 4 is provided on the ceiling inside the retail store, but in reality, it is not limited to this pattern.

[0068] The sensor 4 monitors the container 11 and / or the customer holding the container 11, and transmits the acquired detection data to the control unit 2.

[0069] The control unit 2 determines whether the object A is put into the container 11 based on the detection data acquired by the sensor 4. As a determination method, for example, the method described in paragraphs

[0029] to

[0032] may be used.

[0070] When the control unit 2 determines that the object A has been put into the container 11 based on the detection data acquired by the sensor 4, the control unit 2 transmits a trigger signal for imaging to the first imaging unit 1.

[0071] When the first imaging unit 1 receives the trigger signal from the control unit 2, the first imaging unit 1 performs imaging. After that, the first imaging unit 1 transmits the captured image to the control unit 2.

[0072] Although FIG. 5 describes the imaging method in the first stage, the same imaging method may be used in the second stage.

[0073] Hereinafter, based on FIG. 6, the processing flow of the information processing system according to the second embodiment of the present disclosure will be described.

[0074] <s101>This process is of the first embodiment <s002>corresponds to. After the execution of this process, <s102>Proceed to

[0075] <s102>Based on the detection data acquired by the sensor 4, the control unit 2 predicts the timing at which the purchaser inserts an object into the container 11. Thereafter, <s103>Proceed to

[0076] <s103>The control unit 2 determines whether it is the timing at which the first imaging unit 1 performs imaging based on the prediction result of the timing at which the purchaser throws an object into the container 11. When it is determined that it is the timing at which the first imaging unit 1 performs imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. After that, <s104>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s102>Return to.

[0077] <s104>The first imaging unit 1 captures an image of an object based on a trigger signal from the control unit 2. Thereafter, <s105>Proceed to

[0078] <s105>This process is of the first embodiment <s005>corresponds to. After the execution of this process, <s106>Proceed to

[0079] <s106>This process is of the first embodiment <s006>corresponds to this. When the control unit 2 determines that the recognition result should be added to the first list as a purchase candidate product, <s107>Proceed. If the control unit 2 determines that the recognition result should not be added to the first list as a purchase candidate product, after discarding the recognition result <s102>Return to

[0080] <s107>This process is for the first embodiment <s007>corresponds to. After the execution of this process, <s108>Proceed to

[0081] <s108>This process is of the first embodiment <s003>corresponds to. When the control unit 2 determines that it should shift from the first stage to the second stage, the control unit 2 shifts from the first stage to the second stage. After that, <s109>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s102>Return.

[0082] <s109>Based on the detection data acquired by the sensor 4, the control unit 2 predicts the timing at which the purchaser takes out the object from the container 11. Thereafter, <s110>Proceed to

[0083] <s110>This process is <s103>It corresponds to. When it is determined that it is the timing for the first imaging unit 1 to perform imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. After that, <s111>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s109>Return to.

[0084] <s111>even <s113>These processes are each <s104> 、 <s105>and <s010>Corresponds to. <s111>After the processing of <s112>to, <s112>After the processing of <s113>to <s113>After the processing of <s114>Proceed to

[0085] <s114>This process is of the first embodiment <s011>corresponds to. When the control unit 2 determines that the second stage has ended, <s115>Proceed to. When the control unit 2 does not determine that the second stage has ended, <s109>Return.

[0086] <s115>This process is of the first embodiment <s012>corresponds to. When the recognition results included in the first list and the second list exactly match, <s116>Proceed to. If the recognition results included in the first list and the second list do not exactly match, <s117>Proceed to

[0087] <s116>and <s117>This process is respectively <s013>and <s014>It corresponds to. The control unit 2 performs the execution of each process and the process based on the third list and the fourth list, and then ends the overall process.

[0088] Next, an information processing method of the information processing system according to the third embodiment of the present disclosure will be described. In the information processing methods of the information processing systems according to the first and second embodiments of the present disclosure, a determination process is always performed as to whether the recognition results based on the images captured in the first stage and the second stage match. However, for example, when a high recognition rate is obtained in the recognition result in the first stage, it is assumed that processing based on imaging twice and the captured images is not always necessary. Therefore, in the information processing method of the information processing system according to the third embodiment of the present disclosure, when the recognition rate in object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition (for example, the recognition rate is 90% or more, any threshold value), it is directly added to the third list. When the control unit 2 confirms that all objects have been added to the third list, object recognition based on the image captured in the second stage is not performed. By this method, it is expected that the data processing amount of the control unit 2 will be reduced.

[0089] Hereinafter, the processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG. 7.

[0090] <s201>even <s205>These processes are each of the second embodiment <s101>even <s105>Corresponds to. <s201>After the processing of <s202>to <s202>After the processing of <s203>to, <s203>After the processing of <s204>to <s204>After the processing of <s205>to <s205>After the processing of <s206>Proceed to

[0091] <s206>The control unit 2 determines whether the recognition rate obtained by object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition. When it is determined that the recognition result satisfies the predetermined condition, <s207>Proceed. If it is not determined that the recognition result satisfies a predetermined condition, <s208>Proceed to

[0092] <s207>The control unit 2 adds the recognition result to the third list. After that, <s209> Proceed to

[0093] <s208>The control unit 2 adds the recognition result to the first list. Thereafter, <s209>Return.

[0094] <s209>This process is for the second embodiment <s108>corresponds to. When the control unit 2 determines that it should transition from the first stage to the second stage, the control unit 2 transitions from the first stage to the second stage. After that, <s210>Proceed. When the control unit 2 does not determine that it should shift from the first stage to the second stage, <s202>Return.

[0095] <s210>The control unit 2 determines whether the first list is a null value, that is, whether all recognized objects satisfy a predetermined condition. If it is determined that the first list is a null value, the information processing system simply ends the process. If it is not determined that the first list is a null value, <s211>Proceed to

[0096] <s211>This process is of the second embodiment <s109>corresponds to. After the execution of this process, <s212>Proceed to

[0097] <s212>This process is for the second embodiment <s110>It corresponds to. When it is determined that it is the timing for the first imaging unit 1 to perform imaging, the control unit 2 transmits a trigger signal to the first imaging unit 1. After that, <s213>Proceed. When it is determined that it is not the timing for the first imaging unit 1 to perform imaging, <s211>Return to.

[0098] <s213>even <s215>These processes are respectively those of the second embodiment <s111>even <s113>corresponds to. <s213>After the processing of <s214>to <s214>After the processing of <s215>to <s215>After the processing of <s216>Proceed to

[0099] <s216>This process is of the second embodiment <s114>corresponds to. When the control unit 2 determines that the second stage has ended, <s217>Proceed. When the control unit 2 does not determine that the second stage has ended, <s211>Return to.

[0100] <s217>The control unit 2 determines whether the recognition result included in the second list exactly matches the combination of the recognition results included in the first list and the third list. If the combination of the recognition results included in the first list and the third list exactly matches, <s218>Proceed to. If the combination of the recognition results included in the first list and the third list does not exactly match, <s219>Proceed to

[0101] <s218>and <s219>These processes are each of the second embodiment <s116>and <s117>It corresponds to. By executing each process, the information processing system ends the process.

[0102] The above is the description of the information processing method of the information processing system according to the first to third embodiments.

[0103] In the above description, it has been mentioned up to the point where the control unit 2 adds the recognition result of the object to the third list or the fourth list. However, as described in, for example, paragraphs

[0027] and

[0028] , after the control unit 2 adds the recognition result of the object to the third list or the fourth list, the control unit 2 may transmit a control signal to the notification unit 3 based on the third list or the fourth list.

[0104] For example, the notification unit 3 may display the recognition result of the object for which recognition has failed and the candidate objects for recognition in list form based on the fourth list, and suggest that the customer correct them using the input unit provided in the notification unit 3.

[0105] Also, when the recognition result of the object is included in the fourth list, the notification unit 3 may notify the customer or an employee of the retail store, etc. of an error warning.

[0106] Further, the notification unit 3 may display the total amount based on the third list and the fourth list reflecting the above-mentioned correction by the customer, and suggest to the customer to perform accounting processing.

[0107] It should be noted that so far, regarding the information processing system, the information processing apparatus, and the information processing method, it is easy to make corrections to the present disclosure in various figures.

[0108] For example, in the information processing method of the information processing system according to the first to third embodiments, when the control unit 2 detects that an object already added to the first list is taken out of the container 11 based on the moving direction of the object included in the recognition result of the first imaging unit 1 at the first stage, the control unit 2 may delete the recognition result of the object taken out of the container 11 from the first list.

[0109] Also, in the information processing method of the information processing system according to the first to third embodiments, although the collation with the first list is performed after the generation of the second list is completed, for example, when adding a recognition result to the second list, it may be collated whether the added recognition result is included in the first list at the same time.

[0110] Also, in the information processing method of the information processing system according to the first to third embodiments, although the control unit 2 performs processing based on the first to fourth lists, as shown in FIG. 8, for example, without using the second to fourth lists, as attributes of the recognition results included in the first list, "Recognition in the first imaging" represented by two values of Yes / No (hereinafter Y / N), corresponding to the first list, and "Recognition in the second imaging" corresponding to the second list may be newly used (hereinafter parentheses are omitted). In this case, Y may be given to "Recognition in the first imaging" for the recognition result added in the first stage, and Y may be given to "Recognition in the second imaging" for the recognition result added in the second stage. That is, Y is given to both "Recognition in the first imaging" and "Recognition in the second imaging" for the recognition results of the objects recognized in both the first stage and the second stage, and for the recognition results of the objects recognized in the first stage but not recognized in the second stage, Y is given to "Recognition in the first imaging" and N is given to "Recognition in the second imaging", and for the recognition results of the objects not recognized in the first stage but recognized in the second stage, N is given to "Recognition in the first imaging" and Y is given to "Recognition in the second imaging".

[0111] For example, referring to FIG. 8, for the apple with ID number 1, Y is given to both "Recognition in the first imaging" and "Recognition in the second imaging", and the control unit 2 may process the apple as the final confirmed purchase item corresponding to the third list. On the other hand, for the nothing and tangerine with ID numbers 2 and 3, N is given to either "Recognition in the first imaging" or "Recognition in the second imaging", and the control unit 2 may process the nothing and tangerine as the final recognition failure items corresponding to the fourth list.

[0112] Also, in the information processing method of the information processing system according to the third embodiment, the control unit 2 causes the storage unit 23 to temporarily store, in the first stage, those recognition results of objects that do not satisfy the predetermined conditions, and in the second stage, the control unit 2 may prompt the customer via the notification unit 3 to image only those with a low recognition rate by the first imaging unit 1. That is, for those whose recognition rate of object recognition satisfies the predetermined conditions in the first stage, the control unit 2 may perform the process of finally determining the recognition result as it is, and for those whose recognition rate of object recognition does not satisfy the predetermined conditions in the first stage, the control unit 2 may perform the process based on the image captured by the first imaging unit 1 in the second stage.

[0113] Also, in the information processing method of the information processing system according to the first to third embodiments, although the customer is supposed to use the container 11, the customer does not necessarily have to use the container 11. For example, it is also assumed that the customer holds it by their own hand. In that case, the information processing system may, for example, instead of providing the first imaging unit 1 in the container 11, provide it on a product shelf or ceiling in the store of the retail store. For example, the first imaging unit 1 may constantly track the customer and constantly associate the first list with the customer.

[0114] Also, in the information processing method of the information processing system according to the first to third embodiments, the first imaging unit 1 that captures an object in the first stage and the second stage is the same, but it is not necessarily so. For example, the first imaging unit 1 in the first stage may be the one provided in the container 11, and the first imaging unit 1 in the second stage may be the one provided in the cash register terminal, and different ones may be used respectively.

[0115] Furthermore, as described above, although the solution means of the present disclosure has been described as an information processing system, an information processing apparatus, and an information processing method, the present disclosure can also be realized as an aspect including these, and can also be realized as a program substantially corresponding to these and a storage medium recording the program. It should be understood that these are also included in the scope of the present disclosure.

Description of Reference Numerals

[0116] 1 First imaging unit 2 Control unit 3 Notification unit 4 Sensor 11 Container 12 Communication unit of the container 21 Calculation unit 22 Communication unit of the control unit 23 Storage unit 31 Communication unit of the notification unit 41 Communication unit of the sensor < / s104>

Claims

1. A weight sensor provided on a merchandise shelf, and an imaging unit capable of imaging a customer, comprising: the imaging unit includes a first imaging unit provided on the ceiling, when the weight detected by the weight sensor changes, adding or deleting an item to / from a list, associating the customer photographed by the first imaging unit with the list, the imaging unit includes a second imaging unit also provided on a cash register terminal, based on an image captured by the second imaging unit, transitioning to a stage of performing settlement processing for an item added to the list, the cash register terminal includes an input unit for correcting or determining a recognition result and recognition candidates of an object included in the list An information processing system.

2. The first imaging unit performs constant tracking of the customer The information processing system according to claim 1.

3. When the weight detected by the weight sensor changes, associating the item added to the list with the customer The information processing system according to claim 1.

4. The settlement processing is performed in a specific area The information processing system according to claim 1.

5. Further comprising a notification unit, the notification unit displays a total amount based on the list reflecting the correction The information processing system according to claim 1.

6. The notification unit displays purchase candidate items based on the list The information processing system according to claim 5.

7. A first imaging unit provided on a merchandise shelf for imaging the merchandise on the merchandise shelf, and a second imaging unit provided on the ceiling for imaging a customer, comprising: when the first imaging unit images an item moving, adding or deleting the item to / from a list, associating the customer photographed by the second imaging unit with the list, the second imaging unit includes a third imaging unit also provided on a cash register terminal, based on an image captured by the third imaging unit, transitioning to a stage of performing settlement processing for an item added to the list, the cash register terminal includes an input unit for correcting or determining a recognition result and recognition candidates of an object included in the list An information processing system.

8. When the weight detected by a weight sensor provided on a merchandise shelf changes, adding or deleting an item to / from a list, comprising a control unit for associating the customer imaged by a first imaging unit provided on the ceiling and capable of imaging the customer with the list The control unit shifts to a stage of performing settlement processing for the products added to the list based on the image captured by the second imaging unit provided in the cash register terminal. The cash register terminal includes an input unit for correcting or determining the recognition result and recognition candidates of the objects included in the list. An information processing apparatus. **Claim 9** An information processing method by a computer, the information processing method comprising: When a weight sensor provided on a product shelf detects a change in weight, determining that a product has been added or removed from a customer's hand or container and updating a list; Linking the customer photographed by the first imaging unit provided on the ceiling so as to track the customer with the list; Shifting to a stage of performing settlement processing for the products added to the list based on the image captured by the second imaging unit provided in the cash register terminal; Correcting or determining the recognition result and recognition candidates of the objects included in the list by an input to an input unit included in the cash register terminal. An information processing method.

Images