Information processing system, information processing device, and information processing method

Abstract

To track a commodity until the commodity is put into a storage bag or the like from a container or the like including a shopping basket without manually scanning the commodity by a buyer and determine whether the commodity has been appropriately recognized.SOLUTION: An information processing system includes a first imaging unit 1 and a control unit 2 that performs processing related to an image captured by the first imaging unit 1. The control unit 2 performs processing based on an image captured in each of a first stage in which an article is put into a container 11 and a second stage in which the article is taken out of the container 11.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, and an information processing method. [Background technology]

[0002] In the checkout of purchased goods in the retail industry, for example, Patent Document 1 discloses an invention that tracks the product from the time it is removed from a shopping cart to the time it is placed in a storage bag, and determines whether the product has been scanned properly. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-54038 A Summary of the Invention [Problem to be solved by the invention]

[0004] The invention disclosed herein requires the purchaser to manually scan the product to identify the product, and there is room for improvement in this regard.

[0005] Therefore, in consideration of the above, an object of the present disclosure is to track a product from a container, such as a shopping basket, to being placed in a storage bag, etc., without the need for a purchaser to manually scan the product, and to determine whether the product has been properly recognized. [Means for solving the problem]

[0006] In order to achieve the above-mentioned objective, an information processing system that is one aspect of the present disclosure includes a first imaging unit and a control unit that performs processing on an image captured by the first imaging unit, and the control unit performs processing based on images captured at each of a first stage of placing an object into a container and a second stage of removing the object from the container.

[0007] Moreover, an information processing device that is one aspect of the present disclosure includes a control unit that performs processing on an image captured by a first imaging unit, and the control unit performs processing based on images captured at each of a first stage of placing an object into a container and a second stage of removing the object from the container.

[0008] Moreover, an information processing method that is one aspect of the present disclosure includes a first imaging unit capturing images in a first stage of placing an object into a container, a second stage of removing the object from the container, and performing processing based on images captured by the first imaging unit in each of the first stage and the second stage. Effect of the Invention

[0009] According to the present disclosure configured as described above, it becomes possible to track an item from its shopping basket to its placement in a storage bag or the like, without the purchaser having to manually scan the item, and to determine whether the item has been properly recognized. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a functional block diagram showing a schematic configuration of an information processing system according to the present disclosure. [Diagram 2] FIG. 2 is a conceptual diagram relating to the imaging method etc. in the first stage. [Diagram 3] FIG. 3 is a conceptual diagram relating to the imaging method etc. in the second stage. [Figure 4] FIG. 4 is a flowchart illustrating the information processing method of the information processing system according to the first embodiment of the present disclosure. [Diagram 5] FIG. 5 is an image diagram relating to a modified example of the imaging method etc. in the first stage. [Figure 6] FIG. 6 is a flowchart illustrating an information processing method of the information processing system according to the second embodiment of the present disclosure. [Figure 7]FIG. 7 is a flowchart illustrating an information processing method of the information processing system according to the third embodiment of the present disclosure. [Figure 8] FIG. 8 is an image diagram of a modified example of the first list used in the information processing method of the information processing system according to the first to third embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of an information processing system according to the present disclosure will be described with reference to the drawings.

[0012] 1, the information processing system according to the present disclosure includes a first imaging unit 1 and a control unit 2. The information processing system may also include a notification unit 3 and a sensor 4.

[0013] In the following figures, the solid arrows connecting the functional blocks indicate the flow of control signals or data to be communicated. The communications indicated by the arrows may be either wired or wireless, or may be a combination of both.

[0014] The first imaging unit 1 captures images of an object being put into or taken out of a container 11 that temporarily stores the object.

[0015] The first imaging unit 1 includes an imaging optical system and an imaging element to capture an image of an object. The imaging optical system includes optical components such as one or more lenses and an aperture. The lens may be of any type regardless of focal length, and may be, for example, a general lens, a wide-angle lens including a fisheye lens, or a zoom lens with a variable focal length. The imaging optical system forms an image of a subject on a light receiving surface of the imaging element. The imaging element includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The imaging element captures the image of the subject formed on the light receiving surface to generate an image.

[0016] The first imaging unit 1 may capture still images continuously at any frame, or may capture still images discontinuously at a specific timing. The first imaging unit 1 may also capture a moving image.

[0017] The first imaging unit 1 may be provided in any form as long as it can capture images of objects put in and taken out of the container 11 including a shopping basket or shopping cart. For example, it may be provided in the container 11, or in the form of a portable electronic terminal including a smartphone or a business terminal, or it may be provided in a product shelf, ceiling, cash register, or the like in a retail store. In particular, when the first imaging unit 1 is provided in the container 11, it is easy to capture images of objects put in and taken out of the container 11. When the first imaging unit 1 is provided in a portable electronic terminal, the container 11 may include a holder for holding the portable electronic terminal. In the following description, it is assumed that the first imaging unit 1 is provided in the container 11 (also shown in FIG. 1 as being provided in the container 11).

[0018] Moreover, the image captured by the first imaging unit 1 is transmitted to the control unit 2 via a communication unit 12 provided together with the first imaging unit 1.

[0019] The container 11 and the communication units 12 (hereinafter, container communication unit 12), 22 (control unit communication unit 22), 31 (alarm unit communication unit 31), and 41 (sensor communication unit 41), including those provided in the control unit 2, alarm unit 3, and sensor 4 described below, are configured to include either or both of a wired and / or wireless module. The communication units 12, 22, 31, and 41 do not need to have the same configuration and may each have a different configuration. The communication units 12, 22, 31, and 41 transmit and receive control signals or data.

[0020] The control unit 2 performs processing based on the image captured by the first imaging unit 1 and the like.

[0021] The control unit 2 includes an arithmetic unit 21, a communication unit 22 of the control unit, and a storage unit 23 as internal modules. The control unit 2 physically includes a processor including a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or the like, and in addition includes at least one of a semiconductor memory, a magnetic memory, and an optical memory as the storage unit 23. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include a hard disk and a magnetic tape, for example. The optical memory may include a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray (registered trademark) disc (BD), for example.

[0022] The calculation unit 21 performs processing based on the image captured by the first imaging unit 1, such as control related to imaging by the first imaging unit 1, object detection / recognition processing related to the image captured by the first imaging unit 1, data processing related to the recognition result, and control related to notification by the notification unit 3 related to the data processing result. The data processing related to the recognition result includes, for example, status management of the recognition result. The status management includes, for example, adding or deleting the recognition result of an object to or from a list related to the addition or deletion of an object in the container 11.

[0023] The calculation unit 21 may output the recognition result and attribute information of the recognition result including, for example, the name of the recognized object (product name), the price, the moving direction of the object, and the reliability. The moving direction of the object includes, for example, binary information indicating whether the object was put into the container 11 or taken out of the container 11. For example, the reliability is a numerical representation of how reliable the recognition result is, and it can be said that the higher the reliability, the higher the accuracy of the recognition result.

[0024] The communication unit 22 of the control unit includes transmitting and receiving control signals or data, for example, as follows: 1. Receives an image captured by the first imaging unit 1 from the communication unit 12 of the container. 2. Receives a signal acquired by the sensor 4 from the communication unit 41 of the sensor. 3. Transmits and receives various control signals or data to and from the calculation unit 21. 4. Transmits a control signal of the first imaging unit 1 to the communication unit 12 of the container. 5. Transmits a control signal of the alarm unit 3 to the communication unit 31 of the alarm unit.

[0025] The storage unit 23 stores at least one of the following: an image captured by the first imaging unit 1, a recognition result obtained by object detection / recognition processing performed by the calculation unit 21 based on the image captured by the first imaging unit 1, a processing result of processing performed based on the recognition result, a signal obtained from the sensor 4, or a program related to processing in the information processing system, etc.

[0026] The control unit 2 does not necessarily have to be on-premise, and may perform processing in a cloud on a web service. The control unit 2 may also divide functions so that processing is performed on-premise or in the cloud on a module basis. For example, the calculation unit 21 may perform processing in the cloud. The storage unit 23 may perform processing on-premise.

[0027] The notification unit 3 issues a notification based on a control signal or the like from the control unit 2 via the communication unit 31 of the notification unit. The content of the notification may include, for example, the recognition result obtained by the control unit 2 or a result obtained by processing based on the recognition result.

[0028] The notification unit 3 includes, for example, a visual, auditory, or tactile means for making the above-mentioned notification. Therefore, the notification unit 3 may be configured to include, for example, a display, a lamp, a speaker, a motor, etc. The notification unit 3 may also include an input unit for correcting the notified recognition result or for confirming the result. The input unit may include, for example, a mouse, a keyboard, a microphone, and a touch sensor, and the touch sensor may be provided on the display as a touch display.

[0029] The sensor 4 monitors the movement of a purchaser or an object, and transmits the detection data obtained through the monitoring to the control unit 2.

[0030] The sensor 4 includes a second imaging unit including, for example, a weight sensor, an infrared sensor, an imaging optical system, and an imaging element. The control unit 2 may perform the control or the calculation processing related to the detection data, or the sensor 4 may have a separate component equivalent to the control unit 2 and perform the calculation processing.

[0031] When the sensor 4 is provided in the container 11, it may be provided, for example, as a weight sensor at the bottom of the container 11. When the weight detected by the weight sensor changes, it may be determined that an object has been added to or removed from the container 11, and the control unit 2 or the weight sensor may generate a trigger signal for the first imaging unit 1 to capture an image based on the detection result of the weight sensor.

[0032] Furthermore, when the sensor 4 is provided outside the container 11, it may be provided, for example, on a product shelf or ceiling in a retail store. For example, if a weight sensor is provided on a product shelf as the sensor 4, and if the weight detected by the weight sensor changes, the weight sensor may generate a trigger signal as an object has been added or removed from the container 11. Similarly, if a second imaging unit is provided on the ceiling as the sensor 4, and a specific movement (for example, the movement of a purchaser taking an object out of the product shelf, the trajectory of an object moving from the product shelf, etc.) is detected from an image captured by the second imaging unit, the control unit 2 or the second imaging unit may generate a trigger signal based on the image captured by the second imaging unit as an object has been added or removed from the container 11.

[0033] Next, the imaging methods in the first and second stages will be briefly described with reference to FIGS.

[0034] 2 is an image diagram relating to an imaging method etc. in the first stage of putting an object into the container 11. The first stage here includes the meaning of the stage in which the purchaser selects the product (object) to be purchased. In other words, the first stage can be said to be a pre-recognition confirmation stage before the purchaser confirms the product to be purchased (confirms the product recognition result) in a specific area including a checkout counter where the payment is processed.

[0035] According to FIG. 2, when an object A (object) is put into the container 11, the first imaging unit 1 transmits the captured image of the object A to the control unit 2.

[0036] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains the recognition result.

[0037] 3 is an image diagram of an imaging method etc. in the second stage where the object is taken out of the container 11. The second stage here includes the stage where the purchaser performs the checkout process for the product he / she is going to purchase. In other words, the second stage can be said to be a recognition and confirmation stage where the product to be purchased is confirmed in a specific area.

[0038] 3, in order to move from the first stage to the second stage, for example, the first imaging unit 1 captures an image of a marker provided in a specific area and transmits the captured image to the control unit 2. After that, the control unit 2 moves from the first stage to the second stage. In other words, the first stage is the period before the control unit 2 recognizes a marker provided in a specific area, and the second stage is the period after the control unit 2 recognizes the marker.

[0039] In addition, determining that the container has moved from the first stage to the second stage may be based on other means, such as providing a weight sensor in a specific area and detecting a change in weight when the container 11 is placed on the weight sensor, or providing a separate sensor in the container 11 and detecting a specific signal or object, including a marker provided in a specific area.

[0040] When the object A is taken out of the container 11 after the transition to the second stage, the first imaging unit 1 transmits the captured image of the object A to the control unit 2.

[0041] Next, the control unit 2 performs object detection / recognition processing based on the image and obtains the recognition result.

[0042] Thereafter, the control unit 2 performs a final confirmation process of the recognition result based on the images captured in each of the first stage of putting the object into the container 11 and the second stage of removing the object from the container 11.

[0043] At this time, the control unit 2 may transmit a control signal to cause the notification unit 3 to notify the recognition result after the final confirmation process. The control unit 2 may also transmit a control signal to cause the notification unit 3 to notify the recognition result of the object captured by the first imaging unit 1 in the second stage before the final confirmation process of the recognition result. For example, according to Fig. 3, when the respective recognition results acquired based on the images of the object A captured in the first stage and the second stage match "apple", "apple" is displayed on the notification unit 3.

[0044] Next, an information processing method of the information processing system according to the first embodiment of the present disclosure will be described. In the information processing method of the information processing system according to the first embodiment of the present disclosure, the first imaging unit 1 continuously captures images at an arbitrary frame rate, and the control unit 2 performs object detection and object recognition based on the images captured by the first imaging unit 1. Then, first to fourth lists are used as a method for status management of the recognition results, and the control unit 2 performs final confirmation processing of the recognition results based on these lists.

[0045] Then, in the first stage, the control unit 2 adds the object determined to be the object put into the container 11 as a candidate commodity to be purchased to a first list that stores the recognition results of the candidate commodity.

[0046] Furthermore, in the second stage, the control unit 2 adds the item determined to have been taken out of the container 11 as a purchased item to a second list that stores the recognition results of purchased items.

[0047] Then, the control unit 2 verifies whether the first list and the second list are a perfect match, and if they are a perfect match, adds the recognition results of the objects included in the second list to a third list that stores the recognition results of the finally confirmed purchased items as they are, and confirms the object recognition results. If the first list and the second list do not match completely, the control unit 2 adds the recognition results of the matching objects to the third list, and the recognition results of the non-matching objects to a fourth list that stores the recognition results of the final recognition-failed items.

[0048] The control unit 2 may perform settlement processing based on the recognition results stored in the third list, and may also display the recognition results of objects that have failed to be recognized and recognition candidate objects in list form on the notification unit 3 based on the recognition results stored in the fourth list, and suggest that the customer correct them using an input unit provided in the notification unit 3. The processing based on the third list or the fourth list may be the same in the second and subsequent embodiments.

[0049] Hereinafter, a processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG.

[0050] <s001>The first imaging unit 1 starts capturing images continuously at an arbitrary frame rate. <s002>Proceed to.

[0051] <s002>The control unit 2 determines that it is the first stage. Then, <s003>Proceed to.

[0052] <s003>The control unit 2 judges whether to transition from the first stage to the second stage. As a judgment method, for example, the method described in paragraphs

[0037] to

[0039] may be used. When the control unit 2 judges that the transition from the first stage to the second stage should be made, the control unit 2 transitions from the first stage to the second stage. Then, <s008>If the control unit 2 does not determine that the process should move from the first stage to the second stage, <s004>Proceed to.

[0053] <s004>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. If the control unit 2 detects an object, <s005>If the control unit 2 does not detect an object, <s003>Return to.

[0054] The object detection method may be a publicly known object detection method, such as a method using segmentation, semantic segmentation, and / or bounding boxes, or a method using pattern matching or machine learning including deep learning in combination with object recognition, which will be described later.

[0055] <s005>The control unit 2 performs object recognition based on the object detection result. <s006>Proceed to.

[0056] As the object recognition method, a publicly known object recognition method may be used, for example, pattern matching or machine learning including deep learning.

[0057] <s006>The control unit 2 judges whether the recognition result should be added to the first list as a candidate product for purchase. Criteria for judging whether to add the recognition result include, for example, whether the reliability of the recognition result is equal to or greater than a predetermined threshold, and whether the movement of the object is judged to be an object being dropped into the container 11. When the control unit 2 judges that the recognition result should be added to the first list as a candidate product for purchase, <s007>If the control unit 2 does not determine that the recognition result should be added to the first list as a purchase candidate commodity, the control unit 2 discards the recognition result and proceeds to <s003>Return to.

[0058] <s007>The control unit 2 adds the recognition result to the first list. Then, <s003>Return to.

[0059] <s008>The control unit 2 determines whether an object has been detected based on the image captured by the first imaging unit 1. If the control unit 2 detects an object, <s009>If the control unit 2 does not detect an object, <s011>Proceed to.

[0060] <s009>The control unit 2 performs object recognition based on the object detection result. <s010>Proceed to.

[0061] <s010>The control unit 2 adds the recognition result to the second list. <s011>Proceed to.

[0062] <s011>The control unit 2 judges whether the second stage should be ended. For example, a method of judgment is when no object is detected for a certain period of time. When the control unit 2 judges that the second stage should be ended, <s012>If the control unit 2 does not determine that the second stage is to be ended, <s008>Return to.

[0063] <s012>The control unit 2 judges whether the recognition results included in the first list and the second list completely match. If the recognition results included in the first list and the second list completely match, <s013>If the recognition results in the first list and the second list do not match exactly, <s014>Proceed to.

[0064] <s013>The control unit 2 copies the recognition result contained in the second list and adds it to the third list, after which the information processing system ends the process.

[0065] <s014>For the recognition results of objects that match between the first list and the second list, the control unit 2 copies the recognition results included in the second list and adds them to the third list. For the recognition results of objects that do not match between the first list and the second list, the control unit 2 adds the recognition results of the objects that do not match to the fourth list. After that, the control unit 2 performs processing based on the third list and the fourth list, and then ends the overall processing.

[0066] Next, an information processing method of an information processing system according to a second embodiment of the present disclosure will be described. In the information processing method of an information processing system according to a first embodiment of the present disclosure, images are captured continuously, but there is a concern that the amount of data handled by the control unit 2 will increase. Therefore, in the information processing method of an information processing system according to a second embodiment of the present disclosure, the image capturing timing of the first imaging unit 1 is estimated by predicting the timing at which the purchaser will put an object into the container 11. Then, the first imaging unit 1 captures an image based on the estimated image capturing timing. With this method, it is no longer necessary for the first imaging unit 1 to capture images continuously, and it is expected that the amount of data handled by the control unit 2 will be reduced.

[0067] Fig. 5 is an image diagram of an imaging method etc. in the first stage in the second embodiment. In Fig. 5, a sensor 4 is added as a new configuration element compared to Fig. 2 and the first embodiment. Note that the sensor 4 here is provided on the ceiling inside the retail store, but in reality it is not limited to this pattern.

[0068] The sensor 4 monitors the container 11 and / or the customer holding the container 11 , and transmits the acquired detection data to the control unit 2 .

[0069] The control unit 2 determines whether the object A is to be put into the container 11 based on the detection data acquired by the sensor 4. As a method of determination, for example, the method described in paragraphs

[0029] to

[0032] may be used.

[0070] When the control unit 2 determines that the object A is to be dropped into the container 11 based on the detection data acquired by the sensor 4, it transmits a trigger signal to the first imaging unit 1 to perform imaging.

[0071] The first imaging unit 1 performs imaging when receiving a trigger signal from the control unit 2. After that, the first imaging unit 1 transmits the captured image to the control unit 2.

[0072] Although FIG. 5 illustrates an imaging method in the first stage, a similar imaging method may be used in the second stage.

[0073] Hereinafter, a processing flow of the information processing system according to the second embodiment of the present disclosure will be described with reference to FIG.

[0074] <s101>This process is the same as that of the first embodiment. <s002>After this process is performed, <s102>Proceed to.

[0075] <s102>The control unit 2 predicts the timing when the purchaser will put the object into the container 11 based on the detection data acquired by the sensor 4. Then, <s103>Proceed to.

[0076] <s103>The control unit 2 judges whether it is time for the first imaging unit 1 to capture an image based on the result of the prediction of the timing when the purchaser will put the object into the container 11. If it is judged that it is time for the first imaging unit 1 to capture an image, the control unit 2 transmits a trigger signal to the first imaging unit 1. Then, <s104>If it is determined that it is not time for the first imaging unit 1 to capture an image, <s102>Return to.

[0077] <s104>The first imaging unit 1 captures an image of the object based on a trigger signal from the control unit 2. Then, <s105>Proceed to.

[0078] <s105>This process is the same as that of the first embodiment. <s005>After this process is performed, <s106>Proceed to.

[0079] <s106>This process is the same as that of the first embodiment. <s006>When the control unit 2 determines that the recognition result should be added to the first list as a purchase candidate product, <s107>If the control unit 2 does not determine that the recognition result should be added to the first list as a purchase candidate commodity, the control unit 2 discards the recognition result and proceeds to <s102>Return to.

[0080] <s107>This process is the same as that of the first embodiment. <s007>After this process is performed, <s108>Proceed to.

[0081] <s108>This process is the same as that of the first embodiment. <s003>When the control unit 2 determines that the first stage should be changed to the second stage, the control unit 2 changes the first stage to the second stage. Then, <s109>If the control unit 2 does not determine that the process should move from the first stage to the second stage, <s102>Return to.

[0082] <s109>The control unit 2 predicts the timing when the purchaser will remove the object from the container 11 based on the detection data acquired by the sensor 4. Then, <s110>Proceed to.

[0083] <s110>This process is <s103>When the control unit 2 determines that it is time for the first imaging unit 1 to capture an image, the control unit 2 transmits a trigger signal to the first imaging unit 1. Then, <s111>If it is determined that it is not time for the first imaging unit 1 to capture an image, <s109>Return to.

[0084] <s111>and even <s113>These processes are <s104> 、 <s105>and <s010>Corresponds to. <s111>After processing <s112>To, <s112>After processing <s113>To, <s113>After processing <s114>Proceed to.

[0085] <s114>This process is the same as that of the first embodiment. <s011>When the control unit 2 determines that the second stage is to be ended, <s115>If the control unit 2 does not determine that the second stage is to be ended, <s109>Return to.

[0086] <s115>This process is the same as that of the first embodiment. <s012>If the recognition results in the first list and the second list are an exact match, <s116>If the recognition results in the first list and the second list do not match exactly, <s117>Proceed to.

[0087] <s116>and <s117>This process is <s013>and <s014>The control unit 2 executes each process and processes based on the third list and the fourth list, and then ends the overall process.

[0088] Next, an information processing method of an information processing system according to a third embodiment of the present disclosure will be described. In the information processing methods of the information processing system according to the first and second embodiments of the present disclosure, a determination process is always performed to determine whether the recognition results based on the images captured in the first stage and the second stage match, but it is also assumed that two images are not necessarily captured and processing is not necessarily required based on the captured images, for example, when a high recognition rate is obtained in the recognition result in the first stage. Therefore, in the information processing method of the information processing system according to the third embodiment of the present disclosure, if the recognition rate in the object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition (for example, an arbitrary threshold such as a recognition rate of 90% or more), the object is directly added to the third list. If the control unit 2 confirms that all objects have been added to the third list, object recognition based on the image captured in the second stage is not performed. It is expected that this method will reduce the amount of data processing by the control unit 2.

[0089] Hereinafter, a processing flow of the information processing system according to the first embodiment of the present disclosure will be described with reference to FIG.

[0090] <s201>and even <s205>These processes are the same as those in the second embodiment. <s101>and even <s105>Corresponds to. <s201>After processing <s202>To, <s202>After processing <s203>To, <s203>After processing <s204>To, <s204>After processing <s205>To, <s205>After processing <s206>Proceed to.

[0091] <s206>The control unit 2 judges whether the recognition rate obtained by the object recognition based on the image captured by the first imaging unit 1 in the first stage satisfies a predetermined condition. If it is judged that the recognition result satisfies the predetermined condition, <s207>If the recognition result is not judged to satisfy the predetermined conditions, <s208>Proceed to.

[0092] <s207>The control unit 2 adds the recognition result to the third list. <s209>Proceed to.

[0093] <s208>The control unit 2 adds the recognition result to the first list. Then, <s209>Return to.

[0094] <s209>This process is the same as that of the second embodiment. <s108>When the control unit 2 determines that the first stage should be changed to the second stage, the control unit 2 changes the first stage to the second stage. Then, <s210>If the control unit 2 does not determine that the process should move from the first stage to the second stage, <s202>Return to.

[0095] <s210>The control unit 2 judges whether the first list is null, i.e., whether all recognized objects satisfy a predetermined condition. If it is judged that the first list is null, the information processing system ends the process. If it is not judged that the first list is null, <s211>Proceed to.

[0096] <s211>This process is the same as that of the second embodiment. <s109>After this process is performed, <s212>Proceed to.

[0097] <s212>This process is the same as that of the second embodiment. <s110>When the control unit 2 determines that it is time for the first imaging unit 1 to capture an image, the control unit 2 transmits a trigger signal to the first imaging unit 1. Then, <s213>If it is determined that it is not time for the first imaging unit 1 to capture an image, <s211>Return to.

[0098] <s213>and even <s215>These processes are the same as those in the second embodiment. <s111>and even <s113>Corresponds to. <s213>After processing <s214>To, <s214>After processing <s215>To, <s215>After processing <s216>Proceed to.

[0099] <s216>This process is the same as that of the second embodiment. <s114>When the control unit 2 determines that the second stage is to be ended, <s217>If the control unit 2 does not determine that the second stage is to be ended, <s211>Return to.

[0100] <s217>The control unit 2 judges whether the recognition result included in the second list is a perfect match with the combination of the recognition results included in the first list and the third list. If the combination of the recognition results included in the first list and the third list is a perfect match, <s218>If the combination of the recognition results in the first list and the third list does not result in an exact match, <s219>Proceed to.

[0101] <s218>and <s219>These processes are the same as those in the second embodiment. <s116>and <s117>After each process is executed, the information processing system ends the process.

[0102] This concludes the description of the information processing methods of the information processing systems according to the first to third embodiments.

[0103] Note that the explanation so far has mentioned the control unit 2 adding the object recognition result to the third list or the fourth list, but for example, as described in paragraphs

[0027] and

[0028] , after the control unit 2 adds the object recognition result to the third list or the fourth list, the control unit 2 may transmit a control signal to the alarm unit 3 based on the third list or the fourth list.

[0104] For example, the notification unit 3 may display the recognition results of objects that failed to be recognized and candidate objects for recognition in list format based on the fourth list, and suggest that the customer make corrections using an input unit provided in the notification unit 3.

[0105] Furthermore, when the fourth list contains the object recognition result, the notification unit 3 may notify the customer, a retail store employee, or the like of an error warning.

[0106] Furthermore, the notification unit 3 may display the total amount based on the third list and the fourth list that reflects the above-mentioned correction by the customer, and may suggest to the customer to carry out the accounting process.

[0107] It should be noted that it is easy to make various modifications to the information processing system, the information processing device, and the information processing method disclosed herein.

[0108] For example, in the information processing method of the information processing system according to the first to third embodiments, if the control unit 2 detects in the first stage based on the movement direction of the object included in the recognition result of the first imaging unit 1, that an object already added to the first list is taken out of the container 11, the control unit 2 may delete the recognition result of the object taken out of the container 11 from the first list.

[0109] Furthermore, in the information processing methods of the information processing systems according to the first to third embodiments, after the generation of the second list is completed, the second list is compared with the first list. However, for example, it is also possible to add a recognition result to the second list and at the same time compare whether the added recognition result is included in the first list.

[0110] In the information processing method of the information processing system according to the first to third embodiments, the control unit 2 performs processing based on the first to fourth lists, but instead of using the second to fourth lists, for example, "recognition in the first image capture" corresponding to the first list and "recognition in the second image capture" corresponding to the second list, each of which is expressed by two values, Yes / No (hereinafter Y / N), may be newly used as attributes of the recognition results included in the first list (quotation marks are omitted below). In this case, Y may be added to the recognition results added in the first stage for recognition in the first image capture, and Y may be added to the recognition results added in the second stage for recognition in the second image capture. In other words, the recognition results for objects that were recognized in both the first and second stages are marked with a Y for recognition in the first image and a Y for recognition in the second image, the recognition results for objects that were recognized in the first stage but not in the second stage are marked with a Y for recognition in the first image and an N for recognition in the second image, and the recognition results for objects that were not recognized in the first stage but recognized in the second stage are marked with an N for recognition in the first image and a Y for recognition in the second image.

[0111] 8, for example, an apple with ID number 1 is assigned a Y in both the first and second images, and the control unit 2 may process the apple as a final confirmed purchased product corresponding to the third list. On the other hand, a pear and an orange with ID numbers 2 and 3 are assigned an N in either the first image or the second image, and the control unit 2 may process the pear and orange as final recognition failed products corresponding to the fourth list.

[0112] Furthermore, in the information processing method of the information processing system according to the third embodiment, the control unit 2 may prompt the customer via the notification unit 3 to temporarily store in the storage unit 23 object recognition results in the first stage whose recognition rate does not satisfy a predetermined condition, and to cause the control unit 2 to capture only those objects with low recognition rates in the second stage by the first imaging unit 1. That is, the control unit 2 may execute a process of finalizing the recognition results as they are for those objects whose recognition rate of the object recognition in the first stage satisfies a predetermined condition, and may execute a process based on the image captured by the first imaging unit 1 in the second stage for those objects whose recognition rate of the object recognition in the first stage does not satisfy the predetermined condition.

[0113] In addition, in the information processing method of the information processing system according to the first to third embodiments, the customer uses the container 11, but it is assumed that the customer does not necessarily use the container 11 and may hold it in his / her own hand. In that case, the information processing system may provide the first imaging unit 1 on a product shelf or ceiling in a retail store instead of on the container 11, and the first imaging unit 1 may constantly track the customer and constantly link the first list with the customer.

[0114] Furthermore, in the information processing methods of the information processing systems according to the first to third embodiments, the first imaging unit 1 that captures images of the object in the first and second stages is the same, but this is not necessarily the case. For example, the first imaging unit 1 in the first stage may be provided in the container 11, and the first imaging unit 1 in the second stage may be provided in the cash register terminal.

[0115] Furthermore, as mentioned above, the solutions of the present disclosure have been described as an information processing system, an information processing device, and an information processing method, but the present disclosure can also be realized in forms that include these, and can also be realized as a program that substantially corresponds to these, or a storage medium having a program recorded thereon, and it should be understood that these are also included within the scope of the present disclosure. [Explanation of symbols]

[0116] 1 First imaging unit 2. Control section 3. Notification Department 4 Sensors 11 Container 12 Container communication unit 21 Arithmetic section 22 Communication section of control section 23 Preservation Department 31 Communications Division of the Information Department 41 Sensor communication section < / s104>

Claims

1. an imaging unit that generates a captured image including an image of an object; a control unit that recognizes the object included in the captured image, The control unit recognizes a marker provided in a specific area, and then recognizes an object included in the captured image, and notifies a notification unit of the recognition result. Information processing system.

2. When the control unit recognizes an object, the control unit adds the recognition result to a list. The information processing system according to claim 1 .

3. An imaging unit that captures an image; A control unit that recognizes an object captured by the imaging unit, The control unit causes a display to display a recognition result of the object recognized after a marker included in the captured image is recognized after the imaging unit starts imaging. Information processing system.

4. An imaging unit that captures an image; A control unit that recognizes an object captured by the imaging unit, The control unit causes a display to display a recognition result of the object recognized before a marker included in a captured image is recognized after the imaging unit starts imaging the image, after the marker is recognized. Information processing system.

5. An imaging unit that captures an image; A control unit that recognizes an object captured by the imaging unit, the control unit, after the start of imaging by the imaging unit, determines a recognition result of the object recognized before a marker included in the captured image is recognized as a recognition result after the marker is recognized, and displays the result on a display. Information processing system.

6. The recognition result includes at least an object name of the object.

6. An information processing system according to claim 3.

Images