Information processing program, information processing method, and information processing device

Abstract

To solve the problem that there may be an article hidden behind a side face of a shopping basket or other articles, and an error may be caused in this case when specifying the number of articles in the shopping basket from an image of a monitor camera.SOLUTION: An information processing program causes a computer to execute processing to: acquire an image obtained by photographing a basket containing articles and then analyzing the acquired image to extract a region of the basket included in the image; analyze the acquired image to extract a region of one or a plurality of articles included in the extracted region of the basket; estimate the article weight of the articles contained in the basket based upon the position of the contour of the region of the articles relative to the position of the extracted region of the basket; and specify the number of the articles contained in the basket based upon the specified article weight.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] In retail stores, in order to eliminate the congestion at the cash register, a system is being introduced in which customers themselves scan and register products and perform accounting. Furthermore, in recent years, a system has begun to be introduced in which customers use a terminal lent out in the store of a retail store or an application installed on a terminal owned by the customer to perform product scanning. In such a system in which customers themselves perform product scanning, it is necessary to detect missed scanning of products in order to detect illegal acts such as shoplifting and mistakes.

[0003] On the other hand, as a system for detecting illegal acts of customers in retail stores, for example, a system has been developed that uses surveillance cameras in the store to detect suspicious behavior of customers and illegal acts such as shoplifting. There is also a technique for detecting illegal acts and mistakes by identifying the number of products in the shopping basket from the video of the surveillance camera and comparing the identified number of products with the product information for the products purchased by the customer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, there may be items hidden from view, such as on the side of the shopping cart or hidden by other items. In such cases, errors may occur when determining the number of items in the shopping cart from the surveillance camera footage.

[0006] One aspect of this is the aim to provide an information processing program, information processing method, and information processing device that can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves. [Means for solving the problem]

[0007] In one embodiment, the information processing program causes the computer to perform the following processes: acquire an image of a basket containing goods, analyze the acquired image to extract the basket region contained in the image, analyze the acquired image to extract the regions of one or more goods contained in the extracted basket region, estimate the quantity of goods contained in the basket based on the position of the contour of the goods region relative to the position of the extracted basket region, and determine the number of goods contained in the basket based on the determined quantity. [Effects of the Invention]

[0008] One aspect of this is that in systems where customers scan items themselves, the number of items in the shopping cart can be determined more accurately. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of purchasing products using self-scanning. [Figure 2] Figure 2 shows an example of product detection. [Figure 3] Figure 3 shows an example of a problem in product detection. [Figure 4] Figure 4 shows an example of the configuration of the information processing system according to Example 1. [Figure 5] Figure 5 shows an example of the configuration of the information processing device 10 according to Example 1. [Figure 6] FIG. 6 is a diagram showing an example of data stored in the product information 32 according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of rotational correction of the basket area according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing another example of rotational correction of the basket area according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of estimation of the amount of products in the basket according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing another example of estimation of the amount of products in the basket according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of the difference in the appearance of the basket and products depending on the position of the basket according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing an example of correction of the amount of products depending on the position of the basket according to the first embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of the product number estimation process according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing another example of the flow of the product number estimation process according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing yet another example of the flow of the product number estimation process according to the first embodiment. [Figure 16] FIG. 16 is a diagram for explaining an example of the hardware configuration of the information processing apparatus 10. [Figure 17] FIG. 1十七 is a diagram for explaining an example of the hardware configuration of the information processing terminal 1000. [Figure 18] FIG. 18 is a diagram for explaining an example of the hardware configuration of the self-checkout terminal 400.

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, examples of the information processing program, information processing method, and information processing apparatus according to the present embodiment will be described in detail based on the drawings. Note that the present embodiment is not limited by these examples. Also, the respective examples can be appropriately combined within a non - contradictory range.

[0011] First, using FIG. 1, a method in which a customer scans and registers a product by themselves (hereinafter sometimes referred to as "self - scan") and purchases the product will be described. FIG. 1 is a diagram showing an example of product purchase by self - scan.

[0012] As shown in FIG. 1, first, the customer selects the store visited via an application displayed on the user terminal 100 for scanning and registering products. The application is pre - installed on the user terminal 100. Also, the user terminal 100 may be a mobile terminal such as a smartphone carried by the customer, or may be a dedicated terminal lent to the customer in the store. In the case of a dedicated terminal lent in the store, it may be pre - selected even if the customer does not select the store.

[0013] Next, the customer picks up the product to be purchased and reads, for example, the barcode or two - dimensional code for each product attached to the product or the product shelf using the user terminal 100 (hereinafter sometimes referred to as "product scan"). Thereby, the product to be purchased is registered in the application.

[0014] Next, the customer scans the payment code displayed on the display unit of the self - checkout terminal 400 or the like using the user terminal 100. Then, by paying the amount displayed on the settlement screen of the self - checkout terminal 400, the purchase of the product is completed.

[0015] The above explains product purchase using self-scanning with the help of Figure 1. However, with self-scanning, for example, a customer could place items in their basket without scanning them and avoid paying for them by not using the self-checkout terminal 400. Alternatively, a customer could scan only some of the items they purchased and pay for only those items at the self-checkout terminal 400, thus avoiding payment for some items. In particular, fraudulent activity when there are only a few items is easy for store staff to detect, but it is difficult for store staff to detect when there are many items and some items are not scanned.

[0016] Therefore, there is also technology that identifies the number of items in a shopping cart from surveillance camera footage and compares the identified number of items with product information for the customer's purchases to detect fraudulent activity or errors.

[0017] Figure 2 shows an example of product detection. As shown in Figure 2, for example, shopping carts and products can be detected from surveillance camera footage using existing object detection algorithms. Here, existing object detection algorithms refer to deep learning-based object detection algorithms such as Faster R-CNN (Regions with Convolutional Neural Network). Alternatively, object detection algorithms such as YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), and DETR (DEtection Transformer) may also be used. By counting the number of products detected in the shopping cart, the total number of products in the shopping cart can be determined.

[0018] However, in the example shown in Figure 2, the shopping basket is filmed from above. Depending on the position of the surveillance camera and the shopping basket, the footage may be taken from the side or from a diagonal angle above the basket. In such cases, there may be items hidden from view by the sides of the basket or other items. In this case, errors may occur when determining the number of items in the shopping basket from the surveillance camera footage.

[0019] Figure 3 illustrates an example of a problem in product detection. In the example on the left of Figure 3, there are 7 items in the shopping basket, but some items are hidden on the side of the basket. For example, in a video taken from the side of the shopping basket, the number of items may be identified as fewer than 7. Also, in the example on the right of Figure 3, there are 4 items in the shopping basket, but some items are hidden by other items. For example, the number of items may be identified as fewer than 4. Therefore, one of the objectives of this embodiment is to estimate the amount of products in the shopping basket by taking into account products hidden on the side of the basket and other products, and to more accurately identify the number of products in the shopping basket. [Examples]

[0020] First, the information processing system for implementing this embodiment will be described. Figure 4 is a diagram showing an example configuration of the information processing system according to Embodiment 1. As shown in Figure 4, the information processing system 1 is a system in which an information processing device 10 and user terminals 100-1 to 100-N (where N is any integer; hereinafter collectively referred to as "user terminal 100") are connected to each other via a network 50 so that they can communicate with one another.

[0021] Furthermore, the information processing device 10 is connected to camera devices 200-1 and 200-2 (hereinafter collectively referred to as "camera devices 200") and the store clerk terminal 300 via the network 50, enabling mutual communication. In addition, the information processing device 10 is connected to self-checkout terminals 400-1 to 400-L (M is any integer; hereinafter collectively referred to as "self-checkout terminals 400") via the network 50, enabling mutual communication.

[0022] Network 50 can employ various communication networks, whether wired or wireless, such as an intranet used within a retail store. Furthermore, Network 50 is not a single network; for example, an intranet and the internet may be configured via network devices such as gateways or other devices (not shown). Note that the retail store is not limited to indoor spaces but may include outdoor areas within the premises.

[0023] The information processing device 10 is, for example, an information processing device such as a desktop PC (Personal Computer), a notebook PC, or a server computer, which is installed in a retail store and used by store staff or managers.

[0024] The information processing device 10 receives, for example, multiple captured images from the camera device 200. More precisely, multiple captured images are a series of frames of video footage captured by the camera device 200.

[0025] Furthermore, the information processing device 10 uses, for example, existing object detection technology to detect customers staying in the store (hereinafter sometimes simply referred to as "people"), shopping baskets (hereinafter sometimes simply referred to as "baskets"), and products from the captured image. Object detection of products and other objects from the captured image may be performed by detecting a predetermined area on the captured image for each object, such as a bounding box which is a rectangular area surrounding the product or other object. The information processing device 10 then calculates the number of products contained in the basket based on the baskets and products detected from the captured image.

[0026] Furthermore, the information processing device 10 identifies the self-checkout terminal 400 on which the person will make a purchase, based on, for example, the person's location information and the location information of each of the multiple self-checkout terminals 400-1 to 400-L. The information processing device 10 then determines whether there are any unregistered items that have not been registered as items to be purchased with the self-checkout terminal 400, based on the difference between the number of items placed in the basket and the number of items to be purchased registered with the self-checkout terminal 400. If there are unregistered items, the information processing device 10 notifies the store clerk terminal 300 of an alert.

[0027] In Figure 4, the information processing device 10 is shown as a single computer, but it may also be a distributed computing system composed of multiple computers. Furthermore, the information processing device 10 may be a cloud computing device managed by a service provider offering cloud computing services.

[0028] The user terminal 100 is an information processing terminal used by customers to register purchased items by scanning barcodes or other information of products, for example, in order to purchase goods. The user terminal 100 may be a mobile device such as a smartphone or tablet PC owned by the customer, or it may be a dedicated terminal provided within the store. The user terminal 100 comes pre-installed with an application for scanning and registering products.

[0029] The customer uses the user terminal 100 to scan, for example, the barcode attached to each product or product shelf, and registers the product to be purchased in the user terminal 100. Then, when paying for the product, the customer has the two-dimensional code displayed on the user terminal 100 read by, for example, a self-checkout terminal 400 or a code reader (not shown) installed at the entrance of the self-checkout area where the self-checkout terminal 400 is located. As a result, the product information of the product to be purchased is registered in the self-checkout terminal 400 or the information processing device 10.

[0030] The camera device 200 is, for example, a surveillance camera installed inside or on the premises of a retail store. The camera device 200 may include, for example, a camera device 200-1 for photographing shopping baskets from above and a camera device 200-2 for photographing them from the side. In particular, as shown in Figure 4, when a shopping cart is used that can hold shopping baskets in two tiers, the shopping basket placed on the lower tier is difficult to photograph with the camera device 200-1 that photographs from above, so it is photographed by the camera device 200-2 that photographs from the side. The camera device 200 may also be installed at the entrance of the self-checkout area where the self-checkout terminals 400 are installed, or at the shopping basket storage area for each self-checkout terminal 400. The video captured by the camera device 200 is transmitted to the information processing device 10.

[0031] The employee terminal 300 may be a mobile device such as a smartphone or tablet PC held by a retail store employee, or it may be an information processing device such as a desktop PC or notebook PC installed in a designated location within the store. The employee terminal 300 receives an alert from the information processing device 10 when fraudulent customer behavior or failure to scan products due to errors is detected by the information processing device 10. There may be multiple employee terminals 300 for each employee in the store, but the terminals that receive the alert may be limited to, for example, a terminal held by an employee responsible for security near the self-checkout area.

[0032] The self-checkout terminal 400 is a self-checkout terminal that allows customers to pay for their purchases themselves. The self-checkout terminal 400 receives information about purchased items registered via the user terminal 100. The self-checkout terminal 400 then accepts payment from customers for purchased items using cash, credit cards, electronic money, etc.

[0033] [Functional configuration of the information processing device 10] Next, the functional configuration of the information processing device 10 will be described. Figure 5 is a diagram showing an example configuration of the information processing device 10 according to Embodiment 1. As shown in Figure 5, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0034] The communication unit 20 is a processing unit that controls communication with other information processing devices, such as the user terminal 100 and the camera device 200, and is a communication interface such as a USB (Universal Serial Bus) interface or a network interface card.

[0035] The storage unit 30 has the function of storing various data and programs executed by the control unit 40, and is implemented by a storage device such as memory or a hard disk. The storage unit 30 stores image DB 31 and product information 32, etc.

[0036] Image DB31 stores multiple captured images, which are a series of frames captured by the camera device 200. The captured images stored in Image DB31 are images captured by the camera device 200 and transmitted to the information processing device 10. Image DB31 may also store an identifier to uniquely identify the camera device 200 that captured the images, as well as the date and time of capture. In addition, Image DB31 can store location information of objects in the captured images, such as identified persons, shopping baskets, and products.

[0037] Product information 32 stores information about the product to be purchased that has been registered by self-scanning. Figure 6 shows an example of the data stored in product information 32 according to Embodiment 1. As shown in Figure 6, product information 32 stores the "product name" indicating the name of the product registered by self-scanning, the "quantity" indicating the number of each product, and the "amount" indicating the price of the product, in association with each other. In addition to the "product name," or instead of the "product name," a "product type" indicating the type of product registered by self-scanning may also be stored. Furthermore, product information 32 can store identifiers of images in which the registered product has been captured, and link them to the captured images stored in the image DB 31.

[0038] The above information stored in the memory unit 30 is merely an example, and the memory unit 30 can store various other types of information besides the above.

[0039] The control unit 40 is a processing unit that oversees the entire information processing device 10, and is, for example, a processor. The control unit 40 includes a detection unit 41, a specification unit 42, a calculation unit 43, and a notification unit 44. Each processing unit is an example of an electronic circuit in the processor or an example of a process executed by the processor.

[0040] The detection unit 41 extracts the region of the basket containing the goods from an image, for example, by analyzing an image of the basket captured by the camera device 200. The detection unit 41 also extracts the regions of one or more goods contained within the extracted basket region by analyzing the image. The extraction of basket and product regions from the image may be performed using existing object detection methods, such as Faster R-CNN or YOLO.

[0041] Furthermore, the detection unit 41 may, for example, perform rotation correction on the image of the basket when extracting the basket region, correcting it so that the sides of the basket are horizontal in the image. Figure 7 shows an example of rotation correction of the basket region according to Embodiment 1. As shown in Figure 7, the detection unit 41 may rotate the entire image by a predetermined angle before extracting the basket region.

[0042] Figure 8 shows another example of rotation correction of the cage area according to Embodiment 1. In the example in Figure 8, the detection unit 41 rotates the image at multiple angles, for example, and adopts the one with the smallest area of ​​the detected rectangular region, and extracts the adopted rectangular region as the cage area.

[0043] The identification unit 42 identifies, for example, the position of the contour of the product area relative to the basket area extracted by the detection unit 41. Figure 9 shows an example of estimating the amount of products in the basket according to Embodiment 1. As shown in Figure 9, the product area extracted by the detection unit 41 may be the visible area, excluding the part hidden by the basket.

[0044] Furthermore, as shown on the left side of Figure 9, the identification unit 42 identifies the position of the contour of the product area 70 relative to the basket area 60. For example, as shown on the left side of Figure 9, the position of the upper edge of the contour of the product area 70 is identified, with the upper edge of the basket area 60 set to 1 and the lower edge to 0. The area 80 without products is identified as the lower edge of the basket area 60. The identification unit 42 then estimates the amount of products in the basket as the average value of the upper edges of the product area 70 and the area 80 without products, which in the example on the left side of Figure 9 is 0.2.

[0045] As shown on the right side of Figure 9, even if there are multiple items in the basket, the identification unit 42 estimates the amount of items in the basket as the average value of the top edges of the areas for each item and the areas without items, which is 0.8 in the example on the right side of Figure 9, similar to the left side of Figure 9. Furthermore, as shown in Figure 10, even if the extraction of the item area 70 is performed by instance segmentation instead of by bounding box, the identification unit 42 can still estimate the amount of items in the basket based on the position of the contour of the item area 70 relative to the basket area 60.

[0046] Figure 10 shows another example of estimating the quantity of items in the shopping cart according to Example 1. For example, in the example in Figure 10, as shown in the center of Figure 10, the specific unit 42 uses instance segmentation to mask and extract the product area 70. Even in this case, as shown on the right side of Figure 10, for example, the average value of the top edges of the product area 70 and the area 80 without products can be estimated as the quantity of items in the shopping cart.

[0047] Furthermore, when the specific unit 42 estimates the quantity of goods, it may correct and estimate the quantity of goods based on how the basket and goods appear in the camera image. Figure 11 is a diagram showing an example of the difference in how the basket and goods appear depending on the position of the basket according to Embodiment 1. The three camera images on the left side of Figure 11 were taken by the same camera device 200-2 of the same basket placed on the lower level of a shopping cart. As shown in Figure 11, the appearance of the basket and the goods contained in the basket differs in these three camera images because the position of the shopping cart is different. As shown on the right side of Figure 11, even when the basket region is extracted from each of the three camera images, the appearance of the basket and the goods contained in the basket also differs. Therefore, the estimation of the quantity of goods explained using Figures 9 and 10 may also be subject to errors depending on how the basket and the goods contained in the basket appear. For this reason, in this embodiment, the quantity of goods is corrected and estimated based on how the basket and goods appear in the camera image.

[0048] Figure 12 shows an example of correcting the amount of goods based on the position of the basket according to Example 1. The upper part of Figure 12 shows baskets and goods for which the amount of goods estimated to be 0.65 and 0.5, respectively, was determined using the estimation of the amount of goods explained using Figures 9 and 10. However, the baskets and goods estimated to be 0.65 and 0.5 are actually the same basket and goods, only appearing differently in the camera image. Therefore, the identification unit 42 corrects the amount of goods based on the position of the basket, for example, using the following equation (1).

[0049]

number

[0050] By using equation (1), as shown in the lower part of Figure 12, the position of the basket where the product quantity is 0, i.e., the origin, is corrected, and consequently, the product quantity is also corrected. Therefore, the estimated product quantities of 0.65 and 0.5 are corrected to product quantities of 0.3 and 0.3, respectively. The product quantity estimated by the product quantity estimation method explained using Figures 9 to 12 is converted into the number of products by the calculation unit 43, as will be described later.

[0051] Returning to the explanation of Figure 5, the identification unit 42 identifies the first self-checkout terminal 400 used by the first person holding a shopping basket from a second image captured by the camera device 200 of an area including the self-checkout terminal 400. The identification unit 42 also receives product information to be purchased from the first self-checkout terminal 400 and identifies a third number of products to be purchased based on this product information. The product information 32 received from the first self-checkout terminal 400 may be information generated by scanning and registering a barcode or two-dimensional code attached to a product by the user terminal 100, which is an information processing terminal used by the first person.

[0052] Then, the identification unit 42 determines whether or not there are any products that are not registered as products to be purchased in the self-checkout terminal 400, based on the number of products calculated based on the identified position of the contour of the product area, as explained using Figures 9 to 12, and a third number.

[0053] Furthermore, the identification unit 42 identifies, for example, the personal belongings of the first person who owns the basket, which are included in the extracted basket area. For such identification of personal belongings, a machine learning model may be used, for example, which has been pre-trained using existing machine learning techniques to detect personal belongings such as reusable bags and purses from captured images. Objects identified as personal belongings are not detected as products and are excluded from product area extraction and calculation of product quantity and number of products.

[0054] Furthermore, the identification unit 42 identifies, for example, the type of product contained in the extracted basket area. For this, for example, a machine learning model may be used, which has been pre-trained using existing machine learning techniques to detect products from captured images and output the type of the detected product.

[0055] The calculation unit 43 calculates the number of items contained in the basket, for example, based on the identified position of the contour of the area of ​​the items. This is converted to the number of items by multiplying the estimated amount of items, as explained using Figures 9-12, by a conversion factor such as 20. More specifically, if the estimated amount of items is 0.3, the number of items will be (conversion factor) × (amount of items) = 20 × 0.3 = 6. Note that the conversion to the number of items may also be done by using a nonlinear function f(x) to convert (number of items) = f(amount of items), or by using an estimation method such as a random forest using a decision tree. Furthermore, the number of items may be calculated based on the types of items contained in the area of ​​the basket, as identified by the identification unit 42.

[0056] Furthermore, for example, when a basket contains large items such as toilet paper or tissue boxes, the number of rectangular regions of the extracted items tends to be small, and the area of ​​each rectangular region tends to be large. Therefore, the method described using Figures 9-12 may overestimate the quantity of items, and consequently the number of items. To mitigate this, the estimation error can be reduced by using information about the item rectangles within the visible range, such as their number and size, in conjunction with the conversion to the number of items. Specifically, the calculation unit 43 calculates, for example, the size of the rectangular regions of the items in the image and calculates a second number of items extracted from the image. Then, the calculation unit 43 calculates the number of items contained in the basket based on the calculated region size, the second number of items, and the position identified by the contour identification unit 42 of the item regions. More specifically, the calculation unit 43 calculates the number of items using, for example, the following equations (2) to (4).

[0057]

number

number

number

[0058] Here, we will explain how to calculate the number of items using a machine learning model. By inputting the quantity of items, the area of ​​the rectangular region of each item, and at least one of the number of rectangular regions of each item into a trained machine learning model, the number of items placed in the shopping cart can be determined.

[0059] For example, a trained machine learning model is a machine learning model in which the parameters of a neural network have been modified so that the error between the output result produced by the neural network when the explanatory variable, which is the quantity of goods, is input to the neural network, and the ground truth data, which is the label of the number of goods, is minimized.

[0060] Furthermore, the trained machine learning model may be a machine learning model in which the parameters of the neural network have been modified so that the error between the output result produced by the neural network when explanatory variables such as the quantity of products, the area of ​​the bounding box, and the number of bounding boxes are input to the neural network is small, and the ground truth data which is the label of the number of products. Note that the area of ​​the bounding box may be the area of ​​the image region of the product contained in the basket included in the identified image, which can be obtained from the rectangular region of each product included in the image. Also, the number of bounding boxes may be the sum of the rectangular regions of the identified products, which can be obtained from the number of rectangular regions of the products.

[0061] Now, returning to Figure 9, let's explain how to identify product quantities using bounding boxes. For example, R-CNN or SSD can be used for bounding boxes. R-CNN combines the features of a convolutional neural network with the rectangular regions to detect subsets of those regions (region proposals). Next, CNN features are extracted from the region proposals. Then, the bounding boxes of the region proposals are adjusted using a support vector machine that has been pre-trained with the CNN features.

[0062] SSD is a general object detection algorithm that uses machine learning to determine bounding boxes consisting of rectangular regions called default boxes. It superimposes multiple default boxes of different sizes onto a single image and calculates a predicted value for each box. For each default box, it can predict its position, including how far it is from an object and how much its size differs from the object.

[0063] The detection unit 41 identifies a first bounding box surrounding each product included in the image. The detection unit 41 also identifies the position of the lower boundary line of the second bounding box.

[0064] The identification unit 42 identifies the position of the upper boundary line of the first bounding box for each product included in the image. Then, the identification unit 42 identifies the position corresponding to the entire product contained in the basket from the identified upper boundary line positions for each product included in the image. Furthermore, the identification unit 42 identifies a second bounding box surrounding the basket from the acquired image.

[0065] The calculation unit 43 determines the quantity of goods contained in the basket based on the position of the entire product relative to the position of the lower boundary line. Furthermore, the calculation unit 43 determines the number of goods by inputting the quantity of goods into a trained machine learning model.

[0066] [Process flow] Next, the flow of the product count estimation process performed by the information processing device 10 will be described. Figure 13 is a flowchart showing an example of the product count estimation process according to Embodiment 1. The product count estimation process shown in Figure 13 may be performed, for example, at regular intervals or each time an image is received from the camera device 200.

[0067] First, as shown in Figure 13, the information processing device 10 acquires, for example, an image of a shopping basket captured by the camera device 200 from the image DB 31 (step S101). In the product count estimation process shown in Figure 13, the captured image, more precisely the surveillance video, captured by the camera device 200 is processed in real time, so the captured image is transmitted from the camera device 200 as it occurs and stored in the image DB 31.

[0068] Next, the information processing device 10 detects the shopping basket from the image acquired in step S101, for example, using an existing object detection algorithm (step S102). Note that the shopping basket may be detected by extracting a predetermined area containing the shopping basket using a bounding box or the like.

[0069] Next, the information processing device 10 uses, for example, an existing object detection algorithm to detect the products contained in the shopping basket detected in step S102 from the captured image acquired in step S101 (step S103). Product detection may also be performed by extracting a predetermined area containing the products using a bounding box or the like.

[0070] Next, the information processing device 10 identifies the contour of the area of ​​the product group detected in step S103 (step S104). This involves identifying the position of the upper edge of the contour of the product area 70, with the upper edge of the basket area 60 being set to 1 and the lower edge to 0, as explained using Figure 9. In this case, the upper edge of the area 80 without products may also be identified as 0, as explained using Figure 9.

[0071] Next, the information processing device 10 calculates the quantity of goods based on the positions of the contours of the areas of goods and areas without goods, as identified in step S104 (step S105).

[0072] Then, the information processing device 10 multiplies the quantity of goods calculated in step S105 by a conversion factor such as 20 to convert it into the number of goods (step S106). After step S106 is executed, the product number estimation process shown in Figure 13 is completed.

[0073] Next, we will explain a different product count estimation process from the one shown in Figure 13, following the process flow using Figure 14. Figure 14 is a flowchart showing another example of the product count estimation process for Example 1. Steps S201 to S205 in the product count estimation process shown in Figure 14 are the same as steps S101 to S105 in the product count estimation process shown in Figure 13.

[0074] Next, the information processing device 10 corrects the quantity of goods calculated in step S205 based on the position of the basket, using equation (1), as explained using Figure 12, for example (step S206).

[0075] Then, the information processing device 10 multiplies the product quantity corrected in step S206 by a conversion factor such as 20 to convert it into the number of products (step S207). After step S207 is executed, the product quantity estimation process shown in Figure 14 is completed.

[0076] Next, we will explain a different product quantity estimation process from the one shown in Figures 13 and 14, following the process flow using Figure 15. Figure 15 is a flowchart showing yet another example of the product quantity estimation process according to Example 1. Steps S301 to S304 in the product quantity estimation process shown in Figure 15 are the same as steps S101 to S104 in the product quantity estimation process shown in Figure 13.

[0077] Next, the information processing device 10 calculates the number and size of the product group regions detected in step S303 (step S305).

[0078] Next, the information processing device 10 calculates the quantity of goods based on the positions of the contours of the areas of goods and areas without goods, which were identified in step S304 (step S306).

[0079] Then, the information processing device 10 converts the quantity of goods calculated in step S306 into the number of goods using, for example, the number and size of the areas of the goods group calculated in step S305, and one of the equations (2) to (4) (step S307). After the execution of step S307, the goods number estimation process shown in Figure 15 is completed.

[0080] [effect] As described above, the information processing device 10 acquires an image of a basket containing goods, analyzes the acquired image to extract the basket region included in the image, analyzes the acquired image to extract the regions of one or more goods included in the extracted basket region, estimates the quantity of goods contained in the basket based on the position of the contour of the goods region relative to the position of the extracted basket region, and identifies the number of goods contained in the basket based on the identified quantity of goods. Furthermore, the processing load of the information processing device 10 can be reduced when calculating the number of goods.

[0081] In this way, the information processing device 10 extracts the areas of the shopping basket and products from the captured image and calculates the number of products based on the position of the contours of the product areas relative to the shopping basket area. As a result, the information processing device 10 can more accurately identify the number of products in a shopping basket in a system where customers scan products themselves.

[0082] Furthermore, the information processing device 10 identifies a first bounding box surrounding each product in the image from the acquired image, identifies the position of the upper boundary line of the first bounding box for each product in the image, identifies the position corresponding to the entire product in the basket from the identified upper boundary line of each product in the image, identifies a second bounding box surrounding the basket from the acquired image, identifies the position of the lower boundary line of the second bounding box, identifies the quantity of products in the basket based on the position corresponding to the entire product relative to the lower boundary line, and outputs the number of products in the basket based on the identified quantity.

[0083] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0084] Furthermore, the information processing device 10 detects a first image region surrounding each product in the image from the acquired image, identifies the position of the upper boundary line and the number of first image regions for each product in the image, detects a second image region surrounding the basket from the acquired image, identifies the quantity of products in the basket based on the position of the boundary line of the second image region and the position of the upper boundary line of the first image region, identifies the area of ​​the image region of the products in the basket in the image using the first image region for each product, and identifies the number of products in the basket by inputting the identified quantity of products, the area of ​​the product image region and the number of first image regions into a trained machine learning model.

[0085] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0086] Furthermore, the process for calculating the number of products, performed by the information processing device 10, includes calculating the size of the rectangular area of ​​the product in the image, calculating a second number of products extracted from the image, and calculating the number of products placed in the basket based on the size of the area, the second number of products, and the position of the identified contours.

[0087] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0088] Furthermore, the information processing device 10 identifies a first self-checkout terminal used by a first person holding a shopping basket from a second image captured of an area including a self-checkout terminal, receives product information to be purchased from the first self-checkout terminal, identifies a third number of products to be purchased based on the product information, and determines whether there are any products that are not registered as products to be purchased in the self-checkout terminal based on the number of products calculated based on the position of the identified contours and the third number.

[0089] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0090] Furthermore, the process of receiving product information, which is performed by the information processing device 10, includes the process of receiving product information from a first self-checkout terminal, which is generated when a barcode or two-dimensional code attached to a product is scanned and registered by an information processing terminal used by a first person.

[0091] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0092] Furthermore, the process for calculating the number of items, performed by the information processing device 10, includes identifying a second number of items extracted from the image, calculating the average position of the upper edge of the contour of the item area in the image relative to the extracted basket area, correcting the second number of items based on the average value, and calculating the number of items contained in the basket.

[0093] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0094] Furthermore, the information processing device 10 performs a process to identify the personal belongings of the first person who owns the basket, which are included in the extracted basket area, and the process to calculate the number of items includes a process to calculate the number of items contained in the basket, excluding the identified personal belongings.

[0095] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0096] Furthermore, the process of determining the position of the contour, which is performed by the information processing device 10, includes the process of determining the position of the contour using instance segmentation.

[0097] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0098] Furthermore, the information processing device 10 performs a process to identify the types of products contained in the extracted basket area, and the process to calculate the number of products includes a process to calculate the number of products contained in the basket based on the types of products.

[0099] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0100] Furthermore, the process of extracting the basket region, performed by the information processing device 10, includes the process of extracting the basket region in the rotated image after performing rotation correction on the image.

[0101] As a result, the information processing device 10 can more accurately identify the number of items in a shopping cart in a system where customers scan products themselves.

[0102] [system] The processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and may be changed at will.

[0103] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the diagram. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, all or any part of the processing functions of each device may be implemented by a CPU and the program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0104] [Hardware] Figure 16 illustrates an example of the hardware configuration of the information processing device 10. As shown in Figure 16, the information processing device 10 includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, memory 10c, and a processor 10d. Furthermore, the components shown in Figure 16 are interconnected by a bus or similar means.

[0105] The communication interface 10a is a network interface card or similar device that communicates with other servers. The HDD 10b stores programs and databases that operate the functions shown in Figure 5.

[0106] The processor 10d is a hardware circuit that operates processes that perform the functions described in Figure 5 by reading programs that perform the same processing as each processing unit shown in Figure 5 from the HDD 10b or the like and loading them into memory 10c. In other words, this process performs the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads programs that have the same functions as the detection unit 41, the identification unit 42, the calculation unit 43, and the notification unit 44 from the HDD 10b or the like. Then, the processor 10d executes a process that performs the same processing as the identification unit 42, etc.

[0107] Thus, the information processing device 10 operates as an information processing device that performs operation control processing by reading and executing a program that performs the same processing as each processing unit shown in Figure 5. In addition, the information processing device 10 can also achieve the same functions as in the above-described embodiment by reading a program from a recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 10. For example, this embodiment may also be applied when another information processing device executes a program, or when the information processing device 10 and other information processing devices cooperate to execute a program.

[0108] Furthermore, programs that perform the same processing as the processing units shown in Figure 5 can be distributed via networks such as the Internet. These programs can also be recorded on computer-readable storage media such as hard disks, flexible disks (FDs), CD-ROMs, MOs (Magneto-Optical disks), and DVDs (Digital Versatile Discs), and executed by being read from these media by a computer.

[0109] Figure 17 illustrates an example of the hardware configuration of the information processing terminal 1000. Since the user terminal 100 and the employee terminal 300 may have similar configurations, Figure 17 shows them together as the information processing terminal 1000, illustrating an example of the hardware configuration for both the user terminal 100 and the employee terminal 300. As shown in Figure 17, the information processing terminal 1000 includes a communication interface 1000a, an HDD 1000b, memory 1000c, a processor 1000d, an input device 1000e, and a display device 1000f. Furthermore, the components shown in Figure 17 are interconnected by a bus or similar means.

[0110] The communication interface 1000a is a network interface card or the like, and is used for communication with other information processing devices. The HDD 1000b stores programs and data that operate the various functions of the information processing terminal 1000.

[0111] The processor 100d is a hardware circuit that operates processes that execute the various functions of the information processing terminal 1000 by reading programs that perform the processing of each function of the information processing terminal 1000 from the HDD 1000b or the like and loading them into memory 1000c. In other words, this process performs the same functions as the various processing units of the information processing terminal 1000.

[0112] Thus, the information processing terminal 1000 operates as an information processing device that performs operation control processing by reading and executing programs that execute the processing of each function of the information processing terminal 1000. Furthermore, the information processing terminal 1000 can also realize each function of the information processing terminal 1000 by reading a program from a recording medium using a media reader and executing the read program. It should be noted that the program referred to in this other embodiment is not limited to being executed by the information processing terminal 1000. For example, this embodiment may also be applied to cases where another information processing device executes a program, or where the information processing terminal 1000 and another information processing device cooperate to execute a program.

[0113] Furthermore, the programs that execute the processing of each function of the information processing terminal 1000 can be distributed via a network such as the Internet. These programs can also be recorded on computer-readable storage media such as hard disks, floppy disks, CD-ROMs, MOs, and DVDs, and executed by being read from these media by a computer.

[0114] The input device 1000e detects various user input operations, such as input operations to a program executed by the processor 100d. These input operations include, for example, touch operations and inserting an earphone jack into the information processing terminal 1000. Here, a touch operation refers to various contact actions with the display device 1000f, such as tapping, double-tapping, swiping, and pinching. A touch operation also includes bringing an object, such as a finger, close to the display device 1000f. The input device 1000e may be, for example, a button, a touch panel, or a proximity sensor.

[0115] The display device 1000f displays various visual information based on control by the processor 1000d. The display device 1000f may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), or a so-called organic EL (electroluminescence) display.

[0116] Figure 18 illustrates an example of the hardware configuration of a self-checkout terminal 400. As shown in Figure 18, the self-checkout terminal 400 includes a communication interface 400a, an HDD 400b, memory 400c, a processor 400d, an input device 400e, and an output device 400f. Furthermore, the components shown in Figure 18 are interconnected via a bus or similar means.

[0117] The communication interface 400a is a network interface card or similar device that communicates with other information processing devices. The HDD 400b stores programs and data that operate the various functions of the self-checkout terminal 400.

[0118] The processor 400d is a hardware circuit that operates the processes that execute the functions of the self-checkout terminal 400 by reading programs that perform the processing of each function of the self-checkout terminal 400 from the HDD 400b or the like and loading them into memory 400c. In other words, this process performs the same functions as the processing units that the self-checkout terminal 400 has.

[0119] Thus, the self-checkout terminal 400 operates as an information processing device that performs operation control processing by reading and executing programs that execute the processing of each function of the self-checkout terminal 400. Furthermore, the self-checkout terminal 400 can also realize its functions by reading a program from a recording medium using a media reader and executing the read program. It should be noted that the program referred to in this other embodiment is not limited to being executed by the self-checkout terminal 400. For example, this embodiment may also be applied to cases where another information processing device executes the program, or where the self-checkout terminal 400 and another information processing device cooperate to execute the program.

[0120] Furthermore, the programs that execute the processing of each function of the self-checkout terminal 400 can be distributed via a network such as the Internet. These programs can also be recorded on computer-readable storage media such as hard disks, floppy disks, CD-ROMs, MOs, and DVDs, and executed by being read from these media by a computer.

[0121] The input device 400e detects various user input operations, such as input operations to a program executed by the processor 400d. Such input operations include, for example, touch operations. In the case of touch operations, the self-checkout terminal 400 is further equipped with a display device, and the input operation detected by the input device 400e may be a touch operation on the display device. The input device 400e may be, for example, a button, a touch panel, a proximity sensor, etc.

[0122] The output device 400f outputs data from a program executed by the processor 400d via an external device connected to the self-checkout terminal 400, such as an external display device. Note that if the self-checkout terminal 400 is equipped with a display device, it does not need to be equipped with the output device 400f.

[0123] With regard to embodiments including the above examples, the following additional information is disclosed.

[0124] (Note 1) Obtain an image of the basket in which the product is stored. Extract the area of ​​the basket in the aforementioned image, Extract the area of ​​one or more products included in the area of ​​the extracted basket, The position of the contour of the product area relative to the extracted basket area is identified, Based on the location of the identified contour, the number of items contained in the basket is calculated. An information processing program characterized by having a computer perform the processing.

[0125] (Note 2) From the acquired image, a first bounding box surrounding each product included in the image is identified, Identify the position of the upper boundary line of the first bounding box for each product included in the aforementioned image, From the position of the upper boundary line for each product included in the identified image, the position corresponding to the entire collection of products in the basket is determined. From the acquired image, a second bounding box surrounding the basket is identified. Identify the position of the lower boundary of the second bounding box, Based on the position corresponding to the entire product with respect to the position of the lower boundary line, the quantity of the product contained in the basket is determined. Based on the specified quantity of goods, the number of goods contained in the basket is output. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0126] (Note 3) From the acquired image, a first image region surrounding each product included in the image is detected, The position of the upper boundary line of the first image region and the number of the first image regions are identified for each product included in the aforementioned image. From the acquired image, a second image region surrounding the basket is detected. Based on the position of the boundary line of the second image region and the position of the upper boundary line of the first image region, the quantity of the product contained in the basket is determined. Using the first image region for each product, the area of ​​the image region of the product contained in the basket included in the image is identified. The number of items contained in the basket is determined by inputting the identified quantity of items, the area of ​​the image region of the items, and the number of the first image regions into a trained machine learning model. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0127] (Note 4) The process for identifying the number of the above-mentioned products is: The size of the rectangular area of ​​the product in the aforementioned image is calculated, The second number of the product extracted from the aforementioned image is calculated, Based on the size of the area, the second number of the items, and the location of the identified contour, the number of items contained in the basket is determined. An information processing program as described in Appendix 1, characterized by including processing.

[0128] (Note 5) From the second image, which captures the area including the self-checkout terminal, the first self-checkout terminal used by the first person holding the basket is identified. The system receives product information for purchase from the first self-checkout terminal. Based on the aforementioned product information, a third number of the aforementioned products to be purchased is identified. Based on the number of items calculated based on the location of the identified contour and the third number, it is determined whether or not there are any items that are not registered as items to be purchased in the self-checkout terminal. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0129] (Note 6) The process of receiving the above product information is: The product information generated by scanning and registering the barcode or two-dimensional code attached to the product using the information processing terminal used by the first person is received from the first self-checkout terminal. An information processing program as described in Appendix 3, characterized by including processing.

[0130] (Note 7) The process for determining the quantity of the above-mentioned goods is: The average position of the upper edge of the contour of the product area in the image relative to the extracted basket area is calculated. The average value is corrected based on the position of the basket to estimate the quantity of goods. An information processing program as described in Appendix 1, characterized by including processing.

[0131] (Note 8) The computer is instructed to perform the process of identifying the personal belongings of the first person who owns the basket, which are included in the extracted basket area. The process of identifying the number of the aforementioned products is: Excluding the identified personal items, determine the number of items contained in the basket. An information processing program as described in Appendix 1, characterized by including processing.

[0132] (Note 9) The location of the contour is determined using instance segmentation. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0133] (Note 10) The computer is instructed to perform a process to identify the type of product contained in the extracted basket area. The process for calculating the number of the aforementioned products is: Based on the type of product, the number of products placed in the basket is calculated. An information processing program as described in Appendix 1, characterized by including processing.

[0134] (Note 11) The information processing program according to Note 1, characterized in that the process for extracting the basket region includes the process of performing rotation correction on the image and then extracting the basket region in the rotated image.

[0135] (Note 12) Obtain an image of the basket in which the product is stored. Extract the area of ​​the basket in the aforementioned image, Extract the area of ​​one or more products included in the area of ​​the extracted basket, The position of the contour of the product area relative to the extracted basket area is identified, Based on the location of the identified contour, the number of items contained in the basket is calculated. An information processing method characterized in that the processing is performed by a computer.

[0136] (Note 13) From the acquired image, identify the first bounding box surrounding each product included in the image, Identify the position of the upper boundary line of the first bounding box for each product included in the aforementioned image, From the position of the upper boundary line for each product included in the identified image, the position corresponding to the entire collection of products in the basket is determined. From the acquired image, a second bounding box surrounding the basket is identified. Identify the position of the lower boundary of the second bounding box, Based on the position corresponding to the entire product with respect to the position of the lower boundary line, the quantity of the product contained in the basket is determined. Based on the specified quantity of goods, the number of goods contained in the basket is output. The information processing method according to Appendix 12, characterized in that the processing is performed by the computer.

[0137] (Note 14) From the acquired image, a first image region surrounding each product included in the image is detected, The position of the upper boundary line of the first image region and the number of the first image regions are identified for each product included in the aforementioned image. From the acquired image, a second image region surrounding the basket is detected. Based on the position of the boundary line of the second image region and the position of the upper boundary line of the first image region, the quantity of the product contained in the basket is determined. Using the first image region for each product, the area of ​​the image region of the product contained in the basket included in the image is identified. The number of items contained in the basket is determined by inputting the identified quantity of items, the area of ​​the image region of the items, and the number of the first image regions into a trained machine learning model. The information processing method according to Appendix 12, characterized in that the processing is performed by the computer.

[0138] (Note 15) The process for determining the number of the above-mentioned products is: The size of the rectangular area of ​​the product in the aforementioned image is calculated, The second number of the product extracted from the aforementioned image is calculated, Based on the size of the area, the second number of the items, and the location of the identified contour, the number of items contained in the basket is determined. The information processing method described in Appendix 12, characterized by including processing.

[0139] (Note 16) From the second image, which captures the area including the self-checkout terminal, the first self-checkout terminal used by the first person holding the basket is identified. The system receives product information for purchase from the first self-checkout terminal. Based on the aforementioned product information, a third number of the aforementioned products to be purchased is identified. Based on the number of items calculated based on the location of the identified contour and the third number, it is determined whether or not there are any items that are not registered as items to be purchased in the self-checkout terminal. The information processing method according to Appendix 12, characterized in that the processing is performed by the computer.

[0140] (Note 17) The process of receiving the above product information is: The product information generated by scanning and registering the barcode or two-dimensional code attached to the product using the information processing terminal used by the first person is received from the first self-checkout terminal. The information processing method described in Appendix 14, characterized by including processing.

[0141] (Note 18) The process for estimating the quantity of the goods is as follows: The average position of the upper edge of the contour of the product area in the image relative to the extracted basket area is calculated. The average value is corrected based on the position of the basket to estimate the quantity of goods. The information processing method described in Appendix 12, characterized by including processing.

[0142] (Note 19) The computer performs a process to identify the personal belongings of the first person who owns the basket, which are included in the extracted basket area. The process of identifying the number of the aforementioned products is: Excluding the identified personal items, determine the number of items contained in the basket. The information processing method described in Appendix 12, characterized by including processing.

[0143] (Note 20) The location of the contour is determined using instance segmentation. The information processing method according to Appendix 12, characterized in that the processing is performed by the computer.

[0144] (Note 21) The computer performs a process to identify the type of product contained in the extracted basket area. The process for calculating the number of the aforementioned products is: Based on the type of product, the number of products placed in the basket is calculated. The information processing method described in Appendix 12, characterized by including processing.

[0145] (Note 22) The information processing method according to Note 12, characterized in that the process for extracting the basket region includes the process of performing rotation correction on the image and then extracting the basket region in the rotated image.

[0146] (Note 23) Obtain an image of the basket in which the product is stored. Extract the area of ​​the basket in the aforementioned image, Extract the area of ​​one or more products included in the area of ​​the extracted basket, The position of the contour of the product area relative to the extracted basket area is identified, Based on the location of the identified contour, the number of items contained in the basket is calculated. An information processing apparatus characterized by having a control unit that performs processing.

[0147] (Note 24) From the acquired image, identify the first bounding box surrounding each product included in the image, Identify the position of the upper boundary line of the first bounding box for each product included in the aforementioned image, From the position of the upper boundary line for each product included in the identified image, the position corresponding to the entire collection of products in the basket is determined. From the acquired image, a second bounding box surrounding the basket is identified. Identify the position of the lower boundary of the second bounding box, Based on the position corresponding to the entire product with respect to the position of the lower boundary line, the quantity of the product contained in the basket is determined. Based on the specified quantity of goods, the number of goods contained in the basket is output. The information processing apparatus according to Appendix 23, characterized in that the control unit performs the processing.

[0148] (Note 25) From the acquired image, a first image region surrounding each product included in the image is detected, The position of the upper boundary line of the first image region and the number of the first image regions are identified for each product included in the aforementioned image. From the acquired image, a second image region surrounding the basket is detected. Based on the position of the boundary line of the second image region and the position of the upper boundary line of the first image region, the quantity of the product contained in the basket is determined. Using the first image region for each product, the area of ​​the image region of the product contained in the basket included in the image is identified. The number of items contained in the basket is determined by inputting the identified quantity of items, the area of ​​the image region of the items, and the number of the first image regions into a trained machine learning model. The information processing apparatus according to Appendix 23, characterized in that the control unit performs the processing.

[0149] (Note 26) The process for determining the number of the above-mentioned products is: The size of the rectangular area of ​​the product in the aforementioned image is calculated, The second number of the product extracted from the aforementioned image is calculated, Based on the size of the area, the second number of the items, and the location of the identified contour, the number of items contained in the basket is determined. An information processing apparatus according to Appendix 23, characterized by including processing.

[0150] (Note 27) From the second image, which captures the area including the self-checkout terminal, the first self-checkout terminal used by the first person holding the basket is identified. The system receives product information for purchase from the first self-checkout terminal. Based on the aforementioned product information, a third number of the aforementioned products to be purchased is identified. Based on the number of items calculated based on the location of the identified contour and the third number, it is determined whether or not there are any items that are not registered as items to be purchased in the self-checkout terminal. The information processing apparatus according to Appendix 23, characterized in that the control unit performs the processing.

[0151] (Note 28) The process of receiving the above product information is: The product information generated by scanning and registering the barcode or two-dimensional code attached to the product using the information processing terminal used by the first person is received from the first self-checkout terminal. An information processing apparatus according to Appendix 27, characterized by including processing.

[0152] (Note 29) The process for estimating the quantity of the goods is as follows: The average position of the upper edge of the contour of the product area in the image relative to the extracted basket area is calculated. The average value is corrected based on the position of the basket to estimate the quantity of goods. An information processing apparatus according to Appendix 23, characterized by including processing.

[0153] (Note 30) The control unit performs a process to identify the personal belongings of the first person who owns the basket, which are included in the extracted basket area. The process of identifying the number of the aforementioned products is: Excluding the identified personal items, determine the number of items contained in the basket. An information processing apparatus according to Appendix 23, characterized by including processing.

[0154] (Note 31) The location of the contour is determined using instance segmentation. The information processing apparatus according to Appendix 23, characterized in that it causes the computer to perform the processing.

[0155] (Note 32) The control unit performs a process to identify the type of product included in the extracted basket area. The process for calculating the number of the aforementioned products is: Based on the type of product, the number of products placed in the basket is calculated. An information processing apparatus according to Appendix 23, characterized by including processing.

[0156] (Note 33) The information processing apparatus according to Note 23, characterized in that the process for extracting the basket region includes the process of performing rotation correction on the image and then extracting the basket region in the rotated image.

[0157] (Note 34) Processor and Memory that is operablely connected to the processor and An information processing device comprising, the processor, We obtained an image of the basket that holds the product, By analyzing the acquired image, the region of the basket contained in the image is extracted. By analyzing the acquired image, the regions of one or more products included in the extracted basket region are extracted. Based on the position of the contour of the product area relative to the position of the extracted basket area, the quantity of the product contained in the basket is estimated. Based on the identified quantity of the aforementioned goods, the number of goods contained in the basket is determined. An information processing device characterized by performing processing. [Explanation of symbols]

[0158] 1. Information Processing System 10 Information Processing Devices 10a communication interface 10b HDD 10c memory 10d processor 20 Communications Department 30 Storage section 31 Image Database 32 Product information 40 Control Unit 41 Detection unit 42 Specific part 43 Calculation Section 44 Notification Department 50 Networks 60 Basket Area 70 product categories 80 Areas where no products are available 100 User Terminals 400 self-checkout terminals 400a communication interface 400b HDD 400c memory 400d processor 400e Input Device 400f output device 1000 Information Processing Terminals 1000a communication interface 1000b HDD 1000c memory 1000d processor 1000e Input Device 1000f display device

Claims

1. We obtained an image of the basket that holds the product, By analyzing the acquired image, the region of the basket contained in the image is extracted. By analyzing the acquired image, the regions of one or more products included in the extracted basket region are extracted. Based on the position of the contour of the product area relative to the position of the extracted basket area, the quantity of the product contained in the basket is estimated. Based on the identified quantity of the aforementioned goods, the number of goods contained in the basket is determined. An information processing program characterized by having a computer perform the processing.

2. From the acquired image, a first bounding box surrounding each product included in the image is identified. Identify the position of the upper boundary line of the first bounding box for each product included in the image, From the position of the upper boundary line for each product included in the identified image, the position corresponding to the entire collection of products in the basket is determined. From the acquired image, a second bounding box surrounding the basket is identified. Identify the position of the lower boundary line of the second bounding box, Based on the position corresponding to the entire product with respect to the position of the lower boundary line, the quantity of the product contained in the basket is determined. Based on the specified quantity of goods, the number of goods contained in the basket is output. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

3. From the acquired image, a first image region surrounding each product included in the image is detected. The position of the upper boundary line of the first image region for each product included in the image and the number of the first image regions are identified, respectively. From the acquired image, a second image region surrounding the basket is detected. Based on the position of the boundary line of the second image region and the position of the upper boundary line of the first image region, the quantity of the product contained in the basket is determined. Using the first image region for each product, the area of ​​the image region of the product contained in the basket included in the image is identified. The number of items contained in the basket is determined by inputting the identified quantity of items, the area of ​​the image region of the items, and the number of the first image regions into a trained machine learning model. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

4. The process of identifying the number of the aforementioned products is: The size of the rectangular area of ​​the product in the aforementioned image is calculated, The second number of the product extracted from the aforementioned image is calculated, Based on the size of the area, the second number of the items, and the location of the identified contour, the number of items contained in the basket is determined. The information processing program according to claim 1, characterized by including processing.

5. From a second image capturing the area including the self-checkout terminal, the first self-checkout terminal used by the first person holding the basket is identified. The system receives product information to be purchased from the first self-checkout terminal. Based on the aforementioned product information, a third number of the aforementioned products to be purchased is identified. Based on the number of items calculated based on the location of the identified contour and the third number, it is determined whether or not there are any items that are not registered as items to be purchased in the self-checkout terminal. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

6. The process of receiving the aforementioned product information is: The product information generated by scanning and registering the barcode or two-dimensional code attached to the product using the information processing terminal used by the first person is received from the first self-checkout terminal. The information processing program according to claim 5, characterized in that it includes processing.

7. The process for estimating the quantity of the aforementioned goods is as follows: The average position of the upper edge of the contour of the product area in the image relative to the extracted basket area is calculated. The average value is corrected based on the position of the basket to estimate the quantity of goods. The information processing program according to claim 1, characterized by including processing.

8. The computer is instructed to perform a process to identify the personal belongings of the first person who owns the basket, which are included in the extracted basket area. The process of identifying the number of the aforementioned products is: Excluding the identified personal items, determine the number of items contained in the basket. The information processing program according to claim 1, characterized by including processing.

9. The location of the contour is determined using instance segmentation. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

10. The computer is instructed to perform a process to identify the type of product contained in the extracted basket area. The process for calculating the number of the aforementioned products is: Based on the type of product, the number of products placed in the basket is calculated. The information processing program according to claim 1, characterized by including processing.

11. The information processing program according to claim 1, characterized in that the process for extracting the basket region includes, after performing rotation correction on the image, the process for extracting the basket region in the rotation-corrected image.

12. We obtained an image of the basket that holds the product, By analyzing the acquired image, the region of the basket contained in the image is extracted. By analyzing the acquired image, the regions of one or more products included in the extracted basket region are extracted. Based on the position of the contour of the product area relative to the position of the extracted basket area, the quantity of the product contained in the basket is estimated. Based on the identified quantity of the aforementioned goods, the number of goods contained in the basket is determined. An information processing method characterized in that the processing is performed by a computer.

13. We obtained an image of the basket that holds the product, By analyzing the acquired image, the region of the basket contained in the image is extracted. By analyzing the acquired image, the regions of one or more products included in the extracted basket region are extracted. Based on the position of the contour of the product area relative to the position of the extracted basket area, the quantity of the product contained in the basket is estimated. Based on the identified quantity of the aforementioned goods, the number of goods contained in the basket is determined. An information processing apparatus characterized by having a control unit that performs processing.

Images