A device for generating training data, a device for verifying the number of items, a method for generating training data, a method for verifying the number of items, and a program.

Abstract

A training data generation device (10) comprises an acquisition unit (110) and a selection unit (120). The acquisition unit (110) acquires a plurality of images generated by an imaging means. From the plurality of images acquired by the acquisition unit (110), the selection unit (120) selects, as negative data that does not include a target product, an image satisfying a preset condition for inclusion in at least part of training data. Using a model generated with use of this training data makes it possible to detect with high accuracy the number of products included in an image.

Description

Technical Field

[0001] The present invention relates to a learning data generation device, a product number confirmation device, a learning data generation method, a product number confirmation method, and program and.

Background Art

[0002] In recent years, when a customer purchases a product, the customer may operate a product registration device by themselves. In this case, there is a possibility that the product may not be registered accurately. In contrast, Patent Document 1 describes an information processing device that performs the following processing. First, the information processing device detects a product in a first predetermined area, for example, a product in an area for placing unregistered products. Next, the information processing device determines whether or not the detected product has moved to a second predetermined area, for example, a product scan area, within a predetermined time after moving outside this area.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As described above, when the customer operates the product registration device by themselves, there is a possibility that the product may not be registered accurately. In the technique described in Patent Document 1, only the products arranged in the first predetermined area can be tracked. Therefore, there remains a possibility that the accuracy of the registration result of the product with respect to the product registration device may decrease.

[0005] In contrast, the present inventor considered determining the possibility that a product is not registered accurately by comparing the number of products registered by the product registration device with the number of products included in an image obtained by photographing the area including the product registration device. However, when performing this processing, it is necessary to accurately detect the number of products included in the image.

[0006] One example of the object of the present invention is, in view of the above-mentioned problems, a learning data generation device, a product count verification device, a learning data generation method, a product count verification method, and that can improve the accuracy of detection when processing images to detect the number of products. program The objective is to provide. [Means for solving the problem]

[0007] According to one aspect of the present invention, a training data generation device that generates at least a portion of training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned training data generation device is An acquisition means for acquiring multiple images generated by the aforementioned shooting means, A selection means for selecting images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; A training data generation device equipped with the following is provided.

[0008] According to one aspect of the present invention, it is used together with the above-described learning data generation device, A calculation means that calculates a first number, which is the number of target products, by processing the images captured by the shooting means when a customer uses the product registration device, using the model generated using the training data, An output means that outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number satisfies a criterion, A product count verification device equipped with the following features is provided.

[0009] According to one aspect of the present invention, a computer method for generating training data for generating a model, wherein the computer generates at least a portion of the training data, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned computer, The aforementioned imaging means acquires a plurality of images generated by the imaging means, A method for generating training data is provided, which involves selecting from the aforementioned plurality of images images that satisfy predetermined conditions to be included as negative data that does not include the target product, in at least part of the training data.

[0010] According to one aspect of the present invention, a computer used together with the above-described learning data generation device is Using the model generated with the aforementioned training data, the image captured by the photographing means when a customer uses the product registration device is processed to calculate a first number which is the number of target products. A method for verifying the number of products is provided, which outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number meets a standard.

[0011] According to one aspect of the present invention, a computer, The number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed is calculated. A program that generates at least a portion of the training data needed to create a model. Mu There is, before On the computer, An acquisition function that acquires multiple images generated by the aforementioned shooting means, A selection function that selects images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; Give it program It will be provided.

[0012] According to one aspect of the present invention, a program used in a computer used together with the above-described learning data generation device Mu There is, before On the computer, Using the model generated using the learning data, by processing the image captured by the imaging means when a customer uses the product registration device, a calculation function for calculating a first number, which is the number of target products, an output function for outputting predetermined information when a difference between a second number, which is the number of target products registered in the product registration device, and the first number satisfies a criterion; is provided with program is provided.

Advantages of the Invention

[0013] According to one aspect of the present invention, when processing an image to detect the number of products, it is possible to improve the accuracy of this detection. A learning data generation device, a product number confirmation device, a learning data generation method, a product number confirmation method, and program can be provided.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram showing an overview of a learning data generation device according to an embodiment. [[ID=at]] [Figure 2] It is a diagram showing an overview of a product number confirmation device according to an embodiment. [Figure 3] It is a diagram for explaining the usage environment of a learning data generation device and a product number confirmation device. [Figure 4] It is a diagram showing an example of the arrangement of a product number confirmation device. [Figure 5] [[ID=at]]It is a diagram showing an example of the functional configuration of a learning data generation device. [Figure 6] It is a diagram for explaining an example of a target area of object detection processing by an image processing unit. [Figure 7] It is a diagram showing an example of learning data stored in a learning data storage unit. [Figure 8] It is a diagram showing an example of the functional configuration of a product number confirmation device. [Figure 9] It is a diagram showing an example of the hardware configuration of a learning data generation device. [Figure 10]This flowchart shows an example of the processing performed by the training data generation device. [Figure 11] This flowchart shows an example of the process performed by the product count verification device. [Figure 12] This figure shows a modified example of Figure 4. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0016] Figure 1 shows an overview of the training data generation device 10 according to an embodiment. The training data generation device 10 generates at least a portion of the training data for generating a model. This model calculates the number of target products included in the images generated by the imaging device. The target products are products subject to settlement. The imaging range of the imaging device includes the area where the target products are placed.

[0017] The learning data generation device 10 includes an acquisition unit 110 and a selection unit 120. The acquisition unit 110 captures Device The system acquires multiple images generated by the system. The selection unit 120 selects images from the multiple images acquired by the acquisition unit 110 that satisfy predetermined conditions, in order to include them as negative data that does not include the target product, in at least part of the training data.

[0018] According to the training data generation device 10, the training data includes images that satisfy predetermined conditions, which are negative data that do not contain the target product. Therefore, by using the model generated with this training data, the number of products included in an image can be detected with high accuracy.

[0019] Figure 2 shows an overview of the product count verification device 20 according to an embodiment. The product count verification device 20 is used together with the learning data generation device 10 and comprises a calculation unit 210 and an output unit 220. The calculation unit 210 uses a model generated using the learning data generated by the learning data generation device 10 to process images captured by the camera when a customer uses the product registration device, thereby calculating a first number, which is the number of target products. The output unit 220 outputs predetermined information when the difference between the first number and a second number, which is the number of target products registered in the product registration device, meets a standard.

[0020] The product count verification device 20 has high accuracy in detecting the first number. Furthermore, the product count verification device 20 outputs a predetermined message if the difference between the number of target products included in the image and the number of target products registered in the product registration device does not meet the criteria. Therefore, the administrator or customer of the product count verification device 20 can recognize that there is a possibility that the products are not registered accurately.

[0021] The following describes detailed examples of the training data generation device 10 and the product count verification device 20.

[0022] Figure 3 is a diagram illustrating the operating environment of the learning data generation device 10 and the product count verification device 20. In the example shown in this figure, the product count verification device 20 also serves as a product registration device. The learning data generation device 10 and the product count verification device 20 are used together with the imaging device 30 and the product information storage unit 40.

[0023] The product quantity verification device 20 is installed in a store or office. This store or office contains goods. The product quantity verification device 20 registers the goods a customer intends to purchase and processes the payment for the registered goods. In other words, the product quantity verification device 20 also functions as a product registration device and a payment device. The product quantity verification device 20 may also be a POS terminal. During the registration and payment processes for the target goods, the product quantity verification device 20 is operated by the customer. Furthermore, the product quantity verification device 20 uses the product information storage unit 40 during the registration and payment processes for the target goods.

[0024] However, the product count verification device 20 does not need to perform the registration and settlement processes for the target products. In this case, the product count verification device 20 may be a separate device from the POS terminal, such as a cloud server.

[0025] The product information storage unit 40 stores information necessary for the settlement process of products, such as the price of a product, associated with the product identification information of that product. The product information storage unit 40 is, for example, part of a server installed in the store, but is not limited to that.

[0026] The imaging device 30 operates at least while the product count verification device 20 is powered on. The imaging device 30 may operate 24 hours a day. The imaging device 30 may also generate images with RGB data for each pixel, or images with depth information for each pixel. The imaging device 30 generates images periodically. These images may constitute a video. The frame rate of the imaging device 30 is, for example, 0.1 fps or more and 30 fps or less, but is not limited to these values.

[0027] The mounting position of the camera 30 is arbitrary. The camera 30 may be mounted above the product counting device 20, for example, on the ceiling of the room where the product counting device 20 is located, or it may be mounted on the product counting device 20.

[0028] The shooting range of the shooting device 30 includes the area where the target product is placed when the registration process for the target product is performed. The area where the target product is placed includes at least one of the following: the area where the target product is temporarily placed, and the area where the target product is placed when the product identification information of the target product is registered with the product count confirmation device 20. An example of the former is the area on the platform 50 where the target product is temporarily placed. An example of the latter is the area (space) from which the auxiliary device 60, which reads product identification information from the target product, can read the product identification information.

[0029] Furthermore, the shooting range of the shooting device 30 may also include the product count confirmation device 20 and the product display area. Products that can be registered by the product count confirmation device 20 may be placed only in the product display area included in this shooting range, or they may be placed in the product display area outside this shooting range.

[0030] The training data generation device 10 generates training data for model generation using images generated by the imaging device 30. This model is used to calculate the number of target products located within the imaging range. The training data generation device 10 adds images that meet predetermined conditions from among the images generated by the imaging device 30 to the training data as negative data that does not contain target products.

[0031] In the example shown in this figure, the training data generation device 10 also generates the model described above. However, this model generation process may be performed by a device other than the training data generation device 10.

[0032] The product count verification device 20 calculates the number of target products located within the shooting range, i.e., the first number, by processing the image captured by the shooting device 30. In this process, the product count verification device 20 uses the model generated by the training data generation device 10. The product count verification device 20 then outputs a predetermined value if the difference between the first number and the number of target products registered in the product count verification device 20, i.e., the second number, meets the criteria.

[0033] Figure 4 shows an example of the arrangement of the product count verification device 20. The product count verification device 20 is placed on a stand 50. The stand 50 is sufficiently larger than the product count verification device 20, and a portion of it forms the product display area 510. In the example shown in this figure, the shooting range of the camera 30 includes the product count verification device 20 and the product display area 510. As explained using Figure 3, the target products that can be registered with the product count verification device 20 may also be displayed in locations other than the product display area 510.

[0034] On the platform 50, at least one of the accessory devices 60 of the product count verification device 20 is often arranged, such as a card reader, barcode reader, 2D code reader, short-range wireless communication device for communicating with wireless communication tags attached to products, short-range wireless communication device for communicating with mobile terminals, and a receipt printing device. In addition, objects 70 such as trash cans may be placed around the platform 50. If the accessory devices 60 and objects 70 are included in the shooting range, these accessory devices 60 and objects 70 may be mistakenly identified as target products. The negative data described above is used to prevent these from being mistakenly identified as target products.

[0035] Figure 12 shows a modified example of the shooting range of the shooting device 30. In the example shown in this figure, the shooting range includes the product display area 510 and the auxiliary equipment 60, but does not include the product count confirmation device 20. Thus, the shooting range of the shooting device 30 only needs to include the area where the target products are placed.

[0036] Figure 5 shows an example of the functional configuration of the training data generation device 10. The training data generation device 10 includes an acquisition unit 110, a selection unit 120, an image processing unit 130, a storage processing unit 140, a model generation unit 150, and a communication unit 160. The training data generation device 10 can utilize a training data storage unit 170.

[0037] The acquisition unit 110 acquires multiple images generated by the imaging device 30. The acquisition unit 110 acquires these images, for example, from the imaging device 30, but it may also acquire these images from a storage device that stores them. The acquisition unit 110 may acquire all of the images generated by the imaging device 30, or it may acquire only some of the images. In addition, when acquiring images, the acquisition unit 110 also acquires the date and time data of these images.

[0038] The acquisition unit 110 may also store the acquired images in a storage unit, such as a learning data storage unit 170.

[0039] The selection unit 120 selects images from among the multiple images acquired by the acquisition unit 110 that satisfy predetermined conditions, as negative data that does not include the target product, and stores them in the learning data storage unit 170 as at least part of the learning data. In this process, the selection unit 120 uses the shooting date and time data of each image as necessary. The predetermined conditions are, for example, conditions that indicate a high probability that the product count confirmation device 20 is not being used. An example of a predetermined condition is at least one of the following (1) to (5).

[0040] (1) The difference between the image in question and the reference image satisfies the first criterion. The reference image is, for example, a pre-set background image, or an image generated by the imaging device 30, selected from the image taken a predetermined time ago. before This is an image generated (or a predetermined number of times earlier). Another example of the first criterion is that the difference is less than or equal to a standard value. If this condition is met, there are no moving objects around the product count verification device 20, so it is highly likely that the product count verification device 20 is not in use.

[0041] (2) At the time the image is generated, the state of the product counting device 20 must meet the second criterion. In this case, the acquisition unit 110 acquires information indicating the status of the product count confirmation device 20, i.e., information indicating the change in status, from the product count confirmation device 20 or the device that manages the product count confirmation device 20, on a daily basis. The selection unit 120 uses this information. The second criterion is a state indicating that the product count confirmation device 20 is not being used. An example of the second criterion is as follows. • The product count verification device 20 has not performed the product registration process. • The product counting device 20 is in standby mode and powered off.

[0042] (3) The timing of the image's generation is predetermined. The predetermined dates and times are those when the store is closed or the office is closed, or when the product count verification device 20 is not used according to the operational plan for the product count verification device 20. Note that the dates and times here may also refer to specific time slots on a specific day or day of the week.

[0043] (4) The image contains a specific object. A specific object is one that indicates that the product count verification device 20 cannot be used, such as a sign. In this case, the selection unit 120 has pre-stored the characteristic quantities of this object. The selection unit 120 then performs image processing to determine whether or not an object possessing these characteristic quantities is present.

[0044] (5) At the time the image is generated, there are no people in the predetermined area. The predetermined area is a specific area of ​​the store or office. This area may or may not include the shooting range of the camera 30. In the latter case, the predetermined area may include the area surrounding the shooting range of the product count confirmation device 20, for example, the passage leading to the product count confirmation device 20. Examples of the processes performed here are (5-1) to (5-3) below.

[0045] (5-1) The selection unit 120 performs human detection processing on the images acquired by the acquisition unit 110, and sets images in which no human was detected as negative data.

[0046] (5-2) The selection unit 120 acquires images from a surveillance camera installed in the store or office and performs human detection processing on these images. The selection unit 120 then identifies a time when there are no people in a predetermined area and turns the images generated by the camera 30 at this time into negative data.

[0047] (5-3) The selection unit 120 uses the detection result of a human presence sensor, which has a predetermined detection range, to identify the timing when there are no people in the predetermined area. The human presence sensor is, for example, an infrared sensor. The product count confirmation device 20 then converts the image generated by the shooting device 30 at this timing into negative data.

[0048] The image processing unit 130 performs object detection processing on the image selected by the selection unit 120, i.e., the image that becomes negative data. The algorithm for the object detection processing performed here is preferably the same as the algorithm used by the product count confirmation device 20 to calculate the first number, for example, the model used by the product count confirmation device 20.

[0049] The memory processing unit 140 stores each of the multiple images selected by the product count verification device 20 as training data in the training data storage unit 170. At this time, these multiple images are stored in the training data storage unit 170 as negative data. Specifically, the memory processing unit 140 stores each of the multiple images in the training data storage unit 170, linking it to the processing result of the image processing unit 130 for that image.

[0050] The negative data stored in the learning data storage unit 170 includes an image and data indicating that the image does not contain a product. An example of data indicating that a product is not included is "product location". of The problem is "there is no data to indicate." On the other hand, positive data that includes an image of a product includes the image and data indicating the position of the product within that image. It is preferable that the learning data storage unit 170 also includes this positive data.

[0051] The model generation unit 150 generates or updates a model using the training data stored in the training data storage unit 170. An example of the processing performed here is as follows:

[0052] First, the model generation unit 150 performs object detection processing on the training data using the current model. This process generates values ​​indicating object-likeness for each part of each image. The model generation unit 150 then adjusts the model parameters so that the values ​​indicating object-likeness increase in areas where objects to be detected exist (positive). On the other hand, the model generation unit 150 adjusts the model parameters so that the values ​​indicating object-likeness decrease in areas where objects to be detected do not exist (negative).

[0053] In this process, the model generation unit 150 may use only the negative data described above as training data, or it may include positive data including products. In the negative data, the explanatory variables are at least one of the image and the processed image data (for example, the result of object detection by the image processing unit 130), and the target variable is 0. On the other hand, in the positive data, the explanatory variables are at least one of the image and the processed image data, and the target variable is the number of products included in the image. The model generation unit 150 stores the generated or updated model in the training data storage unit 170.

[0054] The communication unit 160 transmits the model generated or updated by the model generation unit 150 to the product count confirmation device 20.

[0055] Figure 6 illustrates an example of the target area for object detection processing by the image processing unit 130. Processing by the image processing unit 130 may be performed on a predetermined portion of the image to be processed. For example, in the process in which the product counting device 20 calculates a first number, product detection processing may be performed only on a specific portion of the image generated by the imaging device 30. In this case, the image processing unit 130 may perform object detection processing only on this specific portion.

[0056] The predetermined portion is set to exclude the product display area 510 and include the auxiliary equipment 60. For example, the predetermined portion includes the area where the products are placed when the auxiliary equipment 60, acting as a code reader, reads the product codes. Furthermore, this predetermined portion may also be set to include an object 70.

[0057] There are two ways to specify a predetermined area: specifying this area, or specifying an area other than this predetermined area.

[0058] Figure 7 shows an example of training data stored in the training data storage unit 170. In the example shown in this figure, the training data storage unit 170 stores each of multiple images, associating it with the date and time the image was taken and data indicating the result of object detection by the image processing unit 130.

[0059] Figure 8 shows an example of the functional configuration of the product count verification device 20. The product count verification device 20 can utilize a model storage unit 260. The model storage unit 260 stores the model generated by the model generation unit 150 of the training data generation device 10. The model storage unit 260 may be located outside the product count verification device 20 or may be part of the product count verification device 20.

[0060] The product count confirmation device 20 includes a calculation unit 210, an output unit 220, a registration processing unit 230, a settlement processing unit 240, and a communication unit 250.

[0061] The communication unit 250 communicates with the training data generation device 10 to acquire a model. The communication unit 250 then stores the acquired model in the model storage unit 260.

[0062] The registration processing unit 230 performs registration processing for the products subject to settlement, i.e., target products, using, for example, the reading results from the auxiliary device 60. For example, the auxiliary device 60 reads a code assigned to the target product. This code contains product identification information. The registration processing unit 230 performs registration processing for the target product using the code read by the auxiliary device 60. The registration information generated as a result of the registration processing includes at least one piece of product identification information.

[0063] The settlement processing unit 240 calculates the amount the customer must pay using the registration information generated by the registration processing unit 230 and the information stored in the product information storage unit 40. The settlement processing unit 240 then performs the settlement process for this amount. At this time, the settlement processing unit 240 uses the results of reading the card or mobile terminal by the auxiliary device 60 to identify the payment method. The payment method used here is electronic payment, and is at least one of the following: payment using a credit card, payment using electronic money, and payment using a 2D code.

[0064] The calculation unit 210 calculates a first number by processing images generated by the camera 30 when the customer uses the product quantity confirmation device 20, using the model stored in the model storage unit 260. The images used here may be a video showing the customer's actions during the product registration process, i.e., multiple frame images. The calculated first number corresponds to the number of products the customer intends to purchase.

[0065] The images processed by the calculation unit 210 include, for example, images generated while the registration processing unit 230 is operating. At the time these images are generated, the target product may be placed on the stand 50 or held by a customer. The images processed by the calculation unit 210 may also include images generated within a predetermined time period before the registration processing unit 230 begins operation. This predetermined time period is selected from, for example, a range of 1 second to 30 seconds, preferably a range of 5 seconds to 15 seconds.

[0066] The output unit 220 uses the registration information generated by the registration processing unit 230 to calculate the number of target products registered in the product count confirmation device 20, i.e., the second number. The output unit 220 then outputs predetermined information when the difference between the second number and the first number meets a criterion. This criterion number may be 1 or 2 or more. The predetermined information output by the output unit 220 indicates the possibility that the product registration result for the product count confirmation device 20 is incorrect, and is displayed, for example, on the display of the product count confirmation device 20. At this time, the output unit 220 may also output predetermined sound from the speaker of the product count confirmation device 20. A customer operating the product count confirmation device 20 can recognize that there is a possibility that the product registration result for the product count confirmation device 20 is incorrect by recognizing the output from the output unit 220.

[0067] Furthermore, if the product count verification device 20 is used as a separate device from the POS terminal, such as a cloud server, the product count verification device 20 does not have a registration processing unit 230 and a settlement processing unit 240. In this case, a terminal having a registration processing unit 230 and a settlement processing unit 240, such as a POS terminal, is placed separately from the product count verification device 20, for example, in the position of the product count verification device 20 in Figure 5. The output unit 220 of the product count verification device 20 transmits information to this terminal indicating whether the difference between the second number and the first number meets the criteria. When the difference between the second number and the first number meets the criteria, this terminal displays predetermined information on its display or outputs it from its speaker.

[0068] Figure 9 shows an example of the hardware configuration of the training data generation device 10. The training data generation device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0069] Bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060 to send and receive data to and from each other. However, the method of connecting the processor 1020 and the other components to each other is not limited to bus connection.

[0070] The 1020 processor is a processor implemented in components such as the CPU (Central Processing Unit) and GPU (Graphics Processing Unit).

[0071] Memory 1030 is a main memory device implemented using RAM (Random Access Memory), etc.

[0072] The storage device 1040 is an auxiliary storage device implemented as a removable media such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or memory card, or as ROM (Read Only Memory), and has a recording medium. The recording medium of the storage device 1040 stores program modules that realize each function of the learning data generation device 10 (for example, the acquisition unit 110, the selection unit 120, the image processing unit 130, the storage processing unit 140, the model generation unit 150, and the communication unit 160). The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to that program module. The storage device 1040 may also function as a learning data storage unit 170.

[0073] The input / output interface 1050 is an interface for connecting the learning data generation device 10 with various input / output devices.

[0074] The network interface 1060 is an interface for connecting the learning data generation device 10 to a network. This network may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may connect to the network via a wireless connection or a wired connection. The learning data generation device 10 may communicate with the product count confirmation device 20 and the imaging device 30 via the network interface 1060.

[0075] The hardware configuration of the product count verification device 20 is the same as that of the learning data generation device 10 shown in Figure 9. The recording medium of the storage device 1040 stores program modules that implement each function of the product count verification device 20 (for example, the calculation unit 210, the output unit 220, the registration processing unit 230, the settlement processing unit 240, and the communication unit 250). The storage device 1040 may also function as a model storage unit 260.

[0076] Figure 10 is a flowchart showing an example of the processing performed by the training data generation device 10.

[0077] First, the acquisition unit 110 acquires multiple images generated by the imaging device 30 (step S10). Next, the selection unit 120 selects images from the images acquired in step S10 to be used as training data. The images selected here include at least images that should be used as negative data (step S20). Then, the image processing unit 130 and the storage processing unit 140 perform processing to generate or add training data (step S30).

[0078] Subsequently, the model generation unit 150 of the learning data generation device 10 generates or updates a model at a predetermined timing. The communication unit 160 transmits the generated or updated model to the product count confirmation device 20. The timing at which the communication unit 160 transmits the model to the product count confirmation device 20 is, for example, when the product count confirmation device 20 is not performing product registration processing.

[0079] Figure 11 is a flowchart illustrating an example of the process performed by the product count verification device 20. The customer brings the products they wish to purchase to the product count verification device 20, and then operates the auxiliary device 60 of the product count verification device 20 to have the auxiliary device 60 read the product codes.

[0080] The registration processing unit 230 of the product count confirmation device 20 uses the reading results from the attached device 60 to register the products to be settled and generates registration information (step S110). Once this registration process is completed (step S120), the calculation unit 210 of the product count confirmation device 20 processes the images generated by the imaging device 30 during the registration process described above and calculates the first number (step S130).

[0081] The output unit 220 then uses the registration information generated by the registration processing unit 230 to identify the second number. When the difference between the second number and the first number meets the criteria, i.e., when there is a possibility that the registration information is incorrect (step S140: Yes), the output unit 220 outputs predetermined information (step S150) and returns to step S110.

[0082] On the other hand, when the difference between the second number and the first number does not meet the criteria, that is, when the registration information is presumed to be correct (step S140: No), the settlement processing unit 240 performs the settlement process (step S160).

[0083] In step S150, the output unit 220 may further output predetermined information to a terminal operated by the seller of the product, for example, a terminal operated by a store employee.

[0084] As described above, according to this embodiment, the product count confirmation device 20 uses a model when processing images to calculate a first number indicating the number of products. At least a portion of the training data used to generate this model is generated by the training data generation device 10. The training data generation device 10 includes images that satisfy predetermined conditions as negative data that do not contain the target product in the training data. Therefore, the calculation of the first number using this model accuracy It will get more expensive.

[0085] The product count verification device 20 then outputs a predetermined message if the difference between the first number and the second number, which is the number of target products registered in the product count verification device 20, does not meet the criteria. Therefore, the administrator or customer of the product count verification device 20 can recognize that there is a possibility that the products have not been registered accurately.

[0086] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and various other configurations can also be adopted.

[0087] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content. Also, the above embodiments can be combined to the extent that their contents do not conflict.

[0088] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. A training data generation device that generates at least a portion of the training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned training data generation device is An acquisition means for acquiring multiple images generated by the aforementioned shooting means, A selection means for selecting images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; A training data generation device equipped with [the following features]. 2. In the learning data generation device described in item 1 above, A training data generation device wherein at least one of the predetermined conditions is that the difference between the image and the reference image satisfies the first criterion. 3. In the learning data generation device described in 1 or 2 above, A training data generation device wherein at least one of the predetermined conditions is that the state of the product registration device that performs the registration process of the target product at the timing of image generation satisfies the second criterion. 4. In the learning data generation device described in item 3 above, A learning data generation device wherein at least one of the second criteria is that the product registration device does not perform the registration process. 5. In the learning data generation device described in 3 or 4 above, A learning data generation device, wherein at least one of the second criteria is that the product registration device is in standby mode and powered off. 6. In the learning data generation device described in any one of items 1 to 5 above, A training data generation device wherein at least one of the aforementioned predetermined conditions is that the timing of image generation is a predetermined date and time. 7. In the learning data generation device described in any one of items 1 to 6 above, A training data generation device wherein at least one of the predetermined conditions is that the image contains a specific object. 8. In the learning data generation device described in item 7 above, The aforementioned specific object indicates that the product registration device that performs the registration process for the target product cannot be used, and is a training data generation device. 9. In the learning data generation device described in any one of items 1 to 8 above, A training data generation device wherein at least one of the aforementioned predetermined conditions is that there are no people in a predetermined area at the time the image is generated. 10. In the learning data generation device described in item 9 above, The aforementioned predetermined area includes the aforementioned shooting range, and is a learning data generation device. 11. In the learning data generation device described in any one of items 1 to 10 above, Image processing means that performs object detection processing on the image which is the negative data, A storage processing means that stores the image from which the object detection process has been performed as at least part of the training data, linked to data indicating that the product is not included, in a storage means; A training data generation device equipped with the following features. 12. In the learning data generation device described in item 11 above, The image processing means is a training data generation device that performs the object detection process using an algorithm for calculating the number of target products. 13. In the learning data generation device described in item 11 or 12 above, The image processing means is a training data generation device that performs the object detection process on a predetermined portion of the image. 14. In the learning data generation device described in any one of items 1 to 13 above, A training data generation device comprising a model generation means for generating the model using the aforementioned training data. 15. Used in conjunction with the training data generation device described in any one of items 1 to 14 above. A calculation means calculates a first number, which is the number of target products, by processing the image captured by the shooting means when a customer uses a product registration device to perform the registration process of the target products, using the model generated using the training data, An output means that outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number satisfies a criterion, A product count verification device equipped with the following features. 16. A computer method for generating training data to generate at least a portion of the training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned computer, The aforementioned imaging means acquires a plurality of images generated by the imaging means, A method for generating training data, comprising selecting from the aforementioned multiple images images that satisfy predetermined conditions, in order to include them as negative data that does not include the target product, as at least part of the training data. 17. In the training data generation method described in item 16 above, A method for generating training data, wherein at least one of the predetermined conditions is that the difference between the image and the reference image satisfies the first criterion. 18. In the training data generation method described in 16 or 17 above, A method for generating training data, wherein at least one of the predetermined conditions is that the state of the product registration device that performs the registration process of the target product at the timing of image generation satisfies the second criterion. 19. In the training data generation method described in item 18 above, A method for generating training data, wherein at least one of the second criteria is that the product registration device does not perform the registration process. 20. In the training data generation method described in 18 or 19 above, A method for generating learning data, wherein at least one of the second criteria is that the product registration device is in standby mode and powered off. 21. In the training data generation method described in any one of items 16 to 20 above, A method for generating training data, wherein at least one of the aforementioned predetermined conditions is that the timing of image generation is a predetermined date and time. 22. In the training data generation method described in any one of items 16 to 21 above, A method for generating training data, wherein at least one of the predetermined conditions is that the image contains a specific object. 23. In the training data generation method described in 22 above, A method for generating training data, wherein the aforementioned specific object indicates that the product registration device that performs the registration process for the target product cannot be used. 24. In the training data generation method described in any one of items 16 to 23 above, A method for generating training data, wherein at least one of the aforementioned predetermined conditions is that there are no people in a predetermined area at the time the image is generated. 25. In the training data generation method described in 24 above, The predetermined area includes the shooting range, and is a method for generating training data. 26. In the training data generation method described in any one of items 16 to 25 above, The aforementioned computer, Object detection processing is performed on the image which is the negative data. A method for generating training data, comprising: associating the image that has undergone the object detection process with data indicating that it does not contain a product, and storing it in a storage means as at least part of the training data. 27. In the training data generation method described in 26 above, The computer is a method for generating training data to perform the object detection process using an algorithm for calculating the number of target products. 28. In the training data generation method described in 26 or 27 above, The computer is a method for generating training data that performs the object detection process on a predetermined portion of the image. 29. In the training data generation method described in any one of items 16 to 28 above, The computer generates the model using the training data, and the method for generating training data. 30. A computer used in conjunction with the learning data generation device described in any one of items 1 to 14 above, Using the model generated with the aforementioned training data, the image captured by the photographing means when a customer uses the product registration device to perform the registration process for the target products is processed to calculate a first number, which is the number of target products. A method for verifying the number of products, which outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number, satisfies a criterion. 31. A recording medium that stores a program causing a computer to generate at least a portion of the training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The program is installed on the computer. An acquisition function that acquires multiple images generated by the aforementioned shooting means, A selection function that selects images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; A recording medium that provides a specific function. 32. In the recording medium described in 31 above, A recording medium wherein at least one of the aforementioned predetermined conditions is that the difference between the image and the reference image satisfies the first criterion. 33. In the recording medium described in 31 or 32 above, A recording medium wherein at least one of the aforementioned predetermined conditions is that the state of the product registration device satisfies the second criterion at the timing of image generation. 34. In the recording medium described in 33 above, A recording medium wherein at least one of the second criteria is that the product registration device does not perform the registration process. 35. In the recording medium described in 33 or 34 above, A recording medium wherein at least one of the second criteria is that the product registration device is in standby mode and powered off. 36. In the recording medium described in any one of the above paragraphs 31 to 35, A recording medium wherein at least one of the aforementioned predetermined conditions is that the timing of image generation is a predetermined date and time. 37. In the recording medium described in any one of the above paragraphs 31 to 36, A recording medium wherein at least one of the aforementioned predetermined conditions is that the image contains a specific object. 38. In the recording medium described in 37 above, The aforementioned specific object indicates that the product registration device performing the registration process for the target product cannot be used, which is a recording medium. 39. In the recording medium described in any one of the above paragraphs 31 to 38, A recording medium wherein at least one of the aforementioned predetermined conditions is that there are no people in a predetermined area at the time the image is generated. 40. In the recording medium described in item 39 above, The aforementioned predetermined area includes the aforementioned shooting range and is a recording medium. 41. In the recording medium described in any one of items 31 to 40 above, The program is installed on the computer. An image processing function that performs object detection processing on the image which is the negative data, A memory processing function that stores the image from which the object detection process has been performed in a memory means as at least part of the training data, linked to data indicating that the product is not included. A recording medium that provides a storage medium. 42. In the recording medium described in 41 above, The image processing function is a recording medium that performs the object detection process using an algorithm for calculating the number of target products. 43. In the recording medium described in 41 or 42 above, The aforementioned image processing function is a recording medium that performs the object detection process on a predetermined portion of the image. 44. In any of the recording media described in items 31 to 43 above, The program is a recording medium that provides the computer with a model generation function that generates the model using the training data. 45. A recording medium on which a program used in a computer used together with the learning data generation device described in any one of items 1 to 14 above is recorded. The program is installed on the computer. A calculation function that calculates a first number, which is the number of target products, by processing the image captured by the shooting means when a customer uses the product registration device to perform the registration process of the target products, using the model generated using the training data, An output function that outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number satisfies a criterion, A recording medium that provides a specific function. 46. ​​A program described in any one of the above items 41 to 44. 47. The program described in item 45 above. [Explanation of symbols]

[0089] 10. Training data generation device 20. Product Quantity Verification Device 30 Imaging device 40 Product information storage section 50 units 60 Accessories 70 Object 110 Acquisition Department 120 Selection Section 130 Image Processing Unit 140 Memory Processing Unit 150 Model Generation Unit 160 Communications Department 170 Data storage for training Department 210 Calculation Unit 220 Output section 230 Registration Processing Unit 240 Settlement Processing Unit 250 Communications Department 260 Model Memory Unit

Claims

1. A training data generation device that generates at least a portion of the training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned training data generation device is An acquisition means for acquiring multiple images generated by the aforementioned shooting means, A selection means for selecting images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; Equipped with, The aforementioned predetermined conditions indicate a high probability that the product registration device that performs the registration process for the target product is not being used. A device for generating training data.

2. In the learning data generation device according to claim 1, At least one of the aforementioned predetermined conditions is that the difference between the image and the reference image is less than or equal to the standard. The aforementioned reference image includes a pre-set background image or a past image taken prior to the said image. A device for generating training data.

3. In the learning data generation device according to claim 1 or 2, A training data generation device wherein at least one of the aforementioned predetermined conditions is that the product registration device is not in use at the time the image is generated.

4. In the learning data generation device according to any one of claims 1 to 3, A training data generation device wherein at least one of the aforementioned predetermined conditions is that the timing of image generation is a predetermined date and time.

5. In the learning data generation device according to any one of claims 1 to 4, A training data generation device wherein at least one of the predetermined conditions is that the image contains a specific object.

6. Used in conjunction with the learning data generation device described in any one of claims 1 to 5, A calculation means calculates a first number, which is the number of target products, by processing the image captured by the shooting means when a customer uses a product registration device to perform the registration process of the target products, using the model generated using the training data, An output means that outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number exceeds a predetermined threshold, A product count verification device equipped with the following features.

7. A computer method for generating training data, which generates at least a portion of the training data for generating a model, The aforementioned model calculates the number of target products included in an image generated by a photographic means whose shooting range includes the area where the target products subject to settlement are placed, The aforementioned computer, The aforementioned imaging means acquires a plurality of images generated by the imaging means, From among the aforementioned multiple images, images that meet predetermined conditions are selected to be included as negative data that does not include the target product, in at least part of the training data. The aforementioned predetermined conditions indicate a high probability that the product registration device that performs the registration process for the target product is not being used. Method for generating training data.

8. A computer used in conjunction with the learning data generation device described in any one of claims 1 to 5, Using the model generated with the aforementioned training data, the image captured by the photographing means when a customer uses the product registration device to perform the registration process for the target products is processed to calculate a first number, which is the number of target products. A method for verifying the number of products, which outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number exceeds a predetermined threshold.

9. A program that causes a computer to generate at least a portion of training data for generating a model for calculating the number of target products included in an image generated by a photographic means whose shooting range includes an area where the target products subject to settlement are placed, To the aforementioned computer, An acquisition function that acquires multiple images generated by the aforementioned shooting means, A selection function that selects images from among the aforementioned multiple images that satisfy predetermined conditions, to include as negative data that does not include the target product, in at least part of the training data; Give it to him The aforementioned predetermined conditions indicate a high probability that the product registration device that performs the registration process for the target product is not being used. program.

10. A program for use in a computer used in conjunction with a learning data generation device according to any one of claims 1 to 5, To the aforementioned computer, A calculation function that calculates a first number, which is the number of target products, by processing the image captured by the shooting means when a customer uses the product registration device to perform the registration process of the target products, using the model generated using the training data, An output function that outputs predetermined information when the difference between a second number, which is the number of target products registered in the product registration device, and the first number exceeds a predetermined threshold, A program to give it a specific feature.

Images