Product Management System

Abstract

To perform a product management system that differentiates loading from unloading while reducing installation expenses and installation space for product management using an RFID.SOLUTION: The product management system includes: a gate section that has an RFID reader function that reads the identification information of a product from an RF tag attached to the product; a photographing section that photographs around the gate part; and a management section that manages a product passing the gate section based on the data obtained by photographing with the photographing section and the identification information read by the gate section. The management section includes: an image processing unit that based on the image data obtained by photographing with the photographing section, performs a transportation direction determination process for determining a transportation direction that indicates whether the transported product has been loaded or unloaded and a quantity determination process for determining the number of transported products; and an excess and shortage determination unit that compares the quantity of the identification information read by the gate section with the number of transported products determined by the image processing unit and determines the excess or shortage of the transported products for each transportation direction.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a product management system. To Regarding. [Background technology]

[0002] In recent years, product management using RFID (Radio-Frequency Identification) has become more common. RFID is a technology that uses a dedicated reader to read RF tags attached to products without contact. Product management using RFID is used, for example, to manage the entry and exit of products into a warehouse (in-stock / out-stock management). In-stock / out-stock management is carried out, for example, by installing a gate with an RF reader function through which products pass, and reading the RF tags of products as they pass through the gate.

[0003] Technologies relating to product management using RFID are disclosed in the following prior art documents. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-108777 A [Patent Document 2] JP 2006-103812 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, if you try to use RFID to manage incoming and outgoing products separately, you need to install both an incoming gate and an outgoing gate. Because the gates allow multiple products to pass through, they are fairly large, and installing multiple gates can require a large space. In addition, gates are expensive because they are large and have RF reader functions, so installing multiple gates increases costs.

[0006] Therefore, in product management using RFID, a product management system, a product management method, and a product management program are provided that can perform product management by distinguishing between incoming and outgoing products while reducing installation costs and space. [Means for solving the problem]

[0007] According to one embodiment of the present invention, there is provided a product management system having a gate section with an RFID reader function that reads product identification information from an RF tag attached to a product, an imaging section that images the area around the gate section, and a management section that manages products passing through the gate section based on the data captured by the imaging section and the identification information read by the gate section, wherein the management section has an image processing section that executes a conveying direction determination process that determines the conveying direction indicating whether the product being conveyed is entering or leaving the warehouse based on image data captured by the imaging section, and a number determination process that determines the number of the products being conveyed, and an excess / shortage determination section that compares the number of pieces of identification information read by the gate section with the number of the products being conveyed determined by the image processing section, and determines whether there is a surplus or shortage of the products being conveyed for each of the conveying directions. Effect of the Invention

[0008] According to one embodiment of the present invention described above, it is possible to perform product management that distinguishes between incoming and outgoing products while suppressing installation costs and installation space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a product management system 1. [Diagram 2] FIG. 2 is a diagram illustrating an example of the software configuration of the product management system 1. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the product management system 1. [Figure 4] FIG. 4 is a diagram illustrating an example of a sequence of a product management process. [Diagram 5] FIG. 5 is a diagram showing an example of a process flowchart of the product management process S100. [Figure 6] FIG. 6 is a diagram illustrating an example of a processing flowchart of the excess / deficiency determination process S30. [Figure 7] FIG. 7 is a diagram illustrating an example of a process flowchart of the surplus / deficiency determination process S60 in the second embodiment. [Figure 8] FIG. 8 is a diagram showing an example of the incoming / outgoing list and the determination result. [Figure 9] FIG. 9 is a diagram showing an example of a process flowchart of the product management process S200 in the third embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a processing flowchart of the image analysis learning model pre-learning process S300. [Figure 11] FIG. 11 is a diagram showing an example of the processed image data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a mode for carrying out the invention will be described.

[0011] <Product management system 1 configuration example> 1 is a diagram showing an example of the configuration of a product management system 1. The product management system 1 has a management device 100, a gate 200, and a camera 300. The product management system 1 is a system that manages the entry and exit of products passing through the gate 200. A product is determined to have been exited when it is transported from right to left (D1) through the gate 200, and determined to have been entered when it is transported in the opposite direction (D2).

[0012] The products T1 to T3 are packed in, for example, cardboard boxes, and RF tags R1 to R3 are attached to the outer surfaces of the packages, respectively.

[0013] The RF tag has an IC (Integrated Circuit) chip that stores information about the product. The information about the product includes, for example, a product identifier. The RF tag is, for example, a passive tag. A passive tag is a tag that does not have a battery and transmits radio waves to an RFID reader using radio waves transmitted from an RFID reader as a power source. The RF tag may also be an active tab or semi-active tab that transmits radio waves using its own battery as a power source.

[0014] Gate 200 is a device that has an RFID reader function and reads the RF tags of products that pass through the gate. Gate 200, for example, transmits radio waves to read the RF tags and receives radio waves returned from the RF tags to recognize that a product is present in the vicinity (within a specified distance). Note that gate 200 transmits radio waves at any time, at specified intervals, or when instructed by an administrator or another device.

[0015] Camera 300 is a photographing device that photographs images of gate 200 and a predetermined range around it. The images may be, for example, videos, still images, or both. Camera 300 photographs continuously, at predetermined intervals, or at a timing instructed by an administrator or another device. The camera may also be a camera with a motion function (a function capable of determining the direction of movement of a moving object).

[0016] The management device 100 is a device that manages products transported (entering and leaving the warehouse) through the gate 200, and is, for example, a computer or a server machine. The management device 100 acquires image data of a video captured by a camera 300. The management device 100 performs image analysis on the image data, and determines the number of products, the product transport direction (entering or leaving the warehouse), and the like.

[0017] The management device 100 also acquires the RFID that is read by the gate 200 during a series of processes or within a predetermined time period, and manages the entry and exit of products using the determination results of the image analysis and the RFID read results.

[0018] The management device 100, the gate 200, and the camera 300 are connected and communicate with each other via, for example, a network or a cable. The management device 100 communicates with the gate 200 and the camera 300 to obtain image data and RFID read results.

[0019] In FIG. 1, the management device 100, the gate 200, and the camera 300 are different devices, but all or any two of them may be the same device.

[0020] <Example of software configuration for product management system> FIG. 2 is a diagram illustrating an example of the software configuration of the product management system 1.

[0021] The camera 300 includes an image capturing unit 30. The image capturing unit 30 captures an image within a predetermined range, and stores the captured image data in an internal memory or transfers the image data to the image processing unit 10 of the management device 100.

[0022] The gate (gate unit) 200 has an RFID reading unit 20. The RFID reading unit 20 transmits radio waves for reading an RF tag in response to a read trigger, and receives radio waves returned from the RF tag in response to the transmitted radio waves. The RFID reading unit 20 then passes data, including information on the RFIDs whose RF tags have been successfully read (e.g., the number of RFIDs read and information on the product), contained in the received radio waves, to the product management unit 14 of the management device 100.

[0023] The management device (management unit) 100 has an image processing unit 10 and a product management unit 14. The image processing unit 10 acquires image data captured by a camera 300 and performs image analysis. The product management unit 14 manages the entry and exit of products based on the image analysis and the results of reading the RFID.

[0024] The image processing unit 10 includes an image analysis learning model learning unit 11, a number determination unit 12, and a conveying direction determination unit 13.

[0025] The image analysis learning model learning unit 11 learns an image analysis learning model that performs image analysis. The image analysis model is, for example, an AI (Artificial Intelligence) for image analysis. The image analysis learning model learning unit 11 improves the judgment accuracy of the learned image analysis learning model during operation of the product management system 1, or performs pre-learning of the image analysis learning model before operation of the product management system 1.

[0026] The number determination unit 12 uses a learning model for image analysis to determine the number of products shown in the image data. The number determination unit 12 passes the result of the number determination to the product management unit 14.

[0027] The conveying direction determination unit 13 determines the conveying direction (incoming or outgoing) of the product. The conveying direction determination unit 13 passes the determined conveying direction to the product management unit 14.

[0028] The product management unit 14 includes an incoming / outgoing inventory list acquisition unit 15 , a surplus / shortage determination unit 16 , and a learning data creation unit 17 .

[0029] The incoming / outgoing list acquiring unit 15 acquires an incoming / outgoing list that lists information about incoming and outgoing products. The incoming / outgoing list is acquired from a core system connected to the management device 100, for example.

[0030] The surplus / shortage determination unit 16 determines whether there is a surplus or shortage of products to be transported during storage and delivery. The surplus / shortage determination unit 16 determines whether there is a surplus or shortage based on at least the number of products determined by the quantity determination unit 12 and the number of products whose RFIDs have been successfully read by the RFID reading unit 20.

[0031] The learning data creation unit 17 creates teacher data (learning data) for training an image analysis learning model from image data captured during operation of the product management system 1 and the results of the excess / deficiency determination.

[0032] Each unit may be configured, for example, by dedicated hardware, or may be constructed by a processor executing a loaded program.

[0033] <Example of hardware configuration for product management system> 3 is a diagram illustrating an example of the configuration of the product management system 1. The product management system 1 includes a CPU (Central Processing Unit) 110, a storage 120, a memory 130, a camera 300, a gate 200, and a display 160.

[0034] The storage 120 is an auxiliary storage device such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD) that stores programs and data. The storage 120 stores an RFID reading program 121, a photography program 122, an image processing program 123, a product management program 124, and an image analysis learning model learning program 125.

[0035] The memory 130 is an area into which the programs stored in the storage 120 are loaded. The memory 130 may also be used as an area in which the programs store data.

[0036] The CPU 110 is a processor that loads a program stored in the storage 120 into the memory 130, executes the loaded program, configures each unit, and executes each process.

[0037] Camera 300 is a device that captures images of products around gate 200. Camera 300 may be, for example, connected to the outside of a computer that constitutes product management system 1, or may be built into the computer.

[0038] Gate 200 has an RFID reader function and is a device that uses the RFID reader function to receive radio waves transmitted (returned) by an RF tag attached to a product and acquires RF tag information (product information). Gate 200 may be, for example, connected to the outside of a computer constituting product management system 1, or may be built into the computer. Gate 200 may be, for example, a gate type, and reads RFID from the RF tag of a product located inside the gate or within a predetermined distance from gate 200.

[0039] The display 160 is a display unit that displays images and videos. The display 160 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display 160 displays, for example, the result of the excess / shortage determination.

[0040] The CPU 110 executes the RFID reading program 121 to configure the RFID reading unit 20 and perform RFID reading processing. The RFID reading processing is processing for using the RFID reading function of the gate 200 to read product information from the RF tag.

[0041] The CPU 110 executes the photographing program 122 to configure the photographing unit 30 and perform photographing processing. The photographing processing is processing for photographing an image including the periphery of the gate 200 using the camera 300.

[0042] The CPU 110 executes the image processing program 123 to construct the image processing unit 10 and perform image processing. The image processing is a process of analyzing image data and determining the number of products shown in the image data and the carrying direction of the products (warehouse entry or delivery).

[0043] The CPU 110 executes the product management program 124 to construct the product management unit 14 and perform product management processing. The product management processing is processing for determining whether a product is insufficient or insufficient when the product is received or shipped out.

[0044] The CPU 110 executes the image analysis learning model learning program 125 to construct the image analysis learning model learning unit 11 and the learning data creation unit 17, and performs the image analysis learning model learning process. The image analysis learning model learning process performs processes related to the learning of the image analysis learning model, such as selecting teacher data to be learned by the image analysis learning model and inputting the teacher data. Note that the image analysis learning model learning process may include pre-learning of the image analysis learning model before the operation of the product management system 1.

[0045] <Product management processing sequence> 4 is a diagram showing an example of a sequence of a product management process. The process of each unit will be described below.

[0046] The photographing unit 30 photographs the area around the gate 200 (including the products) and transmits the photographed image data to the image processing unit 10 (S10).

[0047] The image processing unit 10 uses the image analysis learning model 50 (S11), executes the number determination process S12, and transmits the execution result to the product management unit 14 (S13). The image analysis learning model 50 is, for example, AI (Artificial Intelligence).

[0048] The number determination process S12 is a process for determining the number of products shown in the image data. The execution result of the number determination process S12 includes the number of products (hereinafter, may be referred to as the number of images). The number determination process S12 determines the number of products that are moving (conveyed). Products that are not moving (products that are staying in the same position) are not considered to be products that are being transported, and are not counted in the number of products to be transported.

[0049] The image processing unit 10 also acquires movement direction information relating to the direction in which the product moves from the motion function-equipped camera (S14), executes a conveying direction determination process S15, and transmits the execution result to the product management unit 14 (S16).

[0050] The conveying direction determination process S15 is a process for determining the conveying direction of the product (whether it is entering or leaving the warehouse) based on the movement direction information. The execution result of the conveying direction determination process S15 includes the fact that it is entering or leaving the warehouse (hereinafter, it may be referred to as the image conveying direction).

[0051] The RFID reader 20 reads the RFIDs and transmits the read results to the product manager 14 (S20). The read results include the number of RFIDs that were successfully read (hereinafter, may be referred to as the number of reads) and product information contained in the read RFIDs.

[0052] When the product management unit 14 receives the number of images, the number of images to be read, and the image transport direction, it performs an excess / shortage determination process S30, and then performs a learning data creation process S40.

[0053] The excess / shortage determination process S30 is a process for determining whether or not there is an excess or shortage of the products being transported for each incoming and outgoing product based on the number of images, the number of readouts, and the image transport direction. The excess / shortage determination process S30 may also include a process for displaying the result of the excess / shortage determination on the display 160.

[0054] The learning data creation process S40 is a process that uses image data used by the image analysis learning model 50 for image analysis and the correct or incorrect results of the image analysis of the image data as teacher data (creates teacher data). The learning data creation process S40 transmits the teacher data to the image analysis learning model learning process S42 (S41). Note that the teacher data created in the learning data creation process S40 is related to the determination of the number of images. When the movement direction is determined by a camera with a motion function, teacher data related to the transport direction determination is not created in the learning data creation process S40.

[0055] The image analysis learning model learning process S42 is a process in which teacher data is input, the image analysis learning model 50 is trained, and the image analysis learning model 50 is updated (S43). The learning may be performed, for example, while the product management system 1 is in operation, or while the product management system 1 is stopped (at night, during system maintenance, etc.). The product management system 1 uses image data that has been subjected to image analysis and repeatedly trains the image analysis learning model 50, thereby improving the accuracy of the image analysis (the proportion of times that the execution result of the number determination process S12 is correct).

[0056] When an image is captured by a camera without a motion function, for example, the image analysis learning model 50 may be used to determine the direction of movement of the product and whether it is entering or leaving the warehouse. In this case, learning data may also be generated for the conveying direction determination process S15 and may be learned.

[0057] Furthermore, learning of the image analysis learning model 50 during system operation does not have to be performed. Examples of not learning during operation include cases where sufficient advance learning has been performed and the accuracy of the AI ​​is quite high, and cases where learning during operation is difficult due to processing load or communication speed. Also, the system does not necessarily have to be configured to have a learning function during operation.

[0058] [First embodiment] A first embodiment will be described. In the first embodiment, video data is used as the captured data. The management device 100 detects the movement of the product based on the video data, and executes the product management process when the detection of the product movement is triggered.

[0059] FIG. 5 is a diagram showing an example of a process flowchart of the product management process S100 in the first embodiment.

[0060] The management device 100 analyzes the video capture data and waits to detect the movement of the products (that the products are about to be carried in or out) (No in S100-1). When the management device 100 detects that the products have started to move (Yes in S100-1), it executes the number determination process S12.

[0061] In the number determination process S12, the management device 100 determines the number of product images from a still image at a certain moment in the video data. The number determination process S12 is, for example, a process of determining the number of images based on a plurality of still images at a plurality of timings. The number determination process S12 may also be a process of determining the number of images using video data captured from a plurality of directions using a plurality of cameras 300. Furthermore, if the camera 300 has a depth measurement function, the number determination process S12 may include a process of using the results of the depth measurement to determine the number of images. The depth measurement function is a function that detects the distance to an object, and is, for example, a function that makes it possible to determine whether an area surrounded by a plurality of cardboard boxes is a space or cardboard boxes.

[0062] The management device 100 executes a conveying direction determination process S15. In the conveying direction determination process S15, the management device 100 determines whether the conveying direction of the product is entering or leaving the warehouse, based on information on the moving direction acquired from the camera. The conveying direction determination process S15 is a process for determining whether the product is entering or leaving the warehouse, based on the moving direction acquired by the motion function of the camera.

[0063] The management device 100 waits for the completion of reading the RFIDs (No in S100-2). For example, reading of the RFIDs is continued for a predetermined time after one RFID is read. Reading of the RFIDs is continued for a predetermined time because, for example, depending on the size of the product itself and the radio wave conditions around the gate 200, it may take a considerable amount of time to successfully read the RI tags of multiple products.

[0064] When the management device 100 has completed reading the RFID (Yes in S100-2), it executes the surplus / deficiency determination process S30. The surplus / deficiency determination process S30 will be described in detail later.

[0065] If there is a shortage or surplus of products in the shortage / surplus determination process S30 (Yes in S100-3), the management device 100 checks the products and takes measures to address the shortage or surplus (S100-4). The process S100-4 includes, for example, displaying on the display 160 or the like that there is a mismatch between the number of images and the number of readouts, and the image conveying direction such as entering or leaving the warehouse, and prompting the manager to check the shortage or surplus of products or take measures.

[0066] The management device 100 creates training data from the image data and the excess / shortage judgment result, executes the image analysis learning model learning process S42, and waits for detection of the product movement again (S100-1). Note that the training data is created, for example, in the learning data creation process S40.

[0067] <Excess / Shortage Judgment Process> 6 is a diagram showing an example of a process flowchart of the excess / deficiency determination process S30. The management device 100 determines whether or not the number of images matches the number of images to be read (S30-1).

[0068] If the number of images and the number of reads match (Yes in S30-1), and if the image transport direction is outbound ("Outbound" in S30-2), the management device 100 determines that there is no surplus or shortage of products at outbound (S30-3), and if the image transport direction is inbound ("Inbound" in S30-2), it determines that there is no surplus or shortage of products at inbound (S30-4), and terminates the processing.

[0069] On the other hand, if the number of images and the number of reads do not match (No in S30-1), if the image transport direction is outbound ("Outbound" in S30-5), the management device 100 determines that there is a surplus or shortage of products at outbound (S30-7), and if the image transport direction is inbound ("Inbound" in S30-5), it determines that there is a surplus or shortage of products at inbound (S30-6), and terminates the processing.

[0070] In the first embodiment, when the management device 100 detects the movement (transport) of products from the video data, it determines the number of images and the image transport direction from the video data. Then, the management device 100 acquires the RFID read results, compares the read number with the number of images, and determines whether there is an excess or shortage of products for each image transport direction. As a result, by installing one gate 200, it is possible to determine the excess or shortage of products in both the incoming and outgoing directions, and to manage products. Furthermore, by having the management device 100 determine incoming and outgoing from images, it is possible to reduce, for example, the effort required for a manager to input incoming or outgoing and the risk of inputting incorrect information.

[0071] [Second embodiment] A second embodiment will be described. In the second embodiment, the management device 100 executes surplus / shortage determination process S60 instead of the surplus / shortage determination process S30 in the first embodiment. The surplus / shortage determination process S60 is a process in which an incoming / outgoing inventory list (product list) including a list of incoming and outgoing products is used for surplus / shortage determination. The incoming / outgoing inventory list includes information such as the conveying direction, identification information including the identifier of the product to be conveyed, and the number of products to be conveyed (quantity information).

[0072] 7 is a diagram showing an example of a processing flowchart of the surplus / shortage determination process S60 in the second embodiment. The management device 100 acquires an incoming / outgoing list (S60-1). The management device 100 acquires the incoming / outgoing list, for example, from a core system that creates and manages the incoming / outgoing list. The management device 100 may also acquire the incoming / outgoing list by input from an administrator, for example.

[0073] The management device 100 judges whether the conveying direction in the incoming / outgoing list matches the image conveying direction (S60-2). If the conveying direction in the incoming / outgoing list does not match the image conveying direction (No in S60-2), the management device 100 judges that there is a shortage or surplus of products (S60-5) and ends the process. Note that the judgment that there is a shortage or surplus of products includes the judgment that an error other than a shortage or surplus of products has occurred, such as an incorrect conveying direction or the moving of excess products.

[0074] On the other hand, if the conveying direction in the incoming / outgoing list matches the image conveying direction (Yes in S60-2), the management device 100 determines whether the number of images, the number of reads, and the number of conveyed items in the incoming / outgoing list all match (S60-3).

[0075] If the number of images, the number of read items, and the number of transported items in the incoming / outgoing list all match (Yes in S60-3), the management device 100 determines that there is no shortage or surplus of products (S60-4) and ends the process.

[0076] On the other hand, if there is a partial mismatch between the number of images, the number of read items, and the number of transported items in the incoming / outgoing list (No in S60-3), the management device 100 determines that there is a surplus or shortage of products (S60-5) and ends the process.

[0077] FIG. 8 is a diagram showing an example of an incoming / outgoing list and a determination result. FIG. 8(A) is a diagram showing an example of a case where it is determined that there is no surplus or shortage. The management device 100 determines that the conveying directions match because the conveying direction in the incoming / outgoing list L1 is incoming and the image conveying direction is also ingoing (Yes in S60-2 of Table 1 and FIG. 7). The management device 100 determines that there is no surplus or shortage because the number of conveyed items in the incoming / outgoing list L1 is 7, the number of images is 7, and the number of read items is 7, which are all the same numbers (Yes in S60-3 of Table 2 and FIG. 7) (S60-4 of FIG. 7).

[0078] Fig. 8(B) is a diagram showing an example of a case where it is determined that there is a surplus or shortage. The management device 100 determines that the conveying directions match because the conveying direction in the incoming / outgoing list L1 is incoming and the image conveying direction is also ingoing (Yes in S60-2 in Table 3 and Fig. 7). The management device 100 determines that there is a surplus or shortage (S60-5 in Fig. 7) because the number of conveyed items in the incoming / outgoing list L1 is 7, the number of images is 6, and the number of read items is 6, which do not all match (No in S60-3 in Table 4 and Fig. 7).

[0079] <Modification> In the surplus / shortage determination process S60, the management device 100 may use the identifier (ID) of the transported product in the incoming / outgoing list instead of the transported number in the incoming / outgoing list. In this case, the management device 100 checks whether the identifiers of the transported products in the incoming / outgoing list and the product identifiers in the RFID readout results all match (match without any surplus or deficiency). If the identifiers all match, the management device 100 then determines whether the number of images matches the transported number in the incoming / outgoing list (the same as the read number). If the identifiers and transported number all match, the management device 100 determines that there is no surplus or shortage of products, and if there is a mismatch even in part, it determines that there is a surplus or shortage of products.

[0080] Also, comparison with the incoming / outgoing list may be performed only at the time of outgoing. Since it is sometimes unclear when and how many products will be brought in for incoming products, an incoming / outgoing list may not be created for incoming products. On the other hand, since outgoing products are delivered to customers, for example, the timing and target products are clear and they are strictly managed, so they are often listed.

[0081] Furthermore, the excess / shortage determination process may be performed only at the time of delivery. This is because, as described above, stricter product management is required at the time of delivery. For example, in the conveying direction determination process S15 in the product management process S100 of FIG. 5, if the image conveying direction is entry, the management device 100 may not execute the excess / shortage determination process S30 (S60) or the number determination process S12.

[0082] [Third embodiment] A third embodiment will be described. In the third embodiment, still image data is used as the photographed data. When the management device 100 detects the reading of the RFID, it determines the conveying direction of the product based on still image data photographed at least at two timings, and executes product management.

[0083] FIG. 9 is a diagram showing an example of a process flowchart of the product management process S200 in the third embodiment.

[0084] The management device 100 waits to detect the reading of the RFID (No in S200-1). When the management device 100 detects the reading of the RFID (Yes in S200-1), it acquires the photographic data (still image data) photographed at multiple timings (S200-2).

[0085] The management device 100 executes a number determination process S12 based on the photographed data. In the number determination process S12, the management device 100 may determine the number of images from one image data among the multiple image data, or may comprehensively determine the number of images from the multiple image data.

[0086] The management device 100 executes a conveying direction determination process S15. In the conveying direction determination process S15, the management device 100 determines whether the conveying direction of the product is entering or leaving the warehouse based on the still image data. The conveying direction determination process S15 determines the moving direction of the product by grasping the position of the captured product in the image data in chronological order.

[0087] The management device 100 executes a surplus / shortage determination process S30. If there is a surplus or shortage of products in the surplus / shortage determination process S30 (Yes in S200-3), the management device 100 checks the products and takes measures to address the surplus or shortage (S200-4).

[0088] The management device 100 creates teacher data from the image data and the excess / deficiency determination result, executes the image analysis learning model learning process S42, and waits for detection of RFID readout again (S200-1).

[0089] In the first embodiment, the management device 100 executes the product management process when the detection of the product movement by image analysis of a moving image is triggered. However, when the captured image is a still image, the product movement may not be detected in real time as in the case of a moving image. Therefore, in the third embodiment, the management device 100 captures still images (continuously) when the RFID is read, thereby making it possible to detect the product movement without constantly capturing still images.

[0090] [Fourth embodiment] A fourth embodiment will be described below. In the fourth embodiment, a learning method for a learning model for image analysis will be described.

[0091] <Learning process for learning model for image analysis> The image analysis learning model learning process S42 will be described below. The image analysis learning model learning process S42 is a process that is executed in the following two patterns.

[0092] The first pattern is a pattern in which learning is performed using image data captured during operation of the product management system 1. For example, the number of images and the number of reads may differ. In this case, it is possible that an error occurred in reading the RFID, such as an RFID read failure or an RFID of a product other than the target product being read by mistake. On the other hand, in this case, it is also possible that the learning model for image analysis has erroneously determined the number of images. Here, for example, assuming that the possibility of an error occurring in which the RFID reads an RFID other than the product is considerably low, if the number of reads is greater than the number of images, it is highly likely that the determination of the number of images is incorrect. Therefore, if the number of reads is greater than the number of images, the management device 100 assumes that the number of reads is correct, and in the learning data creation process S40, performs the learning model learning process for image analysis using the number of reads as the correct number and the image data used for image analysis as learning data (teacher data).

[0093] Furthermore, when the learning model for image analysis determines the image transport direction using an image captured by a camera without a motion function, the management device 100 may learn about the image transport direction (product movement direction). The image transport direction is determined from the product movement direction. Therefore, if the movement direction is determined incorrectly, the image transport direction will also be determined incorrectly. Therefore, the management device 100 learns the product movement direction as the image transport direction learning. Note that when a camera with a motion function is used, the management device 100 does not learn the image transport direction because it obtains the movement direction determination from the camera.

[0094] The second pattern is a pattern in which pre-learning is performed before operation of the product management system 1. Fig. 10 is a diagram showing an example of a processing flowchart of the image analysis learning model pre-learning process S300.

[0095] The management device 100 captures images of a product (plural or singular) from multiple angles (S300-1). In the process S300-1, for example, multiple products are stacked and images are captured from four directions, namely, front, back, left and right.

[0096] The management device 100 captures an image of an individual item (S300-2). The individual items are, for example, products (including packaging) of various shapes and sizes. In addition to products, the individual items also include carts for carrying products, people carrying products, objects in a warehouse, and the like. In other words, the individual items are objects other than products that may exist (be located) in the vicinity of the gate 200. By using the image data of the individual items for pre-learning, the accuracy of determining whether or not an item is a product is improved.

[0097] The management device 100 generates image data in which the boundaries between the products are processed (processed image data) (S300-3). Fig. 11 is a diagram showing an example of processed image data. In process S300-3, the management device 100 processes the image data of Fig. 11(A) into the image data of Fig. 11(B), (C), and (D).

[0098] 11B is a diagram showing an example of processing data in which the lines where products come into contact are emphasized. The boundary line where the product below and the product stacked above come into contact is shown as a thick line.

[0099] Fig. 11(C) is a diagram showing an example of processed data in which the boundaries between products on the image are emphasized. In addition to the boundaries where the products on the bottom and top actually touch, the boundaries where the products overlap on the image are also made thicker.

[0100] 11D is a diagram showing an example of processed data in which all of the product lines are emphasized. For example, if the product is a rectangular parallelepiped, all of the sides of the area in which both products are shown are displayed with thick lines.

[0101] In this way, by using processed data that makes the relationships between products, between products and spaces, or the shapes of products clearer as training data, the accuracy of recognizing the number of products is improved.

[0102] The management device 100 uses the captured data and the processed data as training data to train a learning model for image analysis (S300-4), and then ends the process.

[0103] In this way, the management device 100 does not learn the number of products itself in the pre-learning. This requires a huge amount of training data to learn the number, which increases time and labor. In the fourth embodiment, the management device 100 learns individual products and objects, and learns boundaries using processed data in which some lines are emphasized, making it possible to build a learning model for image analysis that can determine the number of products (recognize individual products) without learning the number of products itself.

[0104] [Other embodiments] The above-mentioned embodiments may be combined and executed. For example, still images may be used instead of the moving images in the first embodiment. In this case, still images may be taken periodically to determine whether the product has moved. In addition, learning may be performed as appropriate, taking into consideration the time and man-hours required for learning, such as pre-learning and learning during operation. [Explanation of symbols]

[0105] 1: Product management system 10: Image processing section 11: Image analysis learning model learning section 12: Number determination section 13: Conveying direction determination unit 14:Product Management Department 15: Incoming and outgoing list acquisition section 16: Excess / deficiency determination section 17: Learning data creation section 20:RFID reading section 30: Photography Department 50: Learning model for image analysis 100: Management device 110: CPU 120: Storage 121: RFID reading program 122: Shooting Program 123: Image processing program 124: Product Management Program 125: Image analysis learning model learning program 130: Memory 160: Display 200: Gate 300: Camera

Claims

1. a gate unit having an RFID reader function for reading product identification information from an RF tag attached to the product; an imaging unit that images the periphery of the gate unit; a management unit that manages products passing through the gate unit based on data photographed by the photographing unit and identification information read by the gate unit, The management unit an image processing unit that executes a conveying direction determination process that determines a conveying direction indicating whether the conveyed product is entering or leaving the warehouse based on image data captured by the photographing unit, and a number determination process that determines the number of the conveyed products; and an excess / shortage determination unit that compares the number of pieces of identification information read by the gate unit with the number of the conveyed products determined by the image processing unit, and determines whether the number of pieces of identification information matches or does not match the number determined by the image processing unit, and then determines whether the conveyed products are incoming or outgoing products, thereby determining an excess or shortage of the conveyed products for each conveying direction. Product management system.

2. A gate unit having an RFID reader function for reading product identification information from an RF tag attached to the product; an imaging unit that images the periphery of the gate unit; a management unit that manages products passing through the gate unit based on data photographed by the photographing unit and identification information read by the gate unit, The management unit an image processing unit that executes a conveying direction determination process that determines a conveying direction indicating whether the conveyed product is entering or leaving the warehouse based on image data captured by the photographing unit, and a number determination process that determines the number of the conveyed products; an excess / shortage determination unit that compares the number of pieces of identification information read by the gate unit with the number of the conveyed products determined by the image processing unit, and determines whether the conveyed products are insufficient or excessive for each conveying direction, The image processing unit determines a conveying direction of the conveyed product by using a model trained by learning data generated from the data photographed by the photographing unit and the result of the conveying direction determined based on the data. Product management system.

3. The image data is video data, When the image processing unit recognizes that the conveyance of the conveyed product has started in the video data, the image processing unit instructs the gate unit to read the RF tag. The product management system according to claim 1 or 2.

4. The surplus / deficiency determination unit further acquires a product list including quantity information regarding the number of the products to be transported, compares the number of the products to be transported in the product list with the number of pieces of identification information read by the gate unit, and the number of the products to be transported determined by the image processing unit, and if all match, determines that there is no surplus or shortage of the products to be transported. The product management system according to claim 1 or 2.

5. the product list includes identification information of the products to be delivered; The surplus / deficiency determination unit determines that there is no surplus or deficiency in the products to be conveyed when the identification information of the products to be conveyed in the product list matches all of the identification information read by the gate unit, and further when the number of the products to be conveyed determined by the image processing unit matches the number of the identification information read by the gate unit. The product management system according to claim 4.

6. The product list includes information regarding a conveying direction, The surplus / deficiency determination unit further determines that there is no surplus or deficiency in the conveyed products when the conveying direction in the product list coincides with the conveying direction determined by the image processing unit. The product management system according to claim 5.

Images