Information processing device, information processing method, and program

Abstract

This invention provides an information processing device, information processing method, and program that appropriately count the number of product faces in a store. [Solution] In a system comprising an imaging device, an information processing device, and a storage device, the information processing device has an image input unit into which an image is input, and a face counting unit that counts the number of faces based on the image using a plurality of counting methods. The face counting unit counts the number of faces using each counting method, determines the final number of faces based on the number of faces from each counting method, and if the number of faces from each counting method is not the same, it selects a determination method to determine the final number of faces according to the reason for the difference.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program. In particular, it relates to counting the number of product faces in a store. [Background Art]

[0002] It is known that when selling products, the number of faces of products displayed in a store affects sales. In order to achieve optimal sales in a limited space, it is necessary to appropriately manage the number of product faces. The number of faces is a value indicating how many products are arranged in the horizontal direction.

[0003] It has been proposed to obtain the number of faces by excluding products displayed in the vertical direction and counting in the horizontal direction (see, for example, Patent Document 1).

[0004] It has been proposed to obtain the number of faces by excluding products stacked on top from the counting (see, for example, Patent Document 2).

[0005] It has been proposed that among the number of faces obtained from a plurality of images captured of the same shelf, the largest number of faces is used as the integrated number of faces (see, for example, Patent Document 3). [Prior Art Documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent No. 6705075 [Patent Document 2] Japanese Unexamined Patent Publication No. 2024-020698 [Patent Document 3] Japanese Patent No. 6798741 [Summary of the Invention] [Problem to be Solved by the Invention]

[0007] Proper management of product face counts requires accurately counting the number of face counts for products currently on display. In this case, low-profile products such as canned coffee are often stacked vertically on shelves. In particular, accurately counting the number of face counts for products stacked vertically is essential. [Means for solving the problem]

[0008] An information processing device in one aspect of the present disclosure includes a video input unit into which video is input, and a face counting unit that counts the number of faces based on the video using a plurality of counting methods.

[0009] An information processing method for an information processing device in one aspect of the present disclosure involves receiving an image as input and counting the number of faces based on the image using a plurality of counting methods.

[0010] A program in one aspect of this disclosure causes a computer to input video and perform information processing that counts the number of faces based on the video using a plurality of counting methods. [Effects of the Invention]

[0011] According to this disclosure, the number of faces of a product can be properly counted. [Brief explanation of the drawing]

[0012] [Figure 1] Overall system configuration [Figure 2] Configuration of an information processing device [Figure 3] Functional block diagram of an information processing device. [Figure 4] Flowchart executed by information processing device [Figure 5] Example flowchart for when an information processing device executes counting method 1 [Figure 6] Example flowchart for when an information processing device performs counting method 2 [Figure 7]Example of a flowchart for a case where an information processing apparatus determines the final number of faces [Figure 8] Another example of a flowchart for a case where an information processing apparatus determines the final number of faces [Figure 9] Figure showing an example of a face counting method based on counting method 1 [Figure 10] Figure showing an example of a face counting method based on counting method 2 [Figure 11] Figure showing another example of a face counting method based on counting method 1 [Figure 12] Figure showing another example of a face counting method based on counting method 2 [Figure 13] Figure showing still another example of a face counting method based on counting method 1 [Figure 14] Figure showing still another example of a face counting method based on counting method 2 DETAILED DESCRIPTION OF EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings.

[0014] (Embodiment 1) FIG. 1 shows the configuration of the overall system 100.

[0015] The system 100 includes an imaging apparatus 110, an information processing apparatus 120, and a storage apparatus 130.

[0016] The imaging apparatus 110 images shelves in a store. The imaging may be a moving image or may be a still image. The imaging apparatus 110 outputs the captured image / video to the information processing apparatus 120.

[0017] The information processing apparatus 120 analyzes an input image / video. The information processing apparatus 120 may receive the image / video from the imaging apparatus 110, or may acquire the image / video stored in the storage apparatus 130. The information processing apparatus 120 may receive the image / video from another apparatus not shown in the figure.

[0018] The storage device 130 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 130 may reside in the cloud.

[0019] Figure 2 shows the configuration of the information processing device 120.

[0020] The information processing device 120 includes a processor 121, an input device 122, and an output device 123. In addition to the devices shown in Figure 2, it may also include a communication device, memory, and storage. Each device, such as the processor 121 and the input device 122, is connected by a bus 124 for communicating information. The bus 124 may be configured using a single bus, or different buses may be configured for each device.

[0021] The processor 121 is composed of a computer including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 121 receives video footage captured by the imaging device 110 via the input device 122 and analyzes the input video footage. The processor 121 may also retrieve video footage by searching the storage device 130.

[0022] The input device 122 may be an input device that receives input information from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.), or it may be an interface or communication device that receives video from the imaging device 110. The input device 122 may be connected to the storage device 130 and other devices not shown.

[0023] The output device 123 may be an output device that outputs user responses to the outside (e.g., a display, speaker, LED (Light Emitting Diode) lamp, or it may be an interface or communication device that transmits data. The output device 123 may be connected to the storage device 130 and other devices not shown.

[0024] The input device 122 and the output device 123 may be configured as an integrated unit (for example, a touch panel), or the communication device may be the input device 122 and / or the output device 123.

[0025] Memory is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory may also be called registers, cache, main memory, etc. Memory can store executable programs (program code), software modules, etc., for carrying out a wireless communication method according to one embodiment of this disclosure.

[0026] Storage is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk (e.g., a floppy disk), a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), or a magnetic strip. Storage may also be called an auxiliary storage device. The aforementioned storage medium may be, for example, a database, server, or other suitable medium that includes at least one of memory and storage.

[0027] A communication device is hardware (input / output device) for communicating between computers via at least one of a wired network and a wireless network, and is also called a network device, network controller, network card, communication module, etc.

[0028] Furthermore, the information processing device 120 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 121 may be implemented using at least one of these hardware components.

[0029] Figure 3 shows a functional block diagram of the information processing device 120.

[0030] The information processing device 120 includes a video input unit 125 and a face counting unit 126. The information processing device 120 may also include a storage device 130.

[0031] The video input unit 125 receives video from the imaging device 110. The input video may be a still image or a video. The video input unit 125 transmits the input video to the face counting unit 126.

[0032] The face counting unit 126 has multiple counting methods. The number of counting methods can be two or more. The face counting unit 126 counts the number of faces in the video received from the video input unit 125 using each counting method. Based on the number of faces counted using each counting method, the face counting unit 126 determines the final number of faces (hereinafter referred to as the "final number of faces"). Details of the method for determining the number of faces will be described later.

[0033] Figure 4 shows a flowchart of the actions performed by the information processing device 120.

[0034] The information processing device 120 determines whether or not video has been input (step S401). Step S401 is performed by the video input unit 125. If no video has been input (step S401, No), the process returns to step S401 and waits for video to be input.

[0035] When video is input to the information processing device 120 (step S401, Yes), it detects objects in the received video (step S402). The information processing device 120 can detect objects such as shelves and products in the video by executing a well-known object detection model on the received video. For example, detected objects are displayed surrounded by a rectangle (bounding box). Object detection using the object detection model can determine the type of object detected, so it is possible to determine whether the detected object is object A or object B. For example, it is possible to distinguish and detect canned coffee from manufacturer A and canned coffee from manufacturer B, or black coffee from manufacturer A and lightly sweetened coffee from manufacturer A, etc.

[0036] The information processing device 120 may detect objects by segmentation. The information processing device 120 may detect objects by recognizing the region (outline) of an object in the image, for example, by semantic segmentation, instant segmentation, or panoptic segmentation. The information processing device 120 may display a rectangle that circumscribing the region of the recognized object.

[0037] The information processing device 120 counts the number of faces using each counting method based on the detected object (step S403). Figures 5 and 6 show flowcharts of how the information processing device 120 counts the number of faces using each counting method in step S403.

[0038] The information processing device 120 determines whether the number of faces has been counted using all counting methods (step S404). If there are any counting methods that have not yet counted the number of faces (step S404, No), the information processing device 120 returns the flow to S403 and counts the number of faces using the counting methods that have not yet counted the number of faces. The information processing device 120 may select multiple counting methods to be used.

[0039] Here, we will explain each method for counting the number of faces.

[0040] Figure 5 shows an example flowchart of when the information processing device 120 executes counting method 1. For example, counting method 1 is a counting method in which identical products that are stacked vertically (up and down) are counted as one, and the number of each product in the horizontal direction is counted.

[0041] If the detected identical products are stacked vertically, the information processing device 120 combines the identical products that are stacked vertically (step S501).

[0042] The information processing device 120 counts the number of identical products in the horizontal direction and calculates the number of faces (step S502).

[0043] Figure 6 shows an example flowchart of the information processing device 120 when it executes counting method 2. Counting method 2 is a counting method that counts products stacked vertically, one row at a time.

[0044] The information processing device 120 counts the number of items horizontally for each layer on which the items are stacked (step S601).

[0045] The information processing device determines whether or not the number of items has been counted for all rows (step S602). If there are rows where the number of items has not been counted (step S602, No), the information processing device 120 returns the flow to S601 and counts the number of items horizontally for the rows that have not yet been counted.

[0046] If the number of products is counted for all rows (Step S602, Yes), the information processing device calculates the maximum count for each row as the number of faces (Step S603).

[0047] The method used to calculate the number of product faces may vary, and there may be three or more methods. For example, the number of products touching the shelf may be counted as the number of product faces.

[0048] If the number of faces has been counted using all counting methods (Step S404, Yes), the information processing device 120 determines the final number of faces based on the number of faces calculated using each counting method (Step S405).

[0049] Here, we will explain how the final number of faces is determined.

[0050] Figure 7 shows an example flowchart of the process when the information processing device 120 determines the final number of faces.

[0051] The information processing device 120 acquires the number of faces calculated by each counting method (step S701).

[0052] The information processing device 120 determines the final number of faces based on the number of faces calculated by each counting method (step S702). For example, the information processing device 120 determines the average value of the number of faces calculated by each counting method as the final number of faces. The information processing device may also determine the maximum or minimum value of the number of faces calculated by each counting method as the final number of faces.

[0053] Figure 8 shows an example of another flowchart for when the information processing device 120 performs the determination of the final number of faces.

[0054] The information processing device 120 acquires the number of faces calculated by each counting method (step S801). Step S801 is the same as step S701.

[0055] The information processing device 120 determines whether the number of faces calculated by each counting method is the same (step S802). If the number of faces calculated by each counting method is different (step S802, No), it identifies the cause of the discrepancy (different) in the number of faces (step S803). For example, it identifies causes such as the products not being aligned on the shelf, another product being placed (presumably in front of the product in question), or products being misaligned when stacked vertically.

[0056] The information processing device 120 selects a determination method for determining the final number of faces based on the cause of the mismatch (step S804). The information processing device 120 may also determine a method for counting the number of faces based on the cause of the mismatch, and use the number of faces calculated based on the determined counting method as the final number of faces.

[0057] The information processing device 120 determines the final number of faces if the number of faces calculated by each counting method is the same (step S802, Yes), or if a determination method is selected in step S804 (step S805).

[0058] <Examples> Figures 9-14 show the state in which the information processing device 120 has detected an object (step S402), that is, the state in which the object detection model has been executed on the input video.

[0059] <Case 1> Figure 9 shows an example of the method for counting the number of faces based on counting method 1. Counting method 1 is a method of counting products that are stacked vertically as one unit. Figure 9 shows an example in which eight products A are detected on a shelf. Counting method 1 combines products A(2) and A(6), A(3) and A(7), and A(4) and A(8), which are stacked vertically, and counts the number of products horizontally. The dotted line in Figure 9 indicates the combined state of the items. In Figure 9, the information processing device 120 calculates the number of faces based on counting method 1 as "5".

[0060] Figure 10 shows an example of the face count method based on counting method 2. Counting method 2 is a method of counting products stacked vertically, one layer at a time. Since counting method 2 is calculated for the same image as counting method 1, the state of the products in Figure 10 is the same as in Figure 9. Counting method 2 counts the number of products horizontally for each layer in which the products are stacked, and the maximum value of the counts for each layer is taken as the face count. The dotted lines in Figure 10 indicate products on the same layer. In Figure 10, the information processing device 120 counts the number of products in the first layer as "5" and the number of products in the second layer as "3". Therefore, the information processing device 120 determines the largest of the counts "5" and "3" for each layer, "5", as the face count.

[0061] <Case Study 2> Figures 11 and 12 show product A(5) from Figures 6 and 7 in a state where it is stuck on the upper shelf for some reason.

[0062] Figure 11 shows another example of the method for counting the number of faces based on counting method 1. In Figure 11, the information processing device 120 calculates the number of faces based on counting method 1 as "5".

[0063] Figure 12 shows another example of the method for counting the number of faces based on counting method 2. In Figure 12, the information processing device 120 counts the number of products as "4" in the first row and counts the number of products as "4" in the second row. Therefore, the information processing device 120 determines the "4" counted in each row as the number of faces.

[0064] The information processing device 120 determines the final number of faces based on the number of faces calculated by each counting method. If the information processing device 120 uses the average value of the number of faces calculated by each counting method as the final number of faces, it determines the final number of faces to be "4.5". If the information processing device 120 uses the maximum value of the number of faces calculated by each counting method as the final number of faces, it determines the final number of faces to be "5". If the information processing device 120 uses the minimum value of the number of faces calculated by each counting method as the final number of faces, it determines the final number of faces to be "4".

[0065] In the cases of Figures 11 and 12, the actual number of faces is "5," so the information processing device 120 should select the maximum value as the final number of faces. Therefore, for example, in step S402, the information processing device 120 instructs the object detection model to detect objects and also to detect whether the objects (products) are at an angle. If there are products that are at an angle, the information processing device 120 determines the maximum value of the number of faces calculated by each counting method as the final number of faces.

[0066] <Case Study 3> Figures 13 and 14 show a situation where products are stacked four to four horizontally and two vertically, with the products on the upper layer being misaligned.

[0067] Figure 13 shows yet another example of the method for counting the number of faces based on counting method 1. Figure 13 shows the case where it is determined that there are no products overlapping vertically. Therefore, the information processing device 120 calculates the number of faces based on counting method 1 as "8".

[0068] Figure 14 shows yet another example of the method for counting the number of faces based on counting method 2. In Figure 14, the information processing device 120 counts the number of products as "4" in the first row and counts the number of products as "4" in the second row. Therefore, the information processing device 120 determines the "4" counted in each row as the number of faces.

[0069] The information processing device 120 determines the final number of faces based on the number of faces calculated by each counting method. In the case of Figures 13 and 14, the actual number of faces is "4", so the information processing device 120 should select the minimum value as the counting method for the final number of faces. In counting method 1, it was determined that there were no products overlapping vertically, which can be inferred to be due to the products on the upper row being misaligned. When the products on the upper row are misaligned, the information processing device 120 determines the minimum number of faces calculated by each counting method as the final number of faces.

[0070] <Variation> Instead of calculating the number of faces, you can also calculate the number of items displayed. The number of items displayed is the number of items as seen from the aisle. In other words, for items that are not stacked vertically, the number of faces and the number of items displayed are the same.

[0071] Instead of counting method 1, the number of items to be displayed is calculated based on VLM (Vision and Language Model). That is, the number of items to be displayed is determined based on VLM and counting method 2. Alternatively, the number of items to be displayed may be determined based on VLM and counting method 1 by calculating the number of items to be displayed based on VLM instead of counting method 2.

[0072] In counting method 2, the number of items to be displayed can be determined by summing the number of items counted on each shelf.

[0073] The final number of items on display can be calculated based on the number of items output by VLM and the number of items determined by counting method 2. The method for counting the final number of items on display is the same as the method for calculating the final number of faces.

[0074] In the embodiments described above, the notation for each component may be replaced with other notations such as "circuitry", "assembly", "device", "unit", or "module".

[0075] Furthermore, this disclosure can be implemented in software, hardware, or software in conjunction with hardware. Each functional block used in the description of the above embodiments may be implemented in part or in whole as an integrated circuit (LSI), and each process described in the above embodiments may be controlled in part or in whole by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may also be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0076] The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0077] Furthermore, if advancements in semiconductor technology or derivative technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possibility. [Industrial applicability]

[0078] This disclosure is useful for information processing equipment that counts the number of product faces in a store. [Explanation of Symbols]

[0079] 100 Systems 110 Imaging device 120 Information Processing Devices 121 processors 122 Input device 123 Output device 124 Bus 125 Video Input Section 126 Face counting unit 130 Storage device

Claims

1. The video input section where video is input, A face counting unit that counts the number of faces using multiple counting methods based on the aforementioned video, An information processing device having

2. The face counting unit counts the number of faces using each counting method and determines the final number of faces based on the number of faces obtained using each counting method. The information processing apparatus according to claim 1.

3. The face counting unit, if the number of faces obtained by each counting method is not the same, selects a determination method to determine the final number of faces according to the reason for the difference. The information processing apparatus according to claim 2.

4. The face counting unit uses the average, minimum, or maximum value of the face count calculated by each of the counting methods as the final face count. The information processing apparatus according to claim 2.

5. The face counting unit executes an object detection model on the video to detect objects and counts the number of faces of the detected objects. The information processing apparatus according to claim 1.

6. The video is input, Based on the aforementioned video, the number of faces is counted using multiple counting methods. Information processing method for information processing equipment.

7. On the computer, Input video, Based on the aforementioned video, the number of faces is counted using multiple counting methods. A program used to perform information processing.

Images