Image processing device, image processing method, and program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Filing Date
- 2026-01-23
- Publication Date
- 2026-04-22
Abstract
Description
Image processing device, image processing method, and program
[0001] The present disclosure relates to an image processing device, an image processing method, and a program.
[0002] As described in Patent Literature 1, product inventory status is monitored from product images taken of shelves on which the products are displayed. In Patent Literature 1, for example, the inventory of products that may be displayed on shelves is monitored by processing images from multiple cameras, including a high-resolution camera.
[0003] Special table 2019-530035 publication
[0004] However, the technology described in Patent Document 1 requires constantly acquiring and processing images captured by a high-resolution camera, which poses a problem of increased device costs and image processing costs.
[0005] Therefore, an object of the present disclosure is to provide an image processing device that can solve the above-mentioned problem of increased device costs and image processing costs when processing product images.
[0006] An image processing device according to one embodiment of the present disclosure includes: an image acquisition unit that acquires a first image captured by a first imaging device and a second image captured by a second imaging device, the second image including at least a portion of the imaging range of the first image and of higher quality than the first image; a product arrangement recognition unit that recognizes an arrangement of products on the second image based on the second image; and a matching unit that matches the arrangement of products on the second image to the first image. Also, an image processing method according to one embodiment of the present disclosure includes: acquiring a first image captured by a first imaging device and a second image captured by a second imaging device, the second image including at least a portion of the imaging range of the first image and of higher quality than the first image; recognizing an arrangement of products on the second image based on the second image; and matching the arrangement of products on the second image to the first image. Furthermore, a program according to one embodiment of the present disclosure is configured to cause a computer to execute the following processes: acquire a first image captured by a first photographing device and a second image captured by a second photographing device, the second image including at least a portion of the photographing range of the first image and of higher quality than the first image; recognize the arrangement of products on the second image based on the second image; and associate the arrangement of products on the second image with the arrangement of products on the first image.
[0007] By being configured as described above, the present disclosure can suppress increases in device costs and image processing costs when processing product images.
[0008] FIG. 1 is a block diagram showing the overall configuration of an image processing system according to the present disclosure. FIG. 1 is a block diagram showing the configuration of an image processing device according to the present disclosure. FIG. 2 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 3 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 4 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 5 is a flowchart showing the processing operation of an image processing device according to the present disclosure. FIG. 6 is a flowchart showing the processing operation of an image processing device according to the present disclosure. FIG. 7 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 8 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 9 is a diagram showing the state of processing by an image processing device according to the present disclosure. FIG. 10 is a block diagram showing the hardware configuration of an image processing device according to the present disclosure. FIG. 11 is a block diagram showing the configuration of an image processing device according to the present disclosure.
[0009] First Embodiment A first embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any embodiment.
[0010] [Configuration] The image processing system in this embodiment is used to manage the status of products sold in a store and products stored in a storage location such as a warehouse. In particular, the image processing system takes images of products displayed on shelves and detects the status of the products, such as the product availability or display clutter, from the product images. Note that in this embodiment, a case is described in which the image processing system is used in a store to detect the status of products sold in the store, but the image processing system may be used in any location and may detect the status of products displayed in any location.
[0011] 1, the image processing system includes two different cameras 21 and 22, each capturing an image within its capture range of the same shelf R on which products are displayed in a store, and an image processing device 10 that processes the images captured by these cameras 21 and 22. In this embodiment, the two cameras 21 and 22 are configured as a surveillance camera 21 (first imaging device) installed on the ceiling or wall of the store, and a mobile camera 22 (second imaging device) mounted on a mobile information processing terminal such as a smartphone operated by a store employee.
[0012] In this case, since the surveillance camera 21 is installed on the ceiling or wall, as shown in Fig. 1, the surveillance camera images g1 and g3 taken by the surveillance camera 21 may not be images taken from the front of the shelf R, but may be images with an angle of view shifted by a predetermined angle from the front of the shelf R, i.e., the product E. Furthermore, because the surveillance camera images g1 and g3 need to be taken and stored for a long time, they are low-quality images with low resolution or grayscale.
[0013] On the other hand, because the mobile camera 22 can be carried by a person, the person can use the mobile camera 22 to take a mobile image g2 from in front of the shelf R. Therefore, the mobile image g2 can be an image taken from the front of the shelf R, i.e., the product E, as shown in FIG. 2 . Furthermore, the mobile image g2 is of higher quality than the above-mentioned surveillance camera images g1 and g3, for example, it has a higher resolution and is a color image than the above-mentioned surveillance camera images g1 and g3. Note that while it is desirable for the mobile image g2 to be an image taken from the front of the shelf R or the product E, it is not necessarily limited to an image taken from the front of the shelf R or the product E, and the image may have any angle of view relative to the shelf R or the product E.
[0014] The above-mentioned surveillance camera images g1 and g3 and mobile phone image g2 are images that include at least a portion of the same shooting range in which the target product E may be displayed. In this embodiment, the surveillance camera images g1 and g3 and the mobile phone image g2 are images that include the same shelf R shown in Fig. 1 in their shooting range. The surveillance camera images g1 and g3 are images taken at different times; for example, the surveillance camera image g1 is taken at a predetermined time, such as when products are replenished, and the surveillance camera image g3 is taken after that.
[0015] The two cameras 21 and 22 described above are not limited to the surveillance camera 21 and the mobile camera 22, but may be other types of cameras. In this case, the images captured by the two cameras 21 and 22 respectively include at least a portion of the same shooting range, and the quality of one image is higher than the quality of the other image.
[0016] The image processing device 10 is composed of one or more information processing devices each including a calculation device and a storage device. As shown in FIG. 2 , the image processing device 10 includes an image acquisition unit 11, a product placement recognition unit 12, a matching unit 13, a product quantity calculation unit 14, and a product status detection unit 15. The functions of the image acquisition unit 11, the product placement recognition unit 12, the matching unit 13, the product quantity calculation unit 14, and the product status detection unit 15 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The image processing device 10 also includes an image storage unit 16 and a storage device for storing association information. The image storage unit 16 and the storage device for storing association information are each configured using a storage device. Each component will be described in detail below.
[0017] The image acquisition unit 11 acquires surveillance camera images g1 and g3 (first image and third image) taken by the surveillance camera 21 and a mobile image g2 (second image) taken by the mobile camera 22, and stores them in the image storage unit 16. At this time, the image acquisition unit 11 acquires and stores the surveillance camera images g1 and g3 and the mobile image g2 together with location information indicating the location where the images were taken and time information indicating the time of the images. For example, in the case of the surveillance camera images g1 and g3, the image acquisition unit 11 acquires identification information of the surveillance camera 21 that took the images and associates it with the images, thereby enabling the predetermined installation location of the surveillance camera 21 identified by the identification information to be used as location information where the images were taken. Furthermore, in the case of the mobile image g2, the image acquisition unit 11 acquires location information acquired by the information processing terminal equipped with the mobile camera 22 that took the image and associates it with the image, thereby enabling the location information to be used as location information where the image was taken. In addition, the image acquisition unit 11 acquires the time information assigned to the image when it was taken by the surveillance camera 21 or the mobile camera 22, or the time information when the image was acquired from the surveillance camera 21 or the mobile camera 22, and associates it with the image, so that such time information can be used as the time when each image was taken.
[0018] The image acquisition unit 11 acquires and stores surveillance camera images g1 and g3 from the surveillance camera 21 at regular time intervals or at any timing, or acquires surveillance camera images g1 and g3 that are constantly being taken and stores the images at regular time intervals. The image acquisition unit 11 also acquires and stores a mobile image g2 transmitted from an information processing terminal equipped with a mobile camera 22. The mobile image g2 is assumed to be an image taken when the products E have been restocked on the shelves R in the store and are lined up and displayed.
[0019] The product placement recognition unit 12 performs image processing on the mobile image g2 to recognize the placement of the products E shown in the mobile image g2. Because the mobile image g2 is a high-quality image, such as a high-resolution or color image, the product placement recognition unit 12 can recognize the product placement, including the positions and types of the products E, from the mobile image g2. For example, the product placement recognition unit 12 extracts feature amounts from each location on the mobile image g2 and compares these feature amounts with pre-registered product feature amounts to recognize the type of each product E on the mobile image g2 and the positions where each product E is placed on the mobile image g2. The product placement recognition unit 12 then generates product placement information G2 including the positions and types of the products on the mobile image g2. Because the mobile image g2 is an image of the shelves R after they have been restocked and aligned as described above, the product placement information G2 recognized and generated by the product placement recognition unit 12 can be said to represent a situation in which a sufficient number of products E are arranged on the shelves R and are aligned neatly.
[0020] An example of the product placement information G2 generated by the product placement recognition unit 12 is shown in Fig. 3. In the example of Fig. 3, the product placement information G2 is generated by associating information about a frame F surrounding a recognized product E on the mobile image g2, which indicates the position of the recognized product E, with the type of the recognized product E within the frame F. For example, for the top shelf of the shelf R shown in the mobile image g2 shown in Fig. 3, the product placement information G2 is generated by associating information indicating the position of each frame F on the mobile image g2 with information indicating the type of product E within the frame F, "coffee."
[0021] The product arrangement recognition unit 12 may recognize the product arrangement from the mobile image g2 by any method. For example, a barcode or two-dimensional code indicating the type of product E that can be displayed may be attached to each position on the shelf R in advance, and the product arrangement recognition unit 12 may recognize the barcode or the like on the mobile image g2 to recognize the product arrangement consisting of the product positions and types on the mobile image g2, and generate the product arrangement information G2.
[0022] The product arrangement recognition unit 12 may also acquire information about the location where the mobile image g2 was taken and recognize the product arrangement as described above according to the location where the image was taken. For example, if the product arrangement recognition unit 12 can identify the category of products that may be displayed on the shelf R installed at the location where the image was taken from the information about the location where the image was taken, the unit may recognize the type of product by narrowing it down to that category. As an example, if the acquired information about the location where the image was taken can identify that the product category is "beverages," the unit may recognize the type of product by limiting it to the types of products that belong to "beverages." In this case, the information about the location where the image was taken can be, for example, acquired by acquiring information that is pre-associated with the mobile image g2 as described above, acquired by being input by the person who took the image, acquired from a QR code affixed to the shelf R, or acquired from the installation location of the associated surveillance camera 21 as described below.
[0023] In the above example, the product arrangement recognition unit 12 recognizes the position and type of the product on the mobile image g2 as the product arrangement, but the product type does not necessarily have to be recognized. In other words, it is sufficient for the product arrangement recognition unit 12 to recognize at least the position of the product on the mobile image g2, that is, the arrangement of the product relative to the shelf R.
[0024] Here, the above-described product placement recognition unit 12 may be realized by a machine-learned model. For example, a machine-learning model may be generated by machine learning learning data in which mobile phone images and product placements are previously associated with each other, and the mobile phone image g2 may be input to the machine-learning model to recognize the placement of the product E on the mobile phone image g2.
[0025] The association unit 13 associates the product placement information G2 on the recognized mobile phone image g2 with the surveillance camera image g1 as described above. Specifically, the association unit 13 first selects the surveillance camera image g1 to which the product placement information G2 is to be associated. At this time, the association unit 13 checks time information, such as the shooting time, associated with the mobile phone image g2 for which the product placement was recognized, and selects the surveillance camera image g1 based on the time information. For example, the association unit 13 selects the surveillance camera image g1 associated with time information closest to the time associated with the mobile phone image g2. As a result, it can be expected that the selected surveillance camera image g1 shows products E displayed on the shelf R in a product placement that is substantially identical to that of the recognized mobile phone image g2.
[0026] Next, the correspondence unit 13 examines the correspondence between each location on the surveillance camera image g1 and the mobile phone image g2. Specifically, the correspondence unit 13 extracts feature amounts of each location on the surveillance camera image g1 and the mobile phone image g2, matches the feature amounts, and calculates the image similarity of each location to identify corresponding locations on the surveillance camera image g1 and the mobile phone image g2. The correspondence unit 13 then calculates a correspondence function for each location on the surveillance camera image g1 and the mobile phone image g2 based on the positional relationship of the corresponding locations on the surveillance camera image g1 and the mobile phone image g2. As an example, the correspondence function (x, y) = f(x', y') is calculated as the correspondence function between the position (x, y) on the surveillance camera image g1 and the position (x', y') on the mobile phone image g2.
[0027] Then, the association unit 13 associates the product arrangement on the mobile phone image g2 with the surveillance camera image g1 using the positional relationship of corresponding locations between the surveillance camera image g1 and the mobile phone image g2, for example, using the association function described above. Specifically, the association unit 13 uses the association function to generate association information G1 that associates each frame F representing the position of each product E recognized in the mobile phone image g2 with the surveillance camera image g1. At this time, the association unit 13 also associates the type of product E associated with each frame F associated with the surveillance camera image g1.
[0028] An example of the association information G1 generated by the association unit 13 is shown in Fig. 4. In the example of Fig. 4, information on each frame F indicating the position of each product E recognized on the mobile phone image g2 is associated with the surveillance camera image g1, and the type of product E is also associated with each frame F to generate the association information G1. This generates information representing the product arrangement on the low-quality surveillance camera image g1, rather than an image taken from the front of the product E.
[0029] The association unit 13 then stores the association information G1 generated as described above in the association information storage unit 17. The association information G1 generated in this manner can be said to represent a situation in the surveillance camera image g1 in which a sufficient number of products E are arranged on the shelf R and are neatly aligned. Therefore, the association information G1 will be used later to detect the status of the products E, and will serve as a reference for product arrangement in the surveillance camera image g1.
[0030] Here, the association unit 13 may be realized by a machine-learned model. For example, a machine-learning model may be generated by machine learning training data in which the surveillance camera image g1 and the mobile image g2 including the product placement information G2 are previously associated with the association information G1, and the association information G1 may be generated by inputting the surveillance camera image g1 and the mobile image g2 including the product placement information G2 into the machine-learning model.
[0031] After generating the association information G1 as described above, the product quantity calculation unit 14 (product estimation unit) estimates the products in the surveillance camera image g3 from another surveillance camera image g3 captured by the surveillance camera 21. At this time, the product quantity calculation unit 14 detects a product area E' in the surveillance camera image g3 and calculates the possible location of the products and the quantity of the products from the product area E'. For example, although the surveillance camera image g3 has a lower resolution and quality than the mobile phone image g2, the product quantity calculation unit 14 can detect the product area E' in the surveillance camera image g3 using a technique such as semantic segmentation and calculate the location and quantity of the products corresponding to the product area E'. In other words, the product quantity calculation unit 14 calculates only the location and quantity of the products without estimating the type of product.
[0032] For example, the product quantity calculation unit 14 generates a product area image G3 as shown in the right diagram of Fig. 5 from a surveillance camera image g3 as shown in the left diagram of Fig. 5, and calculates the position and quantity of the product area E' by calculating the position of the product area E' and the quantity of the product area E' at each position. In the example of Fig. 5, the product area image G3 can be used to calculate the position and quantity of products in an area other than the right side of the top shelf of the shelf R.
[0033] The product status detection unit 15 detects the status of the products on the surveillance camera image g3 using the products estimated from the product area E' on the surveillance camera image g3 as described above and the association information G1. Here, the product status detection unit 15 detects the sufficiency status of the number of products.
[0034] Specifically, the product status detection unit 15 first associates association information G1, which associates product placements on a surveillance camera image g1, with the surveillance camera image g3. Here, as shown in FIG. 6 , the product status detection unit 15 uses the association information G1 to associate product placements, i.e., frames F representing positions where products can be placed, with product types on a product area image G3 generated from the surveillance camera image g3, to generate product detection information G4. The product status detection unit 15 then checks the number of products based on the detection results of the product area E' within each frame F associated with the product detection information G4. At this time, if the number of products corresponding to the product area E' within the frame F is one or more, the product status detection unit 15 detects that there is a sufficient number of products that can be placed in the position of that frame F. On the other hand, if the number of products corresponding to the product area E' within the frame F is zero, the product status detection unit 15 detects that there is a shortage of products that can be placed in the position of that frame F.
[0035] In the example of product detection information G4 shown in the lower part of Fig. 6, the number of products corresponding to product area E' in frame F located on the right side of the top shelf R is 0. Therefore, the product status detection unit 15 detects that there is a shortage of products that can be placed in the position corresponding to this frame F. Furthermore, because the product type "coffee" is associated with this frame F, the product status detection unit 15 can also detect that the type of product that is in short supply is "coffee."
[0036] The product status detection unit 15 may calculate the product fulfillment rate from the number of products in each frame F. For example, if the maximum number of products that can be displayed in the frame F is set in advance, the fulfillment rate may be calculated as the ratio of the number of products on the surveillance camera image g3 to the maximum number.
[0037] [Operation] Next, a description will be given of the operation of the image processing device 10. First, a description will be given of a process for generating association information that serves as a criterion for product placement used when detecting the status of a product, with reference to the flowchart in FIG.
[0038] The image processing device 10 acquires a surveillance camera image g1 taken by the surveillance camera 21 and a mobile phone image g2 taken by the mobile phone camera 22 and stores them in the image storage unit 16 (step S1). At this time, as shown in FIG. 1, the surveillance camera image g1 is not an image taken from the front of the shelf R with a high angle of view and is a low-quality image, while the mobile phone image g2 is an image taken from almost the front of the shelf R and is a high-quality image with a higher resolution than the surveillance camera image g1. Furthermore, the mobile phone image g2 is an image taken when the products E were restocked and aligned on the shelf R.
[0039] Next, the image processing device 10 performs image processing on the mobile image g2 and recognizes the arrangement of the product E on the mobile image g2 (step S2). For example, as shown in Fig. 3, the image processing device 10 recognizes the position and type of the product E on the mobile image g2 using the feature amount in the mobile image g2, and generates product arrangement information G2 representing the arrangement of the product on the mobile image g2.
[0040] Next, the image processing device 10 checks the correspondence between each location on the surveillance camera image g1 and the mobile phone image g2, and associates the surveillance camera image g1 with the mobile phone image g2 (step S3). Then, based on the positional relationship between the corresponding locations on the surveillance camera image g1 and the mobile phone image g2, it associates the product layout on the mobile phone image g2 with the surveillance camera image g1 (step S4). As a result, as shown in FIG. 4, association information G1 is generated, which serves as a reference for the product layout on the camera image g1.
[0041] Next, a process for detecting the status of a commodity from a subsequently captured surveillance camera image g3 using the above-described association information G1 will be described with reference to the flowchart of FIG.
[0042] The image processing device 10 acquires a surveillance camera image g3 captured by the surveillance camera 21 (step S11). The image processing device 10 then estimates the product in the surveillance camera image g3. At this time, as shown in Fig. 5, the image processing device 10 detects a product area E' in the surveillance camera image g3, and calculates the position where the product may exist and the quantity of the product from the product area E' (step S12).
[0043] Next, the image processing device 10 compares the product positions and quantities estimated from the product area E' in the surveillance camera image g3 with the association information G1 (step S13). Then, the image processing device 10 detects the product availability status in the surveillance camera image g3 based on the comparison results (step S14). Specifically, as shown in FIG. 6 , the image processing device 10 generates product detection information G4 by associating the product arrangement, i.e., the frame F, and the type of product E in the surveillance camera image g1 with the product area image G3. Then, the image processing device 10 detects the type of product that is lacking by checking the number of products based on the detection results of the product area E' within each frame F associated with the product detection information G4.
[0044] As described above, in this embodiment, the product layout is recognized from the high-quality mobile phone image g2 and associated with the low-quality surveillance camera image g1. Therefore, using this associated information, the status of the product, such as the number of products, can be detected from a surveillance camera image g3 captured later. As a result, the system for detecting the status of the products can reduce the cost of the device and image processing.
[0045] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any of the embodiments.
[0046] The image processing system of this embodiment has a configuration similar to that of the image processing system of the first embodiment. In addition, the image processing device 10 of this embodiment has the following functions. The following mainly describes the configuration that is different from the above-mentioned embodiment.
[0047] First, as shown in Fig. 9, the product quantity calculation unit 14 of the image processing device 10 detects a product area E' in the surveillance camera image g1 used when generating the association information G1, and calculates the position where the product may be present and the quantity of the product from the product area E'. For example, as described above, the product quantity calculation unit 14 can use a technique such as semantic segmentation to detect the product area E' in the surveillance camera image g1, as shown in image g1' in Fig. 9, and calculate the position and quantity of the product corresponding to the product area E'.
[0048] Furthermore, the association unit 13 of the image processing device 10 in this embodiment has a function of associating product placement information G2 on the mobile image g2 with the surveillance camera image g1, and further associating the product positions and quantities on the surveillance camera image g1. That is, the association unit 13 includes in the association information G1 the product positions and quantities on the surveillance camera image g1 in a situation where a sufficient number of products E are arranged on the shelf R and neatly aligned. As a result, the association unit 13 generates second association information G1' in which the product positions and quantities corresponding to the product area E' on the surveillance camera image g1 are further associated with the association information G1, as shown in FIG. 10 .
[0049] The product status detection unit 15 of the image processing device 10 in this embodiment then detects the product sufficiency status in the surveillance camera image g3 using the second association information G1' and the product positions and quantities (product detection information G4) estimated from the product area E' in the surveillance camera image g3 captured subsequently. Specifically, as shown in FIG. 11 , the product status detection unit 15 compares the product positions and quantities in the surveillance camera image g1 in a situation where a sufficient number of products E are arranged on the shelf R and neatly aligned, with the product positions and quantities in the subsequent surveillance camera image g3. This makes it possible to detect changes in the product positions and quantities in each frame F in the surveillance camera image g1 to the product positions and quantities in each frame F in the surveillance camera image g3, and to calculate the product sufficiency rate in the surveillance camera image g3.
[0050] Third Embodiment Next, a third embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any of the embodiments.
[0051] The image processing system of this embodiment has a configuration similar to that of the image processing system of the first embodiment. In addition, the image processing device 10 of this embodiment has the following functions. The following mainly describes the configuration that is different from the above-mentioned embodiment.
[0052] First, as in the first embodiment, the product quantity calculation unit 14 (product estimation unit) of the image processing device 10 detects a product area E' in a surveillance camera image g3 taken by the surveillance camera 21 after generating the association information G1, as shown in FIG. 12 , to generate a product area image G3, and calculates the possible positions of products and the quantity of products from the product area E'. In particular, the product quantity calculation unit 14 calculates the positions of products as the display status of the products relative to the shelf R. For example, as described above, the product quantity calculation unit 14 can detect the product area E' in the surveillance camera image g3 using a technique such as semantic segmentation, and calculate the positions (display status) and quantity of products corresponding to the product area E'.
[0053] The product situation detection unit 15 of the image processing device 10 in this embodiment detects the product situation in the surveillance camera image g3 using the product positions (display situation) estimated from the product area E' in the surveillance camera image g3 and the association information G1. Here, the product situation detection unit 15 detects the disorganized state of products displayed on the shelf R. Specifically, the product situation detection unit 15 first associates the association information G1 representing the product arrangement in the surveillance camera image g1 with the surveillance camera image g3. Here, as shown in FIG. 12 , the product situation detection unit 15 uses the association information G1 to associate product arrangement information, i.e., frames F representing positions where products may be arranged and the types of products E, with the product area image G3 generated from the surveillance camera image g3, thereby generating product detection information G4. The product situation detection unit 15 then checks whether a product is located in each frame F associated with the product detection information G4 based on the detection result of the product area E'. At this time, the product status detection unit 15 detects that the products are displayed in a disorderly manner if the product corresponding to the product area E' is not located within the frame F, for example, if part of the product is located outside the frame F and the display position is shifted. At this time, the product status detection unit 15 also detects the type of product associated with the frame F where the product display position is shifted.
[0054] In the example of product detection information G4 shown in the lower part of Fig. 12, the position of a product corresponding to product area E' is shifted in frame F' located on the right side of the top shelf R. Therefore, the product status detection unit 15 detects that the display positions of products located near frame F are messy. Furthermore, because the product type "coffee" is associated with the corresponding frame F', the product status detection unit 15 can also detect the product type "coffee" whose display position is messy.
[0055] Fourth Embodiment Next, a fourth embodiment of the present disclosure will be described. In this embodiment, an application example of the image processing system described above will be described.
[0056] In the above-described embodiment, an example was given of an image processing system being used to manage the status of merchandise displayed on shelves for sale in a store, such as the quantity of merchandise, but the image processing system of the present disclosure can also be applied to the medical healthcare field. In the following, a case will be described in which the image processing system of the present disclosure is used to manage the status of pharmaceuticals (merchandise) in a medical facility.
[0057] First, assume that in a medical facility such as a hospital or pharmacy, pharmaceuticals are displayed on shelves, and that a surveillance camera 21 is installed so that the shelves are within its capture range. Furthermore, assume that the shelves are captured by a mobile camera 22 operated by an employee or the like in the medical facility. In this situation, the image processing device 10 according to the embodiment described above can be used to detect the location and number of pharmaceuticals by processing the surveillance camera images g1 and g3 captured by the surveillance camera 21 and the mobile image g2 captured by the mobile camera 22 in the same manner as described above. This allows the availability and display status of pharmaceuticals (products) to be detected in the same manner as described above. Since the type of pharmaceutical can also be detected, it is possible to determine which pharmaceuticals are located where.
[0058] As described above, according to this embodiment, the location, supply status, and display status of each type of medicine can be detected in a medical facility, and by outputting the detection results, the location, supply status, and display status of each type of medicine can be notified to medical personnel such as doctors and nurses. This allows medical personnel to accurately recognize the location, supply status, etc. of each type of medicine, effectively preventing medicine mix-ups and out-of-stock situations, and enabling appropriate medical treatment to be provided.
[0059] Fifth Embodiment Next, a fifth embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the configuration of the image processing device described in the above-mentioned embodiments. Note that Figures 13 and 14 are diagrams for explaining the configuration, and these drawings may be relevant to any of the embodiments.
[0060] First, the hardware configuration of the image processing device 100 will be described with reference to Fig. 13. The image processing device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example: CPU (Central Processing Unit) 101 (arithmetic unit); ROM (Read Only Memory) 102 (storage device); RAM (Random Access Memory) 103 (storage device); programs 104 loaded into RAM 103; storage device 105 storing programs 104; drive device 106 for reading and writing data from and to a storage medium 110 external to the information processing device; communication interface 107 for connecting to a communication network 111 external to the information processing device; input / output interface 108 for inputting and outputting data; and bus 109 for connecting the various components.
[0061] 13 shows an example of the hardware configuration of an information processing device that is the image processing device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with a part of the above-described configuration, such as not including the drive device 106. Furthermore, instead of the above-described CPU, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.
[0062] The image processing device 100 can be equipped with an image acquisition unit 121, a product placement recognition unit 122, and an association unit 123 shown in FIG. 14 by having the CPU 101 acquire and execute the program group 104. The program group 104 is stored in advance in the storage device 105 or the ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, with the drive device 106 reading out the programs and supplying them to the CPU 101. However, the image acquisition unit 121, the product placement recognition unit 122, and the association unit 123 described above may be constructed using dedicated electronic circuits for realizing such means.
[0063] The image acquisition unit 121 acquires a first image captured by a first photographing device and a second image captured by a second photographing device, the second image including at least a portion of the photographing range of the first image and having higher quality than the first image. The product layout recognition unit 122 recognizes the layout of products on the second image based on the second image. The association unit 123 associates the layout of the products on the second image with the layout of the products on the first image.
[0064] As configured above, the present disclosure recognizes product layout from a high-quality second image and associates the product layout with a low-quality first image. Therefore, using this associated information, the status of the product can be detected from a low-quality first image captured later. As a result, the system for detecting the status of the product can reduce device costs and image processing costs.
[0065] In addition, at least one or more of the functions of the image acquisition unit 121, product placement recognition unit 122, and matching unit 123 described above may be executed by an information processing device installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.
[0066] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-RWs, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program can also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.
[0067] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each of the above-described embodiments can be combined with other embodiments as appropriate.
[0068] <Supplementary Notes> Some or all of the above embodiments can also be described as in the following supplementary notes. Below, an outline of the configurations of an image processing device, an image processing method, and a program according to the present disclosure will be described. However, the present disclosure is not limited to the following configurations. (Supplementary Note 1) An image processing device comprising: an image acquisition unit that acquires a first image captured by a first photographing device and a second image captured by a second photographing device, the second image including at least a portion of the photographing range of the first image and having higher quality than the first image; a product arrangement recognition unit that recognizes a product arrangement on the second image based on the second image; and a matching unit that matches the product arrangement on the second image with the first image. (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the image acquisition unit acquires a third image captured by the first photographing device; and a product situation detection unit that matches the product arrangement associated with the first image on the third image, and detects a product situation on the third image based on the third image. (Supplementary Note 3) The image processing device according to Supplementary Note 2, comprising: a product estimation unit that estimates products on the third image based on the third image, wherein the product status detection unit detects the product status on the third image based on product information estimated on the third image and the product arrangement associated on the third image. (Supplementary Note 4) The image processing device according to Supplementary Note 3, wherein the product estimation unit estimates the quantity of products on the third image based on the third image, and the product status detection unit detects the product sufficiency state on the third image based on the quantity of products estimated on the third image and the product arrangement associated on the third image. (Supplementary Note 5) The image processing device according to Supplementary Note 3, wherein the product estimation unit estimates the product display status on the third image based on the third image, and the product status detection unit detects the product clutter state on the third image based on the product display status estimated on the third image and the product arrangement associated on the third image.(Supplementary Note 6) The image processing device according to Supplementary Note 3, wherein the product estimation unit estimates products on the first image based on the first image, and the product status detection unit detects a fulfillment status of the products on the third image based on the product information estimated on the first image and the product arrangement associated on the first image, and the product information estimated on the third image and the product arrangement associated on the third image. (Supplementary Note 7) The image processing device according to Supplementary Note 1, wherein the product arrangement recognition unit recognizes the arrangement position of each product type on the second image, and the association unit associates the arrangement positions of each product type on the second image with the first image. (Supplementary Note 8) The image processing device according to Supplementary Note 1, wherein the association unit associates the arrangement of the products on the second image with the first image based on the corresponding positions on the first image and the second image. (Supplementary Note 9) The image processing device according to Supplementary Note 1, wherein the association unit associates the arrangement of the products on the second image with the first image selected according to the time the second image was captured. (Supplementary Note 10) The image processing device according to Supplementary Note 1, wherein the product arrangement recognition is configured to use a machine learning model constructed in advance by machine learning to input the second image and output the arrangement of the products on the second image, or the association unit is configured to use a machine learning model constructed in advance by machine learning to input the first image and the arrangement of the products on the second image and output information associating the arrangement of the products with the first image. (Supplementary Note 11) An image processing method comprising: acquiring a first image captured by a first imaging device and a second image captured by a second imaging device, which includes at least a part of the imaging range of the first image and is of higher quality than the first image; recognizing the arrangement of the products on the second image based on the second image; and associating the arrangement of the products on the second image with the first image.(Supplementary Note 12) The image processing method according to Supplementary Note 11, comprising: acquiring a third image captured by the first photographing device; associating the product arrangement associated with the first image on the third image; and detecting the product status on the third image based on the third image. (Supplementary Note 13) The image processing method according to Supplementary Note 12, comprising: estimating products on the third image based on the third image; and detecting the product status on the third image based on product information estimated on the third image and the product arrangement associated on the third image. (Supplementary Note 14) The image processing method according to Supplementary Note 13, comprising: estimating the quantity of products on the third image based on the third image; and detecting the product fulfillment status on the third image based on the quantity of products estimated on the third image and the product arrangement associated on the third image. (Supplementary Note 15) A computer-readable storage medium storing a program that causes a computer to execute the following processes: acquiring a first image captured by a first photographing device and a second image captured by a second photographing device, the second image including at least a portion of the photographing range of the first image and of higher quality than the first image; recognizing the arrangement of products on the second image based on the second image; and associating the arrangement of the products on the second image with the arrangement of the products on the first image.
[0069] REFERENCE SIGNS LIST 10 Image processing device 11 Image acquisition unit 12 Product placement recognition unit 13 Correspondence unit 14 Product quantity calculation unit 15 Product status detection unit 16 Image storage unit 17 Correspondence information storage unit 21 Surveillance camera 22 Mobile camera 100 Image processing device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Image acquisition unit 122 Product placement recognition unit 123 Correspondence unit
Claims
1. An image acquisition unit that acquires a first image captured by a first imaging device and a second image captured by a second imaging device, which includes at least a portion of the imaging range of the first image and is of higher quality than the first image, A product placement recognition unit recognizes the placement of products on the second image based on the second image, A correspondence unit that maps the arrangement of the product on the second image to the first image, Equipped with an image processing device.
2. An image processing apparatus according to claim 1, The image acquisition unit acquires the third image captured by the first imaging device, The system includes a product status detection unit that maps the arrangement of the product, which is associated with the first image, onto the third image, and detects the status of the product on the third image based on the third image. Image processing device.
3. An image processing apparatus according to claim 2, The system includes a product estimation unit that estimates the products on the third image based on the aforementioned third image. The product status detection unit detects the status of the product on the third image based on the product information estimated on the third image and the arrangement of the product associated with the third image. Image processing device.
4. An image processing apparatus according to claim 3, The product estimation unit estimates the quantity of products on the third image based on the third image, The product status detection unit detects the product availability status on the third image based on the estimated quantity of products on the third image and the arrangement of the products on the third image. Image processing device.
5. An image processing apparatus according to claim 3, The product estimation unit estimates the display arrangement of the products on the third image based on the third image, The product status detection unit detects the state of clutter in the third image based on the estimated display status of the products on the third image and the arrangement of the products corresponding to the third image. Image processing device.
6. An image processing apparatus according to claim 3, The product estimation unit estimates the product on the first image based on the first image, The product status detection unit detects the availability status of the product on the third image based on the product information estimated on the first image and the arrangement of the product on the first image, and the product information estimated on the third image and the arrangement of the product on the third image. Image processing device.
7. An image processing apparatus according to claim 1, The product placement recognition unit recognizes the placement position of each type of product on the second image, The correspondence unit maps the placement positions of each type of product on the second image to the first image. Image processing device.
8. An image processing apparatus according to claim 1, The matching unit matches the arrangement of the product on the second image to the first image based on the corresponding positions on the first image and the second image. Image processing device.
9. The information processing device is A first image captured by a first imaging device and a second image captured by a second imaging device, which includes at least a portion of the imaging range of the first image and is of higher quality than the first image, are obtained. Based on the second image, the arrangement of the products on the second image is recognized. The arrangement of the product on the second image is mapped to the first image. Image processing methods.
10. A first image captured by a first imaging device and a second image captured by a second imaging device, which includes at least a portion of the imaging range of the first image and is of higher quality than the first image, are obtained. Based on the second image, the arrangement of the products on the second image is recognized. The arrangement of the product on the second image is mapped to the first image. A program that instructs a computer to perform a process.