Imaging device, control method of imaging device, imaging system, and program

Abstract

To appropriately acquire the occupancy ratio of a subject in an imaging area even when a size difference occurs in a subject image projected in a viewing angle of an imaging device.SOLUTION: The imaging device is provided with: an imaging unit for acquiring an image; an image recognizing unit for extracting an area in which one or more specific subjects are located as a second region in a first region, which is a predetermined area in the image; a distance acquiring unit for acquiring distance information between the imaging device and the first region, and distance information between the imaging device and the second region, respectively, and a calculating unit for calculating the occupancy ratio of the second region relative to the first region based on an area size of an area corresponding to the first region and an area size of an area corresponding to the second region in a corrected image generated based on the respective distance information acquired by the distance acquiring unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, a control method for an imaging apparatus, an imaging system, and a program. [Background technology]

[0002] In stores such as convenience stores and supermarkets, if there are not enough products on the shelves, sales opportunities will be lost and sales will be significantly affected. Therefore, it is necessary to carry out product management work such as quickly replenishing products when there is a shortage of products on the shelves.

[0003] In such product management work, a technology is known that determines the need for product replenishment and issues a notification based on images captured (pictured) in the product display area (Patent Document 1). In Patent Document 1, the need for product replenishment is determined by performing image recognition processing on a predetermined monitoring area, and the ratio of the area in which the target product is detected to the area of ​​the monitoring area is calculated as the product occupancy rate. The need for product replenishment is determined and notified by monitoring the product occupancy rate over time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6341388 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in Patent Document 1, the imaging device is not only installed facing the display shelf of the products, but is often installed at an angle of view diagonally to the display shelf in order to monitor a wider range. In such a case, even if the products are the same size, the products further away from the imaging device are photographed smaller depending on the distance from the imaging device. In this case, for example, when a product in front of the imaging device is missing and when a product located far away from the imaging device is missing, there may be a difference in the missing area relative to the monitoring area in the captured image, even though the products of the same size are missing. As a result, there may be cases where the correct product occupancy rate cannot be calculated.

[0006] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to make it possible to appropriately obtain the occupancy rate of a subject in an imaging area even when a difference in size occurs in the subject image projected within the angle of view of an imaging device. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, an imaging device according to one aspect of the present invention includes an imaging unit that acquires an image, an image recognition unit that extracts an area in which one or more specific objects are located in a first area, which is a predetermined area in the image, as a second area, a distance acquisition unit that acquires distance information between the imaging device and the first area and distance information between the imaging device and the second area, respectively, and a calculation unit that calculates the occupancy rate of the second area relative to the first area based on an area of ​​an area corresponding to the first area and an area corresponding to the second area in a corrected image generated based on the respective distance information acquired by the distance acquisition unit. Effect of the Invention

[0008] According to the present invention, even if a difference in size occurs in the subject image projected within the angle of view of the imaging device, it is possible to appropriately obtain the occupancy rate of the subject in the imaging area. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a schematic diagram showing the functional configuration of an imaging device and a display device according to an embodiment. [Diagram 2] 1A and 1B are schematic diagrams illustrating an installation state of an imaging device according to an embodiment and a captured image. [Diagram 3] 5A and 5B are schematic diagrams illustrating processing of the imaging device according to the embodiment. [Figure 4] 4 is a flowchart showing a processing flow according to the embodiment. [Diagram 5] 1 is a flowchart showing a processing flow according to Example 1 of the embodiment. [Figure 6] FIG. 11 is a schematic diagram for explaining a process according to Example 2 of the embodiment. [Figure 7] 13 is a flowchart showing a processing flow according to Example 2 of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the invention related to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, reference numbers are given to the same or similar configurations, and duplicated descriptions are omitted.

[0011] <Embodiment> An embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the functional configuration of an imaging device 100 according to this embodiment and the functional configuration of a display device (information processing device) 190. Although details will be described later, the imaging device 100 in this embodiment also functions as an image processing device. In this embodiment, the imaging device 100 and the display device 190 are connected to each other so as to be able to communicate with each other via a network 181. Also, in this embodiment, the imaging device 100 and the display device 190 form an imaging system.

[0012] The imaging device 100 is composed of an imaging section 110, an image processing section 120, a distance acquisition section 130, a storage section 140, an image recognition section 150, a calculation section 160, a control section 170, and a communication section 180. Although details will be described later, the imaging device 100 is capable of calculating and determining a product occupancy rate as an index showing a stock status, and notifying a user of the same by displaying it on a display device 190 via the communication section 180. The imaging device 100 has the distance acquisition section 130, and is therefore capable of measuring the distance to a product (specific object), and is able to accurately calculate the product occupancy rate by correcting an image using this distance measurement information.

[0013] The imaging unit 110 has an imaging optical system 111 and an imaging element 112. Light transmitted through the imaging optical system 111 is imaged by the imaging element 112 and converted into an electrical signal, which is output as image data through processing by the image processing unit 120. The imaging optical system 111 is composed of a plurality of lenses and a holding member. The imaging element 112 is composed of a solid-state imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In this embodiment, the imaging device 100 captures images of the inside of a store. Specifically, it captures images of a display area in the store. This allows the imaging unit 110 to acquire images (image acquisition) of one or more products arranged in the display area of ​​the store. However, this is not limited to this, and this embodiment can also be applied to cases where, for example, places other than a store, such as a product storage room or a loading area, or a specific object (item) to be detected is monitored.

[0014] The distance acquisition unit 130 has a distance measurement sensor (distance sensor) capable of measuring distances to at least three or more different points within the shooting (imaging) range in order to obtain the surface of the display shelf. The distance acquisition unit 130 can calculate the subject distance (distance information) from the imaging device 100 based on the position relative to the optical axis of the imaging device 100 and on attitude information. Here, the distance measurement sensor is an existing distance measurement sensor capable of acquiring distances by a Time Of Flight (TOF) method, a phase difference detection method, or a triangulation method using light or ultrasonic waves. The distance measurement sensor may be capable of acquiring multiple pieces of distance information, mapping them, and outputting them as a single image, or may be capable of measuring the distance to a single point.

[0015] In addition, in the case of a distance measurement sensor that obtains distance information for only one point, the distance acquisition unit 130 is further configured to have a drive unit (not shown) and to acquire multiple distances by scanning within the shooting range (image capture range). The distance acquisition unit 130 can spatially model the position and height of the display shelves by performing distance measurement in advance within the store. The distance acquisition unit 130 is controlled by a control signal from the control unit 170 as to the timing and position at which the distance is acquired. The distance acquisition unit 130 may be disposed outside the imaging device 100. In that case, it is sufficient that the distance acquisition unit 130 is connected to the imaging device 100 via a line, such as wired or wireless, so as to be able to communicate with the imaging device 100.

[0016] The storage unit 140 is composed of a ROM (Read Only Memory) which is a non-volatile memory and a RAM (Random Access Memory) which is a volatile memory. A control program for the imaging device 100 is stored in the storage unit 140. The storage unit 140 also stores a trained neural network model used in the image recognition unit 150 described later, distance information of a monitoring area (first area), information on product occupancy calculated by the calculation unit 160, and the like. The storage unit 140 may also store size information of products to be displayed, and the like.

[0017] The control unit 170 is composed of a CPU, an MPU, etc., and is connected via a line to each component of the imaging device 100. The control unit 170 also performs overall control of the operation adjustment of each component of the entire imaging device 100 according to a program stored in the storage unit (memory) 140.

[0018] Furthermore, the control unit 170 controls the operation of the distance acquisition unit 130, such as transmitting an execution signal for the distance acquisition unit 130 to acquire a distance. In particular, the control unit 170 controls the distance acquisition unit 130 to acquire at least three or more coordinates that specify the inclination of the monitoring area in which the products are arranged. The area of ​​the monitoring area is treated as a denominator when calculating the product occupancy rate, which is an index of the inventory status of the products. Therefore, it is desirable to set an area in which the same products are displayed as one monitoring area. At least one monitoring area is provided for one image. The monitoring area is set by the user via the operation unit 194 of the display device 190, for example. Furthermore, the monitoring area may be set by executing an image recognition process on the display shelf after the display is completed to determine the area in which the same products are displayed.

[0019] The image recognition unit 150 identifies products using a neural network model (hereinafter, NN model) read from the storage unit 140 for the monitoring area in the captured image (captured image) sent from the image processing unit 120. In this way, the image recognition unit 150 extracts the area where the products are displayed (hereinafter, product area). The NN model is a model that extracts the area where the learned products exist. The extraction result is sent to the calculation unit 160. The area of ​​the product area (second area) is used as the numerator when calculating the product occupancy rate, which is an index of the product inventory status.

[0020] The calculation unit 160 uses the distance to the point on the surface on which the product is placed, acquired by the distance acquisition unit 130, to output an image in which the size difference of the product that occurs in the captured image due to the distance from the imaging device 100 to the product has been corrected. The correction by the calculation unit 160 is performed using coordinate transformation such as projective transformation, which is a well-known technique. Details will be described later. After that, the calculation unit 160 extracts (calculates) the area of ​​the monitoring area and the area of ​​the product area in the image in which the size difference has been corrected, and calculates the product occupancy rate using the following formula (1). The area is calculated from the total number of pixels in the target area.

[0021] Product occupancy rate = (area of ​​product area) / (area of ​​monitoring area) (1)

[0022] The communication unit 180 is a network processing circuit. The communication unit 180 can convert, for example, image data and data indicating a product occupancy rate into a communication signal conforming to a communication protocol and transmit the signal to the network 181.

[0023] The display device 190 is an information processing device that is connected to the imaging device 100 via the network 181 and can communicate with the imaging device 100. The display device 190 may be, for example, a personal computer or a tablet terminal. The display device 190 has, for example, a communication unit 191, a control unit 192, a display unit 193, an operation unit 194, and a storage unit 195.

[0024] The control unit 192 is composed of a CPU, an MPU, and the like, and is connected via a line to each component of the display device 190. The control unit 192 also performs overall control of the operation adjustment of each component of the entire display device 190 according to a program stored in a storage unit (memory) 195.

[0025] Furthermore, the control unit 192 can control the imaging device 100 by transmitting control information (commands, control signals) for controlling the imaging device 100 to the imaging device 100 via the communication unit 191. Note that the control unit 192 may transmit a control signal to the control unit 170 of the imaging device 100 and control the imaging device 100 via the control unit 170, or may transmit a control signal to each unit of the imaging device 100 to control the imaging device 100.

[0026] The display unit 193 can receive and display image data distributed by the imaging device 100 via the communication unit 191. The display device 190 can also store a product occupancy threshold value designated by the user in advance in the storage unit 195, and when the product occupancy rate falls below the threshold value, display a lack of products on the display unit 193 to notify the user. The storage unit 195 is composed of a ROM (Read Only Memory) which is a non-volatile memory and a RAM (Random Access Memory) which is a volatile memory. The storage unit 195 stores a control program for the display device 190. Note that, like the storage unit 140 of the imaging device 100, a trained neural network model, distance information of a monitoring area, product occupancy information, and the like may be stored. In addition, the storage unit 140 may store size information of products to be displayed, and the like.

[0027] The operation unit 194 accepts operation input from the user via the display unit 193. The user can control the imaging device 100 or the display device 190 by operating a mouse, a keyboard, or the like according to a user interface such as icons displayed on the display unit 193. In particular, it is possible to set a monitoring area by specifying a position on the captured image displayed on the display unit 193.

[0028] Hereinafter, the processing of the calculation unit 160 will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a schematic diagram showing an installation state of the imaging device 100 according to this embodiment and a captured image. Fig. 3 is a schematic diagram for explaining the processing of the imaging device 100 according to the embodiment.

[0029] Fig. 2(A) illustrates the relationship between the imaging device 100 and a product display shelf 200. Fig. 2(B) is a diagram showing an example of a captured image 220. It is assumed that the imaging device 100 of this embodiment captures an image of the product display shelf 200 from an oblique direction. On the display shelf 200 shown in Fig. 2(A), products 201, 202, and 203, as well as products 204, 205, and 206 (not shown) are displayed. It is assumed that the products 201 to 206 shown in Fig. 2 are the same product.

[0030] A dashed line 210 indicates the photographing angle of view of the imaging device 100. An image photographed at the photographing angle of view and in the installation environment in FIG. 2(A) is a photographed image 220 shown in FIG. 2(B). As shown in FIG. 2(B), the products 201-206 displayed on the display shelf 200 are photographed from an oblique angle, so that the products 201 and 204 that are farther from the imaging device 100 are photographed smaller, and the products 203 and 206 that are closer to the imaging device 100 are photographed larger. A monitoring area 221 indicates a monitoring area in which the same products 201-206 are displayed (hereinafter, referred to as the monitoring area 221). Points A, B, C, and D shown in FIG. 2(B) explicitly indicate points that specify the monitoring area 221.

[0031] FIG. 3(A) is a diagram showing an example of a captured image showing a state in which the product 201 and the product 202 are out of stock in FIG. 2(B). FIG. 3(B) is a diagram showing a result of image recognition processing performed by the image recognition unit 150 on the captured image 220 in FIG. 3(A). In FIG. 3(B), a product area 301 is extracted by the image recognition processing. Points E, F, G, H, I, and J shown in FIG. 3(B) explicitly show points that specify the product area 301. FIG. 3(C) is a diagram showing a result of superimposing the monitoring area 221 in FIG. 3(A) and the product area 301 in FIG. 3(b) that are projected onto the same plane and the size difference is corrected. The monitoring area 311 shown in FIG. 3(C) is the area after the monitoring area 221 is corrected, and the product area 312 is the area after the product area 301 is corrected.

[0032] In this embodiment, the distance acquisition unit 130 first acquires the distance to points A to D that identify the monitoring area 221 (acquires distance information). Next, the calculation unit 160 detects the surface of the monitoring area 221 using the distance information acquired by the distance acquisition unit 130, and obtains a transformation matrix for projecting and transforming the surface onto a plane perpendicular to the optical axis direction of the imaging device 100. The result of transformation using this transformation matrix is ​​the monitoring area 311 shown in FIG. 3(C). The calculation unit 160 then performs the same process on the product area 301, calculates the product area 312 shown in FIG. 3(C), and outputs this. At this time, the plane onto which the monitoring area 221 and the product area 301 are projected is the same plane.

[0033] Hereinafter, a processing flow of the imaging device 100 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the processing flow according to this embodiment. Each process described below is realized by the control unit 170 of the imaging device 100 executing a program stored in the storage unit 140. Also, by adding an S to the beginning of each process (step), the notation of the process (step) is omitted.

[0034] First, in S401, a transformation matrix (first transformation information) for projecting and transforming onto a plane perpendicular to the optical axis direction of the image capturing device 100 is calculated. Specifically, first, the distance acquisition unit 130 acquires distance information for points A to D of the monitoring area 221 designated for the captured image from the image capturing device 100 (or the image capturing unit 110). Then, based on the acquired distance information, the surface of the monitoring area 221 is detected. That is, the surface of the monitoring area 221 is detected from the acquired distance information, and surface information of the monitoring area 221 is acquired. Then, the distance acquisition unit 130 calculates a transformation matrix (first transformation information) for projecting and transforming onto a plane perpendicular to the optical axis direction of the image capturing device 100, using the acquired surface information of the monitoring area 221. Then, the process proceeds to S402.

[0035] Next, in S402, the image recognition unit 150 executes image recognition processing on the captured image 220 to extract the product area 301. Specifically, the image recognition processing is executed on the captured image 220 to extract the product area 301 from the monitored area 221 as an area different from the monitored area 221. Then, the process proceeds to S403.

[0036] Next, in S403, a transformation matrix (second transformation information) for projecting and transforming onto a plane perpendicular to the optical axis direction of the imaging device 100 is calculated. Specifically, first, the distance acquisition unit 130 acquires distance information for points E to J that identify the product area 301 extracted in S402, and detects the surface of the product area 301. That is, the surface of the product area 301 is detected from the acquired distance information from the imaging device 100 to the product area, and surface information of the product area 301 is acquired. Then, the distance acquisition unit 130 uses the acquired surface information of the product area 301 to calculate a transformation matrix (second transformation information) for projecting and transforming onto a plane perpendicular to the optical axis direction of the imaging device 100. Thereafter, the process proceeds to S404.

[0037] Next, in S404, the calculation unit 160 generates and acquires a corrected image by correcting the image. Specifically, the calculation unit 160 corrects the size difference of the monitoring area 221 in the captured image 220 using the transformation matrix of the monitoring area 221 acquired in S401, and outputs the monitoring area 311. Furthermore, the calculation unit 160 corrects the size difference of the product area 301 in the captured image 220 using the transformation matrix of the product area 301 acquired in S403, and outputs the product area 312. As a result, a corrected image in which the size difference in each area (including the product) is corrected is generated. Then, the process proceeds to S405.

[0038] Next, in S405, the calculation unit 160 extracts the area of ​​the monitoring area 311 and the area of ​​the product area 312 from the corrected image generated in S404. The areas of the monitoring area 311 and the product area 312 can be found (extracted) by the calculation unit 160 counting the number of pixels in the corresponding areas from the corrected image. Then, the extracted areas of the monitoring area 311 and the product area 312 are substituted into the above formula (1) for calculation. This makes it possible to calculate the product occupancy rate. In other words, the calculation unit 160 calculates the occupancy rate of the product area 312 (second area) relative to the corrected monitoring area 311 (first area).

[0039] In addition, when the surface of the product area 312 detected in S403 can be considered to be the same (or approximately the same) as the surface of the monitoring area 221, it is not necessary to detect the surface of the product area 312 and calculate the transformation matrix again. That is, in this case, the monitoring area 221 may also be treated as the product area 312.

[0040] As described above, by performing image recognition processing and then correcting the image, it is possible to appropriately obtain the occupancy rate of the subject in the imaging area while maintaining the accuracy of image recognition, even if a size difference occurs in the subject image projected within the angle of view of the imaging device 100. This makes it possible to reduce false detections. In this way, the imaging device 100 in this embodiment also functions as an image processing device. Below, examples 1 and 2 of this embodiment will be described.

[0041] <Example 1> Fig. 5 is a flowchart showing a process flow of Example 1 of this embodiment. An example of a process flow of a system for managing product inventory status in Example 1 of this embodiment will be described below with reference to Figs. 3 and 5. Each process below is realized by the control unit 170 of the imaging device 100 executing a program stored in the storage unit 140. Also, each process (step) is represented with an S at the beginning to omit the representation of the process (step).

[0042] In the process in Fig. 5, branching based on the distance acquisition unit 130 determining whether there is an increase or decrease in the number of products placed on the shelf (S504), and determining that there is depth in the monitoring area (S508), and a notification process to the display device 190 (S512), etc. are added to the process in Fig. 4. Also, detailed explanations of the processes in Fig. 5 that perform the same operations as those in Fig. 4 will be omitted.

[0043] First, in S501, the control unit 170 judges whether or not the setting information of the monitoring area 221 has been received (whether or not the control signal related to the setting information of the monitoring area 221 has been received). If it is judged that the control signal related to the setting information of the monitoring area 221 has been received, the process proceeds to S502. On the other hand, if it is judged that the control signal related to the setting information of the monitoring area 221 has not been received, the process waits until a command for the setting information of the monitoring area 221 is received. Note that, when setting the monitoring area 221, for example, the user may set the monitoring area 221 by operating the operation unit 194 of the display device 190 on a setting screen displayed on the screen of the display unit 193. When the monitoring area 221 is set, the setting information of the monitoring area 221 is transmitted to the imaging device 100. At this time, the setting information is transmitted from the communication unit 191 of the display device 190 to the imaging device 100 via the network 181. Then, the setting information of the monitoring area 221 transmitted to the imaging device 100 is transmitted to the control unit 170 via the communication unit 180. In this processing flow, the control unit 170 also functions as a determination unit during the determination process.

[0044] Next, in S502, the distance acquisition unit 130 performs the same process as S401. That is, the distance acquisition unit 130 acquires distance information for points A to D of the monitoring area 221 specified for the captured image from the imaging device 100 (or the imaging unit 110), and calculates surface information of the monitoring area 221 from the acquired distance information. Then, the calculated surface information is used to obtain a transformation matrix of the monitoring area 221. After that, the process proceeds to S503.

[0045] Next, in S503, the distance acquisition unit 130 acquires distance information (object distance information) from the imaging device 100 to a commodity placed (displayed) on the display shelf 200. The acquired distance information to the commodity is stored in the storage unit 140, and the process proceeds to S504.

[0046] Next, in S504, the control unit 170 determines whether or not the distance from the imaging device 100 to the product placed on the display shelf 200 has changed. If it is determined that the distance from the imaging device 100 to the product placed on the display shelf 200 has changed, it is determined that the display state (stock status) of the product has changed, and the process proceeds to S505. On the other hand, if it is determined that the distance from the imaging device 100 to the product placed on the display shelf 200 has not changed, the process returns to S503 and repeats the same process.

[0047] Next, in S505, the control unit 170 updates the distance information to the product. The updated distance information to the product is stored in the storage unit 140. Then, the process proceeds to S506. Next, in S506, the image recognition unit 150 performs the same process as S402. That is, the image recognition process is performed on the captured image 220 to extract the product area 301. Then, the process proceeds to S507.

[0048] Next, in S507, the distance acquisition unit 130 performs the same process as in S403. That is, the surface of the product area 301 is detected, and a transformation matrix of the product area 301 is calculated based on surface information of the detected product area 301. Then, the process proceeds to S508.

[0049] Next, in S508, the control unit 170 uses the surface information of the monitoring area 221 detected in S502 to determine whether or not the monitoring area 221 has depth (whether there are products displayed close to and far from the imaging device 100). If it is determined that the monitoring area 221 has depth, the process proceeds to S509. On the other hand, if it is determined that the monitoring area 221 does not have depth, the process proceeds to S510.

[0050] Here, the control unit 170 determines the depth in the monitoring area based on the inclination of the surface of the detected monitoring area 221 with respect to a surface perpendicular to the optical axis of the imaging device 100. A threshold value for this inclination at which the accuracy of the product occupancy rate deteriorates is stored in advance in the storage unit 140. The display device 190 may have a UI (user interface) that allows the user to freely change it later. The process of S508 makes it possible to distinguish, for example, a case in which the imaging device 100 is placed directly facing the display shelf 200, no size difference occurs in the captured image due to the distance to the product, and no image correction is required.

[0051] Next, in S509, the calculation unit 160 performs the same process as S404. That is, a corrected image is generated by correcting the image. Then, the process proceeds to S510. Next, in S510, the calculation unit 160 performs the same process as S405. That is, a product occupancy rate is calculated. Then, the process proceeds to S511.

[0052] Next, in S511, the calculation unit 160 compares the product occupancy rate calculated in S509 with a threshold value, and determines whether or not the product occupancy rate is less than the threshold value. If the product occupancy rate is less than the threshold value, it is determined that product replenishment is necessary, and the process proceeds to S512. On the other hand, if the product occupancy rate is equal to or greater than the threshold value, it is determined that product replenishment is not necessary, and the process proceeds to S513. Here, the threshold value is, for example, a value that can detect the timing when product replenishment is necessary, and can be freely set by the user via the display device 190, etc.

[0053] Next, in S512, the control unit 170 notifies that the product occupancy rate is equal to or greater than the threshold. Specifically, the control unit 170 displays a message or the like indicating that the number of products has fallen below the threshold on the display unit 193 via the communication unit 180, the network 181, and the communication unit 191 of the display device 190. Then, the process proceeds to S513. In S512, the control unit 170 also functions as a display control unit.

[0054] Finally, in S513, the control unit 170 determines whether or not to end the process. As an example, it determines whether a control signal (command) related to an operation to stop shooting by the imaging device 100 or the process related to this flow has been received, or whether an operation to turn off the power supply of the imaging device 100 has been performed. Then, if it is determined that the process is to be ended, the process in this flow ends. On the other hand, if it is determined that the process is not to be ended, the process returns to S503, and the same processes from S503 to S512 are repeated. Note that it may be arranged to wait in S503 until an instruction is received from the user (a control signal to start the process, etc. is transmitted from the display device 190).

[0055] The above is the flow for explaining the processing in the embodiment 1. Furthermore, hereinafter, a variation of the processing in FIG. 5 showing the processing in the embodiment 1 will be explained.

[0056] In S501, when setting the monitoring area, the user designates it using the operation unit 194 of the display device 190, but this is not limiting. For example, the image recognition unit 150 may execute image recognition processing on the display shelf after the display of the products is completed, and designate the area where the same products are displayed as the monitoring area.

[0057] In S504, the control unit 170, for example, monitors the height of the product placed on the display shelf by the distance acquisition unit 130, and determines that the distance has changed when the height has changed, but is not limited to this. For example, the control unit 170 may compare the latest acquired distance information to the product with the distance information to the product acquired immediately before the latest acquired distance information to the product, and determine whether or not the distance from the imaging device 100 to the product placed on the display shelf 200 has changed. That is, the control unit 170 may determine that the distance information to the product placed on the display shelf 200 has changed when there is a difference (change in distance) between the distance information to the product acquired at a predetermined interval. Also, for example, the control unit 170 may perform shooting at a certain time interval, and determine that the distance information to the product placed on the display shelf 200 has changed when a difference occurs between the captured images. Alternatively, during normal times, the control unit 170 may execute a detection process such as object detection, and start the determination process of S504 with an event such as the detection of a human hand as a trigger. Furthermore, the determination process may be started with a regular time as a trigger.

[0058] In S512, a process is performed to display on the display unit 193 of the display device 190 that the inventory is low. During this display control process, the display unit 193 displays a message such as characters or a frame indicating that the inventory is low on the display screen. Alternatively, the display device 190 can store the product occupancy rate transmitted from the imaging device 100 in the storage unit 195 together with the reception time, and display it on the screen of the display unit 193, for example, as a graph with the horizontal axis being time and the vertical axis being product occupancy rate. In addition, the display device 190 may have an alert (not shown) and sound this to notify the operator or worker of the inventory status of the product. The above-mentioned message and graph may be displayed on the screen of the display unit 193 together. Furthermore, the above-mentioned alert may be used to notify the user together with the display of the message and graph, or these may be combined to notify the user.

[0059] In addition, the processing flow shown in Fig. 5 has been described for the case where one monitoring area is set, but multiple monitoring areas can be set on one captured image. When multiple monitoring areas are set, the processing flow shown in Fig. 5 is executed for each monitoring area. The processing for each monitoring area may be performed in parallel or in conjunction with each other, and once the processing for one monitoring area is completed, the processing for the next monitoring area to be processed may be started.

[0060] In addition, some or all of the functions of each functional unit of the imaging device 100 shown in FIG. 1 may be included in a device other than the imaging device 100. Specifically, a server, an information processing device (for example, the display device 190), or a storage device other than the imaging device 100 may have these functional units, and the functions of each embodiment may be realized by communicating with the imaging device 100 based on a wired or wireless connection. For example, the image recognition unit 150, the calculation unit 160, and the control unit 170 of the imaging device 100 may be a server or an information processing device as a device other than the imaging device 100, and in that case, the processing in each of these units may be executed by the server, the information processing device, or the like. In addition, some or all of the functions of each functional unit of the display device 190 may be included in a device other than the display device. Specifically, a server, an information processing device, an imaging device (for example, the imaging device 100), or a storage device other than the display device may have these functional units, and the functions of each embodiment may be realized by communicating with the display device 190 based on a wired or wireless connection. The system of this embodiment can also be realized with the above configuration.

[0061] As described above, according to the imaging device 100 of the first embodiment, even if there is a size difference in the subject image projected within the angle of view of the imaging device, the occupancy rate of the subject in the imaging area can be appropriately acquired. This makes it possible to reduce false detections.

[0062] <Example 2> In Example 2 of this embodiment, a method of calculating the product occupancy rate by volume ratio will be described. Even when calculating the product occupancy rate by volume ratio, it is possible to appropriately obtain the occupancy rate of the subject in the imaging area as in Example 1, and to reduce false positives.

[0063] Hereinafter, a case where a product occupancy rate is calculated based on a volume ratio according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a schematic diagram for explaining processing of the imaging device 100 according to the second embodiment. Fig. 6(A) shows a captured image 220 captured by the imaging device 100. In the captured image 220 shown in Fig. 6(A), a state where a product 611, a product 612, a product 613, and a product 614 are displayed on a display shelf is captured. It is assumed that the products 611, 612, and 613 are the same products stacked in two layers and displayed, and that the product 611 has only one layer with nothing stacked thereon.

[0064] Also, the points specifying the monitoring area 221 are explicitly designated as points A, B, C, and D. FIG. 6(B) shows the result of image recognition processing performed by the image recognition unit 150 on the captured image 220 in FIG. 6(A). In FIG. 6(B), product areas 622 and 623 are extracted by the image recognition processing. Here, the points specifying the product area 622 are designated as points K, L, M, and N. The points specifying the product area 623 are designated as points O, P, Q, and R. FIG. 6(C) shows the result of superimposing the monitoring area 221 in FIG. 6(A) and the product areas 622 and 623 in FIG. 6(B) after projecting and transforming them onto the same plane perpendicular to the optical axis of the imaging device 100 and correcting the size difference. 6C, a monitoring area 631 is the area after the correction of the monitoring area 221, a product area 632 is the area after the correction of the product area 622, and a product area 633 is the area after the correction of the product area 622.

[0065] Fig. 7 is a flowchart showing processing in Example 2 of this embodiment. Processing in Example 2 of this embodiment will be described below with reference to Figs. 6 and 7. Also, a description of operations similar to those in the flow shown in Fig. 4 will be omitted. Each of the following processes is realized by the control unit 170 of the imaging device 100 executing a program stored in the storage unit 140. Also, each process (step) is represented by adding an S to the beginning, and the representation of the process (step) will be omitted.

[0066] First, in S701, the distance acquisition unit 130 performs the same process as S401. That is, the distance acquisition unit 130 acquires distance information for points A to D of the monitoring area 221 specified for the captured image from the imaging device 100 (or the imaging unit 110), and calculates surface information of the monitoring area 221 from the acquired distance information. Then, the calculated surface information is used to obtain a transformation matrix of the monitoring area 221. After that, the process proceeds to S702.

[0067] Next, in S702, the distance acquisition unit 130 acquires distance information to a product placed (displayed) on the display shelf 200. Then, the distance information from the imaging unit 110 to the display shelf (the position where the product is placed) that was previously acquired is subtracted from the distance information to the product. This acquires height information (second height information) of the product itself placed (displayed) on the display shelf 200. Then, the process proceeds to S703.

[0068] Next, in S703, the image recognition unit 150 performs the same process as in S402. That is, the image recognition unit 150 executes image recognition processing on the captured image 220 to extract the product area 622 (second area) and the product area 623 (second area). Then, the process proceeds to S704.

[0069] Next, in S704, the distance acquisition unit 130 obtains a transformation matrix for the product area 622 for each area of ​​the same height using the height information of the product itself obtained in S702. In this case, the transformation matrices for the product area 622 and the product area 623 are obtained. Note that the calculation of the transformation matrix is ​​the same as in S403, so a detailed description will be omitted. Then, proceed to S705.

[0070] Next, in S705, the calculation unit 160 performs the same process as in S404. That is, a corrected image is generated by correcting the image. Specifically, the calculation unit 160 corrects the size difference of the monitoring area 221 using the transformation matrix of the monitoring area 221 acquired in S701, and outputs the monitoring area 631. Similarly, the calculation unit 160 corrects the size difference of the product areas 622 and 623 using the transformation matrices of each product area acquired in S704, and outputs the product areas 632 and 633. Then, the process proceeds to S706.

[0071] Next, in S706, the calculation unit 160 extracts the area of ​​each region using the corrected image generated in S705. Furthermore, the height information of each region is read from the height information of the product acquired in S702, and is multiplied by the extracted area to calculate the product occupancy rate by volume. That is, the calculation unit 160 calculates the volume of the monitoring area 631 based on the extracted area of ​​the monitoring area 631 and the height information of the product itself in the monitoring area 631. Furthermore, the calculation unit 160 calculates the volume of the product area 632 and the product area 633 based on the areas of the product area 632 and the product area 633 and the height information of the product itself in the product area 632 and the product area 633. Then, the calculation unit 160 calculates the product occupancy rate based on the calculated volume of the monitoring area 631 and the volume of the product area 632 and the product area 633. Thereafter, this processing flow ends.

[0072] The height information of the product itself in the monitoring area 631 is acquired at a timing prior to the acquisition of the height information of the product in the product area. The timing of acquisition may be any timing. Furthermore, the height information of the product itself as a reference may be stored in advance in the memory unit 140 before the processing of this flow starts. The height information of the product itself in the monitoring area 631 (first height information) is acquired in the same manner as the height information of the product itself in the product area. That is, the height information of the product itself in the monitoring area 631 is acquired by subtracting the distance information to the product placed on the display shelf 200 and the distance information from the imaging unit 110 to the display shelf that was acquired in advance from the distance information to the product.

[0073] The image correction process in S705 is not limited to the above-mentioned method. For example, the distance to the display shelf in the store is acquired in advance by the distance acquisition unit 130, and size information of the products displayed in the monitoring area 221 is stored in the storage unit 140. Using the distance to the display shelf and the product height information, surface information when the products are stacked in one layer, two layers, three layers, etc. is acquired, and a transformation matrix for each surface is calculated in advance. The calculation unit 160 selects a transformation matrix corresponding to the product height acquired by the distance acquisition unit 130 and corrects the image. In this way, it is not necessary to calculate the transformation matrix of the product area each time in S704, and the calculation can be simplified, thereby reducing the processing load.

[0074] As described above, according to the imaging device 100 of the second embodiment, even when the product occupancy rate is calculated by volume, the occupancy rate of the subject in the imaging area can be appropriately acquired, so that it is possible to reduce false detections.

[0075] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims.

[0076] Furthermore, the functions of the above-described embodiments are not only realized by the computer reading and executing the program, but also include an operating system (OS) running on the computer carrying out all or part of the actual processing based on the instructions of the program.

[0077] Furthermore, a program that realizes all or part of the control in the above-described embodiment may be supplied to the imaging device 100, the display device 190, etc. via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging device 100, the display device 190, etc. may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0078] Furthermore, the functions of the above-mentioned embodiments may be realized by the following method. Program code read from a recording medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of the program code, a CPU provided in the function expansion card or function expansion unit performs some or all of the actual processing. When the present invention is applied to the above-mentioned recording medium, the recording medium stores program codes corresponding to the respective flowcharts described above.

[0079] The disclosure of this embodiment includes the following configuration, method, system, and program.

[0080] (Configuration 1) 1. An imaging device, comprising: An imaging unit that acquires an image; an image recognition unit that extracts, from a first region, which is a predetermined region in the image, a region in which one or more specific objects are located, as a second region; a distance acquisition unit that acquires distance information between the imaging device and the first area and distance information between the imaging device and the second area; a calculation unit that calculates an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired by the distance acquisition unit; and An imaging device comprising:

[0081] (Configuration 2) the distance acquisition unit acquires the distance information from a distance sensor capable of acquiring the distance information; 2. The imaging device according to configuration 1, wherein the distance sensor acquires distance information between the first area and the second area based on information of a plurality of points.

[0082] (Configuration 3) 3. The imaging device according to claim 2, wherein the distance sensor acquires the distance information by scanning within an imaging range of the imaging section.

[0083] (Configuration 4) The imaging device described in any one of configurations 1 to 3, characterized in that the distance acquisition unit detects the surface of the first area based on distance information of multiple points in the first area, and detects the surface of the second area based on distance information of multiple points in the second area.

[0084] (Configuration 5) the distance acquisition unit acquires first conversion information for performing a projection conversion onto a plane perpendicular to an optical axis of the imaging unit, based on a surface of the extracted first region; The imaging device according to configuration 4, further comprising: acquiring second conversion information for performing projection conversion onto a plane perpendicular to the optical axis of the imaging unit, based on a surface of the second region.

[0085] (Configuration 6) The imaging device described in configuration 5, characterized in that the calculation unit generates the corrected image by correcting the size difference of the first region using the first conversion information and correcting the size difference of the second region using the second conversion information.

[0086] (Configuration 7) The imaging device according to any one of configurations 1 to 6, wherein the calculation unit extracts the area of ​​each of the first and second regions by counting the number of pixels in the first and second regions in the corrected image.

[0087] (Configuration 8) the distance acquisition unit acquires object distance information which is information about a distance from the imaging device to the specific object; a determination unit that compares the object distance information most recently acquired by the distance acquisition unit with distance information immediately before the most recently acquired object distance information and determines whether or not the distance from the imaging device to the specific object has changed; 8. The imaging device according to any one of configurations 1 to 7, wherein the determination unit updates the object distance information when it determines that the object distance information has changed.

[0088] (Configuration 9) The determination unit determines whether the first region has depth, 9. The imaging device according to configuration 8, wherein the calculation unit generates the corrected image when the determination unit determines that there is depth.

[0089] (Configuration 10) The calculation unit determines whether an occupancy rate of the second region with respect to the first region is less than a predetermined threshold value; An imaging device described in any one of configurations 1 to 9, characterized in that it has a display control unit that displays a predetermined message on a screen of a display device when it is determined that the occupancy ratio of the second area to the first area is less than a predetermined threshold.

[0090] (Configuration 11) The imaging device described in any one of configurations 1 to 10, characterized in that the calculation unit calculates the volume of the first region based on first height information, which is information on the area of ​​the first region and the height of a specific object in the first region, and calculates the volume of the second region based on the area of ​​the second region and second height information, which is information on the height of a specific object in the second region.

[0091] (Configuration 12) The imaging device according to configuration 11, characterized in that the calculation unit calculates the occupancy rate of the second region relative to the first region based on the volume of the first region and the volume of the second region.

[0092] (Configuration 13) The imaging device described in configuration 11 or 12, characterized in that the distance acquisition unit acquires the first height information and the second height information based on object distance information, which is information on the distance from the imaging device to the specific object, and distance information from the imaging device to a position where the specific object is located, which has been acquired in advance.

[0093] (Configuration 14) 14. The imaging device according to any one of configurations 11 to 13, wherein the first height information is acquired at a timing earlier than the second height information.

[0094] (Configuration 15) 15. The imaging device according to any one of configurations 1 to 14, wherein at least the first height information is stored in a storage unit in advance.

[0095] (Configuration 16) A control method for an imaging device, comprising: an image acquisition step of acquiring an image; An extraction step of extracting, as a second region, a region in which one or more specific objects are located in a first region, which is a predetermined region in the image; a distance acquisition step of acquiring distance information between the imaging device and the first area and distance information between the imaging device and the second area; a calculation step of calculating an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired in the distance acquisition step; 13. A method for controlling an imaging apparatus comprising:

[0096] (Configuration 17) An imaging unit that acquires an image; an image recognition unit that extracts, from a first region, which is a predetermined region in the image, a region in which one or more specific objects are located, as a second region; a distance acquisition unit that acquires distance information between an imaging device and the first area and distance information between the imaging device and the second area; a calculation unit that calculates an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired by the distance acquisition unit; and An operation unit for setting the first area; a display device that displays a setting screen for setting the first region; An imaging system comprising:

[0097] (Configuration 18) A program for causing a computer to function as the imaging device according to any one of configurations 1 to 15. [Explanation of symbols]

[0098] 100 Imaging device 110 Imaging unit 130 Distance acquisition part 140 Storage section 150 Image Recognition Unit 160 Arithmetic section 170 Control section

Claims

1. 1. An imaging device, comprising: An imaging unit that acquires an image; an image recognition unit that extracts, from a first region, which is a predetermined region in the image, a region in which one or more specific objects are located, as a second region; a distance acquisition unit that acquires distance information between the imaging device and the first area and distance information between the imaging device and the second area; a calculation unit that calculates an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired by the distance acquisition unit; An imaging device comprising:

2. the distance acquisition unit acquires the distance information from a distance sensor capable of acquiring the distance information; The imaging device according to claim 1 , wherein the distance sensor acquires distance information between the first area and the second area based on information of a plurality of points.

3. The imaging device according to claim 2 , wherein the distance sensor acquires the distance information by scanning within an imaging range of the imaging section.

4. The imaging device according to claim 1, characterized in that the distance acquisition unit detects the surface of the first area based on distance information of a plurality of points in the first area, and detects the surface of the second area based on distance information of a plurality of points in the second area.

5. the distance acquisition unit acquires first conversion information for performing a projection conversion onto a plane perpendicular to an optical axis of the imaging unit, based on a surface of the extracted first region; 5. The imaging device according to claim 4, further comprising: second conversion information for performing a projection conversion onto a plane perpendicular to an optical axis of the imaging section, based on a surface of the second region.

6. The imaging device according to claim 5, characterized in that the calculation unit generates the corrected image by correcting the size difference of the first region using the first conversion information and correcting the size difference of the second region using the second conversion information.

7. 2 . The imaging device according to claim 1 , wherein the calculation unit extracts the area of ​​each of the first and second regions by counting the number of pixels in the first and second regions in the corrected image.

8. the distance acquisition unit acquires object distance information which is information about a distance from the imaging device to the specific object; a determination unit that compares the object distance information most recently acquired by the distance acquisition unit with distance information immediately before the most recently acquired object distance information and determines whether or not a distance from the imaging device to the specific object has changed; The imaging apparatus according to claim 1 , wherein the determination unit updates the object distance information when it determines that the object distance information has changed.

9. The determination unit determines whether the first region has depth, The imaging device according to claim 8 , wherein the calculation section generates the corrected image when the determination section determines that there is depth.

10. The calculation unit determines whether an occupancy rate of the second region with respect to the first region is less than a predetermined threshold value; The imaging device according to claim 1, further comprising a display control unit that displays a predetermined message on a screen of a display device when it is determined that the occupancy ratio of the second area to the first area is less than a predetermined threshold value.

11. The imaging device according to claim 1, characterized in that the calculation unit calculates the volume of the first region based on first height information, which is information on the area of ​​the first region and the height of a specific object in the first region, and calculates the volume of the second region based on the area of ​​the second region and second height information, which is information on the height of a specific object in the second region.

12. The imaging device according to claim 11 , wherein the calculation unit calculates an occupancy rate of the second region relative to the first region based on a volume of the first region and a volume of the second region.

13. The imaging device according to claim 11, characterized in that the distance acquisition unit acquires the first height information and the second height information based on object distance information, which is information about the distance from the imaging device to the specific object, and distance information from the imaging device to a position where the specific object is located, which has been acquired in advance.

14. The imaging device according to claim 11 , wherein the first height information is acquired at a timing earlier than the second height information.

15. 2. The imaging device according to claim 1, wherein at least the first height information is stored in a storage unit in advance.

16. A control method for an imaging device, comprising: an image acquisition step of acquiring an image; An extraction step of extracting, as a second region, a region in which one or more specific objects are located in a first region, which is a predetermined region in the image; a distance acquisition step of acquiring distance information between the imaging device and the first area and distance information between the imaging device and the second area; a calculation step of calculating an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired in the distance acquisition step; 13. A method for controlling an imaging apparatus comprising:

17. An imaging unit that acquires an image; an image recognition unit that extracts, from a first region, which is a predetermined region in the image, a region in which one or more specific objects are located, as a second region; a distance acquisition unit that acquires distance information between an imaging device and the first area and distance information between the imaging device and the second area; a calculation unit that calculates an occupancy rate of the second region relative to the first region based on an area of ​​a region corresponding to the first region and an area of ​​a region corresponding to the second region in a corrected image generated based on the distance information acquired by the distance acquisition unit; An operation unit for setting the first region; a display device that displays a setting screen for setting the first region; An imaging system comprising:

18. A program for causing a computer to function as the imaging device according to any one of claims 1 to 15.

Images