Image processing device and program

Abstract

To create distance image data useful for specifying human behavior using a plurality of pieces of captured image data generated by a plurality of imaging devices.SOLUTION: An image processing device 1 includes a data acquisition unit 141 that acquires a plurality of pieces of captured image data generated by a plurality of cameras C installed in a predetermined space, a detection unit 142 that detects the position of a person in the predetermined space, a selection unit 143 that selects two pieces of captured image data from the plurality of pieces of captured image data on the basis of the position detected by the detection unit 142, and a distance image creation unit 144 that creates distance image data including distance information on the basis of the two pieces of captured image data selected by the selection unit 143.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a program. [Background technology]

[0002] Conventionally, there is known a technology for monitoring human behavior using a camera. Patent Document 1 discloses a system for analyzing video data captured by a camera in order to prevent shoplifting in a store. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-76171 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional systems analyzed two-dimensional video data captured by each surveillance camera, but because two-dimensional video data does not allow for the determination of depth, there was a problem in that the accuracy of identifying human behavior was low.

[0005] Therefore, the present invention has been made in consideration of these points, and aims to make it possible to create distance image data useful for identifying human behavior using multiple captured image data generated by multiple imaging devices. [Means for solving the problem]

[0006] An image processing device according to a first aspect of the present invention includes a data acquisition unit that acquires multiple captured image data generated by multiple imaging devices installed in a specified space, a detection unit that detects the position of a person within the specified space, a selection unit that selects two captured image data from the multiple captured image data based on the position detected by the detection unit, and a distance image creation unit that creates distance image data including distance information based on the two captured image data selected by the selection unit.

[0007] The selection unit may select the two captured image data generated by two of the imaging devices, the distance between the position of the person detected by the detection unit and the position of the imaging device being within a predetermined range.

[0008] The selection unit may select the two captured image data generated by two of the imaging devices that are closest in distance between the position of the person detected by the detection unit and the positions of the plurality of imaging devices.

[0009] The specified space may have a plurality of areas, and the image processing device may further have an area data storage unit that stores each of the plurality of areas in association with two or more imaging devices, the detection unit may detect an area where a person is present, and the selection unit may select the two captured image data generated by the two imaging devices stored in the area data storage unit in association with the area detected by the detection unit.

[0010] The image processing device may further include a recording processing unit that stores a plurality of pieces of range image data in which the same person is commonly present in the detection data storage unit in association with the person.

[0011] The recording processing unit may store the plurality of pieces of range image data in the detection data storage unit in association with the times at which the captured image data corresponding to the plurality of pieces of range image data were generated.

[0012] The selection unit may select the two or more captured image data based on the result of identifying the direction of travel of the person detected by the detection unit, based on the relationship between multiple positions corresponding to different times identified by the detection unit.

[0013] An image processing device according to a second aspect of the present invention includes a data acquisition unit that acquires multiple captured image data generated by multiple imaging devices installed in a specified space, a detection unit that detects the position of a person within the specified space, a selection unit that selects two imaging devices from the multiple imaging devices based on the position detected by the detection unit, and a distance image creation unit that creates distance image data including distance information based on the two captured image data generated by the two imaging devices selected by the selection unit.

[0014] A third aspect of the program of the present invention causes a computer to function as a data acquisition unit that acquires multiple captured image data generated by multiple imaging devices installed in a specified space, a detection unit that detects the position of a person within the specified space, a selection unit that selects two captured image data sets from the multiple captured image data sets based on the position detected by the detection unit, and a distance image creation unit that creates distance image data including distance information based on the two captured image data sets selected by the selection unit.

[0015] A fourth aspect of the program of the present invention causes a computer to function as a data acquisition unit that acquires multiple captured image data generated by multiple imaging devices installed in a specified space, a detection unit that detects the position of a person within the specified space, a selection unit that selects two imaging devices from the multiple imaging devices based on the position detected by the detection unit, and a distance image creation unit that creates distance image data including distance information based on the two captured image data generated by the two imaging devices selected by the selection unit. [Effects of the Invention]

[0016] According to the present invention, it is possible to create distance image data useful for identifying human behavior using a plurality of captured image data generated by a plurality of imaging devices. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an overview of an image processing system 100. [Figure 2] FIG. 2 is a schematic diagram showing the layout of store S. [Figure 3] FIG. 10 is a diagram showing addresses, which are area IDs, of a plurality of areas in a store S. [Figure 4] 1 is a diagram illustrating a configuration of an image processing device 1. FIG. [Figure 5] FIG. 4 is a diagram illustrating an example of area data. [Figure 6] FIG. 10 is a diagram showing an example of first detection data. [Figure 7] FIG. 10 is a diagram illustrating an example of second detection data. [Figure 8] 10 is a diagram for explaining a method in which distance image creation section 144 creates distance image data. FIG. [Figure 9] This is an example of data associating a customer M who stays at the same location for a long period of time with the location where the customer M stays. [Figure 10] FIG. 10 is a diagram illustrating an example of data indicating suspicious behavior. [Figure 11] FIG. 10 is a diagram showing an example of data associating shelf IDs with customer M's stay times. [Figure 12] 3 is a flowchart showing the flow of processing in the image processing device 1. [Figure 13] 10 is a flowchart showing the flow of processing by a recording processing unit 147 to store distance image data in a detection data storage unit 134. DETAILED DESCRIPTION OF THE INVENTION

[0018] [Overview of image processing system 100] 1 is a diagram illustrating an overview of an image processing system 100 according to this embodiment. The image processing system 100 is a system for monitoring people's behavior, and is used, for example, to prevent shoplifting in a store S or to understand which products people in the store S (hereinafter, sometimes referred to as "customer M") are interested in. In this embodiment, an example is shown in which the image processing system 100 monitors the behavior of customers in the store S, but the image processing system 100 may also be a system for processing images taken of people in other locations.

[0019] 1, a store S is equipped with a plurality of cameras C, which are imaging devices, and the cameras C capture images of customers M inside the store to generate captured image data. The viewing angles of the cameras C are arbitrary, but the camera C in this embodiment is assumed to be a camera capable of capturing images in a 360-degree direction.

[0020] The multiple cameras C transmit captured image data to the image processing device 1 via communication lines. The communication lines may be wired or wireless. In this embodiment, the captured image data is moving image data, but the captured image data may also be still image data.

[0021] The image processing device 1 is a device for performing image processing on captured image data, and is, for example, a computer. The image processing device 1 is, for example, a personal computer installed in an office of the store S, but may also be a computer or a server installed in a location different from the store S.

[0022] FIG. 2 is a schematic diagram showing the layout of store S. FIG. 3 is a diagram showing addresses, which are area IDs, for each of multiple areas in store S. The addresses of multiple areas are area identification information expressed as a combination of horizontal addresses x1 to x8 and vertical addresses y1 to y6. For example, the address of the area in the upper left corner is expressed as (x1, y1). The dashed dotted arrows in FIG. 3 indicate the route taken by customer M1.

[0023] A store S is equipped with product shelves T1 and T2. As shown in Fig. 3, the product shelf T1 is made up of product shelves T11 to T22, and the product shelf T2 is made up of product shelves T23 to T34. Cameras C11 to C15 are installed above the product shelf T1, and cameras C21 to C25 are installed above the product shelf T2. Cameras C11 to C15 and C21 to C25 are used as camera identification information (hereinafter referred to as "camera ID") for identifying the camera C.

[0024] Cameras may be installed above the aisles around the product shelves T1 and T2. Customers (M1 to M5 in FIG. 2) who visit the store can move freely along the aisles. The multiple cameras C capture images within their respective fields of view, generating different captured image data.

[0025] The image processing device 1 creates distance image data by using two pieces of captured image data selected from the captured image data generated by the cameras C and using the distance between the cameras C as parallax. Since the distance image data contains depth information (i.e., information indicating the distance in the depth direction), the image processing device 1 can analyze the behavior of the subject, customer M, with higher accuracy by analyzing the distance image data compared to when only two-dimensional captured image data is used. The configuration and operation of the image processing device 1 will be described in detail below.

[0026] [Configuration of image processing device 1] 4 is a diagram showing the configuration of the image processing device 1. The image processing device 1 has a communication unit 11, a setting reception unit 12, a storage unit 13, and a control unit .

[0027] The communication unit 11 is an interface for transmitting and receiving data to and from an external device, and includes, for example, a LAN (Local Area Network) controller or a wireless communication controller. The communication unit 11 receives captured image data from a plurality of cameras C.

[0028] The setting reception unit 12 is an interface for receiving various settings from a user of the image processing device 1 (for example, a manager of the store S). The setting reception unit 12 includes, for example, an operation device such as a keyboard or a mouse. The setting reception unit 12 stores data indicating the input setting contents in the storage unit 13 and notifies the control unit 14.

[0029] The storage unit 13 has a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), or an SSD (Solid State Drive). The storage unit 13 stores programs executed by the control unit 14. The storage unit 13 also stores data used to execute various processes based on captured image data.

[0030] The storage unit 13 includes a program storage unit 131, a setting data storage unit 132, an area data storage unit 133, and a detection data storage unit 134. The program storage unit 131 stores a program to be executed by the control unit 14.

[0031] The setting data storage unit 132 stores various data set by the user. The setting data storage unit 132 stores data indicating the setting contents accepted by the setting acceptance unit 12. The setting data storage unit 132 stores, for example, setting data accepted by the setting acceptance unit 12 indicating a detection target part, which is a body part of the customer M to be detected from the captured image data. The detection target part is, for example, the face, left hand, right hand, whole body, or dominant hand, and corresponds to a target part that the user wants to monitor.

[0032] The setting data storage unit 132 may store each of the body parts of a plurality of customers M in association with one or more cameras C. For example, the setting data storage unit 132 stores the fact that the head can be photographed in association with the camera ID of a camera C that can photograph a relatively high position, and stores the fact that the hands can be photographed in association with the camera ID of a camera C that can photograph a relatively low position.

[0033] The parts of a person that can be photographed by one camera C may differ depending on the location of customer M. For example, when customer M is at a position (x2, y3) near camera C11, camera C11 may be able to photograph only the head of customer M, but when customer M is at a position (x3, y3) a little distance from camera C11, camera C11 may be able to photograph the entire body of customer M. Therefore, the setting data storage unit 132 may store detectable human parts in association with the area ID and the camera ID.

[0034] The area data storage unit 133 stores area data in which each of a plurality of areas is associated with two or more cameras C. The area data storage unit 133 may store area IDs in association with shelf IDs, which are shelf identification information for identifying product shelves installed in the areas corresponding to the area IDs. The shelf IDs are, for example, character strings such as T11 and T12 in FIG. 3.

[0035] Fig. 5 is a diagram showing an example of area data. In the area data shown in Fig. 5, area addresses and camera IDs are associated with each other. The address of an area that cannot be photographed by multiple cameras C is associated with the camera ID of only one camera C. The address of an area that can be photographed by two or more cameras C is associated with the camera IDs of two or more cameras C.

[0036] The detection data storage unit 134 stores first detection data in which a plurality of captured image data in which customer M detected in the captured image data exists is associated with the detected customer M. For example, for each of the same customers identified by the person identification unit 146 based on features identified by the feature identification unit 145 described below, the detection data storage unit 134 stores the captured image data in which the customer appears in chronological order in the order of the date and time when the captured image data was generated.

[0037] FIG. 6 is a diagram showing an example of first detection data. While FIG. 6 shows first detection data D01, D02, and D03 corresponding to three customers (e.g., customers M1, M2, and M3) detected in the captured image data, the detection data storage unit 134 may store multiple pieces of first detection data corresponding to more customers. In the following description, information such as D01, D02, and D03 for identifying the customer corresponding to the first detection data may be referred to as a feature ID. The detection data storage unit 134 may store the content of the feature in association with the feature ID.

[0038] 6, the first detection data is associated with the start date and time and end date and time of the time when customer M appears in each piece of captured image data, the file name of the captured image data, and the camera ID of the camera C that generated the captured image data, in the order in which the captured image data containing the image of customer M corresponding to each piece of first detection data was generated. The image data name is the file name of the captured image data associated with an address in the detection data storage unit 134, and the control unit 14 can access the captured image data by specifying the image data name.

[0039] The first detection data shown in Fig. 6 corresponds to a plurality of captured image data taken while customer M1 in store S shown in Fig. 3 moves from entering store S to the address (x4, y3) along the route shown by the dashed dotted line. Here, it is assumed that customer M1 moves from one address to another every second.

[0040] As shown in the area data in Figure 5, when customer M1 enters store S, only camera C11 can photograph customer M1 while he is at addresses (x1, y1) and (x1, y2), so for the first two seconds (13:45:00 to 13:45:02), only camera C11 photographs customer M1. After that, when customer M1 moves to address (x1, y3), camera C21 is also able to photograph customer M1, and C21 also photographs customer M1 between 13:45:02 and 13:45:05.

[0041] After that, cameras C12 and C22 photograph customer M1 while he is at address (x2, y3) to (x4, y3) between 13:45:03 and 13:45:06. When customer M1 arrives at address (x3, y3) at 13:45:04, cameras C13 and C23 also photograph customer M1. In this way, the first detection data records captured image data of customer M1 in chronological order, allowing the image processing device 1 to efficiently analyze the behavior of customer M1. Furthermore, since the user can view images of customer M in chronological order, this is suitable for cases where the user wants to check the behavior of each customer M.

[0042] The detection data storage unit 134 also stores second detection data in which a person detected in the distance image data is associated with multiple distance image data pieces in which the person is present. For example, for each customer M, the detection data storage unit 134 stores multiple distance image data pieces in which the customer M is photographed in chronological order in the order of the dates and times when the two captured image data pieces used to create the distance image data were generated.

[0043] Fig. 7 is a diagram showing an example of second detection data. Fig. 7 shows second detection data D11, D12, and D13 corresponding to three customers (e.g., customers M1, M2, and M3) detected in the captured image data, respectively, but the detection data storage unit 134 may store multiple pieces of second detection data corresponding to more customers.

[0044] 7, the second detection data are associated with the start and end dates and times of the time when customer M was included in the distance image data generated based on the captured image data, the file names of the distance image data, and the camera IDs of the cameras that generated the captured image data used to create the distance image data, in the order in which the captured image data containing the image of customer M corresponding to each second detection data was generated. The file names of the distance image data include the camera IDs of the two cameras C that generated the two captured image data used to create the distance image data.

[0045] Since two captured image data are used to create the range image data, two camera IDs are associated with the range image data in the second detection data. However, since customer M1 appears only in the captured image data generated by camera C11 between 13:45:00 and 13:45:02, only the captured image data generated by camera C11, not the range image data, is recorded during this period.

[0046] Customer M1 is photographed by multiple cameras C while he moves from address (x1, y3) to address (x4, y3), and so distance image data created based on two sets of captured image data are recorded. During this time, if customer M1 is photographed by three or more cameras C, two or more sets of distance image data can be created. Therefore, for example, between 13:45:03 and 13:45:05, four sets of distance image data are created based on four sets of two sets of captured image data corresponding to the combination of cameras C11 and C21, the combination of cameras C11 and C12, the combination of cameras C21 and C22, and the combination of cameras C12 and C22.

[0047] By executing a program stored in program storage unit 131, control unit 14 functions as at least one of data acquisition unit 141, detection unit 142, selection unit 143, distance image creation unit 144, feature identification unit 145, person identification unit 146, and record processing unit 147. Control unit 14 may execute the functions of all of these components, or may execute the functions of some of these components.

[0048] The data acquisition unit 141 acquires a plurality of captured image data generated by a plurality of cameras C installed in a predetermined space. As described above, the predetermined space in this embodiment is a space within a store S, for example.

[0049] The captured image data acquired by the data acquisition unit 141 may be moving image data or still image data. The data acquisition unit 141 inputs the acquired multiple captured image data to the detection unit 142. The data acquisition unit 141 may store the multiple captured image data in the detection data storage unit 134 in association with the camera ID of the camera C that transmitted the captured image data.

[0050] The detection unit 142 detects the position of the customer M within the store S. The detection unit 142 detects in which of multiple areas within the store S the person is staying, based on multiple captured image data. The detection unit 142 may detect in front of which product shelf the person is staying in the area where the customer M is staying, based on the multiple captured image data.

[0051] The detection unit 142 may detect the location of the customer M using any method, but the detection unit 142 may identify the location of the customer M based on, for example, captured image data. Specifically, the detection unit 142 identifies areas in the area data storage unit 133 associated with the multiple cameras C that generated the multiple captured image data, thereby detecting which of the multiple areas in the store M the person is staying in.

[0052] In this case, the detection unit 142 first identifies a contour line in which the amount of change in brightness value is equal to or greater than a threshold value in the multiple pieces of captured image data input from the data acquisition unit 141, and identifies the captured image data that includes an image of customer M because the shape of the contour line is similar to a human contour line pre-stored in the storage unit 13. The detection unit 142 references the area data stored in the area data storage unit 133 and identifies the location of the camera C that transmitted the captured image data that has been identified as including an image of customer M, thereby detecting the location of customer M.

[0053] The detection unit 142 may detect the position of the customer M based on signals from motion sensors (not shown) installed at multiple positions in the store S. In this case, the detection unit 142 detects the position of the customer M based on the range in which the motion sensor that transmitted the signal indicating that a person has been detected can detect a person. The detection unit 142 notifies the selection unit 143 of the detected position of the customer M.

[0054] The detection unit 142 may detect the time when the person detected by the detection unit 142 faces a predetermined position in the area where customer M is staying. The detection unit 142 detects, for example, the time when the person faces a product shelf in the area where the person is staying. The detection unit 142 may detect the time when customer M is staying in the same position within a predetermined distance from the product shelf (for example, within reach of the product shelf). The detection unit 142 notifies the person identification unit 146 of the detected time. The detection unit 142 may notify the person identification unit 146 of the shelf ID of the product shelf that customer M is facing or the product shelf closest to customer M, along with the detection time.

[0055] The selection unit 143 selects two pieces of captured image data from the plurality of captured image data based on the position of the customer M detected by the detection unit 142. The selection unit 143 selects two pieces of captured image data generated by the two cameras C stored in the area data storage unit 133 in association with the area that includes the position detected by the detection unit 142. For example, the selection unit 143 selects two cameras C from the plurality of cameras C installed in the store S based on the position of the customer M detected by the detection unit 142, and selects the two pieces of captured image data generated by the two selected cameras C.

[0056] Specifically, the selection unit 143 refers to the area data shown in FIG. 5 and selects two cameras C corresponding to the address of the location where the detection unit 142 detected the customer M. The selection unit 143 selects two captured image data generated by the two selected cameras C. The selection unit 143 inputs the selected two captured image data to the distance image creation unit 144 so that the distance image creation unit 144 can create distance image data. The selection unit 143 may notify the distance image creation unit 144 of information for identifying the two captured image data (e.g., image file names).

[0057] As shown in the area data in FIG. 5, depending on the location, three or more cameras C may be able to capture images of customer M. In such a case, the selection unit 143 selects two pieces of captured image data generated by two imaging devices whose distance between the position of customer M detected by the detection unit 142 and the position of camera C is within a predetermined range. The predetermined range is, for example, a range set as a distance suitable for creating distance image data. The predetermined range may also be a distance range suitable for capturing an image of the detection target area.

[0058] As an example, the selection unit 143 selects two captured image data generated by two cameras C that are closest in distance between the position of customer M detected by the detection unit 142 and each of the multiple cameras C. The selection unit 143 may select captured image data generated by two cameras C that have the smallest average or median value of the distances from the two cameras C to the position of customer M. By the selection unit 143 selecting such two captured image data, distance image data in which customer M is captured prominently is created, thereby increasing the accuracy of identifying customer M's behavior.

[0059] The selection unit 143 may also select two cameras C such that the variation in the distance between the position of customer M and the two cameras C is within a range set as a distance suitable for creating distance image data. If the variation in the distance between the position of customer M and the two cameras C is small, the size of customer M captured by the two cameras C will be equivalent, thereby improving the quality of the created distance image data.

[0060] The selection unit 143 may select two cameras C that can capture images of customer M, the distance between which is within a distance range suitable for creating distance image data that is stored in advance in the storage unit 13. As an example, if customer M is located at address (x3, y3) shown in FIG. 3, the selection unit 143 selects, from among the six cameras C that can capture images of this location, cameras C11 and C12, or cameras C12 and C13, etc., that are closest to each other.

[0061] The selection unit 143 may select two or more captured image data based on the result of identifying the traveling direction of customer M detected by the detection unit 142 based on the relationship between multiple positions corresponding to different times identified by the detection unit 142. The selection unit 143 identifies the traveling direction of customer M1 by identifying the addresses of positions where customer M1 appears in the captured image data in chronological order based on the first detection data shown in Fig. 6, for example, and identifying the direction in which the position changes over time.

[0062] For example, when the selection unit 143 determines that customer M1 is moving in the direction from (x3, y3) to (x4, y3) so that the face of customer M can be recorded, the selection unit 143 selects, from the plurality of captured image data included in the first detection data, two captured image data sets generated by two cameras C, camera C13 and camera C23, aligned in a direction perpendicular to the direction of travel. If there are multiple combinations of two cameras C aligned in a direction perpendicular to the direction of travel, the selection unit 143 selects the two captured image data sets generated by the two cameras C on the side where customer M1 is moving. By the selection unit 143 selecting the two captured image data sets generated by such two cameras C, it becomes easier to create distance image data that captures the face of customer M1.

[0063] The selection unit 143 may select, from the plurality of captured image data, one or more captured image data generated by a camera C capable of capturing an image of the detection target part stored in the setting data storage unit 132. In this case, the selection unit 143 selects, from the plurality of human parts, one or more captured image data generated by one or more cameras C stored in the setting data storage unit 132 in association with the detection target part.

[0064] As an example, if at a location with address (x3, y3), camera C12 closest to the location can only capture the head of customer M, and cameras C11 and C13 located away from the location can capture the entire body of customer M, the selection unit 143 selects one of cameras C11, C13, C21, C22, or C23.

[0065] The selection unit 143 may select multiple cameras C or multiple captured image data based on the direction of travel of customer M and the body part to be detected. When the body part to be detected is a hand, the selection unit 143 selects, from the multiple captured image data, captured image data generated by camera C that can capture the hand of customer M1. When the selection unit 143 determines that the body part to be detected is the right hand and that customer M1 is moving from (x3, y3) to (x4, y3), the selection unit 143 selects, from the multiple captured image data included in the first detection data, two captured image data generated by camera C22 and camera C23, which are two cameras C lined up on the right side of the direction of travel. When the body part to be detected is a face, as described above, the selection unit 143 selects camera C13 and camera C23, which are installed in the direction of travel of customer M and can capture the face of customer M.

[0066] By having the selection unit 143 select the captured image data generated by the camera C that can capture the detection target part in this manner, it is possible to prevent the distance image creation unit 144 from creating distance image data that does not include the detection target part, and to prevent the feature identification unit 145 from analyzing captured image data that does not include the detection target part.

[0067] When selecting captured image data corresponding to a detection target part, the selection unit 143 may select the number of captured image data corresponding to the detection target part. As an example, the storage unit 13 stores information that one piece of captured image data is required when the detection target part is the head, and two pieces of captured image data are required to create distance image data when the detection target part is the hand. In this case, in response to the hand being set as the detection target part, the selection unit 143 selects two or more pieces of captured image data that include the hand. By operating in this manner, the selection unit 143 can select the number of captured image data appropriate for analyzing the detection target part.

[0068] The selection unit 143 selects one or more pieces of captured image data for each time from the plurality of captured image data generated at multiple times. The selection unit 143 may select one or more pieces of captured image data for each unit time (e.g., one second). As an example, the selection unit 143 selects captured image data from cameras C11 and C21 when customer M is at (x1, y3), and selects captured image data from cameras C11, C12, C21, and C22 when customer M is at (x2, y3). The selection unit 143 may select two pieces of captured image data for each time by any of the methods described above.

[0069] The selection unit 143 may select two pieces of captured image data captured in the same direction. By the selection unit 143 selecting such two pieces of captured image data, the accuracy of the distance information included in the distance image data created by the distance image creation unit 144 is improved. When the two cameras C are capable of capturing images within a 360-degree range, the selection unit 143 selects, from the captured image data created by the two cameras C, image data of an area that corresponds to the same direction and that includes the same customer M as the two pieces of captured image data.

[0070] Distance image creation unit 144 creates distance image data including distance information based on the two captured image data selected by selection unit 143. Distance information is information indicating the depth or perspective to the subject. In this embodiment, distance image data is data including, for example, color information and distance information included in the captured image data.

[0071] The two cameras C that generated the two captured image data sets selected by selection unit 143 are considered to constitute a stereo camera, and distance image creation unit 144 creates distance image data using the same image processing method as when a stereo camera creates distance image data. Distance image creation unit 144 creates distance image data by calculating the distance from a line connecting the two cameras C to the subject, assuming that the distance between the two cameras C corresponding to the two captured image data sets selected by selection unit 143 is the distance between the left and right cameras in the stereo camera. The distance image data created by distance image creation unit 144 is stored in detection data storage unit 134 by recording processing unit 147.

[0072] Fig. 8 is a diagram illustrating a method for creating distance image data by distance image creation unit 144. In the example shown in Fig. 8, the focal length of cameras C11 and C12 is F, and the baseline length (distance between the cameras) is B. Distance image creation unit 144 calculates the parallax S, which is the difference between the position of customer M in the captured image data generated by camera C11 and the position of customer M in the captured image data generated by camera C12, and calculates the distance D to customer M using D = B × F / S.

[0073] The feature identification unit 145 identifies the features of customer M contained in the captured image data or distance image data. The feature identification unit 145 identifies features of customer M, such as the face, clothing, and manner of movement, contained in the captured image data or distance image data. The feature identification unit 145 identifies the features of customer M contained in the first captured image data or first moving image data generated by the first camera C and the features of customer M contained in the second captured image data or second moving image data generated by the second camera C.

[0074] The feature identification unit 145 assigns the same feature ID to a customer M whose identified feature matches multiple pieces of captured image data or multiple pieces of range image data. The feature ID is a character string such as D11, D12, or D13 shown in FIG. 7, and a different character string is assigned each time a customer M with a new feature is detected. The feature ID may be a character string corresponding to the date and time when the customer M was first detected in the store S.

[0075] The feature identification unit 145 may identify multiple types of features and assign the same feature ID to customers M with matching combinations of the multiple features. For example, even if customers M are wearing red clothes, if their walking style characteristics are different, the feature identification unit 145 assigns different feature IDs to each customer M.

[0076] The feature identification unit 145 associates the identified features or feature IDs with information for identifying the captured image data (e.g., camera ID and shooting time) and notifies the recording processing unit 147 of the identified features or feature IDs, thereby enabling the recording processing unit 147 to classify the multiple captured image data by customer M and create first detection data and second detection data.

[0077] When identifying the movement characteristics of customer M, the feature identification unit 145 uses multiple captured image data generated at different times or multiple distance image data (i.e., distance video image data) based on multiple captured image data generated at different times. In other words, the feature identification unit 145 identifies the change in position of customer M contained in the video image data as the movement characteristics.

[0078] As an example, the feature identifying unit 145 identifies the feature of the way customer M walks. In this case, the feature identifying unit 145 identifies the feature of the change in head height of customer M contained in a first video data generated at a first time, and identifies the feature of the change in head height of a person contained in a second video data generated at a second time different from the first time, thereby identifying the feature of the way customer M walks. For example, the feature identifying unit 145 analyzes the video data captured within a predetermined time (e.g., 5 seconds) required to identify the feature of the way customer M walks, and identifies the feature of the change in head height.

[0079] Customers M differ in height and the magnitude of vertical movement when walking. Therefore, the feature identifying unit 145 identifies, for example, the height of the head of customer M identified based on the first video data and the second video data and the range of vertical movement as features. The feature identifying unit 145 may also identify, as features, the height of the head of customer M identified based on the first video data and the second video data and the period of vertical movement.

[0080] Specifically, the feature identifying unit 145 identifies the height of the top position of customer M's head in multiple captured image data or multiple distance image data generated at multiple times. The feature identifying unit 145 identifies the period of a waveform obtained by plotting the identified height for each time as a feature indicating the period of up and down movement of customer M's head. The feature identifying unit 145 may use data indicating the waveform obtained by plotting the identified height for each time as data indicating customer M's feature.

[0081] The feature identifying unit 145 may identify the depth direction positions of the head, shoulders, or hands of customer M in the multiple distance image data generated at multiple times. The feature identifying unit 145 identifies the period of a waveform obtained by plotting the identified depth direction positions for each time as a feature indicating the period of left-right fluctuation of customer M. By the feature identifying unit 145 identifying the period of left-right fluctuation as a feature, it is possible to identify the feature of customer M who has a habit of swaying from side to side while walking.

[0082] The feature identification unit 145 may identify the moving speed of the customer M, which is identified based on the first moving image data and the second moving image data, as a feature. The feature identification unit 145 identifies the moving speed of the customer M based on the results of identifying the position of the customer M in the store S at each time, based on the position of the customer M in the captured image data corresponding to multiple times and the position of the camera C that generated the captured image data.

[0083] It is considered that customer M tends to walk fast near products that interest him / her and walk slowly near products that he / she is not interested in. Therefore, the feature identification unit 145 may identify the movement speed of customer M for each position in the store S as the feature of customer M. In this case, the feature identification unit 145 stores data indicating the feature in which, for example, a shelf ID and a movement speed are associated in the storage unit 13.

[0084] The person identification unit 146 identifies the details of the behavior of customer M based on the captured image data or distance image data. In order to identify the details of the behavior of customer M, if the features identified by the feature identification unit 145 based on the first moving image data and the features identified based on the second moving image data match, the person identification unit 146 identifies the person whose features are identified as the same customer M. The person identification unit 146 identifies the details of the behavior of customer M based on the captured image data or distance image data that includes the same customer M.

[0085] The person identification unit 146 identifies the content of the behavior of the customer M included in one or more captured image data corresponding to the multiple times selected by the selection unit 143. The person identification unit 146 may identify the content of the behavior of the customer M included in the distance image data. The person identification unit 146 identifies the content of the behavior for each customer M by, for example, referring to the first detection data shown in FIG. 6 or the second detection data shown in FIG. 7.

[0086] The person identification unit 146 identifies the content of the behavior corresponding to the movement characteristics of customer M in the captured image data or distance image data by referring to movement identification data that is stored in advance in the storage unit 13 and that associates movement characteristics with the content of the behavior. As an example, when the person identification unit 146 identifies the movement of customer M moving his / her hand toward a product shelf, it determines that customer M has taken the action of picking up a product. When customer M picks up a product and then finds that there is no product in the shopping cart that customer M is holding but the product is not in customer M's hand, the person identification unit 146 determines that customer M may have shoplifted.

[0087] The person identification unit 146 may identify the dominant hand of the customer M based on the hand movement of the customer M included in one or more captured image data. For example, when the person identification unit 146 identifies the behavior of the customer M picking up a product, it identifies the hand that picked up the product as the dominant hand. The person identification unit 146 stores data indicating the identified dominant hand in the detection data storage unit 134 via the recording processing unit 147, in association with the feature ID corresponding to the customer M.

[0088] The person identification unit 146 may identify the time that customer M stays in the same area. When the detection unit 142 determines that customer M has been staying in the same area continuously for a predetermined time or more, the person identification unit 146 may associate a feature ID for identifying customer M detected by the detection unit 142 with information for identifying the area (for example, an address or shelf ID) and store them in the detection data storage unit 134.

[0089] Fig. 9 is an example of data associating a customer M who stays at the same location for a long period of time with the location where the customer M stays. In Fig. 9, for each feature ID corresponding to customer M, the length of stay when the customer stays at the same location for 180 seconds or more is associated with a shelf ID indicating the location of the product shelf where the customer stayed. Creating such data by the person identification unit 146 makes it easier for the manager of store S to check the behavior history of customer M who stays at the same location for a long period of time.

[0090] When the person identification unit 146 determines that the dominant hand of customer M has been near a product for a predetermined period of time or longer, it may store the characteristic ID of customer M and information indicating the customer's location in the detection data storage unit 134. When customer M shoplifts, it is likely that the dominant hand will be near a product for a long period of time. Therefore, by having the person identification unit 146 operate in this manner, it becomes easier for the manager of store S to check captured image data that includes the appearance of shoplifting.

[0091] The person identification unit 146 may associate the feature ID of customer M, who the detection unit 142 has detected as staying in the same area continuously for a predetermined period of time or more, with the area ID and the time that the customer M is facing a predetermined position, and store these in the detection data storage unit 134. When customer M is facing the same product shelf position continuously for a predetermined period of time or more, the person identification unit 146 may associate the feature ID of customer M with data indicating that the customer is behaving suspiciously and store this data in the detection data storage unit 134.

[0092] Fig. 10 is a diagram showing an example of data indicating suspicious behavior. In Fig. 10, a feature ID, an area ID corresponding to a location where the stay time was identified as being equal to or longer than a predetermined time, the stay time, the orientation of customer M at that time, and a suspicious flag are associated with each other. The orientation of customer M is represented by setting the orientation of the product as 0 o'clock and the orientation of the side opposite the product as 6 o'clock.

[0093] The suspicious flag indicates whether a person is exhibiting suspicious behavior that may indicate an attempt to shoplift. The person identification unit 146 determines that a person is suspicious if they stay facing the 11:00 to 1:00 o'clock range, which corresponds to the orientation of the merchandise, for 180 seconds or more. In this way, the person identification unit 146 associates data including the suspicious flag with a feature ID corresponding to a customer M who is facing the merchandise for a long period of time and stores the data in the detection data storage unit 134. This allows the manager of the store S to search for captured image data that includes a suspicious customer M, and by checking the captured image data, the manager can identify the features of the customer M who may be likely to shoplift, and pay attention if a customer M with the same features visits the store.

[0094] The person identification unit 146 may further identify characteristics of the customer M detected by the detection unit 142 as staying in the same area continuously for more than a predetermined time, and may store the identified characteristics in the detection data storage unit 134 in association with a feature ID and an area ID. The data indicating the characteristics of the customer M may be image data indicating the type and color of the customer M's clothing, hair length, facial expression, facial shape, and skin color, or may be text data. By the person identification unit 146 storing data indicating the characteristics of the customer M who has stayed in the same area for more than a predetermined time in the detection data storage unit 134, the manager of the store S can grasp the characteristics of the customer M who may be committing shoplifting without checking the captured image data, and can pay attention if a customer M with the same characteristics visits the store.

[0095] The person identification unit 146 may create data associating the shelf ID with the stay time of customer M. Fig. 11 is a diagram showing an example of data associating the shelf ID with the stay time of customer M. In the data shown in Fig. 11, the shelf ID, the number of people staying, the stay time, and the characteristics of customer M who stayed are associated with each other.

[0096] The number of visitors is the number of people who stayed in front of a product shelf for a predetermined time (for example, 10 seconds) or more during a day. The duration of stay is the total duration (in minutes) of people who stayed in front of a product shelf for a predetermined time or more during a day. Customer characteristics include the gender ratio, age group, and proportion of people with children. The person identification unit 146 identifies the behavior of customer M and compiles this data, which is ideal for the manager of store S when considering which products to place on which product shelf. It is also ideal for the manager of store S to understand which product shelves are prone to shoplifting by understanding which product shelves customer M is likely to stay on for long periods of time.

[0097] The recording processing unit 147 stores the captured image data and the distance image data in the detection data storage unit 134. The recording processing unit 147 creates the first detection data shown in FIG. 6 by storing multiple captured image data sets that share the same customer M in the detection data storage unit 134 in association with the customer M. The recording processing unit 147 creates the first detection data shown in FIG. 6 by classifying the multiple captured image data sets for each customer M included in the captured image data. Specifically, the recording processing unit 147 stores the multiple captured image data sets in the detection data storage unit 134 in association with the feature ID assigned by the feature identification unit 145.

[0098] When the feature identification unit 145 detects customer M having the same features in multiple captured image data, the recording processing unit 147 creates the first detection data shown in Figure 5 by associating the multiple captured image data containing the detected customer M and storing them in the detection data storage unit 134.

[0099] 7 by storing multiple pieces of distance image data that share the same customer M in the detection data storage unit 134 in association with the customer M. The recording processing unit 147 creates the second detection data shown in FIG. 7 by classifying the multiple pieces of distance image data for each customer M included in the distance image data.

[0100] Specifically, the recording processing unit 147 arranges the distance image data created using two captured image data pieces including, for example, customer M1 in chronological order and stores them in the detection data storage unit 134. More specifically, the distance image creation unit 144 creates second detection data in which the start date and time and end date and time of the captured image data used to create the distance image data, the camera ID of the camera C that generated the captured image data, and the file name of the distance image data are associated, and stores the distance image data in the detection data storage unit 134 at an address corresponding to the image data name.

[0101] [Processing flow in image processing device 1] 12 is a flowchart showing the flow of processing in the image processing device 1. The data acquisition unit 141 acquires captured image data generated by multiple cameras C (S1). The detection unit 142 identifies an image of customer M in each of the multiple captured image data and detects the position of customer M (S2).

[0102] The detection unit 142 identifies the area where the detected customer M is located based on the position where the customer M is detected in the captured image data and the position of the camera C that generated the captured image data (S3). The detection unit 142 also identifies the direction of travel of the customer M based on changes in the position of the customer M detected in multiple captured image data generated at different times (S4).

[0103] Next, the selection unit 143 selects two cameras C to be used for the distance image data from the multiple cameras C installed in the store S (S5). The selection unit 143 selects, for example, two cameras C that generate two captured image data that can be used to create distance image data suitable for identifying the behavior of customer M at the position where customer M was detected by the detection unit 142. Then, the selection unit 143 selects the two captured image data generated by the two selected cameras C (S6).

[0104] The distance image creation unit 144 creates distance image data using the two captured image data selected by the selection unit 143 (S7). The recording processing unit 147 stores the distance image data for each customer M in the detection data storage unit 134 (S8). Details of the process by which the recording processing unit 147 stores the distance image data in the detection data storage unit 134 will be described later with reference to FIG.

[0105] The person identification unit 146 analyzes the distance image data created by the distance image creation unit 144 and determines whether the time that customer M included in the distance image data has stayed at the same position is equal to or longer than a threshold (S9). If the person identification unit 146 determines that the stay time is equal to or longer than the threshold (YES in S9), it analyzes multiple captured image data (i.e., video image data) taken at different times to identify the behavior of customer M (S10).

[0106] The control unit 14 repeats the processes from S1 to S10 until an instruction to end the analysis of the captured image data is received or until a preset time comes (NO in S11).

[0107] 13 is a flowchart showing the flow of the process (S8) in which the recording processing unit 147 stores the distance image data in the detection data storage unit 134. Before the recording processing unit 147 stores the distance image data in the detection data storage unit 134, the feature identification unit 145 identifies the features of customer M contained in the distance image data (S101).

[0108] When the feature identification unit 145 identifies a feature of customer M based on distance image data based on captured image data photographed at a certain time, the recording processing unit 147 determines whether a feature ID corresponding to the same feature has been previously recorded in the detection data storage unit 134 (S102). If the same feature ID has been previously recorded in the detection data storage unit 134 (YES in S102), the recording processing unit 147 updates the second detection data by storing the distance image data in the detection data storage unit 134 in association with the recorded feature ID (S103).

[0109] If a feature ID corresponding to the same feature has not been recorded in the detection data storage unit 134 within a predetermined period of time in the past (NO in S102), the recording processing unit 147 creates second detection data corresponding to the new feature ID by storing the range image data in the detection data storage unit 134 in association with the new feature ID (S104). The predetermined period of time in the past is a period set by the manager of the store S, for example, and may be one day or one week.

[0110] Furthermore, when the person identification unit 146 detects suspicious behavior (YES in S105), the record processing unit 147 associates a suspicious flag indicating that the behavior is suspicious with the characteristic ID of the customer M whose suspicious behavior was detected in the detection data storage unit 134 (S106), as shown in Fig. 10. When the person identification unit 146 does not detect suspicious behavior, the record processing unit 147 does not store the suspicious flag in the detection data storage unit 134 (S107).

[0111] [Effects of image processing device 1] As described above, in the image processing device 1, the selection unit 143 selects two pieces of captured image data from the multiple captured image data generated by the multiple cameras C installed in the store S based on the position where the detection unit 142 detected the customer M, and the distance image creation unit 144 creates distance image data using the two selected captured image data. By configuring the image processing device 1 in this way, the multiple captured image data generated by the multiple cameras C can be used to create distance image data that is useful for the person identification unit 146 to identify human behavior with high accuracy.

[0112] The feature identification unit 145 identifies the features of the customer M, and the person identification unit 146 identifies the customer M whose features are identified by the feature identification unit 145 based on the first captured image data generated by the first camera C as being the same customer M when the features identified by the feature identification unit 145 based on the second captured image data generated by the second camera C match. Then, the recording processing unit 147 stores at least one of the captured image data or the distance image data for each identified customer M in the detection data storage unit 134. This configuration of the image processing device 1 makes it easier for the image processing device 1 to identify the details of the behavior of each customer M. It also makes it easier for the manager of the store S to search for captured image data or distance image data that can confirm the behavior of a specific customer M.

[0113] Furthermore, the setting data storage unit 132 stores detection target parts, which are parts of the customer M that are to be detected from the captured image data, and the selection unit 143 selects, from the plurality of captured image data, one or more pieces of captured image data generated by one or more cameras C that can capture the detection target parts stored in the setting data storage unit 132. By the selection of captured image data by the selection unit 143 in this manner, the image processing device 1 only needs to analyze captured image data that is useful for identifying the movement of the parts of the customer M that the manager of the store S wants to monitor, thereby improving the efficiency of the image analysis process.

[0114] Furthermore, when the detection unit 142 determines that the customer M has been staying in the same area continuously for a predetermined period of time or more, the person identification unit 146 associates the feature ID for identifying the customer M detected by the detection unit 142 with the area ID and stores them in the detection data storage unit 134. With the person identification unit 146 configured in this way, the image processing device 1 can provide the manager of the store S with information regarding the relationship between the customer and the area.

[0115] Furthermore, the feature identifying unit 145 identifies the feature of the change in the height of the person's head included in the video data generated by each camera C. By the feature identifying unit 145 identifying such a feature as one of the features of customer M, the accuracy of detecting customer M included in common in the multiple video data generated by the multiple cameras C is improved.

[0116] In the above explanation, we have used the example of a case where camera C is installed in store S and the person whose characteristics and behavior are to be identified by control unit 14 is customer M of store S, but the space in which camera C is installed is arbitrary, and the person whose characteristics and behavior are to be identified by control unit 14 is arbitrary.

[0117] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0118] 1. Image processing device 11 Communications Department 12 Settings reception section 13 Storage section 14 Control Unit 100 Image Processing System 131 Program memory unit 132 Setting data storage unit 133 Area data storage unit 134 Detection data storage unit 141 Data Acquisition Unit 142 Detector 143 Selection Section 144 Range image creation unit 144 Range image creation unit 145 Feature Identification Unit 146 Person Identification Department 147 Recording Processing Unit C Camera

Claims

1. a data acquisition unit that acquires three or more pieces of captured image data generated by three or more imaging devices installed in a predetermined space; a detection unit that detects the position of a person within the predetermined space; a selection unit that selects two pieces of captured image data from the three or more pieces of captured image data based on the position detected by the detection unit; a distance image creating unit that creates distance image data including distance information based on the two captured image data selected by the selecting unit; and The selection unit selects the two captured image data generated by two of the three or more imaging devices that can capture images of a person at a position detected by the detection unit, the two imaging devices having the shortest distance between the position of the person detected by the detection unit and the position of the imaging device.

2. the predetermined space has a plurality of areas, the image processing device further includes an area data storage unit that stores each of the plurality of areas in association with two or more image capturing devices; The detection unit detects an area where a person is present, the selection unit selects the two captured image data generated by the two imaging devices and stored in the area data storage unit in association with the area detected by the detection unit. The image processing device according to claim 1 .

3. a recording processing unit that associates a plurality of pieces of distance image data in which the same person is commonly present with the person and stores the pieces in the detection data storage unit; 3. The image processing device according to claim 1 or 2.

4. the recording processing unit stores the plurality of pieces of range image data in the detection data storage unit in association with the times at which the captured image data corresponding to the plurality of pieces of range image data were generated; The image processing device according to claim 3 .

5. the selection unit selects the two sets of captured image data based on a result of specifying a direction of travel of the person detected by the detection unit, based on a relationship between a plurality of positions corresponding to different times specified by the detection unit. The image processing device according to claim 1 .

6. a data acquisition unit that acquires three or more pieces of captured image data generated by three or more imaging devices installed in a predetermined space; a detection unit that detects the position of a person within the predetermined space; a selection unit that selects two of the three or more imaging devices based on the positions detected by the detection unit; a distance image creation unit that creates distance image data including distance information based on the two captured image data generated by the two imaging devices selected by the selection unit; and The selection unit selects the two captured image data generated by two of the three or more imaging devices that can capture images of a person at a position detected by the detection unit, the two imaging devices having the shortest distance between the position of the person detected by the detection unit and the position of the imaging device.

7. Computer a data acquisition unit that acquires three or more pieces of captured image data generated by three or more imaging devices installed in a predetermined space; a detection unit that detects the position of a person within the predetermined space; a selection unit that selects two pieces of captured image data from the three or more pieces of captured image data based on the position detected by the detection unit; a distance image creating unit that creates distance image data including distance information based on the two captured image data selected by the selecting unit; A program for functioning as The selection unit selects the two captured image data generated by two of the three or more imaging devices that can capture an image of a person at a position detected by the detection unit, the two imaging devices having the closest distance between the position of the person detected by the detection unit and the position of the imaging device.

8. Computer a data acquisition unit that acquires three or more pieces of captured image data generated by three or more imaging devices installed in a predetermined space; a detection unit that detects the position of a person within the predetermined space; a selection unit that selects two of the three or more imaging devices based on the positions detected by the detection unit; a distance image creation unit that creates distance image data including distance information based on the two captured image data generated by the two imaging devices selected by the selection unit; A program for functioning as The selection unit selects the two captured image data generated by two of the three or more imaging devices that can capture an image of a person at a position detected by the detection unit, the two imaging devices having the closest distance between the position of the person detected by the detection unit and the position of the imaging device.

Images