Information processing system, information processing method, and program

Abstract

To improve accuracy of statistical attribute information on an object.SOLUTION: A system includes: an attribute estimation unit configured to estimate an attribute of an object from an image feature amount of a partial image of the object in a captured image; a trajectory generation unit configured to scan the captured image by the partial image and generate trajectory information indicating a movement trajectory of the object; a classification unit configured to classify the movement trajectory into any of a plurality of types of routes based on the generated trajectory information; a determination unit configured to determine whether or not an estimated attribute of the object is a target of statistical processing based on the type of the route into which the movement trajectory is classified; and a statistical unit configured to perform statistical processing by using attribute information indicating the estimated attribute of the object determined as the target.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a technique for acquiring attributes of an object from a captured image. [Background technology]

[0002] Patent document 1 (JP 2019-192154 A) describes an image processing device that includes a detection means for detecting multiple people included in an image, a counting means for counting the number of multiple people detected by the detection means, and a determination means for determining, based on the result of the counting means, a person from among the multiple people detected by the detection means whose attribute information is to be estimated.

[0003] Patent document 2 (JP 2020-155089 A) describes an area setting support device that includes a trajectory detection unit that detects the movement trajectory of each person from a video, a trajectory classification unit that classifies the movement trajectories detected by the trajectory detection unit into multiple types based on the direction of the movement trajectory, and a support information generation unit that generates information to support the setting of a survey area for investigating the situation of a person based on the number of movement trajectories of each type classified by the trajectory classification unit.

[0004] Patent document 3 (JP 2022-73511 A) describes an information processing device that includes a detection unit that detects attributes related to people and their movement patterns from moving images captured by a camera, an assignment unit that assigns an ID to the detected people, and a generation unit that generates the movement route of the person to which the ID has been assigned by linking it to the attributes.

[0005] Patent document 4 (JP 2020-198053 A) states that "gender and age, as attributes of a person, can be recognized with high accuracy using a facial image, but when a person is facing away from the camera, it is difficult to infer the person's gender and age because a facial image cannot be obtained. When a person is detected from an image to be facing away from the camera, the confidence level of features related to gender and age may be set to a low value." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-192154 A [Patent Document 2] JP 2020-155089 A [Patent Document 3] Patent Publication No. 2022-73511 [Patent Document 4] JP 2020-198053 A Summary of the Invention [Problem to be solved by the invention]

[0007] For example, a technique is known in which attributes of an object are estimated by processing an image captured by a camera, and statistical attribute information is acquired from the estimation result and used. Therefore, there is a demand for improving the accuracy of the statistical attribute information. However, Patent Documents 1 to 4 do not propose a technique that meets this demand.

[0008] An object of the present disclosure is to provide an information processing system, an information processing method, and a program capable of improving the accuracy of statistical attribute information about an object. [Means for solving the problem]

[0009] The information processing system according to the present disclosure includes an attribute estimation unit that estimates attributes of an object from image features of a partial image of the object in a captured image, a trajectory generation unit that scans the captured image with the partial image and generates trajectory information indicating a movement trajectory of the object, a classification unit that classifies the movement trajectory into one of a plurality of types of routes based on the generated trajectory information, a determination unit that determines whether the estimated attributes of the object are to be subject to statistical processing based on the type of route into which the movement trajectory is classified, and a statistics unit that performs statistical processing using attribute information indicating the estimated attributes of the object determined to be the target.

[0010] In the above information processing system, the object is a person. In the above information processing system, the determination unit corrects the estimated attributes of the object based on a coefficient corresponding to each type of route in the determination.

[0011] In the above information processing system, the captured images include captured images of an object captured from different imaging directions, and the classification unit further classifies, in the classification, trajectory information generated from the captured image corresponding to the first imaging direction into a first route, and classifies trajectory information generated from the captured image corresponding to the second imaging direction into a second route.

[0012] In the above information processing system, the captured images of an object captured from different imaging directions include captured images of the object captured from the different imaging directions during the same time period.

[0013] In the above information processing system, the determination unit, in making a determination, replaces a predetermined type of attribute estimated for an object whose movement trajectory is classified as a first route with the predetermined type of attribute estimated for the object moving on a movement trajectory classified as a second route.

[0014] In the above information processing system, the estimation accuracy of a predetermined type of attribute for an object whose movement trajectory is classified as a first route is lower than the estimation accuracy of the predetermined type of attribute for the object moving on a movement trajectory classified as a second route.

[0015] The information processing system further includes a route setting unit that sets a route. In the above information processing system, the route setting unit includes a GUI that accepts a user operation for designating a route.

[0016] The above-mentioned information processing system identifies an object from image features of a partial image, and the decision unit, in making the decision, decides whether to subject the attributes estimated for the identified object to statistical processing based on the type of route into which the movement trajectory is classified.

[0017] An information processing method implemented by a computer according to the present disclosure includes the steps of estimating attributes of an object from image features of a partial image of the object in a captured image, scanning the captured image with the partial image to generate trajectory information indicating a movement trajectory of the object, classifying the movement trajectory into one of a plurality of types of routes based on the generated trajectory information, determining whether the estimated attributes of the object are to be subject to statistical processing based on the type of route into which the movement trajectory is classified, and performing statistical processing using attribute information indicating the estimated attributes of the object determined to be the target.

[0018] A program according to the present disclosure is a program for causing a computer to execute the information processing method described above. Effect of the Invention

[0019] In the present disclosure, whether or not to subject an estimated attribute of an object to statistical processing is determined based on the type of route into which the object's movement trajectory is classified. As a result, even if the estimation accuracy of the attribute varies depending on the object's movement trajectory, the accuracy of statistical attribute information can be improved. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram showing a hardware configuration of a terminal 200 according to the present embodiment. [Diagram 3] FIG. 2 is a diagram showing a hardware configuration of an information processing device 300 according to the present embodiment. [Figure 4] FIG. 10 is a diagram illustrating a relationship between a travel path and the accuracy of an attribute. [Diagram 5] FIG. 10 is a diagram illustrating a relationship between a travel path and the accuracy of an attribute. [Figure 6] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 7] 7 is a diagram for explaining the process of the attribute estimation unit 60 in FIG. 6. FIG. [Figure 8] 7 is a diagram for explaining the process of the trajectory generating unit 61 in FIG. 6. FIG. [Figure 9] FIG. 2 is a diagram illustrating tracking according to the present embodiment. [Figure 10] 7 is a diagram for explaining the process of the route classification unit 62 in FIG. 6. [Figure 11] FIG. 2 is a diagram illustrating a route table according to the present embodiment. [Figure 12] FIG. 4 is a diagram illustrating a route classification process according to the embodiment. [Figure 13] 7 is a diagram for explaining the process of the determination unit 63 in FIG. 6. [Figure 14] 7 is a diagram for explaining the processing of the statistics output unit 64 in FIG. 6. [Figure 15] FIG. 2 is a diagram for explaining a specific example 1 according to the present embodiment. [Figure 16] FIG. 2 is a diagram for explaining a specific example 1 according to the present embodiment. [Figure 17] FIG. 11 is a diagram illustrating a specific example 2 according to the present embodiment. [Figure 18] FIG. 11 is a diagram illustrating a specific example 2 according to the present embodiment. [Figure 19] FIG. 11 is a diagram illustrating a specific example 3 according to the present embodiment. [Figure 20] FIG. 9 is a diagram illustrating statistical attribute information 950 according to the present embodiment. [Figure 21] FIG. 4 is a diagram illustrating an example of a user interface according to the present embodiment. [Figure 22] 11A and 11B are diagrams illustrating an example of display of statistical attribute information according to an embodiment of the present invention. [Figure 23] 3 is a flowchart of a process according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. In this embodiment, a person is exemplified as an object (target object) whose attribute information should be detected, but the object is not limited to a person as long as it moves.

[0022] <System configuration> FIG. 1 is a diagram showing a schematic configuration of an information processing system according to the present embodiment. Referring to FIG. 1, the information processing system includes different devices that exchange information including images via a network 100. More specifically, the system includes an information processing device 300 connected to the network, a repeater 210 such as a router, and a terminal 220 such as a computer. The system further includes a terminal 200 such as an edge device that communicates with the repeater 210, and one or more cameras 10 that communicate with the terminal 200. The network 100 is configured by appropriately selecting, for example, a cable, an optical fiber, a wired LAN (Local Area Network), a wireless LAN, the Internet, and the like. The camera 10 is installed, for example, on the ceiling or wall of a commercial facility, and has an imaging field of view that allows an overhead image of the space in the commercial facility. The camera 10 corresponds to an imaging unit. The camera 10 images an object such as a person 5 from approximately above and outputs an imaged image composed of frame images that are continuous in time. The camera 10 has an imaging element such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 converts the amount of light incident on the imaging element into a color image (electrical signal) having pixels corresponding to R (Red), G (Green), and B (Blue), and outputs the color image.

[0023] The terminal 200 and the terminal 220 include, for example, a general-purpose PC (Personal Computer), a smartphone, or a tablet. The terminal 200 transfers captured images (time-series frame images) from each camera 10 to the information processing device 300 via the repeater 210. The information processing device 300 processes the image data transferred from the terminal 200 to identify the captured person, stores information based on the person's identification, and transfers the information to the terminal 220 via the network 100. The terminal 220 outputs the received information via a user interface. Information provided from the information processing device 300 to the terminal 220 may include the trajectory of the movement of a person who visited the commercial facility, information on the store and the visitor (person), and the like. The person information includes attribute information indicating the person's attributes, or statistical attribute information obtained by statistical processing based on the attribute information. The information processing device 300 provides the above information to the terminal 220 as Web data, and has a Web browser that outputs the Web data of the terminal 220.

[0024] In the system of FIG. 1, the camera 10 is installed in a commercial facility, but the location where the camera 10 is installed is not limited to a commercial facility.

[0025] Fig. 2 is a diagram showing a hardware configuration of terminal 200 according to the present embodiment. Fig. 3 is a diagram showing a hardware configuration of information processing device 300 according to the present embodiment.

[0026] Referring to FIG. 2, the terminal 200 includes a processor 11, a main memory 12 including a volatile storage medium including a ROM (Read Only Memory) and a RAM (Random Access Memory), a camera interface 13 for connecting the camera 10, a communication interface 15 for connecting the repeater 210, an input interface 16 for connecting an input device 16A such as a keyboard or a mouse, a display interface 17 for connecting a display 17A, an optical drive 19 to which an optical disk 19A, which is an example of a storage medium, is removably mounted, and a storage device 20 including a non-volatile storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). These components are connected to each other so as to be able to communicate with each other via an internal bus 25. The input interface 16 accepts user operations on the terminal 200 via the input device 16A. The display interface 17 includes a driver circuit for driving the display 17A. The storage device 20 stores a program 22 that realizes various processes when read and executed by the processor 11. The camera interface 13 is configured with a communication circuit including a NIC (Network Interface Card) and transmits and receives data to and from the camera 10 by, for example, Wi-Fi (registered trademark) communication. The communication interface 15 communicates with the repeater 210 according to, for example, a Universal Serial Bus (USB). The terminal 200 may further include a microphone, a speaker, and the like.

[0027] The processor 11 realizes various processes according to the present embodiment by expanding the program 22 stored in the storage device 20 into the main memory 12 and executing it. The main memory 12 provides a working memory necessary for the processor 11 to execute the program.

[0028] 3, the information processing device 300 includes a processor 301, a main memory 302 including a volatile storage medium including a ROM and a RAM, a communication interface 303 including a communication circuit including a NIC, a storage device 304 including a non-volatile storage medium such as a HDD and an SSD, an input interface 305 for connecting an input device 305A such as a keyboard and a mouse that can be operated by a user, a display interface 306 for connecting a display 306A, and an optical drive 307. An optical disk 307A, which is an example of a storage medium, is removably mounted in the optical drive 307. These components are connected to each other via an internal bus 319 so as to be able to communicate with each other. The input interface 305 accepts user operations on the information processing device 300 via the input device 305A. The display interface 306 includes a driver circuit for driving the display 306A based on image data so as to display an image.

[0029] The storage device 304 stores software and data that are read and executed by the processor 301. Such software includes a system program 310 and an application program 320. The storage device 304 also stores information 910-940, statistical attribute information 950, a route table 78, a calculation coefficient 76, a replacement condition 77, and route data 781. Details of this information and data will be described later.

[0030] Processor 301 realizes various processes according to the present embodiment by expanding application program 320 stored in storage device 304 into main memory 302 and executing it. Main memory 302 provides a work memory necessary for processor 301 to execute the program.

[0031] The communication interface 303 transmits and receives data and signals between the terminal 200 and the terminal 220 via the network 100. The optical drive 307 reads various programs (e.g., application program 320) or data stored in an external storage medium such as an optical disk 307A, and installs the programs or data in the storage device 304.

[0032] The processor 11 in Fig. 2 and the processor 301 in Fig. 3 are each configured, for example, by at least one integrated circuit. The integrated circuit is configured, for example, by at least one central processing unit (CPU), at least one application specific integrated circuit (ASIC), at least one field-programmable gate array (FPGA), or a combination thereof. In addition, the external storage medium is not limited to the optical disks 19A and 307A, and may be, for example, a USB memory, an SD (registered trademark) card, a CF (CompactFlash, registered trademark) card, or other storage medium.

[0033] The terminal 200 in FIG. 2 or the information processing device 300 in FIG. 3 shows a configuration example in which the necessary programs are installed into the device via an optical drive, but this is not limited to this, and the necessary programs may be downloaded from other devices on the network 100, etc.

[0034] The program 22 or the application program 320 may be provided not as a standalone program but as part of an arbitrary program. In this case, the processing according to the present embodiment is realized in cooperation with the arbitrary program. In addition, some or all of the functions provided by the program 22 and the application program 320 may be realized by dedicated hardware.

[0035] <Relationship between travel path and attribute accuracy> The inventor has found that the accuracy of estimating a person's attributes based on an image captured by the camera 10 varies depending on the person's movement path in the imaging field of view. For example, for an attribute (such as age or glasses) that can be estimated from the features of a person's face image, if the movement path is a path in which the person approaches the camera, the person's front faces the camera, so the accuracy of estimating the attribute is high. In contrast, if the movement path is a path in which the person moves away from the camera, the person's back faces generally face the camera, so the accuracy of estimating the attribute is low.

[0036] 4 and 5 are diagrams that show a schematic relationship between a travel path and the accuracy of attributes. As shown in FIG. 4A, an escalator 5A ascending from 3F (3rd floor) to 4F (4th floor) and an escalator 5B ascending from 2F (2nd floor) to 3F (3rd floor) are arranged in a commercial facility. Camera 10A has an imaging field of view and an imaging direction that can capture an image of a person traveling on escalator 5A and a person traveling on the escalator ascending from 4F to 5F. Camera 10B has an imaging field of view and an imaging direction that can capture an image of a person traveling on escalator 5B and a person traveling on escalator 5A.

[0037] FIG. 4B shows an example of a captured image captured and output by the camera 10A, and FIG. 4C shows an example of a captured image by the camera 10B. From these captured images, a human rectangle 912 showing a rectangular area surrounding a person is detected. Feature amount information including feature amounts related to the attributes of the person is acquired from a partial image of the human rectangle 912. Here, a scene is assumed in which gender or the presence or absence of glasses is estimated (detected) as a person's attribute. FIG. 4B shows a partial image of a person moving forward with respect to the camera 10A, so the accuracy of estimating the gender or glasses attribute from the partial image is high. In contrast, FIG. 4C shows a partial image of a person moving backward with respect to the camera 10A, so the accuracy of estimating the gender or glasses attribute from the partial image is low. In this way, the accuracy of estimating the attributes of a person (gender, presence or absence of glasses, etc.) differs depending on the moving route of the person acquired from the captured image.

[0038] FIG. 5 (A) shows information obtained by statistically processing the gender of a person estimated from an image captured by the camera 10A in a certain time period, and FIG. 5 (B) shows information obtained by statistically processing the gender of a person estimated from an image captured by the camera 10B in the same time period. The information obtained by statistically processing the gender of a person estimated from the image captured by the camera 10A and the image captured by the camera 10B in the same time period should match, but the statistical values ​​shown in FIG. 5 (A) and (B) do not match. This mismatch is caused by the difference in the estimation accuracy of the attribute depending on the moving route described in FIG. 4. Therefore, the information processing system of the present disclosure selects (determines) whether or not to use the attribute information of the person in the statistical processing according to the moving route of the person, or corrects the attribute information of the person and uses it in the statistical processing. This improves the accuracy of the statistics.

[0039] <Module configuration> FIG. 6 is a diagram showing an example of the configuration of an information processing system according to this embodiment. In this embodiment, the information processing system is configured to include one or more modules. The modules may be implemented using any form of hardware, software, or a combination thereof. For example, a module may be configured by implementing one or more hardware circuits such as a processor, a controller, an ASIC, an FPGA, software including the program 22 or the application program 320, or other mechanisms. Each module constituting the information processing system of FIG. 6 is implemented in an information processing device 300, but the implementation form is not limited thereto. For example, the modules of FIG. 6 may be implemented in a concentrated manner in the terminal 200. Alternatively, some of the modules of FIG. 6 may be implemented in the terminal 200, and the other parts may be implemented in the information processing device 300.

[0040] 6, the information processing system processes the video 50 output by one camera 10, and selects or corrects attribute information according to route information. The video 50 is a moving image composed of time-series frame images (hereinafter simply referred to as image data), and has a predetermined frame rate (fps: frames per second). The frame rate of the video 50 is not limited, but is, for example, 60 fps.

[0041] The information processing system includes an attribute estimation unit 60 that estimates attributes of a person, a trajectory generation unit 61 that generates a movement trajectory of the person, a route classification unit 62 that classifies the generated movement trajectories into routes, a determination unit 63, a statistics output unit 64 that performs statistical processing of the attribute information, and a route setting unit 66 that sets route data 781 for classification. The determination unit 63 determines whether or not to use attribute information indicating the estimated attributes for statistical processing, or corrects or replaces the attribute information for use in the statistical processing.

[0042] (Attribute estimation part) The attribute estimation section 60 estimates the attributes of a person from an image feature amount acquired from a partial image corresponding to the person rectangle 912 in a frame image (captured image).

[0043] Fig. 7 is a diagram for explaining the processing of the attribute estimation unit 60 in Fig. 6. In Fig. 7(A), the attribute estimation unit 60 processes the video 50 output from the camera 10, and outputs information 910 shown in Fig. 7(C). As shown in Fig. 7(B), the video 50 is composed of image data at times ti (i=1, 2, 3,...) arranged in chronological order.

[0044] The attribute estimation unit 60 detects a partial image (partial image data) of a person in the image based on image data of each frame constituting the video 50, and outputs data representing a human rectangle 912 indicating the detection result.

[0045] In order to detect a partial image of a person, the attribute estimation unit 60 has a control point information acquisition unit and a feature amount information acquisition unit. Each of these units may be configured, for example, by a GPU (Graphics Processing Unit). A GPU is a computing device (processor) specialized for real-time image processing. In this way, by configuring the control point information acquisition unit and the like with a GPU, a neural network capable of machine learning such as a convolutional neural network (CNN) can be constructed in the GPU, and this neural network can function as the control point information acquisition unit and the feature amount information acquisition unit.

[0046] The control point information acquisition unit acquires information (control point information) about the control points of a predetermined object, here a person, from an image shown by video 50 captured and acquired by camera 10. Video 50 includes an image of a person. Here, a control point refers to a point that defines the person's skeleton in the image. Thus, control points include joint points between the bones of a person, the center point of the skull, and the tips of the bones of the fingers and toes. In addition, the positions of the person's eyes, nose, and ears in the image can also be control points because they define the person's skeleton (size and shape of the skull). Such control point information includes position information that indicates the position (two-dimensional position) of the control point in the image.

[0047] The feature information acquisition unit inputs the video 50, and acquires feature information including a human rectangle 912 and features related to the attributes of a person from an image constituting the video 50 based on the position information included in the control point information. The human rectangle 912 indicates a rectangular area surrounding a person. The types of attributes of a person include the person's orientation with respect to the camera (the attribute value is "front" facing forward toward the camera 10, and "back" facing away from the camera 10), age (the attribute value is young, middle-aged, elderly), sex (the attribute value is male, female), color of clothes (the attribute value is red, blue, etc.), and whether or not the person wears glasses (the attribute value is with glasses, without glasses). Note that the types of attributes are not limited to these. In this embodiment, the attribute information includes one or more pairs of attribute types and attribute values.

[0048] The attribute estimation unit 60 can estimate the attribute value by using machine learning. For example, the attribute estimation unit 60 calculates the variation of the positions of the control points constituting the person's human rectangle 912 over a plurality of frames. For example, men and women walk differently. More specifically, since men swing their arms more when walking, the variation of the positions of the control points between frames for men tends to be greater than the variation of women. The attribute estimation unit 60 has a machine learning model that receives as input the tendency of the variation of the positions of such control points, identifies the variation as being specific to a person, and outputs the attribute value of the person (male or female). The attribute estimation unit 60 also has a machine learning model that receives as input feature amount information, and outputs attribute values ​​of other types of attributes (orientation, age, color of clothes, presence or absence of glasses).

[0049] The attribute estimation unit 60 outputs information 910 in a table format having records storing information acquired from each image (frame image). The record includes a time 911 (time ti) of the corresponding image, data of a human rectangle 912 detected from the image, estimated attribute information 913, and an estimated ID 916 that identifies the attribute information 913. When the image is considered as a two-dimensional coordinate area, the data of the human rectangle 912 includes the coordinates of a rectangular area surrounding a person in the image (the center position (X, Y) of the human rectangle 912, and the size of the rectangular frame (height H and width W).

[0050] The feature information acquisition unit outputs person information including a human rectangle 912 and attribute information identified for each person image detected from the video 50. Such person information constitutes person image feature amounts acquired from the captured image. The attribute estimation unit 60 has a person identification unit 65 that identifies a person using the person information.

[0051] The person identification unit 65 determines (identifies) whether or not the personal information acquired from each person image detected in the image of a plurality of frames is information of the same person, and outputs identification information indicating the determination result. More specifically, the person identification unit 65 inputs the personal information, and accumulates (stores) the personal information in the storage device 304 with a person ID identifier. Each time personal information is output from the feature amount information acquisition unit, the person identification unit 65 compares the personal information with the personal information accumulated in the storage device 304. When the comparison result indicates a similarity exceeding a threshold value, that is, when identification is detected, the person identification unit 65 outputs identification information having the person ID attached to the matched personal information. At this time, the personal information output from the feature amount information acquisition unit is not accumulated (stored) in the storage device 304. On the other hand, when the above comparison result indicates a similarity equal to or lower than the threshold value, that is, when identification is not performed, the person identification unit 65 accumulates (stores) the personal information output from the feature amount information acquisition unit in the storage device 304 with a person ID identifier, and outputs identification information having the person ID.

[0052] In this embodiment, the feature information acquisition unit processes the image data inputted on a frame-by-frame basis. This allows the person identification unit 65 to identify each person among the image data of different frames. As a result, the storage device 304 stores the person information and person ID of each of the identified one or more people.

[0053] Such person identification can be performed without using a face image of the person because it is realized using feature amounts such as position information of the control points, clothing, and clothing color, etc. Also, person identification may be performed using feature amounts of the face image of the person.

[0054] (trajectory generation part) The trajectory generating unit 61 generates trajectory information indicating the trajectory of the person's movement in the imaging field of view by scanning the captured image with the partial image of the person rectangle 912. "Scanning" corresponds to tracking the movement of the partial image in the time-series frame images.

[0055] FIG. 8 is a diagram for explaining the processing of the trajectory generating unit 61 in FIG. 6. FIG. 9 is a diagram for explaining tracking according to the present embodiment. In FIG. 8(A), the trajectory generating unit 61 performs a process of detecting information for each person based on information 910 in FIG. 8(B) output from the attribute estimating unit 60, and a process of tracking the movement of a human rectangle 912 and generating trajectory information based on the tracking result. Based on the results of these processes, the trajectory generating unit 61 outputs information 920 shown in FIG. 8(C) for each person. With reference to FIG. 9, the trajectory generating unit 61 tracks the movement of the center position of the human rectangle 912 at each time ti indicated by the information 910, thereby generating trajectory information in which the center positions are linked in chronological order. The trajectory information is indicated by a set of data (ti, Xi, Yi) (where i=1, 2, 3, . . .) indicating the center position (X, Y) at time ti.

[0056] The trajectory generating unit 61 judges whether or not the partial images of the human rectangles 912 in the trajectory indicated by the trajectory information show images of the same person, based on the degree of overlap between the areas of the human rectangles 912. More specifically, in the judgment, the trajectory generating unit 61 calculates the degree of overlap between the areas of the human rectangles 912 corresponding to time t1 and time t2, and compares the degree of overlap with a threshold. When the trajectory generating unit 61 detects that the degree of overlap exceeds the threshold based on the comparison result, it judges (identifies) that the people corresponding to both human rectangles 912 are the same person. (C) of FIG. 8 shows an example of information 920 generated by such tracking. In the information 920, four records corresponding to the estimated IDs 916 (ID=1, 2, 3, 4) at times t1 to t4 in the information 910 are integrated into one record corresponding to the judged same person. A record of information 920 includes a person ID 921 that identifies a person, a group 922 of corresponding estimated IDs 916, and a final attribute 923 that is an integration of multiple attribute information 913 corresponding to the person in information 910.

[0057] When the trajectory generation unit 61 generates multiple pieces of trajectory information by tracking based on information 910, it determines whether each piece of trajectory information is for the same person by the process of Figure 8, and generates information 920 for each of the determined people.

[0058] (Route Classification Division) The route classification unit 62 classifies the path indicated by the movement trajectory into one of a plurality of types of routes based on the trajectory information generated for the person.

[0059] Fig. 10 is a diagram for explaining the processing of the route classification unit 62 in Fig. 6. Fig. 11 is a diagram for explaining a route table according to this embodiment. Fig. 12 is a diagram for explaining a route classification processing according to this embodiment. In Fig. 10(A), the route classification unit 62 generates information 930 by assigning route classification information 924 to information 920 for each person in Fig. 10(B) output from the trajectory generation unit 61.

[0060] The route classification unit 62 classifies the trajectory information of the person corresponding to the information 920 into one of the routes shown in the route table 78 in Fig. 11(A). In the route table 78, based on a specific floor where the camera 10 outputting the video 50 is installed, a route moving from a floor lower than the specific floor to the specific floor (the same floor) is classified as "No. 1", a route moving from the same floor to a higher floor is classified as "No. 2", and a route moving from a lower floor to a higher floor is classified as "No. 3". Fig. 11(B) shows an example of the route "No. 2".

[0061] Route data 781 corresponding to each type of route in the route table 78 is stored in the storage device 304. The route data 781 is trajectory information of the corresponding route. The route data 781 is represented by a set of data (ti, xi, yi) (where i=1, 2, 3,...) indicating a position (x, y) at time ti.

[0062] For example, when the movement of the center (X, Y) of the human rectangle 912 corresponding to the information 920 is represented by the time series data (ti, Xi, Yi) (where i=1, 2, 3, . . . , 100) shown in FIG. 12A, the trajectory information 120 based on the time series data is shown in FIG. 12B. In FIG. 12C, the trajectory information corresponding to the route data 781 of "No. 1" and "No. 2" is shown. The route classification unit 62 compares the trajectory information 120 in FIG. 12B with the trajectory information corresponding to the route "No. 1" and the route "No. 2" in FIG. 12C, and identifies the route having a greater similarity as a result of the comparison. The similarity between the trajectory information is acquired as the similarity between the direction and the distance (length) indicated by the trajectory information. For example, the route classification unit 62 may approximate the trajectory indicated by the trajectory information to a vector and acquire the similarity between the vectors.

[0063] The route classification unit 62 acquires route classification information 924 including a route No. 73B, which is the type of the identified route, and an identifier 73A of the camera 10 that outputs the video 50. In this embodiment, the video 50 includes the identifier of the camera 10. The route classification unit 62 acquires the identifier 73A of the camera 10 from the video 50.

[0064] (Decision Section) The determination unit 63 determines whether to subject the attribute estimated for the person to statistical processing based on the type of route into which the person's movement trajectory is classified. In this determination, the determination unit 63 performs correction of attribute values ​​or replacement of attribute values, which will be described later.

[0065] Fig. 13 is a diagram for explaining the processing of the determination unit 63 in Fig. 6. The determination unit 63 in Fig. 13(A) determines the final attribute 923 of each person in the information 930 in Fig. 13(C) based on the route classification information 924 of the person and the calculation coefficient 76 or the replacement condition 77. As a result, the information 940 in Fig. 13(D) is output. The user can set the calculation coefficient 76 and the replacement condition 77.

[0066] An example of the calculation coefficient 76 and the replacement condition 77 is shown in FIG. 13B and FIG. 13C. The calculation coefficient 76 in FIG. 13B includes a count coefficient, a gender coefficient, and a glasses coefficient for each route No. indicated by the route table 78. Here, the types of attributes used in the statistical processing are the number of people (count), gender, and the presence or absence of glasses, but are not limited to these types. More specifically, since the people moving on the routes "No. 1" and "No. 3" move facing forward with respect to the camera 10, the estimation accuracy of the gender and the presence or absence of glasses in the attribute estimation unit 60 is high. Therefore, in the calculation coefficient 76, the corresponding coefficient is set to "1". In contrast, the people moving on the route "No. 2" move facing backward with respect to the camera 10, so the estimation accuracy of the gender and the presence or absence of glasses in the attribute estimation unit 60 is low. Therefore, in the calculation coefficient 76, the corresponding coefficient is set to "N / A (NA)" (Not Available, not available for statistical processing). In the calculation coefficient 76 in FIG. 13B, the count coefficient is "1" regardless of the route number, but the count coefficient may also be made different depending on the route number.

[0067] In this way, the estimation accuracy of the attribute values ​​of the predetermined types of attributes (gender and glasses) of the person moving on the movement trajectory classified as the first route (route "No. 2") is low. Therefore, the correction / decision unit 63 corrects these attribute values ​​to N / A based on the calculation coefficient 76, and instructs the statistics output unit 64 not to use the attribute in the statistical processing. This can improve the accuracy of the statistical attribute information 950.

[0068] The replacement condition 77 includes a condition 7A and replacement information. The replacement information includes the type of attribute whose value is to be replaced and the value after replacement. In the replacement condition 77 of Fig. 13(C), the condition 7A is set to "none", so that the corresponding replacement information is not set. The condition 7A may be indicated by a route number, or may be indicated by a combination of the route number and the camera identifier.

[0069] The determination unit 63 corrects the value of each attribute in the final attribute 923 for each person in the information 930 based on the calculation coefficient 76. For example, if the gender, presence or absence of glasses, age, orientation, and clothing color corresponding to the routes "No. 1" and "No. 3" are all set to "N / A" as the calculation coefficient 76, the values ​​of the gender, presence or absence of glasses, age, orientation, and clothing color in the final attribute 923 corresponding to the route No. 73B indicating "No. 1" or "No. 3" in the information 930 are all corrected to "N / A". By such correction, the information 940 in FIG. 13(D) is generated from the information 930.

[0070] In this way, the determination unit 63 determines whether to include or not include (exclude or exclude) the attribute information (final attribute 923) of each person in the information 930 in the statistical processing according to the person's movement route (classified route No.) based on the calculation coefficient 76 or the replacement condition 77. Furthermore, the determination unit 63 selectively determines one or more types of attributes to be used in the statistical processing from among the multiple types of attributes indicated by the person's attribute information (final attribute 923). For example, the determination unit 63 selects one or more types of attributes from the multiple types of attributes based on the person's movement route (classified route No.), and determines to use the selected attributes in the statistical processing.

[0071] (Statistical output section) The statistics output unit 64 performs statistical processing using attribute information indicating the attributes of the person determined by the determination unit 63 to be a subject of statistical processing.

[0072] Fig. 14 is a diagram for explaining the processing of the statistics output unit 64 of Fig. 6. The statistics output unit 64 of Fig. 14(A) processes attribute information based on information 940 of Fig. 14(B) from the determination unit 63, and outputs statistical attribute information 950 as a result of the statistical processing. The statistical attribute information 950 includes information 952 indicating statistical values ​​of attribute information for a field 951 to be monitored in a commercial facility in which the camera 10 is installed. The field 951 corresponds to, for example, the imaging field of view of the camera 10. The field 951 is indicated by the identifier 73A of the camera 10. The information 952 includes a count 724 of the number of people and statistical values ​​725 of the attribute values.

[0073] In the statistical processing, the statistics output unit 64 scans the information 940, detects one or more corresponding final attributes 923 for each identifier 73A of the camera 10, counts the number of detected final attributes 923, and acquires a count value as a count 724. In addition, based on the detected one or more final attributes 923, it calculates a statistical value of the presence or absence of glasses and a statistical value of gender, and acquires the calculation results as attribute statistical value 725. For example, the statistics output unit 64 calculates the ratio (%) of glasses as the statistical value of the presence or absence of glasses, and calculates the male / female ratio as the statistical value of gender.

[0074] <Application Examples> A specific application example of the above-mentioned processing will be described with reference to the drawings.

[0075] (Example 1) 15 and 16 are diagrams for explaining a specific example 1 according to the present embodiment. As shown in (A) of Fig. 15, camera 10A is placed on the 4th floor and camera 10B is placed on the 3rd floor so as to be able to capture an image of a person traveling on an escalator ascending from the 3rd floor to the 4th floor. Camera 10A captures an image of the front of the person traveling on the escalator, and camera 10B captures an image of the back of the person traveling on the escalator.

[0076] Fig. 15B is an example of an image captured by camera 10A, and Fig. 15C is an example of an image captured by camera 10B. The attribute estimation unit 60 estimates the attributes of the person from a partial image of a human rectangle 912 detected from these captured images.

[0077] 16A indicates, for example, a coefficient for attribute information estimated from a captured image corresponding to a combination of camera 10B on 3F and route No. "2" (from the same floor to upper floors). This coefficient includes a coefficient "1" indicating that the attribute information can be used for statistical processing (counting) of the number of people, and a coefficient "N / A" indicating that the attribute information cannot be used for statistical processing of gender and glasses.

[0078] The determination unit 63 generates information 940 (Output) in FIG. 16B, for example, by processing information 930 (Input) in FIG. 16B based on the calculation coefficient 76 in FIG. 16A. This information 940 determines attribute information 741 for a person whose image was captured by the camera 10 with the identifier "3" and whose movement route was classified as a route "2". The attribute information 741 indicates that attributes other than gender and glasses are not subject to statistical processing. As a result, attribute values ​​with low estimation accuracy, that is, the attribute values ​​of gender and glasses of a person who moved along a movement route classified as a route "2", are not used in the statistical processing. As a result, the accuracy of the statistical attribute information 650 is improved.

[0079] (Example 2) 17 and 18 are diagrams for explaining a specific example 2 according to the present embodiment. As shown in (A) of Fig. 17, camera 10A is placed on the 4th floor and camera 10B is placed on the 3rd floor so as to be able to capture an image of a person traveling on an escalator ascending from the 3rd floor to the 4th floor. Camera 10A captures an image of the front of the person traveling on the escalator, and camera 10B captures an image of the back of the person traveling on the escalator.

[0080] FIG. 17(B) is an example of an image captured by camera 10A, and FIG. 17(C) is an example of an image captured by camera 10B. These captured images are taken at the same time of day. The partial images of human rectangle 912 detected from each image show the front and back of the same person. Thus, replacement condition 77 instructs that attribute information corresponding to camera 10B on 3F and route No. "2" (from the same floor to an upper floor) should be replaced with attribute information corresponding to camera 10A on 4F and route No. "1" (from a lower floor to an upper floor), as shown in FIG. 18(A).

[0081] The determination unit 63 processes, for example, information 930 (Input) in FIG. 18B based on the replacement condition 77 to generate information 940 (Output) in FIG. 18B. In the information 940, attribute information 745 of a person whose moving route is classified as route "2" among people captured by the camera 10 with the identifier "3" is replaced with attribute information corresponding to the camera 10 with the identifier "4" and route "1". As a result, although the estimation accuracy of the attribute value of a person moving on route "2" is low, this attribute value is replaced with the attribute value of the person moving on route "1" that is estimated with high accuracy. The statistics output unit 64 can use the highly accurate attribute information 745 for statistical processing.

[0082] (Example 3) FIG. 19 is a diagram for explaining a specific example 3 according to the present embodiment. When a person moving on an escalator is captured by cameras 10A and 10B arranged as shown in (A) to (C) of FIG. 15, camera 10A captures the front of the person and camera 10B captures the back of the person in the same time period. Therefore, among the attribute information estimated from the person image (partial image of person rectangle 912) under the image capture condition of camera 10A, the estimation accuracy of a backpack-type bag, for example, is low, but the estimation accuracy of glasses is high. In contrast, among the attribute information estimated from the image captured by camera 10A, the estimation accuracy of the backpack-type bag is high, but the estimation accuracy of glasses is low. Therefore, the replacement condition 77 in FIG. 19(A) instructs a condition 771 for the person of the attribute information to be replaced and a specification 772 of the attribute information to be replaced.

[0083] 19(A) instructs that, for the attribute information of a person corresponding to the pair of camera 10A on 4F and route No. "1" (from lower floor to upper floor), the attribute value of the bag should be replaced with the attribute value of the bag corresponding to the pair of camera 10B on 3F and route "2" (from the same floor to an upper floor). Also, replacement condition 77 instructs that, for the attribute information of a person corresponding to the pair of camera 10B on 3F and route No. "2" (from the same floor to an upper floor), the attribute value of glasses should be replaced with the attribute value of glasses corresponding to the pair of camera 10A on 4F and route "1" (from lower floor to upper floor).

[0084] The determination unit 63 processes the information 930 (Input) in FIG. 19B based on the replacement condition 77 in FIG. 19A, for example, to generate information 940 (Output) in FIG. 19B including the replaced attribute 746. In this information 940, the attribute value of the glasses of the person whose image was captured by the camera 10 (camera 10B on 3F) with the identifier "3" and whose movement route was classified as route "2" is replaced with the attribute value of the glasses of the camera 10 whose identifier is "4" (camera 10A on 4F) and whose route number is "1". In addition, the attribute value of the bag of the person whose image was captured by the camera 10 (camera 10A on 4F) and whose movement route was classified as route "1" is replaced with the attribute value of the bag of the camera 10 whose identifier is "3" (camera 10B on 3F) and whose route number is "2". The statistics output unit 64 performs statistical processing using the attribute information including the replaced attribute 746 .

[0085] In this way, the calculation coefficients 76 and the replacement conditions 77 are set so that the attribute information can be changed to more accurate information according to the person's movement route. Therefore, the accuracy of the statistical attribute information 650 is improved by statistical processing using the changed attribute information.

[0086] (Advantages of the embodiment) Fig. 20 is a diagram for explaining statistical attribute information 950 according to this embodiment. Fig. 20 shows a case where statistical processing is performed by acquiring person attributes from an image captured by camera 10 installed on 3F shown in Fig. 15(C). Fig. 20(A) shows the movement of people, with 50 people actually moving by escalator from 2F to 3F and 50 people moving by escalator from 3F to 4F, and with male:female ratio of 4:6. At this time, the so-called actual number of people acquired by visually checking the image captured by camera 10 on 3F was 100 people and male:female ratio of 4:6.

[0087] (B) of FIG. 20 and (C) of FIG. 20 show the results of a simulation carried out under the same conditions as (A) of FIG. 20. More specifically, (B) of FIG. 20 shows the number of people and the ratio of men to women based on information 910 estimated using the machine learning model of the attribute estimation unit 60 from the image feature amount detected from the captured image of the camera 10. (C) of FIG. 20 shows statistical information (existing information) and statistical attribute information 950 according to this embodiment. The statistical attribute information 950 shows a value closer to the actual measured value of FIG. 20(A) than the existing information. That is, the statistical attribute information 950 calculated based on the corrected attribute information to which the calculation coefficient 76 or the replacement condition 77 is applied has higher accuracy than the existing statistical information calculated based on the attribute information to which the calculation coefficient 76 and the replacement condition 77 are not applied. In this way, the inventor has found that the accuracy of the statistical attribute information can be improved by determining (selecting) or correcting the attribute information to be used for statistics according to the moving route.

[0088] <User interface> FIG. 21 is a diagram showing an example of a user interface according to the present embodiment. Route setting unit 66 provides a graphical user interface (GUI) of FIG. 21 for receiving a user operation for setting route data 781. FIG. 21 shows a screen displayed on display 306A, which includes a plurality of objects capable of receiving a user operation. The objects include buttons 211-217 and a window W2. Window W2 is an area in which an image captured by camera 10 is displayed, and the size of window W2 corresponds to the size of the imaging field of view. The user sets route data corresponding to each route No. in route table 78 in the area of ​​window W2.

[0089] In FIG. 21A, when the button 211 is operated, the route setting unit 66 displays an image in the window W2 and starts the setting process. When the button 213 is operated, the route setting unit 66 displays grid lines superimposed on the image in the window W2. The route setting unit 66 changes the size of the grid formed by the grid lines according to the user's operation of the button 212. The user is guided by the grid lines and draws a line with a pointer such as a finger on the image in the window W2. When the button 214 is operated, the route setting unit 66 scans the line drawn in the window W2 to obtain time-series coordinate data (ti, xi, yi). When the button 217 is operated, the route setting unit 66 stores the obtained coordinate data (ti, xi, yi) as the route data 781 corresponding to the route number specified by the button 215. Note that the same route number specified by the button 215 can be assigned to multiple route data 781.

[0090] When the user operates the image of the route P drawn in the window W2, the route setting unit 66 changes the route data 781 corresponding to the route P in accordance with the amount of operation. With reference to (B) of Fig. 21, when the button 217 is operated, the route setting unit 66 scans the line of the route P in the window W2 to obtain time-series coordinate data (ti, xi, yi) and rewrites the route data 781 with the obtained coordinate data (ti, xi, yi).

[0091] Via such a GUI, the user can interactively set route data 781 corresponding to the route number for each camera 10. The route setting unit 66 may set the route data 781 using a setting file.

[0092] FIG. 22 is a diagram showing an example of display of statistical attribute information according to an embodiment of the present invention. The terminal 220 outputs the statistical attribute information 950 received from the information processing device 300 via a user interface, for example, as shown in FIG. 22. Information provided from the information processing device 300 to the terminal 220 may include the movement line (movement route) of people who visited the commercial facility, information on the store and the visitors (people), and the statistical attribute information 950. The information processing device 300 provides this information to the terminal 220 as Web data, and the terminal 220 has a Web browser that outputs the Web data. On the screen of FIG. 22, for example, the number of visitors every day, the number of men, the number of women, and an approximate male-female ratio are displayed as the statistical attribute information 950.

[0093] <Flowchart> FIG. 23 is a flowchart of the process according to the present embodiment. The computer (processor 301) of the information processing device 300 receives a captured image from the camera 10 (step S11). The computer detects a partial image of a predetermined object, for example, a human rectangle 912 of a person, from the captured image (step S12), and estimates the attribute of the object from the image feature of the detected partial image (step S13). The computer generates trajectory information indicating the movement trajectory of the object by scanning the captured image with the partial image (step S14), and classifies the movement trajectory into one of a plurality of types of routes based on the generated trajectory information (step S15). The computer determines whether or not to subject the attribute estimated for the object to statistical processing based on the type of route into which the movement trajectory is classified (step S16). The computer performs statistical processing using attribute information indicating the attribute estimated for the object determined to be the target (step S17), and outputs (stores, displays, transfers) the result of the statistical processing (step S18). The processes in steps S12 and S13 may be performed in one step.

[0094] In this embodiment, attributes of people are processed, but the attributes are not limited to people, and may be, for example, livestock (farm animals) including pets, or industrial products in a factory. The information processing system is also capable of processing multiple images 50 output by multiple cameras 10 in parallel. The movement path of a person is not limited to the movement path of a person going up a floor on an upward escalator as shown in FIG. 4, but may be the movement path of a person going down a floor on a downward escalator. The movement of a person is not limited to the use of an escalator. For example, the passage of an attraction in an amusement park, or a passage with a direction of movement determined for infection control or the like may be used for the movement of a person.

[0095] Although the embodiments have been described above, the embodiments disclosed herein are illustrative and not restrictive in all respects. The technical scope of the present disclosure is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0096] 5 Person, 5A,5B Escalator, 7A,771 Condition, 10,10A,10B Camera, 11,301 Processor, 12,302 Main memory, 13 Camera interface, 15,303 Communication interface, 16,305 Input interface, 16A,305A Input device, 17,306 Display interface, 17A,306A Display, 19,307 Optical drive, 19A,307A Optical disk 20,304 storage device, 22 program, 25,319 internal bus, 50 video, 60 attribute estimation unit, 61 trajectory generation unit, 62 route classification unit, 63 determination unit, 64 statistical output unit, 65 person identification unit, 66 route setting unit, 73A identifier, 76 calculation coefficient, 77 replacement condition, 78 route table, 100 network, 120 trajectory information, 200,220 terminal, 210 repeater, 211,212,213,214,215,217 button, 300 information processing device, 310 system program, 320 application program, 950 statistical attribute information, 724 count, statistical value, 741,745,913 attribute information, 746 attribute, 772 designation, 781 Route data,910,920,930,940,952 information,912 person rectangle,923 final attributes,924 route classification information,S11,S12,S13,S14,S15,S16,S17,S18 steps,W2 window.

Claims

1. an attribute estimation unit that estimates attributes of an object from image features of a partial image of the object in a captured image; a trajectory generating unit that scans the captured image with the partial image and generates trajectory information indicating a movement trajectory of the object; A classification unit that classifies the movement trajectory into one of a plurality of types of routes based on the generated trajectory information; a determination unit that determines whether to subject the estimated attribute of the object to statistical processing based on the type of route into which the trajectory is classified; A statistical unit that performs the statistical processing using attribute information indicating estimated attributes of the object determined to be the target.

2. The information processing system according to claim 1 , wherein the object is a person.

3. The determination unit is 3. The information processing system according to claim 1, wherein in the determination, the estimated attributes of the object are corrected based on a coefficient corresponding to each type of the route.

4. the captured images include captured images of the object captured from different imaging directions, The classification unit further includes:

3. The information processing system according to claim 1, wherein in the classification, trajectory information generated from an image captured in a first imaging direction is classified into a first route, and trajectory information generated from an image captured in a second imaging direction is classified into a second route.

5. The information processing system according to claim 4 , wherein the captured images of the object captured from the different imaging directions include captured images of the object captured from the different imaging directions during the same time period.

6. The determination unit is 5. The information processing system according to claim 4, wherein in the determination, a predetermined type of attribute estimated for an object on the movement trajectory classified as the first route is replaced with the predetermined type of attribute estimated for the object moving on a movement trajectory classified as the second route.

7. 7. The information processing system according to claim 6, wherein an estimation accuracy of the predetermined type of attribute for an object whose movement trajectory is classified into the first route is lower than an estimation accuracy of the predetermined type of attribute for the object whose movement trajectory is classified into the second route.

8. The information processing system according to claim 1 , further comprising a route setting unit that sets the route.

9. The route setting unit The information processing system according to claim 8 , further comprising a GUI that accepts a user operation for specifying the route.

10. the information processing system includes an identification unit that identifies the object from an image feature amount of the partial image, The determination unit is The information processing system according to claim 1 , wherein the determining step determines whether or not to subject the attributes estimated for the identified object to statistical processing based on a type of route into which the movement trajectory is classified.

11. 1. A computer-implemented information processing method, comprising: estimating attributes of an object from image features of a partial image of the object in a captured image; scanning the captured image with the partial images to generate trajectory information indicating a movement trajectory of the object; classifying the movement trajectory into one of a plurality of types of routes based on the generated trajectory information; determining whether to subject the estimated attributes of the object to statistical processing based on the type of route into which the movement trajectory is classified; performing the statistical processing using attribute information indicating estimated attributes of the object determined to be the target.

12. A program for causing a computer to execute the information processing method according to claim 11.

Images