Image analysis device, image analysis method, and image analysis program

Abstract

An image analysis device (100) comprises: a time-series image acquisition unit (10) that acquires a time-series image in which multiple persons are captured; a skeletal-point information acquisition unit (20) that acquires multiple skeletal-point information items of the respective persons from each frame image in the time-series image; an attribute information acquisition unit (30) that acquires multiple attribute information items corresponding to the respective skeletal-point information items; an attribute-attached skeletal-point information generation unit (40) that associates the skeletal-point information items with the attribute information items and generates multiple attribute-attached skeletal-point information items; a behavioral analysis unit (50) that conducts behavioral analysis on each of the multiple attribute-attached skeletal-point information items so as to analyze multiple behavioral analysis results in a composite manner; and a scene analysis output unit (60) that outputs an analysis result on a scene in the time-series image.

Description

Image analysis device, image analysis method, and image analysis program

[0001] The present invention relates to an image analysis device, an image analysis method, and an image analysis program.

[0002] Conventionally, a device has been proposed that creates a histogram of the occurrence frequency of behavioral features for each person's attribute and outputs a scene analysis result based on the behavioral features.

[0003] For example, Patent Document 1 discloses an abnormal behavior detection system in which "the system extracts skeletal position information representing the skeletal position of a person from a person included in each frame of a time-series image, which is a time series of photographed images and consists of a certain number of frames or more. The system determines whether the people included in each frame of the time-series image are the same person based on the extracted skeletal position information. The system generates behavioral features for the same person included in a certain number of frames or more of the time-series image based on the extracted skeletal position information. The system generates an attribute histogram by histogramming the appearance frequency of the generated behavioral features for each attribute of the person, and stores the attribute histogram in a memory unit, which can be used to detect abnormal behavior of the person based on the person's behavioral features" (see the abstract of Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2022-17939

[0005] The abnormal behavior detection system disclosed in Patent Document 1 performs complex scene recognition and judgment based only on the attributes and behavior recognition results of individual people. However, it may not be possible to determine whether an individual person is normal or abnormal based on the analysis results alone.

[0006] Specifically, imagine a situation where a teacher and students are in a classroom and the students are walking around the classroom. In this case, if the teacher is writing on the blackboard, the student's walking around the classroom is judged to be a disruption of the class. On the other hand, if the teacher is giving instructions to the students and the students are walking towards the blackboard, the classroom is judged to be a normal lesson.

[0007] In this way, the behavior of students walking around the classroom is recognized as normal or abnormal depending on the interaction with the teacher's behavior. For this reason, there are situations where accurate recognition is not possible based solely on the attributes and behavioral recognition of individual people.

[0008] Therefore, an object of the present invention is to provide an image analysis device, an image analysis method, and an image analysis program that are capable of performing highly accurate scene recognition based on attributed skeleton point information.

[0009] That is, the above-mentioned object of the present invention is achieved by the following configuration: (1) An image analyzing device comprising: a time-series image acquiring unit that acquires time-series images of a plurality of people, a skeleton point information acquiring unit that acquires a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images, an attribute information acquiring unit that acquires a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information, an attribute-attached skeleton point information generating unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information, a behavior analyzing unit that analyzes the behavior of each of the plurality of pieces of attribute-attached skeleton point information and performs a composite analysis of a plurality of behavior analysis results, and a scene analysis output unit that outputs an analysis result of a scene in the time-series images.

[0010] (2) The image analysis device according to (1), wherein the scene analysis output unit outputs a result of the analysis of the scene based on a relationship between different attributes and a result of behavior analysis of the plurality of attribute-attached skeleton point information.

[0011] (3) The image analysis device according to (1), wherein the scene analysis output unit analyzes interactions between the plurality of pieces of attribute-attached skeleton point information when outputting the scene analysis result.

[0012] (4) The image analyzing device according to (3), wherein the analysis of the interaction is outputting an analysis result of the scene based on a behavioral analysis result of the second attributed skeleton point information resulting from the first attributed skeleton point information having first attribute information relating to a first person acting on the second attributed skeleton point information having second attribute information relating to a second person different from the first person.

[0013] (5) The image analysis device according to (2), wherein the attribute information between the different attributes is based on the role of the person.

[0014] (6) The image analysis device according to (1), wherein the attributed skeleton point information generating unit forms a group of attributed skeleton point information to which the same attribute is assigned with respect to the plurality of attributed skeleton point information.

[0015] (7) The image analysis device according to (1), wherein the scene analysis output unit outputs an alert in accordance with a result of the scene analysis.

[0016] (8) The image analysis device according to (1), wherein the behavior analysis unit stores the time-series images obtained when analyzing the plurality of behavior analysis results.

[0017] (9) The image analysis device according to (8), wherein the behavior analysis unit stores the time-series images in a buffer until the processing of the composite analysis is completed, and deletes the time-series images when the processing is completed.

[0018] (10) The image analysis device according to (1), wherein the attribute information acquisition unit acquires the attribute information from the time-series images simultaneously with the skeleton point information acquisition unit acquiring the skeleton point information from the time-series images.

[0019] (11) An image analysis device comprising: a time-series image acquisition unit that acquires time-series images in which a plurality of people are captured; a skeleton point information acquisition unit that acquires a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generation unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; and a scene analysis output unit that outputs an analysis result of a scene in the time-series images based on the plurality of attribute-attached skeleton point information.

[0020] (12) An image analysis method that executes the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; generating a plurality of pieces of attributed skeletal point information by associating the skeletal point information with the attribute information; analyzing the behavior of each of the plurality of pieces of attributed skeletal point information and performing a composite analysis of a plurality of behavioral analysis results; and outputting an analysis result of a scene in the time-series images.

[0021] (13) An image analysis method that executes the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; and outputting an analysis result of a scene in the time-series images based on the plurality of pieces of attributed skeleton point information.

[0022] (14) An image analysis program for causing a computer to execute the following steps: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; generating a plurality of pieces of attributed skeletal point information by associating the skeletal point information with the attribute information; analyzing the behavior of each of the plurality of pieces of attributed skeletal point information and performing a composite analysis of a plurality of behavior analysis results; and outputting an analysis result of a scene in the time-series images.

[0023] (15) An image analysis program for causing a computer to execute the following steps: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; generating a plurality of pieces of attributed skeletal point information by associating the skeletal point information with the attribute information; and outputting an analysis result of a scene in the time-series images based on the plurality of pieces of attributed skeletal point information.

[0024] According to the present invention, highly accurate scene recognition can be performed based on attributed skeleton point information.

[0025] 7 is a functional block diagram showing the functions of an image analyzing device according to a first embodiment. FIG. 8 is a block diagram showing hardware of an image analyzing device according to the first embodiment. FIG. 9 is an explanatory diagram showing a state in which a person is making a fist pump. FIG. 10 is an explanatory diagram showing parts that can be detected by OpenPose. FIG. 11 is a list of parts that can be detected in FIG. 4. FIG. 12 is a diagram showing a skeleton estimation table in which a skeleton point information acquisition unit acquires a human rectangle, skeleton information, and a human ID and estimates a skeleton. FIG. 13 is a diagram showing an attribute information acquisition table in which an attribute information acquisition unit acquires attribute information and a human ID. FIG. 14 is a diagram showing a table in which the attribute information acquisition table of FIG. 7 is expanded to include time and estimated IDs. FIG. 15 is an information integration table in which the skeleton estimation table of FIG. 6 and the attribute information acquisition table of FIG. 16 are integrated. FIG. 16 is a diagram showing a behavior information table in which behavior information is shown for each integrated human ID. FIG. 17 is a diagram showing a behavior analysis table in which the behavior information table is integrated into the information integration table. FIG. 18 is a diagram showing an interaction table generated by a scene analysis output unit. FIG. 19 is a functional block diagram showing the functions of an image analyzing device according to a second embodiment. FIG. 19 is a functional block diagram showing the functions of an image analyzing device according to a third embodiment.

[0026] The following describes in detail embodiments of the present invention. Note that the embodiments described below are merely examples for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0027] 1 is a functional block diagram showing the functions of an image analysis device 100 according to a first embodiment. The image analysis device 100 is generally a device that performs behavioral analysis of attributed skeleton point information, and performs a composite analysis of the behavioral analysis results to output a scene analysis result.

[0028] As shown in FIG. 1, the image analysis device 100 is configured to include a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, a behavior analysis unit 50, and a scene analysis output unit 60.

[0029] The time-series image acquisition unit 10 acquires images of multiple people over time from a camera 200 (see FIG. 2 ), which will be described later. In other words, the time-series image acquisition unit 10 acquires time-series images of multiple people. Therefore, the time-series image acquisition unit 10 can output time-series images in units of one frame. Note that the time-series image acquisition unit 10 may acquire images of multiple people from a storage in which already-shot videos are recorded, and is not particularly limited.

[0030] The skeleton point information acquisition unit 20 acquires multiple skeleton point information for each of multiple people from each frame image of the time-series image, frame by frame. The skeleton point information acquisition unit 20 estimates the skeleton of each of the multiple people for each frame, thereby acquiring skeleton point information and a person ID (IDentification) (for each person). The skeleton point information acquisition unit 20 also acquires a person rectangle. A person rectangle is an area (coordinates) where a person is located, refers to a rectangular image area, and refers to an area for object detection (person detection). The skeleton point information acquisition unit 20 acquires skeleton point information, a person rectangle, and a person ID, which are key points for each of the multiple people, from the frame images, using, for example, Openpose developed at Carnegie Mellon University.

[0031] The attribute information acquisition unit 30 estimates the attributes of the subject (person) from each frame image of the time-series image to acquire attribute information and a person ID. In other words, the attribute information acquisition unit 30 acquires multiple pieces of attribute information corresponding to each of multiple pieces of skeleton point information. Therefore, the attribute information acquisition unit 30 acquires a person rectangle, attribute information, and a person ID (per person). The attribute information acquisition unit 30 acquires the attribute information, person rectangle, and person ID for each of multiple people from the frame images using, for example, FORXAI developed by the applicant of the present application. Furthermore, for example, in analyzing a school, the attribute information acquisition unit 30 may assign an attribute indicating that the person is a teacher or a student based on age information from the attribute information.

[0032] Furthermore, the attribute information acquiring unit 30 may acquire attribute information from the time-series images at the same time that the skeleton point information acquiring unit 20 acquires skeleton point information from the time-series images.

[0033] The attribute-attached skeleton point information generation unit 40 integrates skeleton point information and attribute information of the same person. In other words, the attribute-attached skeleton point information generation unit 40 associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information. The attribute-attached skeleton point information generation unit 40 can output a human rectangle, skeleton points, attribute information, and an integrated human ID (for each person).

[0034] Furthermore, the attribute-attached skeleton point information generating unit 40 may form groups of attribute-attached skeleton point information to which the same attribute is assigned for a plurality of attribute-attached skeleton point information. For example, the attribute-attached skeleton point information generating unit 40 may form a group of attribute-attached skeleton point information to which the attribute of male is assigned, or a group of attribute-attached skeleton point information to which the attribute of middle-aged person is assigned.

[0035] The behavior analysis unit 50 analyzes the behavior of people in a scene based on the skeleton point information and attribute information of the same frame image. In other words, the behavior analysis unit 50 performs a composite analysis of multiple behavior analysis results obtained by analyzing the behavior of each of multiple pieces of attribute-attached skeleton point information.

[0036] For example, when a school is the subject of analysis, a situation is assumed in which a teacher is teaching students in a classroom. In this case, the behavior analysis unit 50 analyzes a combination of the teacher's behavior and the student's behavior. The behavior analysis unit 50 acquires a human rectangle, skeleton points, attribute information, behavior information, and an integrated person ID (per person). Then, the behavior analysis unit 50 analyzes, for example, the following four states:

[0037] The behavior analysis unit 50 analyzes whether the teacher and students in the frame image fall into one of four states: (1) a state in which the teacher is not moving and the students are sleeping, (2) a state in which the teacher is not moving and the students are wandering around, (3) a state in which the teacher is not moving and the students are walking towards the blackboard, or (4) a state in which the teacher is wandering around and the students are not moving. Note that the objects analyzed by the behavior analysis unit 50 are not limited to teachers and students in a school, but may also be store clerks and customers, police officers and a crowd, etc.

[0038] Here, the behavior analysis unit 50 performs a composite analysis of the group belonging to the teacher and the group belonging to the students. The behavior analysis unit 50 of this embodiment analyzes the interactions between groups by fitting them into one-to-one, one-to-many, or many-to-many relationships. In this way, for example, the analysis performed by the behavior analysis unit 50 using the interactions between groups is referred to as a composite analysis.

[0039] The behavior analysis unit 50 may also store time-series images obtained when analyzing a plurality of behavior analysis results in the storage unit 120. The behavior analysis unit 50 may also store the time-series images in a buffer until the processing of the composite analysis is completed, and delete the time-series images when the processing is completed.

[0040] The scene analysis output unit 60 analyzes the scene from the attribute information and the behavior information. In other words, the scene analysis output unit 60 outputs the analysis result of the scene in the time-series images.

[0041] In the case of the school mentioned above, (1) when the teacher is not moving and the students are sleeping means that the class is boring and indicates that the teacher is not teaching well. (2) when the teacher is not moving and the students are wandering around means that the class is falling apart. (3) when the teacher is not moving and the students are walking towards the blackboard means that the class is normal and the students are going to write on the board. (4) when the teacher is wandering around and the students are not moving also means that the class is normal.

[0042] The scene analysis output unit 60 may output an alert depending on the result of scene analysis. For example, in the case of (2), where the teacher is not moving and the students are wandering around, it is desirable to output an alert to the staff room because the class is in disarray.

[0043] Furthermore, the scene analysis output unit 60 may output a scene analysis result based on the relationship between different attributes and the behavior analysis result of multiple pieces of attribute-attached skeleton point information. For example, if the attribute information acquisition unit 30 acquires people with different attributes, such as a teacher and students, the scene analysis output unit 60 can analyze and output that a state in which the teacher is standing and the students are sitting is a normal state indicating a regular class.

[0044] 2 is a block diagram showing the hardware of the image analysis device 100 according to the first embodiment. The image analysis device 100 is connected to a camera 200. The image analysis device 100, when connected to the camera 200, constitutes an image analysis system 300.

[0045] The camera 200 captures an image of a subject and generates an image (called a captured image). The camera 200 generates the captured images over time, thereby generating a continuous time-series image (i.e., a video).

[0046] The image analysis device 100 is configured with a CPU (Central Processing Unit) 110, a memory unit 120, a ROM (Read Only Memory) 130, a RAM (Random Access Memory) 140, an input unit 150, a display unit 160, and a communication unit 170.

[0047] The CPU 110 reads and executes the image analysis program 121 stored in the storage unit 120, thereby realizing the time-series image acquisition unit 10, skeleton point information acquisition unit 20, attribute information acquisition unit 30, attribute-attached skeleton point information generation unit 40, behavior analysis unit 50, and scene analysis output unit 60 shown in FIG. 1 . The CPU 110 also controls the overall operation of the image analysis device 100. At this time, various data stored in the storage unit 120 is referenced. The storage unit 120 is configured, for example, by a large-capacity storage medium such as a semiconductor memory or a hard disk drive.

[0048] The CPU 110 transmits and receives various data to and from external devices connected to a communication network such as a local area network (LAN) or a wide area network (WAN) via the communication unit 170. The CPU 110, for example, acquires and processes captured images transmitted from the camera 200.

[0049] The communication unit 170 is configured by a communication control card such as a NIC (Network Interface Card).

[0050] The input unit 150 includes various operation keys such as a numeric keypad, a start key, etc. The input unit 150 accepts various input operations by the user and outputs operation signals to the CPU 110.

[0051] The display unit 160 is, for example, a liquid crystal display, and displays captured images, characters, figures, and the like.

[0052] Note that the image analysis device 100 shown in FIG. 2 is not limited to the first embodiment shown in FIG. 1 , and the image analysis device 101 according to the second embodiment to the image analysis device 102 according to the fourth embodiment can also be realized by the configuration of the image analysis device 100 shown in FIG. 2 .

[0053] [Specific Example of First Embodiment] For example, the skeleton point information acquisition unit 20 acquires skeleton points including the motion of a standing person S and skeleton points including the motion of a sitting person T. Furthermore, the attribute information acquisition unit 30 acquires an attribute of middle-aged as an attribute of the standing person S and an attribute of child as an attribute of the sitting person T. From this state, the scene analysis output unit 60 of the first embodiment outputs a scene analysis result indicating that a normal lesson is being conducted.

[0054] Here, the skeleton point information acquisition unit 20 acquires skeleton points using, for example, a method (such as Openpose) that extracts a person's pose in real time using deep learning.

[0055] 3 is an explanatory diagram showing a state in which a person is making a fist pump. As shown in FIG. 3, the person is making a fist pump with both hands raised.

[0056] For example, Openpose can detect and acquire point coordinates and joint points of a person using only images (including moving images) captured by camera 200 without using an acceleration sensor or the like.

[0057] Fig. 4 is an explanatory diagram showing parts that can be detected by Openpose, and Fig. 5 is a list of parts that can be detected by Fig. 4.

[0058] As shown in Figures 4 and 5, Openpose estimates the human skeleton to detect the parts corresponding to Figures 4 and 5. That is, Openpose detects a nose 500, a neck 501, a right shoulder 502, a right elbow 503, a right wrist 504, a left shoulder 505, a left elbow 506, a left wrist 507, a middle hip 508, a right hip 509, a right knee 510, a right ankle 511, a left hip 512, a left knee 513, a left ankle 514, a right eye 515, a left eye 516, a right ear 517, a left ear 518, a left thumb 519, a left little finger 520, a left heel 521, a right thumb 522, a right little finger 523, and a right heel 524.

[0059] As shown in FIG. 1, the skeleton point information acquisition unit 20 acquires, for example, a human rectangle, skeleton points, and a human ID for each person from one frame image.

[0060] FIG. 6 is a diagram showing a skeleton estimation table in which the skeleton point information acquisition unit 20 acquires a human rectangle, skeleton information, and a person ID and estimates a skeleton.

[0061] 6 includes columns for time, estimated ID, human rectangle, and skeleton information. For example, at time 1, the estimated ID is person S, the center coordinates (X, Y) of the human rectangle are (100, 100), and the rectangular frame has a height H of 120 and a width W of 20.

[0062] From this state, the skeleton point information acquisition unit 20 estimates, as skeleton information, the center coordinates (X, Y) of the nose to be (100, 110) and the center coordinates of the right heel to be (90, 150).

[0063] At time 2, the estimated ID is person T, the center coordinates (X, Y) of the person rectangle are (110, 160), and the rectangular frame has H of 180 and W of 30.

[0064] On the other hand, the attribute information acquisition unit 30 estimates the attributes and acquires the attribute information and the person ID. When the attribute information acquisition unit 30 estimates the attributes, a known attribute estimation process can be applied.

[0065] FIG. 7 is a diagram showing an attribute information acquisition table in which the attribute information acquisition unit 30 acquires attribute information and person IDs.

[0066] 7, a column for attribute information has been added instead of the skeleton information to the skeleton estimation table of FIG. 6. The attribute information includes whether or not the person is wearing glasses, gender, age, body orientation, and clothing color. For example, the attribute information acquisition unit 30 estimates that, for person S with an estimated ID at time 1, the person is not wearing glasses, is female, is middle-aged, is facing forward, and is wearing red clothing.

[0067] Furthermore, the attribute information acquisition unit 30 estimates that person T, whose estimated ID at time 2, is not wearing glasses, is female, is a child, is facing forward, and is wearing red clothes. Here, the attribute information acquisition unit 30 estimates that the middle-aged woman of person S is a teacher and the child of person T is a student from the age information, and assigns attributes to each of them.

[0068] The attribute information acquiring unit 30 may acquire attribute information from the time-series images at the same time that the skeleton point information acquiring unit 20 acquires skeleton point information from the time-series images.

[0069] The attributed skeleton point information generation unit 40 performs processing to integrate skeleton point information and attribute information of the same person from the skeleton estimation table of Fig. 6 and the attribute information acquisition table of Fig. 7. As an example, the skeleton estimation table of Fig. 6 and the attribute information acquisition table of Fig. 8 are integrated using an expanded version of the attribute information acquisition table of Fig. 7. The attribute information acquisition table of Fig. 8 is an expanded version of the attribute information acquisition table of Fig. 7 with respect to the time and estimated ID.

[0070] FIG. 9 is a diagram showing an information integration table that integrates the skeleton estimation table of FIG. 6 and the attribute information acquisition table of FIG.

[0071] The attributed skeleton point information generation unit 40, for example, integrates the person S whose estimated ID is in the skeleton estimation table of FIG. 6 with the person S whose estimated ID is in the attribute information acquisition table of FIG. 8, and also integrates the person T whose estimated ID is in the skeleton estimation table of FIG. 6 with the person T whose estimated ID is in the attribute information acquisition table of FIG. 8.

[0072] Specifically, first, the attributed skeleton point information generation unit 40 acquires human rectangle information contained in the skeleton point information shown in Fig. 6. Next, the attributed skeleton point information generation unit 40 acquires human rectangle information contained in the attribute information shown in Fig. 8. Then, the attributed skeleton point information generation unit 40 calculates Iu (IoU) from the human rectangle (skeleton points) and the human rectangle (attribute information).

[0073] Here, the attribute-attached skeleton point information generating unit 40 calculates Iu using the following equation (1).

[0074]

[0075] The attributed skeleton point information generating unit 40 searches for the pair with the highest Iu, and if Iu is equal to or greater than a threshold, generates an integrated person ID, but if Iu is less than the threshold, does not form a pair.

[0076] In this way, in Figure 9, the estimated ID of person S in the skeleton estimation table of Figure 6 is integrated with the estimated ID of person S in the attribute information acquisition table of Figure 8, and the estimated ID of person T in the skeleton estimation table of Figure 6 is integrated with the estimated ID of person T in the attribute information acquisition table of Figure 8.

[0077] In this specific example, the behavior analysis unit 50 can integrate behavior information into the information integration table of FIG. 9 by using a behavior recognition technology that uses skeleton detection. For example, as described in Japanese Patent Application No. 2002-095108, behavior can be recognized from the time-series skeleton point information by considering time-series skeleton point information as point cloud data and using the behavior labels indicated by this point cloud data and the point cloud data as training data for a learning model. Note that the behavior recognition technology is not particularly limited.

[0078] 10 is a behavior information table showing behavior information for each integrated person ID. In FIG. 10, a person with an integrated ID of person S is shown standing, and a person with an integrated ID of person T is shown sitting.

[0079] The behavior analysis unit 50 generates a behavior analysis table and an interaction table by integrating the behavior information table of FIG. 10 with the information integration table of FIG.

[0080] FIG. 11 shows a behavior analysis table obtained by integrating the behavior information table of FIG. 10 into the information integration table of FIG.

[0081] The behavior analysis unit 50 generates an interaction table showing interactions between a middle-aged person with the attribute of teacher and a child person with the attribute of student based on the behavior analysis table of FIG.

[0082] Fig. 12 shows an interaction table generated by the behavior analysis unit 50. In Fig. 12, the behavior of middle-aged people whose attribute is teacher is written horizontally, and the behavior of children whose attribute is student is written vertically, with the analysis of the scene written in a matrix.

[0083] The scene analysis output unit 60 can analyze the scene using the interaction table of FIG. 12, and indicates that a situation in which the teacher's behavior is facing backwards and the student's behavior is standing is judged to be abnormal, and that other behaviors are judged to be normal.

[0084] As a result, the scene analysis output unit 60 can output a scene analysis by acquiring the human rectangle, skeleton points, attribute information, behavior information, and integrated person ID (per person) from the attributed skeleton point information generation unit 40 in Figure 1.

[0085] 13 is a functional block diagram showing the functions of an image analyzing device 101 according to a second embodiment. The image analyzing device 101 according to the second embodiment differs from the image analyzing device 100 according to the first embodiment in that the scene analysis output unit 60 includes an interaction detection unit 61. The other configurations are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0086] The interaction detection unit 61 detects interactions. Therefore, when outputting the scene analysis result, the scene analysis output unit 60 analyzes interactions between a plurality of pieces of attribute-attached skeleton point information.

[0087] For example, in the case of a retail business, a situation is assumed in which a store clerk announces a limited-time sale. In this case, the behavior analysis unit 50 analyzes the behavior of multiple customers. Specifically, the behavior analysis unit 50 analyzes the behavior of (1) some customers entering the store, and (2) some customers passing by.

[0088] The interaction detection unit 61 detects a change in customer behavior in response to the store clerk's customer attraction behavior due to the limited-time sale. A change in behavior means a change in a person's behavior. Therefore, the interaction detection unit 61 detects a change in customer behavior due to the announcement of the limited-time sale.

[0089] The scene analysis output unit 60 uses information in the time-series images to determine whether the limited-time sale is effective based on changes in the captured images. For example, the scene analysis output unit 60 uses timestamp information included in the time-series images. The scene analysis output unit 60 may also use timestamp information included in a human rectangle. This allows the scene analysis output unit 60 to output the effect of the event, such as whether the number of customers increased in response to customer attraction behavior due to the limited-time sale.

[0090] The scene analysis results include timestamp information, such as the time, duration, and frame number. The timestamp indicates that the image was captured at the time stamped on the timestamp.

[0091] <Third Embodiment> In the third embodiment, the scene analysis output unit 60 of the second embodiment outputs a scene analysis result based on a behavior analysis result of second attribute-attached skeleton point information resulting from the action of first attribute-attached skeleton point information having first attribute information related to a first person on second attribute-attached skeleton point information having second attribute information related to a second person different from the first person, as an analysis of an interaction.

[0092] For example, in the case of a school, suppose a teacher is writing on the blackboard. In this case, the behavior analysis unit 50 analyzes each of the following states: (1) Student A is taking notes, (2) Student B is not taking notes, and (3) Student C is using his / her mobile phone.

[0093] The interaction detection unit 61 of the scene analysis output unit 60 detects a scene in which a student's hands are not moving while the teacher is writing on the blackboard. The scene analysis output unit 60 determines that Student B and Student C are in a boring state, and outputs the level of boringness of the lesson based on the ratio of the three students excluding Student A (1-1 (Student A) / 3 (Student A to C)).

[0094] For example, in the case of a dance lesson, a situation is assumed in which a teacher is demonstrating a dance to students. In this case, the behavior analysis unit 50 analyzes each state as follows: (1) Students A, B, and C dance at the appropriate timing, (2) Student D dances at a timing later than the teacher, and (3) Student E is sitting and taking a break.

[0095] The interaction detection unit 61 of the scene analysis output unit 60 detects discrepancies in the timing of each of the students A to D dancing in response to the teacher's example. The scene analysis output unit 60 extracts four students (students A to D) who are dancing out of the five students A to E, and outputs the ratio of students dancing at the appropriate timing (3 students (students A to C) / 4 students (students A to D)) as the students' understanding levels.

[0096] For example, in the case of the police, a situation is assumed in which the police are instructing a crowd to evacuate. In this case, the behavior analysis unit 50 analyzes the evacuation behavior of the crowd E in response to the evacuating instructions from the police officers. The attribute information acquisition unit 30 acquires, for example, a relationship between police officers and a crowd as a relationship between different attributes.

[0097] The interaction detection unit 61 of the scene analysis output unit 60 detects evacuation behavior of the crowd E in response to the guidance instructions from the police officers. The scene analysis output unit 60 outputs the ratio (1-crowd E / total number of crowds) obtained by excluding the attributes of the crowd E that has taken evacuating behavior from the total crowd, with the whole crowd being set to 1, as the panic level.

[0098] 14 is a functional block diagram showing functions of an image analysis device 102 according to the fourth embodiment. As shown in Fig. 14, the image analysis device 102 is configured to include a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, and a scene analysis output unit 62.

[0099] The image analyzing device 102 according to the fourth embodiment shown in FIG. 14 differs from the image analyzing device 100 according to the first embodiment shown in FIG. 1 in that it does not include the behavioral analysis unit 50.

[0100] The scene analysis output unit 62 outputs a result of analyzing a scene in time-series images based on a plurality of pieces of attributed skeleton point information. When analyzing a scene, the scene analysis output unit 62 performs processing collectively using behavior analysis and AI (artificial intelligence) based on a plurality of pieces of attributed skeleton point information. More specifically, the scene analysis result can be output by inputting the time-series attributed skeleton point information into a learning model that has undergone machine learning using the time-series attributed skeleton point information (point cloud data) and the scene analysis result (scene label) as training data.

[0101] As a result, the scene analysis output unit 62 can analyze the scene based on a learning model created by machine learning by acquiring the human rectangle, skeleton points, attribute information, behavior information, and integrated human ID (per person) from the attributed skeleton point information generation unit 40 in Figure 14.

[0102] <Actions and Effects> As described above, the image analysis device 100 according to the first embodiment is configured to include a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, a behavior analysis unit 50, and a scene analysis output unit 60.

[0103] In the image analyzing device 100 according to the first embodiment, the behavior analysis unit 50 performs behavior analysis on each of a plurality of pieces of attribute-attached skeleton point information and performs a composite analysis of the plurality of behavior analysis results. The scene analysis output unit 60 outputs the analysis results of scenes in the time-series images.

[0104] As a result, the image analyzing device 100 according to the first embodiment can utilize the interactions between groups by performing a composite analysis of multiple behavioral analysis results using the behavior analysis unit 50. Therefore, the image analyzing device 100 can perform highly accurate scene recognition by utilizing the interactions between groups.

[0105] Furthermore, the image analysis device 100 according to the first embodiment may output a scene analysis result by the scene analysis output unit 60 based on the relationship between different attributes and the behavior analysis result of multiple pieces of attribute-attached skeleton point information.

[0106] The image analysis device 100 according to the first embodiment can acquire the relationships between different attributes using the attribute information acquisition unit 30, and therefore the scene analysis output unit 60 can utilize the interactions between groups in the behavior analysis results of multiple attribute-attached skeleton point information.

[0107] For example, when the attribute information acquisition unit 30 acquires the relationship between different attributes such as a teacher and a student, the scene analysis output unit 60 can analyze and output that a situation in which the teacher is standing and the students are sitting is a normal situation indicating that a regular class is taking place.

[0108] In this way, the image analysis device 100 according to the first embodiment can perform more complex scene recognition.

[0109] Furthermore, the image analyzing device 100 according to the second embodiment includes an interaction detection unit 61 in the scene analysis output unit 60. The interaction detection unit 61 detects interactions. This allows the scene analysis output unit 60 to analyze interactions between multiple pieces of attribute-attached skeleton point information when outputting the scene analysis results.

[0110] For example, the interaction detection unit 61 detects a change in customer behavior in response to a store clerk's customer attraction behavior due to a limited-time sale. This allows the scene analysis output unit 60 to use information in the time-series images to determine from the change in behavior whether the customer attraction behavior due to the limited-time sale is effective.

[0111] Furthermore, the image analyzing device 100 according to the third embodiment outputs a scene analysis result based on the behavioral analysis result of second attribute-attached skeleton point information obtained as a result of the scene analysis performed by the scene analysis output unit 60 according to the second embodiment, in which first attribute-attached skeleton point information having first attribute information relating to a first person acts on second attribute-attached skeleton point information having second attribute information relating to a second person different from the first person.

[0112] For example, in the case of a school, the interaction detection unit 61 of the scene analysis output unit 60 analyzes the scene of a student who is not moving his / her hands while the teacher is writing on the blackboard. The scene analysis output unit 60 determines that Student B and Student C are in a boring state, and outputs the degree of boringness of the lesson based on the ratio of the three students excluding Student A (1-1 (Student A) / 3 (Student A to C)).

[0113] In addition, for example, in the case of a dance lesson, the interaction detection unit 61 of the scene analysis output unit 60 analyzes the difference in the timing of each of the students A to D dancing in response to the teacher's example. The scene analysis output unit 60 can output the ratio of students A to D who dance at the appropriate timing (3 students (students A to C) / 4 students (students A to D)) as the students' understanding levels.

[0114] Also, in the case of the police, for example, the interaction detection unit 61 of the scene analysis output unit 60 detects evacuation behavior of the crowd E in response to guidance instructions from the police officers. The scene analysis output unit 60 can output, as the panic level, the ratio (1-crowd E / total number of crowds) obtained by excluding the attributes of the crowd E that has taken evacuating behavior from the total crowd, with the whole crowd being set to 1.

[0115] In addition, the image analysis device 100 according to the fourth embodiment is configured such that the image analysis device 102 includes a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, and a scene analysis output unit 62.

[0116] The attributed skeleton point information generation unit 40 integrates skeleton point information and attribute information of the same person from the skeleton estimation table and the attribute information acquisition table. In the fourth embodiment, by using a behavior recognition technology (behavior analysis and AI) using skeleton detection instead of the behavior analysis unit 50, the scene analysis output unit 62 outputs the analysis result of the scene in the time-series image based on multiple pieces of attributed skeleton point information.

[0117] In this way, the image analysis device 100 according to the fourth embodiment can perform scene analysis in the scene analysis output unit 62 .

[0118] (Modifications) The present invention is not limited to the above-described embodiment, and modifications can be made without departing from the spirit of the present invention, for example, the following (a) to (f) are possible.

[0119] (a) Attribute information between different attributes may be based on the roles of people. For example, different attributes may include teacher attributes and student attributes. (b) The attribute-attached skeleton point information generation unit 40 may form groups of attribute-attached skeleton point information to which the same attribute is assigned. This allows the image analysis devices 100 to 102 according to the first to fourth embodiments to utilize interactions between groups and perform more complex scene recognition. (c) The scene analysis output unit 60, 62 may output an alert based on the scene analysis results. This allows the image analysis devices 100 to 102 according to the first to fourth embodiments to be applied to, for example, surveillance cameras, people flow systems, and DX (Digital Transformation) in factories. (d) The behavior analysis unit 50 may store time-series images generated when analyzing multiple behavior analysis results in the storage unit 120. Furthermore, the behavior analysis unit 50 may store the time-series images in a buffer until the composite analysis process is completed, and then delete the time-series images when the process is completed. This allows the storage unit 120 to appropriately secure the necessary storage capacity. (e) The attribute information acquisition unit 30 may acquire attribute information from the time-series images simultaneously with the skeleton point information acquisition unit acquiring skeleton point information from the time-series images. This improves real-time performance. Furthermore, the attribute information acquisition unit 30 is not limited to acquiring attribute information from the time-series images. For example, the user may input attribute information for each piece of skeleton information, or the attribute information may be acquired from another database. (f) The image analysis devices 100 to 102 according to the first to fourth embodiments may, for example, link audio to behavior analysis and perform a composite analysis of the behavior analysis results together with the audio. This improves the accuracy of scene recognition.

[0120] REFERENCE SIGNS LIST 10 Time-series image acquisition unit 20 Skeleton point information acquisition unit 30 Attribute information acquisition unit 40 Attribute-attached skeleton point information generation unit 50 Action analysis unit 60, 62 Scene analysis output unit 61 Interaction detection unit 100 Image analysis device 110 CPU 120 Storage unit 121 Image analysis program 130 ROM 140 RAM 150 Input unit 160 Display unit 170 Communication unit 200 Camera 300 Image analysis system

Claims

1. An image analyzing device comprising: a time-series image acquisition unit that acquires time-series images of a plurality of people; a skeleton point information acquisition unit that acquires a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generation unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; a behavior analysis unit that performs behavior analysis on each of the plurality of pieces of attribute-attached skeleton point information and performs a composite analysis of a plurality of behavior analysis results; and a scene analysis output unit that outputs analysis results of scenes in the time-series images.

2. The image analysis device according to claim 1, wherein the scene analysis output unit outputs the analysis result of the scene based on the relationship between different attributes and the behavior analysis result of the multiple attribute-attached skeleton point information.

3. The image analysis device according to claim 1, wherein the scene analysis output unit analyzes interactions between the plurality of attributed skeleton point information when outputting the scene analysis results.

4. The image analyzing device of claim 3, wherein the analysis of the interaction comprises outputting an analysis result of the scene based on a behavioral analysis result of the second attribute-attached skeleton point information resulting from the action of first attribute-attached skeleton point information having first attribute information relating to a first person on second attribute-attached skeleton point information having second attribute information relating to a second person different from the first person.

5. The image analysis device according to claim 2, wherein the attribute information between the different attributes is based on the role of the person.

6. The image analysis device according to claim 1, wherein the attributed skeleton point information generation unit forms a group of attributed skeleton point information to which the same attribute is assigned with respect to the plurality of attributed skeleton point information.

7. The image analysis device according to claim 1, wherein the scene analysis output unit outputs an alert depending on the result of the scene analysis.

8. The image analysis device according to claim 1, wherein the behavior analysis unit stores the time-series images obtained when analyzing the multiple behavior analysis results.

9. The image analysis device according to claim 8, wherein the behavior analysis unit stores the time-series images in a buffer until the processing of the comprehensive analysis is completed, and deletes the time-series images when the processing is completed.

10. The image analysis device according to claim 1, wherein the attribute information acquisition unit acquires the attribute information from the time-series image simultaneously with the skeleton point information acquisition unit acquiring the skeleton point information from the time-series image.

11. An image analysis device comprising: a time-series image acquisition unit for acquiring time-series images of a plurality of people; a skeleton point information acquisition unit for acquiring a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images; an attribute information acquisition unit for acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generation unit for generating a plurality of pieces of attribute-attached skeleton point information by associating the skeleton point information with the attribute information; and a scene analysis output unit for outputting an analysis result of a scene in the time-series images based on the plurality of attribute-attached skeleton point information.

12. An image analysis method comprising the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; associating the skeletal point information with the attribute information to generate a plurality of pieces of attributed skeletal point information; performing behavioral analysis on each of the plurality of pieces of attributed skeletal point information and performing a composite analysis of a plurality of behavioral analysis results; and outputting analysis results of scenes in the time-series images.

13. An image analysis method comprising the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; and outputting an analysis result of a scene in the time-series images based on the plurality of attributed skeleton point information.

14. An image analysis program for causing a computer to execute the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; associating the skeletal point information with the attribute information to generate a plurality of pieces of attributed skeletal point information; performing behavioral analysis of each of the plurality of pieces of attributed skeletal point information and performing a composite analysis of a plurality of behavioral analysis results; and outputting analysis results of scenes in the time-series images.

15. An image analysis program for causing a computer to execute the steps of: acquiring time-series images in which a plurality of people are captured; acquiring a plurality of pieces of skeletal point information for each of the plurality of people from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeletal point information; associating the skeletal point information with the attribute information to generate a plurality of pieces of skeletal point information with attributes; and outputting analysis results of scenes in the time-series images based on the plurality of pieces of skeletal point information with attributes.

Images