IMAGE ANALYSIS DEVICE, IMAGE ANALYSIS METHOD, AND IMAGE ANALYSIS PROGRAM

Abstract

The image analysis device (100) includes a time-series image acquisition unit (10) that acquires time-series images of a plurality of people, a skeleton point information acquisition unit (20) that acquires a plurality of pieces of skeleton point information for each of the plurality of people from each frame image of the time-series images, an attribute information acquisition unit (30) that acquires a plurality of pieces of attribute information corresponding to each of the plurality of pieces of skeleton point information, an attribute-attached skeleton point information generation unit (40) that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information, a behavior analysis unit (50) that performs behavior analysis of each of the plurality of pieces of attribute-attached skeleton point information and performs a composite analysis of the plurality of behavior analysis results, and a scene analysis output unit (60) that outputs analysis results of scenes in the time-series images.

Description

[Technical field]

[0001] The present invention relates to an image analysis device, an image analysis method, and an image analysis program. [Background technology]

[0002] Conventionally, a device has been proposed that creates a histogram of the occurrence frequency of behavioral features for each person's attribute and outputs a scene analysis result based on the behavioral features.

[0003] For example, Patent Document 1 discloses an abnormal behavior detection system that states, "The system extracts skeletal position information representing the skeletal position of a person included in each frame of a time-series image, which is a time series of captured images and consists of a certain number of frames or more. The system determines whether the people included in each frame of the time-series image are the same person based on the extracted skeletal position information. The system generates behavior features for the same person included in a certain number of frames or more of the time-series image based on the extracted skeletal position information. The system generates attribute histograms by histogramming the occurrence frequencies of the generated behavior features for each attribute of the person, and stores the attribute histograms in a storage unit that can be used to detect abnormal behavior of the person based on the person's behavior features" (see the abstract of Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-17939 A Summary of the Invention [Problem to be solved by the invention]

[0005] The abnormal behavior detection system disclosed in Patent Document 1 performs complex scene recognition and judgment based only on the attributes and behavior recognition results of each person. However, there are cases where it is not possible to determine whether an individual is normal or abnormal based on the analysis results of each person alone.

[0006] Specifically, let us assume a situation where a teacher and students are in a classroom and the students are walking around the classroom. In this case, if the teacher is writing on the blackboard, the behavior of the students walking around the classroom is judged as class disruption. On the other hand, if the teacher is giving instructions to the students and the students are walking towards the blackboard, it is judged that the class is proceeding normally in that classroom.

[0007] In this way, the behavior of students walking around the classroom is recognized as normal or abnormal depending on the interaction with the teacher's behavior. For this reason, there are some situations where the behavior cannot be recognized correctly by only the attributes and behavior recognition of individual people.

[0008] Therefore, an object of the present invention is to provide an image analysis device, an image analysis method, and an image analysis program that are capable of performing highly accurate scene recognition based on attributed skeleton point information. [Means for solving the problem]

[0009] That is, the above-mentioned object of the present invention is achieved by the following configuration. (1) a time-series image acquisition unit for acquiring time-series images of a plurality of people; a skeleton point information acquisition unit that acquires a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generating unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; a behavior analysis unit that performs behavior analysis on each of the plurality of pieces of attributed skeleton point information and performs a composite analysis of the plurality of behavior analysis results; a scene analysis output unit that outputs a result of analyzing a scene in the time-series images; An image analysis device comprising:

[0010] (2) The scene analysis output unit outputting an analysis result of the scene based on the relationship between different attributes and the behavior analysis result of the plurality of attribute-attached skeleton point information; The image analysis device according to (1).

[0011] (3) The scene analysis output unit analyzing interactions of the plurality of attributed skeleton point information when outputting the scene analysis result; The image analysis device according to (1).

[0012] (4) The analysis of the interaction includes: and outputting an analysis result of the scene based on a behavior analysis result of the second attribute-attached skeleton point information, which is a result of the first attribute-attached skeleton point information having first attribute information related to a first person acting on the second attribute-attached skeleton point information having second attribute information related to a second person different from the first person. (3) An image analysis device according to the present invention.

[0013] (5) the attribute information between the different attributes is based on the role of the person; (2) An image analysis device according to the present invention.

[0014] (6) The attributed skeleton point information generating unit forming a group of attributed skeleton point information to which the same attribute is assigned with respect to the plurality of attributed skeleton point information; The image analysis device according to (1).

[0015] (7) The scene analysis output unit outputting an alert according to the result of the analysis of the scene; The image analysis device according to (1).

[0016] (8) The behavior analysis unit storing the time-series images obtained when the plurality of behavioral analysis results are analyzed; The image analysis device according to (1).

[0017] (9) The behavior analysis unit storing the time-series images in a buffer until the processing of the composite analysis is completed, and deleting the time-series images when the processing is completed; (8) An image analysis device according to (8).

[0018] (10) At the same time as the skeleton point information acquisition unit acquires the skeleton point information from the time-series image, the attribute information acquisition unit acquires the attribute information from the time-series image. The image analysis device according to (1).

[0019] (11) a time-series image acquisition unit for acquiring time-series images of a plurality of people; a skeleton point information acquisition unit that acquires a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generating unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; a scene analysis output unit that outputs a result of analyzing a scene in the time-series images based on the plurality of attributed skeleton point information; An image analysis device comprising:

[0020] (12) acquiring time-series images of a plurality of people; acquiring a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; performing behavior analysis on each of the plurality of pieces of attributed skeleton point information and performing a composite analysis of the plurality of behavior analysis results; outputting a result of the analysis of the scene in the time series of images; Perform image analysis methods.

[0021] (13) acquiring a time-series image of a plurality of people; acquiring a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; outputting a result of analyzing a scene in the time-series images based on the plurality of pieces of attributed skeleton point information; Perform image analysis methods.

[0022] (14) acquiring time-series images of a plurality of persons; obtaining a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; a step of generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; a step of performing behavior analysis on each of the plurality of pieces of attributed skeleton point information and performing a composite analysis of the plurality of behavior analysis results; outputting a result of the analysis of the scene in the time series images; An image analysis program that allows a computer to execute the above.

[0023] (15) acquiring time-series images of a plurality of persons; obtaining a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; a step of generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; a step of outputting a result of analysis of a scene in the time-series images based on the plurality of pieces of attributed skeleton point information; An image analysis program that allows a computer to execute the above. Effect of the Invention

[0024] According to the present invention, highly accurate scene recognition can be performed based on attribute-attached skeleton point information. [Brief description of the drawings]

[0025] [Figure 1] 1 is a functional block diagram showing functions of an image analysis device according to a first embodiment. FIG. [Diagram 2] 1 is a block diagram showing hardware of an image analysis device according to a first embodiment. [Diagram 3] FIG. 13 is an explanatory diagram showing a person making a fist pump. [Figure 4] FIG. 1 is an explanatory diagram showing areas that can be detected by Openpose. [Diagram 5] Figure 4 lists the sites that can be detected. [Figure 6] 13 is a diagram showing a skeleton estimation table in which a skeleton point information acquisition unit acquires a human rectangle, skeleton information, and a human ID and estimates a skeleton. FIG. [Figure 7] 13 is a diagram showing an attribute information acquisition table in which an attribute information acquisition unit acquires attribute information and a person ID. FIG. [Figure 8] 8 is a diagram showing a table in which the time and estimated ID are expanded from the attribute information acquisition table in FIG. 7. FIG. [Figure 9] 9 is a diagram showing an information integration table obtained by integrating the skeleton estimation table of FIG. 6 and the attribute information acquisition table of FIG. 8. FIG. [Figure 10] FIG. 13 is a diagram showing a behavior information table showing behavior information for each integrated person ID. [Figure 11] 13 is a diagram showing a behavior analysis table in which a behavior information table is integrated into an information integration table. FIG. [Figure 12] FIG. 13 shows an interaction table generated by a scene analysis output unit. [Figure 13] FIG. 11 is a functional block diagram showing functions of an image analyzing device according to a second embodiment. [Figure 14]FIG. 11 is a functional block diagram showing functions of an image analyzing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The following describes in detail the embodiments for carrying out the present invention. Note that the embodiments described below are merely examples for implementing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0027] First Embodiment [Overall configuration of image analysis device] 1 is a functional block diagram showing functions of an image analyzing device 100 according to the first embodiment. The image analyzing device 100 is generally a device that performs behavior analysis of attributed skeleton point information, and performs a composite analysis of the behavior analysis results to output a scene analysis result.

[0028] As shown in FIG. 1, the image analysis device 100 is configured to include a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, a behavior analysis unit 50, and a scene analysis output unit 60.

[0029] The time-series image acquisition unit 10 acquires images of multiple people over time from a camera 200 (see FIG. 2) described later. In other words, the time-series image acquisition unit 10 acquires time-series images of multiple people. Thus, the time-series image acquisition unit 10 can output time-series images in units of frames. Note that the time-series image acquisition unit 10 may acquire images of multiple people from a storage in which a shot video is recorded, and is not particularly limited.

[0030] The skeleton point information acquisition unit 20 acquires a plurality of skeleton point information of a plurality of people from each frame image of the time series image on a frame-by-frame basis. The skeleton point information acquisition unit 20 estimates the skeleton of each of the plurality of people on a frame-by-frame basis to acquire skeleton point information and a person ID (IDentification) (per person). The skeleton point information acquisition unit 20 also acquires a person rectangle. A person rectangle is an area (coordinates) where a person is present, refers to a rectangular image area, and refers to an area for object detection (person detection). The skeleton point information acquisition unit 20 acquires skeleton point information, a person rectangle, and a person ID, which are key points of each of the plurality of people, from the frame images using, for example, Openpose developed at Carnegie Mellon University.

[0031] The attribute information acquisition unit 30 estimates the attributes of the subject (person) from each frame image of the time-series image to acquire attribute information and a person ID. In other words, the attribute information acquisition unit 30 acquires a plurality of pieces of attribute information corresponding to each of a plurality of pieces of skeleton point information. Thus, the attribute information acquisition unit 30 acquires a person rectangle, attribute information, and a person ID (per person). The attribute information acquisition unit 30 acquires attribute information, a person rectangle, and a person ID for each of a plurality of people from a frame image, for example, using FORXAI developed by the applicant of the present application. In addition, the attribute information acquisition unit 30 may, for example, in the analysis of a school, assign an attribute indicating that the person is a teacher or attribute information indicating that the person is a student from age information among the attribute information.

[0032] Furthermore, the attribute information acquiring unit 30 may acquire attribute information from the time-series image at the same time that the skeleton point information acquiring unit 20 acquires skeleton point information from the time-series image.

[0033] The attributed skeleton point information generating unit 40 integrates skeleton point information and attribute information of the same person. In other words, the attributed skeleton point information generating unit 40 generates a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information. The attributed skeleton point information generating unit 40 can output a human rectangle, skeleton points, attribute information, and an integrated human ID (per person).

[0034] Furthermore, the attribute-attached skeleton point information generating unit 40 may form a group of attribute-attached skeleton point information to which the same attribute is assigned with respect to a plurality of attribute-attached skeleton point information. For example, the attribute-attached skeleton point information generating unit 40 can form a group of attribute-attached skeleton point information to which the attribute of a male is assigned, or a group of attribute-attached skeleton point information to which the attribute of a middle-aged person is assigned.

[0035] The behavior analysis unit 50 analyzes the behavior of people in a scene from the skeleton point information and attribute information of the same frame image. In other words, the behavior analysis unit 50 performs a composite analysis of a plurality of behavior analysis results obtained by analyzing the behavior of each of a plurality of pieces of attribute-attached skeleton point information.

[0036] For example, when a school is the subject of analysis, a situation is assumed in which a teacher is teaching students in a classroom. In this case, the behavior analysis unit 50 analyzes a combination of the teacher's behavior and the student's behavior. The behavior analysis unit 50 acquires a human rectangle, skeleton points, attribute information, behavior information, and an integrated human ID (per person). Then, the behavior analysis unit 50 analyzes, for example, the following four states:

[0037] The behavior analysis unit 50 analyzes whether the teacher and students in the frame image fall into any of the following four states: (1) the teacher is not moving and the students are sleeping, (2) the teacher is not moving and the students are wandering around, (3) the teacher is not moving and the students are walking toward the blackboard, and (4) the teacher is wandering around and the students are not moving. Note that the subjects analyzed by the behavior analysis unit 50 are not limited to teachers and students in a school, but may be store clerks and customers, police officers and a crowd, etc.

[0038] Here, the behavior analysis unit 50 performs a composite analysis of the group belonging to the teacher and the group belonging to the students. The behavior analysis unit 50 of this embodiment analyzes the interactions between the groups by fitting them into one-to-one, one-to-many, or many-to-many. In this way, for example, the behavior analysis unit 50 performing an analysis using the interactions between the groups is referred to as a composite analysis.

[0039] The behavior analysis unit 50 may also store time-series images obtained when analyzing a plurality of behavior analysis results in the storage unit 120. The behavior analysis unit 50 may also store the time-series images in a buffer until the processing for performing a composite analysis is completed, and delete the time-series images when the processing is completed.

[0040] The scene analysis output unit 60 analyzes a scene from the attribute information and the behavior information. In other words, the scene analysis output unit 60 outputs the analysis result of the scene in the time-series images.

[0041] In the case of the school mentioned above, (1) when the teacher is not moving and the students are sleeping means that the lesson is boring and shows that the teacher is not good at teaching. (2) when the teacher is not moving and the students are wandering around means that the class is falling apart. (3) when the teacher is not moving and the students are walking towards the blackboard means that the lesson is normal with the students going to write on the board. (4) when the teacher is wandering around and the students are not moving also means that the lesson is normal.

[0042] The scene analysis output unit 60 may output an alert depending on the result of scene analysis. For example, in the state of (2) where the teacher is not moving and the students are wandering around, it is desirable to output an alert to the staff room since the class is disorganized.

[0043] The scene analysis output unit 60 may also output a scene analysis result based on the relationship between different attributes and the behavior analysis result of a plurality of attributed skeleton point information. For example, when the attribute information acquisition unit 30 acquires people with different attributes, such as a teacher and a student, the scene analysis output unit 60 can analyze and output that a state in which the teacher is standing and the student is sitting is a normal state indicating a normal lesson.

[0044] [Hardware configuration of image analysis device] 2 is a block diagram showing hardware of image analysis device 100 according to the first embodiment. Image analysis device 100 is connected to camera 200. Image analysis device 100 configures image analysis system 300 by being connected to camera 200.

[0045] The camera 200 captures an image of a subject and generates an image (called a captured image). The camera 200 generates the captured images over time, thereby generating a continuous time-series image (that is, a video).

[0046] The image analyzing device 100 includes a CPU (Central Processing Unit) 110, a storage unit 120, a ROM (Read Only Memory) 130, a RAM (Random Access Memory) 140, an input unit 150, a display unit 160, and a communication unit 170.

[0047] The CPU 110 reads out and executes an image analysis program 121 stored in the storage unit 120, thereby realizing the time-series image acquisition unit 10, skeleton point information acquisition unit 20, attribute information acquisition unit 30, attributed skeleton point information generation unit 40, behavior analysis unit 50, and scene analysis output unit 60 shown in FIG. 1. The CPU 110 also controls the overall operation of the image analysis device 100. At this time, various data stored in the storage unit 120 are referenced. The storage unit 120 is composed of a large-capacity storage medium such as a semiconductor memory or a hard disk drive.

[0048] The CPU 110 transmits and receives various data to and from external devices connected to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network) via the communication unit 170. The CPU 110, for example, obtains a captured image transmitted from the camera 200 and processes the image.

[0049] The communication unit 170 is configured by a communication control card such as a NIC (Network Interface Card).

[0050] The input unit 150 includes various operation keys such as a numeric keypad, a start key, etc. The input unit 150 accepts various input operations by the user and outputs operation signals to the CPU 110.

[0051] The display unit 160 is, for example, a liquid crystal display, and displays captured images, characters, figures, and the like.

[0052] Note that the image analyzing device 100 shown in FIG. 2 is not limited to the first embodiment shown in FIG. 1, and the image analyzing device 101 according to the second embodiment to the image analyzing device 102 according to the fourth embodiment can also be realized by the configuration of the image analyzing device 100 shown in FIG. 2.

[0053] [Specific example of the first embodiment] For example, the skeleton point information acquisition unit 20 acquires skeleton points including the motion of a standing person S and skeleton points including the motion of a sitting person T. Furthermore, the attribute information acquisition unit 30 acquires an attribute of middle-aged as an attribute of the standing person S and an attribute of child as an attribute of the sitting person T. From this state, the scene analysis output unit 60 of the first embodiment outputs a scene analysis result indicating that a normal lesson is being conducted.

[0054] Here, the skeleton point information acquisition unit 20 acquires skeleton points by, for example, a method (such as Openpose) for extracting a person's pose in real time using deep learning.

[0055] 3 is an explanatory diagram showing a state in which a person is making a fist pump. As shown in FIG 3, the person is making a fist pump with both hands raised.

[0056] For example, in Openpose, it is possible to detect and acquire point coordinates and joint points of a human being only from images (including moving images) captured by the camera 200 without using an acceleration sensor or the like.

[0057] Fig. 4 is an explanatory diagram showing the parts that can be detected by Openpose. Fig. 5 is a list of the parts that can be detected by Fig. 4.

[0058] As shown in Figures 4 and 5, Openpose detects the parts corresponding to Figures 4 and 5 by estimating the human skeleton. That is, Openpose detects a nose 500, a neck 501, a right shoulder 502, a right elbow 503, a right wrist 504, a left shoulder 505, a left elbow 506, a left wrist 507, a middle hip 508, a right hip 509, a right knee 510, a right ankle 511, a left hip 512, a left knee 513, a left ankle 514, a right eye 515, a left eye 516, a right ear 517, a left ear 518, a left thumb 519, a left little finger 520, a left heel 521, a right thumb 522, a right little finger 523, and a right heel 524.

[0059] As shown in FIG. 1, the skeleton point information acquisition unit 20 acquires, for example, a human rectangle, skeleton points, and a human ID for each person from one frame image.

[0060] FIG. 6 is a diagram showing a skeleton estimation table in which the skeleton point information acquisition unit 20 acquires a human rectangle, skeleton information, and a person ID and estimates a skeleton.

[0061] 6 includes columns for time, estimated ID, human rectangle, and skeleton information. For example, at time 1, the estimated ID is person S, the center coordinates (X, Y) of the human rectangle are (100, 100), and the height H of the rectangular frame is 120 and the width W is 20.

[0062] From this state, the skeleton point information acquisition unit 20 estimates, as skeleton information, the center coordinates (X, Y) of the nose to be (100, 110), and the center coordinates of the right heel to be (90, 150).

[0063] At time 2, the estimated ID of person T has a human rectangle with center coordinates (X, Y) of (110, 160), and the rectangular frame has H of 180 and W of 30.

[0064] On the other hand, the attribute information acquiring unit 30 estimates the attribute and acquires the attribute information and the person ID. When the attribute information acquiring unit 30 estimates the attribute, a known attribute estimation process can be applied.

[0065] FIG. 7 is a diagram showing an attribute information acquisition table in which the attribute information acquisition unit 30 acquires attribute information and person IDs.

[0066] In Fig. 7, a column of attribute information is added instead of the skeleton information to the skeleton estimation table of Fig. 6. The attribute information estimates whether or not the person is wearing glasses, the gender, the age, the body orientation, and the color of the clothes. For example, the attribute information acquisition unit 30 estimates that the person S, whose estimated ID is person S at time 1, is not wearing glasses, is female, is middle-aged, is facing forward, and is wearing red clothes.

[0067] The attribute information acquisition unit 30 also estimates that the person T, whose estimated ID at time 2 is not wearing glasses, is female, is a child, is facing forward, and is wearing red clothing. Here, the attribute information acquisition unit 30 estimates the middle-aged woman of person S to be a teacher and the child of person T to be a student from the age information, and assigns attributes to each.

[0068] The attribute information acquiring unit 30 may acquire attribute information from the time-series image at the same time that the skeleton point information acquiring unit 20 acquires skeleton point information from the time-series image.

[0069] The attributed skeleton point information generating unit 40 performs a process of integrating skeleton point information and attribute information of the same person from the skeleton estimation table of Fig. 6 and the attribute information acquisition table of Fig. 7. As an example, the skeleton estimation table of Fig. 6 and the attribute information acquisition table of Fig. 8 are integrated using Fig. 8, which is an expanded version of the attribute information acquisition table of Fig. 7. The attribute information acquisition table of Fig. 8 is an expanded version of the attribute information acquisition table of Fig. 7 with respect to the time and estimated ID.

[0070] FIG. 9 is a diagram showing an information integration table obtained by integrating the skeleton estimation table of FIG. 6 and the attribute information acquisition table of FIG.

[0071] The attributed skeleton point information generation unit 40, for example, integrates the estimated ID of person S in the skeleton estimation table of FIG. 6 with the estimated ID of person S in the attribute information acquisition table of FIG. 8, and also integrates the estimated ID of person T in the skeleton estimation table of FIG. 6 with the estimated ID of person T in the attribute information acquisition table of FIG. 8.

[0072] Specifically, first, the attributed skeleton point information generation unit 40 acquires human rectangle information contained in the skeleton point information shown in Fig. 6. Next, the attributed skeleton point information generation unit 40 acquires human rectangle information contained in the attribute information shown in Fig. 8. Then, the attributed skeleton point information generation unit 40 calculates Iu (IoU) from the human rectangle (skeleton points) and the human rectangle (attribute information).

[0073] Here, the attribute-attached skeleton point information generating unit 40 calculates Iu using the following formula (1).

[0074]

number

[0075] The attributed skeleton point information generating unit 40 searches for the pair with the highest Iu, and if the Iu is equal to or greater than a threshold, generates an integrated person ID, but if it is less than the threshold, does not form a pair.

[0076] In this way, in Figure 9, the estimated ID in the skeleton estimation table of Figure 6 is person S and the estimated ID in the attribute information acquisition table of Figure 8 is person S are integrated, and the estimated ID in the skeleton estimation table of Figure 6 is person T and the estimated ID in the attribute information acquisition table of Figure 8 is person T are integrated.

[0077] In this specific example, the behavior analysis unit 50 can integrate behavior information into the information integration table of Fig. 9 by using a behavior recognition technology using skeleton detection. For example, as described in Japanese Patent Application No. 2002-095108, behavior can be recognized from the time-series skeleton point information by considering the time-series skeleton point information as point cloud data and having a learning model machine-learn using the behavior label indicated by the point cloud data and the point cloud data as teacher data. The behavior recognition technology is not particularly limited.

[0078] Fig. 10 is a behavior information table showing behavior information for each integrated person ID. In Fig. 10, a person with an integrated ID of person S is shown to be in a standing state, and a person with an integrated ID of person T is shown to be in a sitting state.

[0079] The behavior analysis unit 50 generates a behavior analysis table by integrating the behavior information table of FIG. 10 with the information integration table of FIG. 9, and generates an interaction table.

[0080] FIG. 11 shows a behavior analysis table in which the behavior information table of FIG. 10 is integrated into the information integration table of FIG.

[0081] The behavior analysis unit 50 generates an interaction table showing interactions between middle-aged people with the attribute of teacher and children with the attribute of student, based on the behavior analysis table of FIG.

[0082] Fig. 12 shows an interaction table generated by the behavior analysis unit 50. In Fig. 12, the behaviors of middle-aged people whose attribute is teacher are listed horizontally, and the behaviors of children whose attribute is student are listed vertically, with the analysis of the scene being listed in a matrix.

[0083] The scene analysis output unit 60 can analyze the scene using the interaction table of FIG. 12, and indicates that a situation in which the teacher's behavior is facing away from the students and the students' behavior is standing is judged to be abnormal, and other behaviors are judged to be normal.

[0084] As a result, the scene analysis output unit 60 can output a scene analysis by acquiring a human rectangle, skeleton points, attribute information, behavior information, and an integrated human ID (per person) from the attributed skeleton point information generation unit 40 of Figure 1.

[0085] <Second embodiment> 13 is a functional block diagram showing functions of an image analyzing device 101 according to the second embodiment. The image analyzing device 101 of the second embodiment differs from the image analyzing device 100 of the first embodiment in that the scene analysis output unit 60 includes an interaction detection unit 61. The other configurations are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0086] The interaction detection unit 61 detects interactions. Therefore, when outputting the scene analysis result, the scene analysis output unit 60 analyzes interactions between a plurality of pieces of attribute-added skeleton point information.

[0087] For example, in the case of a retail business, a situation is assumed in which a salesperson announces a limited time sale. In this case, the behavior analysis unit 50 analyzes the behavior of multiple customers. Specifically, the behavior analysis unit 50 analyzes the behavior of (1) some customers entering the store, and (2) some customers passing by.

[0088] The interaction detection unit 61 detects a change in customer behavior in response to the store clerk's customer attraction behavior due to the limited-time sale. A change in behavior means a change in a person's behavior. Therefore, the interaction detection unit 61 detects a change in customer behavior due to the announcement of the limited-time sale.

[0089] The scene analysis output unit 60 uses information in the time-series images to determine whether or not a limited-time sale is effective from changes in the captured images. For example, the scene analysis output unit 60 uses time stamp information included in the time-series images. The scene analysis output unit 60 may also use time stamp information included in a human rectangle. This allows the scene analysis output unit 60 to output the effect of the event, such as whether or not the number of customers has increased in response to customer attraction actions due to the limited-time sale.

[0090] The scene analysis results include time stamp information, such as the time, duration, and frame number. The time stamp indicates that the captured image was taken at the time stamped on the time stamp.

[0091] <Third embodiment> In the third embodiment, the scene analysis output unit 60 of the second embodiment outputs a scene analysis result based on a behavior analysis result of second attribute-attached skeleton point information resulting from the action of first attribute-attached skeleton point information having first attribute information relating to a first person on second attribute-attached skeleton point information having second attribute information relating to a second person different from the first person, as an analysis of an interaction.

[0092] For example, in the case of a school, assuming a situation where a teacher is writing on the blackboard, the behavior analysis unit 50 analyzes each of the following situations: (1) Student A is taking notes, (2) Student B is not taking notes, and (3) Student C is using his / her mobile phone.

[0093] The interaction detection unit 61 of the scene analysis output unit 60 detects a scene where the teacher is writing on the blackboard but the students are not moving their hands. The scene analysis output unit 60 outputs the boring lesson level based on the ratio of students B and C excluding student A (1-1 (student A) / 3 (students A-C)) out of the three students, which corresponds to the boring state.

[0094] For example, in the case of a dance lesson, a state in which a teacher shows students how to dance is assumed. In this case, the behavior analysis unit 50 analyzes each state as follows: (1) Students A, B, and C dance at appropriate times, (2) Student D dances later than the teacher, and (3) Student E is sitting and taking a break.

[0095] The interaction detection unit 61 of the scene analysis output unit 60 detects the difference in the timing of each of the students A to D dancing in response to the teacher's example. The scene analysis output unit 60 extracts four students (students A to D) who are dancing out of the five students A to E, and outputs the ratio of students who dance at the appropriate timing (3 students (students A to C) / 4 students (students A to D)) as the students' understanding level.

[0096] Also, for example, in the case of the police, a situation is assumed in which the police instruct the crowd to evacuate. In this case, the behavior analysis unit 50 analyzes the evacuation behavior of the crowd E in response to the evacuating instructions from the police officers. The attribute information acquisition unit 30 acquires, for example, a relationship between the police officers and the crowd as a relationship between different attributes.

[0097] The interaction detection unit 61 of the scene analysis output unit 60 detects the evacuation behavior of the crowd E in response to the guiding instructions from the police officers. The scene analysis output unit 60 outputs the ratio (1-crowd E / total number of crowds) excluding the attributes of the crowd E that has taken evacuating behavior out of the total crowd, with the whole crowd being set at 1, as the panic degree.

[0098] <Fourth embodiment> Fig. 14 is a functional block diagram showing functions of an image analyzing device 102 according to the fourth embodiment. As shown in Fig. 14, the image analyzing device 102 includes a time-series image acquiring unit 10, a skeleton point information acquiring unit 20, an attribute information acquiring unit 30, an attribute-attached skeleton point information generating unit 40, and a scene analysis output unit 62.

[0099] An image analyzing device 102 according to the fourth embodiment shown in FIG. 14 differs from the image analyzing device 100 according to the first embodiment shown in FIG.

[0100] The scene analysis output unit 62 outputs a result of analyzing a scene in a time-series image based on a plurality of pieces of attributed skeleton point information. When analyzing a scene, the scene analysis output unit 62 performs processing by using behavior analysis and AI (Artificial Intelligence) based on a plurality of pieces of attributed skeleton point information. More specifically, the scene analysis result can be output by inputting the time-series attributed skeleton point information to a learning model that has been machine-learned using the time-series attributed skeleton point information (point cloud data) and the scene analysis result (scene label) as teacher data.

[0101] As a result, the scene analysis output unit 62 can analyze the scene based on a learning model using machine learning by acquiring the human rectangle, skeleton points, attribute information, behavior information, and integrated human ID (per person) from the attributed skeleton point information generation unit 40 in Figure 14.

[0102] <Actions and Effects> As described above, the image analysis device 100 according to the first embodiment is configured to include a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attributed skeleton point information generation unit 40, a behavior analysis unit 50, and a scene analysis output unit 60.

[0103] In the image analyzing device 100 according to the first embodiment, the behavior analysis unit 50 performs behavior analysis on each of a plurality of pieces of attribute-attached skeleton point information and performs a composite analysis of the plurality of behavior analysis results. The scene analysis output unit 60 outputs the analysis result of the scene in the time-series image.

[0104] As a result, the image analyzing device 100 according to the first embodiment can utilize interactions between groups by the behavior analysis unit 50 performing a composite analysis of a plurality of behavior analysis results. Therefore, the image analyzing device 100 can perform highly accurate scene recognition by utilizing interactions between groups.

[0105] Furthermore, the image analyzing device 100 according to the first embodiment may output, via the scene analysis output section 60, a scene analysis result based on the relationship between different attributes and the behavior analysis result of a plurality of pieces of attribute-attached skeleton point information.

[0106] The image analysis device 100 according to the first embodiment can acquire relationships between different attributes using the attribute information acquisition unit 30, and therefore the scene analysis output unit 60 can utilize interactions between groups in the results of behavior analysis of multiple attribute-attached skeleton point information.

[0107] For example, when the attribute information acquisition unit 30 acquires a relationship between different attributes such as a teacher and a student, the scene analysis output unit 60 can analyze and output that a state in which the teacher is standing and the students are sitting is a normal state indicating that a regular lesson is taking place.

[0108] In this way, the image analyzing device 100 according to the first embodiment can perform more complicated scene recognition.

[0109] Moreover, the image analyzing device 100 according to the second embodiment includes an interaction detection unit 61 in the scene analysis output unit 60. The interaction detection unit 61 detects interactions. This allows the scene analysis output unit 60 to analyze interactions between a plurality of pieces of attribute-attached skeleton point information when outputting the scene analysis result.

[0110] For example, the interaction detection unit 61 detects a change in customer behavior in response to a store clerk's customer-attracting behavior due to a limited-time sale. This allows the scene analysis output unit 60 to use information in the time-series images to determine from the change in behavior whether the customer-attracting behavior due to the limited-time sale is effective.

[0111] Furthermore, the image analyzing device 100 according to the third embodiment outputs a scene analysis result based on a behavioral analysis result of second attribute-attached skeleton point information resulting from the scene analysis output unit 60 of the second embodiment acting on second attribute-attached skeleton point information having second attribute information relating to a second person different from the first person, as an analysis of an interaction.

[0112] For example, in the case of a school, the interaction detection unit 61 of the scene analysis output unit 60 analyzes the scene of a student who is not moving his / her hands while the teacher is writing on the board. The scene analysis output unit 60 determines that students B and C are in a boring state, and outputs the degree of boring class based on the ratio of students excluding student A (1-1 (student A) / 3 (students A-C)) out of the three students.

[0113] Also, for example, in the case of a dance lesson, the interaction detection unit 61 of the scene analysis output unit 60 analyzes the difference in timing of each of the students A to D dancing in response to the teacher's example. The scene analysis output unit 60 can output the students' understanding levels based on the ratio of students who dance at the appropriate timing among the four students A to D (3 students (students A to C) / 4 students (students A to D)).

[0114] Also, for example, in the case of the police, the interaction detection unit 61 of the scene analysis output unit 60 detects the evacuation behavior of the crowd E in response to the guiding instructions from the police officers. The scene analysis output unit 60 can output, as the panic degree, the ratio (1-crowd E / total number of crowds) excluding the attributes of the crowd E that has taken evacuating behavior out of the total crowd, with the whole crowd being set at 1.

[0115] In addition, the image analysis device 100 of the fourth embodiment is configured to include an image analysis device 102 that includes a time-series image acquisition unit 10, a skeleton point information acquisition unit 20, an attribute information acquisition unit 30, an attribute-attached skeleton point information generation unit 40, and a scene analysis output unit 62.

[0116] The attributed skeleton point information generating unit 40 integrates skeleton point information and attribute information of the same person from the skeleton estimation table and the attribute information acquisition table. In the fourth embodiment, a behavior recognition technique (behavior analysis and AI) using skeleton detection is used instead of the behavior analysis unit 50, and the scene analysis output unit 62 outputs the analysis result of the scene in the time-series image based on multiple pieces of attributed skeleton point information.

[0117] In this way, the image analyzing device 100 according to the fourth embodiment can perform scene analysis in the scene analysis output section 62.

[0118] (Modification) The present invention is not limited to the above-described embodiment, and modifications can be made without departing from the spirit of the present invention, for example, the following (a) to (f).

[0119] (a) The attribute information between different attributes may be based on the role of a person, for example, the different attributes may be a teacher attribute and a student attribute. (b) The attributed skeleton point information generating unit 40 may form groups of attributed skeleton point information to which the same attribute is assigned for a plurality of pieces of attributed skeleton point information. This allows the image analyzing devices 100 to 102 according to the first to fourth embodiments to utilize interactions between groups and perform more complex scene recognition. (c) The scene analysis output units 60 and 62 may output an alert depending on the scene analysis results. This makes it possible for the image analyzing devices 100 to 102 according to the first to fourth embodiments to be applied to, for example, surveillance cameras, people flow systems, and DX (Digital Transformation) in factories. (d) The behavior analysis unit 50 may store the time-series images obtained when analyzing a plurality of behavior analysis results in the storage unit 120. The behavior analysis unit 50 may also store the time-series images in a buffer until the processing for performing the composite analysis is completed, and delete the time-series images when the processing is completed. This allows the storage unit 120 to appropriately secure the necessary storage capacity. (e) The attribute information acquiring unit 30 may acquire attribute information from the time-series image at the same time as the skeleton point information acquiring unit acquires skeleton point information from the time-series image. This can improve real-time performance. The attribute information acquiring unit 30 is not limited to acquiring attribute information from the time-series image, and may, for example, be input by a user for each piece of skeleton information, or may acquire attribute information from another database. (f) The image analyzing devices 100 to 102 according to the first to fourth embodiments may, for example, associate voice with behavioral analysis and perform a composite analysis of the behavioral analysis result together with the voice. This can improve the accuracy of scene recognition. [Explanation of symbols]

[0120] 10 Time series image acquisition section 20 Skeleton point information acquisition unit 30 Attribute information acquisition section 40 Attributed skeleton point information generation unit 50 Behavior Analysis Department 60,62 Scene analysis output section 61 Interaction detection unit 100 Image analysis device 110 CPU 120 Storage section 121 Image Analysis Program 130 ROM 140 RAM 150 Input section 160 Display section 170 Communications Department 200 Cameras 300 Image Analysis System

Claims

1. a time-series image acquisition unit for acquiring time-series images of a plurality of people; a skeleton point information acquisition unit that acquires a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generating unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; a behavior analysis unit that performs behavior analysis on each of the plurality of pieces of attributed skeleton point information and performs a composite analysis of the plurality of behavior analysis results; a scene analysis output unit that outputs a result of analyzing a scene in the time-series images; An image analysis device comprising:

2. The scene analysis output unit includes: outputting an analysis result of the scene based on the relationship between different attributes and the behavior analysis result of the plurality of attribute-attached skeleton point information; The image analysis device according to claim 1 .

3. The scene analysis output unit includes: analyzing interactions of the plurality of attributed skeleton point information when outputting the scene analysis result; The image analysis device according to claim 1 .

4. The analysis of the interaction includes: and outputting an analysis result of the scene based on an action analysis result of the second attribute-attached skeleton point information, which is a result of the first attribute-attached skeleton point information having first attribute information related to a first person acting on the second attribute-attached skeleton point information having second attribute information related to a second person different from the first person. The image analysis device according to claim 3.

5. The attribute information between the different attributes is based on the role of the person. The image analysis device according to claim 2 .

6. The attributed skeleton point information generating unit forming a group of attributed skeleton point information to which the same attribute is assigned with respect to the plurality of attributed skeleton point information; The image analysis device according to claim 1 .

7. The scene analysis output unit includes: outputting an alert according to the result of the analysis of the scene; The image analysis device according to claim 1 .

8. The behavior analysis unit is storing the time-series images obtained when the plurality of behavioral analysis results are analyzed; The image analysis device according to claim 1 .

9. The behavior analysis unit is storing the time-series images in a buffer until the processing of the composite analysis is completed, and deleting the time-series images when the processing is completed; The image analysis device according to claim 8.

10. and at the same time when the skeleton point information acquisition unit acquires the skeleton point information from the time-series image, the attribute information acquisition unit acquires the attribute information from the time-series image. The image analysis device according to claim 1 .

11. a time-series image acquisition unit for acquiring time-series images of a plurality of people; a skeleton point information acquisition unit that acquires a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; an attribute information acquisition unit that acquires a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; an attribute-attached skeleton point information generating unit that associates the skeleton point information with the attribute information to generate a plurality of pieces of attribute-attached skeleton point information; a scene analysis output unit that outputs a result of analyzing a scene in the time-series images based on the plurality of attributed skeleton point information; An image analysis device comprising:

12. acquiring time-series images of a plurality of people; acquiring a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; performing behavior analysis on each of the plurality of pieces of attributed skeleton point information and performing a composite analysis of the plurality of behavior analysis results; outputting a result of the analysis of the scene in the time series of images; Perform image analysis methods.

13. acquiring time-series images of a plurality of people; acquiring a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; outputting a result of analyzing a scene in the time-series images based on the plurality of pieces of attributed skeleton point information; Perform image analysis methods.

14. A step of acquiring time-series images of a plurality of people; obtaining a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; a step of generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; a step of performing behavior analysis on each of the plurality of pieces of attributed skeleton point information and performing a composite analysis of the plurality of behavior analysis results; outputting a result of the analysis of the scene in the time series images; An image analysis program that allows a computer to execute the above.

15. A step of acquiring time-series images of a plurality of people; obtaining a plurality of skeleton point information of each of the plurality of persons from each frame image of the time-series images; acquiring a plurality of pieces of attribute information corresponding to the plurality of pieces of skeleton point information; a step of generating a plurality of pieces of attributed skeleton point information by associating the skeleton point information with the attribute information; a step of outputting a result of analysis of a scene in the time-series images based on the plurality of pieces of attributed skeleton point information; An image analysis program that allows a computer to execute the above.