Video processing device, video processing method, and program

Abstract

To accurately identify person's behavior.SOLUTION: A video acquisition unit 201 of a video processing device 100 acquires a video (S901). A skeleton determination unit 203 determines a plurality of specific positions of a person in the video (S903). An unstable posture determination unit 207 determines whether the person is in an unstable posture based on determination results of the plurality of specific positions of the person and determination results of at least a part of the video in the direction of gravity (S907).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method for determining a person's behavior. [Background technology]

[0002] In recent years, technology has been proposed to detect human behavior from surveillance camera footage, and applications are progressing in analyzing customer behavior in stores and monitoring patients in hospitals. Proposals have also been made to analyze human behavior by estimating the joint positions of people in footage. However, it is difficult to detect different behaviors when the posture appears to be the same from the joint positions. Therefore, Patent Document 1 describes a method of detecting objects (such as chairs) around a person and distinguishing between behaviors such as "squatting" and "sitting on a chair" based on the relationship between the joint positions of the target person and the object positions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-21940 [Non-patent literature]

[0004] [Non-Patent Document 1] J. Redmon,“You Only Look Once: Unified,Real-Time Object Detection”,CVPR2015. [Non-Patent Document 2] Z. Cao, “OpenPose:Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, CVPR2018. [Non-Patent Document 3] Rene Ranftl,et.al.“Vision Transformers for Dense Prediction”arXiv:2103.13413 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, the method of Patent Document 1 has difficulty in distinguishing between a person's "falling on his / her bottom" on the floor and "squatting." In addition, the method of Patent Document 1 has difficulty in distinguishing between a person's "standing" in a passageway and a person's "lying down" with his / her head facing the back in the camera's shooting direction. Thus, with conventional technology, it was sometimes difficult to distinguish between multiple actions that were independent of the object's position.

[0006] The present invention has been made in consideration of the above problems, and an object of the present invention is to provide a method capable of identifying a person's behavior with high accuracy. [Means for solving the problem]

[0007] In order to solve the above problems, the image processing device of the present invention has, for example, the following configuration: namely, an acquisition means for acquiring an image, a position determination means for determining a plurality of specific positions of a person in the image acquired by the acquisition means, a gravity determination means for determining the direction of gravity in at least a portion of the image acquired by the acquisition means, and a posture determination means for determining whether the person is in an unstable posture based on the determination results of the plurality of specific positions of the person by the position determination means and the determination result of the direction of gravity by the gravity determination means. Effect of the Invention

[0008] According to the present invention, the actions of a person can be identified with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a video processing device 100. [Diagram 2] 1 is a diagram showing a functional configuration of a video processing device 100 according to a first embodiment. [Diagram 3]13 is an example of a skeleton determination result. [Figure 4] 13 is a flowchart showing the flow of a process for determining the direction of gravity. [Diagram 5] This is an example of a gravity vector field. [Figure 6] 13 is an example of vector selection for use in determining a gravity vector field. [Figure 7] This is an example of a gravity vector field. [Figure 8] 1 is an example of a user interface. [Figure 9] 4 is a flowchart showing a process flow of the video processing device 100 in the first embodiment. [Figure 10] 13 is an example of an unstable posture determination in a standing position. [Figure 11] 13 is an example of an unstable posture determination in a sitting position. [Figure 12] This is an example of determining an unstable posture when a body support device is present. [Figure 13] FIG. 11 is a diagram showing a functional configuration of a video processing device 100 according to a second embodiment. [Figure 14] 13 is a flowchart showing a process flow of the video processing device 100 in the second embodiment. [Figure 15] 1 is an example of a graph showing a time change in instability. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiment is merely an example, and is not intended to limit the scope of the present invention. In the drawings, the same or similar configurations are given the same reference numerals, and duplicated explanations are omitted. Note that in this embodiment, an "action" is described as being specified by one or more consecutive postures. In other words, in this embodiment, a single posture can also be regarded as one of the actions.

[0011] 1 is a block diagram showing a hardware configuration of a video processing device 100 according to this embodiment. The video processing device 100 according to this embodiment includes a CPU 101, a ROM 102, a RAM 103, a secondary storage device 104, an imaging device 105, an input device 106, a display device 107, a network interface 108 (hereinafter, network I / F 108), and a bus 109.

[0012] The CPU 101 is a processor that executes instructions according to programs stored in the ROM 102 and the RAM 103. The ROM 102 is a non-volatile memory that stores programs for executing processes according to flowcharts described later and other programs and data required for control. The RAM 103 is a volatile memory for temporarily storing video / image data, pattern discrimination results, and the like.

[0013] The secondary storage device 104 is a rewritable secondary storage device such as a hard disk drive or a flash memory. Various information stored in the secondary storage device 104 is transferred to the RAM 103, and the CPU 101 can execute the program according to this embodiment. The imaging device 105 has an imaging lens, an imaging sensor such as a CCD or a CMOS, a video signal processing unit, and the like.

[0014] The input device 106 is a device that accepts input from a user, such as a keyboard or a mouse.

[0015] The display device 107 is a device for displaying processing results to a user, such as a liquid crystal display. The network I / F 108 is a modem or LAN for connecting to a network such as the Internet or an intranet. The bus 109 is an internal bus for connecting the above-mentioned hardware and inputting and outputting data between them.

[0016] First Embodiment The method of determining behavior and posture in this embodiment will be described in detail with reference to the drawings. FIG. 2 is a diagram showing the functional configuration of the image processing device 100. The image processing device 100 of this embodiment determines whether or not a person identified from an image is in an unstable posture. The image processing device 100 has an image acquisition unit 201, a person detection unit 202, a skeleton determination unit 203, a posture type determination unit 204, a skeleton information storage unit 205, a gravity direction determination unit 206, an unstable posture determination unit 207, and a display unit 208.

[0017] The video acquisition unit 201 acquires a video using the imaging device 105. Note that, in this embodiment, an example in which the video processing device 100 has the imaging device 105 will be mainly described, but the video processing device 100 does not necessarily have to have the imaging device 105. In this case, the video acquisition unit 201 of the video processing device 100 acquires a video via a wired or wireless communication medium.

[0018] The person detection unit 202 detects a person's area from the image acquired by the image acquisition unit 201. The person detection unit 202 of this embodiment detects the whole body area of ​​a person.

[0019] The skeleton determination unit 203 executes position determination processing for determining specific positions of a person from the whole body region detected by the person detection unit 202. The specific positions in this embodiment are shown in Fig. 3. As shown in the figure, the specific positions in this embodiment are shoulders 301 and 302, elbows 303 and 304, wrists 305 and 306, waists 307 and 308, knees 309 and 310, ankles 311 and 312, eyes 313 and 314, nose 317, and ears 315 and 316. In other words, the specific positions in this embodiment include the positions of the person's joints and organs.

[0020] The posture type determination unit 204 determines the type of posture of the person (e.g., standing, sitting, etc.) based on the specific position determined (estimated) by the skeleton determination unit 203. The skeleton information storage unit 205 stores skeleton information that associates information on the specific position determined (estimated) by the skeleton determination unit 203 with the type of posture determined by the posture type determination unit 204. The skeleton information storage unit 205 is realized by the RAM 103 or the secondary storage device 104.

[0021] The gravity direction determination unit 206 executes a gravity determination process for determining the gravity direction in at least a partial area in the video based on the skeleton information stored in the skeleton information storage unit 205. The method of determining (estimating) the gravity direction by the gravity direction determination unit 206 will be described later.

[0022] The unstable posture determination unit 207 determines whether or not the posture of the person is unstable based on the direction of gravity determined by the gravity direction determination unit 206 and the skeletal information stored in the skeletal information storage unit 205. The display unit 208 displays the determination result by the unstable posture determination unit 207. The display unit 208 is realized by the display device 107.

[0023] Next, the details of the processing of the video processing device 100 in the first embodiment will be described. Fig. 4 is a flowchart for explaining the processing executed by the video processing device 100 of this embodiment to determine (estimate) the direction of gravity. Each operation shown in Fig. 4 is realized by the CPU 101 of the video processing device 100 reading a necessary program from the ROM 102 to the RAM 103 and executing it. The processing in Fig. 4 is started when a user instructs to start determining the direction of gravity. However, for example, the processing may be automatically started when a preset condition is satisfied.

[0024] 4 starts, the video acquisition unit 201 acquires video of the inside of the store (S401). In addition, in S401, the video acquisition unit 201 associates time information (time stamp or frame ID) with each image frame constituting the video.

[0025] In S402, the person detection unit 202 detects a person's area (person area) from within the frame image. In this embodiment, the person detection unit 202 detects a whole-body area in which the whole body of a person is reflected as the person area. As a method for the person detection unit 202 to detect the person area, for example, there is a method using a Convolutional Neural Network (CNN) as described in Non-Patent Document 1. However, the method is not limited to the method described in Non-Patent Document 1, and other methods may be used as long as they are capable of detecting a person area.

[0026] In this embodiment, the person area is rectangular, and the position of the person area is expressed by the x and y coordinates of the two points, the top left and bottom right, in a coordinate system with the top left of the frame image as the origin. Furthermore, time information of the frame image is added to each person area.

[0027] In S403, the skeleton determination unit 203 determines specific positions of a person for all person regions in the frame image. Then, the skeleton determination unit 203 generates information (joint point list) that lists specific positions for each person region. One method for determining (estimating) a skeleton is to estimate the coordinates of each joint point using CNN as in Non-Patent Document 2, for example, and to obtain the reliability thereof. However, the method is not limited to the method in Non-Patent Document 2, and other methods may be used as long as the coordinates of specific positions can be determined.

[0028] In this embodiment, a joint point list is created for each person region included in a frame image. Each joint point list includes time information of the frame image, coordinates of all specific positions of the person, and reliability in a specific order.

[0029] Note that, in this embodiment, the case where skeletal determination is performed on each person area detected from the entire frame image will be mainly described, but skeletal determination may also be performed on the entire frame image and then grouped into person units based on the relationship between each specific position.

[0030] In S404, the posture type determination unit 204 determines the type of posture of the person (for example, standing, sitting, etc.) from the above-mentioned joint point list. For example, a method for determining the posture type is to use a large number of joint point lists prepared for each posture type as learning data and train a CNN or a Multi Layer Perceptron (MLP) as a class identification problem. In this method, the posture type corresponding to the maximum likelihood among the likelihoods obtained for each type (class) is adopted as the posture type of the person.

[0031] In S405, the skeleton information storage unit 205 stores the joint point list with a posture type ID as skeleton information. The posture type ID is an ID that is preset for each posture type, such as 1 for a standing position, 2 for a sitting position, and so on.

[0032] The image processing device 100 of this embodiment repeats the processes from S401 to S405 until a predetermined time has elapsed. It is assumed that people will appear at various positions within the shooting range as the predetermined time elapses, and the direction of gravity at each position within the shooting range is obtained based on information obtained during the predetermined time. However, the condition for repeating S401 to S405 is not limited to time. Other conditions may include, for example, the number of people detected in the image reaching a predetermined number, or people being detected from all positions within the frame screen.

[0033] The direction of gravity in at least a part of the area in the image is determined (estimated) by the following steps S407 to S409. In the image processing device 100 of this embodiment, the direction in which gravity acts at each position in the image, such as the floor or ground, is expressed by a unit vector. An example of this is shown in FIG. 5. 501 indicates the entire image, with a person 502 standing in a room. The floor 503 is covered by an arrow indicating the direction of gravity, represented by 504. In this embodiment, a specific area in the image frame, such as the floor or ground, is divided into a predetermined size (10 pixels vertically and horizontally), and a gravity direction is associated with each of the divided areas, and a divided area associated with one gravity direction is called a gravity vector field. However, the division method and division size are not limited to the above method.

[0034] In S407, the gravity direction determination unit 206 reads out skeleton information for a predetermined time from the skeleton information storage unit 205. In S408, the gravity direction determination unit 206 selects a direction to be used for determining the gravity direction from the joint point list included in the skeleton information.

[0035] Details of S408 will be described with reference to Fig. 6. Fig. 6 shows an example in which an image 601 contains a person 602 in a standing position and a person 603 in a sitting position. The black dots on each person represent the specific positions shown in Fig. 3. Note that the person 603 is sitting on a chair 604. The process of selecting the direction to be used to determine the direction of gravity for each of the two postures, standing and sitting, will be described.

[0036] For a person 602 in a standing position (posture type ID=1), the gravity direction determination unit 206 selects a direction 611 from a midpoint 607 between waists 605 and 606 toward a midpoint 610 between ankles 608 and 609. When standing, the head and upper body may tilt, but the positions of the waist and feet are often stable to support the body and balance it against the direction of gravity. For this reason, the gravity direction determination unit 206 of this embodiment selects a direction 611 from the center point of the waist of a person in a standing position toward the midpoint of both ankles. However, this is just one example, and the method is not limited to this example as long as it can accurately determine the direction of gravity when standing.

[0037] On the other hand, for the person 603 in a sitting position (posture type ID=2), the gravity direction determination unit 206 selects a direction 618 from a midpoint 614 between the eyes 612 and 613 toward a midpoint 617 between the waists 615 and 616. In the case of a sitting position, the waist supports the upper body and balances the body against the direction of gravity, so that the waists are often in a stable position. Therefore, the gravity direction determination unit 206 of this embodiment selects a direction 618 from the midpoint between the eyes of the person in a sitting position toward the center point of the waist. In this way, the gravity direction determination unit 206 of this embodiment determines (estimates) the direction of gravity in a different manner depending on whether the person is standing or sitting. Note that the method of selecting the direction of the person 603 in a sitting position is not limited to this. For example, instead of using the midpoint 614 between the eyes, the center position between the eyes and the nose may be used. Also, instead of using the midpoint between the waists 615 and 616, the midpoint between the waists and the knees may be used. Although the upper body tends to lean forward relatively easily when sitting, the direction of gravity can be determined with high accuracy by using a sufficient number of vectors.

[0038] Using the method described above, the gravity direction determination unit 206 determines the gravity direction according to the type of person's posture in S408. As a result, when a chair 702 is placed on a screen 701 as shown in Fig. 7, the gravity direction 703 of not only the floor surface but also the chair surface can be determined.

[0039] In S409, the gravity direction determination unit 206 determines a gravity vector field from the gravity direction found in S408. As described above, in this embodiment, a gravity direction is associated with each divided area obtained by dividing a specific surface (floor surface, ground surface, chair surface, etc.) of an image frame into a predetermined size (10 pixels vertically and horizontally). More specifically, the gravity direction determination unit 206 determines, for each divided area, the average of all gravity directions within a predetermined range from that area as the gravity direction of that divided area.

[0040] In this embodiment, the gravity direction determination unit 206 determines the gravity direction based on organ points according to the type of posture of the person, but this is not limited to the above. For example, there is a method of determining depth information and semantic information for each pixel from an image using Vision Transformer (ViT) as described in Non-Patent Document 3. According to this method, the gravity direction relative to the floor can be obtained from the determined relationship between the floor and the wall.

[0041] It is also possible for the user to input the direction of gravity via the input device 106. An example of a user interface is shown in Fig. 8. Reference numeral 801 denotes the entire screen, and the user can specify the floor surface by operating the mouse or the like. The shaded area 802 is an example of the specified floor surface. The user can set the direction of gravity at each position on the floor surface by specifying a position on the screen with the mouse and inputting an angle on the keyboard while visually checking the inclination of vector symbol 803 that serves as a guide.

[0042] Next, details of the process during operation of the video processing device 100 in the first embodiment will be described. Fig. 9 is a flowchart for explaining the process executed by the video processing device 100 of this embodiment to determine (estimate) the posture of a person. Each process shown in Fig. 9 is realized by the CPU 101 of the video processing device 100 reading out a necessary program from the ROM 102 to the RAM 103 and executing it. The process in Fig. 9 is started when a user instructs to execute posture determination. However, for example, the process in Fig. 9 may be automatically started when a preset condition is satisfied as a trigger.

[0043] In S901, the video acquisition unit 201 acquires a video in the same manner as in S401. In S902, the person detection unit 202 detects a person's area (person area) from within a frame image constituting the video in the same manner as in S402. In S903, the skeleton determination unit 203 determines a specific position (position of a joint point, etc.) of the person in the frame image in the same manner as in S403. In S904, the posture type determination unit 204 determines a type of posture of the person (for example, standing, sitting, etc.) from the determination result of the specific position of the person in the same manner as in S404.

[0044] In S905, the unstable posture determination unit 207 determines a reference point used to determine whether the posture of the person is unstable. In addition, in S906, the unstable posture determination unit 207 determines a body supporting region.

[0045] In this embodiment, the reference point is a specific position selected according to the person's posture type, and it is determined whether or not this specific position is supported. In addition, the body support area in this embodiment is an area used to determine whether or not the reference point is supported, and is determined according to the person's posture. The unstable posture determination unit 207 in this embodiment determines the reference point and the body support area based on the specific position according to the person's posture type, similar to the method described in S408.

[0046] FIG. 10 shows a state where one person 1001 of two people who are determined to be in a standing position based on the specific positions is actually standing, and the other person 1002 is actually falling. The unstable posture determination unit 207 of this embodiment determines the midpoint 1005 (and 1013) of the waists 1003 and 1004 (and 1011 and 1012) as the reference point for a person determined to be in a standing position. As described above, this is because in the case of a standing position, the positions of the waist and feet are often more stable in the direction of gravity than the head and shoulders. In addition, the unstable posture determination unit 207 of this embodiment determines the region 1008 (and 1016) including the ankles 1006 and 1007 (and 1014 and 1015) as the body support region for a person determined to be in a standing position. The body support region of this embodiment is a circle whose diameter is a distance obtained by providing a margin of a predetermined percentage to the distance between both ankles and whose center is the midpoint of both ankles.

[0047] The method of determining the reference point and the body supporting region is not limited to the above-mentioned method. For example, the reference point may be determined by including other joint points such as both shoulders in addition to both hips. The body supporting region may be determined by including other joint points such as both knees in addition to both ankles. The body supporting region does not have to be circular, and may be elliptical or rectangular.

[0048] FIG. 11 shows a state where one person 1101 of two people who are determined to be in a sitting position based on a specific position is actually sitting, and the other person 1102 is actually falling. The unstable posture determination unit 207 of this embodiment determines the midpoint 1105 (and 1115) between the eyes 1103 and 1104 (and 1113 and 1114) as the reference point for a person determined to be in a sitting position. This is because, as described above, in the case of a sitting position, both hips are often in a stable position. In addition, the unstable posture determination unit 207 of this embodiment determines the area 1110 (and 1120) including the hips 1106, 1107 (and 1116, 1117) and both knees 1108, 1109 (and 1118, 1119) as the body support area for a person determined to be in a sitting position. The body support area in this embodiment is a circle whose diameter is the distance from the waist to the knees plus a specified margin, and whose center is the midpoint between both knees and both waists.

[0049] The method of determining the reference point and the body supporting region is not limited to the above-mentioned method. For example, the reference point may be determined by including the positions of other organs such as the nose and joint points in addition to the eyes. The body supporting region may be determined by using only the hips. The body supporting region does not need to be circular, but may be elliptical or rectangular, and the shape of the body supporting region may be different between the standing position and the sitting position.

[0050] In S907, the unstable posture determination unit 207 determines whether or not the posture of the target person is unstable based on the positional relationship between the reference point determined in S905 and the body supporting region determined in S906. That is, the unstable posture determination unit 207 determines whether or not the target person has an unstable posture based on the determination results of a plurality of specific positions of the target person by the skeleton determination unit 203 and the determination result of the gravity direction by the gravity direction determination unit 206. The unstable posture determination unit 207 of this embodiment determines whether or not the target person has an unstable posture based on whether or not a line segment extended from the reference point in the gravity direction reaches the body supporting region.

[0051] A case where a person is determined to be in a standing posture will be described with reference to FIG. 10. Since a line segment 1010 extending from a reference point 1005 of a person 1001 in a gravity direction 1009 is within a body supporting area 1008, the person 1001 is determined to be in a stable posture. On the other hand, since a line segment 1018 extending from a reference point 1013 of a person 1002 in a gravity direction 1017 is outside a body supporting area 1016, the person 1002 is determined to be in an unstable posture. That is, when the type of a person's posture is determined to be standing, the unstable posture determination unit 207 of this embodiment determines whether the person is in an unstable posture based on a reference point corresponding to the position of the person's waist. Furthermore, when a line segment extending from a reference point 1005 in a gravity direction 1009 does not fall within a predetermined range (within the body supporting area 1008) specified based on a plurality of specific positions corresponding to the person, the unstable posture determination unit 207 determines that the person is in an unstable posture.

[0052] A case where a person is determined to be in a sitting posture will be described with reference to FIG. 11. A line segment 1112 extending from a reference point 1105 of a person 1101 in a gravity direction 1111 is within a body support area 1110, so that the person 1101 is determined to be in a stable posture. On the other hand, a line segment 1122 extending from a reference point 1115 of a person 1102 in a gravity direction 1121 is outside a body support area 1120, so that the person 1102 is determined to be in an unstable posture. That is, when the type of a person's posture is determined to be a sitting posture, the unstable posture determination unit 207 of this embodiment determines whether the person is in an unstable posture based on a reference point corresponding to the position of the person's eyes. Also, when a line segment extending from a reference point 1115 in a gravity direction 1121 does not fall within a predetermined range (within a body support area 1110) specified based on a plurality of specific positions corresponding to the person, the unstable posture determination unit 207 determines that the person is in an unstable posture.

[0053] In addition, in the case of a person using a tool for supporting the body, such as a cane, as shown in FIG. 12, the unstable posture determination unit 207 detects the body support tool and determines a body support region 1206 based on the sparse detection result so that the tip position of the body support tool 1202 is included. As a method for detecting the body support tool, for example, the method of Non-Patent Document 1 can be used. Since a line segment 1205 extending from a reference point 1203 of a person 1201 using a cane shown in FIG. 12 in a gravity direction 1204 is within the body support region 1206, the posture is determined to be stable. Although the posture of the person 1002 shown in FIG. 10 and the posture of 1201 shown in FIG. 12 are the same, by considering the presence of the body support tool, it is possible to distinguish with higher accuracy whether a person has actually fallen or not.

[0054] In addition, the farther the line segment extending from the reference point in the direction of gravity is from the body supporting region, the more unstable the posture is considered to be. Therefore, the unstable posture determination unit 207 of this embodiment determines the degree of instability (instability level) of each person based on the distance between the line segment and the body supporting region. For example, if the distance from the reference point to the boundary of the body supporting region and the line segment extending from the reference point in the direction of gravity is d and the distance from the reference point to the boundary of the body supporting region is L, the instability level U can be expressed by Equation 1. U=d / L (formula 1)

[0055] The instability level U is expressed as a real number between 0 and 1. When d and L are the same, the person has completely fallen, so 1 indicates the most instability. However, the method of calculating the degree of anxiety is not limited to this. By calculating the instability level in this way, it is possible to switch the content of processing based on the instability level, such as outputting an alert only when a person with an instability level above a certain threshold is detected. However, calculating the instability level is not essential.

[0056] In S908, the display unit 208 displays the determination result by the unstable posture determination unit 207. For example, a message saying "There is a person with an unstable posture" may be displayed, or a rectangle surrounding the person with an unstable posture may be displayed superimposed on the camera image. When calculating the degree of posture instability, a numerical value indicating the degree of instability, a color / bar graph according to the degree of instability, or the like may be displayed. The image processing device 100 of this embodiment repeats the process of FIG. 9 until receiving an end instruction from the user.

[0057] As described above, according to the image processing device 100 of the present embodiment, it is possible to determine whether or not a person's posture is unstable based on the relationship between the reference point and the body supporting area determined based on the specific position of the person. In addition, the reference point and the body supporting area are set by different methods depending on the type of posture. By using the gravity vector field described with reference to FIG. 4, an unstable person can be detected with high accuracy even if the downward direction of the screen differs from the direction of gravity due to the shooting direction or an installed object such as a chair.

[0058] <Second embodiment> Next, a second embodiment of the present invention will be described, focusing on the differences from the first embodiment. Fig. 13 is a diagram showing the functional configuration of a video processing device 100 in the second embodiment. The video processing device 100 in the second embodiment has a person tracking unit 1303, and is capable of handling time changes in the results of determining (estimating) a person's posture. The video acquisition unit 1301 and the video acquisition unit 201, and the person detection unit 1302 and the person detection unit 202 each have the same function.

[0059] The person tracking unit 1303 associates person areas acquired by the person detection unit 1302 in temporally successive image frames and assigns the same person ID. The skeleton determination unit 1304 and the skeleton determination unit 203, the posture type determination unit 1305 and the posture type determination unit 204, and the skeleton information storage unit 1306 and the skeleton information storage unit 205 each have the same function. In addition, the gravity direction determination unit 1307 and the gravity direction determination unit 206, and the unstable posture determination unit 1308 and the unstable posture determination unit 207 each have the same function.

[0060] An unstable posture classification unit 1309 determines the type of unstable posture from the time change in the output of the unstable posture determination unit 1308. A display unit 1310 has the same function as the display unit 208. In addition, the gravity direction determination process in the second embodiment is similar to that in the first embodiment, and therefore will not be described.

[0061] Next, details of the process during operation of the video processing device 100 in the second embodiment will be described with reference to FIG. 14. FIG. 14 is a flowchart for describing the process executed by the video processing device 100 of this embodiment to determine (estimate) the posture of a person. Each process shown in FIG. 14 is realized by the CPU 101 of the video processing device 100 reading a necessary program from the ROM 102 to the RAM 103 and executing it. The process in FIG. 14 is started when a user instructs to execute posture determination. However, for example, the process in FIG. 14 may be automatically started by being triggered by the satisfaction of a preset condition. Note that S1401 is the same as S901, and S1402 is the same as S902, and therefore description thereof will be omitted.

[0062] In S1403, the person tracking unit 1303 determines whether the person area detected in the current frame corresponds to one or more person areas detected in the previous frame. The person tracking unit 1303 assigns the same person ID to the person areas determined to be the same person.

[0063] There are various methods for tracking, such as a method of matching the center position of a person region included in a previous frame with the center position of a person region included in a current frame that is closest to the center position of the person region included in the previous frame. There are also various other methods for matching person regions between frames, such as a pattern matching method that uses the person region in the previous frame as a matching pattern.

[0064] Since S1404 is similar to S903, S1405 is similar to S904, S1406 is similar to S905, and S1407 is similar to S906, the description thereof will be omitted.

[0065] In S1408, the unstable posture determination unit 1308 calculates the instability of the person by the same method as in S907. In addition, the unstable posture determination unit 1308 temporarily stores an instability information list in which the instability is arranged in chronological order for each person (person ID) in the RAM 103. In S1409, the unstable posture classification unit 1309 reads the instability information list temporarily stored in the RAM 103, and performs a type determination of the unstable posture for each person ID.

[0066] FIG. 15 is an example of a graph with time on the horizontal axis and instability on the vertical axis. Graph 1501 shows a state in which instability above a certain level continues. This state is considered to be "unsteadiness" behavior. Graph 1502 shows a state in which instability increases rapidly. This state is considered to be "falling" behavior. However, the method of determining behavior is not limited to this, and behaviors determined based on the change in the degree of instability over time are not limited to "unsteadiness" and "falling".

[0067] In S1409, the display unit 1310 displays the unstable posture determination result. For example, a message such as "There is a person unsteady" may be displayed, or a rectangle surrounding the person in a color corresponding to the type of unstable posture may be superimposed on the camera image. The image processing device 100 of this embodiment repeats the process of FIG. 14 until an end instruction is received from the user. As described above, the image processing device 100 of this embodiment can determine the type of unstable posture from the time change in the instability degree.

[0068] <Other embodiments> In the above embodiment, the determination result by the unstable posture determination unit 207 is displayed, but in addition to this, the video when an unstable person is detected may be recorded, or meta information indicating the time when the unstable person is detected may be added. This makes it easy to search for the video when an unstable person is detected.

[0069] In the above embodiment, the body support region is determined based on a plurality of specific positions of a person, but the present invention is not limited to this. For example, the body support region may be determined as a region obtained by adding a predetermined margin from one of the person's ankles. In this way, even if the person is facing sideways with respect to the camera and only one of the person's feet is captured, it is possible to determine whether the person is in an unstable posture.

[0070] In the above embodiment, the video processing device 100 has all the functions shown in Fig. 2 or 13, but the present invention is not limited to this. For example, the processes from the person detection unit 202 to the unstable posture determination unit 207 may be performed on the cloud, and the results may be displayed on a display installed in a store or the like.

[0071] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0072] 201 Video Acquisition Unit 203 Skeleton Judgment Department 206 Gravity direction determination section 207 Unstable posture determination unit

Claims

1. An acquisition means for acquiring a video; a position determination means for determining a plurality of specific positions of a person in the video image acquired by the acquisition means; a gravity determination means for determining a direction of gravity in at least a part of the image acquired by the acquisition means; a posture determination means for determining whether the person is in an unstable posture based on a result of determination of a plurality of specific positions of the person by the position determination means and a result of determination of a direction of gravity by the gravity determination means; 13. A video processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the specific positions determined by the position determination means include at least one of the positions of one or more joints, a head, eyes, a nose, and a mouth of a person.

3. The image processing device according to claim 1 or 2, characterized in that the posture determination means identifies a reference point and a body support area of ​​the person based on multiple specific positions of the person, and determines whether the person is in an unstable posture based on the positional relationship between the reference point and the body support area.

4. The image processing device according to claim 3, characterized in that the posture determination means determines that the person is in an unstable posture if a line segment extended from a reference point of the person in the direction of gravity does not fall within the specified range identified based on a plurality of specific positions corresponding to the person.

5. a type determining means for determining a type of the posture of the person based on the plurality of specific positions determined by the position determining means, 4. The image processing apparatus according to claim 3, wherein the posture determining means determines whether the person is in an unstable posture based on a reference point corresponding to the type of posture determined by the type determining means.

6. When the type determination means determines that the type of the posture of the person is a standing posture, the posture determination means determines whether the person is in an unstable posture based on the reference point corresponding to a position of the person's waist, When the type determination means determines that the type of the person's posture is a sitting posture, the posture determination means determines whether the person is in an unstable posture based on the reference point corresponding to the position of the person's eyes.

6. The image processing device according to claim 5,

7. 3. The image processing device according to claim 1, wherein the gravity determining means determines a direction of gravity for each of one or a plurality of areas in the image.

8. The image processing device according to claim 1 or 2, characterized in that the posture determination means further determines a degree of instability of the person based on the determination results of the multiple specific positions by the position determination means and the determination result of the gravity direction by the gravity determination means.

9. 9. The image processing apparatus according to claim 8, wherein the posture determining means further determines a type of the unstable posture based on a time change in a degree of instability of the person.

10. a detection means for detecting a body support device from the image acquired by the acquisition means; The image processing device according to claim 1 or 2, characterized in that the posture determination means determines whether the person is in an unstable posture based on the determination results of multiple specific positions of the person by the position determination means, the determination result of the gravity direction by the gravity determination means, and the detection result of the body support device by the detection means.

11. 3 . The image processing device according to claim 1 , wherein the gravity determining means determines the direction of gravity based on a reference point according to a posture of a person in the image acquired by the acquiring means. 4 .

12. A posture determination method performed by a video processing device, comprising: An acquisition step of acquiring an image; a position determination step of determining a plurality of specific positions of a person in the video acquired by the acquisition step; a gravity determination step of determining a direction of gravity in at least a portion of the image acquired by the acquisition step; a posture determination step of determining whether the person is in an unstable posture based on a result of determining a plurality of specific positions of the person in the position determination step and a result of determining a direction of gravity in the gravity determination step; A posture determination method comprising the steps of:

13. The posture determination method according to claim 12, characterized in that in the posture determination process, a reference point and a body support area of ​​the person are identified based on a plurality of specific positions of the person, and whether the person is in an unstable posture is determined based on the positional relationship between the reference point and the body support area.

14. The posture determination method described in claim 13, characterized in that in the posture determination process, if a line segment extended from a reference point of the person in the direction of gravity does not fall within the specified range identified based on a plurality of specific positions corresponding to the person, the person is determined to be in an unstable posture.

15. A program for causing a computer to operate as the image processing device according to claim 1 or 2.

Images