Video image processing apparatus, posture determination method, and storage medium

Abstract

The invention discloses a video image processing apparatus, a posture determination method, and a storage medium. The video image processing apparatus includes: an acquisition unit configured to acquire a video image; a position determination unit configured to determine a plurality of specific positions of a person in the video image acquired by the acquisition unit; a gravity direction determination unit configured to determine a gravity direction in at least a part of a region in the video image acquired by the acquisition unit; and a posture determination unit configured to determine whether the person is in an unstable posture based on a determination result of the plurality of specific positions of the person by the position determination unit and a determination result of the gravity direction by the gravity direction determination unit.

Description

Technical Field

[0001] The invention relates to a method for determining a character's action. Background Art

[0002] In recent years, a technology for detecting the movement of a person from a video image of a surveillance camera has been proposed, and the application of this technology in the behavior analysis of store customers or the behavior monitoring of hospital patients has been expanding. In addition, a technology for analyzing the movement of a person by estimating the joint positions of a person in a video image has been proposed. However, it is difficult to detect the movement of a person when the posture appears to be the same as the posture taken in different actions only from the joint positions of the person. Japanese Patent Gazette No. 2022-21940 discusses a method for detecting an object (such as a chair) in the surrounding environment of a person, and distinguishes between the actions of a person "doing a squat" and "sitting on a chair" based on the relationship between the joint positions of the target person and the position of the object, for example.

[0003] However, according to the method discussed in Japanese Patent Laid-Open No. 2022-21940, it is difficult to distinguish between the action of a person "falling on his buttocks" and the action of a person "doing a squat". In addition, according to the method discussed in Japanese Patent Laid-Open No. 2022-21940, it is also difficult to distinguish between the action of a person "standing in a passage" and the action of a person "lying down with the head of the person facing the back in the shooting direction of the camera". As described above, there are cases where multiple actions that do not depend on the position of the object cannot be distinguished based on conventional technology. Summary of the invention

[0004] In view of the above problems, the present invention relates to a method capable of accurately distinguishing the actions of a person.

[0005] According to one aspect of the present invention, a video image processing device includes: an acquisition unit configured to acquire a video image; a position determination unit configured to determine a plurality of specific positions of a person in the video image acquired by the acquisition unit; a gravity direction determination unit configured to determine the gravity direction in at least a portion of an area in the video image acquired by the acquisition unit; and a posture determination unit configured to determine whether the person is in an unstable posture based on a determination result of the plurality of specific positions of the person by the position determination unit and a determination result of the gravity direction by the gravity direction determination unit.

[0006] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram showing a hardware configuration example of a video image processing device.

[0008] Figure 2 is a block diagram showing the functional configuration of a video image processing apparatus.

[0009] Figure 3 is a diagram showing an example of a skeleton determination result.

[0010] Figure 4 : is a flowchart showing the flow of the gravity direction determination process.

[0011] Figure 5 is a diagram showing an example of a gravity vector field.

[0012] Figure 6 is a diagram showing an example of vector selection for determination of a gravity vector field.

[0013] Figure 7 is a diagram showing an example of a gravity vector field.

[0014] Figure 8 is a diagram showing an example of a user interface.

[0015] Fig. 9 : is a flowchart showing the processing flow of the video image processing device.

[0016] Fig.10 is a diagram showing an example of unstable posture determination in a standing posture.

[0017] Fig.11 is a diagram showing an example of unstable posture determination in a sitting position.

[0018] Fig.12 is a diagram showing an example of unstable posture determination in a case where a character has a body support device.

[0019] Fig.13 is a block diagram showing the functional configuration of a video image processing apparatus.

[0020] Fig.14 : is a flowchart showing the processing flow of the video image processing device.

[0021] Fig.15 : is a diagram showing an example of a graph showing a temporal change in the degree of instability. DETAILED DESCRIPTION

[0022] Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. Note that the following exemplary embodiments are merely examples and are not intended to limit the scope of the present invention. In the accompanying drawings, identical or similar components are given identical reference numerals, and redundant descriptions thereof are omitted. In the exemplary embodiments, a description is given assuming that an "action" is identified by one gesture or a plurality of continuous gestures. In other words, in the exemplary embodiments, a single gesture can be considered as an action.

[0023] Figure 1 1 is a block diagram showing a hardware configuration of a video image processing apparatus 100 according to an exemplary embodiment. The video image processing apparatus 100 according to the present exemplary embodiment includes a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, a secondary storage device 104, an imaging device 105, an input device 106, a display device 107, a network interface (I / F) 108, and a bus 109.

[0024] The CPU 101 is a processor that executes instructions according to a program stored in the ROM 102 or the RAM 103. The ROM 102 is a nonvolatile memory that stores programs for executing processes related to the flowcharts described below and other programs and data required for control. The RAM 103 is a volatile memory for temporarily storing video / image data or mode determination results.

[0025] The secondary storage device 104 is a rewritable secondary storage device such as a hard disk drive and a flash memory. Various information stored in the secondary storage device 104 is transferred to the RAM 103, and the CPU 101 can execute the program according to the present exemplary embodiment. The camera device 105 includes a camera lens, a camera sensor such as a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor, and a video image signal processing unit.

[0026] The input device 106 is a device for receiving input from a user, and is, for example, a keyboard and a mouse.

[0027] The display device 107 is a device for displaying processing results and the like to the user, and is, for example, a liquid crystal display. The network I / F 108 is a modem or a local area network (LAN) for connecting to a network such as the Internet and an intranet. The bus 109 is an internal bus for performing data input and output between the above-mentioned hardware components by connecting them.

[0028] The determination method of the action and the posture in the first exemplary embodiment will be described in detail with reference to the drawings.

[0029] Figure 2 2 is a block diagram showing a functional configuration of the video image processing apparatus 100. The video image processing apparatus 100 according to the present exemplary embodiment determines whether a person recognized from a video image is in an unstable posture. The video image processing apparatus 100 includes a video image acquisition unit 201, a person detection unit 202, a skeleton determination unit 203, a posture type determination unit 204, a skeleton information storage unit 205, a gravity direction determination unit 206, an unstable posture determination unit 207, and a display unit 208.

[0030] The video image acquisition unit 201 acquires a video image using the camera 105. In the present exemplary embodiment, the description is focused on an example in which the video image processing apparatus 100 includes the camera 105, but the video image processing apparatus 100 may not include the camera 105. In this case, the video image acquisition unit 201 of the video image processing apparatus 100 acquires a video image via a wired or wireless communication medium.

[0031] The person detection unit 202 detects the region of a person from the video image acquired by the video image acquisition unit 201. The person detection unit 202 according to the present exemplary embodiment detects the entire body region of a person.

[0032] The skeleton determination unit 203 performs position determination processing for determining the specific position of the person based on the entire body region detected by the person detection unit 202. Figure 3 shown.

[0033] like Figure 3 As shown, the specific positions of this embodiment are shoulders 301 and 302, elbows 303 and 304, wrists 305 and 306, hips 307 and 308, knees 309 and 310, ankles 311 and 312, eyes 313 and 314, nose 317, and ears 315 and 316.

[0034] In other words, the specific positions according to the present exemplary embodiment include the positions of joints and organs of a character.

[0035] The posture type determination unit 204 determines the posture type (e.g., standing posture or sitting posture) of the character based on the specific position determined (estimated) by the skeleton determination unit 203. The skeleton information storage unit 205 stores skeleton information in which information about the specific position determined (estimated) by the skeleton determination unit 203 is associated with the posture type determined by the posture type determination unit 204. The skeleton information storage unit 205 is implemented by the RAM 103 or the secondary storage device 104.

[0036] The gravity direction determination unit 206 performs gravity determination processing for determining the gravity direction in at least a partial region in the video image based on the skeleton information stored in the skeleton information storage unit 205. The method of determining (estimating) the gravity direction by the gravity direction determination unit 206 will be described below.

[0037] The unstable posture determination unit 207 determines whether the posture of the character is unstable based on the gravity direction determined by the gravity direction determination unit 206 and the skeleton information stored in the skeleton information storage unit 205. The display unit 208 displays the determination result of the unstable posture determination unit 207. The display unit 208 is implemented by the display device 107.

[0038] Next, details of processing performed by the video image processing apparatus 100 according to the first exemplary embodiment will be described. Figure 4 : is a flowchart illustrating a process of determining (estimating) the direction of gravity performed by the video image processing device 100 according to the present exemplary embodiment. Figure 4 Each operation shown is realized by the CPU 101 of the video image processing apparatus 100 reading a required program from the ROM 102 into the RAM 103 and executing the program. In addition, when the user issues an instruction to start the gravity direction determination, the start Figure 4 However, for example, when a predetermined condition is satisfied, Figure 4 The processing in can start automatically.

[0039] when Figure 4 When the processing in is started, in step S401, the video image acquisition unit 201 acquires a video image shot in a store.

[0040] Furthermore, in step S401 , the video image acquisition unit 201 associates time information (time stamp or frame identification (ID)) with each image frame constituting the video image.

[0041] In step S402, the person detection unit 202 detects a region of a person (person region) from the frame image. The person detection unit 202 according to the present exemplary embodiment detects the entire body region including the entire body of the person as the person region. Examples of methods by which the person detection unit 202 detects a person region include methods using a convolutional neural network (CNN), as discussed in "J. Redmon "You Only Look Once: Unified, Real Time Object detection", CVPR 2015". However, the method is not limited to this method, and any other method may be used as long as a person region can be detected.

[0042] In this exemplary embodiment, the character region is rectangular, and the position of the character region is represented by the x-coordinate and y-coordinate of two points, one of which is located at the upper left position and the other is located at the lower right position in a coordinate system with the upper left position of the frame image as the origin. In addition, the time information of the frame image is assigned to each character region.

[0043] In step S403, the skeleton determination unit 203 determines the specific position of the character with respect to all character areas in the frame image. Then, the skeleton determination unit 203 generates information (joint point list) listing the specific position of each character area. Examples of skeleton determination (estimation) methods include a method of estimating the coordinates of each joint point using CNN and obtaining its reliability as described in "Z.Cao, "OpenPose: Real-time Multi-Person 2D Poseestimation using Part Affinity Fields", CVPR2018". However, this method is not limited to this method, and any other method may be used as long as the coordinates of a specific position can be determined.

[0044] In this exemplary embodiment, a joint point list is generated for each character region included in the frame image. In addition, each joint point list is a list of time information of the frame image, coordinates of all specific positions of the character, and reliability arranged in a specific order.

[0045] In the present exemplary embodiment, description is focused on the case where skeleton determination is performed on each person region detected from the entire frame image, but skeleton determination may be performed on the entire frame image and may be organized for each person based on the relationship between specific positions.

[0046] In step S404, the posture type determination unit 204 determines the posture type (e.g., standing or sitting) of the character based on the above-mentioned joint point list. An example of a method for determining the posture type includes using a large number of joint point lists prepared for each posture type as training data to train a CNN or a multi-layer perceptron (MLP) as a method for classification recognition problems. Using this method, among the possibilities obtained for each posture type (classification), the posture type corresponding to the maximum possibility is adopted as the posture type of the character.

[0047] In step S405, the skeleton information storage unit 205 stores a list in which the posture type ID is added to the joint point list as the skeleton information. The posture type ID is an ID preset for each posture type, for example, 1 for standing posture and 2 for sitting posture.

[0048] The video image processing device 100 according to the present exemplary embodiment determines whether a predetermined time has passed. In the case where the predetermined time has passed ("Yes" in step S406), the processing proceeds to step S407. On the other hand, in the case where the predetermined time has not passed ("No" in step S406), the processing returns to step S401 to repeat the processing from steps S401 to S405 until the predetermined time has passed. It is assumed that a person appears at various positions in the camera range while the predetermined time has passed, and the gravity direction at each position in the camera range can be acquired based on the information acquired during the predetermined time. However, the condition for repeating the steps from steps S401 to S405 is not limited to the predetermined time. For example, the condition that the number of people detected in the video image reaches a predetermined number, or the condition that a person is detected at all positions in the frame image can be another condition.

[0049] Through the processing in steps S407 to S409, the gravity direction is determined (estimated) in at least a partial area in the video image. The video image processing apparatus 100 according to the present exemplary embodiment indicates the direction in which gravity acts at each position on the floor surface or ground surface in the video image using a unit vector. Figure 5 An example thereof is shown. Screen 501 shows the entire image, and a person 502 stands in a room. Floor surface 503 is covered with an arrow indicating the direction of gravity represented by arrow 504. In the present exemplary embodiment, the gravity direction is associated with each of divided areas obtained by dividing a specific area that is a floor surface or a ground surface in an image frame into areas each having a predetermined size (10 pixels x 10 pixels in the vertical and horizontal directions), and the divided areas associated with one gravity direction are referred to as gravity vector fields. However, the division method and the size of the division are not limited to the above method and size.

[0050] In step S407, the gravity direction determination unit 206 reads the skeleton information corresponding to the predetermined time period from the skeleton information storage unit 205. In step S408, the gravity direction determination unit 206 selects a direction for determining the gravity direction from the joint point list included in the skeleton information.

[0051] Reference Figure 6 The details of the processing performed in step S408 are described. Figure 6 An example of an image 601 including a standing person 602 and a sitting person 603 is shown. The black dots on each person represent Figure 3 The specific position shown. The character 603 is sitting on a stool 604. A process of selecting a direction for determining the direction of gravity for each of two postures, the standing posture and the sitting posture, will be described.

[0052] The gravity direction determination unit 206 selects a direction 611 from the midpoint 607 between the hips 605 and 606 to the midpoint 610 between the ankles 608 and 609 for the character 602 in a standing posture (posture type ID=1). In the case of a standing posture, the head and upper body of the character may be tilted, but the positions of the hips 605 and 606 and the ankles 608 and 609 generally take a stable position to support the body of the character and achieve balance with respect to the gravity direction. Therefore, the gravity direction determination unit 206 according to the present exemplary embodiment selects a direction 611 from the midpoint 607 between the hips 605 and 606 to the midpoint 610 between the ankles 608 and 609. However, this method is merely an example and is not limited thereto, as long as the gravity direction in the standing posture can be accurately acquired.

[0053] On the other hand, the gravity direction determination unit 206 selects a direction 618 from a midpoint 614 between the eyes 612 and 613 to a midpoint 617 between the hips 615 and 616 for the character 603 in a sitting position (posture type ID=2). In the case of a sitting position, since the character 603 supports the upper body with the hips 615 and 616 to achieve balance with respect to the gravity direction, the hips 615 and 616 of the character are usually in a stable position. Therefore, the gravity direction determination unit 206 according to the present exemplary embodiment selects a direction 618 from the midpoint 614 between the eyes 612 and 613 to the midpoint 617 between the hips 615 and 616. As described above, the gravity direction determination unit 206 according to the present exemplary embodiment determines (estimates) the gravity direction using different methods depending on whether the character is in a standing position or a sitting position. However, the method of selecting the gravity direction of the character 603 in a sitting position is not limited to this. For example, the center position between the eyes 612 and 613 and the nose may be used instead of the midpoint 614 between the eyes 612 and 613. Furthermore, the center position between the hips 615 and 616 and the two knees may be used instead of the midpoint 617 between the hips 615 and 616. When the upper body of the character tends to lean in a sitting position, the gravity direction can be obtained with high accuracy by using a sufficient number of vectors.

[0054] By the above method, in step S408, the gravity direction determination unit 206 determines the gravity direction based on the posture type of the character. Figure 7 As shown, when a chair 702 is placed in a screen 701, the gravity direction 703 can be determined not only on the floor surface but also on the surface of the chair.

[0055] In step S409, the gravity direction determination unit 206 determines the gravity vector field based on the gravity direction acquired in step S408. As described above, in the present exemplary embodiment, the gravity direction is associated with each of the divided areas obtained by dividing a specific area in the image frame (e.g., the floor surface, the ground surface, and the stool (chair) surface) into areas each having a predetermined size (10 pixels x 10 pixels in the vertical and horizontal directions). More specifically, the gravity direction determination unit 206 determines the average value of all gravity directions within a predetermined range from each divided area as the gravity direction of the divided area.

[0056] The gravity direction determination unit 206 according to the present exemplary embodiment determines the gravity direction based on the organ point corresponding to the posture type of the character, but is not limited thereto. For example, there is a method of determining the depth information and semantic information in each pixel from an image using a visual transformer (ViT), as described in "Rene Ranftl, et al. "Vision Transformers for Dense Prediction" arXiv:2103.13413". In this method, the gravity direction relative to the floor surface can be obtained based on the determined relationship between the floor surface and the wall.

[0057] The user may also input the gravity direction via the input device 106 . Figure 8 An example of a user interface is shown. Screen 801 shows the entire screen, and the user can specify the floor surface by operating a mouse, etc. A shaded area 802 is an example of a specified floor surface. For example, the user can set the gravity direction at each position on the floor surface by specifying a position on screen 801 with a mouse and inputting an angle using a keyboard to visually confirm the inclination of vector symbol 803 used as a guide while inputting the vector symbol 803.

[0058] Next, details of processing when the video image processing apparatus 100 according to the first exemplary embodiment is in operation will be described. Fig. 9 : is a flowchart illustrating a process of determining (estimating) a posture of a person performed by the video image processing device 100 according to the present exemplary embodiment. Fig. 9 The processing in each step shown in is realized by the CPU 101 of the video image processing apparatus 100 reading a necessary program from the ROM 102 into the RAM 103 and executing the program. In addition, when the user issues an instruction to perform posture determination, the process starts. Fig. 9 However, when, for example, a predetermined condition is met, Fig. 9 The processing in can start automatically.

[0059] In step S901, as in step S401, the video image acquisition unit 201 acquires a video image. In step S902, as in step S402, the person detection unit 202 detects a region of a person (person region) from a frame image constituting the video image. In step S903, as in step S403, the skeleton determination unit 203 determines a specific position (position of a joint, etc.) in the frame image. In step S904, as in step S404, the posture type determination unit 204 determines the posture type (e.g., standing posture or sitting posture) of the person based on the determination result of the specific position of the person.

[0060] In step S905, the unstable posture determination unit 207 determines a reference point for determining whether the posture of the person is unstable. In step S906, the unstable posture determination unit 207 determines the body support area of ​​the person.

[0061] In the present exemplary embodiment, the reference point is a specific position selected based on the posture type of the person, and the unstable posture determination unit 207 determines whether the specific position is supported. In addition, in the present exemplary embodiment, the body support area of ​​the person is an area for determining whether the reference point is supported, and is determined based on the posture of the person. The unstable posture determination unit 207 according to the present exemplary embodiment determines the reference point and the body support area of ​​the person based on the specific position corresponding to the posture type of the person by a method similar to the method described in step S408.

[0062] Fig.10 A state is shown in which one person 1001 is actually standing and the other person 1002 is actually about to fall down, among two people determined to be in a standing position based on a specific position. For the people 1001 and 1002 determined to be in a standing position, the unstable posture determination unit 207 according to the present exemplary embodiment determines the midpoint between the hips 1003 and 1004 as the reference point 1005, and the midpoint between the hips 1011 and 1012 as the reference point 1013, respectively. As described above, this is because, in the case of a standing position, the hips and ankles are generally used as stable positions relative to the direction of gravity compared to the head and shoulders. In addition, the unstable posture determination unit 207 according to the present exemplary embodiment determines the area 1008 including the ankles 1006 and 1007 as the body support area of ​​the person, and determines the area 1016 including the ankles 1014 and 1015 as the body support area of ​​the person of the people 1001 and 1002 determined to be in a standing position, respectively. The body support area of ​​the character according to the present exemplary embodiment is a circle centered at the midpoint between the ankles and having a diameter that is a distance obtained by adding a margin of a predetermined ratio to the distance between the ankles.

[0063] The method for determining the reference point and the body support area of ​​the character is not limited to the above method. For example, in addition to the hip, the reference point can also be determined by including other joint points (such as shoulders). In addition, in addition to the ankle, the body support area of ​​the character can also be determined by including other joint points (such as knees). The body support area of ​​the character is not necessarily circular, but can also be elliptical or rectangular.

[0064] Fig.11 A state is shown in which one person 1101 is actually sitting and the other person 1102 is actually falling backwards, out of two persons determined to be in a sitting position based on a specific position. For the persons 1101 and 1102 determined to be in a sitting position, the unstable posture determination unit 207 according to the present exemplary embodiment determines the midpoint between the eyes 1103 and 1104 as the reference point 1105, and the midpoint between the eyes 1113 and 1114 as the reference point 1115, respectively. As described above, this is because, in the case of a sitting position, the hip is generally used as a stable posture. In addition, the unstable posture determination unit 207 according to the present exemplary embodiment determines the area 1110 including the hips 1106 and 1107 and the knees 1108 and 1109 as the body support area of ​​the person, and determines the area 1120 including the hips 1116 and 1117 and the knees 1118 and 1119 as the body support area of ​​the person determined to be in a sitting position 1101 and 1102. The body support area of ​​the character according to the present exemplary embodiment is a circle centered at the midpoint between the ankle and the hip and having a diameter that is a distance obtained by adding a margin of a predetermined ratio to the distance between the hip and the knee.

[0065] The method for determining the reference point and the body support area of ​​the character is not limited to the above method. For example, the reference point can be determined by including the position of other organs (such as the nose) other than the eyes or the position of other joint points. In addition, the body support area of ​​the character can be determined using only the buttocks. The body support area of ​​the character does not necessarily have to be circular, and can be elliptical or rectangular, and the shape of the body support area of ​​the character can be different between standing and sitting.

[0066] In step S907, the unstable posture determination unit 207 determines whether the posture of the target person is unstable based on the positional relationship between the reference point determined in step S905 and the body support area of ​​the person determined in step S906. More specifically, the unstable posture determination unit 207 determines whether the target person is in an unstable posture based on the determination results of multiple specific positions of the person by the skeleton determination unit 203 and the determination results of the gravity direction by the gravity direction determination unit 206. The unstable posture determination unit 207 according to the present exemplary embodiment determines whether the target person is in an unstable posture based on whether a line segment extending from the reference point in the gravity direction reaches the body support area of ​​the person.

[0067] Reference Fig.10 A case where a person is determined to be in a standing posture is described. The unstable posture determination unit 207 determines that the person 1001 is in a stable posture because a line segment 1010 extending from a reference point 1005 of the person 1001 along a gravity direction 1009 falls within a body support region 1008 of the person. On the other hand, the unstable posture determination unit 207 determines that the person 1002 is in an unstable posture because a line segment 1018 extending from a reference point 1013 of the person 1002 along a gravity direction 1017 falls outside a body support region 1016 of the person. More specifically, in a case where the posture type of the person is determined to be a standing posture, the unstable posture determination unit 207 according to the present exemplary embodiment determines whether the target person is in an unstable posture based on a reference point corresponding to the position of the hip of the target person. In addition, in a case where a line segment extending from the reference point 1005 along a gravity direction 1009 does not fall within a predetermined range (body support region 1008 of the person) determined based on a plurality of specific positions corresponding to the target person, the unstable posture determination unit 207 determines that the target person is in an unstable posture.

[0068] Reference Fig.11A case where the posture type determination unit 204 determines that the person is in a sitting posture is described. The unstable posture determination unit 207 determines that the person 1101 is in a stable posture because the line segment 1112 extending from the reference point 1105 of the person 1101 along the gravity direction 1111 falls within the body support area 1110 of the person. On the other hand, the unstable posture determination unit 207 determines that the person 1102 is in an unstable posture because the line segment 1122 extending from the reference point 1115 along the gravity direction 1121 falls outside the body support area 1120 of the person. In this way, in the case where the posture type determination unit 204 determines that the posture type of the person is a sitting posture, the unstable posture determination unit 207 according to the present exemplary embodiment determines whether the target person is in an unstable posture based on the reference point corresponding to the eye position of the person. In addition, in the case where the line segment 1122 extending from the reference point 1115 along the gravity direction 1121 does not fall within the predetermined range (body support area 1110 of the person) determined based on a plurality of specific positions corresponding to the target person, the unstable posture determination unit 207 determines that the target person is in an unstable posture.

[0069] like Fig.12 As shown, in the case where the target person is a person who uses a body support device 1202 (such as a cane) to support the person's body, the unstable posture determination unit 207 detects the body support device 1202 and determines the body support area 1206 of the person to include the tip position of the body support device 1202 of the person based on the detection result. The unstable posture determination unit 207 detects the body support device 1202 of the person using the method described in, for example, "J. Redmon, "You Only Look Once: Unified, Real Time Object Detection", CVPR 2015". The unstable posture determination unit 207 determines the unstable posture. Fig.12 Figure 1201 with body support device 1202 (cane) is in a stable posture because line segment 1205 extending from reference point 1203 in gravity direction 1204 falls within body support area 1206 of the figure. Fig.10 The posture of the character 1002 is shown and Fig.12 The postures of the characters 1201 shown are the same, but by taking into account the presence of the body support device 1202, it is possible to distinguish with greater accuracy between the character 1002 who is actually about to fall and the character 1201 who is not actually falling.

[0070] In addition, since the line segment extending from the reference point in the direction of gravity exists at a position farther from the body support area of ​​the character, the posture is considered to be more unstable. Therefore, the unstable posture determination unit 207 according to the present exemplary embodiment determines the instability (instability degree) of each character based on the distance between the line segment and the body support area of ​​the character. For example, when the distance between the line segment extending from the reference point in the direction of gravity and the boundary of the body support area of ​​the character is "d", and the distance from the reference point to the boundary of the body support area of ​​the character is "L", the instability degree U can be expressed by Formula 1.

[0071] U=d / L(1)

[0072] The instability degree U is represented by a real number from 0 to 1. When "d" is equal to "L", the character is in a completely lying state, so "1" represents the most unstable state. However, the calculation method of the instability degree is not limited to this. By calculating the instability degree in this way, the processing content can be switched based on the instability degree, for example, to issue an alarm only when a person whose instability degree is a predetermined threshold or more is detected. However, the calculation of the instability degree is not indispensable.

[0073] In step S908, the display unit 208 displays the determination result of the unstable posture determination unit 207. For example, a message "There is an unstable posture of a person" may be displayed, or a rectangle surrounding the person with an unstable posture may be superimposed on the camera video image. In the case where the degree of instability of the posture of the person is calculated, a numerical value indicating the degree of instability or a color / bar graph corresponding to the degree of instability may be displayed. The video image processing apparatus 100 according to the present exemplary embodiment repeats Fig. 9 The process continues until the user issues an end instruction.

[0074] As described above, the video image processing device 100 according to this exemplary embodiment can determine whether the posture of the target person is unstable based on the relationship between the reference point and the body support area of ​​the person determined based on the specific position of the person. In addition, the reference point and the body support area of ​​the person are set by different methods according to the type of posture.

[0075] By using reference Figure 4 The gravity vector field described can accurately detect people with unstable postures even if the downward direction in the picture is different from the gravity direction due to the camera orientation or objects placed in the camera range (such as stools or chairs).

[0076] Next, a second exemplary embodiment of the present invention will be described focusing on the differences from the first exemplary embodiment. Fig.131 is a block diagram showing a functional configuration of a video image processing apparatus 100 according to a second exemplary embodiment. The video image processing apparatus 100 according to the second exemplary embodiment includes a person tracking unit 1303 so as to be able to process a temporal change in a result of determining (estimating) a person's posture. The video image acquisition unit 1301 and the person detection unit 1302 have functions similar to those of the video image acquisition unit 201 and the person detection unit 202, respectively.

[0077] The person tracking unit 1303 associates the person regions acquired by the person detecting unit 1302 in the temporally continuous image frames and assigns the same person ID. The skeleton determining unit 1304, the posture type determining unit 1305, and the skeleton information storage unit 1306 respectively have functions similar to those of the skeleton determining unit 203, the posture type determining unit 204, and the skeleton information saving unit 205. In addition, the gravity direction determining unit 1307 and the unstable posture determining unit 1308 respectively have functions similar to those of the gravity direction determining unit 206 and the unstable posture determining unit 207.

[0078] The unstable posture type determination unit 1309 determines the type of the unstable posture based on the temporal change in the output of the unstable posture determination unit 1308. The display unit 1310 has a function similar to that of the display unit 208. In addition, the determination process of the gravity direction in the second exemplary embodiment is similar to that of the first exemplary embodiment, and thus a description thereof is omitted.

[0079] Next, we will refer to Fig.14 Details of processing performed when the video image processing apparatus 100 according to the second exemplary embodiment is in operation are described. Fig.14 : is a flowchart illustrating a process of determining (estimating) a posture of a person performed by the video image processing device 100 according to the present exemplary embodiment. Fig.14 The processing in each step shown in is realized by the CPU 101 of the video image processing apparatus 100 reading a necessary program from the ROM 102 into the RAM 103 and executing the program. In addition, when the user issues an instruction to perform posture determination, the process starts. Fig.14 However, for example, when a predetermined condition is met, Fig.14 The processing in steps S1401 and S1402 is similar to the processing performed in steps S901 and S902, respectively, and thus description thereof is omitted.

[0080] In step S1403, the person tracking unit 1303 determines whether the person region detected in the current frame corresponds to one or more person regions detected in the previous frame. The person tracking unit 1303 assigns the same person ID to the person regions determined to include the same person.

[0081] There are various methods for performing tracking processing, for example, there is a method of associating a human region in a frame in which the distance between the center position of a human region included in a previous frame and the center position of a human region included in a current frame is shortest. In addition to the above method, there are various methods of associating human regions in frames, such as a pattern matching method using a human region in a previous frame as a reference pattern.

[0082] The processing performed in steps S1404 , S1405 , S1406 , and S1407 are similar to the processing performed in steps S903 , S904 , S905 , and S906 , respectively, and thus descriptions thereof are omitted.

[0083] In step S1408, the unstable posture determination unit 1308 calculates the degree of instability of the character using a method similar to that in step S907. In addition, the unstable posture determination unit 1308 temporarily stores an instability degree information list in which the degree of instability is arranged in chronological order for each character (character ID) in the RAM 103. In step S1409, the unstable posture type determination unit 1309 reads the instability degree information list temporarily stored in the RAM 103, and determines the type of the unstable posture for each character ID.

[0084] Fig.15 Examples of graphs are shown, each of which has a horizontal axis and a vertical axis representing time and instability, respectively. Graph 1501 shows a state in which the degree of instability continues to be at a predetermined level or higher. This state is considered to represent a "swaying" action. In contrast, graph 1502 shows a state in which the degree of instability increases rapidly. This state is considered to represent a "falling" action. However, the method of determining the action is not limited to this, and the action determined based on the time change of the degree of instability is not limited to "swaying" and "falling".

[0085] In step S1410, the display unit 1310 displays the determination result of the unstable posture. For example, a message "There is a swaying person" may be displayed, or a rectangle having a color corresponding to the type of unstable posture and surrounding the person may be superimposed on the camera video image. The video image processing apparatus 100 according to the present exemplary embodiment repeats Fig.14 As described above, the video image processing apparatus 100 according to the present exemplary embodiment can determine the type of unstable posture based on the temporal change of the degree of instability.

[0086] <Other Exemplary Embodiments>

[0087] In the above exemplary embodiment, a description is given of a case where the display unit 1310 displays the determination result of the unstable posture determination unit 207. In addition, a video image captured when an unstable person is detected may be recorded, or meta information indicating the time when the unstable person is detected may be added. In this way, a video image captured when an unstable person is detected can be easily searched.

[0088] Furthermore, in the above exemplary embodiments, the description given focuses on an example of determining a body support area of ​​a person based on a plurality of specific positions of the person, but is not limited thereto. For example, an area obtained by adding a predetermined margin to one of the ankles of the person may be the body support area of ​​the person. In this way, even in a case where the person has turned to the side relative to the camera and only one of the person's legs is visible, it is possible to determine whether the person is in an unstable posture.

[0089] Furthermore, in the above exemplary embodiments, the video image processing apparatus 100 is provided with Figure 2 or Fig.13 The present invention is a description of examples of all functions shown, but is not limited thereto. For example, the processing performed by the person detection unit 202 to the processing performed by the unstable posture determination unit 207 can be performed on the cloud, and the results can be displayed on a display installed in a store or the like. According to the present invention, the actions of people can be accurately distinguished.

[0090] Other embodiments

[0091] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0092] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A video image processing device, comprising: An acquisition unit, configured to acquire a video image; a position determination unit configured to determine a plurality of specific positions of a person in the video image acquired by the acquisition unit; a gravity direction determining unit configured to determine the gravity direction in at least a portion of the area in the video image acquired by the acquisition unit; as well as The posture determination unit is configured to determine whether the person is in an unstable posture based on the determination results of the multiple specific positions of the person by the position determination unit and the determination result of the gravity direction by the gravity direction determination unit.

2. The device according to claim 1, wherein: The multiple specific positions include at least positions about one or more of the joints, head, eyes, nose and mouth of the character.

3. The device according to claim 1, in, identifying reference points of the person and a body support area of ​​the person based on the plurality of specific positions of the person, and Wherein, whether the character is in an unstable posture is determined based on the positional relationship between the reference point and the body support area.

4. The device according to claim 3, wherein: In a case where a line segment extending from the reference point of the character in the gravity direction does not fall within a predetermined range determined based on the plurality of specific positions corresponding to the character, the character is determined to be in an unstable posture.

5. The device according to claim 3, in, The one or more processors are further configured to determine a gesture type of the character based on the plurality of specific positions, and Wherein, whether the character is in an unstable posture is determined based on the reference point corresponding to the determined posture type.

6. The device according to claim 5, in, In a case where the posture type of the character is determined to be a standing posture, determining whether the character is in an unstable posture based on the reference point corresponding to the position of the hip of the character, and Wherein, when the posture type of the character is determined to be a sitting posture, whether the character is in an unstable posture is determined based on the reference point corresponding to the position of the eyes of the character.

7. The device according to claim 1, wherein: The gravity direction is determined for each of one or more regions in the video image.

8. The device according to claim 1, wherein: Based on the determination results of the multiple specific positions and the determination result of the gravity direction, the instability degree of the character is further determined.

9. The device according to claim 8, wherein: Based on the temporal variation of the instability degree of the character, the type of the unstable posture is further determined.

10. The apparatus according to claim 1, further comprising: a detection unit configured to detect a body support device from the video image, and Wherein, based on the determination results of the multiple specific positions of the character, the determination result of the gravity direction and the detection result of the body support device, it is determined whether the character is in an unstable posture.

11. The device according to claim 1, wherein: The gravity direction is determined based on a reference point corresponding to a posture of the person in the video image.

12. A posture determination method, comprising: Acquire video images; determining a plurality of specific locations of a person in the acquired video image; determining a direction of gravity in at least a portion of a region in the acquired video image; as well as Based on the determination results of the plurality of specific positions of the character and the determination result of the gravity direction, it is determined whether the character is in an unstable posture.

13. The method according to claim 12, wherein: A reference point of the character and a body support area of ​​the character are identified based on the plurality of specific positions of the character, and whether the character is in an unstable posture is determined based on a positional relationship between the reference point and the body support area.

14. The method according to claim 13, wherein: In a case where a line segment extending from the reference point of the character in the gravity direction does not fall within a predetermined range determined based on the plurality of specific positions corresponding to the character, the character is determined to be in an unstable posture.

15. A non-transitory computer-readable storage medium storing a program to cause a computer to: Get video images, determining a plurality of specific positions of a person in the acquired video image, determining a direction of gravity in at least a portion of a region in the acquired video image, and Based on the determination results of the plurality of specific positions of the character and the determination result of the gravity direction, it is determined whether the character is in an unstable posture.

Images