Posture abnormality detection system and posture abnormality detection program

Abstract

To reduce overlooked posture abnormalities as compared with a case where posture determination of a person is performed based only on coordinates of skeleton points.SOLUTION: A posture abnormality detection system 80 includes at least one processor. When determining that a posture of a person is not abnormal based on coordinates of skeleton points of the person extracted from the person area of an image captured by an image capturing device 20, the processor further performs posture determination of the person based on probability values of the skeleton points.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a posture abnormality detection system and a posture abnormality detection program. [Background technology]

[0002] The following Patent Document 1 describes a human detection device for reliably identifying a person who has fallen, regardless of the direction of the fall and the positional relationship with the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-33343 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to reduce posture anomalies that are overlooked, compared to when determining a person's posture based only on the coordinates of skeleton points. [Means for solving the problem]

[0005] The posture abnormality detection system of the first embodiment includes at least one processor, and when the processor determines that the posture of the person is not abnormal based on the coordinates of the person's skeleton points extracted from the person area of ​​an image captured by an imaging device, it further performs posture determination of the person based on a probability value of the skeleton points.

[0006] A second aspect of the posture abnormality detection system is the posture abnormality detection system of the first aspect, in which the processor, in determining the posture based on the probability values, determines that the posture is abnormal if at least one of the probability values ​​of the skeleton points indicating the positions of the person's left or right ears is below a threshold value.

[0007] A posture abnormality detection system of a third aspect is the posture abnormality detection system of the second aspect, which determines that the posture is abnormal when both of the probability values ​​are equal to or less than the threshold value.

[0008] A fourth aspect of the posture abnormality detection system is the posture abnormality detection system of the second or third aspect, wherein the processor determines that the posture is abnormal when the probability value is below a threshold value in a plurality of images taken within a predetermined period of time.

[0009] A fifth aspect of the posture abnormality detection system is the posture abnormality detection system of the first aspect, wherein the processor determines that a person's posture is abnormal when, in the posture determination based on the coordinates, the coordinate of a skeletal point indicating the position of the waist is located above the coordinate of a skeletal point indicating the position of the shoulder, and the coordinate of the skeletal point indicating the position of the shoulder is located above the coordinate of the skeletal point indicating the position of the head.

[0010] A sixth aspect of the posture abnormality detection system is the posture abnormality detection system of the first aspect, wherein, in the posture determination based on the coordinates, the processor determines that the person's posture is abnormal if both the angle of the thighs and the angle of the torso are tilted by more than a predetermined angle relative to a predetermined axial direction.

[0011] A seventh aspect of the posture abnormality detection system is the posture abnormality detection system of the first aspect, wherein the processor, in determining the posture based on the coordinates, determines that the person's posture is abnormal in at least one of the following cases: when the coordinate of the skeletal point indicating the position of the waist is located higher than the coordinate of the skeletal point indicating the position of the shoulder, and the coordinate of the skeletal point indicating the position of the shoulder is located higher than the coordinate of the skeletal point indicating the position of the head; and when both the angle of the thighs and the angle of the torso are tilted by more than a predetermined angle relative to a predetermined axial direction.

[0012] An eighth aspect of the posture abnormality detection system is the posture abnormality detection system of the seventh aspect, wherein the processor determines that a person's posture is abnormal in at least one of the following cases: in a plurality of images taken within a predetermined period, the coordinates of a skeletal point indicating the position of the waist are located higher than the coordinates of a skeletal point indicating the position of the shoulders, and the coordinates of the skeletal point indicating the position of the shoulders are located higher than the coordinates of the skeletal point indicating the position of the head; and in a plurality of images taken within a predetermined period, both the angle of the thighs and the angle of the torso are tilted by more than a predetermined angle relative to a predetermined axial direction.

[0013] A posture abnormality detection system of a ninth aspect is the posture abnormality detection system of the seventh or eighth aspect, wherein the processor detects, as the axial direction, a direction perpendicular to a floor or ground surface in the image.

[0014] A tenth aspect of the posture abnormality detection system is the posture abnormality detection system of the seventh or eighth aspect, in which the processor detects, as the axial direction, the trunk direction of the person when standing or sitting, extracted from the skeletal points.

[0015] An eleventh aspect of the posture abnormality detection system is the posture abnormality detection system of the tenth aspect, in which the processor derives the trunk direction from a plurality of images captured within a predetermined period of time.

[0016] The posture abnormality detection program of the twelfth aspect is capable of causing a computer to further perform posture determination of a person based on a probability value of a person's skeleton points when the computer determines that the person's posture is not abnormal based on the coordinates of the person's skeleton points extracted from the person area of ​​an image captured by an imaging device. Effect of the Invention

[0017] According to the posture abnormality detection system of the first aspect and the posture abnormality detection program of the twelfth aspect, it is possible to reduce overlooked posture abnormalities compared to when a person's posture is determined based only on the coordinates of skeleton points.

[0018] According to the posture abnormality detection system of the second aspect, when a person falls backward with respect to the imaging device, posture abnormality can be easily detected.

[0019] According to the posture abnormality detection system of the third aspect, it is possible to reduce false positives of abnormal postures compared to a case in which the posture is determined to be abnormal when at least one of the probability values ​​of the skeleton points is below a threshold value.

[0020] According to the posture abnormality detection system of the fourth aspect, the accuracy of posture abnormality detection is improved compared to a case where the presence or absence of posture abnormality is determined from a single image.

[0021] According to the posture abnormality detection system of the fifth aspect, when a person falls toward the image capturing device, posture abnormality can be easily detected.

[0022] According to the posture abnormality detection system of the sixth aspect, when a person falls at an angle equal to or greater than a preset angle with respect to a preset axial direction, posture abnormality can be easily detected.

[0023] According to the seventh aspect of the posture abnormality detection system, posture abnormality can be easily detected both when a person falls toward the imaging device and when a person falls at an angle equal to or greater than a predetermined angle relative to a predetermined axial direction.

[0024] According to the posture abnormality detection system of the eighth aspect, the accuracy of posture abnormality detection is improved compared to a case where the presence or absence of posture abnormality is determined from a single image.

[0025] According to the posture abnormality detection system of the ninth aspect, even if the imaging device is installed at an angle with respect to the floor or ground, it is difficult to erroneously detect a posture abnormality.

[0026] According to the posture abnormality detection system of the tenth aspect, posture abnormalities can be detected taking into account individual differences in usual posture.

[0027] According to the abnormal posture detection system of the eleventh aspect, the accuracy of detecting the trunk orientation is improved compared to the case where the trunk orientation is detected from a single image. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram showing an example of an overall configuration of a posture abnormality detection system and an electrical configuration of a posture abnormality detection device according to an embodiment. FIG. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a posture abnormality detection device in the posture abnormality detection system according to the embodiment. [Diagram 3] 1 is a conceptual diagram showing an example of a posture abnormality detected by a posture abnormality detection system according to an embodiment. FIG. [Figure 4A] 1 is a schematic diagram showing an example of an image captured by a photographing device and a person area within the image; [Figure 4B] 10 is a schematic diagram showing an example of human skeleton points extracted from a human region; FIG. [Figure 5A] 10 is a schematic diagram showing an example of a person area in an image in which a person has fallen; FIG. [Figure 5B] FIG. 13 is a schematic diagram showing an example of skeleton points extracted from a fallen person. [Figure 6A] FIG. 13 is a schematic diagram showing another example of a person region when a person has fallen in an image. [Figure 6B] FIG. 13 is a schematic diagram showing another example of skeleton points extracted from a fallen person. [Figure 7A] FIG. 2 is a schematic diagram showing an example of the direction of a person's trunk in an image. [Figure 7B] FIG. 13 is a schematic diagram showing another example of the direction of the trunk of a person in an image. [Figure 7C] 10 is a schematic diagram showing an example of a method for deriving a trunk direction from a skeleton point of a person; FIG. [Figure 8A] FIG. 13 is a schematic diagram showing another example of a person region when a person has fallen in an image. [Figure 8B] FIG. 13 is a schematic diagram showing another example of skeleton points extracted from a fallen person. [Figure 8C] FIG. 13 is a schematic diagram showing an example of a state in which a fallen person is being photographed by an imaging device. [Figure 9A] 1A to 1C are schematic diagrams showing examples of human postures to illustrate a method of determining posture abnormality. [Figure 9B] 1A to 1C are schematic diagrams showing examples of human postures to illustrate a method of determining posture abnormality. [Figure 9C] 1A to 1C are schematic diagrams showing examples of human postures to illustrate a method of determining posture abnormality. [Figure 9D] 1A to 1C are schematic diagrams showing examples of human postures to illustrate a method of determining posture abnormality. [Figure 10] 11 is a flowchart illustrating an example of a posture abnormality detection process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, a posture abnormality detection system and a posture abnormality detection program according to an embodiment of the present disclosure will be described with reference to the drawings. Components indicated by the same reference numerals in each drawing are the same components. However, unless otherwise specified in the specification, each component is not limited to one, and may be present in multiple numbers.

[0030] In addition, explanations of configurations and symbols that are duplicated in each drawing may be omitted. Note that the present disclosure is not limited to the following embodiments, and may be implemented by making appropriate modifications, such as omitting configurations, replacing them with different configurations, or combining one embodiment and various modified examples, within the scope of the purpose of the present disclosure.

[0031] <Posture abnormality detection system> 1 is a system constructed in a posture abnormality detection device 10, and is used in combination with an imaging device 20 and a notification device 30. Note that the term "system" in this disclosure includes any system constructed with multiple devices, a system constructed with a single device, or a system constructed within a higher-level device or system.

[0032] That is, the posture abnormality detection system 80 of the present disclosure is not limited to being constructed only in the posture abnormality detection device 10, but may be constructed across a plurality of devices other than the posture abnormality detection device 10. For example, the posture abnormality detection system 80 may include the imaging device 20 and the notification device 30, or may be constructed across a terminal such as a personal computer externally attached to the posture abnormality detection device 10.

[0033] In addition, the various processes performed by the posture abnormality detection system 80 can also be executed on the cloud, an on-premise server, an edge server, an end point, etc.

[0034] The posture abnormality detection system 80 is used to detect posture abnormalities of a person (hereinafter referred to as the person being observed) in, for example, a room in a hospital, a welfare facility for the elderly, a house, etc. The posture abnormality detection system 80 is also used to notify others when it detects an abnormality in the posture of the person being observed. "Others" include staff stationed at a staff station in a hospital, staff stationed in a care staff room or nursing staff room in a welfare facility for the elderly, residents of a house, etc. "Posture abnormalities" include a state of collapse, a state of sitting with head down, etc.

[0035] The posture abnormality detection system 80 may be used to detect posture abnormalities of an observed person in rooms or corridors other than living rooms, or in various buildings other than hospitals, elderly welfare facilities, and houses. Furthermore, the posture abnormality detection system 80 may be used to detect posture abnormalities of an observed person in outdoor spaces other than buildings.

[0036] (Photographing device) The photographing device 20 is, for example, an RGB camera. The photographing device 20 is installed in an area where the observed person is present, for example, a room in a building. When the photographing device 20 is installed in a room, it is preferable that the photographing device 20 is equipped with a lens that can capture within the angle of view the entire range in which the observed person can move within the room. Furthermore, the photographing device 20 is preferably one that can capture video, but it is also possible to use one that captures still images at predetermined intervals. Furthermore, the photographing device 20 can also be a mobile terminal (for example, a smartphone) with a photographing function.

[0037] The photographing device 20 transmits the photographed image to the posture abnormality detection device 10 in response to an instruction from the control unit 11F (see FIG. 2) in the posture abnormality detection device 10. Furthermore, the photographing device 20 can also transmit the photographed image to the posture abnormality detection device 10 at regular time intervals, regardless of the presence or absence of an instruction from the control unit 11F.

[0038] (Alarm device) The notification device 30 is a device for notifying others when an abnormal posture of the observed person is detected. As the notification device 30, an audio device such as a speaker that emits a buzzer sound or an announcement, a light-emitting device such as a pilot lamp that emits light, a display device that outputs images, characters, symbols, etc. on a screen, etc. can be used as the notification means. These audio devices, light-emitting devices, and display devices may be used alone or in combination.

[0039] The notification device 30 is installed in a staff station in a hospital, a care worker room or a nursing staff room in a welfare facility for the elderly, each room in a house (e.g., a living room), etc. Alternatively, the notification device 30 is installed in a disaster prevention center or a manager's room in a building, etc. Alternatively, the notification device 30 may be constructed as an application in a mobile terminal (e.g., a smartphone) owned by a person other than the person being observed.

[0040] The notification device 30, in response to an instruction from the control unit 11F in the posture abnormality detection device 10, notifies the user that there is an abnormality in the posture of the person being observed.

[0041] (Posture abnormality detection device) The posture abnormality detection device 10 judges the presence or absence of posture abnormality of the observed person from the image captured by the imaging device 20. Furthermore, when the posture abnormality of the observed person is detected, the posture abnormality detection device 10 controls the notification device 30 to notify the fact that there is an abnormality in the posture of the observed person.

[0042] Although details will be described later, for example, the posture abnormality detection device 10 can detect a posture abnormality when the observed person falls forward with respect to the shooting direction of the image capturing device 20, as shown as Case 1 in Fig. 3. Similarly, the posture abnormality detection device 10 can detect a posture abnormality when the observed person falls sideways with respect to the shooting direction of the image capturing device 20, as shown as Case 2. Similarly, the posture abnormality detection device 10 can detect a posture abnormality when the observed person falls backward with respect to the shooting direction of the image capturing device 20, as shown as Case 3.

[0043] (Electrical configuration of posture abnormality detection device) As shown in FIG. 1, the posture abnormality detection device 10 includes a CPU (Central Processing Unit: processor) 11, a memory 12 as a temporary storage area, a non-volatile storage unit 13, an input unit 14, a display unit 15 such as a liquid crystal display, a medium read / write device (R / W) 16, a communication interface (I / F) unit 18, and an external I / F unit 19. The CPU 11, the memory 12, the storage unit 13, the input unit 14, the display unit 15, the medium read / write device 16, the communication I / F unit 18, and the external I / F unit 19 are connected to each other via a bus B1. The medium read / write device 16 reads information written in a recording medium 17 and writes information to the recording medium 17. The communication I / F unit 18 is an interface for communicatively connecting the posture abnormality detection device 10 to, for example, a server provided outside the posture abnormality detection device 10 or various terminals (such as personal computers, smartphones, and tablets) used by users. The communication I / F unit 18 uses communication standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), and LAN (Local Area Network).

[0044] (Storage part) The storage unit 13 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. A posture abnormality detection program 13A is stored in the storage unit 13 as a storage medium. The posture abnormality detection program 13A is stored in the storage unit 13 by setting a recording medium 17 in which the posture abnormality detection program 13A is written to the medium reading and writing device 16, and the medium reading and writing device 16 reading the posture abnormality detection program 13A from the recording medium 17. The CPU 11 reads the posture abnormality detection program 13A from the storage unit 13, expands it in the memory 12, and sequentially executes the processes of the posture abnormality detection program 13A. An image database 13B, which will be described later, is stored in this storage unit 13.

[0045] (Input section) An operation for starting and ending the posture abnormality detection program 13A is performed by a user via the input unit 14. The user is, for example, an administrator of the posture abnormality detection system 80.

[0046] (Display) Information for starting and ending the posture abnormality detection program 13A is displayed on the display unit 15. The display unit 15 may be formed in the above-mentioned notification device 30. The display unit 15 of the posture abnormality detection device 10 may also function as the display device in the notification device 30.

[0047] (Functional configuration of posture abnormality detection device) Next, the functional configuration of the posture abnormality detection device 10 according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the posture abnormality detection device 10 includes an image acquisition unit 11A, a skeleton point extraction unit 11B, an axial direction detection unit 11C, a coordinate determination unit 11D, a probability value determination unit 11E, and a control unit 11F. The CPU 11 of the posture abnormality detection device 10 executes a posture abnormality detection program 13A to function as the image acquisition unit 11A, the skeleton point extraction unit 11B, the axial direction detection unit 11C, the coordinate determination unit 11D, the probability value determination unit 11E, and the control unit 11F.

[0048] (Control unit) The control unit 11F can control the photographing device 20 to photograph an image, and can also control the notification device 30 to execute the various notifications described above.

[0049] (Image acquisition section) The image acquisition unit 11A acquires an image captured by the photographing device 20.

[0050] (Skeleton point extraction part) The skeleton point extraction unit 11B defines a person area from the image acquired by the image acquisition unit 11A, and also extracts coordinates of skeleton points of the person from the defined person area.

[0051] Specifically, the skeleton point extraction unit 11B defines a person area PE from the image 20A acquired by the image acquisition unit 11A, as shown in Fig. 4A. Then, the skeleton point extraction unit 11B extracts coordinates of skeleton points P0 to P16 of the observed person from the defined person area PE, as shown in Fig. 4B.

[0052] The coordinates are extracted as coordinates (X coordinate and Y coordinate) in a two-dimensional coordinate system with the horizontal direction of the image 20A as the X axis and the vertical direction as the Y axis. The origin of the coordinates can be set anywhere, but in FIG. 4A, the origin is the lower left corner of the image 20A. Each skeleton point P0 to P16 indicates each part of the person, and the correspondence relationship is as follows:

[0053] P0: Back of the head P1: Left eye P2: Right eye P3: Left ear P4: Right ear P5:Left shoulder P6: Right shoulder P7: left elbow P8: Right elbow P9: Left wrist P10: Right wrist P11: Left hip P12: Right hip P13:Left knee P14: Right knee P15:Left ankle P16: Right ankle

[0054] As an example, the person area PE is acquired from a trained neural network (not shown) that has performed machine learning for identifying the person area PE by inputting the image 20A to the trained neural network.

[0055] In machine learning, a combination of an image of a person and a person area PE is used as training data to generate a trained neural network that outputs a person area PE when an image of a person is input.

[0056] Similarly, as an example, skeleton points P0 to P16 are obtained from a trained neural network (not shown) that has performed machine learning to identify skeleton points P0 to P16 by inputting an image contained in the defined person area PE into the neural network.

[0057] In machine learning, a combination of an image contained in the person area PE in an image of a person and the person's skeletal points P0 to P16 is used as training data to generate a trained neural network that outputs the person's skeletal points when an image of a person is input.

[0058] In addition, it is preferable to generate a daytime model using daytime images as training data and a nighttime model using nighttime images as training data for the neural network.

[0059] When outputting skeleton points P0 to P16 from the neural network, the skeleton point extraction unit 11B outputs a probability value for each skeleton point in addition to the X and Y coordinates. The "probability value" is the probability that the coordinates of the detected skeleton point are accurate, in other words, the probability that the skeleton point whose coordinates are detected is the skeleton point in question. For example, the "probability value of the right ear" is the "probability that the detected object is the right ear."

[0060] The skeleton point extraction unit 11B extracts the coordinates of each of the skeleton points P0 to P16 from the image 20A. For example, even if the right ear is not captured in the image 20A, the skeleton point extraction unit 11B extracts the coordinates of the skeleton point P4 of the right ear. However, since the image acquisition unit 11A also extracts a probability value, it is possible to suggest the accuracy of the coordinates (the probability that the skeleton point P4 is the right ear).

[0061] The above skeleton points P0 to P16 are just examples. In the present disclosure, the positions of the left buttock, right buttock, center of the waist, spine, neck, nose, back of the head, navel, etc. may be extracted as skeleton points, or any of the above skeleton points may be omitted.

[0062] (Axial detection section) The axial direction detector 11C detects an axial direction that is used when a coordinate determiner 11D (to be described later) determines whether or not the posture of the person to be observed is abnormal.

[0063] In addition, the "predetermined axis direction" in the present disclosure refers to the axis direction set at the time when the coordinate determination unit 11D determines whether the posture of the observed person is abnormal or not. In other words, this axis direction also includes the axis direction detected by the axis detection unit 11C immediately before the coordinate determination unit 11D determines whether the posture of the observed person is abnormal or not.

[0064] As an example of the axial direction, the axial direction detector 11C detects a horizontal plane along which the floor or ground in the image 20A runs, as shown in FIG. 6A. The axial direction detector 11C also detects a reference line SL perpendicular to a horizontal line GL included in the horizontal plane as the axial direction. By using such a reference line SL, it is difficult to erroneously detect an abnormal posture even if the image capture device 20 is installed at an angle.

[0065] As another example, the axial direction is the "trunk direction" of the observed person when he / she is standing or sitting. As shown in FIG. 7C, the axial direction detection unit 11C detects a reference line SL along the trunk direction of the observed person using skeleton points P5 (left shoulder), P6 (right shoulder), P11 (left buttock), and P12 (right buttock) from among the extracted skeleton points P0 to P16. In this case, the reference line SL is a straight line passing through the midpoint of a line segment L5 connecting the skeleton points P5 and P6 and the midpoint of a line segment L6 connecting the skeleton points P11 and P12. By using such a reference line SL, it is possible to detect posture abnormalities by taking into account individual differences in normal posture for each observed person.

[0066] When detecting the trunk direction as the axial direction, the axial direction detector 11C can detect a reference line SL in one image 20A as the axial direction. Alternatively, the axial direction detector 11C may detect the reference line SL from "multiple" images 20A as shown in Figs. 7A and 7B. In this case, the axial direction detector 11C detects the reference line SL from each of the multiple images 20A captured within a predetermined period. Then, for example, a direction along an average angle of the multiple reference lines SL (for example, an angle in the image 20A) is derived as the axial direction.

[0067] When the axial direction detector 11C uses a plurality of images 20A, the images 20A are stored in the image database 13B. By deriving the axial direction using a plurality of images 20A in this manner, the accuracy of detecting the trunk direction is improved compared to the case where the trunk direction is detected from a single image.

[0068] The axial direction detector 11C may be capable of using either the "vertical direction" or the "trunk direction" of the subject as the axial direction, or may be capable of using both. When both are capable of being used, the user can arbitrarily select either one by inputting through the input unit 14, etc.

[0069] (Coordinate determination section) The coordinate determination unit 11D determines whether or not the posture of the observed person is abnormal based on the coordinates of skeleton points P0 to P16 of the observed person extracted from the person area PE of the image 20A captured by the photographing device 20. The coordinate determination unit 11D can execute posture determination based on the coordinates by using methods including the following two methods.

[0070] (Coordinate judgment part-judgment 1) In the first type of posture determination based on coordinates (hereinafter, sometimes referred to as determination 1), it is possible to mainly determine whether the observed person has fallen forward in the shooting direction, as shown as case 1 in Fig. 3. In this case, the observed person is often detected in an inverted posture in the image 20A, as shown in Fig. 5A. Note that "mainly" means that other abnormal postures can be detected by this determination method.

[0071] In this judgment 1, as shown in FIG. 5B, if the coordinates (Y-axis coordinate Y3) of skeleton points P11 and P12 indicating the position of the waist are located higher than the coordinates (Y-axis coordinate Y2) of skeleton points P5 and P6 indicating the position of the shoulders, and the coordinates of skeleton points P5 and P6 indicating the position of the shoulders are located higher than the coordinates of skeleton points P0, P1 and P2 (Y-axis coordinate Y1) indicating the position of the head, then the posture of the observed person is judged to be abnormal.

[0072] The Y-axis coordinate Y1 of skeleton points P0, P1, and P2 is the average value of the Y-axis coordinates of skeleton points P0, P1, and P2. Similarly, the Y-axis coordinate Y2 of skeleton points P5 and P6 is the average value of the Y-axis coordinates of skeleton points P5 and P6, and the Y-axis coordinate Y3 of skeleton points P11 and P12 is the average value of the Y-axis coordinates of skeleton points P11 and P12.

[0073] (Coordinate judgment section - Judgment 2) In the second type of posture determination based on coordinates (hereinafter, sometimes referred to as determination 2), it is mainly possible to determine whether or not the observed person is lying sideways relative to the shooting direction, as shown as case 2 in Fig. 3. In this case, the observed person is often detected as leaning in the image 20A (leaning relative to the reference line SL), as shown in Fig. 6A.

[0074] In this judgment 2, as shown in FIG. 6B, if both the angle of the thighs and the angle of the torso are tilted by a predetermined angle or more with respect to the reference line SL indicating the axial direction detected by the axial direction detection unit 11C, the posture of the observed person is judged to be abnormal.

[0075] The thigh angle is the angle of both the left thigh and the right thigh. The left thigh angle is the angle θ1 of the line segment L1 connecting the skeleton point P11 (left hip) and the skeleton point P13 (left knee) with respect to the reference line SL. The right thigh angle is the angle θ2 of the line segment L2 connecting the skeleton point P12 (right hip) and the skeleton point P14 (right knee) with respect to the reference line SL.

[0076] The torso angle is the angle of both the left flank and the right flank. The left flank angle is the angle θ3 of the line segment L3 connecting the skeletal point P11 (left hip) and the skeletal point P5 (left shoulder) with respect to the reference line SL. The right flank angle is the angle θ4 of the line segment L4 connecting the skeletal point P12 (right hip) and the skeletal point P6 (right shoulder) with respect to the reference line SL.

[0077] The axial direction detector 11C determines that the posture of the person being observed is abnormal when all of the angles θ1 to θ4 are inclined with respect to the reference line SL by a preset angle or more. One example of the "preset angle" is 60°. However, this threshold value can be changed as appropriate depending on the position where the photographing device 20 is installed and the angle of view of the photographing device 20. Furthermore, the "preset angles" may be different values ​​for θ1 to θ4. Alternatively, the "preset angles" may be the same value for θ1 and θ2, and the same value for θ3 and θ4.

[0078] The coordinate determination unit 11D can execute two types of posture determination based on the coordinates as described above. In this case, when the posture of the observed person is determined to be abnormal in at least one of the determinations 1 and 2, the coordinate determination unit 11D determines that the posture of the observed person is abnormal as a posture determination result based on the coordinates.

[0079] (Probability value judgment section - judgment 3) If the coordinate determining section 11D determines that the posture of the observed person is not abnormal, the probability value determining section 11E further executes a posture determination of the non-observing person based on the probability values ​​of the skeleton points.

[0080] In posture judgment based on a probability value (hereinafter, sometimes referred to as judgment 3), it is possible to mainly judge whether or not the observed person has fallen backward with respect to the shooting direction, as shown as case 3 in Fig. 3. In this case, the observed person is often detected in the image 20A in a normal posture, but with a small vertical dimension, as shown in Fig. 8A.

[0081] In this determination 3, the probability value determination unit 11E determines that the posture is abnormal when at least one of the probability values ​​of the skeleton points P3 and P4 indicating the positions of the left and right ears of the observed person shown in Fig. 8B is equal to or smaller than a threshold value. This threshold value is a preset value.

[0082] When the observed person is leaning backward with respect to the shooting direction, as shown in Fig. 8C, the left and right ears of the observed person are difficult to photograph because they are blocked by the torso and clothing. Also, compared to when the observed person is leaning "forward" with respect to the shooting direction in the same position as shown in Fig. 8C, the positions of the left and right ears are relatively farther away. Therefore, the probability values ​​of skeleton points P3 and P4 become smaller. Therefore, when at least one of the probability values ​​of skeleton points P3 and P4 is equal to or less than a threshold value, it can be determined that the posture is abnormal.

[0083] (Example of posture abnormality detection) By using the coordinate determination unit 11D and the probability value determination unit 11E, it is possible to detect whether or not the following postures, for example, are abnormal.

[0084] (Example 1) 9A shows an example in which the observed person has shifted from a standing posture to a posture in which he / she bends over and drops his / her head to pick up a baggage placed on the floor. In this posture, in judgment 1, the posture is judged not to be abnormal based on the positions of the observed person's waist, shoulders, and head. In judgment 2, even if the angle of the torso is greater than a predetermined angle, the posture is judged not to be abnormal because the angle of the thighs is equal to or less than the predetermined angle. Therefore, the posture abnormality is not detected in the coordinate judgment.

[0085] If no posture abnormality is detected in the coordinate determination, the probability value determination unit 11E executes a probability value determination (Determination 3). In this posture, the head of the observed person is unlikely to be blocked by the torso, etc., so that the skeleton point extraction unit 11B can easily detect the coordinates of the left and right ears.

[0086] Therefore, both the probability values ​​of the left and right ears are detected as being greater than the threshold. In other words, the condition that "if at least one of the probability values ​​of the skeleton points P3 and P4 indicating the positions of the left and right ears of the observed person is equal to or less than the threshold, the posture is determined to be abnormal" is not met. Therefore, the posture abnormality is not detected by the probability value judgment.

[0087] (Example 2) FIG. 9B shows an example in which the observed person has shifted from a standing posture to a crouching posture (squatting posture). In this posture, in judgment 1, the posture is determined to be normal based on the positions of the observed person's waist, shoulders, and head. In judgment 2, even if the angle of the thighs is greater than a predetermined angle, the posture is determined to be normal because the angle of the torso is equal to or smaller than the predetermined angle. Therefore, no abnormal posture is detected.

[0088] If no posture abnormality is detected in the coordinate determination, the probability value determination unit 11E executes a probability value determination (Determination 3). In this posture, the head of the observed person is unlikely to be blocked by the torso, etc., so that the skeleton point extraction unit 11B can easily detect the coordinates of the left and right ears.

[0089] Therefore, the probability values ​​for both the left and right ears are detected as being greater than the threshold value, and therefore, posture abnormality is not detected in the probability value determination.

[0090] (Example 3) 9C shows an example in which the observed person falls sideways from a sitting position. In this position, in judgment 1, the posture is judged not to be abnormal based on the positions of the observed person's waist, shoulders, and head. In judgment 2, if the angle between the torso and thighs is larger than a predetermined angle, the posture is judged to be abnormal.

[0091] Alternatively, in judgment 2, it may be determined that the posture is not abnormal because at least one of the torso angle and the thigh angle is equal to or smaller than a predetermined angle. In such a case, a posture abnormality is not detected in the coordinate judgment.

[0092] If no abnormal posture is detected in the coordinate determination, the probability value determination unit 11E executes a probability value determination (Determination 3). In this posture, the left and right ears of the observed person are likely to be blocked by the torso, etc., so it is difficult for the skeleton point extraction unit 11B to detect the coordinates of at least one of the left and right ears. In this case, at least one of the probability values ​​of the skeleton points P3 and P4 indicating the positions of the left and right ears of the observed person is likely to be equal to or smaller than the threshold, so there is a high possibility that the posture will be determined to be abnormal.

[0093] (Example 4) FIG. 9D shows an example in which the observed person has fallen from a sitting position onto the desk face down. In this posture, in judgment 1, the posture is judged not to be abnormal based on the positions of the observed person's waist, shoulders, and head. In judgment 2, the posture is judged not to be abnormal because at least one of the angles of the torso and the thighs is equal to or smaller than a predetermined angle. In such a case, the posture abnormality is not detected in the coordinate judgment.

[0094] If no abnormal posture is detected in the coordinate determination, the probability value determination unit 11E executes a probability value determination (Determination 3). In this posture, the left and right ears of the observed person are likely to be blocked by the arms, etc. In addition, since the left and right ears are photographed from the top of the head, it is difficult for the skeleton point extraction unit 11B to detect the coordinates of at least one of the left and right ears, compared to when photographing from the face side. In this case, at least one of the probability values ​​of the skeleton points P3 and P4 indicating the positions of the left and right ears of the observed person is likely to be below the threshold, and there is a high possibility that the posture will be determined to be abnormal.

[0095] <Posture abnormality detection process> The CPU 11 of the posture abnormality detection device 10 executes the posture abnormality detection program 13A when receiving an instruction to execute the posture abnormality detection program 13A from a user via, for example, the input unit 14. This causes the posture abnormality detection process shown in FIG.

[0096] When the posture abnormality detection process is executed, in step S102, the CPU 11 acquires the image 20A captured by the image capturing device 20. After step S102, the process proceeds to step S104.

[0097] In step S104, the CPU 11 detects the axial direction from the image 20A. If the CPU 11 detects the reference line SL from each of the multiple images 20A captured within a predetermined period to derive the axial direction, the CPU 11 repeats steps S102 and S104 to detect the reference line SL and derive the axial direction. After step S104, the process proceeds to step S106.

[0098] In step S106, the CPU 11 detects skeleton points of the person being observed. After step S106, the process proceeds to step S110.

[0099] In step S110, the CPU 11 executes coordinate determination of the posture of the observed person based on the coordinates of the skeleton points of the observed person, and determines whether the posture of the observed person is abnormal or not. If the determination in step S110 is positive, the process proceeds to step S112. On the other hand, if the determination in step S110 is negative, the process proceeds to step S114.

[0100] In step S112, the CPU 11 controls the notification device 30 to notify that the posture of the person being observed is abnormal. After step S112, the process proceeds to step S120.

[0101] In step S114, the CPU 11 executes a probability value judgment on the posture of the observed person based on the coordinates of the skeleton points of the observed person, and judges whether the posture of the observed person is abnormal or not. If the judgment in step S114 is positive, the process proceeds to step S116. On the other hand, if the judgment in step S114 is negative, the process proceeds to step S120.

[0102] In step S116, the CPU 11 controls the notification device 30 to notify that the posture of the person being observed is abnormal. After step S116, the process proceeds to step S120.

[0103] In step S120, the CPU 11 determines whether or not the timing for ending the posture abnormality detection process has arrived. The posture abnormality detection process ends when, for example, an instruction to end the posture abnormality detection program 13A is received from the user via the input unit 14. If the determination in step S120 is positive, the posture abnormality detection process ends. On the other hand, if the determination in step S120 is negative, the process proceeds to step S122.

[0104] In step S122, the CPU 11 acquires the image 20A captured by the photographing device 20. After step S122, the process proceeds to step S106, where skeleton points of the observed person are extracted, and the subsequent processes are executed again.

[0105] <Action and Effects> In the posture abnormality detection system 80 of the present disclosure, when the posture of the observed person is determined not to be abnormal by the coordinate determination, the posture abnormality detection device 10 further executes a probability value determination to determine whether or not the posture of the observed person is abnormal. Therefore, it is possible to reduce the number of posture abnormalities that are overlooked, compared to when the posture of the observed person is determined based only on the coordinates of the skeleton points.

[0106] In addition, in this posture abnormality detection system 80, in posture judgment based on probability values, if at least one of the probability values ​​of the skeleton points (skeleton points P3 and P4) indicating the positions of the observed person's left and right ears is below a threshold value, it is judged that the posture is abnormal.

[0107] This makes it easier to detect abnormal posture compared to a case in which posture is not determined to be abnormal only when "both" of the probability values ​​of the skeleton points indicating the positions of the left and right ears are below a threshold value.

[0108] <Other embodiments> In the above embodiment, the coordinate determination unit 11D determines whether or not the posture of the observed person is abnormal based on the positions of the waist, shoulders, and head in the determination 1, but the embodiment of the present disclosure is not limited to this. For example, it may be determined whether or not the posture is abnormal based on the positions of the lower body, such as the positions of the heels, waist, and head, and the positions of the knees, waist, and head.

[0109] In the above embodiment, the coordinate determining unit 11D can determine whether or not the posture of the person being observed is abnormal by executing the determination 1 and the determination 2 for one image 20A, but the embodiment of the present disclosure is not limited to this.

[0110] For example, the coordinate determination unit 11D may perform judgment 1 and judgment 2 on "multiple" images 20A taken within a predetermined period, and if it is determined that the posture of the observed person is abnormal in all of the images 20A, it may determine that the posture of the observed person is abnormal.

[0111] Alternatively, the coordinate determining section 11D may determine that the posture of the observed person is abnormal when the posture of the observed person is determined to be abnormal in a predetermined ratio or more of the images 20A.

[0112] By using a plurality of images 20A in this way, for example, when the subject temporarily lies down on the floor and then returns to a normal posture, posture abnormality is prevented from being detected. In other words, the accuracy of posture abnormality detection is improved compared to the case where the posture is judged from a single image 20A.

[0113] The coordinate determination unit 11D may execute only one of the determination 1 and the determination 2. In this case, the user can arbitrarily select, by input operation via the input unit 14 or the like, which of the determination 1 and the determination 2 to execute in the posture abnormality detection process.

[0114] In the above embodiment, the probability value determining unit 11E can determine that the posture of the observed person is abnormal when the probability values ​​of the skeleton points P3 and P4 extracted from one image 20A are equal to or smaller than a threshold value. However, the embodiment of the present disclosure is not limited to this.

[0115] For example, the probability value determining section 11E may determine that the posture of the person being observed is abnormal when the probability values ​​of the skeleton points P3 and P4 in "multiple" images taken within a predetermined period of time are equal to or smaller than a threshold value.

[0116] By using a plurality of images 20A in this way, for example, when the subject temporarily lies down on the floor and then returns to a normal posture, posture abnormality is prevented from being detected. In other words, the accuracy of posture abnormality detection is improved compared to the case where the posture is judged from a single image 20A.

[0117] In the above embodiment, the probability value determining unit 11E can determine that the posture is abnormal when at least one of the probability values ​​of the skeleton points (skeleton points P3 and P4) indicating the positions of the left and right ears of the observed person is equal to or less than a threshold value. However, the embodiment of the present disclosure is not limited to this.

[0118] For example, the probability value determining unit 11E may determine that the posture is abnormal only when "both" of the probability values ​​of the skeleton points indicating the positions of the left and right ears of the observed person are equal to or less than a threshold value. This reduces erroneous detection of abnormal posture compared to the case where the posture is determined to be abnormal when at least one of the probability values ​​of the skeleton points is equal to or less than a threshold value.

[0119] In the above embodiment, the axial direction used in determination 2 in the coordinate determination is the direction perpendicular to the floor or ground in the image 20A or the direction of the trunk of the observed person, but the embodiment of the present disclosure is not limited to this.

[0120] For example, the vertical direction (Y direction) in the image 20A may be set as the axial direction. Even in this case, if the installation angle of the image capturing device 20 with respect to the floor or ground is specified, the direction perpendicular to the floor or ground can be set as the axial direction.

[0121] In the above embodiment, for example, the hardware structure of the processing unit that executes each process of the image acquisition unit 11A, the skeleton point extraction unit 11B, the axial direction detection unit 11C, the coordinate determination unit 11D, the probability value determination unit 11E, and the control unit 11F may be any of the following various processors: As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as a processing unit, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0122] The processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the processing unit may be configured with a single processor.

[0123] As an example of configuring the processing unit with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as represented by computers such as client and server, and this processor functions as the processing unit. Second, there is a form in which a processor is used that realizes the functions of the entire system including the processing unit with one IC (Integrated Circuit) chip, as represented by System On Chip (SoC), etc. In this way, the processing unit is configured using one or more of the above various processors as a hardware structure.

[0124] Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit combining circuit elements such as semiconductor elements can be used. In this manner, the present disclosure can be embodied in various forms.

[0125] <Additional Notes> (((1))) at least one processor; The processor, When it is determined that the posture of the person is not abnormal based on the coordinates of the skeleton points of the person extracted from the person area of ​​the image captured by the image capture device, further performing a posture determination of the person based on the probability values ​​of the skeleton points; Posture abnormality detection system.

[0126] (((2))) The processor, In the posture determination based on the probability value, determining that the posture is abnormal when at least one of the probability values ​​of the skeleton points indicating the positions of the left and right ears of the person is equal to or less than a threshold value; The posture abnormality detection system according to (((1))).

[0127] (((3))) If both of the probability values ​​are equal to or less than a threshold, the posture is determined to be abnormal. The posture abnormality detection system according to (((2))).

[0128] (((4))) The processor, determining that the posture is abnormal when the probability value is equal to or less than a threshold value in a plurality of images captured within a predetermined period of time; The posture abnormality detection system according to (((2))) or (((3))).

[0129] (((5))) The processor, In the posture determination based on the coordinates, When the coordinates of the skeleton point indicating the position of the waist are located higher than the coordinates of the skeleton point indicating the position of the shoulder, and the coordinates of the skeleton point indicating the position of the shoulder are located higher than the coordinates of the skeleton point indicating the position of the head, Determine that the person's posture is abnormal; 2. The posture abnormality detection system according to claim 1 , wherein the posture abnormality detection system is a posture abnormality detection system for detecting a posture of the user.

[0130] (((6))) The processor, In the posture determination based on the coordinates, When both the thigh angle and the torso angle are inclined by a predetermined angle or more with respect to a predetermined axial direction, Determine that the person's posture is abnormal; 2. The posture abnormality detection system according to claim 1 ,

[0131] (((7))) The processor, In the posture determination based on the coordinates, If the coordinates of the skeleton point indicating the position of the waist are located higher than the coordinates of the skeleton point indicating the position of the shoulder, and the coordinates of the skeleton point indicating the position of the shoulder are located higher than the coordinates of the skeleton point indicating the position of the head, And, When both the thigh angle and the torso angle are inclined by a predetermined angle or more with respect to the predetermined axial direction, and determining that the posture of the person is abnormal in at least one of the above cases. 2. The posture abnormality detection system according to claim 1 , wherein the posture abnormality detection system is a posture abnormality detection system for detecting a posture of the user.

[0132] (((8))) The processor, In a plurality of images captured within a predetermined period, when the coordinates of a skeleton point indicating the position of the waist are located above the coordinates of a skeleton point indicating the position of the shoulders, and the coordinates of the skeleton point indicating the position of the shoulders are located above the coordinates of a skeleton point indicating the position of the head, And, In a plurality of images taken within a predetermined period, when both the angle of the thigh and the angle of the torso are inclined by a predetermined angle or more with respect to a predetermined axial direction, and determining that the posture of the person is abnormal in at least one of the above cases. The posture abnormality detection system according to (((7))).

[0133] (((9))) The processor, As the axial direction, Detecting a vertical direction with respect to a floor or ground in the image; The posture abnormality detection system according to (((7))) or (((8))).

[0134] (((10))) The processor, As the axial direction, detecting a trunk direction of the person when standing or sitting, the trunk direction being extracted from the skeleton points; The posture abnormality detection system according to (((7))) or (((8))).

[0135] (((11))) The processor, deriving the trunk direction from a plurality of images taken within a predetermined period of time; The posture abnormality detection system according to (((10))).

[0136] (((12))) On the computer, When it is determined that the posture of the person is not abnormal based on the coordinates of the skeleton points of the person extracted from the person area of ​​the image captured by the image capture device, and further performing a posture determination of the person based on the probability values ​​of the skeleton points. Posture abnormality detection program.

[0137] The posture abnormality detection system of (((1))) and the posture abnormality detection program of (((12))) can reduce the number of posture abnormalities that are overlooked, compared to when a person's posture is determined based only on the coordinates of skeleton points.

[0138] According to the posture abnormality detection system (((2))), it is easy to detect posture abnormality when a person falls backward relative to the imaging device.

[0139] According to the posture abnormality detection system of (((3))), it is possible to reduce false positives of abnormal posture compared to a case in which the posture is determined to be abnormal when at least one of the probability values ​​of the skeleton points is below the threshold value.

[0140] According to the posture abnormality detection system of (((4))), the accuracy of posture abnormality detection is improved compared to a case where the presence or absence of posture abnormality is determined from a single image.

[0141] According to the posture abnormality detection system of (((5))), it is easy to detect posture abnormality when a person falls toward the imaging device.

[0142] According to the posture abnormality detection system of (((6))), when a person falls at an angle equal to or greater than a preset angle with respect to a preset axial direction, a posture abnormality can be easily detected.

[0143] The posture abnormality detection system of (((7))) makes it easy to detect posture abnormalities in both cases when a person falls toward the imaging device and when a person falls at an angle equal to or greater than a preset angle relative to a preset axial direction.

[0144] According to the posture abnormality detection system of (((8))), the accuracy of posture abnormality detection is improved compared to a case where the presence or absence of posture abnormality is determined from a single image.

[0145] According to the posture abnormality detection system of (((9))), it is difficult to erroneously detect posture abnormalities even if the imaging device is installed at an angle relative to the floor or ground.

[0146] According to the posture abnormality detection system of (((10))), posture abnormalities can be detected taking into account individual differences in usual posture.

[0147] According to the abnormal posture detection system of (((11))), the accuracy of detecting the trunk direction is improved compared to the case where the trunk direction is detected from a single image. [Explanation of symbols]

[0148] 11 CPU (Processor) 20 Imaging Equipment 20A Images P0 skeleton point P1 skeleton point P2 skeleton point P3 skeleton point P4 Skeleton point P5 Skeleton point P6 Skeleton Point P7 Skeleton point P8 Skeleton points P9 Skeleton point P10 skeleton point P11 Skeleton point P12 Skeleton point P13 Skeleton point P14 Skeleton point P15 Skeleton point P16 Skeleton point

Claims

1. At least one processor; The processor, When it is determined that the posture of the person is not abnormal based on the coordinates of the skeleton points of the person extracted from the person area of ​​the image captured by the image capture device, further performing a posture determination of the person based on the probability values ​​of the skeleton points; Posture abnormality detection system.

2. The processor, In the posture determination based on the probability value, determining that the posture is abnormal when at least one of the probability values ​​of the skeleton points indicating the positions of the left and right ears of the person is equal to or less than a threshold value; The posture abnormality detection system according to claim 1 .

3. If both of the probability values ​​are equal to or less than the threshold value, the posture is determined to be abnormal. The posture abnormality detection system according to claim 2 .

4. The processor, determining that the posture is abnormal when the probability value is equal to or less than a threshold value in a plurality of images captured within a predetermined period of time; The posture abnormality detection system according to claim 2 or 3.

5. The processor, In the posture determination based on the coordinates, When the coordinates of the skeleton point indicating the position of the waist are located higher than the coordinates of the skeleton point indicating the position of the shoulder, and the coordinates of the skeleton point indicating the position of the shoulder are located higher than the coordinates of the skeleton point indicating the position of the head, Determine that the person's posture is abnormal; The posture abnormality detection system according to claim 1 .

6. The processor, In the posture determination based on the coordinates, When both the thigh angle and the torso angle are inclined by a predetermined angle or more with respect to a predetermined axial direction, Determine that the person's posture is abnormal; The posture abnormality detection system according to claim 1 .

7. The processor, In the posture determination based on the coordinates, If the coordinates of the skeleton point indicating the position of the waist are located higher than the coordinates of the skeleton point indicating the position of the shoulder, and the coordinates of the skeleton point indicating the position of the shoulder are located higher than the coordinates of the skeleton point indicating the position of the head, And, When both the thigh angle and the torso angle are inclined by a predetermined angle or more with respect to the predetermined axial direction, and determining that the posture of the person is abnormal in at least one of the above cases. The posture abnormality detection system according to claim 1 .

8. The processor, In a plurality of images captured within a predetermined period, when the coordinates of a skeleton point indicating the position of the waist are located above the coordinates of a skeleton point indicating the position of the shoulders, and the coordinates of the skeleton point indicating the position of the shoulders are located above the coordinates of a skeleton point indicating the position of the head, And, In a plurality of images taken within a predetermined period, when both the angle of the thigh and the angle of the torso are inclined by a predetermined angle or more with respect to a predetermined axial direction, and determining that the posture of the person is abnormal in at least one of the above cases. The posture abnormality detection system according to claim 7 .

9. The processor, As the axial direction, Detecting a vertical direction with respect to a floor or ground in the image; The posture abnormality detection system according to claim 7 or 8.

10. The processor, As the axial direction, detecting a trunk direction of the person when standing or sitting, the trunk direction being extracted from the skeleton points; The posture abnormality detection system according to claim 7 or 8.

11. The processor, deriving the trunk direction from a plurality of images taken within a predetermined period of time; The posture abnormality detection system according to claim 10.

12. On the computer, When it is determined that the posture of the person is not abnormal based on the coordinates of the skeleton points of the person extracted from the person area of ​​the image captured by the image capture device, and further performing a posture determination of the person based on the probability values ​​of the skeleton points. Posture abnormality detection program.

Images