Information processing apparatus, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program which, with a wide-angle camera, can detect fall of a target person.SOLUTION: An information processing apparatus according to the present invention has an acquiring unit for acquiring a target image information relating to a target image obtained by capturing a target person with a wide angle camera and distortion information relating to a distortion state of the image at each of the position in the image captured by the wide-angle camera, a generation unit for generating, based on the acquired target image information, a target positional information relating to a position of the target person in the target image and a target skeleton information relating to a skeleton state of the target person in the target image, and a fall detection unit for, based on the generated target position information, the generated target skeleton information, and the distortion information, detecting fall of the target person.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, cameras have been installed in factories and nursing care facilities to ensure the safety of workers in factories and users of nursing care facilities. Images captured by these cameras are subjected to image processing such as object recognition and used for various purposes.

[0003] A wide-angle camera has a wider viewing angle than a normal camera. Therefore, a wide-angle camera can capture images of a wide area in a factory, nursing care facility, etc. For example, Patent Document 1 discloses a technology related to object recognition in an image captured using a fisheye camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-230546 Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to use such a wide-angle camera to detect falls of a subject, which would allow a single camera to capture a wider area, such as in a factory or nursing home.

[0006] The present invention has been made in view of the above circumstances, and a main object of the present invention is to provide an information processing device, an information processing method, and an information processing program that are capable of detecting a fall of a subject using a wide-angle camera. [Means for solving the problem]

[0007] The above-mentioned problems of the present invention are solved by the following means.

[0008] (1) An information processing device comprising: an acquisition unit that acquires target image information regarding a target image of a subject photographed by a wide-angle camera and distortion information regarding the distortion state of the image at each position within the image photographed by the wide-angle camera; a generation unit that generates target position information regarding the position of the subject in the target image and target skeletal information regarding the skeletal state of the subject in the target image based on the acquired target image information; and a fall detection unit that detects a fall of the subject based on the generated target position information and target skeletal information and the distortion information.

[0009] (2) The information processing device described in (1) above, wherein the fall detection unit detects a fall of the subject when the angle between a predetermined line formed by the subject's skeleton in the target image and a reference line set in the target image exceeds a threshold value.

[0010] (3) The information processing device according to (2), wherein the fall detection unit determines the threshold value based on the target position information and the distortion information.

[0011] (4) The information processing device described in (3) above, wherein the distortion information is generated based on reference position information regarding the position of the object in a reference image photographed by the wide-angle camera and reference skeletal information regarding the skeletal state of the object in the reference image.

[0012] (5) The information processing device according to (2), wherein the distortion information associates each position in the image with a correction angle of the threshold.

[0013] (6) The information processing device according to (1) above, wherein the distortion information is stored in a storage unit.

[0014] (7) The information processing device according to (6) above, further comprising a change unit that changes the distortion information stored in the storage unit.

[0015] (8) The information processing device according to (7), wherein the change unit changes the distortion information based on the target position information and the target skeletal information.

[0016] (9) The information processing device according to (7), wherein the change unit changes the distortion information in response to an input from a user.

[0017] (10) The information processing device according to (1) above, further comprising an output unit that outputs fall information relating to a detected fall of the subject.

[0018] (11) The information processing device according to (10), further comprising an image generating unit that generates a distortion correspondence image that visualizes the distortion information, and the output unit further outputs the generated distortion correspondence image.

[0019] (12) The information processing device according to (11), wherein the distortion-corresponding image includes at least one of a grayscale image and a colorscale image.

[0020] (13) An information processing method using an information processing device, comprising: acquiring target image information relating to a target image of a subject photographed by a wide-angle camera and distortion information relating to the distortion state of the image at each position within the image photographed by the wide-angle camera; generating target position information relating to the position of the subject in the target image and target skeletal information relating to the skeletal state of the subject in the target image based on the acquired target image information; and detecting a fall of the subject based on the generated target position information, the target skeletal information, and the distortion information.

[0021] (14) An information processing program for causing the information processing device to execute processing including the information processing method described in (13) above. [Effects of the Invention]

[0022] In the information processing device, information processing method, and information processing program according to the present invention, a fall of a subject is detected using distortion information in addition to target position information and target skeletal information. This makes it possible to detect with high accuracy even if the subject falls in a distorted part of the target image captured by a wide-angle camera. Therefore, it becomes possible to detect a fall of a subject using a wide-angle camera. [Brief explanation of the drawings]

[0023] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for purposes of illustration only and are not intended to be limiting. [Figure 1] 1 is a diagram illustrating a schematic configuration of a fall detection system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram illustrating an example of an installation state of the imaging device illustrated in FIG. [Figure 3] 2 is a block diagram illustrating a schematic configuration of the imaging device shown in FIG. 1. FIG. [Figure 4] 2 is a schematic block diagram illustrating a hardware configuration of the information processing device shown in FIG. 1. [Figure 5A] FIG. 1 is a diagram illustrating an example of an image captured by a wide-angle camera. [Figure 5B] FIG. 5B is a diagram illustrating distortion of the image shown in FIG. 5A. [Figure 6A] FIG. 10 is a diagram illustrating another example of an image captured by a wide-angle camera. [Figure 6B] FIG. 6B is a diagram illustrating distortion of the image shown in FIG. 6A. [Figure 7A] 10A and 10B are diagrams illustrating other examples of images captured by a wide-angle camera. [Figure 7B] FIG. 7B is a diagram illustrating distortion of the image shown in FIG. 7A. [Figure 8] 2 is a diagram showing a reference image captured by the imaging device shown in FIG. 1 together with detected joint points. [Figure 9]9 is a diagram for explaining a method of generating distortion information based on the reference image shown in FIG. 8. FIG. [Figure 10] 2 is a schematic block diagram illustrating a functional configuration of the information processing device shown in FIG. 1. [Figure 11] 2 is a diagram showing an image of a target captured by the imaging device shown in FIG. 1 together with detected joint points. [Figure 12] 5 is a diagram illustrating an example of distortion information stored in an auxiliary storage unit illustrated in FIG. 4. FIG. [Figure 13] 2 is a flowchart illustrating a processing procedure of a fall detection method performed by the information processing device shown in FIG. 1. [Figure 14] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a first modified example. [Figure 15] 15 is a flowchart illustrating a processing procedure of a fall detection method performed by the information processing device shown in FIG. 14. [Figure 16] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a second modification. [Figure 17] 17 is a diagram showing an example of a distortion-adjusted image generated by the image generating unit shown in FIG. 16. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0025] <Embodiment> Overall Configuration of Fall Detection System 100 1 is a schematic diagram showing an example of the configuration of a fall detection system 100 according to an embodiment. The fall detection system 100 includes, for example, an imaging device 200 and an information processing device 300. The imaging device 200 is connected to the information processing device 300 via a communication network so that they can communicate with each other. The fall detection system 100 detects a fall, for example, of a worker in a factory or a user in a nursing care facility, and notifies a manager, caregiver, or the like.

[0026] FIG. 2 shows an example of an installation state of the imaging device 200. The imaging device 200 is installed so as to be able to capture an image of a subject 400. The imaging device 200 captures, for example, moving images. The imaging device 200 is installed, for example, in a factory or a nursing care facility. The subject 400 is a worker in a factory or the like or a user of a nursing care facility or the like, and moves along a path PA on a floor surface FL. The field of view of the imaging device 200 includes this path PA.

[0027] 3 is a block diagram illustrating a schematic configuration of imaging device 200. Imaging device 200 includes, for example, control unit 210, communication unit 220, and wide-angle camera 230. Control unit 210, communication unit 220, and wide-angle camera 230 are connected to one another by bus 201.

[0028] The control unit 210 includes, for example, a CPU, RAM, and ROM. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory. The CPU controls each unit of the imaging device 200 and performs arithmetic processing according to a program stored in the memory, for example.

[0029] The control unit 210 transmits a plurality of captured images obtained by the wide-angle camera 230 capturing an image of a predetermined area to the information processing device 300 via the communication unit 220. The predetermined area is, for example, a three-dimensional area including the route PA.

[0030] The communication unit 220 includes an interface circuit for communicating with the information processing device 300 and the like via a communication network, for example.

[0031] The wide-angle camera 230 is a camera with a wider angle of view than a standard camera. The wide-angle camera 230 includes a fisheye camera, etc. By including the wide-angle camera 230 in the imaging device 200, it becomes possible to capture images with a wider angle of view. Therefore, it becomes possible to detect a fall of the subject 400 over a wider area. The wide-angle camera 230 may be disposed in a position where it captures the subject 400 from a bird's-eye view, or may be disposed in a position where it captures the subject 400 from above. The wide-angle camera 230 may be disposed in a position where it captures the subject 400 horizontally.

[0032] Wide-angle camera 230 is configured, for example, by a visible light camera. The visible light camera can capture an image of a predetermined area by receiving visible light reflected by objects within the predetermined area using a CMOS sensor. CMOS is an abbreviation for Complememtary Metal Oxide Semiconductor.

[0033] The wide-angle camera 230 may be configured as a near-infrared camera. The infrared camera captures an image of a predetermined area by irradiating the predetermined area with near-infrared light using an LED and receiving the near-infrared light reflected by objects within the predetermined area using a CMOS sensor. In this case, the image captured by the wide-angle camera 230 may be a monochrome image in which each pixel has a reflectance of near-infrared light. LED is an abbreviation for Light Emitting Device. The wide-angle camera 230 may include a near-infrared camera and a visible light camera.

[0034] 4 is a schematic block diagram illustrating an example of the hardware configuration of the information processing device 300. The information processing device 300 is configured, for example, by a personal computer, a server, a smartphone, or a tablet terminal. The information processing device 300 has, for example, a CPU 310, a RAM 320, a ROM 330, an auxiliary storage unit 340, a communication I / F 350, an operation display unit 360, and an alarm 370. The components of the information processing device 300 are connected to each other by a bus 301. Specific functions of the CPU 310 will be described later. The RAM 320 and ROM 330 are memories.

[0035] The auxiliary storage unit 340 may be configured with, for example, an SSD or an HDD. SSD is an abbreviation for Solid State Drive. HDD is an abbreviation for Hard Disk Drive. The auxiliary storage unit 340 stores, for example, an information processing program for detecting a fall of the subject 400, images captured by the imaging device 200, the capture date and time of the images, distortion information, and the like.

[0036] The distortion information is information relating to the state of distortion of an image at each position within the image captured by wide-angle camera 230. In an image captured by wide-angle camera 230, image distortion is more likely to occur as the image approaches the edge of the image.

[0037] 5A and 5B show an example of an image captured from a bird's-eye view by a wide-angle camera, where the image is distorted outward at the edges, as indicated by the arrows in Fig. 5B.

[0038] 6A and 6B show an example of an image captured horizontally by a wide-angle camera. The image is distorted at the edges, as indicated by the arrows in Fig. 6B. Specifically, the bottom of the image is distorted outward, and the top of the image is distorted inward.

[0039] 7A and 7B show an example of an image taken from above by a wide-angle camera, where the edges of the image are distorted inward, as indicated by the arrows in Fig. 7B.

[0040] The distortion information associates each position in an image captured by such a wide-angle camera with the direction and magnitude of the distortion, etc. A method for generating the distortion information will be described later.

[0041] The communication I / F 350 is an interface circuit for communicating with other devices, such as the imaging device 200, via a network. The interface circuit complies with standards such as LAN, USB, and MIPI. LAN is an abbreviation for Local Area Network. USB is an abbreviation for Universal Serial Bus. MIPI is an abbreviation for Mobile Industry Processor Interface.

[0042] The operation display unit 360 has an input unit and an output unit. The input unit is configured with, for example, a touch panel, a keyboard, or a mouse, etc. The output unit is configured with, for example, a display.

[0043] For example, the alarm 370 outputs an alarm when it detects that the subject 400 has fallen. The alarm 370 has, for example, a speaker and can output a warning sound or voice from the speaker as the alarm. The alarm 370 also has a lamp or LED and can light or flash the lamp or LED as the alarm.

[0044] [Method for generating distortion information] The information processing device 300 generates distortion information, for example, in the following manner.

[0045] The information processing device 300 first acquires reference image information related to a reference image. The reference image is an image of an object captured by the imaging device 200. This object is, for example, a person. Next, the information processing device 300 generates reference position information related to the position of the object captured in this reference image within the image, and reference skeleton information related to the skeletal state of the object in the reference image. Here, the skeleton is formed, for example, by lines connecting predetermined joint points with each other, or lines connecting a joint point with the center point of two joint points.

[0046] The information processing device 300 may acquire the reference image information from the imaging device 200, or may acquire the reference image information by reading the reference image from the auxiliary storage unit 340 or the like. The information processing device 300 may acquire the reference image information by reading the reference image from an external storage device or the like. The information processing device 300 may acquire the reference image information offline.

[0047] FIG. 8 shows an example of a reference image IM captured by the imaging device 200. This reference image IM captures a person HM in a standing position at position P0 as an object. The information processing device 300, for example, performs predetermined preprocessing on reference image information related to the reference image IM, and then generates reference position information and reference skeletal information of the person HM. The preprocessing performed on the reference image information includes, for example, convolution integral and gradation correction using a one-dimensional lookup table. By performing preprocessing on the reference image information, the person HM is highlighted even if there are parts of the person HM that are difficult to view in the reference image IM, thereby improving the accuracy of detecting the joint points of the person HM.

[0048] The information processing device 300 generates reference position information and reference skeleton information of the person HM, for example, by analyzing the reference image IM and detecting the joint points of the person HM. The reference skeleton information may include, for example, information on the position of each of a plurality of joint points, such as the eyes, nose, neck, shoulders, elbows, wrists, waist, knees, and ankles, which are skeletal feature points of the person HM, and information on line segments connecting the joint points. The information on the position of each of the plurality of joint points is, for example, the position coordinates of each of the plurality of joint points in the reference image IM. The position coordinates are, for example, represented by X and Y coordinates provided in the reference image IM. The reference position information of the person HM is, for example, information on a position P0, and may include, for example, information on the position coordinates of one or more joint points of the person HM.

[0049] The information processing device 300 detects the joint points of the person HM from a human rectangle BX including the person HM, for example, using a trained model. The human rectangle BX is an area containing the joint points and nodes of the person HM in the reference image IM. The information processing device 300 may estimate the certainty of each joint point along with the joint points. It is desirable that the information processing device 300 use joint points with a certainty equal to or greater than a predetermined value. This is because, due to the nature of machine learning, a low certainty increases the likelihood of an incorrect detection result.

[0050] The information processing device 300 detects the joint points of the person HM using a trained model such as OpenPose (https: / / arxiv.org / abs / 1812.08008), DeepPose (https: / / arxiv.org / abs / 1312.4659), and ResNet (https: / / arxiv.org / abs / 1512.03385), for example.

[0051] Next, the information processing device 300 calculates the angle between a predetermined line formed by the skeleton of the person HM and the reference line based on the generated reference skeleton information. The information processing device 300 generates distortion information by, for example, associating this angle with the position P0.

[0052] 9 shows reference images IM1, IM2, and IM3 superimposed on each other. Reference images IM1, IM2, and IM3 are images of the same area captured by imaging device 200 from the same position and at the same angle. In reference image IM1, a person HM in a standing position is located at position P1, in reference image IM2, a person HM in a standing position is located at position P2, and in reference image IM3, a person HM in a standing position is located at position P3. For example, position P1 is the center of the image, position P2 is the left side of the image, and position P3 is the right side of the image.

[0053] The information processing device 300 sets, for example, an X-axis and a Y-axis in the reference images IM1, IM2, and IM3. The reference line is, for example, a straight line parallel to the Y-axis. In each of the reference images IM1, IM2, and IM3, joint points 0 to 17 of the person HM are detected. The predetermined lines formed by the skeleton of the person HM include, for example, a 0-1 line connecting joint points 0 and 1, a 9-10 line connecting joint points 9 and 10, and a 12-13 line connecting joint points 12 and 13. The 0-1 line corresponds to the neck of the person HM. The 9-10 line corresponds to the right shin of the person HM. The 12-13 line corresponds to the left shin of the person HM.

[0054] For example, the information processing device 300 calculates, for the reference image IM1, the angle α11 between the 0-1 line and a line parallel to the Y axis, the angle α12 between the 9-10 line and a line parallel to the Y axis, and the angle α13 between the 12-13 line and a line parallel to the Y axis. The angles α11, α12, and α13 are each approximately 0 degrees. The information processing device 300 stores the angles α11, α12, and α13 and the position P1 in association with each other. The position P1 is represented, for example, by the position coordinates of at least one of the joint points 0 to 17 in the reference image IM1.

[0055] For example, the information processing device 300 calculates, for the reference image IM2, an angle α21 between the 0-1 line and a line parallel to the Y axis, an angle α22 between the 9-10 line and a line parallel to the Y axis, and an angle α23 between the 12-13 line and a line parallel to the Y axis. The information processing device 300 stores the angles α21, α22, and α23 in association with a position P2. The position P2 is represented by the position coordinates of at least one of the joint points 0 to 17 in the reference image IM2.

[0056] For example, the information processing device 300 calculates, for the reference image IM3, an angle α31 between the 0-1 line and a line parallel to the Y axis, an angle α32 between the 9-10 line and a line parallel to the Y axis, and an angle α33 between the 12-13 line and a line parallel to the Y axis. The information processing device 300 stores the angles α31, α32, and α33 in association with a position P3. The position P3 is represented by the position coordinates of at least one of the joint points 0 to 17 in the reference image IM3.

[0057] In this manner, the information processing device 300 generates distortion information by, for example, associating the angles α11, α12, and α13 with the position P1, associating the angles α21, α22, and α23 with the position P2, and associating the angles α31, α32, and α33 with the position P3. The information processing device 300 stores the generated distortion information in the auxiliary storage unit 340.

[0058] [Functions of the information processing device 300] 10 is a block diagram illustrating an example of the functional configuration of information processing device 300. Information processing device 300 functions as, for example, an acquisition unit 311, a generation unit 312, a fall detection unit 313, and an output unit 314. These functions are realized by CPU 310 executing a program.

[0059] The acquisition unit 311 acquires target image information and distortion information related to a target image. The target image is an image of the subject 400 captured by the imaging device 200. The acquisition unit 311 acquires the target image information, for example, by receiving the target image from the imaging device 200 via the communication I / F 350. As described above for the reference image, the acquisition unit 311 may acquire the target image information from the auxiliary storage unit 340 or an external storage device, or may acquire the target image information offline. As described above for the reference image, the acquisition unit 311 may perform predetermined preprocessing on the target image. The acquisition unit 311 acquires distortion information from, for example, the auxiliary storage unit 340. The acquisition unit 311 may also acquire distortion information from an external storage device, etc.

[0060] 11 shows an example of a target image IM4 captured by the imaging device 200. In the target image IM4, a target person 400 in a standing position is captured at a position P4. The acquisition unit 311 acquires, for example, target image information relating to the target image IM4.

[0061] The generation unit 312 generates target position information and target skeletal information based on the target image information acquired by the acquisition unit 311. The target skeletal information is information about the skeletal state of the target person 400 in the target image IM4. The target skeletal information may include, for example, information about the positions of each of a plurality of joint points, such as the eyes, nose, neck, shoulders, elbows, wrists, waist, knees, and ankles, which are skeletal feature points of the target person 400, and information about line segments connecting the joint points. The information about the positions of each of the plurality of joint points is, for example, the position coordinates of each of the plurality of joint points in the target image IM4. The position coordinates are expressed, for example, by X and Y coordinates set in the target image IM4. The target position information of the target person 400 is information about position P4 and may include, for example, information about the position coordinates of one or more joint points of the target person 400. The generation unit 312 generates the target position information and the target skeletal information, for example, using a method similar to that used for the reference position information and reference skeletal information.

[0062] The fall detection unit 313 detects a fall of the subject 400 based on the subject position information and subject skeletal information generated by the generation unit 312 and the distortion information acquired by the acquisition unit 311. The fall detection unit 313 detects a fall of the subject 400, for example, when the angle between a predetermined line formed by the subject 400's skeleton in the target image IM4 and a reference line set in the target image IM4 exceeds a threshold. The reference line is, for example, a straight line parallel to the Y axis. The predetermined line formed by the subject 400's skeleton is preferably the same as the line used to generate the distortion information. The predetermined line formed by the subject 400's skeleton includes, for example, a 0-1 line connecting joint point 0 and joint point 1, a 9-10 line connecting joint point 9 and joint point 10, and a 12-13 line connecting joint point 12 and joint point 13.

[0063] The fall detection unit 313 calculates the angle α41 between the 0-1 line and a line parallel to the Y axis, the angle α42 between the 9-10 line and a line parallel to the Y axis, and the angle α43 between the 12-13 line and a line parallel to the Y axis, and compares these with the respective thresholds. The fall detection unit 313 detects a fall of the subject 400, for example, when at least one of the angles α41, α42, and α43 exceeds the threshold. The fall detection unit 313 may also detect a fall of the subject 400 when all of the angles α41, α42, and α43 exceed the threshold.

[0064] In this embodiment, this threshold is determined for each position of the target image IM4 based on the distortion information. As will be described in detail later, this allows the fall of the target person 400 to be detected without a decrease in accuracy even if the image capturing device 200 includes the wide-angle camera 230.

[0065] FIG. 12 shows an example of the relationship between coordinate positions in target image IM4 and correction angles for the threshold. For example, when the coordinate position (X, Y) of joint point 0 in target image IM4 is (6, 1), the fall detection unit 313 compares the threshold with angle α41 without correcting it. For example, when the coordinate position (X, Y) of joint point 0 in target image IM4 is (2, 5), the fall detection unit 313 corrects the threshold by −4 degrees and compares it with angle α41. In this way, the distortion information may associate each position in the image with a correction angle for the threshold. The threshold when the correction angle is 0 degrees is stored in, for example, the auxiliary storage unit 340.

[0066] When the fall detection unit 313 detects that the subject 400 has fallen, the output unit 314 outputs fall information regarding the fall of the subject 400. The output unit 314 outputs the fall information, for example, by transmitting the fall information to the alarm 370. The output unit 314 may output the fall information by transmitting the fall information to another terminal device through communication via a network. For example, the output unit 314 may send an email with an image of the subject 400 who has fallen to the other terminal device. The image to be transmitted may be the captured image itself, or may be an image that has been processed from the captured image.

[0067] The output unit 314 may output the fall information by transmitting the fall information to the operation and display unit 360. Upon receiving the fall information, the operation and display unit 360 displays, for example, a warning screen and an image of the subject person 400 who has fallen.

[0068] [Method for detecting a fall of the information processing device 300] Fig. 13 is a flowchart illustrating the processing procedure of a fall detection method by information processing device 300. That is, Fig. 13 shows an information processing method executed by information processing device 300. The processing of the flowchart shown in the figure is realized by CPU 310 executing an information processing program.

[0069] (Step S101) The information processing device 300 first acquires target image information and distortion information. The information processing device 300 acquires the target image information from the imaging device 200 and the distortion information from the auxiliary storage unit 340, for example.

[0070] (Step S102) Next, the information processing device 300 generates target position information and target skeletal information based on the target image information acquired in step S101. The information processing device 300 generates the target position information and target skeletal information, for example, by detecting joint points of the target person 400 in the target image. For example, the information processing device 300 sets a detection frame in the target image and detects joint points of the target person 400 that appear within this detection frame.

[0071] (Step S103) Next, the information processing device 300 determines an angle threshold for detecting a fall of the subject 400 based on the subject position information generated in step S102 and the distortion information acquired in step S101. For example, as shown in Fig. 12, the distortion information associates a position in the image with a correction angle for the threshold. The information processing device 300 obtains a correction angle according to the position of the subject 400 in the subject image and determines the threshold.

[0072] (Step S104) Next, the information processing device 300 detects a fall of the subject 400. Specifically, when the angle formed between a predetermined line formed by the skeleton of the subject 400 in the target image and a reference line set in the target image exceeds the threshold determined in step S103, the information processing device 300 detects a fall of the subject 400. When the angle formed between the predetermined line formed by the skeleton of the subject 400 in the target image and the reference line set in the target image falls within the threshold determined in step S103, the information processing device 300 determines that a fall of the subject 400 has not been detected. When a fall of the subject 400 has not been detected, that is, when step S104 is NO, the information processing device 300 returns to the processing of step S101.

[0073] (Step S105) When a fall of the subject person 400 is detected, that is, when step S104 is YES, the information processing device 300 outputs fall information and ends the process.

[0074] [Operational Effects of the Information Processing Device 300 and the Fall Detection System 100] In the information processing device 300 and fall detection system 100 according to this embodiment, a fall of a subject is detected using distortion information in addition to subject position information and subject skeletal information. This distortion information reflects the distortion state of the image at each position within the image captured by the imaging device 200, including the wide-angle camera 230. This makes it possible to detect the fall of the subject 400 with high accuracy even if the subject 400 falls in a distorted portion of the target image captured by the imaging device 200. Therefore, it is possible to detect the fall of the subject 400 using the wide-angle camera 230. The effects of this are described below.

[0075] 5A to 7B, in an image captured using a wide-angle camera, image distortion tends to occur as the distance from the center of the image increases. For this reason, when a subject in a standing position is captured at the edge of an image captured with a wide-angle camera, the system may recognize the subject as leaning, which may result in a false detection of the subject having fallen.

[0076] In contrast, the information processing device 300 and the fall detection system 100 use distortion information to detect a fall of the subject 400. In other words, the information processing device 300 and the fall detection system 100 detect a fall of the subject 400 by taking into account distortion in the image captured by the imaging device 200. Therefore, the information processing device 300 and the fall detection system 100 can detect a fall of the subject 400 with high accuracy from the target image captured by the imaging device 200 including the wide-angle camera 230.

[0077] Moreover, the distortion information is generated from a reference image captured by the imaging device 200. Therefore, the information processing device 300 can generate distortion information corresponding to various wide-angle cameras 230. This improves the degree of freedom in selecting the wide-angle camera 230.

[0078] Furthermore, the strain information is stored in advance in the auxiliary storage unit 340, etc. This allows the user to easily and accurately know that the subject person 400 has fallen, without having to perform complex operations.

[0079] As described above, the information processing device 300 and fall detection system 100 according to this embodiment detect a fall of a subject using distortion information in addition to subject position information and subject skeletal information. This makes it possible to detect a fall of the subject 400 with high accuracy even if the subject 400 falls in a distorted part of the target image captured by the imaging device 200. Therefore, it is possible to detect a fall of the subject 400 using a wide-angle camera.

[0080] Modifications of the above embodiment will be described below, but the same components as those in the above embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0081] <Variation 1> 14 shows the functional configuration of an information processing device 300 according to Modification 1. This information processing device 300 functions as a change unit 315 in addition to an acquisition unit 311, a generation unit 312, a fall detection unit 313, and an output unit 314. Except for this point, the information processing device 300 according to this modification has the same configuration as the information processing device 300 described in the above embodiment, and achieves the same effects.

[0082] The modification unit 315, for example, modifies the distortion information stored in the auxiliary storage unit 340. The modification unit 315 modifies the distortion information using, for example, a target image captured by the imaging device 200. Specifically, the modification unit 315 modifies the distortion information based on target position information and target skeletal information generated from target image information.

[0083] FIG. 15 is a flowchart illustrating a processing procedure of a fall detection method performed by the information processing device 300.

[0084] (Steps S201 to S203) The information processing device 300 performs the processes of steps S201 to S203 in the same manner as steps S101 to S103 described in the above embodiment.

[0085] (Step S204) Next, the information processing device 300 detects a fall of the subject 400. Specifically, when the angle between a predetermined line formed by the skeleton of the subject 400 in the target image and a reference line set in the target image exceeds the threshold determined in step S203, the information processing device 300 detects a fall of the subject 400. When the angle between the predetermined line formed by the skeleton of the subject 400 in the target image and a reference line set in the target image falls within the threshold determined in step S203, the information processing device 300 determines that a fall of the subject 400 has not been detected. If a fall of the subject 400 has not been detected, that is, if step S204 is NO, the information processing device 300 proceeds to processing in step S206. If a fall of the subject 400 has been detected, that is, if step S204 is YES, the information processing device 300 proceeds to processing in step S205.

[0086] (Step S205) The information processing device 300 outputs the fall information and ends the process.

[0087] (Step S206) The information processing device 300 stores the target position information and target skeleton information generated in the processing of step S202, for example, in the auxiliary storage unit 340. The information processing device 300 may store in the auxiliary storage unit 340 an angle formed between a predetermined line formed by the skeleton of the subject 400 in the target image and a reference line provided in the target image.

[0088] (Step S207) After storing the target position information and target skeletal information, the information processing device 300 determines whether or not a change in distortion information is necessary. For example, the information processing device 300 compares the target position information and target skeletal information stored in step S206 with the distortion information stored in the auxiliary storage unit 340 to determine whether or not a change in distortion information is necessary. When the information processing device 300 determines that a change in distortion information is unnecessary, i.e., when step S207 is NO, the information processing device 300 returns to the processing of step S201. When the information processing device 300 determines that a change in distortion information is necessary, i.e., when step S207 is YES, the information processing device 300 proceeds to the processing of step S208.

[0089] (Step S208) After changing the distortion information stored in the auxiliary storage unit 340, the information processing device 300 returns to the processing of step S201.

[0090] In the information processing device 300 according to this modification, a fall of the subject is detected using distortion information in addition to the subject position information and the subject skeletal information, as in the information processing device 300 according to the above embodiment. This makes it possible to detect the fall of the subject 400 with high accuracy even if the subject 400 falls in a distorted part of the target image captured by the imaging device 200. Therefore, it is possible to detect the fall of the subject 400 using a wide-angle camera.

[0091] Furthermore, in this information processing device 300, the distortion information is changed based on the target position information and target skeletal information generated from the target image. Therefore, the distortion information can be updated in response to changes in the imaging device 200, etc., and a fall of the target person 400 can be detected with higher accuracy.

[0092] <Variation 2> 16 shows the functional configuration of an information processing device 300 according to Modification 2. This information processing device 300 functions as an acquisition unit 311, a generation unit 312, a fall detection unit 313, and an output unit 314, as well as a change unit 315 and an image generation unit 316. Except for this point, the information processing device 300 according to this modification has the same configuration as the information processing device 300 described in the above embodiment, and achieves the same effects.

[0093] For example, the change unit 315 changes the distortion information stored in the auxiliary storage unit 340. For example, the change unit 315 changes the distortion information in response to an input from the user. The user inputs a change to the distortion information via the operation display unit 360, for example.

[0094] The image generation unit 316 generates a distortion response image that visualizes the distortion information. The distortion response image includes, for example, at least one of a grayscale image and a color scale image. The output unit 314 outputs the distortion response image generated by the image generation unit 316. For example, the output unit 314 outputs the distortion response image by transmitting the distortion response image to the operation and display unit 360. The operation and display unit 360 displays the received distortion response image on a display.

[0095] 17 shows an example of a distortion correspondence image 3161 displayed on the operation display unit 360. This distortion correspondence image 3161 is a grayscale image. The horizontal axis of the distortion correspondence image 3161 corresponds to the X coordinate of the image, and the vertical axis corresponds to the Y coordinate of the image. In this distortion correspondence image 3161, the direction and magnitude of the correction angle are represented by the shading of each area.

[0096] For example, the user inputs a change to the distortion information while checking this distortion-related image 3161. For example, the user selects the area (X, Y) (1, 3) and inputs a change from a correction angle of -2 degrees to a correction angle of -4 degrees. For example, after this input, the user selects the apply button, and the change unit 315 changes the distortion information.

[0097] In the information processing device 300 according to this modification, a fall of the subject is detected using distortion information in addition to the subject position information and the subject skeletal information, as in the information processing device 300 according to the above embodiment. This makes it possible to detect the fall of the subject 400 with high accuracy even if the subject 400 falls in a distorted part of the target image captured by the imaging device 200. Therefore, it is possible to detect the fall of the subject 400 using a wide-angle camera.

[0098] Furthermore, the information processing device 300 generates a distortion correspondence image that visualizes the distortion information. Therefore, the user can input changes to the distortion information while checking this distortion correspondence image. Therefore, the user can easily adjust the distortion information.

[0099] The configuration of the fall detection system 100 described above is a description of the main configuration in explaining the features of the above embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims. Furthermore, configurations that are included in general fall detection systems are not excluded.

[0100] For example, in the above embodiment, a case where a person's joint points are detected by machine learning has been described, but the present invention is not limited to this case. For example, a configuration may be adopted in which a person's joint points are detected by pattern matching using feature points.

[0101] In the above example, a person is used as an example of a target for detecting a fall, but the present invention is not limited to this. The target may be, for example, an animal, a robot, or a structure having a skeleton.

[0102] In addition, although the above example shows a target image and a reference image each showing one person, the target image and the reference image may show multiple people, and the fall detection system 100 may detect falls of multiple subjects.

[0103] Furthermore, the distortion information may be generated by a device other than the information processing device 300. For example, the image capturing device 200 may generate the distortion information, or an external server or the like may generate the distortion information.

[0104] Furthermore, the number of predetermined lines formed by the skeleton of subject 400 may be one, or may be four or more. The predetermined lines formed by the skeleton of subject 400 may be lines other than the lines shown as examples. The reference line may be a straight line parallel to the X-axis, or may be another straight line.

[0105] The means and methods for performing the various processes in the fall detection system 100 described above can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred to and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a server or other device.

[0106] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims. [Explanation of symbols]

[0107] 100 Fall detection system, 200 imaging device, 201 bus, 210 control section, 220 Communications Department, 230 cameras, 300 information processing device, 310 CPUs, 320 RAM, 330 ROM, 340 Auxiliary storage unit, 350 communication I / F, 360 operation display section, 311 Acquisition Department; 312 generation section, 313 Fall detection unit, 314 output section, 315 Changes Division, 316 Image generation unit.

Claims

1. an acquisition unit that acquires target image information relating to a target image of a target person photographed by a wide-angle camera and distortion information relating to a distortion state of the image at each position within the image photographed by the wide-angle camera; a generating unit that generates, based on the acquired target image information, target position information relating to a position of the target person in the target image and target skeletal information relating to a skeletal state of the target person in the target image; a fall detection unit that detects a fall of the subject based on the generated subject position information, the subject skeletal information, and the distortion information; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the fall detection unit detects a fall of the subject when an angle formed between a predetermined line formed by the subject's skeleton in the target image and a reference line set in the target image exceeds a threshold value.

3. The information processing apparatus according to claim 2 , wherein the fall detection unit determines the threshold value based on the target position information and the distortion information.

4. 4. The information processing device according to claim 3, wherein the distortion information is generated based on reference position information relating to the position of the object in a reference image captured by the wide-angle camera, and reference skeleton information relating to the skeleton state of the object in the reference image.

5. The information processing apparatus according to claim 2 , wherein the distortion information associates each position in the image with a correction angle of the threshold value.

6. The information processing apparatus according to claim 1 , wherein the distortion information is stored in a storage unit.

7. The information processing apparatus according to claim 6 , further comprising a change unit that changes the distortion information stored in the storage unit.

8. The information processing apparatus according to claim 7 , wherein the change unit changes the distortion information based on the target position information and the target skeletal information.

9. The information processing apparatus according to claim 7 , wherein the change unit changes the distortion information in response to an input from a user.

10. The information processing device according to claim 1 , further comprising an output unit that outputs fall information regarding a detected fall of the subject.

11. an image generating unit that generates a distortion-corresponding image by visualizing the distortion information; The information processing apparatus according to claim 10 , wherein the output unit further outputs the generated distortion-adjusted image.

12. The information processing apparatus according to claim 11 , wherein the distortion-corresponding image includes at least one of a grayscale image and a colorscale image.

13. An information processing method by an information processing device, Acquiring target image information relating to a target image of a target person photographed by a wide-angle camera and distortion information relating to a distortion state of the image at each position within the image photographed by the wide-angle camera; generating target position information relating to a position of the target person in the target image and target skeletal information relating to a skeletal state of the target person in the target image based on the acquired target image information; Detecting a fall of the subject based on the generated subject position information, the subject skeletal information, and the distortion information; An information processing method including:

14. An information processing program for causing the information processing device to execute a process including the information processing method according to claim 13.

Images