Fall prevention system

Abstract

A fall prevention system (1) is characterized by being provided with: an imaging unit (2) that captures an image of a space to be monitored; a skeleton model generation unit (8b) that generates a skeleton model indicating a person included in the image captured by the imaging unit (2); a determination unit (8c) that determines, on the basis of the skeleton model generated by the skeleton model generation unit (8b), a state in which the person stands and a state in which the person sits separately as a state of the person corresponding to the skeleton model; and an operation processing unit (8d) capable of executing, on the basis of the determination result of the determination unit (8c), a fall prevention process for preventing a person from falling, which is a process corresponding to the determination result.

Description

Technical Field

[0001] The invention relates to a fall prevention system. Background Art

[0002] As a technology related to the conventional fall prevention system, for example, Patent Document 1 discloses a fall detection device that extracts appearance features based on a human body image representing a detection target, ie, a human body, and detects a human fall based on the appearance features.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6217635 Summary of the invention

[0006] Technical problem that the invention aims to solve

[0007] However, the fall detection device described in Patent Document 1 described above has room for further improvement, for example, in terms of more appropriately preventing falls.

[0008] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a fall prevention system that can appropriately prevent a fall.

[0009] Technical solutions to the problem

[0010] In order to achieve the above-mentioned purpose, the anti-fall system involved in the present invention is characterized in that it comprises: a shooting unit, which shoots an image of a monitored object space; a skeleton model generating unit, which generates a skeleton model representing a person contained in the image captured by the shooting unit; a judgment unit, which distinguishes between a standing state of the person and a sitting state of the person as the state of the person corresponding to the skeleton model based on the skeleton model generated by the skeleton model generating unit; and an action processing unit, which, based on the judgment result of the judgment unit, can execute a processing corresponding to the judgment result, i.e., an anti-fall processing for preventing the person from falling.

[0011] In addition, the above-mentioned anti-fall system may be equipped with a guiding unit for guiding, and when the judgment unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit executes the processing of controlling the guiding unit and guiding the person to sit down as the anti-fall processing.

[0012] In addition, the above-mentioned fall prevention system may be equipped with a notification unit for notification, and when the determination unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit executes a first notification process of controlling the notification unit and notifying other people different from the person that the person is in a standing state as the fall prevention process.

[0013] In addition, the above-mentioned anti-fall system may be equipped with an external communication unit, which can communicate with an external device outside the monitored object space. After the action processing unit executes the guidance processing, when the judgment unit determines that the person corresponding to the skeleton model is in a standing state after a predetermined period of time, the action processing unit executes a first notification processing as the anti-fall processing to control the external communication unit and notify the external device that the person is in a standing state.

[0014] In addition, the above-mentioned anti-fall system may be provided with an internal communication unit, and the internal communication unit can communicate with a vehicle-mounted device, and the vehicle-mounted device is installed in a vehicle whose interior space is set as the monitored object space. When the determination unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit executes a prohibition start processing of controlling the vehicle-mounted device via the internal communication unit and prohibiting the starting of the vehicle as the anti-fall processing.

[0015] In addition, the above-mentioned anti-fall system may be provided with: a notification unit, which performs a notification action; and an external communication unit, which can communicate with an external device outside the monitored object space, wherein the determination unit, based on the skeleton model generated by the skeleton model generation unit, in addition to distinguishing between the standing state and the sitting state of the person, also distinguishes between the fallen state of the person as the state of the person corresponding to the skeleton model, and when the determination unit determines that the person corresponding to the skeleton model is in a fallen state, the action processing unit executes a second notification processing of controlling the notification unit and notifying other people different from the person that the person is in a fallen state, or executes a second notification processing of controlling the external communication unit and notifying the external device that the person is in a fallen state.

[0016] Effects of the Invention

[0017] The fall prevention system involved in the present invention captures an image of a monitored object space by a capturing unit, and generates a skeleton model representing a person included in the image by a skeleton model generating unit. Then, the determination unit distinguishes between a standing state of the person and a sitting state of the person as the state of the person corresponding to the skeleton model based on the skeleton model generated by the skeleton model generating unit. Moreover, the action processing unit can perform fall prevention processing corresponding to the determination result according to the determination result of the determination unit. In this case, the fall prevention system can determine the state of the person in more detail and accurately by distinguishing between a standing state of the person and a sitting state of the person based on the skeleton model as described above. As a result, the fall prevention system has the following effect, that is, it can appropriately implement fall prevention based on more accurate state determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a block diagram showing the schematic structure of the fall prevention system involved in the embodiment.

[0019] Figure 2 It is a schematic cross-sectional view showing an example of mounting the fall prevention system according to the embodiment.

[0020] Figure 3 A schematic diagram illustrating an example of state determination based on a skeleton model in the fall prevention system according to the embodiment.

[0021] Figure 4 It is a flowchart which shows an example of control of the fall prevention system involved in embodiment.

[0022] Figure 5 It is a flowchart which shows an example of control of the fall prevention system involved in embodiment.

[0023] Explanation of symbols

[0024] 1. Fall prevention system

[0025] 2. Filming Department

[0026] 3 Information output unit (guidance unit)

[0027] 3a, 203a Display

[0028] 3b, 203b Speakers

[0029] 4.204 Microphone

[0030] 5. Notification Department

[0031] 6 External Communications Department

[0032] 7 Internal Communications Department

[0033] 8.205 Control device

[0034] 8A, 205A interface

[0035] 8B, 205B Storage

[0036] 8C, 205C Processing Department

[0037] 8a Information Processing Department

[0038] 8b Skeleton model generation unit

[0039] 8c Judgment Department

[0040] 8d Motion Processing Unit

[0041] 100 ECU

[0042] 101 Vehicle Equipment

[0043] 201 External Devices

[0044] 202 Ministry of Communications

[0045] 203 Information output unit

[0046] BB Bounding Box

[0047] I Image

[0048] IN Vehicle Interior Space

[0049] MDL Skeleton Model

[0050] N Network

[0051] P, P1, P2 people

[0052] RF Roof Parts

[0053] S Seat

[0054] SP Monitoring target space

[0055] sys Collaboration System DETAILED DESCRIPTION

[0056] Hereinafter, the embodiments of the present invention will be described in detail based on the accompanying drawings. In addition, the present invention is not limited to the embodiments. In addition, the constituent elements in the following embodiments include constituent elements that can be easily replaced by those skilled in the art or substantially the same constituent elements.

[0057] [Implementation Method]

[0058] Figure 1 , Figure 2The fall prevention system 1 of the present embodiment shown is a system that monitors the state of a person P existing in a monitoring target space SP to prevent the person P from falling. The fall prevention system 1 of the present embodiment is a vehicle-mounted system mounted on a vehicle V that can carry and transport the person P. The vehicle interior space IN of the vehicle V is set as the monitoring target space SP of the fall prevention system 1. In other words, the monitoring target space SP of the present embodiment is the vehicle interior space IN of the vehicle V that can transport the person P.

[0059] The fall prevention system 1 of this embodiment is a structural module that monitors the above-mentioned monitoring target space SP and performs fall prevention processing corresponding to the state of the person P. Here, the fall prevention system 1 and the external device 201 installed outside the vehicle V together constitute a collaborative system sys. The collaborative system sys includes: the fall prevention system 1 installed in the vehicle V; and the external device 201, which is installed outside the vehicle V and can exchange information with the fall prevention system 1.

[0060] Furthermore, the fall prevention system 1 of the present embodiment is configured to generate a skeleton model MDL representing a person P (see Figure 3 ) and judges the state of the person P in the monitoring target space SP, thereby improving the accuracy of state judgment and realizing more appropriate fall prevention based on this. Figure 1 The components shown are implemented by being mounted on a vehicle V. Hereinafter, each structure of the fall prevention system 1 will be described in detail with reference to the drawings.

[0061] In addition, Figure 1 In the illustrated fall prevention system 1, the connection method between the components for power supply, control signals, and transmission and reception of various information, etc., can be any of a wired connection and a wireless connection unless otherwise specified. A wired connection refers to a connection via wiring materials such as electric wires and optical fibers. A wireless connection refers to a connection based on wireless communication, contactless power supply, etc.

[0062] In addition, the vehicle V to which the anti-fall system 1 is applied may be any vehicle that uses an electric motor or an engine as a driving source, such as an electric vehicle (EV), a hybrid vehicle (HEV), a plug-in hybrid vehicle (PHEV), a gasoline vehicle, a diesel vehicle, etc. In addition, the driving of the vehicle V may be any of manual driving by a driver, semi-automatic driving, and fully automatic driving. In addition, the vehicle V may be any of a so-called private car owned by an individual, a rental car, a shared car, a bus, a taxi, and a shared car. In the following description, as an example, the vehicle V is a bus, a taxi, a shared car, etc., which is managed by a specific operator, etc., and is used by an unspecified number of users and automatically moves to the boarding and alighting locations reserved by the users, and is described. In the case of fully automatic driving, there may also be a situation where the driver himself is not in the vehicle V. In addition, the external device 201 is, for example, set in a facility of an operator of the operating vehicle V, etc.

[0063] Specifically, if Figure 1 , Figure 2 As shown, the fall prevention system 1 includes a photographing unit 2, an information output unit 3, a microphone 4, a notification unit 5, an external communication unit 6, an internal communication unit 7, and a control device 8. The information output unit 3, the notification unit 5, the external communication unit 6, the internal communication unit 7, etc. constitute an action unit that performs various fall prevention actions to prevent the person P from falling in the fall prevention system 1. The fall prevention system 1 can be, for example, an overhead module (OHM: Over-Head Module) or a roof module (RM: Roof Module) that integrates various functions by assembling these structural elements in a shell, etc. and unitizing them and then setting them on the roof part (roof) RF of the vehicle V. In addition, the fall prevention system 1 can also be, for example, each part can be set separately on the vehicle V.

[0064] The photographing unit 2 photographs an image of the monitored space SP. The photographing unit 2 may be, for example, a monocular camera capable of photographing a two-dimensional image, or a stereo camera capable of photographing a three-dimensional image. In addition, the photographing unit 2 may be a so-called TOF (Time of Flight) camera or the like. The photographing unit 2 is disposed at a position where it can photograph all passengers in the vehicle interior space IN constituting the monitored space SP. The photographing unit 2 is, for example, disposed above the vehicle interior space IN of the vehicle V, and is disposed on the roof member RF, and the field of view angle is set in such a manner that the entire area of ​​the vehicle interior space IN is included in the photographing range. The photographing unit 2 may also be configured such that, when one photographing unit 2 cannot cover the entire area of ​​the vehicle interior space IN, a plurality of photographing units 2 are provided, and the entire area of ​​the vehicle interior space IN is covered by the plurality of photographing units 2. The image captured by the photographing unit 2 is typically used by the control device 8 for determining the state of the person P.

[0065] The information output unit 3 can output various information toward the vehicle interior space IN (monitored space SP). The information output unit 3 of this embodiment constitutes a guidance unit that performs the guidance action described later, and performs various guidance (broadcasting) by outputting various information. As an example, the information output unit 3 is configured to include a display 3a and a speaker 3b.

[0066] The display 3a displays (outputs) image information (visual information) toward the vehicle interior space IN. The display 3a is composed of, for example, a thin liquid crystal display, a plasma display, an organic EL display, etc. The display 3a displays the image information at a position where the person P in the vehicle interior space IN can see it.

[0067] The speaker 3b outputs sound information (auditory information) toward the vehicle interior space IN. For example, the speaker 3b may be a speaker with high directivity so that only a specific person P who wishes to be presented with the information in the vehicle interior space IN can hear the sound information.

[0068] Furthermore, the information output unit 3 may be configured to include a display lamp, an indicator, and the like in addition to the display 3 a and the speaker 3 b .

[0069] The microphone 4 is a sound collecting device that converts the sound generated in the vehicle interior space IN into an electric signal. The microphone 4 can be used, for example, to communicate with a person (for example, an operator described later) outside the vehicle interior space IN.

[0070] The notification unit 5 performs the notification operation described later, and can notify various information to the person P inside and outside the vehicle interior space IN (monitored space SP). The notification unit 5 typically outputs various information to a person different from the person P monitored by the fall prevention system 1, such as a person in charge of service in the vehicle V, a driver in the case where the vehicle V is a manual drive vehicle, etc., thereby performing various notifications. The notification unit 5 is similar to the above-mentioned information output unit 3, and is configured to include, for example, a speaker, a display, a display light, an indicator, etc. In addition, the notification unit 5 can also be used as the above-mentioned information output unit 3.

[0071] The external communication unit 6 is a wireless communication functional component mounted on the vehicle V and can communicate with an external device 201 outside the vehicle interior space IN (monitored space SP). The external communication unit 6 is a communication module (Data Communication Module) that connects the vehicle V to a network N outside the vehicle V by wireless.

[0072] Here, the network N connects the fall prevention system 1 and the external device 201 of the vehicle V in a communicable manner. The network N can use any communication network regardless of whether it is wired or wireless. The external communication unit 6 transmits and receives information between the device connected to the network N and the fall prevention system 1 .

[0073] The external communication unit 6 can communicate with devices outside the vehicle V via the network N through various wireless communication methods such as wide-area wireless and narrow-area wireless. Here, the wide-area wireless method is, for example, radio (AM, FM), TV (UHF, 4K, 8K), TEL, GPS, WiMAX (registered trademark), etc. In addition, the narrow-area wireless method is, for example, ETC / DSRC, VICS (registered trademark), wireless LAN, millimeter wave communication, etc. The external communication unit 6 of the present embodiment at least sends and receives information with the external device 201 connected to the network N via the network N. That is, the external communication unit 6 of the present embodiment is configured to be able to communicate with at least the external device 201 that is a device outside the vehicle V.

[0074] The internal communication unit 7 is a communication relay function component mounted on the vehicle V, and can communicate with the vehicle-mounted device 101 mounted on the vehicle V. The internal communication unit 7 is configured to be able to communicate with the ECU (Electronic Control Unit) 100 and the vehicle-mounted device 101 mounted on the vehicle V via the vehicle network or the like. That is, the internal communication unit 7 is configured to be able to send and receive various information with the ECU 100 and the vehicle-mounted device 101 mounted on the vehicle V.

[0075] Here, ECU 100 is an electronic control unit that performs unified control of the entire vehicle V including the vehicle-mounted devices 101 and the like. The ECU 100 may also be configured as a part of the anti-fall system 1. In other words, the anti-fall system 1 may also be configured as a structure having ECU 100. The vehicle-mounted devices 101 are devices that are mounted on the vehicle V and implement various functions. The internal communication unit 7 transmits and receives information between the multiple vehicle-mounted devices 101 connected to the ECU 100 via the ECU 100. The multiple vehicle-mounted devices 101 of this embodiment are configured to include, for example, a travel system actuator as various devices for driving the vehicle V. The travel system actuator is typically configured to include a travel powertrain, a steering device, a braking device, and the like. The travel powertrain is a drive device that drives the vehicle V. The steering device is a device that steers the vehicle V. The braking device is a device that brakes the vehicle V. In addition, the vehicle-mounted devices 101 may also be configured to include lighting equipment, air conditioning equipment, display equipment, audio equipment, and the like of the vehicle V.

[0076] The control device 8 controls the various parts of the anti-fall system 1 in a unified manner. The control device 8 monitors the status of the vehicle interior space IN and executes various calculation processes for various response processes according to the status. The control device 8 is configured as an electronic circuit having a well-known microcomputer as the main body including a central calculation processing unit including a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc., a ROM (Read Only Memory), a RAM (Random Access Memory), and an interface. The control device 8 is electrically connected to the imaging unit 2, the information output unit 3, the microphone 4, the notification unit 5, the external communication unit 6, the internal communication unit 7, etc. The control device 8 can send and receive various detection signals, various electrical signals such as drive signals for driving each part, etc. between the various parts.

[0077] Specifically, the control device 8 is configured to include an interface unit 8A, a storage unit 8B, and a processing unit 8C in terms of functional concept. The interface unit 8A, the storage unit 8B, and the processing unit 8C can mutually transmit and receive various information with various electrically connected devices.

[0078] The interface unit 8A is an interface for transmitting and receiving various information with each unit of the fall prevention system 1, such as the imaging unit 2, the information output unit 3, the microphone 4, the notification unit 5, the external communication unit 6, and the internal communication unit 7. The interface unit 8A has a function of performing wired communication of information with each unit via a wire or the like, a function of performing wireless communication of information with each unit via a wireless communication unit or the like, and the like.

[0079] The storage unit 8B is a storage device that stores various information. The storage unit 8B can be, for example, a relatively large-capacity storage device such as a hard disk, SSD (Solid State Drive), or an optical disk, or a semiconductor memory that can rewrite data such as RAM, flash memory, or NVSRAM (NonVolatile Static Random Access Memory). The storage unit 8B stores conditions and information required for various processes in the control device 8, various programs, applications, control data, etc. executed by the control device 8. The storage unit 8B stores, for example, a learned mathematical model used for determining the state of a person P in the vehicle interior space IN (monitored space SP). In addition, the storage unit 8B can also temporarily store, for example, image information of an image captured by the imaging unit 2, output information output from the information output unit 3 and the notification unit 5, and various information received by the external communication unit 6 and the internal communication unit 7. The storage unit 8B reads this information as needed through the processing unit 8C, etc.

[0080] The processing unit 8C is a unit that executes various programs stored in the storage unit 8B based on various input signals and the like, outputs output signals to various units by operating the programs, and performs various processes for realizing various functions.

[0081] The processing unit 8C of this embodiment can use the skeleton model MDL (see Figure 3 ) to determine the state of the person P existing in the vehicle interior space IN (monitoring object space SP), and execute fall prevention processing corresponding to the determination result.

[0082] In order to execute the above-mentioned various processes, the processing unit 8C of this embodiment is configured to include an information processing unit 8a, a skeleton model generating unit 8b, a determining unit 8c, and an action processing unit 8d in terms of functional concept.

[0083] The information processing unit 8a is a part having a function of being able to perform processing related to various information used in the fall prevention system 1. For example, the information processing unit 8a controls the imaging unit 2 and performs processing for capturing an image of the vehicle interior space IN (monitoring target space SP) of the vehicle V. Then, the information processing unit 8a performs processing for storing the image information of the vehicle interior space IN captured by the imaging unit 2 in the storage unit 8B.

[0084] In addition, the information processing unit 8a controls the external communication unit 6 and can perform processing for sending and receiving various information between the external device 201. The information processing unit 8a can perform processing for reading information sent by the external communication unit 6 from the storage unit 8B and for causing the storage unit 8B to store information received by the external communication unit 6. For example, the information processing unit 8a can perform processing for sending image information captured by the camera 2 and sound information collected by the microphone 4 to the external device 201 via the external communication unit 6. Moreover, the information processing unit 8a can perform processing for receiving output information to be output by the display 3a and the speaker 3b from the external device 201 via the external communication unit 6. The fall prevention system 1 can exchange information (for example, voice information or image information, etc.) with the external device 201 through these processes implemented by the information processing unit 8a.

[0085] The information processing unit 8a controls the internal communication unit 7 and can perform processing for transmitting and receiving various information with the vehicle-mounted device 101. The information processing unit 8a performs processing for reading information to be transmitted by the internal communication unit 7 from the storage unit 8B and for causing the storage unit 8B to store information received by the internal communication unit 7. The information processing unit 8a performs processing for transmitting and receiving various information with the vehicle-mounted device 101 via the ECU 100, for example. For example, the information processing unit 8a performs processing for transmitting information for causing the vehicle-mounted device 101 to operate via the action processing unit 8d described later to the vehicle-mounted device 101 via the internal communication unit 7.

[0086] The skeleton model generating unit 8b is a unit capable of executing the generation of the skeleton model MDL (see Figure 3 ), the skeleton model MDL represents the person P included in the image of the vehicle interior space IN (monitored space SP) captured by the imaging unit 2. The skeleton model MDL is a human body model that three-dimensionally represents the human skeleton including the head, eyes, nose, mouth, shoulders, waist, feet, knees, elbows, hands, and various joints of the person P.

[0087] Figure 3 This is an example of an image I of the vehicle interior space IN captured by the imaging unit 2. The skeleton model generating unit 8b estimates and generates a skeleton model MDL representing a person P included in the image I captured by the imaging unit 2 using various well-known methods such as background difference method, mean shift method, pattern matching, and various machine learning methods.

[0088] For example, after first detecting the person P, the skeleton model generation unit 8b can generate a skeleton model MDL representing the person P included in the image I by estimating the skeleton of the person P from top to bottom. In this case, the skeleton model generation unit 8b uses various well-known object recognition technologies to identify the person P in the image I, and performs processing to surround the outside of the area where the identified person P exists with a bounding box BB in the image I. Here, the bounding box BB refers to a rectangular frame with a size necessary to surround the person P identified in the image I. Then, the skeleton model generation unit 8b detects the three-dimensional position coordinates of various skeletal parts (human body parts) of the human body, such as the head, eyes, nose, mouth, shoulders, waist, feet, knees, elbows, hands, and various joints of the person P in the bounding box BB, and combines them to generate the skeleton model MDL of the person P. Figure 3 The skeleton model MDL shown as an example is generated by symbolically representing various skeleton parts of the human body such as the head, eyes, nose, mouth, shoulders, waist, feet, knees, elbows, hands, and joints of the person P with "points" and connecting these points with "lines". In the case where the image I includes multiple persons P, the skeleton model generation unit 8b generates multiple skeleton models MDL according to the number of the persons P.

[0089] In addition, the skeleton model generation unit 8b may also generate a skeleton model representing the person P included in the image I by estimating the skeleton of the person P from the bottom up after detecting all the skeleton parts of the human body in the image I, for example, without using the bounding box BB, etc. In this case, the skeleton model generation unit 8b first detects the three-dimensional position coordinates of all the skeleton parts such as the head, eyes, nose, mouth, shoulders, waist, feet, knees, elbows, hands, and joints of the human body in the image I using various known object recognition technologies. After that, the skeleton model generation unit 8b generates the skeleton model MDL of each person P by matching and connecting the detected skeleton parts for each person P.

[0090] The determination unit 8c is a part having a function of executing a process of determining the state of the person P corresponding to the skeletal model MDL generated by the skeletal model generation unit 8b based on the skeletal model MDL. As the state of the person P corresponding to the skeletal model MDL generated by the skeletal model generation unit 8b, the determination unit 8c of the present embodiment distinguishes between a state in which the person P is standing and a state in which the person P is sitting. Figure 3 The person P1 shown is in a standing position. Figure 3The person P2 shown is in a state of sitting on a seat S in the vehicle interior space IN. As the state of the person P corresponding to the skeleton model MDL, the determination unit 8c of the present embodiment distinguishes between the state of the person P standing and the state of the person P sitting and the state of the person P falling down based on the skeleton model MDL generated by the skeleton model generation unit 8b. When the person P is included in the image I of the vehicle interior space IN (monitoring target space SP) captured by the imaging unit 2, the determination unit 8c performs these state determinations on the person P.

[0091] The anti-fall system 1, for example, uses the relative position relationship, relative distance, size of the bounding box BB, etc. of each skeletal part in the skeletal model MDL as parameters, and learns the standing state, sitting state, and falling state of the person P in advance through various machine learning. For example, the anti-fall system 1 uses the "relative position relationship, relative distance, size of the bounding box BB, etc. of each skeletal part in the skeletal model MDL" collected in advance as explanatory variables, and uses "is it a standing state, is it a sitting state, is it a falling state" as the target variable to perform machine learning. As machine learning, for example, various forms of algorithms that can be applied to the present embodiment, such as logistic regression, support vector machine, neural network, random forest, etc., can be used. The anti-fall system 1 stores the mathematical model for state judgment obtained through the machine learning in advance in the storage unit 8B.

[0092] Then, the determination unit 8c determines the state of the person corresponding to the skeleton model MDL by classification / regression based on the learned state determination mathematical model stored in the storage unit 8B as described above. More specifically, the determination unit 8c inputs the relative position relationship, relative distance, and size of the bounding box BB of each skeletal part obtained from the skeleton model MDL of the person P included in the actually captured image I into the mathematical model for state determination. Thus, as the state of the person P corresponding to the skeleton model MDL, the determination unit 8c distinguishes between the state of the person P standing, the state of the person P sitting, and the state of the person P falling.

[0093] The action processing unit 8d is a unit having a function of executing a process for controlling the actions of each unit of the fall prevention system 1. The action processing unit 8d of this embodiment can execute a fall prevention process for preventing the person P from falling corresponding to the determination result of the determination unit 8c. The action processing unit 8d controls the information output unit 3, the notification unit 5, the external communication unit 6, the internal communication unit 7, etc. constituting the action unit based on the determination result of the determination unit, and executes the fall prevention process according to the determination result.

[0094] For example, after the person P gets on the vehicle V, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state, the action processing unit 8d performs a guidance process as a fall prevention process. The guidance process refers to a process of controlling the information output unit 3 to guide the person P to take a seat. The action processing unit 8d performs the guidance process as a fall prevention process, thereby, for example, controlling the display 3a constituting the information output unit 3, and implementing a guidance display that prompts the person P to take a seat and reminds the person to prevent falling through the display 3a. In addition, the action processing unit 8d may also perform a guidance process as a fall prevention process, for example, controlling the speaker 3b constituting the information output unit 3, and providing voice guidance to the person P through the speaker 3b to prompt the person P to take a seat and remind the person to prevent falling.

[0095] In this case, the action processing unit 8d may further perform a first notification process as a fall prevention process after executing the guidance process and after a predetermined period of time has passed, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state. The first notification process is a process of controlling the external communication unit 6 to notify the external device 201 that the person P is in a standing state.

[0096] Here, the external device 201 is briefly described. As described above, the external device 201 is, for example, installed in the facilities of the operator of the operating vehicle V. The external device 201 constitutes a cloud service device installed on the network N outside the vehicle V, and constitutes the so-called cloud service center. The external device 201 can send and receive information with the anti-fall system 1 (vehicle V) via the network N. The external device 201 can also be configured by installing applications that implement various processing in a known computer system such as a PC or workstation. The anti-fall system 1 can communicate and cooperate with the external device 201 via the external communication unit 6, thereby receiving various cloud services from the external device 201.

[0097] As an example, Figure 1As shown, the external device 201 includes a communication unit 202, an information output unit 203, a microphone 204, and a control device 205. The communication unit 202 is a communication module that can communicate with the network N, similar to the external communication unit 6. Here, it can communicate with the anti-fall system 1 via the network N in a wired / wireless manner. The information output unit 203 is similar to the information output unit 3, and can output various information. It is configured to include a display 203a, a speaker 203b, etc. The microphone 204 is a sound collection device that converts sound into an electrical signal, similar to the microphone 4, and is used to exchange sounds with the person P in the vehicle V. The control device 205 controls the various parts of the external device 201 in a unified manner. Similar to the control device 8, it is configured to include an interface unit 205A, a storage unit 205B, and a processing unit 205C in terms of functional concept. The interface unit 205A is an interface for sending and receiving various information with the various parts of the external device 201 such as the communication unit 202, the information output unit 203, and the microphone 204. The storage unit 205B is a storage device that stores various information. The processing unit 205C is a unit that executes various programs stored in the storage unit 205B based on various input signals, etc., and outputs output signals to each unit by running the programs, thereby executing various processes for realizing various functions.

[0098] The action processing unit 8d performs the first notification processing as a fall prevention processing, for example, notifying the operator of the operator, etc., via the external device 201 configured as described above that the person P sitting in the vehicle V is still in a standing state. In addition, the action processing unit 8d may also perform the processing of the information processing unit 8a or communicate with the external device 201 of the notification destination via the external communication unit 6 after the notification. In this case, the action processing unit 8d exchanges various information (for example, sound information, image information, etc.) between the external device 201 and the vehicle V via the information output unit 3, 203, the microphone 4, 204, etc., thereby enabling the person P on the vehicle V side and the operator, etc. on the external device 201 side to directly communicate with each other. As a result, the action processing unit 8d can, for example, directly remind the person P to take a seat from the operator, etc. on the external device 201 side.

[0099] In addition, for example, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state, the action processing unit 8d may perform the first notification processing as a fall prevention processing. The first notification processing refers to a processing for controlling the notification unit 5 to notify other persons different from the person P that the person P is in a standing state. The action processing unit 8d performs the first notification processing as a fall prevention processing, thereby, for example, controlling the speaker, display, display light, indicator, etc. constituting the notification unit 5 to notify other persons different from the person P, such as a person in charge of service in the vehicle V, a driver when the vehicle V is a manual drive vehicle, etc., that the person P is in a standing state.

[0100] Furthermore, the action processing unit 8d may, for example, execute a start prohibition process as a fall prevention process when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state. The start prohibition process refers to a process of prohibiting the start of the vehicle V by controlling the vehicle-mounted device 101 via the internal communication unit 7. The action processing unit 8d executes the start prohibition process as a fall prevention process, thereby controlling, for example, a travel system actuator constituting the vehicle-mounted device 101, so that the vehicle V does not start.

[0101] Furthermore, for example, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a fallen state, the operation processing unit 8d may execute the second notification process or the second notice process.

[0102] The second notification process is a process of controlling the notification unit 5 to notify other persons different from the person P that the person P is in a fallen state. The action processing unit 8d controls, for example, a speaker, a display, a display lamp, an indicator, etc. constituting the notification unit 5 by executing the second notification process, and notifies a person different from the person P, for example, a person providing service in the vehicle V, a driver in the case where the vehicle V is a manual drive vehicle, etc., that the person P is in a fallen state.

[0103] On the other hand, the second notification process is a process of controlling the external communication unit 6 to notify the external device 201 that the person P is in a fallen state. The action processing unit 8d performs the second notification process, thereby notifying the operator of the operator, etc., through the external device 201 configured as described above, that the person P riding in the vehicle V is in a fallen state. The action processing unit 8d may perform the process of the information processing unit 8a or communicate with the external device 201 of the notification destination through the external communication unit 6 after the notification, similarly to the case of the first notification process. Thus, the action processing unit 8d can allow the operator, etc. on the external device 201 side to directly confirm the state of the fallen person P, for example.

[0104] Next, refer to Figure 4 , Figure 5 An example of control in the fall prevention system 1 is described with reference to the flowchart of FIG. Figure 4 control and Figure 5 The control can be executed in parallel or one can be combined with the other.

[0105] exist Figure 4 In the control, first, when a person P is sitting in the vehicle interior space IN (monitored space SP) of the vehicle V, the information processing unit 8a of the control device 8 controls the shooting unit 2 to shoot an image I of the vehicle interior space IN, and stores the shot image information in the storage unit 8B (step S1).

[0106] Next, the skeleton model generation unit 8b of the control device 8 detects the positions of the skeleton parts of the person P through object recognition and skeleton estimation based on the image I of the vehicle interior space IN stored in the storage unit 8B (step S2), and generates the skeleton model MDL of the person P (step S3).

[0107] Next, the determination unit 8c of the control device 8 detects whether the person P corresponding to the skeleton model MDL is in a standing or sitting state based on the skeleton model MDL generated in the processing of step S3, using a mathematical model for state determination learned through machine learning, etc. (step S4).

[0108] Then, the determination unit 8c determines whether the person P corresponding to the skeleton model MDL is standing (step S5). If the determination unit 8c determines that the person P is not standing, that is, sitting (step S5: No), the control ends.

[0109] When the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state (step S5: Yes), the motion processing unit 8d of the control device 8 executes a fall prevention process (step S6). The motion processing unit 8d executes, for example, the above-mentioned guidance process, the first notification process, the start prohibition process, etc. as the fall prevention process.

[0110] Then, after the fall prevention process is executed in step S6, after a predetermined period (e.g., several minutes) has passed, the determination unit 8c determines whether the person P corresponding to the skeleton model MDL is still in a standing state (step S7). If the determination unit 8c determines that the person P is not in a standing state, that is, in a sitting state (step S7: No), the control ends.

[0111] When the determination unit 8c determines that the person P corresponding to the skeleton model MDL is still standing (step S7: Yes), the operation processing unit 8d executes the above-mentioned first notification process (step S8) as a fall prevention process and ends the control.

[0112] exist Figure 5 In the control, first, when a person P is sitting in the vehicle interior space IN (monitored space SP) of the vehicle V, the information processing unit 8a controls the shooting unit 2 to shoot an image I of the vehicle interior space IN, and stores the shot image information in the storage unit 8B (step S101).

[0113] Next, the skeleton model generation unit 8b detects the positions of the skeleton parts of the person P by object recognition and skeleton estimation based on the image I of the vehicle interior space IN stored in the storage unit 8B (step S102), and generates the skeleton model MDL of the person P (step S103).

[0114] Next, the determination unit 8c of the control device 8 detects whether the person P corresponding to the skeleton model MDL is in a fallen state based on the skeleton model MDL generated in the process of step S103, using a mathematical model for state determination learned through machine learning, etc. (step S104).

[0115] Then, the determination unit 8c determines whether the person P corresponding to the skeleton model MDL is in a fallen state (step S105). If the determination unit 8c determines that the person P is not in a fallen state, that is, is in a sitting state or a standing state (step S105: No), the control ends.

[0116] When the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a fallen state (step S105: Yes), the action processing unit 8d executes the above-mentioned second notification process or second notice process, etc. (step S106), and ends the control.

[0117] The fall prevention system 1 described above captures an image I of the monitoring target space SP by the capturing unit 2, and generates a skeleton model MDL representing a person P included in the image I by the skeleton model generating unit 8b. Then, the judging unit 8c distinguishes between the state of the person P standing and the state of the person P sitting based on the skeleton model MDL generated by the skeleton model generating unit 8b as the state of the person P corresponding to the skeleton model MDL. Moreover, the action processing unit 8d can perform fall prevention processing corresponding to the determination result based on the determination result of the judging unit 8c. In this case, as described above, the fall prevention system 1 can more accurately and precisely determine the state of the person P by distinguishing between the state of the person P standing and the state of the person P sitting based on the skeleton model MDL. In addition, the judging unit 8c determines the posture and state based not only on the image including the person P but also on the skeleton model MDL of the person P, so that, for example, even when the person P performs various movements, the state of the person P can be more appropriately determined. As a result, the fall prevention system 1 can appropriately implement fall prevention based on more accurate state determination.

[0118] In addition, in the above-described fall prevention system 1, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state, the action processing unit 8d can perform a guidance process as a fall prevention process. In this case, the fall prevention system 1 can guide the standing person P to sit down, thereby preventing the person P from falling.

[0119] In addition, in the above-described fall prevention system 1, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state, the action processing unit 8d can perform the first notification process as a fall prevention process. In this case, the fall prevention system 1 can notify a person different from the standing person P, such as a person providing services in the vehicle V, a driver when the vehicle V is a manual drive vehicle, etc., that there is a standing person P. As a result, the fall prevention system 1 can prevent the person P from falling before it happens through the person providing services, the driver, etc.

[0120] In addition, the fall prevention system 1 described above can also perform the first notification process as a fall prevention process by the action processing unit 8d after executing the guidance process and after a predetermined period of time, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state. In this case, the fall prevention system 1 can notify the operator of the business operator, etc., of the presence of the standing person P via the external device 201. Thus, the fall prevention system 1 can, for example, enable the operator of the external device 201 side, etc. to directly prompt the person P to sit down, and can prevent the person P from falling before it happens.

[0121] In addition, in the above-described fall prevention system 1, when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a standing state, the action processing unit 8d can execute the start prohibition process as the fall prevention process. In this case, when there is a person P standing in the vehicle interior space IN as the monitoring target space SP, the fall prevention system 1 can prohibit the start of the vehicle V, so that the person P can be prevented from falling.

[0122] In addition, the fall prevention system 1 described above can also be executed by the action processing unit 8d when the determination unit 8c determines that the person P corresponding to the skeleton model MDL is in a state of falling. In this case, the fall prevention system 1 can quickly convey the fact that the person P has fallen to the person providing the service in the vehicle V, the driver when the vehicle V is a manual driving vehicle, the operator on the side of the external device 201, etc., so that the treatment can be carried out quickly and appropriately.

[0123] In addition, the fall prevention system according to the above-mentioned embodiment of the present invention is not limited to the above-mentioned embodiment, and various changes can be made within the scope described in the claims.

[0124] In the above description, as the state of the person P represented by the skeleton model MDL, it is described that the determination unit 8c distinguishes between the state of the person P standing and the state of the person P sitting and also distinguishes between the state of the person P falling, but the present invention is not limited thereto. As the state of the person P represented by the skeleton model MDL, the determination unit 8c only needs to be able to distinguish between the state of the person P standing and the state of the person P sitting.

[0125] In the above description, the fall prevention system 1 includes the information output unit 3, the microphone 4, the notification unit 5, the external communication unit 6, and the internal communication unit 7, but the present invention is not limited thereto and may include none of them.

[0126] In the above description, the anti-fall processing is described as a guidance processing, a notification processing, a first notification processing, a start prohibition processing, etc., but it is not limited to this, as long as it is a processing to prevent a person from falling, it can also be other processing.

[0127] In the following description, as an example, the vehicle V is described as a bus, taxi, shared car, etc., which is managed by a specific operator and can be fully automatically driven, but the present invention is not limited thereto. The vehicle V may also be a manually driven vehicle whose behavior is controlled according to the driving operation of the driver of the vehicle V.

[0128] In the following description, the fall prevention system 1 is mounted on a vehicle V and the monitored space SP is the vehicle interior space IN, but the invention is not limited thereto. The fall prevention system 1 may also be applied to facilities other than the vehicle V, such as nursing facilities, and the monitored space SP may be a corridor space, a living room space, etc. of the facility instead of the vehicle interior space IN.

[0129] The control device 8 described above may be constructed by separate parts, which are connected in a manner that can send and receive various electrical signals to each other, or a part of the function may be realized by other control devices. In addition, the programs, applications, various data, etc. described above may be updated appropriately, or stored in a server connected to the anti-fall system 1 via an arbitrary network. The programs, applications, various data, etc. described above may also be downloaded in whole or in part as needed. In addition, for example, with respect to the processing functions possessed by the control device 8, all or any part thereof may be realized, for example, by a CPU, etc. and a program interpreted and executed by the CPU, etc., or may be realized as hardware based on wired logic, etc.

[0130] The fall prevention system according to the present embodiment may be configured by appropriately combining the components of the above-described embodiments and modifications.

Claims

1. A fall prevention system, characterized in that: have: a photographing unit that photographs an image of a monitored space; a skeleton model generating unit configured to generate a skeleton model representing a person included in the image captured by the imaging unit; a determination unit that distinguishes between a state in which the person is standing and a state in which the person is sitting as states of the person corresponding to the skeleton model, based on the skeleton model generated by the skeleton model generation unit; as well as An action processing unit is configured to execute, based on the determination result of the determination unit, a process corresponding to the determination result, that is, a fall prevention process for preventing the person from falling.

2. The fall prevention system according to claim 1, characterized in that: The fall prevention system includes a guiding part for guiding. When the determination unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit performs a guidance process of controlling the guidance unit to guide the person to sit down as the fall prevention process.

3. The fall prevention system according to claim 1 or 2, characterized in that: The fall prevention system includes a notification unit for making notifications. When the determination unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit performs a first notification process of controlling the notification unit and notifying other persons different from the person that the person is in a standing state as the fall prevention process.

4. The fall prevention system according to claim 2, characterized in that: The fall prevention system includes an external communication unit capable of communicating with an external device outside the monitored space. After the action processing unit has executed the guidance processing, if the judgment unit determines that the person corresponding to the skeleton model is in a standing state after a predetermined period of time, the action processing unit executes a first notification processing that controls the external communication unit and notifies the external device that the person is in a standing state as the anti-fall processing.

5. The fall prevention system according to any one of claims 1 to 4, characterized in that: The fall prevention system includes an internal communication unit capable of communicating with an in-vehicle device mounted on a vehicle whose interior space is set as the monitoring target space. When the determination unit determines that the person corresponding to the skeleton model is in a standing state, the action processing unit performs a start prohibition process of controlling the in-vehicle device via the internal communication unit to prohibit starting of the vehicle as the fall prevention process.

6. The fall prevention system according to any one of claims 1 to 5, characterized in that: have: a notification unit, the notification unit performing a notification action; and an external communication unit capable of communicating with an external device outside the monitored space, The determination unit distinguishes between a state in which the person is standing and a state in which the person is sitting, and further distinguishes between a state in which the person is falling as a state of the person corresponding to the skeleton model, based on the skeleton model generated by the skeleton model generation unit. When the determination unit determines that the person corresponding to the skeleton model is in a state of falling, the action processing unit performs a second notification process of controlling the notification unit and notifying other persons different from the person that the person is in a state of falling, or performs a second notification process of controlling the external communication unit and notifying the external device that the person is in a state of falling.

Images