Fall detection device, system, method, and program

Abstract

A fall detection device (1) according to the present invention comprises a first detection unit (11) that detects a set of skeletal points of a person outside a vehicle traveling on a road from a first image captured by a first onboard camera mounted on the vehicle, a second detection unit (12) that detects a road area indicating the area of the road from the first image, and a determination unit (13) that determines whether or not the person is falling on the road on the basis of the positional relationship between the skeletal points and the road area. In particular, for the positional relationship, the determination unit (13) uses the distance from the ground to the skeletal points calculated from the position information of the skeletal points and the position information of the road area.

Description

Technical Field

[0001] The present invention relates to a fall detection device, a system, a method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for detecting the behavior of an elderly person from a monitoring target image captured by an in-vehicle camera. Further, Patent Document 2 discloses a technique for determining the possibility of a two-wheeled vehicle falling from an image of a two-wheeled vehicle ahead captured by an in-vehicle camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, there is room for improvement in the accuracy of detecting a fallen person outside the vehicle from a captured image of an in-vehicle camera.

[0005] An object of the present disclosure is to provide a fall detection device, a system, a method, and a program for improving the accuracy of detecting a fallen person outside the vehicle from a captured image of an in-vehicle camera in view of the above-described problems.

Means for Solving the Problems

[0006] The fall detection device according to the first aspect of the present disclosure is a first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, a second detection means for detecting a road area indicating the area of the road from the first image, A determination means for determining whether or not the person is lying on the road, based on the positional relationship between the skeletal point and the road area, It is equipped with.

[0007] The fall detection system according to the second aspect of this disclosure is: The first in-car camera, Equipped with a fall detection device, The aforementioned tipping detection device is A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by the first in-vehicle camera, A second detection means for detecting a road area indicating the road area from the first image, A determination means for determining whether or not the person is lying on the road, based on the positional relationship between the skeletal point and the road area, It is equipped with.

[0008] The fall detection method according to the third aspect of this disclosure is: Computers From the first image captured by the first in-vehicle camera, the skeletal points of a person outside the vehicle are detected. From the first image, a road area indicating the road region is detected, Based on the positional relationship between the skeletal point and the road area, it is determined whether or not the person is lying on the road.

[0009] Non-temporary computer-readable media relating to the fourth aspect of this disclosure is, A first detection process that detects the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, A second detection process for detecting a road area from the first image, A determination process that determines whether or not the person is lying on the road based on the positional relationship between the skeletal point and the road area, A fall detection program that causes the computer to execute is stored here. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide a fall detection device, system, method, and program for improving the accuracy of detecting a fallen person outside the vehicle from a captured image of an in-vehicle camera.

Brief Description of the Drawings

[0011] [Figure 1] It is a block diagram showing the configuration of the fall detection device according to Embodiment 1. [Figure 2] It is a flowchart showing the flow of the fall detection method according to Embodiment 1. [Figure 3] It is a block diagram showing the overall configuration including the fall detection system according to Embodiment 2. [Figure 4] It is a block diagram showing the configuration of the fall detection device according to Embodiment 2. [Figure 5] It is a flowchart showing the flow of the fall detection process according to Embodiment 2. [Figure 6] It is a diagram for explaining the concept of fall detection according to Embodiment 2. [Figure 7] It is a block diagram showing the overall configuration including the fall detection system according to Embodiment 3. [Figure 8] It is a block diagram showing the configuration of the fall detection device according to Embodiment 4. [Figure 9] It is a flowchart showing the flow of the fall detection process according to Embodiment 4. [Figure 10] It is a block diagram showing the overall configuration including the fall detection system according to Embodiment 5. [Figure 11] It is a diagram showing an example of the positional relationship between a person on the road and a plurality of vehicles according to Embodiment 5. [Figure 12] It is a flowchart showing the flow of the fall detection process according to Embodiment 5.

Modes for Carrying Out the Invention

[0012] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations will be omitted where necessary for clarity.

[0013] <Embodiment 1> Figure 1 is a block diagram showing the configuration of the fall detection device 1 according to this embodiment 1. The fall detection device 1 is an information processing device for detecting when a person falls. The fall detection device 1 comprises a first detection unit 11, a second detection unit 12, and a determination unit 13.

[0014] The first detection unit 11 detects the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera (not shown). The second detection unit 12 detects a road area from the first image. Here, the first detection unit 11 and the second detection unit 12 may each detect the skeletal points and the road area using image recognition processing. The determination unit 13 determines whether or not a person has fallen on the road based on the positional relationship between the skeletal points and the road area.

[0015] Figure 2 is a flowchart showing the flow of the fall detection method according to this embodiment 1. First, the first detection unit 11 detects the skeletal points of a person outside the vehicle from the first image captured by the first in-vehicle camera (S11). Next, the second detection unit 12 detects a road area indicating the area of ​​the road from the first image (S12). Then, the determination unit 13 determines whether or not a person has fallen on the road based on the positional relationship between the skeletal points and the road area (S13).

[0016] As described above, the fall detection device 1 according to this embodiment detects the posture of a person outside the vehicle by analyzing images captured by an in-vehicle camera and detecting the skeletal points of the person. Therefore, the fall detection device 1 can identify the positional relationship of the person's body parts on the road, for example, the coordinate information of each part in three-dimensional space. Furthermore, the fall detection device 1 can analyze the same captured images to identify the coordinate information of the road area, that is, the ground in three-dimensional space. Since the fall detection device 1 can grasp the positional relationship between the skeletal points and the road area, it can accurately determine whether or not a person has fallen on the road and accurately detect the person who has fallen.

[0017] The tip-over detection device 1 includes a processor, memory, and storage device (not shown in the diagram). The storage device stores a computer program that implements the tip-over detection method according to this embodiment. The processor loads the computer program from the storage device into the memory and executes the computer program. As a result, the processor realizes the functions of the first detection unit 11, the second detection unit 12, and the determination unit 13.

[0018] Alternatively, each component of the fall detection device 1 may be implemented with dedicated hardware. Furthermore, some or all of the components of each device may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be comprised of a single chip or multiple chips connected via a bus. Some or all of the components of each device may be implemented by a combination of the aforementioned circuits, etc., and programs. Additionally, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc., can be used as the processor.

[0019] <Embodiment 2> This second embodiment is a specific example of the first embodiment described above. Figure 3 is a block diagram showing the overall configuration including the fall detection system 1000 according to this second embodiment. The fall detection system 1000 is an information system for detecting a person who has fallen over among people outside the vehicle 1001. Figure 3 shows an example in which the fall detection system 1000 detects whether or not a person U on the roadway while the vehicle 1001 is traveling has fallen over. However, person U may also be on the sidewalk. Therefore, in the following description, "road" includes roadways, sidewalks, etc.

[0020] Vehicle 1001 is driven equipped with an on-board camera 100 and a rollover detection device 200. The on-board camera 100 is a camera that captures images in at least one of the following directions: the direction of travel (forward), the rear, or the side of the vehicle 1001. The on-board camera 100 captures images within a predetermined range at predetermined intervals and outputs the captured images to the rollover detection device 200. The on-board camera 100 is an example of the first on-board camera described above.

[0021] The rollover detection device 200 is an example of the rollover detection device 1 described above, and is an information processing device mounted on the vehicle 1001. The rollover detection device 200 is, for example, an ECU (Electronic Control Unit) that controls the vehicle 1001. Alternatively, the rollover detection device 200 may be redundantly implemented using multiple computers, and each functional block may be implemented using multiple computers.

[0022] The fall detection device 200 is connected to the emergency system 300 via network N in a communicative manner. Here, network N is a communication network including wireless communication lines, mobile phone lines, etc. Network N may also include the Internet. Furthermore, the communication network is not limited by the type of communication protocol.

[0023] The fall detection device 200 analyzes the image received from the in-vehicle camera 100, determines whether the person U in the image is a person who has fallen, and if it determines that the person has fallen, it notifies the emergency system 300 via the network N that a person who has fallen has been detected and their current location.

[0024] The emergency medical system 300 is an information system that, in response to notifications from the fall detection device 200, issues instructions to dispatch an ambulance or other emergency vehicle to the location indicated by the current position.

[0025] Figure 4 is a block diagram showing the configuration of the fall detection device 200 according to this second embodiment. The fall detection device 200 comprises a storage unit 210, a memory 220, an IF (Interface) unit 230, and a control unit 240. The storage unit 210 is an example of a storage device such as a hard disk or flash memory. The storage unit 210 stores a fall detection program 211, a first threshold 212, and a second threshold 213. The fall detection program 211 is a computer program on which the fall detection processing according to this second embodiment is implemented. The first threshold 212 is the distance between the head of person U and the ground for determining that person U has fallen. The second threshold 213 is the distance between the knees of person U and the ground for determining that person U has fallen.

[0026] Memory 220 is a volatile storage device such as RAM (Random Access Memory), and is a storage area for temporarily holding information when the control unit 240 is operating. IF unit 230 is a communication interface between the inside of the fall detection device 200 and the in-vehicle camera 100 and network N.

[0027] The control unit 240 is a processor, or control device, that controls each component of the fall detection device 200. The control unit 240 loads the fall detection program 211 from the storage unit 210 into the memory 220 and executes the fall detection program 211. In this way, the control unit 240 realizes the functions of the acquisition unit 241, the first detection unit 242, the second detection unit 243, the determination unit 244, the calculation unit 245, and the output unit 246.

[0028] The acquisition unit 241 acquires the images captured and output by the in-vehicle camera 100.

[0029] The first detection unit 242 is an example of the first detection unit 11 described above. The first detection unit 242 detects skeletal points of a person U outside the vehicle from a first image captured by the in-vehicle camera 100. The first detection unit 242 detects the region of the person U's body shape in the image through a first image recognition process, and detects a set of skeletal points of person U from the detected region. Here, skeletal points are information indicating characteristic parts of the person U's skeleton, i.e., representative points. Skeletal points may include coordinate information (information indicating position) in three-dimensional space and information indicating body parts. In other words, skeletal points may include information indicating the position when the road is considered a plane, as well as information on the height from the road. Body parts include, for example, the head, shoulders, elbows, hands, waist, knees, feet, etc., but are not limited to these. The first detection unit 242 may also detect the posture of person U through a first image recognition process. Furthermore, the first detection unit 242 may detect a set of skeletal points of person U from multiple images, i.e., video data, that are continuously captured by the in-vehicle camera 100.

[0030] The first image recognition process may use known skeleton detection algorithms, posture estimation algorithms, etc. Alternatively, the first image recognition process may use a first AI (Artificial Intelligence) model that takes an image of a person as input and outputs a set of skeletal points of the person. In this case, the first AI model should be a trained model that has been machine-learned using training data that includes images of various people in postures such as falling, crouching, and standing, and ground truth data of the set of skeletal points of the person in each image.

[0031] The second detection unit 243 is an example of the second detection unit 12 described above. The second detection unit 243 detects road regions from the first image. The second detection unit 243 detects road regions in the image by second image recognition processing. Here, the road regions may be coordinate information (position information) in the same three-dimensional space as the skeletal points described above. However, if there is no difference in elevation in the road, two-dimensional coordinate information with height information omitted may be used as position information.

[0032] The second image recognition process may employ segmentation techniques such as instance segmentation or semantic segmentation. Alternatively, the second image recognition process may involve labeling each pixel in the image with an indication of the region type. For example, the second image recognition process may use a second AI model that outputs a label indicating whether or not each pixel in the input image is a road. In this case, the second AI model is preferably a trained model that has been machine-trained using training data that includes multiple images of roads captured by various in-vehicle cameras, and ground truth data in which each pixel in each image is labeled as a road.

[0033] The determination unit 244 is an example of the determination unit 13 described above. The determination unit 244 determines whether or not person U is lying on the road based on the positional relationship between the skeletal point and the road area. In particular, the determination unit 244 determines the positional relationship by using the distance from the ground at the skeletal point, which is calculated from the position of the skeletal point and the position of the road area.

[0034] Furthermore, the determination unit 244 may determine whether or not a set of skeletal points has been detected from the image by the first detection unit 242. Also, the determination unit 244 may determine whether or not a road area has been detected from the image by the second detection unit 243.

[0035] Furthermore, the determination unit 244 determines whether or not the skeletal points of the head have been detected. If the skeletal points of the head have been detected, the determination unit 244 determines whether or not the first distance between the head and the ground, calculated by the calculation unit 245 (described later), is less than or equal to the first threshold 212. If the first distance is less than or equal to the first threshold 212, the determination unit 244 determines that person U has fallen over.

[0036] The determination unit 244 determines whether or not a skeletal point at the knee is detected. If a skeletal point at the knee is detected, the determination unit 244 determines whether or not the second distance between the knee and the ground, calculated by the calculation unit 245, is less than or equal to the second threshold 213. If the second distance is less than or equal to the second threshold 213, the determination unit 244 determines that person U has fallen.

[0037] In particular, if the first detection unit 242 does not detect the skeletal points of the head of person U, the determination unit 244 may determine whether the second distance is less than or equal to the second threshold 213. This allows the determination process using the second threshold 213 to be omitted if a head is detected, thereby reducing processing costs.

[0038] The calculation unit 245 is an example of the first and second calculation means. When the first detection unit 242 detects the skeletal points of the head of person U, the calculation unit 245 calculates a first distance from the ground at the skeletal points of the head based on the road area. Specifically, the calculation unit 245 calculates the first distance from the position of the skeletal points of the head and the position of the road area. For example, the calculation unit 245 may compare the position coordinates of the head on a plane with the position coordinates of the road area on a plane and calculate the difference in the height coordinates at the matching point as the first distance. Alternatively, the calculation unit 245 may calculate the shortest distance between the position of the head and the position of the road area as the first distance.

[0039] Furthermore, if the calculation unit 245 detects the skeletal points of the knee portion of person U by the first detection unit 242, it calculates a second distance from the ground at the skeletal points of the knee portion based on the road area. Specifically, the calculation unit 245 calculates the second distance by replacing the skeletal points of the head with the skeletal points of the knee portion in the first distance calculation process described above.

[0040] If the determination unit 244 determines that person U in the image is a person who has fallen, the output unit 246 outputs at least a message indicating that a person has fallen. For example, if the vehicle 1001 is equipped with a display device, the output unit 246 may output the image including person U and information indicating that person U is a person who has fallen to the display device and display them on the screen of the display device. Also, if the vehicle 1001 is capable of acquiring current location information (current location) using a GPS (Global Positioning System) function or the like, it may transmit the message indicating that a person has fallen and its current location to the emergency system 300 via the network N.

[0041] Figure 5 is a flowchart showing the flow of the fall detection process according to this embodiment 2. First, the in-vehicle camera 100 photographs the road and outputs the captured image to the fall detection device 200. In response, the fall detection device 200 acquires the image captured by the in-vehicle camera 100 (S101). Next, the fall detection device 200 detects a set of skeletal points of a person outside the vehicle from the image (S102). The fall detection device 200 also detects the road area from the image (S103). Here, it is assumed that a set of skeletal points of a person and the road area have been detected from the image.

[0042] After steps S102 and S103, the fall detection device 200 determines whether or not a skeletal point of the head has been detected (S104). If a skeletal point of the head is detected (YES in S104), the fall detection device 200 calculates a first distance between the head and the ground (S105). Then, the fall detection device 200 determines whether or not the first distance is less than or equal to a first threshold (S106). If the first distance is less than or equal to the first threshold (YES in S106), the fall detection device 200 determines that person U in the image is a person who has fallen (S110). This is because if the head is relatively close to the ground, there is a very high probability that person U has fallen. Therefore, by prioritizing the determination of the skeletal point over other parts, the processing load of calculating and determining distances for other parts is reduced, and the speed of detecting fallen persons is also improved.

[0043] If no skeletal points of the head are detected in step S104 (NO in S104), the fall detection device 200 determines whether or not skeletal points of the knee area are detected (S107). If skeletal points of the knee area are detected (YES in S107), the fall detection device 200 calculates a second distance between the knee area and the ground (S108). Then, the fall detection device 200 determines whether or not the second distance is less than or equal to a second threshold (S109). If the second distance is less than or equal to the second threshold (YES in S109), the fall detection device 200 determines that person U in the image is a person who has fallen (S110). For example, depending on the direction in which person U has fallen, the head may not be captured by the onboard camera 100, and skeletal points of the head may not be detected. In such cases, if other body parts, such as the knee area, are relatively close to the ground, there is a high probability that person U has fallen or at least is on their knees. Therefore, if the skeletal points of the head are not detected from the image, a more accurate determination can be made by using the skeletal points of the knee area. Furthermore, if the head is detected, the processing load can be reduced by not performing the calculation and determination process for the second distance using the skeletal points of the knee area.

[0044] On the other hand, if the first distance in step S106 is greater than the first threshold, or if the second distance in step S109 is greater than the second threshold, the fall detection device 200 does not determine that person U in the image is a person who has fallen and terminates processing. Also, if no skeletal points are detected in the knee area in step S107, the fall detection device 200 terminates processing. However, if no skeletal points are detected in the knee area in step S107, the fall detection device 200 may calculate a third distance between other skeletal points and the ground, and if the third distance is less than or equal to the third threshold, it may determine that person U in the image is a person who has fallen. Other body parts include, but are not limited to, the shoulders, elbows, hands, waists, and feet. Furthermore, the knee area detection determination in step S107 may be performed before the head detection determination in step S104. Alternatively, the detection determination of other body parts other than the head and knee area may be performed before step S104.

[0045] After step S110, the fall detection device 200 outputs a message indicating that a person has fallen, as described above (S111).

[0046] Figure 6 is a diagram illustrating the concept of fall detection according to this second embodiment. Captured image 5 is an example of an image captured by the on-board camera 100 of the vehicle 1001. Here, it is shown that the first detection unit 242 detects a set of skeletal points 51 from captured image 5, and the second detection unit 243 detects a road area 52 from captured image 5. The set of skeletal points 51 includes multiple skeletal points of person U. In particular, the set of skeletal points 51 includes the skeletal point 511 of the head, and the skeletal points 512 and 513 of the knee area.

[0047] In the example shown in Figure 6, the first distance between the head skeletal point 511 and the ground is assumed to be below the first threshold, and person U is detected as a person who has fallen. In this case, since no calculation or determination is performed using the knee skeletal points 512 and 513, the person who has fallen can be detected more quickly. Furthermore, even if the head skeletal point 511 is not detected, and the second distance between the detected knee skeletal point 512 or 513 and the ground is below the second threshold, person U can still be detected as a person who has fallen. Therefore, the accuracy of detecting a person who has fallen outside the vehicle from images captured by an in-vehicle camera can be improved.

[0048] <Embodiment 3> This third embodiment is a modification of the second embodiment described above. Figure 7 is a block diagram showing the overall configuration including the fall detection system 1000a according to this third embodiment. The fall detection device 200a according to this third embodiment is an example in which it is installed as a server outside the vehicle 1001, compared to Figure 3 described above. In other words, the fall detection device 200a is connected to the in-vehicle camera 101 and the emergency system 300 via the network N so as to be able to communicate. The in-vehicle camera 101 has a wireless communication function in addition to the functions of the in-vehicle camera 100 described above. Therefore, the in-vehicle camera 101 is connected to the network N via wireless communication. The in-vehicle camera 101 transmits the captured image and current location to the fall detection device 200a via the network N.

[0049] The fall detection device 200a acquires an image from the in-vehicle camera 101 via the network N. The fall detection device 200a performs fall detection processing on the acquired image in the same manner as steps S102 to S110 in Figure 5 described above. If person U in the image is determined to be a person who has fallen, the fall detection device 200a transmits a message indicating that a person has fallen and their current location to the emergency system 300 via the network N. The fall detection device 200a may also transmit a message indicating that a person has fallen to the vehicle 1001 via the network N and display this message on a display device in the vehicle 1001.

[0050] Thus, the same effects as those of the above-described embodiment 2 can be achieved in this embodiment 3. Furthermore, in this embodiment 3, there is no need to install an advanced on-board device equivalent to the rollover detection device 200 inside the vehicle 1001. In addition, in this embodiment 3, since the rollover detection process is performed outside the vehicle 1001, the power consumption of the vehicle 1001 can be suppressed.

[0051] <Embodiment 4> This fourth embodiment is an improved version of the second or third embodiment described above. While the second embodiment and others described above can accurately detect a person who has fallen, if the person who has fallen gets up immediately, the need to notify the emergency system 300 is low. In other words, even if the occupants of the vehicle 1001 (driver and passengers) detect a person who has fallen using the fall detection device 200, they may be unsure whether to notify the emergency system 300. For example, the occupants of the vehicle 1001 may be unsure whether to call the emergency system 300 if they see a person lying on the road. Alternatively, even if the fall detection device 200 detects a person who has fallen using the fall detection device 200, the occupants of the vehicle 1001 may hesitate to call the emergency system 300, considering the possibility that someone else might provide assistance or call for help. As a result, there is a possibility of delays in responding to a person who has fallen. Therefore, the fall detection device according to this embodiment 4 calculates the duration of the fall from a number of images continuously captured by an in-vehicle camera, and notifies the emergency system 300, etc., when the duration of the fall exceeds a predetermined time.

[0052] Figure 8 is a block diagram showing the configuration of the fall detection device 200b according to this embodiment 4. Compared to Figure 4 described above, the fall detection device 200b has been modified in terms of the fall detection program 211b, the calculation unit 245b, and the output unit 246b. The fall detection program 211b is a computer program that implements the fall detection processing, etc., according to this embodiment 4. The control unit 240 loads the fall detection program 211b from the storage unit 210 into the memory 220 and executes the fall detection program 211b. As a result, the control unit 240 realizes the functions of the calculation unit 245b and the output unit 246b, in addition to the functions of the acquisition unit 241, the first detection unit 242, the second detection unit 243, and the determination unit 244 described above.

[0053] The calculation unit 245b is an example of a third calculation means that, if it is determined that person U has fallen based on the positional relationship between the skeletal points detected from the first image and the road area, calculates the duration of the person U's fall based on a first subsequent image taken after the first image. The output unit 246b is an example of a notification means that notifies a predetermined recipient if the duration of the fall is longer than a predetermined time. The predetermined time is set in advance in the fall detection device 200b and is a value that can be changed as appropriate.

[0054] Figure 9 is a flowchart showing the flow of the fall detection process according to this embodiment 4. First, the acquisition unit 241 acquires a first image captured by the in-vehicle camera 101 (S101). Then, steps S102 to S109 are executed as described above. After that, the determination unit 244 determines whether or not the person U in the first image is a person who has fallen (S110b). If it was determined to be YES in step S106 or S109, the determination unit 244 determines that the person U in the first image is a person who has fallen. In that case, the calculation unit 245b calculates the duration of the fall (S112). For example, if the person U is determined to be a person who has fallen for the first time in the first image, the calculation unit 245b starts counting the duration of the fall. Then, the determination unit 244 determines whether or not the duration of the fall is longer than a predetermined time (S113).

[0055] Furthermore, if it is determined in step S110b that person U in the first image is not the person who fell, the calculation unit 245b clears the fall duration (S114). After step S114, or if it is determined in step S113 that the fall duration is less than a predetermined time, the process returns to step S101.

[0056] Then, the acquisition unit 241 acquires the first subsequent image captured by the on-board camera 101 (S101). The first subsequent image is an image captured by the on-board camera 101 after the first image described above. Then, the tipping detection device 200b performs steps S102 to S109 with respect to the first subsequent image.

[0057] Then, in step S110b, if person U in the first subsequent image is determined to be a person who has fallen, the calculation unit 245b calculates the duration of the fall from the first image (S112). For example, the calculation unit 245b adds the time interval between the first image and the first subsequent image to the duration of the fall calculated immediately before. In other words, the calculation unit 245b updates the duration of the fall. Then, the determination unit 244 determines whether the updated duration of the fall is equal to or greater than a predetermined time (S113).

[0058] If, in step S110b, it is determined that person U in the first subsequent image is not a person who has fallen, the calculation unit 245b clears the fall duration (S114). However, even if it is determined that the person has not fallen, the calculation unit 245b does not have to immediately clear the fall duration. For example, the calculation unit 245b may calculate the number of consecutive times in step S110b that the person has not fallen, and if that number exceeds a predetermined number, it may clear the fall duration. This makes it possible to exclude temporary noise, such as when the person is incorrectly determined not to have fallen. At this time, the calculation unit 245b also clears the count. After step S114, or if it is determined in step S113 that the fall duration is less than a predetermined time, the process returns to step S101.

[0059] On the other hand, if step S113 determines that the duration of the fall is longer than a predetermined time, the output unit 246b notifies that a person who has fallen has been detected, as described above (S111b). Specifically, the output unit 246b transmits the fact that a person who has fallen has been detected and the current location of the vehicle 1001 to the emergency system 300 via the network N. The output unit 246b may also output the fact that a person who has fallen has been detected to the display device of the vehicle 1001.

[0060] In this embodiment 4, the duration of the fall is measured, and if the duration of the fall exceeds a predetermined time, a notification is sent to the emergency system 300, etc. This enables notification to the emergency system 300 regarding fallen persons who require rescue. Furthermore, since the accuracy of the fall detection process is the same as in embodiment 2, etc. described above, the chances of overlooking a fallen person (person in need of rescue) can be reduced. In addition, the fall detection device 200b notifies the current location information of the vehicle 1001, similar to embodiment 2, etc. described above. Therefore, the emergency system 300 can more accurately determine the location of the person in need of rescue. Moreover, since the fall detection device 200b automatically notifies the emergency system 300 without operation by the occupants of the vehicle 1001, the notification time is shortened.

[0061] Furthermore, if a person is detected as having fallen in the first image, but is determined not to have fallen in a subsequent image taken later, the emergency system 300 will not be notified. For example, if person U falls once on the road but gets up shortly afterward, no notification will be sent. In other words, if someone only falls slightly, no notification will be sent. This helps to suppress excessive notifications.

[0062] <Embodiment 5> This fifth embodiment is a modification of embodiments 2 to 4 described above. The fall detection device according to this fifth embodiment uses multiple images captured by multiple on-board cameras mounted on multiple vehicles to more accurately determine whether or not a person has fallen on the road. In particular, the determination unit determines whether or not a person has fallen on the road based on the positional relationship between the skeletal points detected from a second image captured by a second on-board camera mounted on a second vehicle other than the first vehicle on which the first on-board camera is mounted, and the road area.

[0063] Figure 10 is a block diagram showing the overall configuration including the fall detection system 1000c according to this embodiment 5. The fall detection system 1000c comprises vehicles 1001 to 100n (where n is a natural number of 2 or more), a fall detection device 200c, and an emergency system 300. Vehicle 1001 is equipped with an on-board camera 101, vehicle 1002 is equipped with an on-board camera 102, ... vehicle 100n is equipped with an on-board camera 10n. The functions of the on-board cameras 101 to 10n are equivalent. Each of the on-board cameras 101 to 10n, the fall detection device 200c, and the emergency system 300 are connected to each other via a network N so as to be able to communicate.

[0064] The fall detection device 200c is installed outside the vehicle as a server, similar to the fall detection device 200a shown in Figure 7 above. Therefore, each of the in-vehicle cameras 101 to 10n transmits the captured images to the fall detection device 200c via the network N. The fall detection device 200c analyzes multiple images acquired from each of the in-vehicle cameras 101 to 10n within a predetermined area to comprehensively determine whether a person in the image has fallen.

[0065] Figure 11 shows an example of the positional relationship between a person U on a roadway and multiple vehicles 1001 to 1003 according to this embodiment 5. The example in Figure 11 is an intersection, and the person U is assumed to be in the direction of travel of vehicles 1001 and 1002. Therefore, the shooting range of the onboard camera 101 of vehicle 1001 and the onboard camera 102 of vehicle 1002 includes the roadway and the person U. Vehicle 1003 is traveling from the opposite direction to the direction of travel of vehicle 1001. Therefore, the shooting range of the onboard camera 103 of vehicle 1003 also includes the roadway and the person U. In other words, the first image taken by onboard camera 101, the second image taken by onboard camera 102, and the third image taken by onboard camera 103 each include the roadway (intersection) and the person U. Although Figure 11 shows an example where the person U is on a roadway, this embodiment can also be applied when the person U is on a road other than a roadway, such as a sidewalk.

[0066] Figure 12 is a flowchart showing the flow of the fall detection process according to this embodiment 5. First, the fall detection device 200c acquires multiple images taken by multiple in-vehicle cameras within a predetermined area (S101c). Next, the fall detection device 200c detects a set of skeletal points of a person outside the vehicle from each image (S102c). The fall detection device 200c also detects the road area from each image (S103c). For example, the fall detection device 200c acquires first to third images taken by in-vehicle cameras 101 to 103. Then, the fall detection device 200c detects a set of skeletal points of person U from each of the first to third images. For example, suppose that some of the skeletal points of person U are detected from the first image, and also from the second and third images. Then, the fall detection device 200c combines these detected skeletal points to form a set of skeletal points. Therefore, some of the skeletal points may overlap in position. Furthermore, the tipping detection device 200c detects road areas from each of the first to third images. The tipping detection device 200c then combines the positions of these detected road areas and detects them as a single road area.

[0067] After steps S102c and S103c, steps S104 to S111 are executed, similar to Figure 5 described above. Therefore, if person U is determined to be a person who has fallen, the fall detection device 200c transmits a message indicating that a person has fallen and the location information of vehicle 1001, etc., to the emergency system 300 via the network N. In addition, the fall detection device 200c may calculate the duration of the fall based on the determination results (detection results) from images of multiple in-vehicle cameras, as in Embodiment 4 described above.

[0068] As described above, in this embodiment 5, the fall detection device 200c collects images from on-board cameras of multiple vehicles and detects a person who has fallen from the collected images. In other words, the fall detection device 200c aggregates images from multiple on-board cameras and performs fall detection processing. Therefore, compared to using images from a single on-board camera, it is possible to detect the skeletal points of a person and the road area from multiple angles, and to detect a person who has fallen with greater accuracy.

[0069] <Other Embodiments> In the examples described above, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0070] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit. Furthermore, this disclosure may be implemented by combining the respective embodiments as appropriate.

[0071] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note A1) A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, A second detection means for detecting a road area indicating the road area from the first image, A determination means for determining whether or not the person is lying on the road, based on the positional relationship between the skeletal point and the road area, A fall detection device equipped with the following features. (Appendix A2) If the first detection means detects the skeletal points of the person's head, the system further comprises a first calculation means for calculating a first distance from the ground to the skeletal points of the head based on the road area. The determination means determines that the person has fallen if the first distance is less than or equal to the first threshold. The tip-over detection device described in Appendix A1. (Note A3) If the first detection means detects the skeletal point of the knee portion of the person, the system further comprises a second calculation means for calculating a second distance from the ground at the skeletal point of the knee portion based on the road area. The determination means determines that the person has fallen if the second distance is less than or equal to the second threshold. The tip-over detection device described in Appendix A1 or A2. (Note A4) The determination means determines whether the second distance is less than or equal to the second threshold if the first detection means does not detect the skeletal points of the person's head. The tip-over detection device described in Appendix A3. (Note A5) If it is determined that the person has fallen based on the positional relationship between the skeletal points detected from the first image and the road area, a third calculation means calculates the duration of the person's fall based on a first subsequent image taken after the first image. If the duration of the fall exceeds a predetermined time, a notification means is provided to notify a designated recipient of that fact. A fall detection device according to any one of the appendices A1 to A4, further comprising: (Note A6) The determination means further determines whether or not the person has fallen on the road, based on the positional relationship between the skeletal points detected from a second image captured by a second on-board camera mounted on a second vehicle other than the first vehicle on which the first on-board camera is mounted, and the road area. A tip-over detection device as described in any one of the appendices A1 to A5. (Note A7) The determination means uses the distance from the ground at the skeletal point, calculated from the position of the skeletal point and the position of the road area, as the positional relationship. A tipping detection device as described in any one of the appendices A1 to A6. (Note B1) The first in-car camera, Equipped with a fall detection device, The aforementioned tipping detection device is A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by the first in-vehicle camera, A second detection means for detecting a road area indicating the road area from the first image, A determination means for determining whether or not the person is lying on the road, based on the positional relationship between the skeletal point and the road area, A fall detection system equipped with the following features. (Note B2) The first detection means detects at least the skeletal points of the knee area of ​​the person, The determination means determines that the person has fallen if, based on the road area, the distance from the ground at the skeletal point of the knee is less than or equal to a first threshold. The tip-over detection system described in Appendix B1. (Note C1) Computers From the first image captured by the first in-vehicle camera, the skeletal points of a person outside the vehicle are detected. From the first image, a road area indicating the road region is detected, Based on the positional relationship between the aforementioned skeletal point and the aforementioned road area, it is determined whether or not the person is lying on the road. Method for detecting falls. (Note D1) A first detection process that detects the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, A second detection process for detecting a road area from the first image, A determination process that determines whether or not the person is lying on the road based on the positional relationship between the skeletal point and the road area, A non-temporary computer-readable medium containing a fall detection program that causes a computer to execute.

[0072] Although the present invention has been described above with reference to embodiments (and examples), the present invention is not limited to the above embodiments (and examples). Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention. [Explanation of Symbols]

[0073] 1. Tipping detection device 11 First detection unit 12 Second detection unit 13 Judgment section 1000 Fall Detection System 1001 Vehicle 1002 vehicles 1003 Vehicle 100n vehicle 100 In-Car Cameras 101 In-car camera 102 In-car camera 103 In-car camera 10n In-Car Camera 200 Tipping detection device 210 Storage section 211 Fall Detection Program 212 First threshold 213 Second threshold 220 memory 230 IF section 240 Control Unit 241 Acquisition Department 242 First detection unit 243 Second detection unit 244 Judgment section 245 Calculation Unit 246 Output section 300 Emergency Systems N Network U person 5. Captured images 51 A collection of skeletal points 511 skeletal points 512 skeletal points 513 Skeletal points 52 Road area 1000a Tipping Detection System 200a Tipping detection device 200b Tipping detection device 211b Fall detection program 245b Calculation part 246b Output section 1000c Tipping Detection System 200c tip-over detection device

Claims

1. A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, A second detection means for detecting a roadway area and a sidewalk area from the first image, A determination means for determining whether or not a person is lying down in the roadway area, based on the positional relationship between the skeletal points of a person located in the roadway area and the roadway area, If the first detection means detects the skeletal point of the knee portion of the person, the system includes a second calculation means that calculates a second distance from the ground at the skeletal point of the knee portion based on the road area, The determination means determines that the person has fallen if the second distance is less than or equal to the second threshold, The determination means determines whether the second distance is less than or equal to the second threshold if the first detection means does not detect the skeletal points of the person's head. Fall detection device.

2. A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by a first in-vehicle camera, A second detection means for detecting a roadway area and a sidewalk area from the first image, A determination means for determining whether or not a person is lying down in the roadway area, based on the positional relationship between the skeletal points of a person located in the roadway area and the roadway area, Equipped with, The determination means further determines whether or not the person has fallen in the roadway area, based on the positional relationship between the skeletal points detected from a second image captured by a second on-board camera mounted on a second vehicle other than the first vehicle on which the first on-board camera is mounted, and the roadway area. Fall detection device.

3. If the first detection means detects the skeletal points of the person's head, the system further comprises a first calculation means that calculates a first distance from the ground to the skeletal points of the head based on the roadway area. The determination means determines that the person has fallen if the first distance is less than or equal to the first threshold. The fall detection device according to claim 1.

4. If it is determined that the person has fallen based on the positional relationship between the skeletal points detected from the first image and the roadway area, a third calculation means calculates the duration of the person's fall based on a first subsequent image taken after the first image. If the duration of the fall exceeds a predetermined time, a notification means is provided to notify a designated recipient that a person who has fallen has been detected. The fall detection device according to claim 1, further comprising:

5. The determination means determines whether or not the person is lying down in the sidewalk area based on the positional relationship between the skeletal points of the person in the sidewalk area and the sidewalk area. The fall detection device according to claim 1.

6. The first in-car camera, Equipped with a fall detection device, The aforementioned tipping detection device is A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by the first in-vehicle camera, A second detection means for detecting a roadway area and a sidewalk area from the first image, A determination means for determining whether or not a person is lying down in the roadway area, based on the positional relationship between the skeletal points of a person located in the roadway area and the roadway area, If the first detection means detects the skeletal point of the knee portion of the person, the system includes a second calculation means that calculates a second distance from the ground at the skeletal point of the knee portion based on the road area, The determination means determines that the person has fallen if the second distance is less than or equal to the second threshold, The determination means determines whether the second distance is less than or equal to the second threshold if the first detection means does not detect the skeletal points of the person's head. Fall detection system.

7. The first in-car camera, Equipped with a fall detection device, The aforementioned tipping detection device is A first detection means for detecting the skeletal points of a person outside the vehicle from a first image captured by the first in-vehicle camera, A second detection means for detecting a roadway area and a sidewalk area from the first image, A determination means for determining whether or not a person is lying down in the roadway area, based on the positional relationship between the skeletal points of a person located in the roadway area and the roadway area, Equipped with, The determination means further determines whether or not the person has fallen in the roadway area, based on the positional relationship between the skeletal points detected from a second image captured by a second on-board camera mounted on a second vehicle other than the first vehicle on which the first on-board camera is mounted, and the roadway area. Fall detection system.

8. Computers From the first image captured by the first in-vehicle camera, the skeletal points of a person outside the vehicle are detected. From the first image described above, a roadway area indicating the roadway area and a sidewalk area indicating the sidewalk area are detected. Based on the positional relationship between the skeletal points of the person located in the roadway area and the roadway area, it is determined whether or not the person is lying down in the roadway area. If the skeletal point of the knee of the person is detected, a second distance from the ground at the skeletal point of the knee is calculated based on the road area. If the second distance is less than or equal to the second threshold, it is determined that the person has fallen. If no skeletal points of the person's head are detected, it is determined whether the second distance is less than or equal to the second threshold. Method for detecting falls.

9. Computers From the first image captured by the first in-vehicle camera, the skeletal points of a person outside the vehicle are detected. From the first image described above, a roadway area indicating the roadway area and a sidewalk area indicating the sidewalk area are detected. Based on the positional relationship between the skeletal points of the person located in the roadway area and the roadway area, it is determined whether or not the person is lying down in the roadway area. Based on the positional relationship between the skeletal points detected from a second image captured by a second on-board camera mounted on a second vehicle other than the first vehicle on which the first on-board camera is mounted, and the roadway area, it is determined whether or not the person has fallen in the roadway area. Method for detecting falls.

10. From the first image captured by the first in-vehicle camera, the skeletal points of a person outside the vehicle are detected. From the first image described above, a roadway area indicating the roadway area and a sidewalk area indicating the sidewalk area are detected. Based on the positional relationship between the skeletal points of the person located in the roadway area and the roadway area, it is determined whether or not the person is lying down in the roadway area. If the skeletal point of the knee of the person is detected, a second distance from the ground at the skeletal point of the knee is calculated based on the road area. If the second distance is less than or equal to the second threshold, it is determined that the person has fallen. If no skeletal points of the person's head are detected, it is determined whether the second distance is less than or equal to the second threshold. A fall detection program that has a computer perform the necessary processing.

Images