Bed Area Extraction Device

Abstract

To provide a bed area extraction device, a bed area extraction method, a bed area extraction program and a watching support system, capable of detecting a bed area from a bed image obtained by imaging a bed from any direction even in a state where a part of the bed is covered.SOLUTION: A watching support system extracts as a bed partial image 702 a bounding box including a part including a bed 20 from a bed image 701 obtained by imaging the bed 20 to be used by a person to be watched, detects positions of four bed corners 41A, 41B, 41C and 41D of the bed 20 in the bed partial image 702 from a feature of the whole bed 20 included in the bed partial image 702, and extracts a bed area by converting coordinate values in the bed partial image 702 into coordinate values in the bed image 701. The extracted bed area is used for detection of action of the person to be watched such as waking-up, leaving the bed, falling and tumbling.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a bed area extraction device. To The present invention relates to a technology for extracting a bed area from a bed image captured from an arbitrary direction even when a part of the bed is covered. [Background technology]

[0002] In recent years, developed countries have been experiencing a declining birthrate and an aging population, and the United Nations predicts that by 2050, OECD (Organization for Economic Co-operation and Development) countries will become super-aging societies. Japan is ahead of other countries in this trend, and as the elderly population increases, the number of people requiring care is also increasing. At the same time, there is a noticeable shortage of nurses and caregivers who can provide care, and there is a need to develop technology to provide appropriate care services.

[0003] For example, in nursing care facilities, there are cases where the person being cared for falls out of bed or gets out of bed but is unable to return and ends up wandering around. Continuing to watch over the person being cared for in such cases places a heavy burden on the caregiver, and it can be difficult to adequately deal with such situations, especially at night when there are fewer staff on duty.

[0004] To address these issues, a behavior detection device has been proposed that captures an image of a care recipient on a bed and analyzes the resulting image to detect when the care recipient gets up, gets out of bed, falls, etc. Specifically, the device identifies the outer edge of the top surface of the bed on the image as the bed area, and determines the positional relationship between the bed area and the care recipient (see, for example, Patent Document 1).

[0005] In this way, the caregiver only needs to be notified and deal with a specific situation when the care recipient falls out of bed, etc., and this significantly reduces the burden of care compared to when the caregiver checks the care recipient's condition by himself / herself. Also, since changes in the care recipient's condition are less likely to be overlooked, the quality of care services can be improved. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2016 / 199495 [Patent Document 2] JP 2018-143333 A [Patent Document 3] JP 2017-041079 A [Patent Document 4] JP 2019-144996 A [Non-patent literature]

[0007] [Non-Patent Document 1] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified. Real-time Object Detection", arXiv: 1506.02640v5, 9 May 2016. [Non-Patent Document 2] Zhe Cao, Gines Hidalgo, Thomas Simon, Yaser Sheikh, "Open Pose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", arXiv: 1812.08008v2, 30 May 2019. Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the above conventional technology, in order to specify the bed area, the user or manager of the behavior detection device must manually input the positions of the four bed corners on the image in advance. This may result in excessive input work in nursing facilities with a large number of beds. In addition, if the positional relationship between the camera capturing the image of the care recipient and the bed changes as a result of moving the bed, the positions of the four bed corners must be reset, which is even more time-consuming.

[0009] To address this problem, a monitoring support system has been proposed that detects the bed using a classifier that uses classical Haar-like features or HoG (Histogram of Gaussian) features, or a classifier that uses Faster R-CNN (Regions of Convolutional Neural Networks), then performs edge extraction processing using an extraction method such as a differential filter or a Laplacian filter, and further detects the outline of the bed using a line detection algorithm such as a Hough transform (see, for example, Patent Document 2).

[0010] In addition, a monitoring support system has been proposed that uses two types of classifiers similar to those described above to detect the two left and right corners at the foot of the bed, and then performs edge extraction processing and straight line detection processing similar to those described above to detect line segments extending downward from each corner of the image as the outline of the bed (also see Patent Document 2).

[0011] Furthermore, a motion recognition device has been proposed that determines the bed area by applying a template matching technique to a grayscale image obtained by capturing an image of the bed from above (see, for example, Patent Document 3).

[0012] In addition, a device has been proposed in which a distance imaging camera is placed in a position where the entire bed fits within the image, the distance from the distance imaging camera to the object to be imaged is measured for each pixel, a distance image is captured in which the obtained distance is the pixel value of that pixel, and the bed area is extracted using the obtained distance image (see, for example, Patent Document 4).

[0013] However, since a bed is usually covered with a comforter or a pillow, or the care recipient is lying on the bed, the outer edge of the bed is often hidden and cannot be seen in the captured image of the bed. Therefore, even if edge detection processing is performed, the outer edge of the bed cannot be detected as an edge, and it is difficult to accurately detect the outline of the bed even if line detection processing is performed using the image obtained by edge detection processing.

[0014] In addition, the above conventional technology is premised on the fact that the positional relationship between the bed and the camera is fixed, since it is necessary to detect the two left and right corners of the foot side of the bed and to detect a line segment starting from the corner and extending downward in the image. Therefore, it cannot be applied when the positional relationship between the bed and the camera is not fixed, and therefore it is not possible to reduce the effort required to reset the bed area in such a case.

[0015] Similarly, when the outer edge of the bed is hidden, it is difficult to accurately determine the bed area using the template matching method. If the bed corners and bed area cannot be accurately determined, for example, there is a risk that the care recipient may be erroneously determined not to have fallen out of bed when in fact they have. This is very dangerous.

[0016] When using distance images, the user is asked to select a specific point on the bed, which reduces the user's workload somewhat compared to when the user is asked to specify the positions of all four bed corners. However, when extracting the bed area, points where the pixel values ​​(distance from the distance image camera) of pixels adjacent to the specific point change suddenly are detected, and then points where the points where the pixel values ​​change suddenly are distributed in a straight line are detected as edge parts.

[0017] For this reason, when the corners of the bed are hidden, pixel values ​​often change suddenly away from the corners of the bed, and in the case of a duvet, for example, the points where the pixel values ​​change suddenly are not distributed in a straight line, making it difficult to detect the edge parts; even if the edge parts can be detected, the corners of the bed cannot be extracted with high accuracy.

[0018] Furthermore, when a normal camera is used for detecting the behavior of the care recipient, a separate distance imaging camera must be installed, which inevitably results in high costs.

[0019] The present disclosure has been made in consideration of the above-mentioned problems, and provides a bed area extraction device that can extract a bed area from a bed image captured from an arbitrary direction, even if a part of the bed is covered. Place The purpose is to provide. [Means for solving the problem]

[0020] In order to achieve the above object, a bed region extraction device according to an embodiment of the present disclosure includes an acquisition unit that acquires a bed image of a bed, and a bed corner detection unit that detects four positions of bed corners from the bed image, and the bed corner detection unit uses a bed corner classifier that has been machine-learned using a machine-learning image including a bed and the positions of the four bed corners in the machine-learning image as training data, and there are multiple training data, The machine learning includes a process of correcting the parameters of the bed corner classifier by a backpropagation method using errors in the positions of the bed corners across multiple training data so that the value of the loss function becomes smaller. fruit, A position calculation unit is provided that calculates the position of the bed corner using the center of the position of the bed corner included in each of the plurality of teacher data, the size of the bed, the camera parameters of the camera that captured the image of the bed, and the distance from the bed to the camera, and the error indicates how much each of the positions of the bed corner included in the plurality of teacher data is deviated from the position of the bed corner calculated by the position calculation unit when the position of the bed corner is set as a target value. It is characterized by:

[0021] Ma The device may further include a position display means for displaying the position of the bed corner detected by the bed corner detection unit together with the bed image, and a position correction means for accepting corrections to the position of the bed corner, and the error may indicate how much the corrected bed corner position is different from the positions of the bed corners included in multiple teaching data when the corrected bed corner position is set as a target value. In this case, it is preferable that the machine learning image included in any one of the plurality of teacher data includes the positions of all four bed corners. It is also preferable that any one of the plurality of teacher data includes an image in which at least one bed corner is hidden as the machine learning image, and it is preferable that any one of the plurality of teacher data includes an image in which all four bed corners are exposed as the machine learning image.

[0022] In addition, the bed image may include the bed By detecting the bounding box The bed corner detecting unit may detect four positions of the bed corners from the bed partial image of the bed image.

[0023] In addition, the bed partial image extraction unit may extract the bed partial image using a bed partial image classifier that has been machine-learned using a machine learning image including a bed and a specification of an area including the bed as training data.

[0024] The bed partial image extraction section may extract, as the bed partial image, an image including at least an entire top surface of a bed.

[0025] A bed region extraction device according to an embodiment of the present disclosure includes: an acquisition unit that acquires a bed image obtained by capturing an image of a bed; and a bed corner detection unit that detects four positions of the bed corners from the bed image. The bed corner detection unit may further include a position determination unit that determines whether the position of the bed corner detected by the bed corner detection unit is appropriate by using the center of the bed corner position detected by the bed corner detection unit, the size of the bed, camera parameters of a camera that captured an image of the bed, and the distance from the bed to the camera.

[0026] In this case, it is preferable that the position determination unit includes a position acquisition means for receiving input of the position of the bed corner when it is determined that the position of the bed corner is not appropriate.

[0027] The bed corner detection unit may further include a position calculation unit that calculates the position of the bed corner using the center of the bed corner position detected by the bed corner detection unit, the size of the bed, the camera parameters of the camera that captured the image of the bed, and the distance from the bed to the camera, and when the position determination unit determines that the position of the bed corner is inappropriate, the bed corner detection unit may set the position calculated by the position calculation unit as the position of the bed corner instead of the position detected by the bed corner identifier.

[0029] The bed corner detecting unit may also include a position memory unit which stores the position of the bed corner detected by the bed corner detecting unit, and a deviation determining means which compares a position of the bed corner previously detected by the bed corner detecting unit with a position of the bed corner newly detected by the detecting unit and determines whether or not the position differs by a predetermined amount. Effect of the Invention

[0033] In this way, bed corner identification is performed using machine learning images containing a bed and the positions of the four bed corners in the main mechanical image as training data, so the bed area can be extracted with high accuracy even if the bed corners are obscured by a person, a duvet, etc. [Brief description of the drawings]

[0034] [Figure 1]FIG. 2 is a diagram showing the main system configuration of the monitoring support system MS. [Diagram 2] 1 is a diagram illustrating an example of the state of a room RM where a person being watched over Ob is staying. [Diagram 3] FIG. 2 is a diagram showing the main functional configuration of the sensor box SB. [Figure 4] (a) illustrates an example of a background image of the bed 20 captured when the person being monitored Ob is not present, (b) illustrates an example of an input image of the bed 20 captured when the person being monitored Ob is present, and (c) illustrates an example of a background difference image obtained by subtracting the background image from the input image. [Diagram 5] FIG. 2 is a diagram showing the main functional configuration of a management server SV. [Figure 6] 13 is a flowchart illustrating a bed area extraction process executed by the management server SV. [Figure 7] (a) is a diagram illustrating an example of a bed image, (b) is a diagram illustrating an example of a bed partial image detected from the bed image, (c) is a diagram illustrating a bed corner position detected from the bed partial image, and (d) is a diagram illustrating a screen display for visual confirmation. [Figure 8] 8A is a diagram illustrating a main portion of a bed image in which a duvet 801 hides bed corners 821, 822, and FIG. 8B is a diagram illustrating the state of a duvet 841 on a bed 20 equipped with a footboard 831. [Figure 9] 13 is a diagram illustrating an example of a region setting table stored in the region setting storage unit 223 for each combination of the sensor box SB and the bed 20. FIG. [Figure 10] 13 is a flowchart illustrating a machine learning procedure for a bed corner classifier. [Figure 11] (a) is a diagram illustrating an example of a bed image captured from above the bed 20, (b) is a diagram illustrating an example of a bed image captured from the side looking down on the bed 20, and (c) is a diagram illustrating an example of a bed image captured from the foot side looking down on the bed 20. [Figure 12]FIG. 2 is an external perspective view illustrating a transformable bed 20 and its bed corners 41A, 41B, 41C, and 41D. [Figure 13] 13A is a diagram for explaining the configuration of a convolutional neural network 1300, and FIG. 13B is a diagram for explaining the configuration of an output layer 1302 of the convolutional neural network 1300. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Hereinafter, embodiments of a bed area extraction device, a bed area extraction method, a bed area extraction program, and a monitoring support system according to the present disclosure will be described with reference to the drawings. [1] Configuration of the monitoring support system First, the configuration of the watching support system according to the present embodiment will be described.

[0036] The monitoring support system MS is a system for supporting caregivers in monitoring those being cared for in nursing facilities, and as shown in Figure 1, is configured such that mobile terminals TA installed in the rooms RM of each monitored person Ob, the nurse station ST, and each caregiver NS are connected to each other via a communication network NW so that they can communicate with each other. (1-1) Sensor Box SB Sensor boxes SB (SB-1 to SB-4) are provided in the rooms RM (RM-1 to RM-4) of the persons being watched over Ob (Ob-1: Person A to SB-4: Person D). As shown in Fig. 2, the sensor boxes SB are installed on the ceiling or walls of the rooms RM so as to be able to watch over the person being watched over Ob on the bed 20. As shown in Fig. 3, the sensor box SB includes a sensor unit 11, an SB voice input / output unit 12, a nurse call input unit 13, an SB control processing unit 14, an SB communication IF (Interface) unit 15, and an SB storage unit 16.

[0037] The sensor unit 11 includes a camera, and captures an image of the space in the room RM, including the bed 20, under the control of the SB control processor 14. When the person being watched over Ob is on the bed 20 or near the bed 20, the person being watched over is also imaged. In this way, the sensor unit 11 generates an image of an overhead view of the top surface of the bed 20 (hereinafter referred to as a "bed image"). Since there is a high probability that the entire person being watched over Ob can be imaged, it is desirable to place the camera directly above the head position (usually the position of the pillow) when the person being watched over Ob is lying on the bed 20.

[0038] As the camera, since it is necessary to monitor the person Ob to be watched over even when he / she is asleep, it is preferable to use a high-sensitivity camera that can obtain a clear image of the bed even with the low light level when the lights are off. Also, an infrared camera can be used to capture an image of the person Ob to be watched over when the lights are off. Also, it is preferable that the optical system of the camera is a wide-angle optical system (a so-called wide-angle lens (including a fisheye lens)) in order to capture as wide an area as possible in the room RM.

[0039] The SB communication IF unit 15 executes processing for the SB control processing unit 14 to communicate with the management server SV, the mobile terminal TA, and the fixed terminal device SP described below via the communication network NW. The SB communication IF unit 15 may further include an interface for inputting and outputting data to and from an external device using standards such as a mobile phone communication network, the WiFi standard, the Bluetooth (registered trademark) standard, the IrDA (Infrared Data Association) standard, and the USB (Universal Serial Bus) standard.

[0040] The SB storage unit 16 is connected to the SB control processing unit 14, and stores various programs and various data under the control of the SB control processing unit 14. The various programs include, for example, a monitoring processing program that executes information processing related to monitoring the person being watched over Ob. The monitoring processing program includes a notification processing program that notifies the outside when a predetermined event occurs. The various data include image data of a bed image captured by a camera, data required to execute various programs, data required to monitor the person being watched over Ob, and the like.

[0041] The SB storage unit 16 includes, for example, a ROM (Read Only Memory) which is a nonvolatile storage element, an EEPROM (Electrically Erasable Programmable Read Only Memory) which is a rewritable nonvolatile storage element, etc. The SB storage unit 16 also includes a RAM (Random Access Memory) which serves as a so-called working memory of the SB control processing unit 14 and stores data generated during execution of the program, etc.

[0042] The SB control processing unit 14 detects the behavior of the watching target Ob. The SB control processing unit 14 is composed of, for example, a CPU (Central Processing Unit) and its peripheral circuits. The SB control processing unit 14 executes various programs to perform functions such as an SB control unit 141, an area setting processing unit 142, a behavior detection processing unit 143, and a notification processing unit 144.

[0043] The SB control unit 141 monitors and controls the operation of the entire sensor box SB.

[0044] The area setting processing unit 142 executes processing for identifying the range of the upper surface of the bed 20 (bed area) in the bed image captured by the sensor unit 11. Specifically, in response to a request from the management server SV described later, the sensor unit 11 captures the bed image, and transmits the obtained bed image to the management server SV via the communication network NW.

[0045] Thereafter, when the SB control unit 141 receives the positions (coordinate values) of the four bed corners in the bed image as the bed area from the management server SV, it stores the received positions of the bed corners as area settings in the area setting storage unit 161 of the SB storage unit 16. The area settings stored in the area setting storage unit 161 are used by the behavior detection processing unit 143, which will be described next, to detect the behavior of the person being watched over Ob.

[0046] In this embodiment, the bed area is specified by the four corners (bed corners) of the bed 20. The top surface of the bed 20 is rectangular, and by specifying the four bed corners, the edge of the top surface of the bed 20 can be specified, so that the shape of the entire top surface of the bed 20 can be specified.

[0047] The behavior detection processing unit 143 detects the behavior of the person being watched over Ob, particularly getting up, getting out of bed, falling over, and falling down, based on the output of the sensor unit 11, and notifies the notification processing unit 144. The behavior detection processing unit 143 extracts a moving object area as the range in which the person being watched over Ob is captured from the bed image captured by the camera of the sensor unit 11, detects the posture of the person being watched over Ob (e.g., standing, sitting, lying down, etc.) from the aspect ratio of the obtained moving object area, and detects the position of the person being watched over Ob from the position of the moving object area, thereby determining the state of the person being watched over Ob.

[0048] For example, as shown in Fig. 4(a), the behavior detection processing unit 143 acquires in advance a background image 30A in which the watching target Ob is not captured, by the sensor unit 11. Thereafter, as shown in Fig. 4(b), when the behavior detection processing unit 143 acquires an input image 30B from the sensor unit 11, it generates a background subtraction image 30C by subtracting the background image 30A from the input image 30B.

[0049] The pixel values ​​of the portions of the input image 30B other than the watching target Ob are generally similar to those of the background image 30A, so the pixel values ​​of the portions other than the watching target Ob are small in the background difference image 30C. On the other hand, for the portion of the watching target Ob, the difference in pixel values ​​between the background image 30A and the input image 30B is large, so that in the background difference image 30C, the pixel values ​​are large only in the portion of the watching target Ob, as shown in FIG. 4(c).

[0050] Therefore, by setting an appropriate threshold value and detecting an area in the background differential image 30C where the pixel value is greater than the threshold value, it is possible to detect the part of the person being watched over Ob, i.e., the moving object area 70. Therefore, it is possible to detect the posture of the person being watched over Ob from the aspect ratio of the moving object area 70, and to detect the position of the person being watched over Ob from the position of the moving object area 70.

[0051] Furthermore, the behavior of the person being watched over Ob can be determined by comparing the bed area 40 defined by the bed corners 41A, 41B, 41C, and 41D with the moving object area 70. In the example of Fig. 4, since the upper part S2 of the moving object area 70 is within the bed area 40 and the lower part S1 is outside the bed area 40, it can be seen that the person being watched over Ob is sitting on the edge of the bed 20.

[0052] Needless to say, the moving object region 70 may be detected using other methods.

[0053] The behavior detection processing unit 143 further determines that the person being watched over Ob has woken up if the person being watched over Ob is on the bed 20 and the posture of the person being watched over Ob has changed from lying down to sitting up. Also, if the position of the person being watched over Ob has moved from on the bed 20 to outside the bed 20 and the posture of the person being watched over Ob has changed from sitting to standing up, it determines that the person being watched over Ob has left the bed.

[0054] Moreover, if the position of the person being watched over Ob moves from on the bed 20 to outside the bed 20 and the posture of the person being watched over Ob changes to lying on his / her side, it is determined that the person being watched over Ob has fallen off the bed 20. Furthermore, if the position of the person being watched over Ob is outside the bed 20 and the posture of the person being watched over Ob changes to lying on his / her side, it is determined that the person being watched over Ob has fallen.

[0055] The behavior detection processing unit 143 may store daily behaviors for each person being watched over Ob, and when the person being watched over Ob behaves differently from the daily behaviors, may notify the mobile terminal TA of the behavior. Also, the notification may be sent when the person being watched over Ob behaves in a specific manner other than the daily behaviors, such as a behavior due to dementia, which is set in advance.

[0056] When the behavior detection processing unit 143 detects a predetermined behavior of the person being watched over Ob, such as getting up, getting out of bed, falling, or tripping, the notification processing unit 144 executes a process for notifying another device, for example, a mobile terminal TA, of the detection. The notification to the mobile terminal TA may be made directly from the sensor box SB or via the management server SV. When making a direct notification, the sensor box SB selects the mobile terminal TA as the notification destination.

[0057] When the notification is sent via the management server SV, the management server SV selects the mobile terminal TA to be the notification destination. The mobile terminal TA distributed to the caregiver NS assigned to each person being watched over Ob or each room RM may be selected as the notification destination, or the work schedule (shift schedule) of the caregiver NS may be taken into consideration. The notification may also be sent to the fixed terminal device SP.

[0058] The notification to the mobile terminal TA includes the type of behavior of the person being watched over Ob, and the identifiers of the sensor box SB that detected the behavior and the bed 20. Furthermore, the person being watched over Ob may be identified from the input image 30B, and the identifier of the person being watched over Ob may be notified.

[0059] The notification processing unit 144 may transmit a bed image captured by the sensor unit 11. The bed image may be transmitted in response to a request from another device such as the mobile terminal TA or the management server SV, or may be transmitted together with a notification when a behavior of the person being watched over Ob is detected. (1-2) Management server SV and fixed terminal device SP A management server SV and a fixed terminal device SP are provided in the nurse station ST.

[0060] As shown in FIG. 5, the management server SV includes an SV control processing unit 21, an SV storage unit 22, and an SV communication IF unit 23.

[0061] The SV communication IF unit 23 executes processing for the management server SV to communicate with other devices such as the sensor box SB, the mobile terminal TA, and the fixed terminal device SP via the communication network NW.

[0062] The SV storage unit 22 stores various programs and various data under the control of the SV control processing unit 21. The various programs include, for example, a bed area extraction program that, when receiving image data of a bed image from the sensor box SB, extracts the bed area 40 included in the image data and returns the generated area setting to the sensor box SB.

[0063] The various data include image data of the bed image received from the sensor box SB, an area setting generated by extracting the bed area 40 from the image data, parameters used by the bed area extraction program to generate the area setting, etc. The SV storage unit 22 includes a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0064] The SV storage unit 22 includes a parameter storage unit 221, an image data storage unit 222, and an area setting storage unit 223. The parameter storage unit 221 stores parameters used by the area extraction program to generate an area setting from image data. Since the area extraction program according to the present embodiment generates an area setting from image data using a neural network, the parameter storage unit 221 stores parameters of the neural network.

[0065] Needless to say, the region extraction program may generate a region setting from image data using a method other than a neural network, and may generate the region setting using a machine-learned classifier other than a neural network.

[0066] The image data storage unit 222 stores image data of the bed image received from the sensor box SB, and the region setting storage unit 223 stores the region setting generated from the image data. The image data stored in the image data storage unit 222 and the region setting stored in the region setting storage unit 223 are used for machine learning as teacher data for the neural network used by the region extraction program. The region setting stored in the region setting storage unit 223 is also used for detecting misalignment of the bed 20, which will be described later.

[0067] The SV control processing unit 21 receives image data of the bed image from the sensor box SB, and returns the area setting generated from the image data to the sensor box SB. The SV control processing unit 21 includes, for example, a CPU and its peripheral circuits, and executes various programs read from the SV storage unit. In this way, the SV control processing unit 21 realizes the functions of the SV control unit 211, bed area extraction unit 212, and parameter update unit 213.

[0068] The SV control unit 211 is responsible for the overall control of the management server SV. The management server SV is connected to an SV input unit 24, an SV output unit 25, and an SVIF unit 26 as necessary. The SV input unit 24 accepts inputs such as various commands and various data. The SV output unit 25 outputs various information to the operator of the SV input unit 24. The SVIF unit 26 is an interface for transmitting and receiving various information to and from other devices.

[0069] When the bed area extraction unit 212 receives image data of the bed image from the sensor box SB, it executes a bed area extraction program to generate an area setting from the image data. In this embodiment, four bed corners on the upper surface of the bed 20 are extracted as the area setting.

[0070] The parameter update unit 213 performs machine learning of the neural network used by the bed area extraction program using the image data of the bed image received from the sensor box SB and the area setting generated from the image data as training data, and generates parameters for the neural network.

[0071] The fixed terminal device SP is, for example, a personal computer, and functions as a user interface (UI) of the monitoring support system MS. It is preferable that the fixed terminal device SP is capable of grasping the state of the room through the function of the sensor box SB in all rooms where the sensor box SB is installed. (1-3) Mobile terminal TA The mobile terminals TA (TA-1, TA-2) are portable communication terminal devices such as tablet computers, smartphones, or mobile phones, and are distributed to each caregiver NS (NS-1: NA, NS-2: NB).

[0072] The mobile terminal TA accesses the network NW by wireless communication via the access point AP, and communicates with the sensor box SB and the management server SV. Through this communication, the mobile terminal TA receives notifications regarding the behavior of the monitored person Ob. The mobile terminal TA is equipped with a user interface such as a touch panel and speaker, and can display a screen or output audio in response to the received notification.

[0073] A notification regarding the behavior of the person being watched over Ob may be sent directly from the sensor box SB to the mobile terminal TA, or may be sent from the sensor box SB to the management server SV once, and then from the management server SV to the mobile terminal TA. When the management server SV relays the notification, the working status of the caregivers, which mobile terminal TA each caregiver carries, and which person being watched over Ob each caregiver is in charge of may be registered in the management server SV, and the mobile terminal TA to which the notification should be sent may be determined based on this registered information.

[0074] In addition, the mobile terminal TA may be equipped with a means for acquiring location information such as a Global Positioning System (GPS), and the location of the mobile terminal TA may be managed by the management server SV. In the event of an emergency, such as when the person being watched over Ob falls off the bed 20 or tumbles, a notification may be sent to the mobile terminal TA that is closest to the person being watched over Ob. [2] Extraction operation of bed area 40 As described above, in order to detect the behavior of the person being watched over Ob, it is necessary to extract the bed area (the range of the upper surface of the bed 20) 40 in advance. For this purpose, the bed area extraction unit 212 of the SV control processing unit 21 constituting the management server SV acquires image data of the bed image captured by the sensor unit 11 of the sensor box SB. The image data may be acquired periodically, or an instruction may be received at the user interface (SV input unit 24) of the management server SV. Alternatively, the caregiver NS may give an instruction using the mobile terminal TA.

[0075] In this embodiment, the sensor unit 11 captures an image of the entire bed 20 and the periphery of the bed 20 from above. The direction in which the sensor unit 11 captures the image of the bed 20 and the person being watched over Ob is not limited to a bird's-eye view, as long as at least the bed area 40 is captured. For example, when multiple people being watched over Ob share one room RM, at least the number of beds 20 for the number of people being watched over Ob are installed in the room RM, and when more than the number of beds 20 are installed, the beds exceeding the number of people being watched over become vacant beds.

[0076] When one sensor box SB detects the behavior of multiple monitoring targets Ob, the imaging angle of the sensor unit 11 may be adjusted so that all beds 20 in the room RM are captured in one image. If it is difficult to capture an image of all beds 20 in the room RM in one image due to the arrangement of the sensor unit 11 and the beds 20, the imaging angle may be changed for each bed 20. In this case, it goes without saying that vacant beds 20 may be excluded from the image capture target.

[0077] Furthermore, if an omnidirectional camera is used, it is possible to capture images of all the beds 20 without changing the capturing angle. In this case, it is desirable not only to generate a bed partial image 702 for each bed 20 from the bed image 701 captured in all directions, but also to correct distortion of the bed partial image 702 by performing coordinate conversion processing on the bed partial image 702 so that the bed region extraction processing and behavior detection processing are not adversely affected by distortion of the bed partial image 702.

[0078] As shown in Fig. 6, when the bed region extraction unit 212 acquires an image of a bed image from the sensor box SB (S601), it cuts out a portion including the bed 20 from the bed image to acquire a bed partial image (S602). When a bed image 701 as shown in Fig. 7(a) is acquired, a bed partial image 702 as shown in Fig. 7(b) is acquired.

[0079] As a bed partial image classifier that acquires the bed partial image 702 from the bed image 701, for example, YOLO (You Only Look Once, see Non-Patent Document 1) may be used to detect a bounding box including the bed 20 using the bed image 701 as input data, and the bounding box may be used as the bed partial image 702. When YOLO is used as the bed partial image classifier, the machine learning of the bed partial image classifier also conforms to the machine learning of YOLO. Needless to say, the bed partial image classifier may acquire the bed partial image 702 using a machine learning model other than YOLO.

[0080] By acquiring the bed partial image 702, the accuracy of detecting the position of the bed corners can be improved since it is possible to avoid the influence of image objects outside the bed partial image 702 compared to detecting the position of the bed corners of the bed 20 directly from the entire bed image 701. Also, the processing load for detecting the position of the bed corners can be reduced.

[0081] Next, the positions of the four bed corners of the bed 20 included in the bed partial image 702 are detected (S603). As a bed corner identifier for detecting the positions of the bed corners, for example, a convolutional neural network (CNN) can be used. In this case, the bed partial image 702 is input, and the positions (coordinates) of the four bed corners are output.

[0082] In addition, the position of the bed corners may be detected using a machine learning model other than a convolutional neural network, but it is preferable to detect the position of the bed corners from the characteristics of the entire bed area, including the positions of at least four bed corners, rather than just the vicinity of the bed corners.

[0083] As shown in FIG. 7 of the above-mentioned conventional technology (Patent Document 1), when a classifier is used to detect each bed corner individually by referring to a detection window 70 that includes only a portion of the bed 20, the characteristics of the entire bed 20 cannot be utilized. Therefore, if the relevant portion is obscured by the person being watched over Ob, the duvet, etc., it becomes difficult to detect the bed corner, and erroneous detection is unavoidable.

[0084] 8(a), the comforter 801 is thick and has a small curvature at corner portions 811, 812 of the comforter 801, making it difficult to identify the corner portions 811, 812 of the comforter in the first place. If one attempts to detect the corners by referring only to the local features of the corner portions 811, 812, there is a high possibility that the detection of the corner portions 811, 812 of the comforter 801 will fail. Even if the corner portions 811, 812 of the comforter 801 are detected, there is no guarantee that they are close to the bed corners 821, 822.

[0085] Furthermore, as illustrated in FIG. 8(b), when a footboard 831 is provided on the bed 20, the duvet 841 pressed against the footboard 831 often protrudes significantly to the side (in the direction of arrow A). In such a case, there is a risk that the corner portion of the duvet 841 that is located further away from the bed corners 851, 852 will be erroneously detected as the bed corners 851, 852.

[0086] On the other hand, in this embodiment, four bed corners are detected from features extracted from the entire bed portion image 702, so that even if the bed corners are hidden by the person being watched over Ob or the duvet, or even if the way the duvet is draped changes due to a footboard, etc., the bed corners can be detected with high accuracy.

[0087] The bed partial image 702 includes parts whose positional relationship with the bed corners is unlikely to change, such as the side rails 861 and 862 shown in Fig. 8(a) and the foot board 831 shown in Fig. 8(b). If feature amounts are extracted from the entire bed partial image 702, the bed corners can be detected by taking into consideration the features of parts whose spatial (three-dimensional) positional relationship with the bed corners is unlikely to change, such as the side rails 861 and 862 and the foot board 831, due to the structure of the bed 20. In this sense, the detection accuracy of the bed corners can be improved.

[0088] Furthermore, if a convolutional neural network is trained using bed images of the bed 20 captured from various directions as training data, the bed area can be extracted without being limited to a specific direction in which the bed 20 is captured, because the convolutional neural network has a generalization ability.

[0089] Then, the coordinate values ​​of the bed corners in the bed partial image 702 obtained using a convolutional neural network are converted into coordinate values ​​in the bed image 701.

[0090] Next, it is confirmed whether the positions of the bed corners detected in step S603 are appropriate (S604). In this embodiment, first, the center position of the four bed corners detected in step S603 is calculated. Next, the size of the bed 20, the camera parameters of the camera of the sensor unit 11, and the distance from the center of the camera of the sensor unit 11 to the bed 20 are acquired, and the positions of the bed corners relative to the center position of the bed corners calculated earlier are calculated.

[0091] The position of the bed corner calculated in this manner is compared with the position of the bed corner detected in step S603 to confirm whether the detected position of the bed corner is appropriate.

[0092] If it is determined that the position of the bed corner is not appropriate and needs to be corrected (S605: YES), the position calculated in step S604 is used instead of the position detected in step S603 for the bed corner determined to need correction or for all four bed corners (S606).

[0093] If it is determined that the position of the bed corner is appropriate and does not need to be corrected (S605: NO), and after completing the process of step S606, the management server SV refers to the area setting storage unit 223 of the SV storage unit 22 and reads out the previously set position of the bed corner. Since multiple beds 20 may be placed in the room RM of the person being watched over Ob, the area setting storage unit 223 records an area setting table 9 that stores the X and Y coordinates of four bed corners for each set date and time as area settings for each bed 20 imaged by the sensor box SB, as shown in Fig. 9.

[0094] The ID column of the area setting table 9 is an identifier for associating the coordinate values ​​of the four bed corners with the bed partial image 702 stored in the image data storage unit 222, and is referenced during machine learning of the bed corner classifier described later.

[0095] When the combination of the sensor box SB and the bed 20 is specified and the previously set position of the bed corner is read from the area setting storage unit 223, it is compared with the currently extracted position of the bed corner, and if the position of the bed corner deviates by a predetermined threshold or more, it is determined that a deviation of the bed corner position has been detected. Even if the previously set position of the bed corner is not stored in the area setting storage unit 223, it is determined that a deviation of the bed corner position has been detected.

[0096] If no deviation in the position of the bed corner is detected (S607: NO), the bed area remains the same and there is no need to set a new bed area, so the process ends.

[0097] When a misalignment of the bed corners is detected (S607: YES), an image display is performed in which icons representing the positions of the bed corners are superimposed on the bed image 701, and visual confirmation is performed (S608). In the screen display 703 illustrated in Fig. 7(d), the positions of the four bed corners 41A, 41B, 41C, and 41D are shown by displaying cross-shaped icons superimposed on the bed image 701. Needless to say, the icons representing the positions of the bed corners 41A, 41B, 41C, and 41D are not limited to cross-shaped icons, and the positions of the bed corners 41A, 41B, 41C, and 41D may be represented by using icons of other shapes.

[0098] The cross-shaped icons indicating the positions of the bed corners 41A, 41B, 41C, and 41D can be moved by mouse operation or touch operation. If the person checking judges that the positions of the bed corners 41A, 41B, 41C, and 41D are correct, he or she simply operates the OK button 704 to input that effect.

[0099] On the other hand, if it is determined that the positions of the bed corners 41A, 41B, 41C, and 41D are incorrect, the cross icon is moved to the correct position, and then the correction button 705 is operated to input that the positions should be corrected.

[0100] The screen display may be performed using the SV output unit 25 of the management server SV, or may be performed using a fixed terminal device SP. Also, the screen display may be performed using a portable terminal TA carried by the caregiver. The device to be used for visual confirmation may be set in advance in the management server SV. Also, the sensor box SB and its subordinate bed 20 may be specified from the fixed terminal device SP or the portable terminal TA to request the management server SV to extract the bed area. In such a case, it is desirable to perform the visual confirmation on the requesting fixed terminal device SP or portable terminal TA.

[0101] If the result of the visual confirmation is that the newly detected bed corner position is not to be adopted (S609: NO), then resetting of the bed corner position is accepted (S610). If the SV output unit 25 of the management server SV is used for the screen display for visual confirmation, the SV input unit 24 of the management server SV may be used to accept the resetting. Similarly, if the screen display for visual confirmation is performed on the fixed terminal device SP (or the mobile terminal TA), it is desirable to also accept the resetting on the fixed terminal device SP (or the mobile terminal TA).

[0102] If the newly detected bed corner position is to be adopted as a result of the visual confirmation (S609: YES), the sensor box SB is notified of the adopted bed corner position, and if a reset of the bed corner position is accepted in step S610, the sensor box SB is notified of the reset bed corner position (S611). The sensor box SB that has been notified of the bed corner position stores the bed corner position in the area setting memory unit 223 of the SB memory unit 16, and uses it in the subsequent behavior detection process of the person being watched over Ob.

[0103] The management server SV further stores the image data of the bed image 701 received from the sensor box SB and the bed partial image 702 acquired from the bed image 701 in the image data storage unit 221 of the SV storage unit 22, and stores the position of the bed corner in the area setting storage unit 223 in association with the bed image and the bed partial image (S612), and ends the process. The stored image data and the position of the bed corner are used for machine learning of a bed partial image classifier and a bed corner classifier.

[0104] In the machine learning of the bed partial image classifier, the teacher data used are a bed image 701, the center coordinates, width and height of a bed partial image 702 to be obtained from the bed image, and the type of object representing the bed 20. The bed image 701 may be a bed image received from the sensor box SB, or another image capturing a bed may be used.

[0105] In the machine learning of the bed-corner classifier, the parameter update unit 213 of the SV control processing unit 21 of the management server SV, as shown in Figure 10, first initializes the working variable epoch, which indicates the number of times the parameters of the bed-corner classifier are revised, to 0 (S1001), and further initializes both the working variable i, which indicates the identification number of the training data, and the total error value S calculated by inputting the training data into the bed-corner classifier, to 0 (S1002).

[0106] Next, as image data for machine learning among the i-th teacher data, image data of the i-th bed partial image is read from the image data memory unit 222 of the SV memory unit 22 of the management server SV, and the position of the i-th bed corner corresponding to the i-th bed partial image is read from the area setting memory unit 223 (S1003).

[0107] The image data of the read-out bed partial image is input to a bed corner identifier, and the position of the bed corner is calculated (S1004). In this case, the bed corner identifier uses the current parameters read from the parameter storage unit 221 of the SV storage unit 22 of the management server SV. An error δ between the bed corner position calculated in this way and the bed corner position (target value) read from the area setting storage unit 223 is calculated (S1005), and the obtained error δ is added to the total error value S (S1006).

[0108] Then, the value of the working variable i is incremented by 1 (S1007), and the total number Nd of training data used for machine learning is compared with the working variable i. If the working variable i is smaller than the total number Nd of training data (S1008: YES), the process proceeds to step S1003, where the error δ when the next training data (i-th training data) is used is calculated and added to the total error value S.

[0109] When the working variable i reaches the total number Nd of training data (S1008: NO), the parameters of the bed-corner classifier are corrected by the backpropagation method using the total error value S (S1009). In this case, the parameters of the bed-corner classifier are corrected by referring to the total error value S so that the value of the loss function becomes smaller.

[0110] Then, the value of the working variable epoch is incremented by 1 (S1010), and compared with the number of times (learning times) Ne for correcting the parameters of the bed-corner classifier. If the working variable epoch is smaller than the number of corrections Ne (S1011: YES), the process proceeds to step S1002, and the training data is read again to further correct the parameters of the bed-corner classifier.

[0111] When the working variable epoch reaches the number of revisions Ne (S1011: NO), the machine learning of the bed-corner classifier is terminated.

[0112] In machine learning of a bed-corner classifier, in order to be able to detect the position of a bed corner even in an image of one or more bed corners that are obscured by the person being watched over Ob, a duvet, a pillow, etc., it is desirable to include images of bed corners with hidden corners in the training data.

[0113] Also, a bed partial image in which all four bed corners are exposed can be used as training data. Regardless of whether the bed corners are hidden or exposed, it is preferable to use a bed partial image that includes all four positions to be detected as bed corners as training data. In this case, the entire top surface of the bed 20 is included in the bed partial image.

[0114] In addition, depending on the room RM, restrictions may be imposed on the placement of the camera of the sensor unit 11 of the sensor box SB, and the imaging direction of the bed 20 may vary. For this reason, it is preferable to use bed partial images obtained by imaging the bed 20 from various directions as training data. For example, bed partial images obtained from a bed image 1101 captured from above as shown in FIG. 11(a) may be used.

[0115] Furthermore, the detection accuracy of the bed corner classifier can be improved by using a bed partial image acquired from a bed image 1102 captured from a side view as shown in FIG. 11(b) or a bed partial image acquired from a bed image 1103 captured from a side view as shown in FIG. 11(c).

[0116] In the bed images 1101 and 1102 shown in Fig. 11(a) and (b), all four corners of the bed are exposed, whereas in the bed image 1103 shown in Fig. 11(c), only one corner of the bed is hidden by the person being watched over Ob. Bed images captured from various angles in which the corner of the bed is hidden by a comforter or the like, other than the person being watched over Ob, may also be used.

[0117] Also, a bed image including exposed bed corners may be combined with people or a blanket to hide the bed corners, and the bed image may be used as training data. In this way, the accuracy of detecting the four bed corners can be improved even from a bed partial image in which all four bed corners are hidden and only the middle part of the bed 20 is exposed. Furthermore, the entire bed image may be used as training data instead of the bed partial image.

[0118] In the above-mentioned conventional technology (Patent Document 1), the line segment extending downward from the bed corner on the foot side is detected as the outer edge of the bed area, so the positional relationship between the bed and the camera must be fixed. On the other hand, in this embodiment, the positional relationship between the bed 20 and the camera is not fixed, because four bed corners are detected regardless of the imaging direction of the bed 20.

[0119] Therefore, when a single camera is used to capture images of multiple beds 20, the image capturing direction differs for each bed 20, but even in such a case, the bed corners can be detected according to this embodiment. Therefore, there is no need to install individual cameras for all beds 20, and the number of cameras required can be reduced compared to the above-mentioned conventional technology, thereby reducing the cost of the device for monitoring the person being watched over Ob.

[0120] Furthermore, since the resolution of the imaging elements mounted on cameras has increased significantly in recent years, the behavior of the person being watched over Ob can be detected for each bed partial image by using bed partial images cut out for each bed 20 from bed images captured of multiple beds 20.

[0121] Needless to say, the positions of the four bed corners are detected using images captured by a camera used in the behavior detection process of the monitored person Ob, which makes it possible to reduce costs, unlike the above-mentioned conventional technology (Patent Document 4), which requires a separate distance imaging camera. [3] Variations While the present disclosure has been described above based on the embodiments, it goes without saying that the present disclosure is not limited to the above-described embodiments, and the following modified examples can be implemented. (3-1) In the above embodiment, a case has been described where the position of a newly detected bed corner is deviated from the position of a previously detected bed corner and the position of the bed corner is visually confirmed. However, it goes without saying that the present disclosure is not limited to this, and visual confirmation may be performed regardless of whether a misalignment of the bed corner is detected.

[0122] For example, if the position of the bed corners is erroneously detected and it is erroneously determined that the bed corners are not misaligned when in fact they are, this may hinder the detection of the behavior of the person being monitored Ob. In preparation for such a case, it is effective to always perform a visual check regardless of whether a misalignment of the bed corners has been detected. (3-2) In the above embodiment, the example of installing the monitoring support system MS in a nursing facility has been described. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

[0123] For example, the monitoring support system MS may support nurses in watching over patients in a hospital, or may support watching over people living alone through security services, etc. Also, the monitoring support system MS may support watching over young children in a childcare facility. Regardless of the facility in which the monitoring support system MS is installed and the person Ob to be watched over, the monitoring support system MS can support watching over the person Ob to be watched over who uses a bed, such as by getting up, getting out of bed, or falling. (3-3) In the above embodiment, the mattress of the bed 20 used by the person being watched over Ob maintains a fixed shape, but the present disclosure is not limited to this, and the shape of the mattress may be variable. Some nursing beds have a mattress placed on a movable bottom part, and the backrest and footrest can be reclined by moving the bottom part by remote control.

[0124] For this reason, as shown in Fig. 12, for example, the mattress 1201 may be deformed into a complex shape other than a rectangular parallelepiped shape. Even with such a complex shape, if the bed corner classifier is trained in advance by machine learning, the bed corners 41A, 41B, 41C, and 41D can be detected with high accuracy. In Fig. 12, the bed corners 41B and 41D are hidden by the footboard 1202, but according to the present disclosure, the features are extracted from the overall shape of the bed 20, so the positions of the hidden bed corners 41B and 41D can also be detected.

[0125] In addition, for bed 20 that can be transformed by remote control operation in this manner, the presence or absence of remote control operation can be monitored by sensor box SB, and if remote control operation is detected, bed area extraction processing can be performed, thereby extracting the bed area in accordance with the transformation of bed 20.

[0126] Furthermore, when the bed 20 is deformed by reclining various parts of the bed 20, the behavior of the person being watched over Ob is more likely to be restricted and the behavioral pattern may change, compared to when the bed 20 is a rectangular parallelepiped. Therefore, it is desirable to perform the behavior detection process of the person being watched over Ob in accordance with the deformation state of the bed 20. (3-4) As shown in FIG. 13(a), the convolutional neural network used as the bed corner classifier may extract features of the bed partial image 702 using a convolutional layer 1301, and input the output of the convolutional layer 1301 to a fully connected layer (output layer) 1302 to detect the positions (coordinates) of the four bed corners.

[0127] This convolutional layer 1301 detects characteristic patterns such as edges and corners at various points in the bed partial image 702, and detects various characteristic combinations of patterns. What patterns and combinations of patterns to detect are learned by machine learning using various bed images captured of the bed 20 as training data. The output data of the convolutional layer 1301 includes information on where in the bed partial image 702 the combination of patterns to be detected is located, so the output layer 1302 can detect the position of the bed corners.

[0128] As shown in Fig. 13(b), the output layer 1302 has three layers for each of the bed corners 41A, 41B, 41C, and 41D: a layer for detecting the X coordinates, a layer for detecting the Y coordinates, and a layer for expressing the reliability of the coordinate values, for a total of 12 layers. Each layer for detecting the coordinate values ​​is divided into N x M grids, and the X and Y coordinates are expressed by expressing where the bed corner coordinates are located within the grid with an output value between 0 and 1.

[0129] The reliability may also be expressed, for example, by an output value greater than or equal to 0 and less than or equal to 1, with the reliability being higher as the output value approaches 1. It goes without saying that the output format of the coordinate values ​​of the bed corners 41A, 41B, 41C, and 41D is not limited to the above, and the coordinate values ​​of the bed corners 41A, 41B, 41C, and 41D may be output in other formats.

[0130] In this way, when the reliability of each bed corner coordinate value is output in the output layer 1302, when checking whether the position of the bed corner is appropriate in the bed area extraction process shown in FIG. 6 (step S604), the reliability of each bed corner coordinate can be compared with a predetermined threshold value, and if there is a coordinate whose reliability is lower than the threshold value, it can be determined that the bed corner has not been detected, is not appropriate, and requires confirmation or correction.

[0131] In addition to the neural network having the above-mentioned configuration, for example, a technology is known that estimates the skeleton from the appearance of the human body covered by clothes or skin (see Non-Patent Document 2, for example), and the reliability of the detection results can also be evaluated using such a neural network. In this way, even if the neural network has the above-mentioned configuration, it is possible to detect the positions of the four bed corners and evaluate the reliability of the detection results by applying a neural network with an appropriate configuration. (3-5) Although not specifically mentioned in the above embodiment, the camera of the sensor unit 11 of the sensor box SB may capture still images or may capture moving images. When capturing moving images, a still image may be extracted from the captured moving images and used as the bed image 701. In addition, in order to detect the four bed corners, it is desirable that the positions (coordinates) of at least the four bed corners are included in the bed image 701.

[0132] For this reason, it is advisable to adjust the imaging conditions, such as the position of the bed 20 and the position of the camera of the sensor unit 11, so that the entire top surface of the bed 20 is within the imaging range. In this case, the entire top surface of the bed 20 does not need to be exposed, and some parts of the top surface of the bed 20 may be hidden by a person, a comforter, a headboard, a footboard, a side rail, a care bar, a table, etc. In addition, the hidden parts do not need to be constant, and may change each time imaging is performed. (3-6) In the above embodiment, the camera of the sensor unit 11 captures an image of the bed 20 from a direction overlooking the bed 20, but the present disclosure is not limited to this, and the bed 20 may be captured from a direction other than a bird's-eye view instead. Regardless of the image capturing direction of the bed 20, the bed area extraction unit 212 according to the present disclosure can detect the positions of the four bed corners 41A, 41B, 41C, and 41D of the bed 20 and extract the bed area. (3-7) In the above embodiment, an example has been described in which the camera parameters of the camera in the sensor unit 11 are used to determine whether the positions of the bed corners detected by the bed corner classifier are appropriate or should be corrected. In order to calculate the three-dimensional coordinates of the four bed corners detected by the bed corner classifier from the bed image 701, the distortion aberration parameters (lens distortion correction coefficients), the horizontal center pixel position u0 on the imaging plane, the vertical center pixel position v0, etc. can be used.

[0133] After determining the three-dimensional coordinates of the four bed corners, if, for example, the size of the bed 20 calculated from the three-dimensional coordinates differs from the size of the bed 20 that has been registered in advance, it is determined that the position of the bed corner detected by the bed corner identifier is not appropriate and requires correction.

[0134] In addition, the range in which the bed corner may exist in relation to the center position of the bed corner in the bed image 701 may be calculated from the distance from the center of the camera of the sensor unit 11 to the bed 20 and the size of the bed 20, and it may be determined that if the position of the bed corner detected by the bed corner identifier is within this range, it is appropriate, and if it is outside this range, correction is required. (3-8) Although not specifically mentioned in the above embodiment, the bed partial image 702 acquired by the bed partial image classifier from the bed image 701 may have a different size depending on the type and imaging conditions of the bed 20. For this reason, before inputting the bed partial image 702 to the bed corner classifier, the size of the bed partial image 702 may be converted to match the size of the input layer of the bed corner classifier.

[0135] For example, if the size of the bed partial image 702 output by the bed partial image classifier is H1 pixels in height and W1 pixels in width, and the size of the input layer of the bed corner classifier is H2 pixels in height and W2 pixels in width, the height of the bed partial image 702 may be multiplied by H2 / H1 and the width by W2 / W1.

[0136] If the bed partial image 702 is input to the bed corner classification layer as is without size conversion, it is necessary to adjust the size of the input layer of the bed corner classifier to the maximum size that the bed partial image classifier may output, in other words, the same size as the bed image 701. In this way, if the size of the bed partial image 702 is smaller than that of the bed image 701, there will be places where image data is not input to the input layer of the bed corner classifier, and this will unnecessarily generate a processing load on the bed corner classifier that is not actually necessary for detecting the position of the bed corner.

[0137] In addition, if the size of the bed partial image 702 is unnecessarily large for detecting the position of the bed corner, the size of the bed partial image 702 can be converted to a smaller size to reduce the processing load of the bed corner classifier without reducing the detection accuracy of the bed corner position. (3-9) As described above, the monitoring support system MS is a computer system including a microprocessor and a memory. The memory stores a computer program, and the microprocessor operates according to the computer program.

[0138] In the above embodiment, the sensor box SB, management server SV, fixed terminal device SP and mobile terminal TA that constitute the monitoring support system MS are each a computer system, and realize the functions described above by executing a computer program.

[0139] Here, a computer program is a combination of multiple instruction codes that indicate commands for a computer to achieve a specific function.

[0140] The computer program may be recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, an optical disk, or a semiconductor memory.

[0141] The computer program may also be transmitted via a wired or wireless telecommunications line, a network such as the Internet, data broadcasting, or the like. (3-10) The present disclosure may be a bed area extraction method executed by a computer according to a bed area extraction program. By implementing the bed area extraction method according to the present disclosure, it is possible to obtain the same effects as those of the above-mentioned embodiment. [Industrial Applicability]

[0142] The bed area extraction device, bed area extraction method, bed area extraction program, and monitoring support system disclosed herein are useful as technologies for detecting a bed area from a bed image captured from any direction with part of the bed covered, for the purpose of detecting actions by a monitored person Ob, such as getting up, getting out of bed, falling, and tripping. [Explanation of symbols]

[0143] MS: Monitoring support system NS……………………………………Caregiver Ob……………………………………Person to be watched over SB……………………………………Sensor box SV……………………………………Management Server TA…………………………………Mobile terminal 11....................................................................Sensor section 20……………………………………Bed 21...................................................SV control processing section 24...................................................................SV input section 25……………………………………SV output section 41A, 41B, 41C, 41D…Bed corner 142...................................................................Area setting processing unit 143...…………………………Behavior detection processing unit 161……………………………………Area setting storage section 212...………………………………Bed area extraction unit 213...................................................................Parameter update section 221...................................................Parameter storage section 222.....Image data storage unit 223……………………………………Area setting storage section 701, 1201~1203... Bed images 702…………………………………Bed part image

Claims

1. An acquisition unit that acquires a bed image obtained by capturing an image of a bed; a bed corner detection unit that detects four positions of the bed corners from the bed image, The bed corner detection unit uses a bed corner classifier that has been machine-learned using a machine-learning image including a bed and the positions of four bed corners in the machine-learning image as training data, There are multiple teacher data, The machine learning method includes: The method includes a process of correcting parameters of the bed corner classifier by a backpropagation method using errors in the positions of the bed corners across multiple training data so that the value of the loss function becomes smaller; a position calculation unit that calculates a position of the bed corner using the center of the position of the bed corner included in each of a plurality of teaching data, the size of the bed, the camera parameters of the camera that captured the image of the bed, and the distance from the bed to the camera; The error indicates how much each of the positions of the bed corners included in the plurality of teaching data is deviated when the position of the bed corner calculated by the position calculation unit is set as a target value. A bed area extraction device characterized by the above.

2. Among the plurality of training data, any one of the training data includes an image in which at least one bed corner is hidden as the machine learning image.

2. The bed area extraction device according to claim 1.

3. Among the plurality of training data, one of them includes an image in which all four bed corners are exposed as the machine learning image.

3. The bed area extraction device according to claim 1 or 2.

4. A bed partial image extraction unit extracts a bed partial image by detecting a bounding box including the bed from the bed image, The bed corner detection unit detects four positions of the bed corners from a bed partial image of the bed image.

4. The bed area extraction device according to claim 1, wherein the bed area extraction device is a bed area extracting device.

5. The bed corner classifier constitutes a convolutional neural network, The bed partial image extraction unit extracts the bed partial image using a bed partial image classifier that has been machine-learned using a machine-learning image including a bed and a specified range including the bed as training data.

5. The bed area extraction device according to claim 4.

6. The bed partial image extraction unit extracts an image including at least the entire upper surface of the bed as the bed partial image.

6. The bed area extraction device according to claim 4 or 5.

7. a position storage unit that stores the position of the bed corner detected by the bed corner detection unit; The bed corner detection unit includes a deviation determination unit that compares a position of a bed corner previously detected with a position of a bed corner newly detected by the detection unit and determines whether or not the difference is greater than a predetermined value.

7. The bed area extraction device according to claim 1, wherein the bed area extraction device is a bed area extracting device.

8. An acquisition unit that acquires a bed image obtained by capturing an image of a bed; a bed corner detection unit that detects four positions of the bed corners from the bed image using a bed corner classifier that has been machine-trained using a machine learning image including a bed and the positions of four bed corners in the machine learning image as training data; a position determination unit that determines whether the position of the bed corner detected by the bed corner detection unit is appropriate, using the center of the position of the bed corner detected by the bed corner detection unit, the size of the bed, the camera parameters of the camera that captured the image of the bed, and the distance from the bed to the camera; A bed area extraction device comprising:

9. The position determination unit includes a position acquisition unit that receives input of the position of the bed corner when the position determination unit determines that the position of the bed corner is not appropriate.

9. The bed area extraction device according to claim 8.

10. a position calculation unit that calculates a position of a bed corner using the center of the position of the bed corner detected by the bed corner detection unit, the size of the bed, camera parameters of a camera that has captured an image of the bed, and a distance from the bed to the camera; When the position determination unit determines that the position of the bed corner is not appropriate, the bed corner detection unit sets the position calculated by the position calculation unit as the position of the bed corner instead of the position detected by the bed corner identifier.

9. The bed area extraction device according to claim 8.

Images