Information processing device and information processing program

Abstract

To provide an information processing device and information processing program capable of identifying an outer shape of bedding in an image.SOLUTION: An information processing device comprises: an acquisition section 111 for acquiring image information related to an image obtained by photographing an area including bedding to be used by an object person and respiration information related to a respiration of the object person; an estimation section 112 for estimating a candidate area of the bedding in the image based on the acquired image information; a determination section 113 for determining whether the object person exists on the bedding based on the acquired respiration information; and an identification section 114 for identifying an outer shape of the bedding in the image based on the estimated candidate area of the bedding when it is determined that the object person exists on the bedding.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] In nursing care facilities such as hospitals and elderly care facilities, there is a risk that care recipients may fall while walking, fall out of bed and injure themselves, or wander around and go missing. Therefore, development of monitoring systems for monitoring the sleep, behavior, etc. of care recipients is underway. For example, Patent Document 1 discloses an invention related to a system for monitoring the behavior of care recipients. The monitoring system uses, for example, images captured by a camera or the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-144996 Summary of the Invention [Problem to be solved by the invention]

[0004] In such a monitoring system, it is desirable to identify the outline of the bedding in the captured image, which makes it easier for caregivers to understand the daily life of the care recipient, focusing on the bedding, such as when the care recipient wakes up and gets out of bed.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide an information processing device and an information processing program that are capable of identifying the outline of bedding in an image. [Means for solving the problem]

[0006] The above object of the present invention can be achieved by the following means.

[0007] (1) An information processing device comprising: an acquisition unit that acquires image information regarding an image in which an area including bedding that can be used by a subject and respiratory information regarding the subject's breathing; an estimation unit that estimates a candidate area of ​​the bedding in the image based on the acquired image information; a judgment unit that determines whether the subject is on the bedding based on the acquired respiratory information; and an identification unit that, when it is determined that the subject is on the bedding, identifies the outline of the bedding in the image based on the estimated candidate area of ​​the bedding.

[0008] (2) The information processing device described in (1) above, in which the subject's breathing is detected by a breathing detection unit installed at a position away from the bedding.

[0009] (3) The information processing device according to (2) above, wherein the acquisition unit further acquires distance information relating to the distance between the breathing detection unit and the subject.

[0010] (4) The information processing device according to (3), wherein the estimation unit further estimates a candidate position of the subject in the image based on the distance information.

[0011] (5) The information processing device described in (4) above, wherein the identification unit, when it is determined that the subject is on the bedding, identifies the outline of the bedding based on the estimated candidate area of ​​the bedding and the candidate position of the subject.

[0012] (6) The information processing device according to (3) above, wherein the respiration detection unit includes a Doppler sensor.

[0013] (7) The information processing device described in (6) above, wherein the breathing information and the distance information include information regarding the frequency detected by the Doppler sensor.

[0014] (8) The information processing device described in (5) above, wherein the identification unit identifies the outline of one area selected from a plurality of divided areas that overlaps with the estimated candidate position of the subject as the outline of the bedding.

[0015] (9) An information processing device according to (8) above, wherein one area selected from the plurality of divided areas is the largest area among the areas overlapping with the estimated candidate position of the subject.

[0016] (10) An information processing device as described in (8) above, wherein one area selected from the multiple divided areas is an area having a predetermined shape among areas that overlap with the estimated candidate position of the subject.

[0017] (11) The information processing device according to (10), wherein the predetermined shape is a rectangle having a predetermined ratio between the long side and the short side.

[0018] (12) An information processing program for causing a computer to execute a process including acquiring image information regarding an image in which an area including bedding that may be used by a subject and respiratory information regarding the subject's breathing, estimating a candidate area of ​​the bedding in the image based on the acquired image information, determining whether the subject is present on the bedding based on the acquired respiratory information, and, when it is determined that the subject is present on the bedding, identifying the outline of the bedding in the image based on the estimated candidate area of ​​the bedding. [Effects of the Invention]

[0019] In the information processing device and information processing program of the present invention, a candidate region of bedding in an image is estimated based on image information. Furthermore, whether or not a subject is present on the bedding is determined based on breathing information. Then, when it is determined that a subject is present on the bedding, the outline of the bedding in the image is identified based on the candidate region of bedding. Therefore, it is possible to identify the outline of the bedding in the image. [Brief explanation of the drawings]

[0020] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for purposes of illustration only and are not intended to be limiting. [Figure 1] 1 is a schematic diagram illustrating a care system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a detection unit. [Figure 3] FIG. 2 is a block diagram showing the configuration of a terminal device. [Figure 4] FIG. 1 is a block diagram showing a configuration of an information processing device. [Figure 5] FIG. 2 is a diagram showing an example of an image captured by an imaging unit. [Figure 6] FIG. 10 is a diagram illustrating an example of an object rectangle estimated by an estimation unit. [Figure 7] FIG. 10 is a diagram illustrating an example of candidate locations of a resident estimated by an estimation unit. [Figure 8] FIG. 10 is a diagram illustrating an example of Doppler data decomposed into frequency components. [Figure 9] FIG. 10 is a diagram showing an example of the respiratory rate per unit time calculated from Doppler data. [Figure 10] 10 is a flowchart illustrating an example of an information processing method executed by the information processing device. [Figure 11] FIG. 10 is a block diagram showing an imaging unit of a nursing care system according to a first modified example. [Figure 12] 12 is a subroutine flowchart showing an example of processing executed by the care system shown in FIG. [Figure 13] FIG. 10 is a diagram showing height information. [Figure 14] FIG. 13 is a schematic diagram for explaining the processing shown in FIG. 12. [Figure 15] FIG. 13 is another schematic diagram for explaining the processing shown in FIG. 12. [Figure 16] 10 is a subroutine flowchart showing an example of processing executed by a care system according to a second modification. [Figure 17]FIG. 17 is a schematic diagram for explaining the processing shown in FIG. 16. [Figure 18] 10 is a subroutine flowchart showing an example of processing executed by a care system according to a third modification. [Figure 19] FIG. 19 is a schematic diagram for explaining the processing shown in FIG. 18. [Figure 20] 10 is a flowchart showing an example of processing executed by a care system according to a fourth modification. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0022] <Embodiment> [Configuration of nursing care system 1000] FIG. 1 is a schematic diagram showing the general configuration of a care system 1000 according to one embodiment. The care system 1000 is used, for example, to monitor a person requiring care in a care facility. Hereinafter, a person requiring care residing in the care facility will be referred to as a resident 71. Here, the resident 71 corresponds to a specific example of a subject of the present invention.

[0023] In the nursing care facility, staff care for the resident 71. The nursing care system 1000 includes, for example, an information processing device 10, a detection unit 20, and a terminal device 30. The information processing device 10, the detection unit 20, and the terminal device 30 are communicatively connected to each other via a network 50.

[0024] The information processing device 10 is a server or a PC. PC is an abbreviation for personal computer. The information processing device 10 may be an on-premise server installed in a nursing care facility, or may be a cloud server using a commercial cloud service. The information processing device 10 may be installed in each nursing care facility, or may be shared by multiple nursing care facilities.

[0025] The detection unit 20 is installed, for example, in the room of the resident 71. The room of the resident 71 contains, for example, a bed 81 and a wheelchair used by the resident 71. The detection unit 20 sets the interior of the room as an observation area and monitors the movements of the resident 71 within the observation area. The detection unit 20 constantly senses the resident 71 in the observation area in real time using a plurality of various sensors. As a result, sensing data is generated in the detection unit 20. The nursing care system 1000 uses the sensing data generated by the detection unit 20 to monitor the condition of the resident 71. The condition of the resident 71 includes, for example, whether the resident 71 has woken up, fallen, fallen over, or is asleep. The monitoring results are recorded in the information processing device 10. Here, the bed 81 corresponds to one specific example of bedding of the present invention.

[0026] A nursing care facility has, for example, a plurality of rooms. A detection unit 20 is installed in, for example, each of the plurality of rooms. For example, a device ID is assigned to each of the plurality of detection units 20. For example, a subject ID is assigned to each of the plurality of residents 71. The device ID and the subject ID are linked to each other and stored in the information processing device 10 or the terminal device 30.

[0027] The terminal device 30 is, for example, a PC. The terminal device 30 is operated by users such as staff and managers of the nursing care facility. The terminal device 30 may be a mobile terminal such as a smartphone used by the staff. The staff performs their work using the terminal device 30, for example, during working hours. When the detection unit 20 detects a specific state such as getting up, falling, or tripping, the specific state is notified to the terminal device 30 as an event. In response to this notification, the staff takes appropriate measures for the resident 71.

[0028] 2 is a block diagram showing an example of the configuration of the detection unit 20. The detection unit 20 includes, for example, a control unit 21, a communication unit 22, an imaging unit 23, a Doppler sensor 24, and a care call unit 25. The imaging unit 23 and the Doppler sensor 24 are provided, for example, on the ceiling of a room. The care call unit 25 is provided, for example, near a bed 81. The imaging unit 23 and the Doppler sensor 24 may be provided on a wall of the room at a position close to the ceiling. In other words, the Doppler sensor 24 is installed at a position away from the bed 81. Here, the Doppler sensor 24 corresponds to a specific example of a respiration detection unit of the present invention.

[0029] The control unit 21 is composed of, for example, a CPU, RAM, and ROM. The control unit 21 controls each unit of the detection unit 20 and performs calculation processing according to a program. The communication unit 22 is an interface circuit for communicating with other devices such as the terminal device 30 via the network 50. The interface circuit is, for example, a LAN card. The control unit 21 may include a large-capacity memory.

[0030] The imaging unit 23 captures the interior of the resident's room as a capture area and outputs two-dimensional image data. The image data is, for example, a moving image. The image data may also be a still image. The imaging unit 23, which is installed on the ceiling of the room, captures, for example, a bird's-eye view of the room. The imaging unit 23 includes, for example, at least one of a near-infrared camera and a visible light camera. The image data output by the imaging unit 23 includes image information. The image information is information relating to an image of an area including the bed 81.

[0031] The Doppler sensor 24 detects the Doppler shift of the electromagnetic waves by transmitting and receiving the electromagnetic waves, and generates Doppler data. The Doppler sensor 24 includes, for example, an FMCW millimeter wave radar. FMCW is an abbreviation for Frequency Modulation Continuous Wave. The Doppler sensor 24 transmits, for example, electromagnetic waves toward the bed 81 and receives the electromagnetic waves reflected near the bed 81. The electromagnetic waves are, for example, millimeter waves or microwaves.

[0032] For example, the Doppler sensor 24 calculates the change between the frequency of the transmitted wave and the frequency of the received wave. This makes it possible to generate information related to the body movement of the resident 71. The body movement of the resident 71 is, for example, the body movement of the chest associated with the breathing and heartbeat of the resident 71. In other words, the Doppler data generated by the Doppler sensor 24 includes respiratory information related to the breathing of the resident 71.

[0033] For example, the Doppler sensor 24 calculates the time difference between the transmitted wave and the received wave. This makes it possible to generate information about the distance between the Doppler sensor 24 and an object that reflects the electromagnetic waves. The object that reflects the electromagnetic waves is, for example, a resident 71. In other words, the Doppler data generated by the Doppler sensor 24 includes distance information about the distance between the Doppler sensor 24 and the resident 71.

[0034] The care call unit 25 includes a push button switch, and detects a care call when the switch is operated by the resident 71.

[0035] The detection unit 20 may further include other sensors. The detection unit 20 may include, for example, at least one of a bed sensor, a mat sensor, a thermal sensor, and an infrared sensor. The bed sensor is a sensor that can be attached to the bed 81. For example, the bed sensor detects the load on the bed 81. The bed sensor may detect whether or not a person is present on the bed 81. The bed sensor may detect the sleep state and abnormal micro-movements of the resident 71. The abnormal micro-movements may be caused by, for example, cardiac arrest. The mat sensor detects the presence of a person in each area of ​​the floor of the room. The infrared sensor is also called a human presence sensor. The infrared sensor detects, for example, whether or not a person is present in the room. For example, the infrared sensor is placed over the entire room or on the bed 81 as the observation area.

[0036] FIG. 3 is a block diagram showing the configuration of the terminal device 30. The terminal device 30 includes, for example, a control unit 31, a memory unit 32, a communication unit 33, a display unit 34, and an operation input unit 35. The control unit 31 is composed of a CPU, RAM, ROM, etc. The control unit 31 controls each unit of the terminal device 30 and performs calculations according to a program. The memory unit 32 is composed of a hard disk, etc., which stores various programs and data. The communication unit 33 is an interface circuit for communicating with other devices via a network 50, either wired or wirelessly. The display unit 34 is composed of an LCD display, a touch sensor, etc., and displays various information and operation screens. The operation input unit 35 is an input device such as a keyboard, a mouse, or a touch sensor. The operation input unit 35 accepts inputs from, for example, staff, unit staff leaders, and facility managers. A unit is also referred to as a responsible area or responsible group. Hereinafter, the staff, unit staff leaders, and facility managers are collectively referred to as managers, etc.

[0037] 4 is a block diagram showing the configuration of the information processing device 10. The information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is composed of a CPU, RAM, ROM, etc. The control unit 11 controls each unit of the information processing device 10 and performs arithmetic processing according to a program. Specific functions of the control unit 11 will be described later.

[0038] The storage unit 12 is configured with a hard disk etc. that stores various programs and various data. The storage unit 12 stores trained models, bed-related data, resident information, staff information, event lists etc.

[0039] The trained model stored in the memory unit 12 is used, for example, in the process of detecting people and objects from image data. The objects may be moving objects. In the process of detecting people and objects, an area in the image data where objects, including people, exist is detected as an object presence area. Then, a confidence score is calculated for each predetermined category of objects included in the detected object presence area. The confidence score is the likelihood of the target object. The confidence score can be calculated using a known technique using a DNN. DNN is an abbreviation for Deep Neural Network. Examples of predetermined categories include people, chairs, furniture, and beds. In the process of detecting people and objects, the object presence area with the highest confidence score in the person category is detected as a person rectangle. The person rectangle is also called a bounding box. In the process of detecting people and objects, a resident 71 may be detected as an object. Similarly, in a predetermined object category, the object presence area with the highest confidence score is detected as the object rectangle, etc., of the category with the highest confidence score.

[0040] This trained model may be adjusted to improve the accuracy of detecting the bed 81. For example, a trained model that has been trained in advance on a large dataset is applied to a task specialized in detecting the bed 81. This adjustment is performed, for example, by retraining the trained model. Adjustment is also called transfer learning or fine tuning.

[0041] The bed-related data includes, for example, information about the height of the bed 81, information about the size of the bed 81, and information about the outer shape of the bed 81. The information about the height of the bed 81 is, for example, information such as 50 cm to 70 cm, and is information about a numerical range representing the height of the bed 81. The information about the size of the bed 81 is, for example, 180 cm in length and 90 cm in width, and is a numerical value representing each of the long and short sides of the bed 81. The information about the height of the bed 81 and the information about the size of the bed 81 are, for example, default values, and are applied commonly to each room in the facility. The information about the height of the bed 81 and the information about the size of the bed 81 may be set for each room by operating the screen via the terminal device 30. The information about the outer shape of the bed 81 is generated and registered by processing by the information processing device 10, which will be described later.

[0042] The resident information is information relating to the resident 71. The resident information includes, for example, the room number in which the resident 71 resides, family information, the age, sex, medical history, and medical conditions of the resident 71.

[0043] The staff information is information about staff working at the nursing care facility, and includes, for example, the staff member's name, the unit to which they belong, and their work schedule.

[0044] The event list is a list of events detected by the nursing care system 1000. Events include, for example, getting up, getting out of bed, falling, tripping, and sleeping. Getting up is an event in which the resident 71 gets up from the bed 81. Getting out of bed is an event in which the resident 71 leaves the bed 81. Falling is an event in which the resident 71 falls from the bed 81. Tripping is an event in which the resident 71 falls onto the floor or the like. Sleeping is an event in which the resident 71 sleeps on the bed 81. The event list includes, for example, information such as the type of each event, the time of occurrence, the event cause, and the staff's response to the event. The event list may include information on the room number instead of or in addition to the event cause information.

[0045] The communication unit 13 is an interface circuit for communicating with other devices via the network 50 .

[0046] The following describes each function of the control unit 11. The control unit 11 functions as an acquisition unit 111, an estimation unit 112, a determination unit 113, an identification unit 114, an event determination unit 115, and an event notification unit 116, for example, by reading a program and executing processing.

[0047] The acquisition unit 111 acquires image information. The image information is information relating to an image of an area including the bed 81. The image information is included in, for example, image data output by the imaging unit 23. The acquisition unit 111 acquires the image data from, for example, the detection unit 20.

[0048] 5 shows an example of an image 23Im of the room of the resident 71 captured by the imaging unit 23. The image 23Im shows, for example, a bed 81 and a chair installed in the room of the resident 71.

[0049] The acquisition unit 111 further acquires breathing information and distance information. The breathing information is information related to the breathing of the resident 71. The distance information is information related to the distance between the Doppler sensor 24 and the resident 71. The breathing information and distance information are included in, for example, Doppler data output by the Doppler sensor 24. The Doppler data includes information related to frequency.

[0050] The estimation unit 112 estimates a candidate area for the bed 81 in the image 23Im based on the image information acquired by the acquisition unit 111. The estimation unit 112 executes a process for detecting people and objects in the image 23Im, for example, by using a trained model stored in the storage unit 12. As a result, a candidate area for the bed 81 in the image 23Im is estimated.

[0051] FIG. 6 is a diagram showing an example of the results of the person and object detection process in the image 23Im. Through the person and object detection process, for example, an object rectangle 231 of the bed 81 and object rectangles 232, 233 of the chairs are created in the image 23Im. A reliability score may be assigned to each of the object rectangle 231 of the bed 81 and the object rectangles 232, 233 of the chairs. The object rectangle 231 of the bed 81 may be assigned information indicating the extent of the rectangle, for example, the XY coordinates of the four corners. For example, the object rectangle 231 of the bed 81 is a candidate region for the bed 81 in the image 23Im. The estimation unit 112 may estimate multiple candidate regions in the image 23Im.

[0052] The estimation unit 112 further estimates a candidate position of the resident 71 in the image 23Im based on the distance information acquired by the acquisition unit 111.

[0053] 7 shows an example of estimated candidate positions 23pt of the resident 71. The candidate positions 23pt of the resident 71 are, for example, a plurality of positions that are at a predetermined distance from the Doppler sensor 24. For example, the candidate positions 23pt of the resident 71 are estimated in a circular shape with the Doppler sensor 24 at the center.

[0054] The determination unit 113 determines whether or not the resident 71 is present on the bed 81 based on the respiratory information acquired by the acquisition unit 111. For example, when the resident 71 is asleep on the bed 81, the determination unit 113 determines that the resident 71 is present on the bed 81. The determination unit 113 determines whether or not the resident 71 is present on the bed 81, for example, in the following manner. First, the determination unit 113 decomposes the Doppler data into frequency components.

[0055] FIG. 8 shows an example of Doppler data decomposed into frequency components. The horizontal axis of FIG. 8 represents frequency, and the vertical axis of FIG. 8 represents signal strength. The unit of frequency is Hz. For example, the Doppler data can be decomposed into frequency components by performing FFT on the signal detected by the Doppler sensor 24. FFT is an abbreviation for Fast Fourier Transform.

[0056] Next, the determination unit 113 extracts frequency components corresponding to human breathing from the Doppler data decomposed into frequency components. The frequency corresponding to human breathing is, for example, approximately 0.2 Hz to 0.5 Hz. The determination unit 113 extracts, for example, data in the frequency range 24R shown in FIG. 8.

[0057] Thereafter, the determination unit 113 converts the data in the frequency range 24R into a respiratory rate per unit time. For example, the determination unit 113 determines that a signal having a substantially constant period among signals included in the data in the frequency range 24R is a signal derived from the breathing of the resident 71.

[0058] FIG. 9 shows an example of data converted into the respiratory rate per unit time. This data indicates that the respiratory rate per minute of the resident 71 is approximately 20 breaths. For example, when the respiratory rate per unit time is equal to or greater than a predetermined value, the determination unit 113 determines that the resident 71 is present on the bed 81. At this time, when the respiratory rate per unit time is smaller than the predetermined value, the determination unit 113 determines that the resident 71 is not present on the bed 81. The determination unit 113 may determine that the resident 71 is present on the bed 81 when the respiratory rate per unit time is within a predetermined range. At this time, when the respiratory rate per unit time is outside the predetermined range, the determination unit 113 determines that the resident 71 is not present on the bed 81.

[0059] When the determination unit 113 determines that the resident 71 is present on the bed 81, the identification unit 114 identifies the outline of the bed 81 in the image 23Im. At this time, the identification unit 114 identifies the outline of the bed 81 in the image 23Im, for example, based on the object rectangle 231 of the bed 81 estimated by the estimation unit 112 and the candidate position 23pt of the resident 71. For example, when the object rectangle 231 of the bed 81 overlaps with the candidate position 23pt of the resident 71, the identification unit 114 identifies the outline of the object rectangle 231 of the bed 81 as the outline of the bed 81. The identified outline of the bed 81 is, for example, the XY coordinates of the four corners of the bed 81. The identification unit 114 may identify, as the outline of the bed 81, the outline of a candidate area that overlaps with the candidate position 23pt of the resident 71, from among the multiple candidate areas estimated by the estimation unit 112.

[0060] The event determination unit 115 determines an event from the image data output by the imaging unit 23. The event determination unit 115 detects the silhouette of the entire body of the resident 71, for example, from multiple frames of image data. The silhouette of the entire body of the resident 71 will be referred to as a human silhouette hereinafter. A head silhouette may also be used in addition to or instead of the human silhouette. The human silhouette can be detected, for example, by extracting a range of pixels with a relatively large difference between multiple pieces of image data captured at different times. The human silhouette may also be detected using a background subtraction method.

[0061] The event determination unit 115 determines the type of event, for example, using the outline of the bed 81 in the image 23Im identified by the identification unit 114. The event determination unit 115 determines events such as getting up and getting out of bed, for example, based on the relative positional relationship between the human silhouette and the outline of the bed 81. The events determined by the event determination unit 115 are recorded in an event list.

[0062] When the type of the event that has occurred is of a specific type, the event notification unit 116 notifies the terminal device 30 of the staff member on duty. Examples of specific types of events include a fall and getting out of bed. For example, some of the multiple staff members who have received the notification from the terminal device 30 will respond to the event. The staff member who responded to the event inputs to the terminal device 30 that they have responded to the event or will respond to the event. Upon receiving this input from the staff member via the terminal device 30, the control unit 11 changes the status of the event from "not responded to" to "responded to" and records it in the event list.

[0063] [Information processing method by care system 1000] Fig. 10 is a flowchart showing an example of an information processing method by the information processing device 10. Specifically, Fig. 10 shows a method for identifying the outer shape of the bed 81.

[0064] (Step S01) The information processing device 10 first determines whether it is time to update the outline of the bed 81 in the image. For example, when a command is input by the user via the terminal device 30, the information processing device 10 determines that it is time to update the outline of the bed 81 in the image. The information processing device 10 may periodically determine that it is time to update the outline of the bed 81 in the image. For example, the information processing device 10 may determine that it is time to update the outline of the bed 81 in the image at the same time every day. When the information processing device 10 determines that it is time to update the outline of the bed 81 in the image, that is, if the answer is YES, the information processing device 10 proceeds to the processing of step S02. When the information processing device 10 determines that it is not time to update the outline of the bed 81 in the image, that is, if the answer is NO, the information processing device 10 repeats the processing of step S01. The outline of the bed 81 is updated, for example, for each room.

[0065] (Step S02) The information processing device 10 acquires image data captured by the imaging unit 23 and Doppler data detected by the Doppler sensor 24. The image data includes image information. The Doppler data includes respiration information and distance information. The information processing device 10 acquires the image data and Doppler data from the detection unit 20, for example.

[0066] (Step S03) The information processing device 10 estimates a candidate region of the bed 81 in the image. The information processing device 10 estimates the candidate region of the bed 81 by detecting an object rectangle of the bed 81 in the image using, for example, a machine learning model.

[0067] (Step S04) The information processing device 10 determines whether the resident 71 is in bed 81 based on the respiratory information acquired in step S01. If it is determined that the resident 71 is in bed 81, that is, if the result is YES, the information processing device 10 proceeds to the processing of step S05. If it is determined that the resident 71 is not in bed 81, that is, if the result is NO, the information processing device 10 returns to the processing of step S02.

[0068] (Step S05) The information processing device 10 estimates a candidate position of the resident 71 in the image based on the distance information acquired in step S01. The information processing device 10 may execute the process of step S05 before the process of step S04. The information processing device 10 may execute the process of step S05 before the process of step S03.

[0069] (Step S06) Information processing device 10 identifies the outline of bed 81 in the image based on the candidate area of ​​bed 81 in the image estimated in step S03 and the candidate position of resident 71 in the image estimated in step S05.

[0070] (Step S07) The information processing device 10 registers the outer shape of the bed 81 identified in step S06 in the storage unit 12, and ends the process. The information processing device 10 determines various events, for example, using information related to the outer shape of the registered bed 81.

[0071] [Effects of the information processing device 10 and the care system 1000] In the information processing device 10 and the nursing care system 1000 according to this embodiment, a candidate region of the bed 81 in the image 23Im is estimated based on image information. Furthermore, whether or not the resident 71 is present on the bed 81 is determined based on the respiratory information. Then, when it is determined that the resident 71 is present on the bed 81, the outline of the bed 81 in the image 23Im is identified based on the candidate region of the bed 81. Therefore, it is possible to identify the outline of the bed 81 in the image 23Im. The effect of this will be described below.

[0072] Nursing care facilities have many rooms, each equipped with a camera for monitoring the residents. Nursing care facilities may have, for example, tens to hundreds of cameras installed. To monitor the behavior of residents using these cameras, it is necessary to set the positions of bedding, such as beds, in the images captured by the cameras. For example, staff at the nursing care facility set the positions of the bedding in the images.

[0073] Setting the position of bedding for all of the dozens to hundreds of cameras is not an easy task. Furthermore, the position of bedding in a room may change at irregular intervals, such as when a resident changes. Every time the position of the bedding changes, it is necessary to reset the position of the bedding in the images.

[0074] In contrast, in the information processing device 10 and the nursing care system 1000, the outer shape of the bed 81 in the image 23Im is identified using the information detected by the detection unit 20. Therefore, work by staff of the nursing care facility or the like is not required. Therefore, it is possible to more efficiently identify the outer shape of the bed 81 in the image 23Im.

[0075] Modifications of the above embodiment will be described below, but the same components as those in the above embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0076] <Variation 1> 11 shows an example of the configuration of the imaging unit 23 of the nursing care system 1000 according to Modification 1. This imaging unit 23 is, for example, a stereo camera having a first camera 23A and a second camera 23B. In this respect, the nursing care system 1000 according to Modification 1 differs from the nursing care system 1000 of the above embodiment. Except for this point, the nursing care system 1000 according to Modification 1 has the same configuration as the nursing care system 1000 of the above embodiment and provides the same effects.

[0077] The first camera 23A and the second camera 23B are installed in the room of the resident 71 so that their optical axes are parallel to each other. The distance between the two optical axes is, for example, several tens of centimeters, e.g., 20 cm. A distance image is generated from two sets of image data captured simultaneously by the first camera 23A and the second camera 23B. In the distance image, the distance value from the subject to the camera calculated based on the parallax information is stored in each pixel. The distance image includes, for example, height information. The distance image is converted to height information using the following method. The distance value is determined by the distance from the camera, while the height information is the distance from a reference surface. The reference surface is, for example, the floor of the room. Therefore, the height information is converted by subtracting the distance value of each pixel from the distance to the floor set for each pixel in the distance image.

[0078] The focal length, pixel size, optical axis direction, distortion, and optical axis distance of the optical system of the imaging unit 23 are registered in advance and stored in the storage unit 12 or the storage area of ​​the control unit 21. Furthermore, by performing calibration, the real space in the room is associated with the camera coordinate system. For example, calibration is performed by the user inputting the distance and direction to multiple markers captured by the imaging unit 23. Furthermore, the distance from the floor to each pixel is registered by the calibration.

[0079] The information processing device 10 may use the height information detected by the imaging unit 23 to estimate a candidate region for the bed 81 in the image.

[0080] 12 is a subroutine flowchart showing the process of estimating the candidate area for the bed 81. This subroutine flowchart corresponds to the subroutine flowchart of step S03 in FIG.

[0081] (Step S301) The information processing device 10 may acquire height information of an area corresponding to the object rectangle of the bed 81, for example.

[0082] Fig. 13 shows an example of height information acquired by the information processing device 10. In Fig. 13, height is converted into density values ​​within a range of 0 to 256 cm from the floor surface.

[0083] (Step S302) The information processing device 10 extracts pixels having a height within a predetermined range from the image. For example, the information processing device 10 extracts pixels having a height between 50 cm and 70 cm.

[0084] (Step S303) The information processing device 10 estimates a candidate region for the bed 81 based on the pixels extracted in step S302.

[0085] 14 and 15 show an example of the processing of step S303. For example, the information processing device 10 extracts the outer edges of the pixels extracted in step S302 and detects multiple corners 231c from the edges. Then, the information processing device 10 specifies the four corners 231c that are farthest from the center in four directions as the coordinates of the four corners that define the candidate area for the bed 81. For example, this obtains the coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the four corners that indicate the candidate area.

[0086] The nursing care system 1000 according to the first modification uses height information to estimate a candidate area for the bed 81 in the image. Like the nursing care system 1000 of the above embodiment, this nursing care system 1000 can also identify the outline of the bed 81 in the image 23Im. The nursing care system 1000 may use the height information as well as the object rectangle of the bed 81 to estimate a candidate area for the bed 81 in the image.

[0087] <Variation 2> 16 and 17 show an example of processing by the nursing system 1000 according to Modification 2. This nursing system 1000 classifies height information and estimates a candidate area for the bed 81 using the classified height information. In this respect, the nursing system 1000 according to Modification 2 differs from the nursing system 1000 according to Modification 1. Fig. 16 corresponds to the subroutine flowchart shown in Fig. 12.

[0088] (Step S321) The information processing device 10 may acquire height information of an area corresponding to the object rectangle of the bed 81, for example.

[0089] (Step S322) The information processing device 10 divides the acquired height information into sections 1 to n. Each section is divided by the same interval. For example, the information processing device 10 divides the range of 50 cm to 70 cm of the area corresponding to the object rectangle of the bed 81 into three sections: section 1, section 2, and section 3.

[0090] (Step S323) Next, the information processing device 10 calculates the areas b1 to b3 of the sections 1 to 3. The area b may be calculated using the number of pixels that belong to the section itself, or may be calculated by converting the xy area of ​​the shooting region according to the angle of view and the number of pixels into the area.

[0091] (Step S324) The information processing device 10 extracts the section with the highest occupancy rate from sections 1 to n. The occupancy rate is, for example, the ratio of area b to area a of the object rectangle of the bed 81. That is, occupancy rate = b / a.

[0092] 17 is a schematic diagram showing an example of the area a of the object rectangle of bed 81 and the areas b1 to b3 of each section. For example, section 1 has the largest area b1 and the highest occupancy rate. The information processing device 10 extracts section 1, which has the largest occupancy rate.

[0093] (Step S325) The information processing device 10 determines whether the area b of the extracted section is equal to or greater than a predetermined value v. The predetermined value v is stored in, for example, the storage unit 12. For example, the predetermined value v is set based on the area of ​​16,200 cm^2 of a typical bed with a major axis length of 180 cm and a minor axis length of 90 cm. For example, the predetermined value v is set to 50% of the typical area of ​​a bed. When the area b1 is equal to or greater than the predetermined value v, the information processing device 10 proceeds to the processing of step S326. When the area b1 is less than the predetermined value v, the information processing device 10 ends the processing. At this time, the information processing device 10 may perform an error determination.

[0094] (Step S326) The information processing device 10 estimates the candidate region of the bed 81 from the pixels of the section extracted in step S325. This process is similar to the process of step S303 above, for example.

[0095] The nursing care system 1000 according to the second modification estimates the area with the largest occupancy rate among the areas corresponding to the object rectangle of the bed 81 as the candidate area of ​​the bed 81. Like the nursing care system 1000 according to the above embodiment, this nursing care system 1000 can also identify the outline of the bed 81 in the image 23Im.

[0096] <Variation 3> 18 and 19 show an example of processing by the nursing system 1000 according to Modification 3. This nursing system 1000 executes processing of steps S343 and S344 instead of the processing of steps S323 and S324 in the above-described Modification 2. In this respect, the nursing system 1000 according to Modification 3 differs from the nursing system 1000 according to the above-described Modification 3. FIG. 18 corresponds to the subroutine flowchart shown in FIG. 12.

[0097] (Steps S341, S342) The information processing device 100 executes the processes of steps S341 and S342 in the same manner as the processes of steps S321 and S322 shown in FIG.

[0098] (Step S343) The information processing device 10 clusters the pixels in each of sections 1 to 3 and calculates the aspect ratios AR1 to AR3 of the resulting clusters. The aspect ratio AR is calculated using the length in the X-axis direction and the length in the Y-axis direction. The aspect ratio is calculated by dividing the minor axis length by the major axis length, with the longer side being the major axis and the shorter side being the minor axis. The directions of the X-axis and Y-axis are set in advance and are along the side walls of the room. Note that there may be cases where the bed 81 is placed at an angle rather than along the wall of the room. Therefore, the information processing device 10 may rotate the cluster by a predetermined angle, calculate the aspect ratio AR at each angle, and use the largest aspect ratio AR.

[0099] (Step S344) The information processing device 10 extracts the category in which the aspect ratio AR calculated in step S343 is closest to a predetermined value w. The predetermined value w is set in advance and stored in the storage unit 12. For example, the predetermined value w is set to 0.5, which is the ratio of the length and width of a typical bed, 90 cm / 180 cm.

[0100] (Steps S345, S346) The information processing device 10 executes the processes of steps S345 and S346 in the same manner as the processes of steps S325 and S326 shown in FIG.

[0101] The nursing care system 1000 according to the third modification estimates, as a candidate area for the bed 81, an area having an aspect ratio AR that is closest to a predetermined value w among areas corresponding to the object rectangle of the bed 81. This nursing care system 1000, like the nursing care system 1000 according to the above embodiment, is also capable of identifying the outline of the bed 81 in the image 23Im.

[0102] <Variation 4> 20 shows an example of processing by the nursing care system 1000 according to Modification 4. The nursing care system 1000 may accept a change in the default value of the bed 81.

[0103] (Step S10) The information processing device 10 accepts various setting changes, for example, regarding the planar size, height, etc. of the bed 81. The planar size, height, etc. of the bed 81 are changed irregularly for each room. The settings of the various beds 81 are changed, for example, when the resident 71 first moves into the room or when the beds 81 in the room are replaced.

[0104] The information processing device 10 accepts setting inputs from a user such as a staff member via, for example, an operation screen displayed on the terminal device 30. For example, at least one of the following settings is accepted. The accepted settings are stored in the storage unit 12 as setting values ​​for each room.

[0105] (Steps S11 to S17) The information processing device 10 reads out various settings set for each room from the storage unit 12 and performs the processes of steps S11 to S17. The processes of steps S11 to S17 are the same as steps S01 to S07 in the embodiment shown in FIG.

[0106] In Modification 4, the changed setting value may be applied according to the bed in each room. Note that the process of step S10 may be performed by the information processing device 10 if an error occurs in the process of step S325.

[0107] The above-described configurations of the nursing care system 1000 and the information processing device 10 are the main configurations described in explaining the features of the above-described embodiment and modified examples, but are not limited to the above configurations and can be variously modified within the scope of the claims. Furthermore, configurations provided in general information processing devices are not excluded.

[0108] For example, the embodiments and modifications may be combined and applied. Also, some of the functions of the control unit 11 may be performed by the detection unit 20.

[0109] Furthermore, in the above embodiment and modified examples, the bed 81 has been described as an example of bedding, but the information processing device 10 may also identify the outer shape of bedding such as a futon and a mattress.

[0110] Furthermore, the information processing device 10 may specify the outline of one area that overlaps with the candidate position of the estimated resident 71 and is selected from the plurality of divided areas as the outline of the bed 81. The information processing device 10 may divide the image into a plurality of areas based on, for example, height information. In this case, the one area selected from the plurality of divided areas may be the largest area among the areas that overlap with the candidate position of the estimated resident 71. The one area selected from the plurality of divided areas may be an area that has a predetermined shape among the areas that overlap with the candidate position of the estimated resident. The predetermined shape is, for example, a rectangle with a predetermined ratio between the long side and the short side.

[0111] Furthermore, in the above embodiment and variant examples, an example has been described in which the information processing device 10 identifies the outer shape of the bed 81 using image information, breathing information, and distance information, but the information processing device 10 may also identify the outer shape of the bed 81 using image information and breathing information.

[0112] Furthermore, in the above embodiment and variant examples, an example has been described in which the information processing device 10 mainly estimates the candidate area of ​​the bed 81 using the object rectangle of the bed 81 detected from the image, but the information processing device 10 may also estimate the candidate area of ​​the bed 81 using other methods.

[0113] The nursing care system 1000 may be configured with only the information processing device 10.

[0114] Furthermore, the means and methods for performing various processes in the nursing care system or information processing device according to the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD-ROM, or online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a device such as a detection unit.

[0115] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims. [Explanation of symbols]

[0116] 1000 Care System 10. Information processing equipment 11 Control section 111 Acquisition Department 112 Estimation Department 113 Judgment Department 114 Specific section 115 Event Judgment Unit 116 Event Notification Section 20 Detector 21 Control section 22 Communications Department 23 Imaging unit 24 Doppler sensor 25 Care Call Department 30 Terminal Equipment 31 Control Unit 32 Storage section 33 Communications Department 34 Display section 35 Operation input section

Claims

1. an acquisition unit that acquires image information relating to an image of an area including bedding that may be used by the subject and respiratory information relating to the subject's breathing; an estimation unit that estimates a candidate area of ​​the bedding in the image based on the acquired image information; a determination unit that determines whether the subject is on the bedding based on the acquired respiratory information; an identification unit that identifies an outline of the bedding in the image based on the estimated candidate area of ​​the bedding when it is determined that the subject is on the bedding; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the breathing of the subject is detected by a breathing detection unit installed at a position away from the bedding.

3. The information processing device according to claim 2 , wherein the acquisition unit further acquires distance information relating to a distance between the breathing detection unit and the subject.

4. The information processing device according to claim 3 , wherein the estimation unit further estimates a candidate position of the subject person in the image based on the distance information.

5. The information processing device according to claim 4, wherein the identification unit, when it is determined that the subject is on the bedding, identifies the outline of the bedding based on the estimated candidate area of ​​the bedding and the candidate position of the subject.

6. The information processing device according to claim 3 , wherein the respiration detection unit includes a Doppler sensor.

7. The information processing device according to claim 6 , wherein the respiration information and the distance information include information relating to a frequency detected by the Doppler sensor.

8. The information processing device according to claim 5 , wherein the identification unit identifies, as the outer shape of the bedding, an outer shape of one area that overlaps with the estimated candidate position of the subject and is selected from a plurality of divided areas.

9. The information processing apparatus according to claim 8 , wherein the one area selected from the plurality of divided areas is the largest area among areas that overlap with the estimated candidate position of the subject person.

10. The information processing device according to claim 8 , wherein one area selected from the plurality of divided areas is an area having a predetermined shape among areas that overlap with the estimated candidate position of the subject person.

11. The information processing device according to claim 10 , wherein the predetermined shape is a rectangle having a predetermined ratio between long and short sides.

12. acquiring image information relating to an image of an area including bedding that may be used by the subject and respiratory information relating to the subject's breathing; estimating a candidate region of the bedding in the image based on the acquired image information; determining whether the subject is present on the bedding based on the acquired respiratory information; When it is determined that the subject is on the bedding, specifying the outline of the bedding in the image based on the estimated candidate area of ​​the bedding; An information processing program for causing a computer to execute processing including the above.

Images