Observation system, observation system control method, and program

Abstract

To provide an observation system and the like which can secure a wider observation range while avoiding or suppressing the occurrence of a stress of an observation object person.SOLUTION: An observation system comprises: an imaging unit which images an observation object space; a drive unit which drives the imaging unit; and a control unit which changes an imaging range of the imaging unit to any of a preset number (e.g. three) of imaging ranges F1, F2, F3 by controlling driving of the imaging unit by the drive unit. The control unit observes an observation object person on the basis of a photographed image captured by the imaging unit in such a state that the imaging range of the imaging unit is changed to a partial imaging range of the prescribed number of imaging ranges F1, F2, F3 and driving of the imaging unit by the drive unit is stopped.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an observation system for observing a person to be observed and to techniques related thereto. [Background technology]

[0002] There are observation systems that observe a person to be observed (see Patent Document 1, etc.). For example, Patent Document 1 describes a monitoring system (observation system) that watches over residents (persons to be observed) in a care facility or the like using images captured by a camera, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-010581 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned observation system, it is preferable to monitor the entire room of the person being observed (with as few blind spots as possible). In order to build such a technology (technology that ensures a wider observation range), one idea is to drive the observation (monitoring) camera to change the camera's orientation and change the camera's shooting range (camera orientation, etc.).

[0005] However, if the position of a person (person being observed) is detected and the camera direction (direction of the camera's line of sight) is constantly tracked to point at the person (position), the camera's driving noise will be generated frequently (continuously) in response to the person's movements, which is likely to cause stress (psychological burden) to the person.

[0006] Therefore, an object of the present invention is to provide an observation system and related technology that can ensure a wider observation range while avoiding or suppressing the occurrence of stress in the person being observed. [Means for solving the problem]

[0007] In order to solve the above problem, an observation system according to the present invention includes: an imaging unit that captures an image of an observation target space; Rotate the imaging unit to change the imaging range. a driving unit that drives the an illumination unit that emits illumination light for illuminating a photographing range of the imaging unit; The driving unit rotate The driving is controlled to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance. and controls the emission of illumination light by the illumination unit. a control unit, wherein the control unit changes the imaging range of the imaging unit to one of the predetermined number of imaging ranges and controls the driving unit to rotate With the drive stopped, The illumination unit emits illumination light having an amount of light emission corresponding to the one of the plurality of photographing ranges, among the amount of light emission predetermined for each of the plurality of photographing ranges, from the illumination unit; The person to be observed is selected based on the captured image taken by the imaging unit. Perform the observation process It is characterized by:

[0008] The control unit detects the position of the person to be observed in the captured image, and when the position of the person to be observed is within a predetermined area in the captured image, the control unit does not drive the drive unit, and in a state where the imaging range of the imaging unit is maintained at the one imaging range, the control unit detects the position of the person to be observed based on the captured image captured by the imaging unit. Perform the observation process You may do so.

[0009] The predetermined number of shooting ranges may include a first shooting range and a second shooting range, and the second shooting range may be a shooting range that can capture the person being observed that is farther away from the imaging unit than the first shooting range.

[0010] The predetermined number of shooting ranges may include a first shooting range and a second shooting range, the imaging unit may be installed near and above the bed, the first shooting range may be a shooting range that can capture the person being observed who is near the bed, and the second shooting range may be a shooting range that can capture the person being observed who is near an entrance away from the bed.

[0011] before The amount of light emission determined in correspondence with the second imaging range may be greater than the amount of light emission determined in correspondence with the first imaging range.

[0012] The program according to the present invention includes an imaging unit that captures an image of an observation target space and a Rotate the imaging unit to change the imaging range. A driving unit and an illumination unit that emits illumination light for illuminating the imaging range of the imaging unit; a) driving the imaging unit; rotate a) controlling the driving of the imaging unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance; and b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and controlling the driving of the imaging unit. rotate With the drive stopped, The illumination unit emits illumination light having an amount of light emission corresponding to the one of the plurality of photographing ranges, among the amount of light emission predetermined for each of the plurality of photographing ranges, from the illumination unit; The person to be observed is selected based on the captured image taken by the imaging unit. Perform the observation process The present invention is characterized in that it is a program for executing the steps.

[0013] The control method according to the present invention includes an imaging unit that captures an image of an observation target space, and the imaging unit Rotate the imaging unit to change the imaging range. A driving unit and an illumination unit that emits illumination light for illuminating the imaging range of the imaging unit; a) controlling the imaging unit by the driving unit; rotate a) controlling the driving of the imaging unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance; and b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and controlling the driving of the imaging unit. rotate With the drive stopped, The illumination unit emits illumination light having an amount of light emission corresponding to the one of the plurality of photographing ranges, among the amount of light emission predetermined for each of the plurality of photographing ranges, from the illumination unit; The person to be observed is selected based on the captured image taken by the imaging unit. Perform the observation process The method is characterized by comprising the steps of: The observation system according to the present invention comprises: The imaging device comprises an imaging unit that captures an image of a space to be observed, a drive unit that rotates the imaging unit to change the imaging range of the imaging unit, and a control unit that controls the rotation of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance, wherein the control unit sets the imaging range of the imaging unit to one of the predetermined number of imaging ranges, and then detects the position of the person to be observed in the captured image that corresponds to the one imaging range, and if the position of the person to be observed is within a predetermined area in the captured image, performs an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit. When the position of the person being observed no longer falls within the specified area, the control unit may rotate the imaging unit to change the shooting range of the imaging unit to a shooting range other than the one shooting range. The observation process of the observed person may include a process of detecting a behavior-related event of the observed person. The program of the present invention is characterized in that it is a program for causing a computer that controls an observation system having an imaging unit that captures an image of a space to be observed and a drive unit that rotates the imaging unit to change the imaging range of the imaging unit to execute the following steps: a) controlling the rotational drive of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of the person to be observed in the captured image that corresponds to the one imaging range; and c) if the position of the person to be observed falls within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit. The control method of the present invention is a control method for an observation system that includes an imaging unit that captures an image of a space to be observed and a drive unit that rotates the imaging unit to change the imaging range of the imaging unit, and is characterized by comprising the steps of: a) controlling the rotational drive of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of the person to be observed in the captured image that corresponds to the one imaging range; and c) when the position of the person to be observed falls within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit. [Effects of the Invention]

[0014] According to the present invention, it is possible to ensure a wider observation range while avoiding or suppressing the occurrence of stress in the person being observed. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram showing an observation system. [Figure 2] FIG. 2 is a functional block diagram showing a schematic configuration of an observation device. [Figure 3] FIG. 10 is a conceptual diagram showing how information on the three-dimensional position of each specific part of a photographed person is acquired (calculated) based on a photographed image and depth information. [Figure 4] FIG. 10 is a diagram showing three attitude angles (three preset attitude angles) of the camera unit. [Figure 5] FIG. 10 is a diagram showing three imaging ranges (three preset imaging ranges). [Figure 6] 10A and 10B are diagrams illustrating a process for changing the shooting range (selected range). [Figure 7] 10A and 10B are diagrams illustrating a process for changing the shooting range (selected range). [Figure 8] FIG. 10 is a diagram showing the light emission amount (light emission amount for each shooting range) previously set by the pre-adjustment process. [Figure 9] 10 is a flowchart showing the observation process (watching process) in detail. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] <1. System Overview> Fig. 1 is a schematic diagram showing an observation system 1. As shown in Fig. 1, the observation system 1 includes a plurality of observation devices 10 and a plurality of terminal devices 70, 80. The terminal device 70 is also referred to as a management device 70, and the terminal device 80 is also referred to as a mobile terminal device 80. Note that Fig. 1 illustrates some of the plurality of observation devices 10 and the plurality of terminal devices 70, 80, namely, the devices 10, 70, 80.

[0018] Here, the example mainly shows a mode in which the observation system 1 is used in a care facility. However, the present invention is not limited to this, and the observation system 1 may also be used in a care facility (hospital, etc.) or an ordinary home.

[0019] As shown in FIG. 1 , each observation device 10 and each terminal device 70, 80 are connected to each other via a network 108. The network 108 is configured by a LAN (Local Area Network), the Internet, etc. Furthermore, the connection to the network 108 may be a wired connection or a wireless connection. For example, the management device 70 is connected to the network 108 by wire, and each observation device 10 and each mobile terminal device 80 is connected to the network 108 wirelessly. Alternatively, all of the devices 10, 70, 80 may be connected to the network 108 wirelessly.

[0020] Each observation device 10 is placed in each room 90 (for example, each care recipient's private room) of each person being observed (here, a care recipient). Each observation device 10 (and observation system 1) is a device that observes the person being observed (such as a care recipient) based on captured images of the person being observed. In detail, each observation device 10 detects (acquires) skeletal information (such as positional information of multiple specific body parts) (described below) of the person being observed, positional information of the person being observed, and various "behavior-related events" related to the person being observed, based on the captured images.

[0021] "Behavior-related events" include a person's behavior (movement) itself and / or a state related to the person's behavior. "Behavior-related events" are events that should be detected (events that are the target of detection processing) regarding the observed person, and are also called detection target events. Examples of "behavior-related events" of a person include events such as "normal lying position (normal lying position (state of lying normally on the bed))", "sitting up (upper body) on the bed", "edge sitting position", "edge lying position", "slipping off the bed", "normal standing position", "normal sitting position", and "falling".

[0022] The observation device 10 is also called a detection device, and the observation system 1 is also called a detection system. In addition, since the observation device 10 and the observation system 1 are used to monitor the behavior of people, they are also called monitoring devices and monitoring systems.

[0023] <2. Overview of the observation device 10> 1 and 2, the observation device 10 includes a camera unit 20, a drive section 27, and a processing unit 30. Note that Fig. 2 is a functional block diagram showing a schematic configuration of the observation device 10.

[0024] The camera unit 20 is a camera that captures an image of the space to be observed (here, the space inside the living room 90).

[0025] 1, camera unit 20 is installed near or above bed 92 in room 90 (on the ceiling or upper part of a wall), and is capable of capturing images of the interior of room 90. Processing unit 30 is capable of acquiring three-dimensional positions (particularly time-series information) of multiple specific parts of the person being observed (eyes, ears, nose, neck, chest, waist, shoulders, elbows, wrists, knees, ankles, etc.) based on images (particularly moving images) captured by camera unit 20. Note that, although camera unit 20 and processing unit 30 are provided separately here, this is not limiting, and camera unit 20 and processing unit 30 may also be provided as an integrated unit.

[0026] The camera unit 20 also includes an imaging section (sensor section) 23 and an illumination section (light emitting section) 25 (see FIG. 2).

[0027] The driving unit 27 is a processing unit that can (mechanically) drive the camera unit 20 (more specifically, the imaging unit 23, the illumination unit 25, etc.) to change the attitude of the camera unit 20. The driving unit 27 is configured to include a driving mechanism (motors, gears, etc.). The driving operation by the driving unit 27 is controlled by a controller 31 (described later) of the processing unit 30.

[0028] Here, the driving unit 27 is configured with a driving mechanism (motor, gears, etc.) capable of realizing a rotational driving operation around a predetermined axis. The attitude (attitude angle) of the camera unit 20 is changed by the driving (rotational driving operation) of the driving unit 27. For example, the camera unit 20 is installed (placed) on a ceiling or the like in a state in which it can rotate around an axis parallel to the horizontal direction (for example, the X axis in FIG. 4). Then, as the driving unit 27 performs a rotational driving operation, the camera unit 20 rotates around the X axis. Furthermore, as the rotational driving operation is performed, the imaging angle (attitude angle) of the camera unit 20 is changed, and the field of view (imaging range) of the imaging unit 23 is changed. However, without being limited to this, the driving unit 27 may drive the camera unit 20 (imaging unit 23) by a rotational driving operation around two axes (or three axes), or may perform a translational driving operation. Furthermore, the driving unit 27 is not limited to being provided external to the camera unit 20, but may be built into the camera unit 20 to drive the imaging unit 23, etc.

[0029] Here, the imaging range (field of view) of the imaging unit 23 can, in principle, change to an infinite number of imaging ranges (field of view) as the rotational drive operation proceeds. However, as will be described later, in this embodiment, of the infinite number of imaging ranges, only a predetermined number of imaging ranges (here, three imaging ranges F1, F2, and F3) are used for the imaging range (field of view) of the imaging unit 23 (see FIG. 5). In other words, of the infinite number of attitude angles for the camera unit 20, only three attitude angles θ1, θ2, and θ3 for the camera unit 20 are used (see FIG. 4).

[0030] The camera unit 20 is a three-dimensional camera. The camera unit 20 acquires a captured image with depth information. Specifically, the camera unit 20 captures a captured image (e.g., infrared image) 110 (see FIG. 3 ) of a subject object (e.g., a person, a wall, a floor 91, a bed 92), and acquires depth information (depth distance information) 120 for each pixel in the captured image 110. The depth information 120 for each pixel in the captured image is information on the distance (distance from the camera unit 20) to the subject object (e.g., a person, a wall, a floor, a bed) corresponding to each pixel in the captured image 110, and is information on the distance in a direction perpendicular to the captured image 110. In other words, the depth information is distance information (depth distance information) in the normal direction of the plane of the captured image. The depth information 120 for multiple pixels can also be expressed as a captured image with depth information.

[0031] Here, a TOF (Time of Flight) 3D camera is used as the camera unit 20. As described above, the camera unit 20 includes an imaging unit (sensor unit) 23 (FIG. 2) and an illumination unit (light-emitting unit) 25. More specifically, the imaging unit 23 is configured as a camera (infrared camera) having optical elements (optical system) such as lenses and an imaging element (infrared image sensor) that receives light (infrared light). The illumination unit 25 emits illumination light (infrared light in this case) to illuminate the imaging range of the imaging unit 23. More specifically, the illumination unit 25 is a light-emitting unit that emits (emits) infrared light for distance measurement (also as illumination light) in the TOF method. The imaging unit 23 and the illumination unit 25 are configured as an integrated unit with a fixed positional relationship. The illumination unit 25 is driven by a drive unit 27 in conjunction with the imaging unit 23, and can illuminate the changed imaging range. The camera unit 20 also includes a controller 21. The controller 21 has the same hardware configuration as the controller 31 (described later) and the like.

[0032] A TOF (Time of Flight) 3D camera is a camera that measures the distance to a subject by utilizing the time of flight of light (here, infrared light). Specifically, the distance from the imaging unit 23 to the subject (depth distance information) is calculated (acquired) for each pixel in a captured image based on the time from when infrared light is emitted (irradiated) from the illumination unit (light-emitting unit) 25 to when the infrared light reaches the subject and the reflected light from the subject returns to the imaging unit (image sensor). The calculation process of the depth distance information is executed by the controller 21 or the like incorporated in the camera unit 20. Note that the sensor unit 23 may be provided with an RGB image sensor or the like that captures visible light images (color images, etc.), and color images may be captured.

[0033] In this way, the camera unit 20 acquires a captured image (captured image information) 110 (see FIG. 3) of the subject person and depth information 120 for each pixel in the captured image (information on the distance to the object corresponding to each pixel (depth distance information)).

[0034] FIG. 3 is a conceptual diagram showing how information on the three-dimensional position of each specific part of a subject person is acquired (calculated) based on a captured image 110 and depth information 120. Note that FIG. 3 shows the captured image 110 and the depth information 120 in a schematic manner. For example, in the actual captured image 110, the person is captured as is, whereas in the captured image 110 of FIG. 3, the person is represented (simplified) by a figure made up of connected ellipses. The same is true for other depth information 120. Furthermore, while the actual depth information 120 includes information on the distance to an object corresponding to each pixel, in the depth information 120 of FIG. 3, the distance to the object corresponding to each pixel is expressed by converting it into the density of each pixel. In detail, the magnitude of the distance is expressed by the magnitude of density (shade).

[0035] First, the processing unit 30 (particularly the controller 31) analyzes the captured image 110 and acquires skeletal information (skeletal model information) 140 (see the lower left portion of FIG. 3 ) of the captured person based on the texture information, etc., of the captured image 110. The skeletal information 140 is information that (simplified) represents the skeleton of the captured person using a plurality of specific parts (specifically, the chest, nose, neck, shoulders, elbows, wrists, waist, knees, ankles, eyes, ears, etc.) (mainly joints) of the captured person and skeletal lines (links) connecting the plurality of specific parts. In the skeletal information 140 of FIG. 3 , each specific part is represented by a "point," and each skeletal line (connecting line) is represented by a "line segment." The skeletal information 140 based on the captured image includes information about a plurality of specific parts (here, B0 to B17) (such as planar position information of each part within the image).

[0036] In addition, the processing unit 30 acquires planar position information (two-dimensional position information within the captured image in the camera coordinate system) of each specific part (for example, each representative position) of the captured person within the captured image 110 based on the skeletal information 140 of the captured person.

[0037] Furthermore, the processing unit 30 also acquires distance information (depth position information in the camera coordinate system) from the camera unit 20 to each specific part based on depth information 120 of one or more pixels at a plane position (in the photographed image 110) corresponding to the specific part. The distance information can also be expressed as depth information in the normal direction (camera line of sight direction) of the photographed image plane.

[0038] In this way, the processing unit 30 acquires information regarding the planar position of each specific part of the subject person within the captured image (two-dimensional position information within the captured image) and distance information (depth information) to each specific part of the subject person.

[0039] Then, the processing unit 30 acquires three-dimensional position information 150 (three-dimensional position information within the living space) of each specific part of the subject person, based on information about the planar position of each specific part within the captured image and distance information (depth information) to each specific part, by coordinate transformation or the like. Specifically, the processing unit 30 converts the position information in the camera coordinate system Σ1 (planar position within the captured image and depth position in the normal direction of the captured image) into three-dimensional position information 150 in a coordinate system Σ2 fixed with respect to the living room 90 (more specifically, the bed 92). The camera coordinate system Σ1 is, for example, a three-dimensional Cartesian coordinate system based on three orthogonal axes, namely, two orthogonal axes parallel to the plane of the captured image and one axis extending in a direction perpendicular to the plane of the captured image. Furthermore, the coordinate system Σ2 after the conversion is, for example, a three-dimensional Cartesian coordinate system based on three orthogonal axes, namely, two orthogonal axes parallel to the horizontal plane and one axis extending in the vertical direction (height direction). In this observation system 1 (controller 31, etc.), the position of camera unit 20 in real space is acquired in advance, and adjustments (calibration related to camera position, etc.) are performed in advance based on the position of camera unit 20. In other words, the positional relationship between the two coordinate systems Σ1 and Σ2 is adjusted in advance.

[0040] More specifically, such calibration between the coordinate systems Σ1 and Σ2 is performed for each of the three attitude angles θi (i=1, 2, 3) (see FIG. 4 (described later)) of the camera unit 20. In other words, the calibration is performed for each of the three imaging ranges Fi.

[0041] Here, the controller 31 of the processing unit 30 generates the skeletal information 140 and the three-dimensional position information 150 of each specific part, but this is not limiting.

[0042] For example, the controller 21 of the camera unit 20 may generate the skeletal information 140. Alternatively, the controller 21 of the camera unit 20 may generate three-dimensional position information 150 of each specific part. In particular, the controller 21 may acquire information about the position of each specific part of the subject person in the captured image and distance information (depth information) to each specific part, and acquire the three-dimensional position information 150 of each specific part by coordinate transformation or the like. The controller 31 of the processing unit 30 may then acquire the three-dimensional position information 150 of each specific part from the camera unit 20. Alternatively, the controllers 21, 31 of the camera unit 20 and the processing unit 30 may cooperate to execute these various processes. In other words, the skeletal information 140 and the three-dimensional position information 150 may be generated by both or one of the controllers 21, 31.

[0043] Additionally, although a TOF (Time of Flight) camera is exemplified as a 3D camera here, the present invention is not limited to this. The 3D camera may be a stereoscopic camera, or may be a 3D camera using a pattern illumination method that acquires depth information for each pixel based on the irradiation of a dot pattern of infrared light. Any type of 3D camera may acquire a captured image of a person being observed and depth information (such as depth information for each pixel) that is information on the distance to a subject object in the captured image.

[0044] Moreover, since the observation device 10 is a device that executes image processing on captured images, etc., it is also expressed as an image processing device. Furthermore, the observation system 1 is also expressed as an image processing system, etc.

[0045] <3. Processing unit 30> As shown in FIG. 2, the processing unit 30 of the observation device 10 includes a controller (also referred to as a control unit) 31, a storage unit 32, a communication unit 34, and an operation unit 35.

[0046] The controller 31 is a control device that is built into the processing unit 30 and controls the observation device 10 .

[0047] The controller 31 is configured as a computer system including one or more hardware processors (for example, a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (more specifically, a group of program modules) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the observation device 10. Alternatively, the program may be downloaded via a communication network or the like and installed in the observation device 10.

[0048] The controller 31 acquires captured image information about a subject person (such as a person to be observed) from the camera unit 20, and also acquires distance information (depth information) to each specific part of the subject from the camera unit 20. Then, the controller 31 acquires three-dimensional position information of each specific part based on the captured image information (particularly, planar position information of each specific part of the subject in the captured image) and the distance information (depth information of the subject).

[0049] Furthermore, the controller 31 uses the learning model 400 (FIG. 2) to detect behavior-related events of the target person.

[0050] As the learning model 400, for example, a neural network model composed of multiple layers is used. Then, weighting coefficients (learning parameters) between multiple layers (input layer, (one or more) intermediate layers, and output layer) in the neural network model are adjusted using a predetermined machine learning method. The learning model 400 after being trained by machine learning is also referred to as a trained model. Specifically, the learning parameters of the learning model (learner) 400 are adjusted using a predetermined machine learning method, and a trained learning model (trained model) 400 is generated.

[0051] Therefore, first, the controller 31 executes the learning stage process in machine learning. Specifically, the controller 31 performs machine learning in advance on the learning model 400 using a plurality of training data. As each training data, training data is used that receives information indicating the positional relationships of a plurality of specific body parts of a person as input and outputs behavior-related events related to the person. Then, a predetermined machine learning method is used to adjust weighting coefficients (learning parameters) between multiple layers (input layer, (one or more) intermediate layers, and output layer) in the neural network model. As a result, a trained learning model 400 (trained model) is generated (produced).

[0052] Thereafter, the controller 31 executes processing at the inference stage using the trained learning model 400. Specifically, the controller 31 uses the learning model 400 (trained model) that has been machine-learned using the plurality of teacher data to detect behavior-related events related to the target person (also referred to as a person to be detected or a person to be determined) based on information indicating the positional relationships of a plurality of specific parts of the target person. More specifically, the controller 31 inputs the positional relationships of a plurality of specific parts of the target person, etc., into the trained model 400, and obtains output from the trained model 400 (processing results of event detection processing related to the behavior-related events of the target person).

[0053] Here, the observation device 10 (controller 31) performs both the learning stage process and the inference stage process in machine learning. However, this is not limited to this, and for example, the learning stage process and the inference stage process may be performed by different devices.

[0054] Furthermore, the controller 31 cooperates with the communication unit 34 and the like to transmit the detection results to the terminal devices 70, 80, and the terminal devices 70, 80 output the detection results (such as display output and / or audio output).

[0055] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 32 stores (memorizes) the above-mentioned programs and various data. For example, the storage unit 32 stores (memorizes) time-series data such as the three-dimensional positions of multiple specific parts of the observed person, as well as various data and programs used for learning and using the learning model 400.

[0056] The communication unit 34 is capable of performing network communication via the network 108. This network communication utilizes various protocols, such as TCP / IP (Transmission Control Protocol / Internet Protocol). By utilizing this network communication, the observation device 10 can exchange various data with desired destinations (e.g., terminal devices 70 and 80).

[0057] The operation unit 35 includes an operation input unit 35a that receives operation inputs to the observation device 10, and a display unit 35b that displays and outputs various types of information.

[0058] <4. Processing Overview> Next, an outline of the processing in this embodiment will be described.

[0059] As a technique different from the embodiments of the present application (also referred to as a comparative example), as described above, it is conceivable to detect the position of a person (person being observed) and perform constant tracking control so that the camera direction (direction of the camera's line of sight) is always directed toward the person (position). More specifically, it is conceivable to perform a process (constant tracking process) of constantly tracking the person being observed by driving the camera unit 20 with a driving unit 27 (motor, etc.) so that the person being observed is always in the center of the captured image. However, in such a technique (comparative example), the driving sound of the camera is generated frequently (continuously) in response to the person's movements. Therefore, there is a high possibility that this will cause stress (psychological burden) to the person.

[0060] Therefore, in this embodiment, of the infinite number of shooting ranges, only a predetermined number (three in this case) of shooting ranges are used as the shooting range (field of view) of the image capturing section 23 (see FIG. 5). In other words, of the infinite number of attitude angles related to the camera unit 20, only three attitude angles related to the camera unit 20 (three predetermined attitude angles) are used (see FIG. 4). In detail, the controller 31 changes the shooting range of the image capturing section 23 to one of the predetermined number (three in this case) of shooting ranges that have been set in advance. Then, the controller 31 basically executes a specific observation operation (particularly, an event detection process related to the person being observed) only in one of the three shooting ranges.

[0061] More specifically, the controller 31 changes the imaging range of the imaging unit 23 to one of the predetermined number of imaging ranges and stops driving the imaging unit 23 by the drive unit 27, and then observes the observed person based on the captured images (newly and sequentially) captured by the imaging unit 23. Specifically, during the period when driving of the imaging unit 23 is stopped, the controller 31 executes, as the observation process of the observed person, a process of detecting skeletal information of the observed person (specific part detection process, etc.), a process of detecting the position of the observed person (person position detection process), and a process of detecting a behavior-related event related to the observed person (event detection process). However, it is not necessary to execute all of these various processes, and only some of the processes may be executed. Alternatively, the observation process of the observed person may only execute a process of acquiring (newly and sequentially) captured images by the imaging unit 23 and displaying the captured images on the display unit 35b, etc.

[0062] In this embodiment, the process of acquiring a captured image by the imaging unit 23, and the process of detecting skeletal information and the process of detecting a person's position based on the captured image continue even during the driving period (non-stop period) of the imaging unit 23 (camera unit 20) by the driving unit 27. On the other hand, the process of detecting an event related to the person being observed is temporarily suspended during the driving period (non-stop period) of the imaging unit 23, and is resumed after the driving is completed (after stopping).

[0063] Fig. 4 is a diagram showing three attitude angles (three preset attitude angles) θ1, θ2, and θ3 of the camera unit 20. Here, only the three attitude angles θ1, θ2, and θ3 shown in Fig. 4 are used out of the infinite number of attitude angles of the camera unit 20. Note that the attitude angle θ of the camera unit 20 is expressed as the angle (depression angle (angle looking down from the horizontal direction)) between the horizontal direction and the line of sight of the camera unit 20 (imaging section 23).

[0064] In the bottom row of Fig. 4, camera unit 20 installed on the ceiling captures an image of the area around the bed directly below (see the bottom row of Fig. 5). At this time, angle θ1 of camera unit 20 is, for example, 90 degrees (depression angle).

[0065] In the middle of Fig. 4, camera unit 20 has rotated several tens of degrees (for example, 25 degrees) counterclockwise around the X axis (in a direction that reduces the depression angle) from a state having angle θ1, transitioning to a state having angle θ2. Angle θ2 of camera unit 20 is, for example, 65 degrees (depression angle). At this time, camera unit 20 (θ = θ2) is mainly capturing an area that is shifted by a predetermined amount toward door (entrance) 93 (to the right in Fig. 4) from the bed (see the middle of Fig. 5).

[0066] In the top row of FIG. 4, camera unit 20 has rotated a further few tens of degrees (for example, 25 degrees) around the X axis (in a direction that reduces the depression angle) from a state having angle θ2 to a state having angle θ3. Angle θ3 of camera unit 20 is, for example, 40 degrees (depression angle). At this time, camera unit 20 (θ=θ3) is mainly capturing an image of an area further shifted by a predetermined amount toward door (entrance / exit) 93 (to the right in FIG. 4) (see the top row of FIG. 5).

[0067] The specific values ​​of these three angles θ1, θ2, and θ3 are not limited to the above values, and other values ​​may be used as appropriate.

[0068] FIG. 5 is a diagram showing three imaging ranges (three preset imaging ranges) related to the imaging unit 23. Here, of the infinite number of imaging ranges (field of view ranges) related to the imaging unit 23, only three imaging ranges (field of view ranges) F1, F2, and F3 shown in FIG. 5 are used. The three imaging ranges F1, F2, and F3 correspond to the attitude angles θ1, θ2, and θ3 (of the camera unit 20) in FIG. 4, respectively. That is, the imaging range F1 is the imaging range corresponding to the attitude angle θ1, and the imaging range F2 is the imaging range corresponding to the attitude angle θ2. The imaging range F3 is the imaging range corresponding to the attitude angle θ3. In other words, these imaging ranges F1, F2, and F3 are realized by changing the attitude angle θ of the camera unit 20 to angles θ1, θ2, and θ3, respectively.

[0069] The shooting range F1 in the bottom row of Figure 5 is a shooting range that shows the area around the bed directly below as seen from the ceiling. The shooting range F1 is a shooting range that can capture a person near the bed (at a relatively close position), and in particular, a shooting range that can capture the entire view of the bed. In other words, the shooting range F1 is a shooting range of a close view captured by a camera installed above the room (on the ceiling or upper part of a wall, etc.) looking down on the area directly below.

[0070] 5 is a shooting range that can capture an image of a person who is slightly away from the bed toward the door (room entrance) 93. The shooting range F2 is also a shooting range that can capture an image of a person who is near the bed. However, compared to the shooting range F1, the shooting range F2 is a shooting range that can capture an image of a person who is farther away from the bed (and from the imaging unit 23).

[0071] The imaging range F3 at the top of Figure 5 is a range that can capture an image of a person who has moved far away from the bed (especially a person who has moved closer to door 93). The imaging range F3 can also be expressed as a range that can capture an image of a person (at a relatively distant location) near an entrance (room entrance) away from the bed. In other words, the imaging range F3 is a distant imaging range captured by a camera installed at the top of the room, looking slightly downward at the distant entrance to the room. In this way, the imaging range F3 is a range that can capture an image of a person who is farther away from the bed (also from the imaging unit 23) than the imaging ranges F1 and F2.

[0072] In this way, the three imaging ranges F1, F2, and F3 are imaging ranges (discontinuous imaging ranges) of predetermined stages (three stages) that change in stages from a close view (see the bottom row of FIG. 5) to a distant view (see the top row of FIG. 5). In other words, the imaging ranges F3, F2, and F1 are imaging ranges that can capture a person who is farther away from the imaging unit 23 in this order. In detail, the imaging range F3 is the imaging range that can capture a person who is located farthest from the imaging unit 23 (or an origin position close to the imaging unit 23 (here, the bed position)) among the imaging ranges F3, F2, and F1. Furthermore, the imaging range F1 is the imaging range that can capture a person who is located closest to the imaging unit 23 (or an origin position close to the imaging unit 23 (here, the bed position)) among the imaging ranges F3, F2, and F1.

[0073] These three imaging ranges F1, F2, and F3 are set in advance in a pre-adjustment process (described later) that is performed before the observation process (described later) in step S20 (see FIG. 9). Furthermore, in this pre-adjustment process, a light emission amount setting process, image pre-processing, bed area setting process, etc. are executed for each of these three imaging ranges. The pre-adjustment process will be described later.

[0074] Although three shooting ranges are exemplified as the predetermined number of shooting ranges set in advance here, the number is not limited to this. For example, the predetermined number of shooting ranges set in advance may be two shooting ranges, or four or more shooting ranges.

[0075] <5. Details of observation processing> <Changing the shooting range> In this embodiment, one shooting range is selected from a predetermined number (three in this case) of shooting ranges, and the person to be observed is observed based on the one shooting range in a stopped state in which the image capture unit 23 is not driven by the drive unit 27. In other words, the shooting range of the image capture unit 23 is changed to one shooting range (one selected range) from the predetermined number of shooting ranges, and the drive of the image capture unit 23 by the drive unit 27 is stopped, and the person to be observed is observed based on a (newly) captured image by the image capture unit 23.

[0076] In detail, the position of the person to be observed in the captured image is detected from time to time (at a predetermined sampling time interval) based on the captured image. Only when the detected position (position of the person to be observed) moves into a trigger area G (described later) in the captured image, is the imaging unit 23 driven by the driving unit 27. Specifically, a process of changing the imaging range of the imaging unit 23 (a process of changing from one imaging range to another imaging range) is performed. Then, the person to be observed is observed based on the captured image captured in the changed imaging range.

[0077] On the other hand, if the position of the person to be observed is within a non-trigger area C in the captured image (area C excluding a trigger area G (described later)), the driving unit 27 does not drive the imaging unit 23. Then, in a state where the imaging range of the imaging unit 23 is maintained at the one imaging range without driving the driving unit 27 (camera drive stopped state), the person to be observed is observed based on the captured image (further) captured by the imaging unit 23.

[0078] This process suppresses the occurrence of drive operations, making it possible to avoid or suppress the occurrence of stress (in the observed person) due to drive sounds (motor drive sounds, etc.).

[0079] 6 and 7 are diagrams illustrating the process of changing the shooting range (selected range). This change process can also be expressed as a process of switching (selecting) between the three shooting ranges F1, F2, and F3.

[0080] As shown in Figures 6 and 7, a "trigger area" (area that triggers an update of the shooting range) G and a "non-trigger area" (area other than the trigger area) C are set for each shooting range F. Note that in Figures 6 and 7, the trigger area G is marked with diagonal hatching or cross hatching, and the non-trigger area C is marked with sand hatching.

[0081] Specifically, within the photographing range F1, the upper end region (for example, the upper 20% of the photographing range F1) is set as a trigger region G (specifically, G11). On the other hand, the region other than the trigger region G11 (the region excluding the upper edge portion G11) is set as a non-trigger region C (specifically, C1).

[0082] In the photographing range F2, the upper end region (for example, the upper 20% of the photographing range F2) is set as a trigger region G (specifically, G21). Also, the lower end region (for example, the lower 20% of the photographing range F2) is set as a trigger region G (specifically, G22). Meanwhile, the region other than the trigger regions G21 and G22 (the central region) is set as a non-trigger region C (specifically, C2).

[0083] In the photographing range F3, the lower end area (for example, 20% from the upper end to the lower end) of the photographing range F1 is set as a trigger area G (specifically, G32). On the other hand, the area other than the trigger area G32 is set as a non-trigger area C (specifically, C3).

[0084] The upper trigger areas G11 and G21 are used to change the shooting range (sequentially) to a relatively distant view (F1 → F2 → F3). Conversely, the lower trigger areas G32 and G22 are used to change the shooting range (sequentially) to a relatively close view (F3 → F2 → F1).

[0085] The position of the observed person is acquired as planar position information of a predetermined part within the captured image based on skeletal information 140 of the observed person. Furthermore, when changing the shooting range to a relatively distant view and when changing the shooting range to a relatively close view, the positions of different parts may be used as the position of the observed person. Here, when changing the shooting range to a relatively distant view, the position of the person's neck is used as the position of the observed person, and the position of the person's waist is used as the position of the observed person.

[0086] <Example of operation> A more specific example of the operation will be described below.

[0087] 6, when the imaging range of imaging unit 23 is imaging range F1, the positional relationship between the position of the person's neck in the captured image captured in imaging range F1 and trigger area G11 is determined. Then, only when the detected position (position of the person's neck) moves into trigger area G11 in the captured image, driving unit 27 drives imaging unit 23, and the imaging range of imaging unit 23 is changed to imaging range F2.

[0088] For example, when the imaging range of imaging unit 23 is imaging range F1 (see the bottom row of FIG. 6), consider a situation in which a person (person to be observed) is sleeping in bed, or a situation in which the person has gotten out of bed and is standing next to the bed. In such a situation, the position of the person (specifically, the position of their neck) is determined to still be within non-trigger area C1. Then, driving unit 27 does not drive imaging unit 23, and the imaging range of imaging unit 23 is maintained at imaging range F1.

[0089] Meanwhile, assume that the person gets off the bed and walks toward the center of the room. In this situation, if it is determined that the position of the person (specifically, the position of the person's neck) has moved into trigger area G11, driving unit 27 drives imaging unit 23, and the imaging range of imaging unit 23 is changed from imaging range F1 to imaging range F2. Within the updated imaging range F2, the person is present (photographed) near the center. The position of the person is within non-trigger area C2 of imaging range F2.

[0090] Similarly, when the imaging range of the imaging unit 23 is imaging range F2 (see the middle part of FIG. 6), the positional relationship between the position of the person's neck and the trigger areas G21 and G22 in the captured image captured in imaging range F2 is determined. Then, only when the detected position (position of the person's neck) moves into trigger area G21 (, G22) in the captured image, the driving unit 27 drives the imaging unit 23, and the imaging range of the imaging unit 23 is changed to imaging range F3 (, F1).

[0091] For example, assume that the imaging range of imaging unit 23 is imaging range F2 (see the middle part of FIG. 6 ) and a person is present near a chair in the center of a living room. In this situation, the position of the person (specifically, the position of the person's neck) is determined to still be within non-trigger area C2. Then, driving unit 27 does not drive imaging unit 23, and the imaging range of imaging unit 23 is maintained at imaging range F2. In a situation where a person is sitting in the chair and relaxing, driving operation by driving unit 27 is not performed.

[0092] On the other hand, in a situation where the person (observed person) moves further toward door 93, it is determined that the position of the person (specifically, the position of the person's neck) has moved into trigger area G21. In response to this, drive unit 27 drives imaging unit 23, and the shooting range of imaging unit 23 is changed from shooting range F2 to shooting range F3. Within the updated shooting range F3, the person is present (photographed) near the center. The position of the person is present within non-trigger area C3 of shooting range F3.

[0093] Furthermore, in a situation where the person is positioned within the non-trigger area C3 of the shooting range F3 (such as when the person is doing exercises in the non-trigger area C3 or sitting on a chair (not shown)), the driving unit 27 does not drive the imaging unit 23, and the shooting range of the imaging unit 23 is maintained within the shooting range F3.

[0094] Conversely, as shown in Figure 7, when a person moves from near the door 93 to the bed 92, the positional relationship between the lower trigger areas G32, G22 and the person's position (more specifically, the position of the person's waist) is determined, and the shooting range of the imaging unit 23 is sequentially changed from shooting range F3 to shooting range F1 (F3 → F2 → F1).

[0095] For example, assume that the imaging range of the imaging unit 23 is imaging range F3 (see the top row of FIG. 7), and a person is walking from the door (entrance) 93 toward the bed.

[0096] In this situation, when the position of the person (more specifically, the position of the person's waist) is within non-trigger area C3, no driving is performed by drive unit 27, and imaging range F3 is maintained. On the other hand, when the position of the person (more specifically, the position of the person's waist) moves into trigger area G32, driving is performed by drive unit 27, and the imaging range of imaging unit 23 is changed from imaging range F3 to imaging range F2 (see the middle part of FIG. 7). Immediately after the change, the position of the person is within non-trigger area C2 of imaging range F2.

[0097] Thereafter, when the position of the person (more specifically, the position of the person's waist) is within the non-trigger area C2, the driving unit 27 does not drive the person, and the imaging range F2 is maintained. On the other hand, when the position of the person (more specifically, the position of the person's waist) moves into the trigger area G22, the driving unit 27 drives the person, and the imaging range of the imaging unit 23 is changed from the imaging range F2 to the imaging range F1 (see the bottom row of FIG. 7).

[0098] Note that the following mainly illustrates an example in which the imaging range of the imaging unit 23 changes from imaging range F1 to imaging range F3 (F1 → F2 → F3) (see FIG. 6) or from imaging range F3 to imaging range F1 (F3 → F2 → F1) (see FIG. 7). However, this is not limiting, and the imaging range of the imaging unit 23 may change from imaging range F1 to imaging range F2 and then return to imaging range F1. Alternatively, the imaging range of the imaging unit 23 may change from imaging range F3 to imaging range F2 and then return to imaging range F3.

[0099] <Processing flow details> Fig. 9 is a flowchart showing the above-described observation process (monitoring process) in detail. The process of Fig. 9 (also referred to as the process of step S20) is executed by the controller 31. It is assumed that the pre-adjustment process (process of step S10) described below has already been executed prior to the process of step S20.

[0100] First, in step S21, various initial processes are executed. Specifically, a process for setting an initial image capture range (for example, a process for setting the image capture range F1 as the image capture range of the image capture unit 23 in response to an input from a caregiver when the care recipient is lying in bed 92) is executed. Furthermore, various processes according to the initial image capture range are executed. For example, various settings prepared in advance corresponding to the image capture range F1 are adopted. In detail, the light emission amount (light emission amount of the illumination unit 25), image pre-processing parameters (brightness adjustment parameters, etc.), bed area, etc. prepared in advance corresponding to the image capture range F1 are adopted.

[0101] Next, in step S22, a captured image (a captured image of the imaging range F1) captured by the imaging unit 23 is acquired, and an observation process for observing a person is executed based on the captured image. The observation process may include, for example, a process for detecting skeletal information of the person, a process for detecting the position of the person, and a process for detecting an event related to the person's behavior.

[0102] In step S23, it is determined whether the person (person to be observed) is present in the trigger area G or the non-trigger area C.

[0103] If it is determined that a person is present in the non-trigger area C, the camera driving stopped state continues (step S24), and the process returns to step S22. The observation process with the camera driving stopped continues through the loop process of steps S22, S23, and S24.

[0104] On the other hand, if it is determined that a person is present within trigger area G, the process proceeds from step S23 to step S25, where the imaging range of image capture unit 23 is changed by a driving operation of drive unit 27. For example, a transition from imaging range F1 to imaging range F2 is performed. Furthermore, in step S26, a change process is performed to various settings that have been prepared in advance corresponding to the changed imaging range. In detail, a change process is performed to the light emission amount (light emission amount of illumination unit 25), image pre-processing parameters, bed area, etc., that have been prepared in advance corresponding to the changed imaging range (for example, imaging range F2).

[0105] <Effects of the embodiment> As described above, in the above embodiment, the shooting range of the imaging unit 23 is changed to one of a predetermined number of shooting ranges (one selected range) and the driving of the imaging unit 23 by the driving unit 27 is stopped, and the person to be observed is observed based on the captured image captured by the imaging unit 23.

[0106] This process makes it possible to avoid or suppress stress (in the person being observed) caused by the driving noise (motor driving noise, etc.) of the drive unit 27. Also, by switching between a predetermined number of shooting ranges, it is possible to ensure a wider observation range.

[0107] Furthermore, since a relatively small number of cameras (one in this case) are sufficient to ensure a wide observation range, costs can be reduced compared to when multiple cameras are provided.

[0108] In particular, in the above embodiment, when the detected position of the person being observed falls within the non-trigger area C in the captured image, the driving unit 27 is not driven and the imaging range of the imaging unit 23 is maintained at one imaging range. In other words, only when the detected position (detected position of the person being observed) moves into the trigger area G (described later) in the captured image is the driving unit 27 driving the imaging unit 23.

[0109] This keeps the drive stopped even if the person moves slightly within the shooting range, making it possible to more reliably avoid or suppress stress (in the person being observed) caused by the drive noise (motor drive noise, etc.) of the drive unit 27.

[0110] <7. Pre-adjustment processing> Next, the pre-adjustment process (step S10 (not shown)) for a predetermined number of shooting ranges (here, three shooting ranges F1, F2, and F3) will be described. This pre-adjustment process (also referred to as pre-setting process) is executed prior to the observation process (watching process) described above (step S20 in FIG. 9).

[0111] First, as described above, the posture angles θi of the cameras corresponding to the three shooting ranges Fi (i = 1, 2, 3) are respectively set. Specifically, the angle θ1 is set for the shooting range F1, the angle θ2 is set for the shooting range F2, and the angle θ3 is set for the shooting range F3.

[0112] Furthermore, settings such as the light emission amount setting, the image preprocessing parameter setting, and the bed area setting are executed for each shooting range Fi.

[0113] The light emission amount setting is a process of adjusting and setting the light emission amount of the lighting unit 25 (specifically, the light emission amount of the infrared light for distance measurement in the TOF method) for each shooting range Fi. Appropriate light emission amounts Li (see FIG. 8) for each shooting range Fi are respectively set in advance. By such a light emission amount setting, the light emission amount Li corresponding to the shooting range Fi is determined in advance (before step S20).

[0114] The shooting range F1 is a shooting range with a relatively short distance to the observed person (shorter than the other shooting ranges). Conversely, the shooting range F3 is a shooting range with a relatively long distance to the observed person (longer than the other shooting ranges). Reflecting such circumstances, the light emission amount L3 corresponding to the shooting range F3 is set to a value larger than the light emission amount L1 corresponding to the shooting range F1. Also, as the light emission amount L2 corresponding to the shooting range F2, an intermediate value is set (L1 < L2 < L3). In cases where there is no significant difference in the appropriate light emission amount, L1 = L2 or L2 = L3 may also be possible.

[0115] This allows for easier adjustment of the light emission amount, since adjustments need only be made for a predetermined number (three in this example) of shooting ranges Fi, rather than an infinite number of shooting ranges. It also allows for appropriate setting of light amounts suitable for each of the limited number of shooting ranges Fi. For example, when shooting range F3 in step S20 (see FIG. 9), a sufficiently large light emission amount L3 is used, thereby ensuring sufficient light (infrared light amount for distance measurement) to reach a relatively distant subject (such as a person). On the other hand, when shooting range F1, a (relatively small) light emission amount L1 is used, thereby allowing for appropriate suppression of the light emission amount.

[0116] Image pre-processing parameter setting is a process for adjusting the brightness, etc., of a captured image. Since the distance to the subject differs for each shooting range Fi and the light emission amount Li differs for each shooting range Fi, it is preferable to set parameters that appropriately adjust the brightness, etc., of a captured image. In the image pre-processing parameter setting, appropriate parameters are set for each shooting range Fi. For example, adaptive histogram smoothing processing, etc., is performed and appropriate parameters (brightness adjustment parameters) are set.

[0117] In this embodiment, image pre-processing parameter settings only require adjustment for a predetermined number of (here, three) imaging ranges Fi, rather than an infinite number of imaging ranges, which makes it possible to simplify the adjustment.

[0118] Bed area setting is a process of setting (designating) a bed area in a captured image. The bed area setting is performed by, for example, drawing and designating the range of the upper surface area of ​​the bed 92 within each imaging range Fi using a polygon (rectangle) or the like within the captured image of each imaging range Fi. Furthermore, the three-dimensional position of the bed 92 is identified by, for example, designating the bed height by numerical input. A behavior-related event related to the person (observed person) is detected using, for example, the positional relationship between the set bed area (three-dimensional bed space) and each specific part of the person.

[0119] Regarding bed area setting, in this embodiment, it is only necessary to perform setting operations for only three imaging ranges Fi, not an infinite number of imaging ranges. Therefore, it is possible to simplify bed area setting. In particular, when a caregiver (care staff) sets the bed area, it is possible to reduce the burden on the caregiver.

[0120] Furthermore, for a limited number of imaging ranges Fi, it is possible to specify each bed area relatively easily by two-dimensionally specifying the bed area.

[0121] <8. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0122] <Unmoving object detection> For example, in the above embodiment, a moving object determination (stationary object determination) may be further performed using a differential image in each shooting range Fi. This makes it possible to accurately distinguish a person (stationary object) in a poster posted on a wall (or floor, etc.) from a living person (moving object).

[0123] Specifically, in each shooting range Fi, a moving object portion is detected based on a difference image between an image (still image) taken at a certain time point and an image (still image) taken at another time point. More specifically, a difference occurrence region (different region) in the difference image is detected as a moving object portion.

[0124] Specifically, when a human skeleton is detected in a certain area within the imaging range Fi based on image features, a difference image is calculated between the area where the skeleton is detected (detection area) in an image captured at a certain time and the area at the same position in an image captured at another time. If it is determined that there is a significant difference between the two areas based on the difference image, it is determined that a moving object exists in the detection area. In other words, it is determined that a living person exists.

[0125] When only a predetermined number (here, three) of imaging ranges Fi are used for person detection, rather than an infinite number of imaging ranges, the same imaging range Fi continues for a certain period of time or more, and images captured at different times by the same imaging range Fi are obtained. Therefore, it is possible to realize moving object (stationary object) determination using differential images in each imaging range Fi. In other words, by employing a determination process using differential images, it is possible to realize moving object (stationary object) determination relatively easily.

[0126] <2D camera> Furthermore, in the above embodiment, a three-dimensional camera is used as the camera unit 20, but the present invention is not limited to this, and the camera unit 20 may be a two-dimensional camera.

[0127] <Learning model for each shooting range> In the above embodiment, the same learning model 400 is used in common for the three imaging ranges Fi, but this is not limiting. For example, a different learning model 400 may be used for each of the three imaging ranges Fi. Specifically, an individual learning model specialized for each imaging range Fi may be used. Such a modification will be described below.

[0128] In this modification, for the imaging range F1, a learning model 400 (401 (not shown)) suitable for the pose of the person in the image captured from a viewpoint looking down from above (the positional relationship between the joints) is used. On the other hand, for the imaging range F3, a learning model 400 (403 (not shown)) suitable for the pose of the person in the image captured from a viewpoint looking from the side (the positional relationship between the joints) is used. Furthermore, for the imaging range F2, a learning model 400 (402 (not shown)) suitable for the pose of the person in the image captured from an intermediate viewpoint (the positional relationship between the joints) is used. Note that it is not necessary to use a different learning model 400 for each imaging range Fi. For example, for the imaging range F2, the same learning model 401 (or 403) as for the imaging range F1 (or F3) may be used.

[0129] For example, in a captured image of the imaging range F1, the joints of a person lying on a bed (a person in a recumbent position), such as the neck, shoulders, waist, knees, and ankles, are arranged in a plane with a relatively large gap between them and a specific positional relationship (two-dimensional (planar) positional relationship). On the other hand, the joints of a person in a standing position, such as the neck, shoulders, waist, knees, and ankles, are relatively close to each other and have a specific positional relationship in the captured image. Furthermore, the joints of the neck, chest, and waist of a person in a sitting position have a positional relationship similar to that in the standing position, but the joints of the knees and ankles are located slightly away from the waist (in a certain direction). For the imaging range F1, a learning model 401 may be used that machine-learns such specific positional relationships in a plane within the captured image (specific positional relationships when viewed from above).

[0130] Conversely, in the captured image of the imaging range F3, the joints of the neck, shoulders, waist, knees, ankles, etc. of a person in a standing posture are arranged at relatively large intervals in the vertical direction (so as to have a specific positional relationship). Furthermore, the joints of the neck, chest, and waist of a person in a sitting posture are arranged in the vertical direction, while the joints of the waist and knees are arranged in a direction other than the vertical direction (such as the horizontal direction). For the imaging range F3, a learning model 403 may be used that machine-learns such specific planar positional relationships (specific positional relationships when viewed from the side) within the captured image. Furthermore, because the imaging range F3 hardly includes the area near the bed, the learning model 403 may not need to detect some events, such as "slipping off the bed."

[0131] If one learning model (common learning model) is constructed to correspond to an infinite number of shooting ranges F corresponding to an infinite number of shooting angles θ, the learning model 400 will often become a relatively large (complex) model (a model with a large number of parameters and / or a large processing load) in order to appropriately reflect the diverse situations corresponding to the extremely different shooting angles.

[0132] On the other hand, in the above-described modified example, a learning model 400 is constructed according to an image captured from a specific shooting angle (specific shooting range), more specifically, a learning model 400 specialized for each shooting range Fi. This makes it possible to construct relatively small (simple) learning models 401, 402, 403 (models with a small processing load and / or a small number of parameters) that appropriately reflect the unique features of each shooting range Fi.

[0133] Therefore, it is possible to reduce the processing load on the CPU and / or reduce the memory capacity, and as a result, it is possible to operate each of the learning models 401, 402, and 403 at high speed (by reducing the processing time) on a CPU with the same performance. [Explanation of symbols]

[0134] 1. Observation System 10 Observation equipment 20 Camera Unit 23 Imaging unit 25 Lighting Department 27 Drive unit 30 processing units 90 Room 91 beds 92 beds 93 Door (Entrance / Exit) θi Camera unit attitude angle (photography angle) Fi, F1, F2, F3 shooting range C, C1, C2, C3 Non-trigger area G, G11, G21, G22, G32 trigger area Li luminous output

Claims

1. an imaging unit that captures an image of an observation target space; a drive unit that rotates the imaging unit to change the imaging range of the imaging unit; an illumination unit that emits illumination light for illuminating a photographing range of the imaging unit; a control unit that controls the rotational driving of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance, and also controls the emission of illumination light by the illumination unit; Equipped with an observation system characterized in that the control unit changes the shooting range of the imaging unit to one of the predetermined number of shooting ranges and stops the rotational driving of the imaging unit by the drive unit, causes the lighting unit to emit illumination light of an amount of light emission corresponding to the one shooting range from among the light emission amounts predetermined for each of the predetermined number of shooting ranges, and performs observation processing of the person being observed based on the captured image captured by the imaging unit.

2. The control unit Detecting the position of the person to be observed within the captured image; The observation system described in claim 1, characterized in that when the position of the person being observed falls within a predetermined area in the captured image, the driving unit is not driven and the capturing range of the imaging unit is maintained at the one capturing range, and an observation process of the person being observed is performed based on the captured image captured by the imaging unit.

3. the predetermined number of imaging ranges includes a first imaging range and a second imaging range, 3. The observation system according to claim 1, wherein the second shooting range is a shooting range that is capable of capturing an image of the person being observed that is farther away from the imaging unit than the first shooting range.

4. the predetermined number of imaging ranges includes a first imaging range and a second imaging range, The imaging unit is installed near and above the bed, the first imaging range is an imaging range in which the person to be observed who is present near the bed can be imaged, 3. The observation system according to claim 1, wherein the second imaging range is a imaging range that can capture an image of the person to be observed who is present near an entrance away from the bed.

5. 5. The observation system according to claim 3, wherein the amount of light emitted determined in correspondence with the second imaging range is greater than the amount of light emitted determined in correspondence with the first imaging range.

6. a computer that controls an observation system that includes an imaging unit that captures an image of a space to be observed, a drive unit that rotates and drives the imaging unit to change the imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit; a) controlling the rotational driving of the imaging unit by the driving unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance; b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and stopping the rotational driving of the imaging unit by the drive unit, causing the lighting unit to emit illumination light of an amount of light emission corresponding to the one imaging range from among light emission amounts predetermined for each of the predetermined number of imaging ranges, and executing an observation process of the person to be observed based on the captured image captured by the imaging unit; A program to execute.

7. A control method for an observation system including an imaging unit that captures an image of an observation target space, a drive unit that rotationally drives the imaging unit to change an imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit, a) controlling the rotational driving of the imaging unit by the driving unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance; b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and stopping the rotational driving of the imaging unit by the drive unit, causing the lighting unit to emit illumination light of an amount of light emission corresponding to the one imaging range from among light emission amounts predetermined for each of the predetermined number of imaging ranges, and executing an observation process of the person to be observed based on the captured image captured by the imaging unit; A control method for an observation system, comprising:

8. an imaging unit that captures an image of an observation target space; a drive unit that rotates the imaging unit to change the imaging range of the imaging unit; a control unit that controls the rotational driving of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that are set in advance; Equipped with The control unit After setting the imaging range of the imaging unit to one of the predetermined number of imaging ranges, detecting the position of an observed person in a captured image corresponding to the one imaging range; An observation system characterized in that, when the position of the person being observed falls within a predetermined area in the captured image, the imaging unit is not rotated, and the imaging range of the imaging unit is maintained at the one imaging range, and observation processing of the person being observed is performed based on the captured image.

9. The observation system described in claim 8, characterized in that when the position of the person being observed no longer falls within the specified area, the control unit rotates the imaging unit to change the shooting range of the imaging unit to a shooting range other than the one shooting range.

10. 10. The observation system according to claim 8, wherein the observation process of the observed person includes a process of detecting a behavior-related event of the observed person.

11. a computer for controlling an observation system including an imaging unit that captures an image of an observation target space and a drive unit that rotationally drives the imaging unit to change the imaging range of the imaging unit; a) controlling the rotational driving of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of a person to be observed within a captured image corresponding to the one capturing range; c) when the position of the person to be observed is within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit; A program to execute.

12. A control method for an observation system including an imaging unit that captures an image of an observation target space and a drive unit that rotationally drives the imaging unit to change an imaging range of the imaging unit, a) controlling the rotational driving of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of a person to be observed within a captured image corresponding to the one capturing range; c) when the position of the person to be observed is within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit; A control method for an observation system, comprising:

Images