Image Processing Method

Abstract

To detect a moving body accurately from a range image.SOLUTION: An image processing apparatus 10 includes: a follow-up processing unit 12 that detects an object which is determined to be identical with a moving body to be followed, on the basis of a preset follow-up criterion, on a range image, and executes processing to follow the detected object as an object to be followed. When failing to follow the moving body to be followed, the follow-up processing unit 12 executes processing to modify the follow-up criterion to detect an object from the range image and follow the detected object as a moving body to be followed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method and an image processing device for performing processing to detect a moving object from a distance image. [Background technology]

[0002] In recent years, the elderly population has increased, and the demand for nursing care has risen. On the other hand, nursing care requires many caregivers, which causes problems such as an increase in the caregiver's workload and labor costs. Therefore, many monitoring systems have been developed to monitor people such as care recipients using distance images captured by a distance image sensor, as described in Patent Document 1.

[0003] In the surveillance system disclosed in Patent Document 1, the position of the bed is extracted from a distance image, and the area of ​​the person is detected to determine the person's movement. Specifically, the position of the bed is obtained from the frequency distribution of the height in the longitudinal and lateral directions of the bed. Furthermore, the movement of the person is determined by detecting the position of an object relative to a predetermined height position, and the person's movement, such as lying down, sitting up, or standing up, is determined. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-030042 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, as described above, when detecting a person's motion from a distance image, each part of the person, such as the head, torso, and legs, is detected and the positional relationship of each part is identified. Therefore, if the orientation or position of the person changes, the size of each part of the person also changes, making it impossible to detect each part with high accuracy, and it becomes difficult to detect the motion of the person with high accuracy. In addition, the same problem occurs when detecting the motion of any moving object, not limited to a person.

[0006] Therefore, an object of the present invention is to solve the above-mentioned problem that it is difficult to detect the movement of a moving object with high accuracy from a distance image. [Means for solving the problem]

[0007] An image processing method according to one aspect of the present invention includes: Detecting an object that is determined to be the same as the moving object to be tracked based on a preset tracking criterion from the range image, and performing a process of tracking the detected object as the moving object to be tracked; When the tracking of the moving object to be tracked has failed, the tracking criterion is changed, an object is detected from the distance image, and the detected object is tracked as the moving object to be tracked. The structure is as follows.

[0008] In addition, in the image processing method, detecting an object that is determined to be substantially the same size as the moving object to be tracked based on the tracking criterion from the distance image, and tracking the detected object as the moving object to be tracked; When the tracking of the moving object to be tracked fails, the tracking criterion is changed, an object smaller than the moving object to be tracked is detected from the distance image, and the detected object is tracked as the moving object to be tracked. The structure is as follows.

[0009] In addition, in the image processing method, detecting an object determined to be substantially the same size as each part of the moving body to be tracked based on the tracking criterion from the distance image, and tracking the detected object as each part of the moving body to be tracked; when tracking of the moving object has failed, the tracking criterion is changed, an object smaller than the size of each part of the moving object to be tracked is detected from the distance image, and the detected object is tracked as each part of the moving object to be tracked. The structure is as follows.

[0010] In addition, in the image processing method, detecting an object that is determined to be located at approximately the same height as the moving object to be tracked based on the tracking criterion from the distance image, and performing a process of tracking the detected object as the moving object to be tracked; when the tracking of the moving object to be tracked has failed, the tracking criterion is changed, an object located at a height position lower than the moving object to be tracked is detected from the distance image, and the detected object is tracked as the moving object to be tracked. The structure is as follows.

[0011] In addition, in the image processing method, detecting an object that is determined to be located at substantially the same height as each part of the moving body to be tracked based on the tracking criterion from the distance image, and tracking the detected object as each part of the moving body to be tracked; when the tracking of the moving object to be tracked fails, the tracking criterion is changed, and an object located at a position lower than the height position of each part of the moving object to be tracked is detected from the distance image, and the detected object is tracked as each part of the moving object to be tracked. The structure is as follows.

[0012] In addition, in the image processing method, detecting an object determined to have substantially the same shape as each part of the moving object to be tracked based on the tracking criterion from the distance image, and tracking the detected object as each part of the moving object to be tracked; Even if tracking of the moving object to be tracked fails, an object that is determined to have substantially the same shape as each part of the moving object to be tracked is detected, and the detected object is tracked as each part of the moving object to be tracked. The structure is as follows.

[0013] Further, an image processing device according to one aspect of the present invention comprises: a tracking processing unit that detects an object that is determined to be the same as the moving object to be tracked based on a preset tracking criterion from the distance image, and performs processing to track the detected object as the moving object to be tracked; the tracking processing unit, when failing to track a moving object to be tracked, performs a process of changing the tracking criterion, detecting an object from a distance image, and tracking the detected object as the moving object to be tracked. The structure is as follows.

[0014] In addition, in the image processing device, The tracking processing unit is detecting an object that is determined to be substantially the same size as the moving object to be tracked based on the tracking criterion from the distance image, and tracking the detected object as the moving object to be tracked; When the tracking of the moving object to be tracked fails, the tracking criterion is changed, an object smaller than the moving object to be tracked is detected from the distance image, and the detected object is tracked as the moving object to be tracked. The structure is as follows.

[0015] In addition, in the image processing device, The tracking processing unit is detecting an object that is determined to be located at approximately the same height as the moving object to be tracked based on the tracking criterion from the distance image, and performing a process of tracking the detected object as the moving object to be tracked; when the tracking of the moving object to be tracked has failed, the tracking criterion is changed, an object located at a height position lower than the moving object to be tracked is detected from the distance image, and the detected object is tracked as the moving object to be tracked. The structure is as follows.

[0016] In addition, a program according to one aspect of the present invention includes: In the information processing device, Detecting an object that is determined to be the same as the moving object to be tracked based on a preset tracking criterion from the range image, and performing a process of tracking the detected object as the moving object to be tracked; When the tracking of the moving object to be tracked has failed, the tracking criterion is changed, an object is detected from the distance image, and the detected object is tracked as the moving object to be tracked. Execute the process, The structure is as follows. Effect of the Invention

[0017] With the above-described configuration, the present invention can detect a moving object from a range image with higher accuracy. [Brief description of the drawings]

[0018] [Figure 1] 1 is a schematic diagram showing a configuration of an information processing system according to a first embodiment of the present invention. [Diagram 2] 2 is a block diagram showing a configuration of the monitoring server disclosed in FIG. 1. [Diagram 3] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1; [Figure 4] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1; [Diagram 5] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1; [Figure 6] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1; [Figure 7] 2 is a flowchart showing a processing operation of the monitoring server disclosed in FIG. 1; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Embodiment 1> A first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 7. Fig. 1 and Fig. 2 are diagrams showing the configuration of an information processing system. Fig. 3 to Fig. 7 are diagrams for explaining the processing operation of the information processing system. Below, the configuration and operation of the information processing system will be described with reference to each diagram.

[0020] The information processing system of the present invention is an information processing system for monitoring the behavior of a person P who is a monitoring target, such as a care recipient or a monitored person. The information processing system is configured to include a monitoring server 10, a distance image camera C, a monitoring terminal 30, and a mobile terminal 40.

[0021] The distance image camera C (imaging device) is disposed in a position capable of capturing distance images within a room R1 (first area), which is an area in which a bed B on which a person P to be monitored is lying is disposed. For example, the distance image camera C is provided on the ceiling of a room R1 in a medical facility, a care facility, a house, or the like, and is disposed in a position where most of the room R1 fits within the image. Note that the distance image camera C is not necessarily limited to being installed on the ceiling, and may be installed anywhere, such as on a wall or a stand. However, the bed B installed in the room R1 may be any type of sleeping surface, such as a futon.

[0022] The distance imaging camera C captures distance images with pixel values ​​as distance values. For example, the distance imaging camera C may be of a type that measures distance from the time it takes for an infrared laser to travel back and forth when projected onto a target, and may be any type of imaging device. The distance imaging camera C has a function of capturing distance images at regular time intervals or when an instruction to capture images is received, and transmitting the captured distance images to a monitoring server 10, which will be described later. The function of the monitoring server 10, which will be described later, may also be incorporated in the distance imaging camera C. In other words, the function of the monitoring server 10 may be realized by an information processing device incorporated in the distance imaging camera C.

[0023] The monitoring terminal 30 (monitoring device) is an information processing terminal operated by a caregiver who cares for the person P who is being cared for, or a user U who is a supervisor who monitors the person P. The monitoring terminal 30 is installed in a place (second area) away from the room R1 (first area) of the person P, such as a monitoring room R2 where the user U, such as a caregiver, is present, and can be operated by multiple users U. The monitoring terminal 30 displays a distance image on the display screen, enables input of operation instructions related to monitoring, and has a function of transmitting such operation instructions to the monitoring server 10 and receiving a notification of the status of the person P issued from the monitoring server 10 and notifying the users U.

[0024] The mobile terminal 40 (monitoring device) is an information processing terminal such as a smartphone held and operated by the user U. The mobile terminal 40 is capable of communicating with the monitoring server 10 via wireless communication, and is located in a place (second area) away from the room R1 (first area) of the person P, such as a monitoring room R2 where the user U, such as a caregiver, is present or in the rooms of other care recipients, and can be operated by multiple users U. The mobile terminal 40 has a function of displaying a distance image on a display screen, enabling input of operation instructions related to monitoring, transmitting such operation instructions to the monitoring server 10, and receiving a notification of the status of the person P issued from the monitoring server 10 and notifying the user U.

[0025] The monitoring server 10 mainly has a function of detecting a specific motion state of the person P in the room of the person P by using a distance image from the distance image camera C. The monitoring server 10 is composed of one or more information processing devices having a calculation device and a storage device. Specifically, as shown in FIG. 2, the monitoring server 10 includes an image acquisition unit 11, a motion detection unit 12, and a notification unit 13, which are constructed by the calculation device executing a program. The monitoring server 10 also includes a criterion storage unit 16 formed in the storage device. Each component will be described in detail below, and the operation of the monitoring server 10 will also be described with reference to the flowchart of FIG. 7. The function of the monitoring server 10 can also be mounted on the distance image camera C. That is, the function of the monitoring server 10 may be realized by an information processing device mounted on the distance image camera C.

[0026] The image acquisition unit 11 acquires a distance image of the inside of the room R1 including the bed B, which has been photographed by the distance image camera C as described above, from the distance image camera C. The acquired distance image is then passed to the motion detection unit 12, where it is used for processing described later. Note that the image acquisition unit 11 may always transmit the acquired distance image as is to the monitoring terminal 30 or the mobile terminal 40, and output it to be displayed on the display device of the monitoring terminal 30 or the mobile terminal 40.

[0027] The motion detection unit 12 (tracking processing unit) performs a process of detecting the person P in the distance image. Specifically, the motion detection unit 12 first detects, as a background portion, a portion where the difference between distance images obtained in advance and backward in time is smaller than a predetermined value (step S1 in FIG. 7). For example, in a distance image of the room R1, a portion where a fixed object such as a floor surface, a wall surface, or a bed B is reflected is detected as a background portion. At this time, the motion detection unit 12 detects a bed area, which is an area of ​​the bed B (a sleeping place), from the detected background portion. For example, the motion detection unit 12 detects a plane extending horizontally with a predetermined width at a predetermined height position from the distance image, and detects, from the plane, a plane area corresponding to the bed information stored in advance as the bed area. For example, the bed information is stored in the determination criterion storage unit 16, and the bed information includes information such as the width of the bed, the lengths of the long and short sides of the rectangular outer shape, and the height from the floor surface F. Then, the motion detection unit 12 detects, in particular, an edge portion that is an edge that forms the outer shape of the bed area. For example, the motion detection unit 12 detects, as an edge portion, a linear portion located near the outer shape of the bed B where the distance values ​​between adjacent pixels change abruptly.

[0028] Then, the motion detection unit 12 calculates the difference in distance between each pixel of the newly captured distance image and the background, and detects the presence of a moving object (moving body) in a location where the difference is greater than a predetermined value (step S2 in FIG. 7). The motion detection unit 12 further detects a person P based on the shape of the detected object, and tracks the person P in distance images captured thereafter (step S3 in FIG. 7). For example, the motion detection unit 12 detects a person P consisting of multiple body parts, such as a pair of a head and a torso, based on the shape of the detected object (e.g., aspect ratio), the positional relationship between the objects, and the volume ratio (area ratio). Then, for example, the motion detection unit 12 tracks parts of the person P (such as the head and torso) that can be determined to be the same by pattern matching the shapes of the body parts in the new distance image, or by using the optical flow of the feature points of the parts. At this time, the motion detection unit 12 detects an object existing near the position of the person P detected immediately before in the new distance image, and tracks the object using pattern matching or the like.

[0029] Here, a specific example of the tracking process by the motion detection unit 12 in this embodiment will be described with reference to Fig. 3 to Fig. 6. Note that in the examples of Fig. 3 to Fig. 6, the motion detection unit 12 will be described taking as an example the process of tracking the head of a person P. However, the motion detection unit 12 detects each part of the person P, such as the head and torso, and tracks the combination of these parts as the person P to be tracked.

[0030] First, the motion detection unit 12 sets a detection area A for a new distance image as shown in Figs. 3(1) and (2), and detects an object that can be determined to be the head of the person P to be tracked within the detection area A. The detection area A is a surrounding area of ​​the position where the head of the person P to be tracked is tracked in the previous distance image as shown in Fig. 3(1), and is a region of a preset size centered on the head position. Then, the motion detection unit 12 detects an object of the same shape and size as the head of the person P to be tracked within the detection area A. For example, the motion detection unit 12 prepares a rectangular template T that is approximately the same shape and size as the head of the person P tracked in the previous distance image shown in Fig. 3(1), and detects an object that matches the template T in the subsequent distance image shown in Fig. 3(2). At this time, the motion detection unit 12 detects objects that differ in size within a preset allowable range for the template T as objects of approximately the same size. For example, even if an object has a different length within an allowable range in both the vertical and horizontal directions of the template T, it is detected as an object of approximately the same size. Moreover, the motion detection unit 12 detects objects of approximately the same shape by rotating the orientation of the template T at all angles to detect objects of approximately the same size. In this way, in the example of Fig. 3(2), the motion detection unit 12 tracks the object detected in the new distance image as the head of the person P. Note that the above-mentioned method of tracking the body parts of the person P is set in advance as a tracking standard.

[0031] On the other hand, if person P shown in Fig. 4(1) suddenly moves from a standing state to a crouching state shown in Fig. 4(2), the distance from the distance image camera C to the head becomes longer and the orientation of the head relative to the distance image camera C changes, causing the size of the head in the distance image to become smaller. As a result, the size of the head in the distance image is not determined to be substantially the same as that of template T, and tracking of the head fails (Yes in step S4 in Fig. 7). In this case, the motion detection unit 12 changes the tracking criterion, in particular the size of template T (step S11 in Fig. 7), and tracks the head again (step S3 in Fig. 7).

[0032] For example, the motion detection unit 12 reduces the size of the template T and detects an object that is determined to be approximately the same size as the modified template T'. In other words, when the motion detection unit 12 fails to track the head, it detects an object that is smaller than the head that has been tracked up to now. As an example, the motion detection unit 12 reduces the length and width of the rectangular template T to 2 / 3 or 1 / 2, respectively, without changing the ratio, and reduces the size of the template T'. Note that the template T before the change and the template T' after the change can be said to have approximately the same shape by not changing the aspect ratio. However, the shape of the template T' may be changed from the template T before the change. By reducing the size of the template T' in this way, the motion detection unit 12 can detect an object that is the head of the person P even if the size of the head of the person P suddenly becomes smaller as shown in FIG. 5(2) and can track it as the head of the person P (No in step S4 of FIG. 7). Note that the motion detection unit 12 may change the size of the template T multiple times so that it becomes smaller in stages, and detect an object using each changed template T'.

[0033] On the other hand, if the person P shown in FIG. 6(1) moves from a standing state to a state where the person P shown in FIG. 6(2) falls to the floor, the position of the head in the distance image will be out of the detection area A. Then, the motion detection unit 12 will fail to track the head (Yes in step S4 of FIG. 7). In this case, the motion detection unit 12 changes the tracking criterion and tracks the head again, particularly at a position lower than the detection area A in the height direction in the distance image (step S11 of FIG. 7). That is, the motion detection unit 12 normally detects an object located in the detection area A as an object located at approximately the same height as the head position detected immediately before, but if tracking fails, it will detect an object located at a position lower than the detection area A (step S3 of FIG. 7).

[0034] For example, the motion detection unit 12 detects an object that is determined to be approximately the same size as the template T, targeting the entire range below the bottom end of the detection area A. At this time, the motion detection unit 12 may perform the detection by rotating the template T, as described above, or may perform the detection by reducing the size of the template T. In this way, the motion detection unit 12 can detect an object at a lower position than the head position that was tracked immediately before, and can detect the object that is the head and track it as the head of person P even if the head position of person P suddenly moves to a lower position as shown in FIG. 6(2) (No in step S4 of FIG. 7).

[0035] In this manner, when the motion detection unit 12 has succeeded in tracking a body part such as the head of the person P (No in step S4 of FIG. 7), it detects the motion state of the person P that has been tracked (step S5 of FIG. 7) and judges whether the motion state is a specific motion state (step S6 of FIG. 7). For example, the motion detection unit 12 detects the motion state of the person P based on the positional relationship of the body parts (head, torso, etc.) of the person P that have been tracked. As an example, the motion detection unit 12 detects the motion state of the person P based on the positional relationship between the height of the person P that can be detected from the head, the center of gravity position that can be detected from the torso, and the bed B in the distance image, and judges whether the motion state is a specific motion state that has been set in advance. For example, when the person P is lying on the bed B (supine state), it only recognizes that the person P is in such a state (lying on the bed B), and does not judge that the motion state is a specific motion state (No in step S6 of FIG. 7). On the other hand, when the person P is located at the boundary of the bed B, when the person P is crouching away from the bed B, or when the person P is lying down away from the bed B, the motion detection unit 12 determines that the motion state of the person P is a specific motion state that may be determined as abnormal (Yes in step S6 of FIG. 7). Note that the criteria for determining that the motion state of the person P is a specific motion state based on the height, center of gravity, positional relationship with the bed B, etc., as described above, are stored in advance in the determination criteria storage unit 16.

[0036] When the motion detection unit 12 determines that the person P is in a specific motion state (Yes in step S6 in FIG. 7), the notification unit 13 notifies the user U that the person P may be in an abnormal state (step S7 in FIG. 7). For example, the notification unit 13 transmits notification information to the monitoring terminal 30 or the mobile terminal 40 to notify that the person P may be in an abnormal state. However, the specific motion state determined in the present invention is not limited to the above-mentioned states, and other motion states may be determined as the specific motion state. For example, a state in which the person P stands on the bed B (standing state) and a state in which the person P sits on the bed B (sitting up state) may be determined as the specific motion state that may be determined as abnormal.

[0037] As described above, according to the present invention, even if the size of a part of person P being followed suddenly changes, the size of the part to be followed is reduced, the height of the tracking position is lowered, etc., and the tracking criterion for the part to be followed is changed to track that part. This makes it possible to detect a moving object more accurately from a distance image.

[0038] The above-mentioned program is stored in a storage device or a computer-readable recording medium. For example, the recording medium is a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0039] Although the present invention has been described with reference to the above-mentioned embodiment, the present invention is not limited to the above-mentioned embodiment. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0040] 10 Monitoring Server 11 Image acquisition section 12 Motion detection unit 13 Notification Department 16 Judgment criteria storage section 30 Monitoring terminal 40 Mobile Devices B Bed C. Range imaging camera P person U User

Claims

1. Detecting an object that is determined to be the same as the moving object to be tracked and located at approximately the same height position as the moving object to be tracked based on a preset tracking criterion from the distance image, and performing a process of tracking the detected object as the moving object to be tracked; when the tracking of the moving body to be tracked has failed, the tracking criterion is changed, an object located at a position lower than the approximately same height position is detected from a distance image, and the detected object is tracked as the moving body to be tracked. Image processing methods.

2. The image processing method according to claim 1, when failing to track the moving body to be tracked, changing the tracking criterion, detecting an object located at a position lower than the lower end of the substantially same height position set in an area of ​​a predetermined size from a distance image, and performing a process of tracking the detected object as the moving body to be tracked; Image processing methods.

3. The image processing method according to claim 1, when failing to track the moving body to be tracked, changing the tracking criterion, detecting from the distance image all objects located in a range lower than the lower end of the substantially same height position set in an area of ​​a predetermined size, and performing a process of tracking the detected objects as the moving body to be tracked; Image processing methods.

4. An image processing method according to any one of claims 1 to 3, The moving object to be tracked is the head of a person. Image processing methods.

5. a tracking processing unit that detects an object that is determined to be the same as the moving object to be tracked and located at approximately the same height position as the moving object to be tracked based on a preset tracking criterion in the distance image, and performs processing to track the detected object as the moving object to be tracked; when the tracking processing unit fails to track the moving body to be tracked, the tracking processing unit changes the tracking criterion, detects an object located at a position lower than the approximately same height position from a distance image, and performs a process of tracking the detected object as the moving body to be tracked. Image processing device.

6. An image processing device according to claim 5, when the tracking processing unit fails to track the moving body to be tracked, the tracking processing unit changes the tracking criterion, detects an object located at a position lower than the lower end of the substantially same height position set in an area of ​​a predetermined size from a distance image, and performs a process of tracking the detected object as the moving body to be tracked. Image processing device.

7. In the information processing device, Detecting an object that is determined to be the same as the moving object to be tracked and located at approximately the same height position as the moving object to be tracked based on a preset tracking criterion from the distance image, and performing a process of tracking the detected object as the moving object to be tracked; when the tracking of the moving body to be tracked has failed, the tracking criterion is changed, an object located at a position lower than the approximately same height position is detected from a distance image, and the detected object is tracked as the moving body to be tracked. A program for executing a process.

Images