Image Processing Method

Abstract

To follow a mobile body accurately from a distance image.SOLUTION: An image processor 10 of the present invention detects a mobile body in a distance image, provides a label to the mobile body, and relates single distance image information as distance image information of the mobile body to the label. When the image processor determines that a predetermined mobile body is combined with another mobile body on the basis of the distance image, the distance image information in a combination state of the combined mobile body is newly related as combination distance image information to the label of the predetermined mobile body while the single distance image information is being related to the label of the mobile body.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing method and an image processing device for detecting a moving object from a distance image. [Background technology]

[0002] In recent years, the elderly population has been increasing, and the demand for nursing care has been rising. On the other hand, nursing care requires many caregivers, which causes problems such as increased workload for caregivers and increased labor costs. To address this issue, a monitoring system has been developed that monitors people such as care recipients using distance images captured by a distance image sensor, as described in Patent Document 1.

[0003] The monitoring system disclosed in Patent Document 1 extracts the position of a bed from a distance image, detects the area of ​​a person, and determines the person's movements. Specifically, the position of the bed is determined from the frequency distribution of the height in the longitudinal and lateral directions of the bed. Furthermore, the person's movements, such as lying down, sitting up, and standing up, are determined by detecting the position of an object relative to a predetermined height position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-030042 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, when detecting the movement of a person from a distance image as described above, it is necessary to track the person itself as a moving object. However, when tracking a moving object such as a person, multiple people may overlap and separate, causing the moving objects being tracked to repeatedly combine and separate. When the moving objects repeatedly combine and separate in this way, it becomes difficult to track the moving objects, and it becomes difficult to accurately detect the movement of the moving objects.

[0006] Therefore, an object of the present invention is to solve the above-mentioned problem that it is difficult to track a moving object from a distance image, and therefore it is not possible to accurately detect the movement of the moving object. [Means for solving the problem]

[0007] An image processing method according to one aspect of the present invention includes: Detecting a moving object in the range image and assigning a label to it, and associating the label with independent range image information that is range image information of the moving object; When it is determined based on the distance image that a predetermined moving object has been combined with another moving object, distance image information in a combined state of the combined moving object is newly associated as combined distance image information with the label of the predetermined moving object while the individual distance image information is still associated with the label of the predetermined moving object. The structure is as follows.

[0008] In addition, in the image processing method, When it is determined that a plurality of moving bodies have been combined, the same combined distance image information in the combined state of the combined moving bodies is newly associated with the label of each of the combined moving bodies. The structure is as follows.

[0009] In addition, in the image processing method, When it is determined that a plurality of moving bodies have been combined, the individual range image information associated with the label of each of the combined moving bodies is updated to the range image information of each moving body at the time of combination. The structure is as follows.

[0010] In addition, in the image processing method, When it is determined that a plurality of moving bodies have been combined, and when the combined distance image information is associated with the label of the combined moving bodies, the combined distance image information is updated to the same distance image information in the combined state of the combined moving bodies. The structure is as follows.

[0011] In addition, in the image processing method, When it is determined that another moving object has joined a joined moving object formed by joining multiple moving objects, the joined distance image information associated with the labels of all moving objects included in the joined moving object is updated to the same distance image information in the joined state of the joined moving objects. The structure is as follows.

[0012] In addition, in the image processing method, When it is determined based on the distance image that the moving object associated with the combined distance image information has separated into a plurality of moving objects, a single moving object belonging to the separated moving object is selected based on the single distance image information associated with the moving object before separation and the distance image information of the moving object after separation. The structure is as follows.

[0013] In addition, in the image processing method, If the separated moving body is a combined moving body consisting of a plurality of independent moving bodies, the combined distance image information associated with the labels of the independent moving bodies belonging to the separated moving body is updated to the distance image information of the combined moving body after separation. The structure is as follows.

[0014] In addition, in the image processing method, If the single distance image information associated with the moving body before separation corresponds to the distance image information of one moving body after separation, associate the label of the moving body before separation and the single distance image information with the one moving body, and discard the combined distance image information associated with the label. The structure is as follows.

[0015] In addition, in the image processing method, When it is determined based on the distance image that a moving body that is not associated with the combined distance image information has separated into multiple moving bodies, if the individual distance image information associated with the moving body before separation corresponds to the distance image information of one moving body after separation, the label of the moving body before separation and the individual distance image information are associated with the one moving body, and a new label is assigned to the other moving body after separation, and the distance image information of the other moving body is associated with the label as the new individual distance image information. The structure is as follows.

[0016] In addition, in the image processing method, the distance image information of the moving object includes at least one piece of information on the center of gravity position, height, and volume of the moving object that can be detected from the distance image; The structure is as follows.

[0017] Furthermore, an image processing device according to one aspect of the present invention includes: means for detecting a moving object in a range image and assigning a label to the detected moving object, and for associating the label with individual range image information that is range image information of the moving object; a means for associating, when it is determined based on the distance image that a predetermined moving object has been combined with another moving object, distance image information in the combined state of the combined moving object as combined distance image information, while keeping the independent distance image information associated with the label of the predetermined moving object; Equipped with The structure is as follows.

[0018] Furthermore, a program according to one aspect of the present invention includes: In the information processing device, Detecting a moving object in the range image and assigning a label to it, and associating the label with independent range image information that is range image information of the moving object; When it is determined based on the distance image that a predetermined moving object has been combined with another moving object, distance image information in a combined state of the combined moving object is newly associated as combined distance image information with the label of the predetermined moving object while the individual distance image information is still associated with the label of the predetermined moving object. Execute the process, The structure is as follows. [Effects of the Invention]

[0019] With the above-described configuration, the present invention makes it easy to track a moving object from a distance image, and can accurately detect the movement of the moving object. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a schematic diagram illustrating a configuration of an information processing system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the configuration of the monitoring server disclosed in FIG. 1. [Figure 3] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 4] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 5] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 6] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 7] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 8] 2 is a diagram showing how a distance image is processed by the monitoring server disclosed in FIG. 1. FIG. [Figure 9] 2 is a flowchart showing the processing operation of the monitoring server disclosed in FIG. 1; [Figure 10] 2 is a flowchart showing the processing operation of the monitoring server disclosed in FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0021] <Embodiment 1> A first embodiment of the present invention will be described with reference to Figs. 1 to 10. Figs. 1 and 2 are diagrams showing the configuration of an information processing system. Figs. 3 to 10 are diagrams for explaining the processing operation of the information processing system. The configuration and operation of the information processing system will be described below with reference to each diagram.

[0022] The information processing system of the present invention is an information processing system for monitoring the behavior of a person P who is a monitoring target, such as a care recipient or a monitored person. The information processing system is configured to include a monitoring server 10, a distance imaging camera V, a monitoring terminal 30, and a mobile terminal 40.

[0023] The distance imaging camera V (photography device) is placed in a position where it can capture distance images within a room R1 (first area), which is an area where a bed W on which a person P to be monitored is lying is located. For example, the distance imaging camera V is installed on the ceiling of a room R1 in a medical facility, nursing home, or home, and is placed in a position where most of the room R1 fits within the image. Note that the distance imaging camera V does not necessarily have to be installed on the ceiling, and may be installed anywhere, such as on a wall or a stand. However, the bed W installed in the room R1 may be any type of bed, such as a futon.

[0024] The distance imaging camera V captures distance images in which pixel values ​​represent distance values. For example, the distance imaging camera V may be of a type that measures distance from the round-trip time when an infrared laser is projected onto a target, or may be any type of imaging device. The distance imaging camera V has the function of capturing distance images at regular time intervals or when a shooting instruction is received, and transmitting these distance images to a monitoring server 10, which will be described later. The functions of the monitoring server 10, which will be described later, can also be incorporated into the distance imaging camera V. In other words, the functions of the monitoring server 10 may be realized by an information processing device incorporated in the distance imaging camera V.

[0025] The monitoring terminal 30 (monitoring device) is an information processing terminal operated by a caregiver who provides care for person P, who is a care recipient, or by a user U who is a supervisor who monitors person P. The monitoring terminal 30 is installed in a location (second area) separate from person P's room R1 (first area), such as a monitoring room R2 where user U, such as a caregiver, is present, and can be operated by multiple users U. The monitoring terminal 30 displays a distance image on its display screen, enables input of operation instructions related to monitoring, and has the function of transmitting such operation instructions to the monitoring server 10 and receiving notifications of person P's status issued by the monitoring server 10 and notifying users U.

[0026] The mobile terminal 40 (monitoring device) is an information processing terminal such as a smartphone held and operated by the user U. The mobile terminal 40 is capable of communicating with the monitoring server 10 via wireless communication, is located in a location (second area) separate from the person P's room R1 (first area), such as a monitoring room R2 where the user U, such as a caregiver, is present, or the rooms of other care recipients, and can be operated by multiple users U. The mobile terminal 40 displays a distance image on its display screen, enables input of operation instructions related to monitoring, and has the function of transmitting such operation instructions to the monitoring server 10 and receiving notifications of the status of the person P issued by the monitoring server 10 and notifying the users U.

[0027] The monitoring server 10 (image processing device) primarily has the function of detecting a specific motion state of a person P in the person P's room using a distance image captured by a distance imaging camera V. The monitoring server 10 is composed of one or more information processing devices each having a computing device and a storage device. Specifically, as shown in FIG. 2, the monitoring server 10 includes an image acquisition unit 11, a motion detection unit 12, and a notification unit 13, which are constructed by the computing device executing a program. The monitoring server 10 also includes a criterion storage unit 16 and a tracking information storage unit 17, which are formed in the storage device. Each component will be described in detail below. The processing of distance images by the monitoring server 10 will also be described with reference to FIGS. 3 to 8, and the operation of the monitoring server 10 will also be described with reference to the flowcharts in FIGS. 9 and 10. The functions of the monitoring server 10 can also be implemented in the distance imaging camera V. That is, the functions of the monitoring server 10 may be implemented by an information processing device installed in the distance imaging camera V.

[0028] The image acquisition unit 11 acquires a distance image of the interior of the room R1 including the bed W, which has been captured by the distance image camera V as described above, from the distance image camera V. The acquired distance image is then passed to the motion detection unit 12, where it is used for processing, which will be described later. Note that the image acquisition unit 11 may always transmit the acquired distance image as is to the monitoring terminal 30 or the mobile terminal 40, and output it to be displayed on the display device of the monitoring terminal 30 or the mobile terminal 40.

[0029] The motion detection unit 12 (tracking processing unit) performs a process to detect person P in the distance image. Specifically, the motion detection unit 12 first uses the difference between two distance images in succession to detect, as background areas, areas where the difference is smaller than a predetermined value (step S1 in FIG. 9). For example, in a distance image of room R1, areas that show fixed objects such as the floor, walls, and bed W are detected as background areas. At this time, the motion detection unit 12 detects a bed area, which is the area of ​​bed W (a sleeping surface), from the detected background area. For example, the motion detection unit 12 detects a plane extending horizontally at a predetermined height from the distance image, and detects, as the bed area, a plane area of ​​the plane that corresponds to pre-stored bed information. For example, the bed information is stored in the determination criterion storage unit 16, and the bed information includes information such as the size of the bed, the lengths of the long and short sides of its rectangular outline, and its height from the floor. The motion detection unit 12 then particularly detects the edges that form the outline of the bed area. For example, the motion detection unit 12 detects, as an edge portion, a linear portion located near the outer shape of the bed W where the distance values ​​between adjacent pixels change abruptly.

[0030] The motion detection unit 12 then calculates the distance difference between each pixel of the newly captured distance image and the background, and detects the presence of a moving object (moving body) where the difference is greater than a predetermined value (step S2 in FIG. 9). The motion detection unit 12 then detects person P based on the shape, size, etc. of the detected object, and tracks person P in distance images captured subsequently (step S3 in FIG. 9). For example, the motion detection unit 12 detects person P consisting of multiple body parts, such as a pair of head and torso, based on the shape (e.g., aspect ratio) of the detected object, the positional relationship between the objects, and the volume ratio (area ratio). The motion detection unit 12 then tracks body parts (e.g., head, torso) of person P that can be determined to be the same in the new distance image by, for example, pattern matching the shapes of the body parts or using the optical flow of the feature points of the parts. At this time, the motion detection unit 12 detects an object that exists near the position of the person P that was detected immediately before in the new distance image, and tracks the object using pattern matching or the like.

[0031] 3 to 8, a specific example of the tracking process by the motion detection unit 12 in this embodiment will be described. In the following, the motion detection unit 12 recognizes a combination of the head and torso of person P as a single moving object T, as indicated by the symbol T in FIG. 3, and tracks this moving object T as a tracking target. However, the motion detection unit 12 may also track only each part of person P, such as the head or torso, as the moving object T to be tracked. In this embodiment, a case will be described in particular in which multiple moving objects T are recognized as overlapping and combined into a single moving object T on a distance image, and a case in which a moving object T recognized as a single object on a distance image is recognized as being separated into multiple objects.

[0032] First, referring to FIG. 4, a case will be described in which two independent moving objects T are recognized as overlapping and combined into one moving object T. In the upper left diagram of FIG. 4, it is assumed that two moving objects, moving object A (a predetermined moving object) and moving object B (another moving object), are being tracked by the motion detection unit 12 (step 3 of FIG. 9). In this case, the motion detection unit 12 assigns different labels to each of the moving objects A and B and calculates distance image information based on the detected distance images of each of the moving objects A and B. For example, the distance image information of each of the moving objects A and B includes the center of gravity position, height, and volume of each of the moving objects A and B calculated based on the distance images extracted as parts of each of the moving objects A and B. However, the distance image information of each of the moving objects A and B may be at least one of the center of gravity position, height, and volume, or may be any value calculated from the distance images extracted as parts of each of the moving objects A and B. Then, each time the motion detection unit 12 tracks each of the moving objects A and B, it associates distance image information with a label and stores the associated distance image information in the tracking information storage unit 17. At this time, the distance image information associated with the label is stored as individual distance image information. In the example shown in the upper left diagram of FIG. 4, the label A and individual distance image information (center of gravity position A, height A, volume A) are associated and stored as the information of the moving object A, and the label B and individual distance image information (center of gravity position B, height B, volume B) are associated and stored as the information of the moving object B. Similarly, as shown in the lower left diagram of FIG. 4, the labels and individual distance image information of the moving objects C and D are also associated and stored.

[0033] Thereafter, moving object A and moving object B are tracked, and as shown in the upper right diagram of FIG. 4, the distance d between moving object A and moving object B decreases, and it is determined that moving object A and moving object B have combined to become a combined moving object E. At this time, the motion detection unit 12 determines whether the moving objects are separated or combined, for example, based on the extracted distance d between moving objects A and B (step S4 of FIG. 9). As an example, when the distance d between moving objects A and B decreases to 50 cm or less, the motion detection unit 12 determines that these moving objects A and B have combined into one ("combined" in step S4 of FIG. 9). Note that the motion detection unit 12 may use any determination criterion to determine whether the moving objects have combined, and the determination criterion is stored in the determination criterion storage unit 16.

[0034] Then, the motion detection unit 12 performs a process of associating new combined distance image information, which is the distance image information of the combined moving body E, with each of the moving bodies A and B that make up the combined moving body E. Specifically, the motion detection unit 12 first checks whether combined distance image information is associated with the labels of each of the moving bodies A and B that are combined into the combined moving body E. Since combined distance image information has not yet been associated with each of the moving bodies A and B, the motion detection unit 12 determines that each of the moving bodies A and B before combination is a single moving body (Yes in step S5 of Figure 9). Then, the motion detection unit 12 calculates distance image information based on the distance image of the combined moving body E in the combined state of the moving bodies A and B, and associates this distance image information as combined distance image information with each of the labels of each of the moving bodies A and B (step S6 of Figure 9). In the example shown in the upper right diagram of Figure 4, for each label A and B of each moving body A and B that make up the combined moving body E, the center of gravity position E, height E, and volume E are calculated from the distance image of the combined moving body E part as distance image information of the combined moving body E, and the same combined distance image information (center of gravity position E, height E, volume E) is associated with each label A and B and stored in the tracking information memory unit 17.

[0035] Furthermore, because each of the moving bodies A and B before being combined into the combined moving body E was a single moving body, the motion detection unit 12 updates the individual distance image information associated with each label A and B to the distance image information of each of the moving bodies A and B at the time of combination (step S7 in FIG. 9). That is, the motion detection unit 12 updates the content of the individual distance image information associated with each label A and B of each of the moving bodies A and B while maintaining the associated state. In the example shown in the upper right diagram in FIG. 4, when the distance image information of the moving body A at the time of combination is calculated as (center of gravity position A', height A', volume A'), the individual distance image information associated with label A is updated to this distance image information. Similarly, for moving body B, the individual distance image information is updated to the distance image information of the moving body B at the time of combination (center of gravity position B', height B', volume B'). Note that the motion detection unit 12 may update the distance image information of each of the moving bodies A and B immediately before combination as the individual distance image information.

[0036] In this way, when two independent moving bodies A and B combine into one combined moving body E as shown in the upper part of Figure 4, the distance image information (center of gravity position E, height E, volume E) of the combined moving body E is associated as new combined distance image information with each label A and B of each moving body A and B. In addition, the independent distance image information originally associated with each label A and B of each moving body A and B is updated to the distance image information of each moving body A and B at the time of combination. Note that when two independent moving bodies C and D shown in the lower part of Figure 4 combine into one combined moving body F, similarly, new combined distance image information is associated with each label C and D of each moving body C and D, and the originally associated independent distance image information is updated.

[0037] Next, with reference to FIG. 5, a case will be described in which two combined moving bodies E and F are recognized as being combined into one combined moving body G. In the left diagram of FIG. 5, it is assumed that two combined moving bodies, combined moving body E and combined moving body F, are being tracked by the motion detection unit 12 (step 3 of FIG. 9). In this case, each time the motion detection unit 12 tracks each combined moving body E and F, it calculates distance image information based on the distance image of each combined moving body E and F, and updates the combined distance image information associated with the labels A, B, C, and D of each individual moving body A, B, C, and D belonging to each combined moving body E and F to the distance image information calculated for the corresponding combined moving body E and F. For example, for combined moving body E, the motion detection unit 12 updates the combined distance image information (center of gravity position E, height E, volume E) associated with the labels A and B of each moving body A and B belonging to it to the distance image information of the combined moving body E each time the motion detection unit 12 tracks the combined moving body E.

[0038] Thereafter, as shown in the right diagram of Figure 5, the distance d between the combined mobile object E and the combined mobile object F becomes smaller, and it is determined that the combined mobile object E and the combined mobile object F have combined to form a new combined mobile object G. At this time, the motion detection unit 12 determines whether the mobile objects are separated or combined, for example, based on the extracted distance d between the combined mobile objects E and F (step S4 of Figure 9). Here, since the distance d between the combined mobile objects E and F has become smaller than the threshold, it is determined that they have combined into one ("combined" in step S4 of Figure 9).

[0039] Then, the motion detection unit 12 performs a process of updating the combined distance image information associated with each of the individual moving bodies A, B, C, and D that make up the combined moving body G. Specifically, the motion detection unit 12 first checks whether combined distance image information is associated with the labels of each of the individual moving bodies A, B, C, and D that are combined into the combined moving body G. Since combined distance image information is already associated with each of the individual moving bodies A, B, C, and D, the motion detection unit 12 determines that each of the combined moving bodies E and F before combining is a combined moving body (No in step S5 of Figure 9). Then, the motion detection unit 12 calculates distance image information based on the distance image of the combined moving body G in the combined state, and updates this distance image information as combined distance image information associated with each of the labels A, B, C, and D of each of the moving bodies A, B, C, and D (step S8 of Figure 9). In the example on the right side of Figure 5, the center of gravity position G, height G, and volume G are calculated from the distance image of the combined moving body G portion, and the combined distance image information associated with each label A, B, C, and D of each individual moving body A, B, C, and D that make up the combined moving body G is updated to the distance image information of the combined moving body G. In other words, the combined distance image information associated with each label A, B, C, and D (however, labels C and D are omitted from Figure 5) is updated to the same distance image information (center of gravity position E, height E, volume E). Note that here, the individual distance image information associated with each label A, B, C, and D of each individual moving body A, B, C, and D is not updated.

[0040] In this way, even if the individual moving bodies A, B, C, and D repeatedly combine, the motion detection unit 12 can distinguish between the individual moving bodies and the combined moving bodies and track them simply by associating and updating the distance image information of the combined moving body to which each of the individual moving bodies A, B, C, and D belongs with each label of the individual moving bodies A, B, C, and D.

[0041] Although the above example illustrates a case where two moving objects are combined, the same processing as described above can also be applied to a case where three or more moving objects are combined at the same time. For example, when three moving objects are combined, the labels of all the individual moving objects belonging to the combined moving object can be newly associated with or updated with the range image information of the combined moving object.

[0042] Next, with reference to Figures 6 and 7, a case will be described in which a combined moving object, which has been recognized as a single moving object as described above, is recognized as having separated into two moving objects again. In the left diagram of Figure 6, it is assumed that one moving object, the combined moving object G, is being tracked by the motion detection unit 12 (step 3 of Figure 9). At this time, the combined moving object G is composed of four individual moving objects A, B, C, and D, and distance image information (center of gravity position G, height G, volume G) of the combined moving object G is associated with each of the labels A, B, C, and D of the individual moving objects A, B, C, and D. Note that the individual distance image information of the moving object itself is also associated with each of the labels A, B, C, and D of the individual moving objects A, B, C, and D.

[0043] Thereafter, as shown in the right diagram of FIG. 6, it is assumed that it is determined that the joined moving object G has separated into moving object H and moving object I, which are separated by a distance d from each other. For example, the motion detection unit 12 determines whether the moving objects are separated or joined based on the distance d between the moving objects H and I, which are detected as separate objects (step S4 of FIG. 9). As an example, if the distance d between the moving objects H and I exceeds 50 cm, the motion detection unit 12 determines that the joined moving object G has separated into two moving objects H and I ("Separated" in step S4 of FIG. 9). However, the motion detection unit 12 may determine whether the moving objects have separated based on any determination criterion, and the determination criterion is stored in the determination criterion storage unit 16.

[0044] The process then proceeds to the separation process shown in the flowchart of Fig. 10. The motion detection unit 12 calculates distance image information based on the distance images of each of the separated moving bodies H and I. In this case, the distance image information (center of gravity position H, height H, volume H) of the moving body H and the distance image information (center of gravity position I, height I, volume I) of the moving body I are calculated. The motion detection unit 12 then matches the distance image information of each of the separated moving bodies H and I with the individual distance image information of each of the individual moving bodies A, B, C, and D that make up the combined distance image G before separation (step S11 of Fig. 10). Specifically, the motion detection unit 12 compares the individual distance image information of each of the individual moving bodies A, B, C, and D with the distance image information of each of the separated moving bodies H and I to determine whether they correspond to each other. Furthermore, the motion detection unit 12 compares a combination of multiple independent moving objects A, B, C, and D with each of the moving objects H and I to determine whether the combination of multiple independent moving objects corresponds to each of the moving objects H and I. At this time, the motion detection unit 12 comprehensively compares the distance image information of each moving object, which includes the center of gravity position, height, and volume, to determine whether the moving objects correspond to each other. For example, the motion detection unit 12 comprehensively compares the independent distance image information of the independent moving object A (center of gravity position A', height A', volume A') with the distance image information calculated from the separated moving object H (center of gravity position H, height H, volume H) to determine whether they correspond to each other, i.e., whether they match. Alternatively, the motion detection unit 12 comprehensively compares the independent distance image information of the two independent moving objects A and B (center of gravity position A', height A', volume A') (center of gravity position B', height B', volume B') with the distance image information calculated from the separated moving object H (center of gravity position H, height H, volume H) to determine whether they correspond to each other. As an example, each element of the distance image information (center of gravity position, height, volume) is weighted, and the degree of match is scored. If the score exceeds a threshold, the two moving objects are determined to correspond to each other. In this way, individual moving objects A, B, C, and D belonging to each of the separated moving objects H and I are selected (step S12 in FIG. 10).

[0045] In the example of FIG. 6, it is assumed that, based on the comparison of the distance image information described above, the separated moving body H is determined to be a combined moving body to which the combination of the individual moving bodies A and B belongs, and the separated moving body I is determined to be a combined moving body to which the combination of the individual moving bodies C and D belongs (No in step S13 of FIG. 10). In this case, the motion detection unit 12 updates the calculated distance image information of each of the separated combined moving bodies H and I as combined distance image information associated with the labels of the individual moving bodies belonging to it (step S8 of FIG. 10). For example, for the combined moving body H, the combined distance image information associated with the labels A and B of the individual moving bodies A and B belonging to it is updated to the distance image information (center of gravity position H, height H, volume H) of the separated moving body H. Note that, although not shown, the combined distance image information associated with the labels C and D of the individual moving bodies C and D belonging to it is similarly updated to the distance image information (center of gravity position I, height I, volume I) of the separated moving body I.

[0046] Next, a case will be described in which the combined moving object H is separated into moving objects A and B, which are separated by a distance d, as shown in FIG. 7. The motion detection unit 12 calculates distance image information based on the distance images of the separated moving objects A and B, and matches this information with the individual distance image information of the individual moving objects A and B that constitute the combined distance image H before separation (step S11 in FIG. 10). Here, it is assumed that the individual moving objects A and B are selected to correspond to the separated moving objects A and B, respectively (step S12 in FIG. 10). Then, each of the separated moving objects A and B is an individual moving object (Yes in step S13 in FIG. 10), and is selected from the individual moving objects belonging to the moving object H before separation (Yes in step S14 in FIG. 10). In this case, the motion detection unit 12 associates the labels A and B associated with the individual moving objects A and B before separation and the individual distance image information with each of the separated individual moving objects A and B, and inherits them. At this time, the motion detection unit 12 calculates distance image information for each of the individual moving bodies A and B based on the distance image after separation, and updates this distance image information as individual distance image information for each moving body (step S15 in FIG. 10). Furthermore, the motion detection unit 12 discards the combined distance image information that was associated with the labels A and B of each of the individual moving bodies A and B before separation. As a result, for example, as shown in FIG. 7, the individual moving body A after separation inherits the label A and individual distance image information of the individual moving body A before separation, but the individual distance image information is updated to the distance image information (center of gravity position A, height A, volume A) of the individual moving body A after separation, and the combined distance image information is discarded.

[0047] In this way, as shown in Figure 7, the individual moving objects A and B separated from the combined moving object H can inherit the same labels A and B and distance image information as the original individual moving objects A and B. As a result, although they were once combined, they can continue to be tracked as the original individual moving objects A and B.

[0048] In the above, we have illustrated a case where the combined moving body H is formed by combining two individual moving bodies A and B, and these two individual moving bodies A and B are separated from the combined moving body H, but the same processing as described above can also be applied to a case where the combined moving body is formed by combining three or more moving bodies, and these three or more moving bodies are separated from the combined moving body. In that case, for example, each of the three or more individual moving bodies after separation will inherit the labels and individual distance image information of each individual moving body before separation.

[0049] Next, with reference to Figure 8, we will explain the case where one moving body A that is not a combined moving body separates into two moving bodies A and B. In the left diagram of Figure 8, it is assumed that one moving body A is being tracked by the motion detection unit 12 (step 3 of Figure 9). At this time, it is assumed that the label A and individual range image information (center of gravity position A, height A, volume A) of the moving body A are associated with the moving body A, but the combined range image information is not associated with the moving body A.

[0050] Thereafter, as shown in the right diagram of Fig. 8, it is assumed that it is determined that moving object A has separated into moving object A and moving object B that are separated by a distance d. For example, the motion detection unit 12 determines whether the moving objects are separated or joined based on the distance d between moving objects A and B that are detected as separate objects (step S4 of Fig. 9). As an example, if the distance d between moving objects A and B exceeds 50 cm, the motion detection unit 12 determines that moving object A has separated into two moving objects A and B ("Separated" in step S4 of Fig. 9).

[0051] The motion detection unit 12 then calculates distance image information based on the distance images of each of the separated moving bodies A and B, and matches it with the individual distance image information of the single moving body A before separation (step S11 in FIG. 10). Here, it is assumed that the separated moving body A is selected as the single moving body A before separation (Yes in step S13 and Yes in step S14 in FIG. 10). In this case, the motion detection unit 12 associates the label A and the individual distance image information associated with the single moving body A before separation with the single moving body A after separation, and inherits them, and further updates the individual distance image information (step S15 in FIG. 10). On the other hand, the separated moving body B is a single moving body because no combined distance image information is associated with the moving body before separation (Yes in step S13 in FIG. 10), but it has not been selected as the single moving body before separation (No in step S14 in FIG. 10), so it can be said to be a new single moving body. For this reason, the motion detection unit 12 assigns a new label B to the moving object B, and stores the distance image information (center of gravity position B, height B, volume B) of the moving object B at that time in association with the label B as new single distance image information.

[0052] In this way, as shown in Figure 8, one of the moving bodies A and B separated from moving body A can inherit the same label and individual range image information as moving body A before separation, and the other can be assigned a new label B and new individual range image information. As a result, it is possible to continue tracking one of the separated moving bodies A, and also to track the other moving body B.

[0053] While the above example illustrates a case where one moving object A separates into two moving objects A and B, the same processing as described above can also be applied when one moving object separates into three or more moving objects. In this case, for example, one moving object after separation will inherit the label and individual range image information of the moving object before separation, and each of the remaining two or more moving objects will be assigned a label different from the label of the moving object before separation, and different labels will also be assigned to each of the separated moving objects.

[0054] Thereafter, the motion detection unit 12 detects the motion state of the person P corresponding to each of the moving objects T that have been tracked, based on the positional relationship of the body parts (such as the head and torso) of the person P. As an example, the motion detection unit 12 detects the motion state of the person P based on the positional relationship between the height of the person P that can be detected from the head, the center of gravity position that can be detected from the torso, and the bed W in the distance image, and determines whether the motion state is a specific motion state that has been set in advance. At this time, the motion detection unit 12 determines that the motion state of the person P is a specific motion state that may be determined to be abnormal, for example, when the person P is located at the boundary of the bed W, when the person P is crouching away from the bed W, or when the person P is lying down away from the bed W. Note that, as described above, the criteria for determining that the motion state of the person P is a specific motion state based on the height, center of gravity position, positional relationship with the bed W, etc., of the person P are stored in advance in the determination criteria storage unit 16.

[0055] When the motion detection unit 12 determines that the person P is in a specific motion state as described above, the notification unit 13 notifies the user U that the person P may be in an abnormal state. For example, the notification unit 13 transmits notification information to the monitoring terminal 30 or the mobile terminal 40 to notify that the person P may be in an abnormal state. However, the specific motion state determined in the present invention is not limited to the above-mentioned states, and other motion states may also be determined as the specific motion state. For example, a state in which the person P is standing on the bed W (standing state) or a state in which the person P is sitting on the bed W (sitting up state) may be determined as the specific motion state that may be determined as abnormal.

[0056] As described above, according to the present invention, even if the moving objects being tracked merge or separate due to people overlapping or separating on the distance image, each person can be tracked appropriately, and as a result, the movement of the person after tracking can be detected with higher accuracy.

[0057] The above-mentioned program is stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0058] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0059] 10 Monitoring Server 11 Image acquisition unit 12 Motion detection unit 13 Notification Department 16 Judgment criteria storage section 17 Tracking information storage unit 30 Monitoring terminal 40 Mobile Devices P person T Mobile U User V Range Imaging Camera Double Bed

Claims

1. Detecting a moving object in the range image and assigning a label to it, and associating the label with independent range image information that is range image information of the moving object; When it is determined based on the distance image that a predetermined moving body has been combined with another moving body, distance image information in the combined state of the combined moving bodies is newly associated as combined distance image information while the single distance image information remains associated with the label of the predetermined moving body, and the same combined distance image information in the combined state of the combined moving bodies is newly associated with the labels of each of the combined moving bodies. Image processing methods.

2. 2. The image processing method according to claim 1, When it is determined that a combined moving body formed by combining a plurality of moving bodies has been combined with another combined moving body formed by combining a plurality of other moving bodies, the combined distance image information associated with the labels of all moving bodies included in the combined moving body is updated to the same distance image information in the combined state of the combined moving bodies. Image processing methods.

3. 3. The image processing method according to claim 2, When it is determined based on the distance image that the moving body associated with the combined distance image information has separated into multiple moving bodies, and the separated moving body is a combined moving body consisting of multiple individual moving bodies, the combined distance image information associated with the label of each of the individual moving bodies belonging to the separated moving body is updated to the distance image information of the combined moving body after separation. Image processing methods.

4. 4. The image processing method according to claim 3, When it is determined based on the distance image that the moving body associated with the combined distance image information has separated into multiple moving bodies, a single moving body belonging to the separated moving body is selected based on the single distance image information associated with the moving body before separation and the distance image information of the moving body after separation, and a combination of the single distance image information of the multiple single moving bodies associated with the moving body before separation is compared with the distance image information of the moving body after separation to select a combination of multiple single moving bodies belonging to the separated moving body. Image processing methods.

5. 5. The image processing method according to claim 4, If the single distance image information associated with the moving body before separation corresponds to the distance image information of one moving body after separation, associate the label of the moving body before separation and the single distance image information with the one moving body, and discard the combined distance image information associated with the label. Image processing methods.

6. 6. An image processing method according to claim 1, When it is determined based on the distance image that a moving body that is not associated with the combined distance image information has separated into multiple moving bodies, if the individual distance image information associated with the moving body before separation corresponds to the distance image information of one moving body after separation, the label of the moving body before separation and the individual distance image information are associated with the one moving body, and a new label is assigned to the other moving body after separation, and the distance image information of the other moving body is associated with the label as the new individual distance image information. Image processing methods.

7. 7. An image processing method according to claim 1, the distance image information of the moving object includes at least one piece of information on the center of gravity position, height, and volume of the moving object that can be detected from the distance image; Image processing methods.

Images