Correction system, correction method, and correction program

Abstract

To provide a correction system for improving accuracy of detecting an object regardless of whether the object is moving.SOLUTION: A correction system includes: an acquisition unit which acquires an image obtained by imaging a predetermined region; a detection unit which detects position information of a person and an object from an image including the person and the object, using a trained model trained to detect position information of a person and an object from an image; a switching unit which switches correction processing for the position information of the object in accordance with the detected position information of the object; and a correction unit which corrects the detected position information of the object by correction processing switched by the switching unit.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a correction system, a correction method, and a correction program.

Background Art

[0002] In Japan, due to the improvement of living standards, the improvement of sanitary environment, and the improvement of medical standards accompanying the high economic growth after the war, the aging population has become remarkable. Therefore, combined with the decline in the birth rate, it has become an aging society with a high aging rate. In such an aging society, an increase in the number of care recipients who require care, such as due to illness, injury, and aging, is assumed.

[0003] Care recipients and the like are at risk of falling while walking or falling from a bed and getting injured in facilities such as hospitals and elderly welfare facilities. Therefore, in order for staff such as nurses and caregivers to rush to the care recipients immediately when they are in such a state, the development of a system for detecting the state of care recipients and the like from the photographed images is underway. In order to detect the state of care recipients and the like with such a system, it is necessary to accurately detect the posture and behavior of the person to be detected from the image.

[0004] The following Patent Document 1 discloses the following prior art. An object moving in a moving image is detected, the movement line of the detected object is obtained, and the type or attribute of the object is identified based on the shape of the movement line. Thereby, the type or attribute of the moving object is identified.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the above prior art has a problem that although it can identify the type or attribute of a moving object, it cannot cope with improving the detection accuracy of an object regardless of whether the object is moving or not.

[0007] The present invention has been made to solve such problems. That is, an object of the present invention is to provide a correction system, a correction method, and a correction program that can improve the detection accuracy of an object regardless of whether the object is moving or not.

Means for Solving the Problems

[0008] The above problems of the present invention are solved by the following means.

[0009] (1) An acquisition unit that acquires an image in which a predetermined area is photographed, and a learned model that has been learned to detect the position information of the person and the object from the image including the person and the object, and uses the learned model to detect the position information of the person and the object from the image A detection unit, a switching unit that switches the correction process of the position information of the object according to the position information of the object detected by the detection unit, and the position information of the object detected by the detection unit is corrected by the correction process after switching by the switching unit A correction system having a correction unit.

[0010] (2) The switching unit includes a moving object determination unit that determines whether or not the object can move based on the position information of the object detected by the detection unit, and according to the determination result by the moving object determination unit, the correction unit The correction system according to the above (1), further comprising a correction process switching unit that switches the correction process of the position information of the object.

[0011] (3) The position information includes the respective attribute classes and positions of the person and the object. The detection unit includes an attribute class identification unit that identifies the attribute classes of the person and the object based on the image, and a position identification unit that identifies the positions of the person and the object based on the image. The moving object determination unit determines whether the object can move by using a reference table in which the relationship between the attribute class and the possibility of movement is set, based on the attribute class of the object identified by the attribute class identification unit, in the correction system according to (2) above.

[0012] (4) The switching unit includes a movement likelihood calculation unit that calculates the likelihood of the ease of movement of the object from the position information of the object. The moving object determination unit identifies whether the object can move based on the likelihood of the object calculated by the movement likelihood calculation unit, in the correction system according to (2) above.

[0013] (5) When the correction unit is switched by the switching unit to the correction process of the position information of the object that does not move, and when it is determined that there is contact between the person and the object, the correction unit includes a stationary object correction unit that performs correction to delete misdetection of the position information of the object, and when the correction unit is switched by the switching unit to the correction process of the position information of the object that moves, and when it is determined that there is contact between the person and the object, the correction unit includes a moving object correction unit that performs correction to complement non-detection of the position information of the object, in the correction system according to (1) above.

[0014] (6) The stationary object correction unit determines misdetection based on the position information of the object detected from a past image captured before the image in which the position information of the object was detected, among the objects determined to be in contact with the person, and performs correction to delete the position information of the object determined to be misdetected. The moving object correction unit does not delete the position information of the object determined to be in contact with the person, even when it is determined that there is contact between the person and the object, in the correction system according to (5) above.

[0015] (7) When the moving object correction unit determines that there is contact between the person and the object, in the past image in which the predetermined area was captured before the image in which the position information was detected, it tracks the object determined to have contact with the person, and corrects to complement the position information of the object that cannot be confirmed by the tracking. The correction system according to (5) or (6) above.

[0016] (8) A step (a) of acquiring an image in which a predetermined area is captured, and using a learned model that has been learned to detect the position information of the person and the object from the image including the person and the object, detecting the position information of the person and the object from the image in step (b), a step (c) of switching the correction process of the position information of the object according to the position information of the object detected in step (b), and a step (d) of correcting the position information of the object detected in step (b) by the correction process after the switching in step (c). A correction method comprising:

[0017] (9) The step (c) includes a step (c1) of determining whether the object can move based on the position information of the object detected in step (b), and a step (c2) of switching the correction process of the position information of the object in step (d) according to the determination result in step (c1). The correction method according to (8) above.

[0018] (10) The position information includes the respective attribute classes and positions of the person and the object. The step (b) includes a step (b1) of specifying the attribute classes of the person and the object based on the image, and a step (b2) of specifying the positions of the person and the object based on the image. The step (c1) determines whether the object can move using a reference table in which the relationship between the attribute class and the possibility of movement is set based on the attribute class of the object specified in step (b1). The correction method according to (9) above.

[0019] (11) The step (c) includes a step (c3) of calculating the likelihood of the ease of movement of the object from the position information of the object, and the step (c1) specifies the possibility of the movement of the object based on the likelihood of the object calculated in the step (c3). The correction method according to (9) above.

[0020] (12) When the step (d) is switched to the correction process of the position information of the object that does not move in the step (c), when it is determined that there is contact between the person and the object, the step (d1) of performing correction to delete the false detection of the position information of the object; when the step (d) is switched to the correction process of the position information of the object that moves in the step (c), when it is determined that there is contact between the person and the object, the step (d2) of performing correction to complement the undetected position information of the object. The correction method according to (8) above.

[0021] (13) In the step (d1), based on the position information of the object detected from a past image taken before the image in which the position information of the object was detected among the objects determined to be in contact with the person, false detection is determined, and correction is performed to delete the position information of the object determined to be false detection. In the step (d2), even when it is determined that there is contact between the person and the object, the position information of the object determined to be in contact with the person is not deleted. The correction method according to (12) above.

[0022] (14) In the step (d2), when it is determined that there is contact between the person and the object, in a past image in which the predetermined area was taken before the image in which the position information was detected, the object determined to be in contact with the person is tracked, and correction is performed to complement the position information of the object that cannot be confirmed by tracking. The correction method according to (12) or (13) above.

[0023] A correction program for causing a computer to execute the correction method according to any one of (8) to (14) above.

Advantages of the Invention

[0024] Using a learned model that has been trained to detect the position information of people and objects from images including people and objects, according to the position information of the objects detected from the images in a predetermined area, the correction process of the position information of the objects is switched, and the position information of the objects is corrected by the corrected process after switching. Thereby, the detection accuracy of the objects can be improved regardless of whether the objects move.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Best Mode for Carrying Out the Invention

[0026] Hereinafter, with reference to the drawings, a correction system, a correction method, and a correction program according to an embodiment of the present invention will be described. In the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may be different from the actual ratios.

[0027] (First Embodiment) FIG. 1 is a diagram showing a schematic configuration of the detection system 10.

[0028] The detection system 10 includes a photographing device 100, a server 200, a communication network 300, and a mobile terminal 400. The photographing device 100 is communicably connected to the server 200 via the communication network 300. The mobile terminal 400 can be connected to the communication network 300 via an access point 310. The server 200 constitutes a correction system. Note that part or all of the functions of the server 200 may be executed by the photographing device 100. In this case, the photographing device 100 can constitute the correction system alone or together with the server 200.

[0029] (Photographing Device 100) FIG. 2 is a block diagram showing the configuration of the photographing device 100. As shown in the example of FIG. 2, the photographing device 100 includes a control unit 110, a communication unit 120, and a camera 130, which are interconnected by a bus.

[0030] The control unit 110 is composed of a CPU (Central Processing Unit) and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and controls and performs arithmetic processing on each part of the imaging device 100 according to a program. The control unit 110 transmits an image 600 (see FIGS. 5, 7, and 8) obtained by the camera 130 photographing a predetermined area to a server 200 or the like via the communication unit 120. Hereinafter, the image 600 photographed by the camera 130 is also simply referred to as "image 600". The predetermined area is, for example, a three-dimensional area including the entire floor surface of the living room of the person 500.

[0031] The communication unit 120 is an interface circuit (such as a LAN card) for communicating with a mobile terminal 400 or the like via the communication network 300.

[0032] The camera 130 is, for example, a wide-angle camera. The camera 130 is installed at a position overlooking the predetermined area when the imaging device 100 is installed on the ceiling or the like of the living room of the person 500, and photographs the predetermined area. The person 500 is, for example, a person who requires care or nursing by a staff or the like and is a target person for behavior detection. The camera 130 may be a standard camera with a narrower angle of view than a wide-angle camera. The image 600 may include the person 500 and objects 700 such as a bed 710 and a chair 720 as images. The image 600 includes still images and moving images. The camera 130 is a near-infrared camera, irradiates the imaging area with near-infrared rays by an LED (Light Emitting Device), and can photograph the predetermined area by receiving the reflected light of the near-infrared rays reflected by the objects in the imaging area with a CMOS (Complememtary Metal Oxide Semiconductor) sensor. The image 600 may be a monochrome image with the reflectance of near-infrared rays as each pixel. The camera 130 may use a visible light camera instead of the near-infrared camera, or these may be used in combination.

[0033] (Server 200) FIG. 3 is a block diagram showing the configuration of the server 200. The server 200 includes a control unit 210, a communication unit 220, and a storage unit 230. Each component is interconnected by a bus.

[0034] Since the basic configurations of the control unit 210 and the communication unit 220 are the same as those of the corresponding components of the imaging device 100, namely the control unit 110 and the communication unit 120, duplicate explanations are omitted. The storage unit 230 is composed of a RAM, a ROM, a HDD (Hard Disc Drive), etc.

[0035] FIG. 4 is a functional block diagram of the control unit 210. The control unit 210 functions as an acquisition unit 211, a detection unit 212, a switching unit 213, and a correction unit 214. Note that the control unit 210 also functions as a behavior estimation unit (not shown).

[0036] The acquisition unit 211 acquires the image 600 from the imaging device 100 by receiving it through the communication unit 220. When the image 600 received from the imaging device 100 is stored in the storage unit 230, the acquisition unit 211 may also acquire it by reading the image 600 from the storage unit 230. The image 600 includes an image 600 including a person 500 and an object 700. The objects include an object 700 that may move (hereinafter referred to as a "moving object") and an object 700 that has no possibility of moving (hereinafter referred to as a "stationary object"). The moving objects include, for example, wheelchairs and walkers. The stationary objects include a chair 720 and a bed 710. Note that the stationary objects also include objects 700 with a relatively low possibility of moving. The moving objects constitute the moving objects. The stationary objects constitute the non-moving objects.

[0037] The detection unit 212 detects the position information of the person 500 and the object 700 from the image 600 including the person 500 and the object 700 using a learned model that has been learned to detect the position information of the person 500 and the object 700. As the learned model, for example, a Region Proposal Network (RPN) which is a neural network model can be used.

[0038] The position information is information on the respective attribute classes and positions of the person 500 and the object 700. That is, the position information is information indicating the position for each attribute class. The attribute class is a classification of the person 500 and each object 700, and examples thereof include "person", "wheelchair", "chair", "walker", and "bed". The position can be coordinates on the image 600.

[0039] The detection unit 212 may include an attribute identification unit 2121 and a position identification unit 2122. The attribute identification unit 2121 identifies (estimates) the attribute classes of the person 500 and the object 700 based on the image 600. The position identification unit 2122 identifies (estimates) the positions of the person 500 and the object 700 (that is, the positions for each attribute class) based on the image 600. The attribute identification unit 2121 constitutes an attribute class identification unit.

[0040] The attribute identification unit 2121 may detect a candidate rectangle that is a region including the person 500 or the object 700 from the image 600, and identify (estimate) the attribute class for each candidate rectangle. The candidate rectangle including the person 500 is also referred to as a "person rectangle 610" (see FIGS. 5, 7, and 8). The candidate rectangle including the object 700 is also referred to as an "object rectangle 620" (see FIGS. 5 and 8).

[0041] The position identification unit 2122 identifies the coordinates of the person rectangle 610 and the object rectangle 620 (for example, the coordinates of a set of opposite corners of the rectangle) as the positions of the person rectangle 610 and the object rectangle 620, respectively.

[0042] The switching unit 213 switches the correction process of the position information of the object 700 according to the position information of the object 700 detected by the detection unit 212 (specifically, the attribute class of the object rectangle 620). The switching unit 213 may include a moving object determination unit 2131 and a correction process switching unit 2132.

[0043] The moving object determination unit 2131 determines whether the object 700 can move based on the position information of the object 700. Specifically, the moving object determination unit 2131 determines whether the object 700 detected as the object rectangle 620 is a moving object or a stationary object based on the position information of the object 700.

[0044] Specifically, the moving object determination unit 2131 can determine whether the object 700 can move by using a reference table in which the relationship between the attribute class and the possibility of movement is set based on the attribute class of the object 700 specified by the attribute specification unit 2121. That is, the moving object determination unit 2131 determines whether the object 700 detected as the object rectangle 620 is a moving object or a stationary object by using the reference table.

[0045] The correction process switching unit 2132 switches the correction process of the position information of the object 700 by the correction unit 214 according to the determination result by the moving object determination unit 2131. Specifically, the correction process switching unit 2132 switches the correction process by the correction unit 214 to either the correction process of the position information of the moving object or the correction process of the position information of the stationary object. The correction process switching unit 2132 outputs correction process information for controlling which correction process of the correction process of the position information of the moving object or the correction process of the position information of the stationary object is to be performed to the correction unit 214.

[0046] The correction unit 214 corrects the position information of the object 700 detected by the detection unit 212 by the correction process after switching by the switching unit 213, and outputs the corrected position information. That is, the correction unit 214 corrects the position information of the object 700 among the position information detected by the detection unit 212 by the correction process after switching, and outputs the corrected position information. The correction unit 214 may include a stationary object correction unit 2141 and a moving object correction unit 2142. Note that the attribute class of the object 700 output from the detection unit 212 to the switching unit 213 and the position information output from the detection unit 212 to the correction unit 214 are associated with each other by a unique ID or the like.

[0047] When the still object correction unit 2141 is switched by the switching unit 213 to the correction process of the position information of the still object, if it is determined that there is contact between the person 500 and the object 700, the still object correction unit 2141 performs correction to delete the false detection of the object 700 with respect to the position information of the object 700 (hereinafter referred to as "correction process related to still object"). When the moving object correction unit 2142 is switched by the switching unit 213 to the correction process of the position information of the moving object, the moving object correction unit 2142 performs correction to complement the undetected of the object 700 with respect to the position information of the object 700 (hereinafter referred to as "correction process related to moving object"). The correction process related to the moving object can be performed on the past image 600. The past image 600 may be an image 600 taken before the image 600 in which the position information that is the basis for the switching of the correction process by the switching unit 213 is detected.

[0048] FIG. 5 is an explanatory diagram for explaining the correction process related to still object. In FIG. 5, for simplicity of explanation, a person rectangle 610 and an object rectangle 620 are shown on the image 600.

[0049] As shown in FIG. 5, in the current image 600, an object 700 is detected as an object rectangle 620. However, if the detected object 700 is a stationary object, it may be misdetected for the reasons described below. Whether it is a misdetection can be determined by whether an object 700 of the same attribute class is detected at the same position (including substantially the same position) in the past image 600 that is a certain time back. The predetermined time can be set to an appropriate value through experiments from the perspective of the detection accuracy of the object 700. In the past image 600 that is the moving (while the person 500 is moving) image 600, since there is no object rectangle 620 (the object 700 is not detected), it can be determined that the object 700 detected in the current image 600 is a misdetection. This is because for a stationary object (for example, a chair), since the stationary object does not move, although the object 700 detected in the current image 600 should be detected at the same position in the past image 600 as well, it is not detected. Therefore, in the correction process for stationary objects, when the object 700 detected in the current image 600 is not detected in the past image 600, the position information of the object 700 included in the position information detected in the current image 600 is deleted. That is, the stationary object correction unit 2141 determines a misdetection based on the position information of the object 700 detected from a past image 600 taken before the image 600 in which the position information of the object 700 was detected among the objects 700 determined to be in contact with the person 500, and performs correction to delete the position information of the object determined to be a misdetection. In FIG. 5, the object rectangle 620 corresponding to the position information of the object 700 to be deleted is shown as a dashed rectangle.

[0050] Note that in FIG. 5, although an object 700 is detected in the image 600 before movement (before the person 500 moves), since the object 700 is determined to be a stationary object, correction to delete the object 700 can be performed in the same manner as the correction for the current image 600.

[0051] The correction process for stationary objects corresponds to the correction process for correcting the misdetection of the object 700.

[0052] Examples of reasons for misdetection of stationary objects will be described with reference to FIGS. 6 to 8.

[0053] FIG. 6 is an explanatory diagram showing an image 650 of training data used for training the learned model of the detection unit 212. In FIG. 6, for simplicity of explanation, a person rectangle 610 and an object rectangle 620 indicating the positions of a person 500 and an object 700 (chair 720), which are correct labels, are shown together on the image 650.

[0054] As shown in the example of FIG. 6, by using a relatively large amount of training data of a combination of an image 650 including a person 500 and an object 700 and correct labels (correct attribute classes and positions) of the person rectangle 610 and the object rectangle 620 to train the neural network model, a learned model can be generated.

[0055] FIG. 7 is an explanatory diagram showing an example of position information normally detected by the detection unit 212 together with an image 600 acquired by the acquisition unit 211. In FIG. 7, for simplicity of explanation, the person rectangle 610, which is the detected position information, is shown on the image 600. When the image 600 acquired by the acquisition unit 211 includes a person 500 but does not include an object 700, if the position information is normally detected, the position of the person 500 etc. is detected as the person rectangle 610, and the non-existent object is not detected as the object rectangle 620.

[0056] FIG. 8 is an explanatory diagram showing an example of position information including an object rectangle 620 erroneously detected (abnormally detected) by a detection unit 212 together with an image 600 acquired by an acquisition unit 211. In FIG. 8, for simplicity of explanation, a detected person rectangle 610 and an object rectangle 620 are shown on the image 600. In the example shown in FIG. 8, when the image 600 acquired by the acquisition unit 211 includes a person 500 and does not include an object 700, the position of the person 500 and the like are normally detected as the person rectangle 610. On the other hand, a non-existent object 700, a chair 720, is erroneously detected as an object rectangle 620 of the chair 720. Such an erroneous detection of the object 700 is considered to be caused by, for example, using an image including both the person 500 and the object 700 in the training data image 650 used for training the learned model, so that the features of the person 500 and the features of the object 700 around the person 500 are associated and learned. This indicates that depending on the movement and posture of the person 500, there is a possibility of erroneously detecting an object 700 in the space around the person 500 where there is nothing.

[0057] When such a non-existent object 700, such as a chair 720, is erroneously detected, in the estimation of the posture of the person 500 based on the position information of the person 500 and the object 700, the floor sitting posture may be erroneously estimated as the chair sitting posture. As a result, in the estimation of the action of the person 500 based on the posture of the person 500 and the like, an action (or floor sitting posture) in which the person 500 falls may be erroneously estimated as an action (or chair sitting posture) of sitting on the chair 720.

[0058] The correction process for a stationary object can determine an erroneous detection of the object 700 based on whether an object 700 of the same attribute class has been detected at the same position in a past image 600 when it is determined that there is contact between the person 500 and the object 700. That is, upon the occasion of contact between the person 500 and the object 700, a process for determining whether there is an erroneous detection of the object 700 can be started. Whether there is contact between the person 500 and the object 700 can be determined based on whether the size of the overlap between the person rectangle 610 and the object rectangle 620 exceeds a predetermined threshold. The predetermined threshold can be set to an appropriate value by experiment from the viewpoint of the detection accuracy of the object 700.

[0059] FIG. 9 is an explanatory diagram for explaining correction processing related to a moving object. In FIG. 9, for simplicity of explanation, a human rectangle 610 and an object rectangle 620 are shown on an image 600.

[0060] As shown in FIG. 9, in the current image 600, an object 700 is detected as the object rectangle 620. When the detected object 700 is a moving object, for example, like a wheelchair, it is assumed that the human rectangle 610 and the object rectangle 620 are in positions close to each other. Also, since the object 700 moves together with the person 500, it is naturally assumed that in a past image 600 that is a predetermined time back, an object 700 of the same attribute class is not detected at the same position (including substantially the same position). Therefore, in the correction processing related to the moving object, even when it is determined that there is contact between the person 500 and the object 700, the position information of the object 700 determined to have contact with the person 500 is not deleted. Therefore, in FIG. 9, the object 700, which is a moving object detected as the object rectangle 620 from the images 600 before and after movement (current), is not deleted in the correction processing related to the moving object.

[0061] In the correction processing related to the moving object, correction is performed to complement an undetected object 700 (an object 700 that actually exists but is not detected) with the position information detected from past images 600. Specifically, when the moving object correction unit 2142 determines that there is contact between the person 500 and the object 700, in a past image 600 in which a predetermined area was photographed before the image 600 where the position information was detected, the object 700 determined to have contact with the person 500 is tracked, and correction is performed to complement the position information of the object 700 that cannot be confirmed by the tracking. Tracking of the object 700 in the past image 600 can be performed using known tracking means. For example, SORT (Simple Online and Realtime Tracking) can be used for tracking the object 700 in the past image 600.

[0062] In FIG. 9, object 700 is not detected as object rectangle 620 in the moving image 600. Therefore, as shown by the dashed rectangle, the object rectangle 620 of the moving object is complemented in the position information detected from the moving image 600. That is, the position information of object 700, which is the moving object, is complemented in the position information detected from the moving image 600.

[0063] The complementation of object 700 in the correction process for the moving object can be performed as follows. For example, the position information of object 700 is complemented to the position information detected from the past image 600 so that the relative positional relationship between person 500 and object 700 in contact with each other is maintained in the past image 600 in which the face object 700 is not detected.

[0064] In this way, the correction process for the moving object corresponds to the correction process for correcting the undetected state of object 700.

[0065] The corrected position information can be used for detecting the action of person 500 by an action estimation unit (not shown).

[0066] The action estimation unit estimates the action of person 500 based on the corrected position information. Specifically, the action estimation unit can estimate the action of person 500 as follows. The action estimation unit estimates the joint points based on the person rectangle 610 (more specifically, the image included in the person rectangle 610). The action estimation unit can estimate the joint points of person 500 using a neural network model that has been learned to estimate the joint points of person 500 from the person rectangle 610. As the neural network model, a known model such as Deep Pose can be used. The action estimation unit can estimate the action of person 500 based on the relationship between the joint points and the object rectangle 620. Specifically, the action of person 500 can be estimated based on the relationship between the posture of person 500 indicated by the joint points and the position and attribute class of object 700 indicated by the object rectangle 620. For example, when the object indicated by the object rectangle 620 is chair 720 and the posture indicated by the joint points is a sitting position, the action estimation unit can estimate that person 500 is performing the action of sitting on the chair when a predetermined number or more of the joint points overlap with the object rectangle 620.

[0067] When the control unit 210 determines that the detected behavior of the person 500 corresponds to an event preset as a behavior that needs to be notified to the care staff such as a fall or a tumble, the control unit 210 may transmit an event notification including the behavior corresponding to the event, the name of the person 500, the room number, etc. to the mobile terminal 400. Further, the control unit 210 may notify the mobile terminal 400 of the detected behavior of the person 500 periodically or when the behavior changes.

[0068] (Mobile terminal 400) FIG. 10 is a block diagram showing the configuration of the mobile terminal 400. The mobile terminal 400 includes a control unit 410, a wireless communication unit 420, a display unit 430, an input unit 440, and an audio input / output unit 450. Each component is interconnected by a bus. The mobile terminal 400 may be constituted by a communication terminal device such as, for example, a tablet computer, a smartphone, or a mobile phone.

[0069] The control unit 410 has a basic configuration such as a CPU, a RAM, and a ROM, similar to the configuration of the control unit 110 of the imaging device 100.

[0070] The wireless communication unit 420 has a function of performing wireless communication according to standards such as Wi-Fi and Bluetooth (registered trademark), and performs wireless communication with each device via the access point 310 or directly. The wireless communication unit 420 receives an event notification from the server 200.

[0071] The display unit 430 and the input unit 440 are a touch panel, and a touch sensor as the input unit 440 is provided on the display surface of the display unit 430 composed of liquid crystal or the like. The display unit 430 displays the actions and event notifications of the target person 510 received from the server 200. Note that the display unit 430 and the input unit 440 may display an input screen for prompting the target person 510 regarding the event notification, and receive the intention of the staff to respond to the event notification input on the input screen and transmit it to the server 200. In this case, the server 200 determines one of the staff as the staff in charge of responding to the event notification, and transmits a notification notifying all the staff including the determined staff to the mobile terminals 400 of all the staff in charge of responding to the event notification.

[0072] The voice input / output unit 450 is, for example, a speaker and a microphone, and enables voice calls between staff through the wireless communication unit 420 and other mobile terminals 400.

[0073] The operation of the detection system 10 will be described.

[0074] FIG. 11 is a flowchart showing the operation of the detection system 10. This flowchart can be executed by the control unit 210 of the server 200 according to a program. Note that when a part or all of the functions shown in FIG. 4 are executed by the imaging device 100, a part or all of this flowchart may be executed by the control unit 110 of the imaging device 100 according to a program.

[0075] The control unit 210 acquires it by receiving the image 600 from the imaging device 100 (S101).

[0076] The control unit 210 detects the position information of the person 500 and the object 700 from the acquired image 600 using a learned model that has been learned to detect the position information of the person 500 and the object 700 from the image 600 including the person 500 and the object 700 (S102).

[0077] Based on the position information of the object 700, the control unit 210 determines whether the object 700 can move by using a reference table (S103).

[0078] The control unit 210 determines whether the object 700 is movable (S104). That is, the control unit 201 determines whether the object 700 is a moving object.

[0079] When the control unit 210 determines that the object 700 is not a moving object (when it determines that the object is a stationary object) (S104: NO), it performs correction processing for the stationary object on the position information detected in step S102 (S105).

[0080] When the control unit 210 determines that the object 700 is a moving object, it performs correction processing for the moving object on the position information detected in step S102 (S106).

[0081] (Second Embodiment) The second embodiment will be described. The difference between this embodiment and the first embodiment is as follows. In the first embodiment, based on the attribute class included in the position information of the object 700, the reference table is referred to determine whether the object 700 can move. On the other hand, in this embodiment, from the position information of the object 700, the likelihood of the object 700 being easy to move (hereinafter referred to as "movement likelihood") is calculated, and based on the calculated likelihood, it is determined whether the object 700 can move. Other points are the same as those in the first embodiment, so duplicate explanations are omitted.

[0082] FIG. 12 is a functional block diagram of the control unit 210. The control unit 210 functions as an acquisition unit 211, a detection unit 212, a switching unit 213, and a correction unit 214.

[0083] The switching unit 213 switches the correction processing of the position information of the object 700 according to the position information of the object 700 detected by the detection unit 212. The switching unit 213 may include a movement likelihood calculation unit 2133, a moving object determination unit 2131, and a correction processing switching unit 2132.

[0084] The movement likelihood calculation unit 2133 calculates a movement likelihood from the position information of the object 700 (i.e., the attribute class and position of the object 700). The movement likelihood calculation unit 2133 can detect the position information of the person 500 and the object 700 using a learned model that has been learned to calculate the movement likelihood from the position information of the object 70 (i.e., the attribute class and position of the object 70).

[0085] The moving object determination unit 2131 determines whether the object 700 can move based on the movement likelihood calculated by the movement likelihood calculation unit 2133. That is, the moving object determination unit 2131 determines whether the object 700 is a moving object or a stationary object based on the movement likelihood of the object 700 detected as the object rectangle 620. Specifically, an object 700 with a movement likelihood greater than or equal to a predetermined threshold is determined as a moving object, and an object 700 with a movement likelihood less than the predetermined threshold is determined as a stationary object.

[0086] The embodiment has the following effects.

[0087] Using a learned model that has been learned to detect the position information of a person and an object from an image including the person and the object, according to the position information of the object detected from the image of a predetermined area, the correction process of the position information of the object is switched, and the position information of the object is corrected by the corrected process after switching. Thereby, the detection accuracy of the object can be improved regardless of whether the object moves or not.

[0088] Furthermore, based on the position information of the object detected by the detection unit, it is determined whether the object can move, and according to the determination result, the correction process of the position information of the object is switched. Thereby, the detection accuracy of the object can be improved simply and efficiently.

[0089] Furthermore, the position information includes the respective attribute classes and positions of people and objects. Based on an image, the attribute classes of people and objects are identified, and based on the image, the positions of people and objects are identified. Then, based on the identified attribute class of the object, the possibility of movement of the object is identified using a reference table in which the relationship between the attribute class and the possibility of movement is set. Thereby, the possibility of movement of the object can be identified more easily.

[0090] Furthermore, from the position information of the object, the likelihood of the ease of movement of the object is calculated, and based on the calculated likelihood of the ease of movement of the object, the possibility of movement of the object is identified. Thereby, the possibility of movement of the object can be identified more easily.

[0091] Furthermore, when switching to the correction process of the position information of the non-moving object, when it is determined that there is contact between a person and the object, a correction is made to delete the false detection of the position information of the object. And when switching to the correction process of the position information of the moving object, when it is determined that there is contact between a person and the object, a correction is made to complement the undetected position information of the object. Thereby, the detection accuracy of the object can be improved more easily and effectively.

[0092] Furthermore, for stationary objects, among the objects determined to have contact with a person, false detection is determined based on the position information of the object detected from a past image taken before the image in which the position information of the object was detected, and a correction is made to delete the position information of the object determined to be false detection. For moving objects, even when it is determined that there is contact between a person and the object, the position information of the object determined to have contact with a person is not deleted. Thereby, false detection of stationary objects can be suppressed, and a decrease in the detection accuracy of moving objects can be suppressed.

[0093] Furthermore, when it is determined that there is contact between a person and the object, in a past image in which a predetermined area was taken before the image in which the position information was detected, the object determined to have contact with a person is tracked, and a correction is made to complement the position information of the object that cannot be confirmed by the tracking. Thereby, the detection accuracy of the moving object can be improved.

[0094] The configuration of the system described above explains the main configuration in describing the features of the above-described embodiments, and is not limited to the above-described configuration, and various modifications can be made within the scope of the claims. Also, it does not exclude the configurations provided in general human and object detection systems.

[0095] Also, the means and methods for performing various processes in the system described above can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in a storage unit such as a hard disk. Also, the above program may be provided as a single application software, or may be incorporated as a single function into the software of a server or other device.

Explanation of Reference Numerals

[0096] 10 Detection system, 100 Photographing device, 110 Control unit, 120 Communication unit, 130 Camera, 200 Server, 210 Control unit, 220 Communication unit, 230 Storage unit, 300 Communication network, 400 Portable terminal, 500 Person, 600 Image, 610 Person rectangle, 620 Object rectangle, 700 Object, 710 Bed, 720 Chair.

Claims

1. An acquisition unit that acquires an image of a predetermined area; A detection unit that detects the position information of the person and the object from the image including the person and the object, using a learned model that has been learned to detect the position information of the person and the object respectively; A switching unit that switches the correction process of the position information of the object according to the position information of the object detected by the detection unit; A correction unit that corrects the position information of the object detected by the detection unit by the correction process after switching by the switching unit; A correction system having the above.

2. The switching unit includes: A moving object determination unit that determines whether the object can move based on the position information of the object detected by the detection unit; A correction process switching unit that switches the correction process of the position information of the object by the correction unit according to the determination result by the moving object determination unit; The correction system according to claim 1, having the above.

3. The position information includes the respective attribute classes and positions of the person and the object, The detection unit includes: An attribute class identification unit that identifies the attribute classes of the person and the object based on the image; A position identification unit that identifies the positions of the person and the object based on the image, and has; The moving object determination unit determines whether the object can move using a reference table in which the relationship between the attribute class and the possibility of movement is set, based on the attribute class of the object identified by the attribute class identification unit. The correction system according to claim 2.

4. The switching unit has a movement likelihood calculation unit that calculates the likelihood of the ease of movement of the object from the position information of the object, The moving object determination unit specifies whether the object can move based on the likelihood of the object calculated by the movement likelihood calculation unit. The correction system according to claim 2.

5. The correction unit includes: When the switching unit switches to the correction process of the position information of the non-moving object, a stationary object correction unit that performs correction to delete the false detection of the position information of the object when it is determined that there is contact between the person and the object; When switched by the switching unit to the correction process of the position information of the moving object, if it is determined that there is contact between the person and the object, there is a moving object correction unit that performs correction to complement the undetected position information of the object. The correction system according to claim 1.

6. The stationary object correction unit determines a false detection based on the position information of the object detected from a past image captured before the image in which the position information of the object was detected among the objects determined to be in contact with the person, and corrects by deleting the position information of the object determined to be a false detection. The moving object correction unit does not delete the position information of the object determined to be in contact with the person even when it is determined that there is contact between the person and the object. The correction system according to claim 5.

7. When it is determined that there is contact between the person and the object, the moving object correction unit tracks the object determined to be in contact with the person in a past image in which the predetermined region was captured before the image in which the position information was detected, and corrects by complementing the position information of the object that cannot be confirmed by the tracking. The correction system according to claim 5 or 6.

8. Step (a) of acquiring an image in which a predetermined region was captured; Step (b) of detecting the position information of the person and the object from the image including the person and the object using a learned model that has been learned to detect the position information of the person and the object from the image; Step (c) of switching the correction process of the position information of the object according to the position information of the object detected in step (b); Step (d) of correcting the position information of the object detected in step (b) by the correction process after the switching in step (c). A correction method having these steps.

9. Step (c) includes: Step (c1) of determining whether the object can move based on the position information of the object detected in step (b); Step (c2) of switching the correction process of the position information of the object in step (d) according to the determination result in step (c1). The correction method according to claim 8, having these steps.

10. The position information includes the respective attribute classes and positions of the person and the object. The step (b) includes: Step (b1) of identifying the attribute classes of the person and the object based on the image; Step (b2) of identifying the positions of the person and the object based on the image, and The step (c1) is the correction method according to claim 9, which determines the mobility of the object by using a reference table in which the relationship between the attribute class and the mobility is set based on the attribute class of the object identified in the step (b1).

11. The step (c) includes a step (c3) of calculating the likelihood of the ease of movement of the object from the position information of the object. The step (c1) is the correction method according to claim 9, which determines the mobility of the object based on the likelihood of the object calculated in the step (c3).

12. The step (d) includes: When switching to the correction process of the position information of the object that does not move in the step (c), if it is determined that there is contact between the person and the object, step (d1) of performing a correction to delete the false detection of the position information of the object; When switching to the correction process of the position information of the object that moves in the step (c), if it is determined that there is contact between the person and the object, step (d2) of performing a correction to complement the undetected position information of the object, and the correction method according to claim 8.

13. In the step (d1), based on the position information of the object detected from a past image taken before the image in which the position information of the object is detected among the objects determined to be in contact with the person, a false detection is determined, and a correction is made to delete the position information of the object determined to be a false detection. In the step (d2), even if it is determined that there is contact between the person and the object, the position information of the object determined to be in contact with the person is not deleted, and the correction method according to claim 12.

14. In the step (d2), when it is determined that there is contact between the person and the object, in the past image in which the predetermined region was photographed before the image in which the position information was detected, the object determined to be in contact with the person is tracked, and correction is performed to supplement the position information of the object that cannot be confirmed by the tracking. The correction method according to claim 12 or 13.

15. A correction program for causing a computer to execute the correction method according to any one of claims 8 to 14.

Images