IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND IMAGE PROCESSING SYSTEM

Abstract

To use a corrected object detection result for behavior recognition of a person by correcting an object detection result with other detection results and knowledge set in advance in an image processing device, an image processing method, a program, and an image processing system.SOLUTION: An image processing device 100 includes a person detection unit 112 for detecting a person from an input image, an object detection unit 113 for detecting an object from the input image, a correction unit 114 for correcting object recognition information detected by the object detection unit 113, and a determination unit 115 for determining a behavior of the person on the basis of the object recognition information corrected by the correction unit 114 and the person recognition information detected by the person detection unit 112.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and an image processing system. [Background technology]

[0002] In recent years, human behavior has been recognized using the results of human detection and object detection. By using the results of object detection in combination, it is possible to more accurately recognize what a person is doing. For example, when a person lying down and a bed are detected, it is possible to more reliably recognize that the person is sleeping by using the knowledge that a bed is for sleeping.

[0003] However, when a person approaches an object or when an object is in the shadow of a person, the object may be erroneously detected. False object detection may result in erroneous recognition of the person's actions.

[0004] Patent Document 1 discloses an image processing apparatus including "an image acquisition unit that acquires an image generated by an imaging device, a human region detection unit that detects a human region including a person in the image, a posture estimation unit that performs image analysis of the image using a trained classifier and estimates a posture type of the person together with an object type of surrounding objects reflected in the human region, and a behavior discrimination unit that determines a behavior class of the person based on the posture type of the person and the object type of the surrounding objects, wherein the classifier has undergone a learning process by associating a combination of the posture type of the person and the object type of the surrounding objects with the image of training data" (see, for example, claim 1 of Patent Document 1).

[0005] Patent Document 2 also discloses a "wake-up monitoring method for monitoring and reporting when a cared-for person sleeping in bed has woken up, the method comprising: setting a monitoring area for determining whether the cared-for person has woken up; capturing an image of the bed from the side including the monitoring area with a camera; and determining that the cared-for person has woken up if the size of the image area of ​​the cared-for person within the monitoring area in the image obtained from the camera is equal to or larger than a predetermined value" (see, for example, claim 1 of Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-206321 A [Patent Document 2] JP 2006-175082 A Summary of the Invention [Problem to be solved by the invention]

[0007] Claim 1 of Patent Document 1 is based on the premise that a person and an object are close to each other, and is therefore prone to false detection during object detection. On the other hand, claim 1 of Patent Document 2 may result in a decrease in the accuracy of human behavior recognition due to false recognition of the object recognition result. In this case, it becomes necessary to correct the object recognition result appropriately.

[0008] Therefore, an object of the present invention is to correct object detection results using other detection results or preset knowledge in an image processing device, an image processing method, a program, and an image processing system, and to use the corrected object detection results for human behavior recognition. [Means for solving the problem]

[0009] That is, the above-mentioned object of the present invention is achieved by the following configuration. (1) a person detection unit that detects a person from an input image; an object detection unit that detects an object from the input image; a correction unit that corrects the object recognition information detected by the object detection unit; a determination unit that determines the behavior of the person based on the object recognition information corrected by the correction unit and the person recognition information detected by the person detection unit; Equipped with The correction unit is correcting the likelihood of an object class related to the object recognition information; child An image processing device comprising:

[0011] ( 2 ) a person detection unit that detects a person from an input image; an object detection unit that detects an object from the input image; a correction unit that corrects the object recognition information detected by the object detection unit; a determination unit that determines a behavior of the person based on the object recognition information corrected by the correction unit and the person recognition information detected by the person detection unit, The correction unit is The object recognition information is replaced with object recognition information when the object was detected in the past. 13. An image processing device comprising:

[0012] ( 3 ) The correction unit is correcting the likelihood of an object class related to the object recognition information so as to detect an undetected object; Characterized by (1) Or (2) The image processing device according to claim 1 .

[0013] ( 4 ) The correction unit is correcting the object recognition information on the condition that the person approaches the object or the object is occluded by the person; The features of (1) to ( 3 ) An image processing device according to any one of the above.

[0014] ( 5 ) The correction unit is correcting the object recognition information according to time or lighting conditions; The features of (1) to ( 4 ) An image processing device according to any one of the above.

[0015] ( 6 ) The correction unit is correcting the object recognition information using stability of a detection result in frames of a certain period of object detection by the object detection unit; The features of (1) to ( 5 ) An image processing device according to any one of the above.

[0016] ( 7 The object to be detected by the object detection unit is It is used by a person in a seated position, The features of (1) to ( 6 ) An image processing device according to any one of the above.

[0017] ( 8 The object to be detected by the object detection unit is It is used by a person in a lying position, The features of (1) to ( 7 ) An image processing device according to any one of the above.

[0018] ( 9 ) the human detection unit further detects a posture of a person, The determination unit determines the behavior of the person based on the posture information detected by the person detection unit and the object recognition information corrected by the correction unit. (1) to ( 8 ) An image processing device according to any one of the above.

[0019] ( 10 ) The correction unit is The correction conditions or the correction contents vary depending on the position coordinates of the person or the object in the input image. The features of (1) to ( 9 ) An image processing device according to any one of the above.

[0020] (11 ) an imaging unit that captures the input image and is attached so as to look down from above, The further feature is (1) to (10) 13. An image processing device according to claim 12,

[0021] ( 12 ) detecting a person from an input image and setting the person recognition information; detecting an object from the input image and setting the object as object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; Equipped with In the step of correcting, correcting the likelihood of an object class related to the object recognition information; child An image processing method comprising: (13) detecting a person from the input image and determining the person recognition information; detecting an object from the input image and setting the object as object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the step of correcting, The object recognition information is replaced with object recognition information when the object was detected in the past. 13. An image processing method comprising:

[0022] ( 14 ) A procedure for detecting a person from an input image and obtaining person recognition information; A step of detecting an object from the input image and generating object recognition information; correcting the object recognition information; determining the behavior of the person based on the corrected object recognition information and the person recognition information; and causing a computer to execute the above. In the correction step, correcting the likelihood of an object class related to the object recognition information; of 、 A program characterized by being executed by a computer. (15) A step of detecting a person from an input image and obtaining person recognition information; A step of detecting an object from the input image and generating object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the correction step, replacing the object recognition information with object recognition information when the object was detected in the past; A program for causing a computer to execute the above.

[0023] ( 16 ) an imaging unit; a person detection unit that detects a person from an input image captured by the imaging unit; an object detection unit that detects an object from the input image; a correction unit that corrects content indicated by the object recognition information detected by the object detection unit; a determination unit that determines the behavior of the person based on the corrected object recognition information and the person recognition information detected by the person detection unit; Equipped with The correction unit is correcting the likelihood of an object class related to the object recognition information; child An image processing system comprising: (17) An imaging unit; a person detection unit that detects a person from an input image captured by the imaging unit; an object detection unit that detects an object from the input image; a correction unit that corrects content indicated by the object recognition information detected by the object detection unit; a determination unit that determines the behavior of the person based on the corrected object recognition information and the person recognition information detected by the person detection unit, The correction unit is The object recognition information is replaced with object recognition information when the object was detected in the past. 1. An image processing system comprising: Effect of the Invention

[0024] According to the present invention, in an image processing device, an image processing method, a program, and an image processing system, the results of object detection can be corrected using other detection results or preset knowledge, and the corrected object detection results can be used for human behavior recognition. [Brief description of the drawings]

[0025] [Figure 1]FIG. 1 is an explanatory diagram illustrating an example of a main configuration of an information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of the image processing device. [Diagram 3] 11 is a flowchart showing a process in which an image processing device corrects object recognition information and determines human behavior based on object recognition information and posture information. [Figure 4] 13 is a flowchart showing a process in which a CPU corrects the likelihood of an object class based on the posture of a person detected by a person detection unit, using a correction unit. [Diagram 5] 13 is a flowchart showing a process in which a CPU corrects, by a correction unit, an object class likelihood associated with an object approached or occluded by a person. [Figure 6] 11 is a flowchart showing a process in which a CPU corrects, by a correction unit, object recognition information related to an object that a person approaches or blocks. [Figure 7] This is an illustration showing the concept of resolving the dilemma of behavior recognition by using information from both people and objects. [Figure 8] FIG. 13 is an image diagram showing the concept of correcting the current bed position using past bed position information. [Figure 9] 11 is a flowchart showing a process in which a CPU replaces object recognition information with predetermined object recognition information by a correction unit. [Figure 10] FIG. 13 is an image diagram showing the concept of correcting the current bed recognition result using the past bed recognition result. [Figure 11] 11 is a flowchart showing a process in which a CPU corrects an object class likelihood related to object recognition information using a correction unit. [Figure 12] 11 is a flowchart showing a process in which the CPU uses the stability of the detection results of the object detection by the object detection unit in frames over a certain period of time to replace the detection results with object recognition information from when an object was detected in the past, using the correction unit. [Figure 13]11 is a flowchart showing a process in which a CPU, using a correction unit, corrects the likelihood of an object class related to the object recognition information by using the stability of detection results in frames over a certain period of object detection by the object detection unit. [Figure 14A] The camera captured an image of the center of the room from the center of the ceiling. [Figure 14B] FIG. 13 is an explanatory diagram showing an image of a room from the side. [Figure 15] FIG. 11 is a block diagram showing an image processing system according to another embodiment. [Figure 16] FIG. 11 is an explanatory diagram showing a recognition result by the image processing device of Comparative Example 1. [Figure 17] 10A to 10C are explanatory diagrams showing desirable correction contents using the person detection result and object recognition information as a result of detecting a person or object in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The following describes in detail the embodiments for carrying out the present invention. Note that the embodiments described below are merely examples for implementing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0027] <Overview of Comparative Examples> In an algorithm for detecting the position of an object, a chair CH may be erroneously detected as a desk, or the position of a bed BD may be erroneously detected as being in a different location.

[0028] FIG. 16 is an explanatory diagram showing a recognition result by the image processing device of Comparative Example 1. As shown in FIG.

[0029] This photographed image is, for example, an image taken by a surveillance camera installed in a room in a nursing home. In this room, there are people HM1, HM2, and HM3, and a chair CH and a bed BD1 are installed. A door DR is visible at the top of the photographed image, and a window and curtains CT are visible at the bottom of the photographed image. The photograph was taken in broad daylight, with direct sunlight streaming in through the window and illuminating the floor of the room. Person HM1 is sitting in chair CH. Person HM2 is lying on bed BD1. Person HM3 has fallen in the area where direct sunlight is streaming in.

[0030] Comparative Example 1 shows object recognition information 301, in which a chair CH is erroneously detected as a desk. Due to this erroneous detection, the image processing device cannot recognize the action of the person HM1 sitting on the chair CH.

[0031] In Comparative Example 1, the object recognition information 302 is shown for the portion irradiated by direct sunlight coming in through the window. Here, since the portion irradiated by direct sunlight is rectangular, the image processing device erroneously recognizes this as a bed and sets the object recognition information 302. And the bed BD1 is not recognized as a bed.

[0032] Therefore, the image processing device erroneously recognizes the action of the person HM2 sleeping on the bed BD1 as a fall.Furthermore, the image processing device erroneously recognizes the action of the person HM3 falling as an action of sleeping on the bed.

[0033] FIG. 17 is an explanatory diagram showing desirable correction contents using the person detection result and object recognition information as a result of the image processing device of this embodiment detecting people HM1, HM2, HM3 and objects. Since the image processing device of this embodiment is in a sitting state, it determines that the object recognition information 303 is highly likely not a desk and is highly likely to be a chair. Furthermore, the person HM2 is often in a lying position at this position during the night. Therefore, the image processing device determines that there is a high possibility that a bed is installed at the position where the person is lying down for a long time, and sets the object recognition information 304. Furthermore, the image processing device reduces the possibility of erroneous detection due to sunlight by adopting the object detection result when the curtains CT are closed.

[0034] In this manner, in the embodiment of the present invention, the object detection result can be corrected using other detection results or preset knowledge, and the corrected object detection result can be used for human behavior recognition.

[0035] <Present embodiment> [Overall configuration of image processing device] 1 is an explanatory diagram illustrating an example of the main configuration of an image processing system according to this embodiment. The image processing system according to this embodiment includes an image processing device 100 and a camera 180 (imaging unit).

[0036] The image processing device 100 includes a CPU (Central Processing Unit) 110, a storage unit 120, a ROM (Read Only Memory) 130, a RAM (Random Access Memory) 140, an input unit 150, a display unit 160, and a communication unit 170. The image processing device 100 is configured as, for example, a board-shaped computer to which a camera 180 is connected, and functions as an edge terminal.

[0037] The CPU 110 is a central processing unit, and executes a control program 121 stored in the storage unit 120 or the ROM 130 to implement each process (function) shown in Fig. 2. The processes implemented by the CPU 110 will be described later with reference to Fig. 2.

[0038] The storage unit 120 is a large-capacity storage device, and is configured, for example, by a hard disk drive, a non-volatile memory, etc. The storage unit 120 stores a control program 121.

[0039] The RAM 140 functions as a work area for temporarily storing various programs that are read from the ROM 130 and can be executed by the CPU 110, input data, output data, parameters, and the like during various processes executed and controlled by the CPU 110.

[0040] The input unit 150 is configured with a keyboard equipped with cursor keys, numeric input keys, various function keys, etc., and a pointing device such as a mouse. The input unit 150 outputs a press signal of a key pressed on the keyboard or an operation signal by a mouse as an input signal to the CPU 110. The CPU 110 executes various processes based on the operation signal from the input unit 150.

[0041] The display unit 160 is configured to include a monitor such as a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display). The display unit 160 displays various screens according to instructions of a display signal input from the CPU 110. A touch panel display can also be used as the display unit 160 and the input unit 150.

[0042] The communication unit 170 includes a communication interface and communicates with external devices on a network. In this example, the communication unit 170 is connected to an external camera 180 and receives images captured by the camera 180.

[0043] The internal bus 190 connects the components of the image processing device 100 to each other.

[0044] On the other hand, camera 180 is installed on the ceiling of the room so as to capture an input image and to look down from above into the room.

[0045] Next, functions of the CPU 110 of the image processing device 100 according to this embodiment will be described with reference to FIG.

[0046] Fig. 2 is a functional block diagram showing functions of the CPU 110 of the image processing device 100 according to this embodiment. The CPU 110 embodies an image acquisition unit 111, a human detection unit 112, an object detection unit 113, a correction unit 114, and a determination unit 115 shown in Fig. 2 by executing a control program 121 in Fig. 1.

[0047] The image acquisition unit 111 acquires an image captured by the camera 180. The image acquisition unit 111 sends the acquired image to the human detection unit 112 and the object detection unit 113.

[0048] The person detection unit 112 receives an image (input image) from the image acquisition unit 111. The person detection unit 112 detects people from the input image. The person detection unit 112 sends the detected person recognition information to the correction unit 114.

[0049] The object detection unit 113 receives an image (input image) from the image acquisition unit 111. The object detection unit 113 detects an object from the input image. The object detection unit 113 sends the detected object recognition information to the correction unit 114.

[0050] The correction unit 114 corrects the object recognition information detected by the object detection unit 113. Various correction modes are possible for the correction unit 114, and these various correction modes will be described later.

[0051] The determination unit 115 determines a human behavior based on the object recognition information corrected by the correction unit 114 and the human recognition information detected by the human detection unit 112.

[0052] Here, the various correction modes in the correction unit 114 are, for example, that the correction unit 114 can correct the likelihood of an object class related to the object recognition information. The correction unit 114 may also replace the object recognition information with object recognition information when an object was detected in the past. The correction unit 114 may also correct the likelihood of an object class related to the object recognition information so as to detect an undetected object. The correction unit 114 may also correct the object recognition information under the condition of the approach between a person and an object, or the occlusion of an object by a person. That is, the correction unit 114 may correct the likelihood related to an object or replace it with object recognition information when an object was detected in the past under the condition of the approach between a person and an object, or the occlusion of an object by a person. The correction unit 114 may also correct the object recognition information according to time or lighting conditions. That is, the correction unit 114 may correct the likelihood of an object class related to each object recognition information or replace it with object recognition information when an object was detected in the past according to time or lighting conditions. Furthermore, correction unit 114 may correct the object recognition information using the stability of the detection result in frames over a certain period of object detection by object detection unit 113. That is, correction unit 114 may replace the object recognition information with object recognition information when an object was detected in the past, or correct the likelihood of the object class related to the object recognition information, using the stability of the detection result in frames over a certain period of object detection by object detection unit 113. Note that the replacement of object recognition information is included in the correction of object recognition information.

[0053] The detection target detected by the object detection unit 113 may be an object used by a person in a sitting position. Also, the detection target detected by the object detection unit 113 may be an object used by a person in a lying position.

[0054] The human detection unit 112 may detect the posture of a person, and the determination unit 115 may determine the behavior of the person based on the posture information detected by the human detection unit 112 and the object recognition information corrected by the correction unit 114.

[0055] Correction section 114 may vary the correction conditions or the correction contents depending on the position coordinates of a person or object in the input image.

[0056] [Processing of information processing device] Next, the processing of the CPU 110 of the image processing device 100 according to this embodiment will be described. FIG. 3 is a flowchart showing a process in which the image processing device 100 according to this embodiment corrects the object recognition information and determines a human action based on the object recognition information and the posture information.

[0057] First, camera 180 is attached so as to look down from above in a room, for example, and captures an image of a user and objects in the room. Camera 180 sends the captured image to image acquisition unit 111 of CPU 110.

[0058] The human detection unit 112 detects the posture of a person from an image (input image) input by the camera 180, and sets the detected posture as posture information (step S001).

[0059] The human detection unit 112 acquires, for example, circumscribed rectangle coordinates, coordinates of each joint, human detection likelihood, and posture class likelihood as posture information indicated by the human detection result. The human detection unit 112 acquires, for example, the human detection likelihood as "0.8." Here, the human detection likelihood is a scale indicating plausibility, with the likelihood decreasing as the likelihood approaches 0, and increasing as the likelihood approaches 1. In addition, the human detection unit 112 acquires, for example, "0.1" for standing, "0.7" for sitting, and "0.2" for lying down as posture information indicating posture class likelihood.

[0060] Next, object detection unit 113 detects an object from the image (input image) input by camera 180, and sets the detected object as object recognition information (step S003).

[0061] For example, the object detection unit 113 acquires circumscribed rectangle coordinates, object detection likelihood, and object class likelihood as object recognition information indicated by the object detection result of detecting an object. For example, the object detection unit 113 acquires the object detection likelihood as "0.5" and the object class likelihood as "0.4" for a desk, "0.3" for a chair, and "0.3" for a bed.

[0062] The correction unit 114 corrects the object recognition information detected by the object detection unit 113 (step S005).

[0063] For example, when a person is close to an object detection (overlapping state) and the person is in a sitting posture, the correction unit 114 increases the likelihood of an object to sit on, such as a chair. Specifically, the likelihood of a chair is corrected from "0.3" to "0.8" as an object class likelihood. Since it is considered that the object detection unit 113 has erroneously detected a chair as a desk, the correction unit 114 corrects the object recognition information indicating the object detection result. That is, the correction unit 114 can correct the likelihood of an object class related to the object recognition information.

[0064] The determination unit 115 determines the human behavior based on the object recognition information corrected by the correction unit 114 and the human recognition information detected by the human detection unit 112 (step S007). In this embodiment, when the determination unit 115 of the CPU 110 determines the human behavior, the process shown in FIG. 3 is terminated.

[0065] As described above, the CPU 110 of the image processing device 100 according to this embodiment embodies the human detection unit 112, the object detection unit 113, the correction unit 114, and the determination unit 115. The human detection unit 112 detects a person from the input image. The object detection unit 113 detects an object from the input image. The correction unit 114 corrects the object recognition information detected by the object detection unit 113. The determination unit 115 determines the behavior of a person based on the object recognition information corrected by the correction unit 114 and the human recognition information detected by the human detection unit 112.

[0066] As a result, the CPU 110 of the image processing device 100 according to this embodiment can correct even erroneous detection results of objects. That is, the image processing device 100 according to this embodiment can correct the object detection results using other detection results or preset knowledge, and can use the corrected object detection results for human behavior recognition.

[0067] Furthermore, the CPU 110 can cause the correction unit 114 to correct the likelihood of the object class related to the object recognition information.

[0068] <Identifying detection targets> The CPU 110 of the image processing device 100 according to this embodiment can detect a detection target detected by the object detection unit 113 as an object used by a person in a sitting position or an object used by a person in a lying position.

[0069] 4 is a flowchart showing a process in which the CPU 110 of this embodiment corrects the likelihood of an object class by the correction unit 114 based on the posture of a person detected by the person detection unit 112. This process corresponds to the process of step S005 in FIG.

[0070] As shown in FIG. 4, CPU 110 determines, by correction unit 114, whether the posture of the person detected by person detection unit 112 is a sitting position or not (step S011). If the posture of the person is a sitting position (Yes in step S011), correction unit 114 proceeds to step S013. In step S013, correction unit 114 increases the object class likelihood related to a chair or a wheelchair. In this case, correction unit 114 corrects the likelihood of a chair from "0.3" to "0.8", for example. Then, CPU 110 proceeds to step S015. Here, the object for which correction unit 114 increases the object class likelihood is not limited to a chair or a wheelchair, and may be any object associated with the sitting position, which is the posture of the person.

[0071] On the other hand, if the person is not in a sitting position (No in step S011), the CPU 110 proceeds to step S015.

[0072] In step S015, correction unit 114 determines whether the posture of the person detected by human detection unit 112 is a lying position. If the posture of the person is a lying position (Yes in step S015), correction unit 114 proceeds to step S0117 and increases the object class likelihood of a bed. In this case, correction unit 114 corrects the likelihood of a bed from "0.3" to "0.8", for example. Then, CPU 110 ends the process shown in FIG. 4.

[0073] Note that the object for which the correction unit 14 increases the object class likelihood is not limited to a bed, but may be any object associated with the lying position, which is the posture of the person.

[0074] On the other hand, if the person's posture is not the lying position (No in step S015), the correction unit 114 ends the process shown in FIG.

[0075] As described above, the image processing device 100 of this embodiment can correct the likelihood of the object class related to the object recognition information based on the person recognition information (posture information) detected by the person detection unit. In this way, the image processing device 100 can correct the likelihood of the object class related to the sitting position or lying position in steps S013 and S017 so as to detect object candidates that have been undetected by threshold judgment, for example. In addition, past detection results around a person may be restored to detect a chair or bed. Such a modified example will be described later with reference to FIG. 6.

[0076] <Example 1 of improved detection accuracy> The CPU 110 of the image processing device 100 according to this embodiment can correct the likelihood associated with an object using the correction unit 114, on the condition that a person approaches the object, or that an object is occluded by a person.

[0077] 5 is a flowchart showing a process in which CPU 110 of this embodiment corrects the object class likelihood related to an object approached or blocked by a person using correction unit 114. This process corresponds to the process of step S005 in FIG.

[0078] 5, CPU 110 uses correction unit 114 to compare the person recognition information with the object recognition information to identify object recognition information related to an object that a person has approached or blocked (step S021). Object detection unit 113 identifies object recognition information related to an object that a person has approached or blocked based on the distance between the center of gravity coordinates of the person recognition information and the object recognition information, or a threshold value of IoU (Intersection over Union) indicating an evaluation index for the person recognition information and the object recognition information.

[0079] CPU 110 increases the object class likelihood in this object recognition information by correction unit 114 (step S023), and ends the process shown in Fig. 5. This makes it possible to complement disturbances in object detection caused by the approach or occlusion of a person.

[0080] <Example 2 of improved detection accuracy> The CPU 110 of the image processing device 100 according to this embodiment can correct, by the correction unit 114, object recognition information relating to an object that a person has approached or blocked, by replacing it with past object recognition information.

[0081] 6 is a flowchart showing a process in which CPU 110 of this embodiment corrects object recognition information related to an object that a person approaches or blocks, using correction unit 114. This process corresponds to the process of step S005 in FIG.

[0082] As shown in FIG. 6, CPU 110 causes correction unit 114 to compare person recognition information with object recognition information to identify object recognition information relating to an object that a person has approached or blocked (step S031).

[0083] CPU 110 causes correction unit 114 to replace this object recognition information with object recognition information when this object was detected in the past (step S033), and ends the process shown in FIG.

[0084] Fig. 7 is an image diagram showing the concept of resolving the dilemma of behavior recognition by using information on both people and objects. Note that Fig. 7 shows an example of the processing result of Fig. 4 or Fig. 5.

[0085] As shown in Fig. 7, a chair CH and a bed BD are provided in a room, with a person HM1 sitting in the chair CH and a person HM2 lying on the bed BD. In behavior recognition that uses information on both people and objects, contact between people and objects occurs, but detection accuracy can decrease if the object is hidden by the person.

[0086] Therefore, when the person is in a sitting or lying position, the CPU 110 uses the person's posture and past results to maximize the detection of chairs CH and beds BD around the person.

[0087] For example, according to the process shown in Fig. 4, since the posture of the person HM1 is sitting, the object class likelihood of a chair, etc. related to the object recognition information of the surroundings is increased. Also, since the posture of the person HM2 is lying down, the object class likelihood of a bed, etc. related to the object recognition information of the surroundings is increased. This makes it possible to preferably detect chairs and beds.

[0088] Fig. 8 is an image diagram showing the concept of correcting the current bed position using past bed position information. Note that Fig. 8 is an example of the processing result of Fig. 6.

[0089] In FIG. 8, the diagram on the left shows a past recognition result RS1, the diagram in the center shows a current recognition result RS2, and the diagram on the right shows a corrected current recognition result RS3.

[0090] In the past recognition result RS1, a chair CH and a bed BD are provided in the room, and curtains CT are hung on the windows. A person HM has just entered the room and is standing in front of the door DR. This recognition result RS1 shows object recognition information 311, which is the result of detecting the chair CH, and object recognition information 312, which is the result of detecting the bed BD.

[0091] After the image processing device 100 outputs the recognition result RS1, the person HM opens the curtains CT in the room and sits on the chair CH. Then, the image processing device 100 outputs the next recognition result RS2.

[0092] In the room in the current recognition result RS2, sunlight is entering because the curtains CT are open. As a result, the bed BD is not recognized as a bed, and instead the sunlight by the window is detected as the object recognition information 314 relating to the bed. In other words, depending on the time of day and lighting conditions, there may be times when the accuracy of object detection and human behavior recognition is low.

[0093] Furthermore, in this room, the chair CH is hidden by the person HM1, and is therefore erroneously recognized as a desk in the object recognition information 313. As a result, the person HM1 is erroneously recognized as sitting at the desk. In other words, the accuracy of the object detection results when a person is in contact with an object tends to be low because the object is hidden.

[0094] Therefore, as shown in the diagram on the right, the correction unit 114 replaces the current recognition result RS2 for the object recognition information with, for example, the object recognition information 311, 312 of the recognition result RS1 at the time when the person HM entered the room, to obtain the current recognition result RS3. The current position of the bed BD is corrected so that the bed BD can be correctly detected based on the past object recognition information 311, and the chair CH is corrected so that it can be correctly detected based on the past object recognition information 312. This enables the image processing device 100 to accurately recognize the behavior of the person HM1 sitting on the chair CH.

[0095] 9 is a flowchart showing a process in which the CPU 110 of this embodiment replaces object recognition information with predetermined object recognition information by the correction unit 114. This process corresponds to the process of step S005 in FIG.

[0096] As shown in FIG. 9, CPU 110 determines whether the current time is daytime or not using correction unit 114 (step S041). If the current time is daytime (Yes in step S041), correction unit 114 proceeds to step S043, replaces the object recognition information with object recognition information detected at night, and proceeds to step S045. Since the accuracy of object detection decreases during the day due to sunlight, correction unit 114 performs correction according to the time period. In this way, correction unit 114 can compensate for the decrease in accuracy of object detection. Note that the current time is not limited to whether it is daytime or not, and correction unit 114 may replace the recognition result of any time period with object recognition information that was suitably detected in the past.

[0097] On the other hand, if the current time is not daytime (No in step S041), the correction unit 114 proceeds to step S045.

[0098] In step S045, the correction unit 114 determines whether the luminance value around the person is equal to or greater than the threshold. If the luminance value around the person is equal to or greater than the threshold (Yes in step S045), the correction unit 114 proceeds to step S047, replaces the luminance value around the person with the object recognition information detected when the luminance value around the person is equal to or less than the threshold, and ends the process shown in FIG.

[0099] On the other hand, if the luminance value around the person is not equal to or greater than the threshold value (No in step S045), the correction unit 114 ends the process shown in FIG.

[0100] Note that, instead of determining whether the luminance value around the person is equal to or greater than a threshold value, correction unit 114 may determine whether the luminance value around the person is brighter than a predetermined range or darker than a predetermined range.

[0101] Fig. 10 is an image diagram showing the concept of correcting a current recognition result RS5 using a past recognition result RS4. Note that Fig. 10 shows an example of the processing result of Fig. 9.

[0102] In FIG. 10, the diagram on the left shows a past recognition result RS4, the diagram in the center shows a current recognition result RS5, and the diagram on the right shows a corrected current recognition result RS6.

[0103] In the room in the past recognition result RS4, a chair CH and a bed BD are provided. A person HM2 is lying in the bed BD. This recognition result RS4 shows object recognition information 321 which is a result of detecting the chair CH, and object recognition information 322 which is a result of detecting the bed BD.

[0104] After the image processing device 100 outputs the recognition result RS4, the person HM stands up, opens the curtains CT in the room, and sits down on the chair CH. Then, the image processing device 100 outputs the next recognition result RS5.

[0105] In the room in the current recognition result RS5, sunlight is entering because the curtains CT are open. As a result, the bed BD is not recognized as a bed, and instead the sunlight by the window is detected as the object recognition information 324 relating to the bed. In other words, there may be times when the object detection accuracy is low, depending on the time of day and lighting conditions.

[0106] Furthermore, the chair CH in this room is hidden by the person HM1, and is therefore erroneously recognized as a desk in the object recognition information 323. As a result, the person HM1 is erroneously recognized as sitting at the desk. In other words, the accuracy of the object detection results when a person is in contact with an object tends to be low because the object is hidden.

[0107] Therefore, as shown in the diagram on the right, the correction unit 114 corrects the object recognition information by replacing the current recognition result RS5 with the object recognition information 321, 322 of the past recognition result RS4. Correction is performed so that the bed BD can be correctly detected from the past object recognition information 321, and correction is performed so that the chair CH can be correctly detected from the past object recognition information 312. This enables the image processing device 100 to accurately recognize the behavior of the person HM1 sitting on the chair CH.

[0108] Also, instead of replacing the object recognition information, the object class likelihood may be corrected.

[0109] 11 is a flowchart showing a process in which the CPU 110 of this embodiment corrects the object class likelihood related to the object recognition information by the correction unit 114. This process corresponds to the process of step S005 in FIG.

[0110] 11, CPU 110 determines whether the current time is daytime or not (step S051) by correction unit 114. If the current time is daytime (Yes in step S051), correction unit 114 proceeds to step S053, where it increases the object class likelihood associated with each piece of object recognition information, and proceeds to step S055.

[0111] On the other hand, if the current time is not daytime (No in step S051), the correction unit 114 proceeds to step S055.

[0112] In step S055, the correction unit 114 determines whether the luminance value around the person is equal to or greater than the threshold value. If the luminance value around the person is equal to or greater than the threshold value (Yes in step S055), the correction unit 114 proceeds to step S057, increases the object class likelihood associated with each piece of object recognition information, and ends the process shown in FIG.

[0113] On the other hand, if the luminance value around the person is not equal to or greater than the threshold value (No in step S055), the correction unit 114 ends the process shown in FIG.

[0114] In addition, the CPU 110 according to this embodiment may use the stability of the detection results of object detection by the object detection unit 113 over a certain period of frames to replace the detection results with object recognition information from when an object was detected in the past, via the correction unit 114.

[0115] 12 is a flowchart showing a process in which CPU 110 of this embodiment replaces the detection result with object recognition information when an object was detected in the past, using the stability of the detection result in frames over a certain period of time of object detection by object detection unit 113, by correction unit 114. This process corresponds to the process of step S005 in FIG.

[0116] 12, CPU 110 causes correction unit 114 to associate objects that are close to each other in each frame as the same object (step S071). Then, CPU 110 causes correction unit 114 to determine whether the variance of the object detection region size is equal to or smaller than a predetermined value (step S073).

[0117] If the variance of the object detection region size is equal to or smaller than the predetermined value (Yes in step S073), correction unit 114 proceeds to step S075, replaces this object recognition information with a result when the variance is small, and proceeds to step S077.

[0118] On the other hand, if the variance of the object detection region size is greater than the predetermined value (No in step S073), correction unit 114 proceeds to step S77.

[0119] In step S077, CPU 110 determines whether or not the variance of the center of gravity position of the object detection is equal to or less than a predetermined value by correction unit 114. If the variance of the center of gravity position of the object detection is equal to or less than the predetermined value (Yes in step S077), correction unit 114 proceeds to step S079, replaces this object recognition information with a result when the variance is small, and ends the process shown in FIG.

[0120] On the other hand, if the variance of the center of gravity positions of the object detection is greater than the predetermined value (No in step S077), correction unit 114 ends the process shown in FIG.

[0121] In this embodiment, instead of replacing the object recognition information, the object class likelihood may be corrected. This modification is shown in FIG.

[0122] The CPU 110 according to this embodiment can use the correction unit 114 to correct the likelihood of the object class related to the object recognition information, using the stability of the detection result of the object detection by the object detection unit 113 in frames over a certain period of time.

[0123] 13 is a flowchart showing a process in which CPU 110 of this embodiment corrects the likelihood of an object class related to the object recognition information by using correction unit 114 the stability of the detection result in frames for a certain period of object detection by object detection unit 113. This process corresponds to the process of step S005 in FIG.

[0124] 13, CPU 110 causes correction unit 114 to associate objects that are close to each other in each frame as the same object (step S081). CPU 110 causes correction unit 114 to determine whether the variance of the object detection region size is equal to or smaller than a predetermined value (step S083).

[0125] If the variance of the object detection region sizes is equal to or smaller than the predetermined value (Yes in step S083), correction unit 114 proceeds to step S85, where it increases the object class likelihood related to each piece of object recognition information, and proceeds to step S087.

[0126] On the other hand, if the variance of the object detection region size is greater than the predetermined value (No in step S083), correction unit 114 proceeds to step S87.

[0127] In step S087, CPU 110 determines whether or not the variance of the center of gravity positions of the object detection is equal to or less than a predetermined value, using correction unit 114. If the variance of the center of gravity positions of the object detection is equal to or less than the predetermined value in step S087 (Yes in step S087), correction unit 114 proceeds to step S089, where it increases the object class likelihood related to each piece of object recognition information, and ends the process shown in FIG.

[0128] On the other hand, if the variance of the center of gravity positions of the object detection is greater than the predetermined value (No in step S087), correction unit 114 ends the process shown in FIG.

[0129] Furthermore, the correction unit 114 may vary the conditions for correction or the content of correction depending on the position coordinates of a person or object in the input image.

[0130] Fig. 14A is an image of the center of the room captured by camera 180 from the center of the ceiling. Fig. 14B is an explanatory diagram showing the image of the room as seen from the side.

[0131] 14A and 14B, for example, a case where a person is at position PT1 is compared with a case where a person is at position PT2. When a person is at position PT1, the person is directly below camera 180, so human area TA1 and chair area TA2 are captured overlapping at approximately the same position. On the other hand, when a person is at position PT2, the person is located away from camera 180, so human area TA3 appears to be shifted in the line of sight of camera 180 compared to chair CH area TA4.

[0132] Therefore, the correction unit 114 may change the correction conditions and the correction contents according to the change in the appearance of the image. For example, if a person overlaps an object directly under the camera 180, the object that the person overlaps may be recognized using past object recognition information. This makes it possible to recognize the object even if it is completely hidden by the person.

[0133] Also, if a person overlaps with an object at a position away from the camera 180, the object may be recognized by increasing the object class likelihood. This makes it possible to correctly recognize the object even if it is moved.

[0134] <Other embodiments> Although the present embodiment has been realized by an image processing system equipped with the image processing device 100 and the camera 180 indoors, the present invention is not limited to this. For example, the image processing system may be configured such that a server on a cloud that functions as the image processing device 100 and the camera 180 are connected via a network.

[0135] Fig. 15 is a block diagram showing an image processing system of another embodiment. The image processing system of another embodiment shown in Fig. 15 differs from the image processing system shown in Fig. 2 in that an image processing device 100, which is a server on the cloud, is connected to a camera 180 installed on the ceiling of a room via a network. The same components are given the same reference numerals and the description will be omitted.

[0136] 15, in the image processing system of another embodiment, a camera 180 (imaging unit) and an image processing device 100, which is a server on the cloud, are connected via a network. As a result, the image processing system can include a human detection unit 112 that detects a person from an input image captured by the camera 180, an object detection unit 113 that detects an object from the input image, a correction unit 114 that corrects the contents indicated by the object recognition information detected by the object detection unit 113, and a determination unit 115 that determines the behavior of a person based on the corrected object recognition information and the human recognition information detected by the human detection unit 112.

[0137] As a result, in the image processing systems of other embodiments, the object detection results can be corrected with other detection results or preset knowledge, and the corrected object detection results can be used for human behavior recognition. [Explanation of symbols]

[0138] 100 Image processing device 110 CPU 111 Image acquisition unit 112 Human detection unit 113 Object detection unit 114 Correction section 115 Judgment section 120 Storage section 121 Control Program 130 ROM 140 RAM 150 Input section 160 Display section 170 Communications Department 180 Camera

Claims

1. a person detection unit that detects a person from an input image; an object detection unit that detects an object from the input image; a correction unit that corrects the object recognition information detected by the object detection unit; a determination unit that determines a behavior of the person based on the object recognition information corrected by the correction unit and the person recognition information detected by the person detection unit, The correction unit is correcting the likelihood of an object class related to the object recognition information; 13. An image processing device comprising:

2. A person detection unit that detects a person from an input image; an object detection unit that detects an object from the input image; a correction unit that corrects the object recognition information detected by the object detection unit; a determination unit that determines a behavior of the person based on the object recognition information corrected by the correction unit and the person recognition information detected by the person detection unit, The correction unit is The object recognition information is replaced with object recognition information when the object was detected in the past.

13. An image processing device comprising:

3. The correction unit is correcting the likelihood of an object class related to the object recognition information so as to detect an undetected object; 3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

4. The correction unit is correcting the object recognition information on the condition that the person approaches the object or the object is occluded by the person; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

5. The correction unit is correcting the object recognition information according to time or lighting conditions; 5. The image processing device according to claim 1,

6. The correction unit is correcting the object recognition information using stability of a detection result in frames of a certain period of object detection by the object detection unit; 6. The image processing device according to claim 1,

7. The detection target detected by the object detection unit is It is used by a person in a seated position, 7. The image processing device according to claim 1,

8. The detection target detected by the object detection unit is It is used by a person in a lying position, 8. The image processing device according to claim 1,

9. The human detection unit further detects a posture of a person, The determination unit determines the behavior of the person based on the posture information detected by the person detection unit and the object recognition information corrected by the correction unit. The image processing device according to claim 1 .

10. The correction unit is The correction conditions or the correction contents vary depending on the position coordinates of the person or the object in the input image.

10. The image processing device according to claim 1,

11. An imaging unit that captures the input image and is attached so as to look down from above, The image processing device according to claim 1 , further comprising:

12. A step of detecting a person from an input image and obtaining person recognition information; detecting an object from the input image and setting the object as object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the step of correcting, correcting the likelihood of an object class related to the object recognition information; 13. An image processing method comprising:

13. A step of detecting a person from an input image and obtaining person recognition information; detecting an object from the input image and setting the object as object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the step of correcting, The object recognition information is replaced with object recognition information when the object was detected in the past.

13. An image processing method comprising:

14. A procedure for detecting a person from an input image and obtaining person recognition information; A step of detecting an object from the input image and generating object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the correction step, correcting the likelihood of an object class related to the object recognition information; A program for causing a computer to execute the above.

15. A step of detecting a person from an input image and obtaining person recognition information; A step of detecting an object from the input image and generating object recognition information; correcting the object recognition information; determining a behavior of the person based on the corrected object recognition information and the person recognition information; In the correction step, replacing the object recognition information with object recognition information when the object was detected in the past; A program for causing a computer to execute the above.

16. An imaging unit; a person detection unit that detects a person from an input image captured by the imaging unit; an object detection unit that detects an object from the input image; a correction unit that corrects content indicated by the object recognition information detected by the object detection unit; a determination unit that determines the behavior of the person based on the corrected object recognition information and the person recognition information detected by the person detection unit, The correction unit is correcting the likelihood of an object class related to the object recognition information; 1. An image processing system comprising:

17. An imaging unit; a person detection unit that detects a person from an input image captured by the imaging unit; an object detection unit that detects an object from the input image; a correction unit that corrects content indicated by the object recognition information detected by the object detection unit; a determination unit that determines the behavior of the person based on the corrected object recognition information and the person recognition information detected by the person detection unit, The correction unit is The object recognition information is replaced with object recognition information when the object was detected in the past.

1. An image processing system comprising:

Images