Image processing device and computer program

Abstract

To minimize a decrease in detection accuracy due to the fact that a portion of a detection target is hidden when detecting the state of the detection target appearing in an image.SOLUTION: An image processing device 10 is provided, comprising: an image acquisition unit 20 for acquiring an input image captured by a camera 11; a state detection unit 21 for detecting a predetermined state of a moving object appearing in the input image; a concealment level computation unit 22 for computing the concealment level of the moving object; and a determination unit 23 configured to determine reliability of a detection result of the predetermined state by the state detection unit 21 on the basis of the concealment level computed by the concealment level computation unit 22.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a computer program. [Background technology]

[0002] Conventionally, technologies for detecting the state (posture, behavior, etc.) of a detection target captured in an image have been proposed. For example, Patent Document 1 below discloses a security system that identifies suspicious behavior of a person captured on a surveillance camera. In this security system, if an occlusion occurs in a person area in a camera image where a person exists, the system prohibits suspicious behavior determination processing for the occluded person area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-086471 Summary of the Invention [Problem to be solved by the invention]

[0004] If a part of the detection target is occluded by something in an image showing the detection target, the accuracy of detecting the state of the detection target may decrease. For example, even if the person being detected is not in a sitting position (e.g., a sitting position such as bending, squatting, or kneeling), the state of the detection target may be erroneously detected as sitting. Furthermore, if the determination process is prohibited when occlusion occurs, as in Patent Document 1, it becomes impossible to detect the state of the detection target at all. The present invention aims to suppress a decrease in detection accuracy caused by occlusion of a part of a detection target when detecting the state of the detection target shown in an image. [Means for solving the problem]

[0005] An image processing device according to one embodiment of the present invention includes an image acquisition unit that acquires an input image captured by a camera, a mode detection unit that detects a predetermined mode of a moving object captured in the input image, an obscuration degree calculation unit that calculates the obscuration degree of the moving object, and a determination unit that determines the reliability of the detection result of the predetermined mode by the mode detection unit based on the obscuration degree calculated by the obscuration degree calculation unit. [Effects of the Invention]

[0006] According to the present invention, when detecting the state of a detection target appearing in an image, it is possible to suppress a decrease in detection accuracy due to the part of the detection target being occluded. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic configuration diagram of an example of an image processing apparatus according to an embodiment; [Figure 2] FIG. 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to an embodiment. [Figure 3] FIG. 10 is an explanatory diagram of an example of a method for detecting an occluded portion. [Figure 4] FIG. 10 is a diagram illustrating an example of setting a determination threshold for each type of state. [Figure 5] FIG. 10 is a diagram illustrating an example of setting an influence coefficient for each mode type. [Figure 6] 1A is a schematic diagram of an example of the optical axis direction and angle of view of a unidirectional camera, and FIG. 1B is a schematic diagram of the direct image area in an image captured by the unidirectional camera. [Figure 7] 1A is a schematic diagram of an example of the optical axis direction and angle of view of an omnidirectional camera, and FIG. 1B is a schematic diagram of the direct image area in an image captured by the omnidirectional camera. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0009] (First embodiment) 1 is a schematic configuration diagram of an example of an image processing device according to an embodiment. The image processing device 10 detects the type of appearance of a detection target in a monitored space 1. As an example, assume that a person 2 and a structure 5 exist in the monitored space 1. In this specification, a case where the type of appearance of the person 2 as a detection target existing in the monitored space 1 is detected is exemplified.

[0010] The image processing device 10 detects the posture and behavior of the person 2 as the type of appearance of the person 2. The type of appearance to be detected includes detection based on the posture and behavior of the person 2 alone, and detection based on the postures and behavior of multiple people. The image processing device 10 detects abnormal posture and abnormal behavior of the person 2 as an abnormal appearance of the person 2 that notifies a monitor or the like of the monitored space 1 that an abnormal situation has occurred.

[0011] Examples of postures and behaviors to be detected as abnormal include "threatening (a person raising their arms, etc.)," "fighting (a person hitting another person with their fist, etc.)," "destruction (a person swinging their arms, etc.)," "falling (a person who has collapsed from a standing position and changed to a lying position)," "holding up (a person who continues to raise both hands due to being threatened, etc.)," "shoving (a person who has extended their arms toward another person (e.g., a person pointing a weapon, etc. at another person))," "crouching (a person who continues to crouch with their knees or waist bent)," "dogeza (a person who bows to another person)," "crouching (a person who is prostrating and bowing to another person)," "crouching (a person who is lying on their stomach)," and "dogeza (a person who is lying down and sleeping)." Note that "crouching" and "dogeza" are examples of a "first posture type classified as a sitting position" as defined in the claims, and "threatening" and "holding up" are examples of a "second posture type classified as a standing position."

[0012] Furthermore, a type of state other than the abnormal state may be detected. For example, postures such as "standing," "standing upright," and "sitting," or actions such as "walking" and "running" may be detected. The image processing device 10 may detect the state type of a detection target other than the person 2. For example, the detection target may be a machine having a manipulator or a non-humanoid robot. For example, the image processing device 10 may detect the orientation of the manipulator or its posture, such as its bending or stretching state, as the state type, or may detect a movement, which is a change in posture, as the state type.

[0013] The image processing device 10 includes a camera 11, an input unit 12, a storage unit 13, a control unit 14, and an output unit 15. Of these, the storage unit 13 and the control unit 14 may be realized by a so-called computer, and the input unit 12 and the output unit 15 may be realized as peripheral devices of the computer. The camera 11 is placed in the monitored space 1 to generate images of objects present in the monitored space 1. The camera 11 may be a unidirectional camera with a horizontal angle of view and a vertical angle of view of about 90 degrees, for example. The camera 11 may also be an omnidirectional camera with a shooting area (monitoring area) in all directions (360 degrees). Note that the image processing device 10 may not be equipped with the camera 11 and may instead acquire images from an external shooting device.

[0014] The input unit 12 includes a user interface such as a keyboard, a mouse, etc. that is operated by a user to input data, etc. The input unit 12 is connected to the control unit 14, converts user operations into operation signals, and outputs the signals to the control unit 14. The input unit 12 may also include a DVD (Digital Versatile Disc) drive and a USB (Universal Serial Bus) interface. The input unit 12 inputs data to the control unit 14 as a file, and outputs data from the control unit 14 as a file. The storage unit 13 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. The storage unit 13 is connected to the control unit 14, and inputs and outputs this information to and from the control unit 14.

[0015] The control unit 14 is composed of arithmetic devices such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an MCU (Micro Control Unit). The control unit 14 is connected to the storage unit 13, and operates as various processing units by reading and executing computer programs from the storage unit 13, and stores and reads various data in the storage unit 13. The functions of the image processing device 10 described below are realized by the control unit 14 executing computer programs stored in the storage unit 13.

[0016] The output unit 15 is connected to the control unit 14 and outputs the detection result of the mode type by the control unit 14. The output unit 15 may include a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The output unit 15 may also include an audio signal output device such as a speaker or a buzzer. The output unit 15 may also include a network interface or the like that transmits and receives data between the image processing device 10 and an external device via wired or wireless communication.

[0017] The control unit 14 acquires a captured image generated by the camera 11 by capturing an image of the person 2 present in the monitored space 1, and detects the type of appearance of the person 2 appearing in the captured image. In this case, if a part of person 2 is hidden from the view of camera 11 and this part does not appear in the captured image generated by camera 11, there is a risk that the accuracy of detecting the appearance of person 2 will decrease depending on the degree to which person 2 is hidden from the view of camera 11 (hereinafter, this may be simply referred to as "degree of hiding"). Note that in this specification, "hiding" is explained to mean not only intentionally hiding a part of person 2, but also accidentally hiding a part of person 2.

[0018] For example, actions such as "kowtowing," "bending," and "crouching" tend to result in a greater decrease in detection accuracy compared to other actions (such as "holding up"), even if the degree of concealment is the same. Therefore, the image processing device 10 of the embodiment detects the type of appearance of the person 2 appearing in the captured image 3, calculates the degree of concealment of the person 2, and determines the reliability of the detected type of appearance based on the detected type of appearance and the calculated degree of concealment.

[0019] 2 is a block diagram showing an example of the functional configuration of the image processing device 10 according to the embodiment. The image processing device 10 includes the camera 11 and output unit 15, an image acquisition unit 20, a state detection unit 21, an obscuration degree calculation unit 22, and an output determination unit 23.

[0020] The image acquisition unit 20 acquires an image generated by the camera 11 by capturing an image of an object present in the monitored space 1 as an input image. The manner detection unit 21 detects the manner type of the person 2 appearing in the input image acquired by the image acquisition unit 20. For example, the manner detection unit 21 may detect the type of behavior of the person 2 as the manner type. In this case, the manner detection unit 21 first detects the posture of the person 2 appearing in the input image.

[0021] For example, the posture detection unit 21 may estimate the posture of person 2 by inputting the input image into a learning model (hereinafter, sometimes referred to as "posture detection AI (Artificial Intelligence)") generated by machine learning, which uses image data as input and postures of people appearing in the image data as output data. For example, the posture detection AI may identify the positions of joint points as joint information of person 2 appearing in the input image, and estimate the posture of person 2 based on feature quantities of the positions of the joint points. Note that the posture may also be estimated based on skeletal information (position of the skeleton) of person 2 as joint information.

[0022] Furthermore, for example, the manner detection unit 21 may detect the posture of the person 2 by means other than a learning model (for example, a rule base, etc.). For example, the manner detection unit 21 may extract a silhouette image of the person 2 by a background subtraction method, and detect the posture by obtaining a score from the similarity between the extracted silhouette image and silhouette images of each posture registered in advance, or may extract the positions of the person's joint points from the silhouette of the person extracted by the background subtraction method, and detect the posture by obtaining a score from the similarity between the positions of the joint points of each posture registered in advance and the positions of the extracted joint points of the person.

[0023] Next, the manner detection unit 21 determines the behavior of the person 2 based on the time-series change information of the detected posture. For example, the manner detection unit 21 may determine the behavior of the person 2 using a rule base based on the time-series change information of the detected posture. For example, if the posture of the person 2 changes in the order of standing upright, collapsing, and lying down, the type of behavior of the person 2 may be determined to be a falling behavior. The manner detection unit 21 may output a "certainty" that is the degree of accuracy (correctness probability) of the detected posture. For example, the type of the detected behavior and its certainty may be output, such as "the probability that it is a falling behavior is 80%."

[0024] Furthermore, for example, the behavior detection unit 21 may determine the behavior of each frame of the input image based on time-series change information of the posture between the target frame and the previous frame, and detect the type of behavior of person 2 by majority vote of the postures detected in each frame for the same person appearing in multiple frame images. Furthermore, the manner detection unit 21 may detect the type of behavior of the person 2 using a learning model generated by machine learning that uses time-series information of moving images and joint points as input and person behavior as output data.

[0025] The manner detection unit 21 may output not only a single manner type but also a plurality of manner types and their certainty levels (for example, "standing upright" 90%, "walking" 80%, etc.). Furthermore, for example, the manner detection unit 21 may detect, as the manner type, the type of posture of the person 2. In this case, the manner detection unit 21 may input an input image to a posture detection AI to estimate the posture of the person 2, or may detect the posture of the person 2 by means other than the posture detection AI (for example, a rule base, etc.).

[0026] The concealment degree calculation unit 22 calculates the concealment degree, which is the degree to which the person 2 is concealed from the field of view of the camera 11. For example, the concealment degree calculation unit 22 calculates the proportion of the parts of the person 2 (head, neck, wrists, elbows, shoulders, etc.) that are concealed from the field of view of the camera 11 as the concealment degree. For example, the concealment degree calculation section 22 may calculate the degree to which the joint points of the person 2 are concealed as each part of the person 2 as the concealment degree.

[0027] For example, the concealment degree calculation unit 22 may calculate, as the concealment degree, the proportion of joint points that are hidden from the field of view of the camera 11 (i.e., invisible from the camera 11) among the joint points of the entire body of the person 2. For example, if the total number of joint points of the entire body is 14 and the number of joint points that are invisible from the camera 11 is 2, the concealment degree may be calculated as 2 / 14=0.14. In the following description, a portion hidden from the view of the camera 11 may be referred to as a "hidden portion."

[0028] For example, the concealment degree calculation unit 22 may detect areas obscured by fixed obstacles, areas obscured by moving objects (e.g., other people), or areas of person 2 obscured by person 2's own body (areas obscured by so-called self-occlusion). For example, when detecting a part obscured by a fixed obstacle, the obscuration degree calculation unit 22 may store in advance position information of the fixed obstacle such as the structure 5 in the monitored space 1, and determine which part of the person 2 is obscured based on the positional relationship between the position of each part of the person 2 and the position of the obstacle. For example, the image processing device 10 may store the position information of the fixed obstacle such as the structure 5 in the storage unit 13 of FIG. 1. The degree of concealment is not limited to the calculation based on the proportion of concealed parts as described above, but may be calculated based on the area proportion of the image region of the concealed human parts. For example, if the area of the concealed human parts on the image is estimated (calculated) to be 20% of the area of the whole body when the whole body is visible from the camera, the degree of concealment may be calculated to be 0.2.

[0029] The concealment degree calculation unit 22 may acquire position information of each part of the person 2 from the manner detection unit 21. The manner detection unit 21 may detect the position of each part of the person 2 when estimating the posture of the person 2, and output the position information to the concealment degree calculation unit 22. For example, the manner detection unit 21 may output position information of the joints of the person 2 to the concealment degree calculation unit 22. For example, when detecting parts obscured by a moving object, the obscuration degree calculation unit 22 may detect the moving object from the input image acquired by the image acquisition unit 20, and determine which parts of the person 2 are obscured based on the positional relationship between the positions of each part of the person 2 and the position of the obstacle.

[0030] Furthermore, for example, when detecting a part obscured by self-occlusion, the obscuration degree calculation unit 22 may detect the obscured part based on a learning model that has learned the positions of the parts of the person 2 and whether or not the parts are obscured. 3 is an explanatory diagram of another example of a method for detecting occluded parts due to self-occlusion. In Fig. 3, circles (e.g., ja, jb) indicate joints of person 2 detected by manner detection unit 21, and lines (e.g., c1, c2) connecting the joints indicate connections between the joint parts.

[0031] For example, when the posture detection unit 21 calculates the score of the connection relationship between joint parts using existing posture estimation software such as OpenPose, the obscuration calculation unit 22 may determine that the joint part jb is an obscured part when the connection c1 between the joint parts ja and jb is broken or the connection c1 is weak. Alternatively, for example, the concealment degree calculation unit 22 may track a joint using history information of the past joint, and predict whether the joint will be concealed. In this case, if three-dimensional part information is estimated, more accurate prediction is possible based on the anteroposterior relationship of the parts.

[0032] See Fig. 2. The output determination unit 23 determines the reliability of the detection result for the type of state detected by the state detection unit 21. As described above, when a part of the person 2 is hidden from the view of the camera 11, there is a risk that the accuracy of detecting the appearance of the person 2 will decrease depending on the degree of hiding.

[0033] Therefore, the output determination unit 23 determines the reliability of the result of the detection of the state type by the state detection unit 21 based on the degree of obscuration calculated by the degree of obscuration calculation unit 22 . For example, the output determination unit 23 may determine the reliability of the detection result of the mode type detected by the mode detection unit 21 based on parameters that are set in advance for each mode type and stored in the memory unit 13 and the degree of obscuration calculated by the degree of obscuration calculation unit 22.

[0034] As a method of determining the reliability of the state type detected by the state detection unit 21, for example, the output determination unit 23 may determine whether the state type detected by the state detection unit 21 is a false detection (whether the state type can be trusted). In this case, the storage unit 13 may store a determination threshold for each state type as the parameter. Fig. 4 is a diagram showing an example of setting the determination threshold for each state type. The determination threshold illustrated in Fig. 4 is set to "0.6" for the second state type "hold up" classified as a standing position, and set to "0.3" for the first state type "kneel" classified as a sitting position.

[0035] The output determination unit 23 may determine whether the mode type detected by the mode detection unit 21 is an erroneous detection or not, based on a comparison between a determination threshold corresponding to the mode type detected by the mode detection unit 21 and the degree of obscuration calculated by the degree-of-obscuration calculation unit 22. For example, if the degree of obscuration calculated by the degree-of-obscuration calculation unit 22 exceeds the determination threshold, the output determination unit 23 may determine that the mode type detected by the mode detection unit 21 is an erroneous detection, and if the degree of obscuration is equal to or less than the determination threshold, the output determination unit 23 may determine that the mode type detected by the mode detection unit 21 is an erroneous detection. For example, if the type of condition detected by the condition detection unit 21 is “hold-up” and the degree of concealment calculated by the degree of concealment calculation unit 22 is 0.4, it can be determined that the detection is not erroneous because the degree of concealment of 0.4 does not exceed the judgment threshold value of 0.6.

[0036] In addition, the first posture type classified as a sitting position tends to have a more significant decrease in detection accuracy than the second posture type classified as a standing position even if the degree of obscuration is the same. For this reason, the determination threshold for the first posture type classified as a sitting position may be set to a smaller value than the determination threshold for the second posture type classified as a standing position. By setting the determination threshold to a smaller value in this way, the reliability of the manner type detected by the manner detection unit 21 can be determined more strictly. Strict determination means that even if the degree of obscuration is low, it is determined to be a false detection. This prevents the person 2 from being determined to be a manner type that is not easy to detect.

[0037] See Fig. 2. The output determination unit 23 determines the output from the output unit 15 for an abnormal state among the state types detected by the state detection unit 21, depending on the determination result of the reliability of the state type detected by the state detection unit 21. For example, when the output determination unit 23 determines that the mode type detected by the mode detection unit 21 is a mode type previously stored in the storage unit 13 as an abnormal mode and that the mode type is not an erroneous detection, the output determination unit 23 outputs the mode type from the output unit 15. On the other hand, even if the mode type detected by the mode detection unit 21 is a mode type previously stored in the storage unit 13 as an abnormal mode, when the output determination unit 23 determines that the mode type is an erroneous detection, the output of the mode type is prohibited.

[0038] For example, the output determination unit 23 may determine an output destination device to which the output unit 15 outputs the mode type based on the determination result of the reliability of the mode type detected by the mode detection unit 21. For example, the output unit 15 may output the mode type determined not to be a false detection to an external device of the image processing device 10 via wired or wireless communication. The external device may be, for example, a center terminal or local monitoring terminal used by a monitor monitoring the monitored space 1, or a customer mobile terminal. For example, the output determination unit 23 may determine the mode of information to be output from the output unit 15 based on the determination result of the reliability of the mode type detected by the mode detection unit 21. For example, when the output unit 15 determines that the reliability is low (false detection or low certainty) due to a high degree of obscuration, the output unit 15 may add auxiliary information indicating that the reliability is low and output the information. For example, to visually identify the low reliability, the output may display a bounding box (a frame) of a color corresponding to the reliability surrounding the person.

[0039] The output determination unit 23 may switch to which of these terminal devices the mode type is to be output, depending on the determination result of the reliability of the mode type detected by the mode detection unit 21. For example, if the mode detection unit 21 determines that the mode type detected is not an erroneous detection, the output determination unit 23 may output the mode type to the center terminal or the local monitoring terminal, and if the mode detection unit 21 determines that the detection is an erroneous detection, the output determination unit 23 may output the mode type to the customer mobile terminal.

[0040] (Second embodiment) The storage unit 13 of the second embodiment stores, as the parameter, an influence coefficient indicating the susceptibility to the influence of concealment in the detection of the mode type for each mode type. The output determination unit 23 of the second embodiment determines the reliability of the mode type detected by the mode detection unit 21 based on the influence coefficient corresponding to the mode type detected by the mode detection unit 21, among the influence coefficients stored in the storage unit 13, and the concealment degree calculated by the concealment degree calculation unit 22. In the following description, the influence degree calculated from the influence coefficient corresponding to the mode type detected by the mode detection unit 21 and the degree of obscuration detected by the degree of obscuration calculation unit 22 may be simply referred to as the "degree of influence."

[0041] As a method for determining the reliability of the mode type detected by the mode detection unit 21, for example, it may be determined whether or not the mode type detected by the mode detection unit 21 is a false detection. For example, the output determination unit 23 may determine that the mode type detected by the mode detection unit 21 is a false detection (the mode type is unreliable) when the degree of influence exceeds a predetermined determination threshold, and may determine that the mode type detected by the mode detection unit 21 is not a false detection (the mode type is reliable) when the degree of influence is equal to or less than the determination threshold.

[0042] Fig. 5 is a diagram showing an example of setting the influence coefficient for each manner type. The influence coefficient shown in Fig. 5 is set to "0.2" for the second manner type "hold up" classified as a standing position, and "0.6" for the first manner type "kneel" classified as a sitting position. For example, the output determination unit 23 may multiply the degree of obscuration calculated by the degree of obscuration calculation unit 22 by an impact coefficient corresponding to the type of mode detected by the mode detection unit 21, and obtain the multiplication result (product) as the degree of influence.

[0043] For example, if the mode type detected by the mode detection unit 21 is "hold-up," the obscuration level calculated by the obscuration level calculation unit 22 is 0.4, and the predetermined judgment threshold is 0.1, the output determination unit 23 may multiply the obscuration level of 0.4 by the influence level of 0.2 to obtain a multiplication result of 0.4 × 0.2 = 0.08 as the influence level. By comparing the obtained influence level with a predetermined judgment threshold pre-stored in the storage unit 13, the output determination unit 23 determines whether the mode type detected by the mode detection unit 21 is a false positive. For example, because the influence level of 0.8 does not exceed the predetermined judgment threshold of 0.1, the output determination unit 23 determines that the mode type detected by the mode detection unit 21 is not a false positive.

[0044] As described above, the first state type classified as a sitting state tends to have a more significant decrease in detection accuracy than the second state type classified as a standing state. For this reason, the influence coefficient for the first state type classified as a sitting state may be set to a larger value than the influence coefficient for the second state type classified as a standing state. By setting the influence coefficient to a larger value in this way, the reliability of the state type detected by the state detection unit 21 can be strictly determined. In the above embodiment, the influence degree is calculated based on the concealment degree and the influence coefficient, but the reliability of the mode type detected by the mode detection unit 21 may be determined without calculating the influence degree. For example, for each combination of the concealment degree and the influence coefficient stored in advance, information indicating whether the mode type detected by the mode detection unit 21 is reliable or not (for example, whether the detection result is erroneous, what the degree of certainty is, etc.) may be associated and stored in advance in the storage unit 13, and the reliability of the mode type detected by the mode detection unit 21 may be determined based on the information.

[0045] (Variation) Modifications of the embodiments will be described below, which are applicable to both the first and second embodiments. (1) The output determination unit 23 may determine the reliability of the manner type detected by the manner detection unit 21 based on the above parameters (i.e., the determination threshold and the influence coefficient) that are modified according to at least one of the image position, which is the position of the person 2 on the input image, and the installation conditions of the camera 11.

[0046] The detection accuracy of the state type by the state detection unit 21 may be reduced depending on the image position of the person 2. For example, in a direct-below image area in which the vicinity of the area directly below the camera 11 is captured, the detection accuracy of the state type by the state detection unit 21 may be reduced. Figure 6(a) is a schematic diagram of an example of the optical axis OA direction and angle of view θf of the camera when the camera 11 is a unidirectional camera, and Figure 6(b) is a schematic diagram of the direct image area 3D in the captured image 3 of the unidirectional camera 11.

[0047] When the camera 11 is a unidirectional camera, the vertical angle of view θf of the camera 11 is approximately 70 to 90 degrees, and the camera 11 is installed above the monitored space 1, with the direction of the optical axis OA set to have a depression angle θd. In the captured image 3 of such a unidirectional camera 11, the nadir image area 3D is located at the bottom edge of the captured image 3.

[0048] Fig. 7(a) is a schematic diagram of an example of the direction of the optical axis OA of the camera and the shooting range when the camera 11 is an omnidirectional camera, and Fig. 7(b) is a schematic diagram of the direct-down image area 4D in the image 4 captured by the omnidirectional camera 11. The hatched area RF in Fig. 7(a) indicates the shooting range of the omnidirectional camera 11. The omnidirectional camera 11 has a shooting range in all directions (360 degrees), is installed above the monitored space 1, and the direction of the optical axis OA is directed directly downward. The image 4 captured by such an omnidirectional camera 11 is a circular image as a whole, and the direct-under image area 4D is located at the center of the captured image 4.

[0049] If Person 2 appears in the 3D or 4D direct image area, the lower half of Person 2's body will be hidden, making it easier for Person 2's posture to be mistakenly detected. For example, when posture detection AI is used to detect a person's posture from the position of each joint point, the posture detection AI will infer invisible joint points and detect the posture including these inferred joint points. If Person 2 appears in the 3D or 4D direct image area, it will often infer that the joint points of the lower half of the body are bent, resulting in a false detection of Person 2 kneeling even though they are not.

[0050] Therefore, the above parameters may be modified according to the image position, which is the position on the input image of person 2. For example, when person 2 is captured in the 3D or 4D direct-behind image region, the determination threshold may be modified to be smaller and the influence coefficient may be modified to be larger than when person 2 is captured in a region other than the 3D or 4D direct-behind image region. For example, the output determination unit 23 may modify the parameters depending on whether the person 2 appears in the direct-behind image areas 3D and 4D. Alternatively, the parameters modified depending on whether the person 2 appears in the direct-behind image areas 3D and 4D may be set in advance and stored in the storage unit 13.

[0051] Furthermore, when the camera 11 is a unidirectional camera, the parameters may be modified according to the depression angle θd of the optical axis OA of the camera 11, for example, as an installation condition of the camera 11. This is because the greater the depression angle θd, the more easily the lower body parts are concealed. For example, the larger the depression angle θd, the smaller the determination threshold value may be modified, or the larger the influence coefficient may be modified.

[0052] (2) The output determination unit 23 may correct the certainty of the mode type detected by the mode detection unit 21 as a method of determining the reliability of the mode type detected by the mode detection unit 21. The output determination unit 23 may output the certainty corrected by the output determination unit 23, or may switch the output destination device of the mode type detected by the mode detection unit 21 in accordance with the certainty corrected by the output determination unit 23.

[0053] The output determination unit 23 may correct the confidence level output by the manner detection unit 21 in accordance with the degree of obscuration calculated by the degree-of-obscuration calculation unit 22 . For example, the output determination unit 23 may correct the certainty factor output by the manner detection unit 21 so that the greater the degree of concealment, the smaller the certainty factor.

[0054] (3) The output determination unit 23 may output the detection result of the mode type by the mode detection unit 21 to an output destination according to the degree of obscuration calculated by the degree of obscuration calculation unit 22. For example, the output destination device to which the detection result of the mode type by the mode detection unit 21 is output from the output unit 15 may be switched according to the degree of obscuration. For example, if the degree of concealment is relatively small and the mode detection unit 21 determines that the mode type detected is not a false detection, the output determination unit 23 may output the mode type to both the remote center terminal and the local monitoring terminal. On the other hand, if the degree of concealment is relatively large and the mode detection unit 21 determines that the mode type detected is a false detection, the output determination unit 23 may output the mode type only to the local monitoring terminal. Also, for example, the type of device to which the output is to be sent may be switched depending on the degree of concealment. For example, the device to which the output is to be sent may be switched between a personal computer, a tablet device, and a smartphone depending on the degree of concealment.

[0055] (4) If the state detection unit 21 is unable to detect the same state type continuously for a predetermined duration or longer, the output determination unit 23 may determine that the state type detected by the state detection unit 21 is a false detection, and if the state detection unit 21 is able to detect the same state type continuously for a predetermined duration or longer, the output determination unit 23 may determine that the state type detected by the state detection unit 21 is not a false detection. The output determination section 23 may set the predetermined duration in accordance with the degree of obscuration calculated by the obscuration degree calculation section 22. For example, the output determination section 23 may set the predetermined duration to be longer as the degree of obscuration increases.

[0056] Similarly, the output determination unit 23 may determine whether the mode type detected by the mode detection unit 21 is a false detection, depending on whether the proportion of the time during which the mode detection unit 21 detected the same mode type in a predetermined period of time is less than a predetermined proportion threshold. The output determination unit 23 may set the predetermined proportion threshold depending on the degree of obscuration calculated by the obscuration degree calculation unit 22. For example, the output determination unit 23 may set a longer predetermined ratio threshold value as the degree of hiding increases.

[0057] (5) When detecting a manner type from multiple frames in an input image, the manner detection unit 21 may detect a manner type detected in a number of frames equal to or greater than a threshold among a predetermined number of consecutive frames as the manner type of person 2, and determine the reliability of the manner type detected by the manner detection unit 21. In the following description, the threshold value for the number of frames required to detect a manner type among a predetermined number of consecutive frames may be referred to as the “required frame number threshold value.”

[0058] For example, the threshold value of the number of required frames when the mode detection unit 21 detects a predetermined mode type may be set according to the degree of obscuration calculated by the degree-of-obscuration calculation unit 22. For example, the greater the degree of obscuration, the greater the threshold value of the number of required frames may be set. Also, for example, the threshold value of the number of required frames may be set when the degree of obscuration exceeds a predetermined threshold, and no threshold value may be set for the number of required frames for detecting a mode type when the degree of obscuration is equal to or less than the predetermined threshold.

[0059] (6) The manner detection unit 21 may determine the reliability of the posture of the person 2 detected by the posture detection AI according to the degree of obscuration calculated by the degree-of-obscuration calculation unit 22. That is, the manner detection unit 21 may be part of the output determination unit 23, and the manner detection unit 21 may detect the type of posture of the person and determine the reliability of the detected type of posture according to the degree of obscuration. As a method for determining the reliability of the detected posture, the posture detection unit 21 may determine whether the posture detected by the posture detection AI is correct depending on the degree of obscuration. For example, when the degree of obscuration is smaller than a threshold value stored in advance in the storage unit 13 and the detected posture is determined to be correct, the posture detection unit 21 may output the behavior detected based on time-series change information of the posture detected by the posture detection AI to the output determination unit 23 as the posture type of the person 2. In an embodiment in which the posture type output to the output determination unit 23 is a posture, the posture detected by the posture detection AI may be output to the output determination unit 23 as the posture type of the person 2.

[0060] If the degree of obscuration is greater than a threshold value stored in advance in the memory unit 13 and it is determined that the detected posture is incorrect, the posture detection AI is prohibited from using the detected posture to detect the behavior of person 2 and from outputting the detected posture as the state type of person 2. At this time, the posture detection unit 21 may determine whether the posture of the person 2 detected by the posture detection AI is correct or not, depending on the angles and movement amounts of joint points (skeleton) and the like recognized from the image of the person 2.

[0061] For example, the posture detection unit 21 may determine that the threatening posture detected by the posture detection AI is correct when the number of times the arms are swung down is equal to or greater than a threshold. In this case, the posture detection unit 21 may set a larger threshold for the number of times the arms are swung down as the degree of concealment increases. Furthermore, for example, the posture detection unit 21 may determine that the sitting posture detected by the posture detection AI is correct when the angle between the line connecting the waist and knee and the line connecting the knee and ankle is equal to or less than a threshold. In this case, the posture detection unit 21 may set a smaller angle threshold as the degree of concealment increases. Furthermore, for example, the posture detection unit 21 may determine that the fighting posture detected by the posture detection AI is correct if the amount of movement of the line connecting the shoulder and arm is equal to or greater than a threshold, or if the number of times that the line connecting the shoulder and elbow and / or the elbow and wrist moves away from and back to the periphery of person 2 is equal to or greater than a threshold. In this case, the posture detection unit 21 may set a larger threshold for the amount of movement or the number of times that the line moves away from and back to the periphery as the degree of obscuration increases.

[0062] Furthermore, for example, the manner detection unit 21 may determine the reliability of the posture detected by the posture detection AI by changing the confidence level of the posture detected by the posture detection AI. For example, the manner detection unit 21 may set the confidence level so that the greater the degree of obscuration, the lower the confidence level. For example, the manner detection unit 21 may set stricter requirements for obtaining the same confidence level as the degree of obscuration becomes higher. For example, if the posture of person 2 detected by the posture detection AI is a threatening posture, the posture detection unit 21 may determine the confidence level of the posture detected by the posture detection AI based on the number of times the person swings their arms down.The number of times required for the confidence level to reach 100% may be set to increase as the degree of obscuration increases.

[0063] (7) It should be noted that the reliability of the detected state type may be determined not only based on the "degree of concealment" but also on the degree to which the person 2 is revealed (exposed) from the field of view of the camera 11 (hereinafter, this may be simply referred to as "degree of expression"). In this case, the reliability is determined based on the opposite criteria to the "degree of concealment". For example, the smaller the degree of expression, the more likely it is that the detected state type is a false detection (the state type is unreliable). Furthermore, the confidence level may be corrected to be smaller as the degree of expression becomes smaller.

[0064] (Effects of the embodiment) (1) The image processing device 10 includes an image acquisition unit 20 that acquires an input image captured by the camera 11, a mode detection unit 21 that detects a predetermined mode of a moving object captured in the input image, an obscuration degree calculation unit 22 that calculates the obscuration degree of the moving object, and an output determination unit 23 that determines the reliability of the detection result of the predetermined mode by the mode detection unit 21 based on the obscuration degree calculated by the obscuration degree calculation unit 22. This makes it possible to suppress a decrease in detection accuracy due to the part of the detection target being occluded when detecting the state of the detection target shown in the image.

[0065] (2) The manner detection unit 21 may detect a manner type that indicates the type of manner of the moving object. The image processing device 10 may further include a storage unit 13 that stores parameters for each manner type. The output determination unit 23 may determine the reliability of the manner type detection result by the manner detection unit 21 based on the parameter corresponding to the manner type detected by the manner detection unit 21 and the degree of obscuration. This makes it possible to determine the reliability according to the difference in the state type when detecting the state of the detection object captured in the image, and prevents a decrease in the detection accuracy of the state type of the moving object.

[0066] (3) The output determination unit 23 may determine the reliability of the detection result of the mode type by the mode detection unit 21 based on parameters that are modified depending on at least one of the installation conditions of the camera 11 and the image position, which is the position of the moving object on the input image. This makes it possible to prevent a decrease in the detection accuracy of the state type of the moving object depending on the photographing conditions of the camera 11.

[0067] (4) The storage unit 13 may store a determination threshold for each mode type as a parameter. The output determination unit 23 may determine the reliability of the detection result of the mode type by the mode detection unit 21 based on a comparison between the determination threshold corresponding to the mode type detected by the mode detection unit 21 and the concealment degree of the moving object. This makes it possible to determine the reliability according to the difference in type of state when detecting the state of the detection target captured in the image.

[0068] (5) The manner detection unit 21 may determine a manner type that indicates a type of manner, which is a posture or behavior of a person who is a moving body. The manner type may include a first manner type classified as a sitting position and a second manner type classified as a standing position. The output determination unit 23 may determine that the detection result of the manner type by the manner detection unit is erroneous when the degree of obscuration is greater than a determination threshold. The storage unit 13 may store the determination threshold for the first manner type as a value smaller than the determination threshold for the second manner type. This makes it possible to prevent a decrease in the detection accuracy of the person's condition type when the degree of concealment of the lower body parts of the person is high.

[0069] (6) The storage unit 13 may store, as a parameter, an impact coefficient indicating the susceptibility to the influence of concealment for each mode type. The output determination unit 23 may determine the reliability of the detection result of the mode type by the mode detection unit 21 based on the impact degree calculated from the impact coefficient corresponding to the mode type detected by the mode detection unit 21 and the concealment degree. This makes it possible to determine the reliability according to the difference in type of state when detecting the state of the detection target captured in the image.

[0070] (7) The output determination unit 23 may output the detection result of the predetermined mode by the mode detection unit 21 to an output destination according to the degree of concealment. This allows, for example, when the reliability of the detection result by the behavior detection unit 21 differs depending on the degree of concealment of the moving object, the detection result can be output to a device that is appropriate for the reliability of the detection result, or the detection result can be notified to a person in charge that is appropriate for the reliability of the detection result.

[0071] An image processing apparatus according to an embodiment of the present invention can contribute to solving social issues such as a decline in the working population. [Explanation of symbols]

[0072] 1...Monitored space, 2...Person (detection target), 10...Image processing device, 11...Camera, 12...Input unit, 13...Memory unit, 14...Control unit, 15...Output unit, 20...Image acquisition unit, 21...Aspect detection unit, 22...Opacity calculation unit, 23...Output determination unit

Claims

1. an image acquisition unit that acquires an input image captured by a camera; a state detection unit that detects a predetermined state of a moving object captured in the input image; an obscuration degree calculation unit that calculates an obscuration degree of the moving object; a determination unit that determines reliability of a detection result of the predetermined mode by the mode detection unit based on the degree of obscuration calculated by the degree of obscuration calculation unit; An image processing device comprising:

2. the state detection unit detects a state type that indicates a type of state of the moving object; the image processing device further includes a storage unit that stores parameters for each of the aspect types; the determination unit determines reliability of a result of detection of the mode type by the mode detection unit based on a parameter corresponding to the mode type detected by the mode detection unit and the concealment degree.

2. The image processing device according to claim 1, wherein:

3. The image processing device according to claim 2, characterized in that the determination unit determines the reliability of the detection result of the mode type by the mode detection unit based on the parameters that are modified according to at least one of the installation conditions of the camera and the image position, which is the position of the moving object on the input image.

4. the storage unit stores a determination threshold for each of the manner types as the parameter; The image processing device according to claim 2 or 3, characterized in that the determination unit determines the reliability of the detection result of the state type by the state detection unit based on a comparison between the determination threshold corresponding to the state type detected by the state detection unit and the concealment degree of the moving body.

5. the manner detection unit determines a manner type that indicates a type of manner, which is a posture or behavior, of the person who is the moving object; The posture type includes a first posture type classified into a sitting position and a second posture type classified into a standing position, The determination unit determines that the reliability of the detection result of the mode type by the mode detection unit is low when the degree of obscuration is greater than the determination threshold, and The storage unit stores the determination threshold for the first mode type as a value smaller than the determination threshold for the second mode type.

5. The image processing device according to claim 4.

6. the storage unit stores, as the parameter, an influence coefficient indicating susceptibility to influence by concealment for each of the manner types; 4. The image processing device according to claim 2, wherein the determination unit determines the reliability of the detection result of the mode type by the mode detection unit based on the influence coefficient and the degree of obscuration corresponding to the mode type detected by the mode detection unit.

7. the manner detection unit determines a manner type that indicates a type of manner, which is a posture or behavior, of the person who is the moving object; The posture type includes a first posture type classified into a sitting position and a second posture type classified into a standing position, the determination unit determines that the reliability of the detection result of the mode type by the mode detection unit is low when an influence degree calculated from the influence coefficient corresponding to the mode type detected by the mode detection unit and the concealment degree is greater than a predetermined value; the storage unit stores the influence coefficient of the first aspect type as a value greater than the influence coefficient of the second aspect type; 7. The image processing device according to claim 6,

8. the mode detection unit calculates a confidence level indicating a degree of accuracy of the detected mode; The determination unit determines the reliability based on the degree of obscuration calculated by the degree-of-obscuration calculation unit and the certainty factor.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

9. The image processing device according to any one of claims 1 to 3, characterized in that the determination unit outputs the detection result of the predetermined mode by the mode detection unit to an output destination according to the degree of obscuration, or outputs it in an output mode according to the degree of obscuration.

10. an image acquisition process for acquiring an input image captured by the camera; A behavior detection process for detecting a predetermined behavior of a moving object captured in the input image; a concealment degree calculation process for calculating a concealment degree of the moving object; a determination process for determining reliability of a detection result of the predetermined mode by the mode detection process based on the degree of obscuration calculated by the degree of obscuration calculation process; A computer program characterized by causing a computer to execute the above.

Images