Image processing apparatus, image processing method, storage medium, and computer program product

Abstract

The invention provides an image processing apparatus, an image processing method, a storage medium, and a computer program product. The image processing apparatus includes: one or more memories storing instructions; and one or more processors configured, upon execution of the stored instructions, to: specify a location of an organ of a person in the image; acquiring posture estimation information related to the posture of the person on the basis of the position of the specified organ; estimating an action of the person on the basis of the acquired posture estimation information; determining a state of the person based on the acquired posture estimation information; and changing information for indicating the estimated motion of the person in a case where the estimated motion of the person is a predetermined motion and it is determined that the person is in a predetermined state.

Description

Technical Field

[0001] The present disclosure relates to techniques for action recognition. Background Art

[0002] In order to detect the actions and conditions of people, posture estimation technology and motion recognition technology have been developed. In posture estimation technology, the positions of organs (such as joints, eyes, and ears, etc.) of people in an image can be detected by using a convolutional neural network (CNN) using an image as input. In motion recognition technology, actions can be classified by using a CNN using the positions of organs detected by posture estimation technology as input. Motion recognition technology is used, for example, for purchase analysis in stores and detection of fallen people in hospitals, etc. However, in posture estimation technology, when an organ is blocked by an object such as a shelf, the image features of the organ cannot be obtained. Therefore, the accuracy of posture estimation may be reduced.

[0003] In pose estimation technology, as a means for solving the occlusion of the organs of the target person, the following method is generally used: creating an image including the occluded organ by image synthesis, etc., and using the data extension of the occlusion to enhance the training data to improve performance. Z. Zhong, "Random Erasing Data Augmentation", CVPR 2017 discussed the following method: generating pseudo occlusion by synthesizing masks at random positions in the image of the training data to improve the accuracy of organ detection for occlusion.

[0004] Using the method discussed in Z. Zhong, "Random Erasing Data Augmentation", CVPR 2017, the accuracy of posture estimation in a state where a part of a person's organ is blocked and the position of the organ is not obtained can be improved. However, for example, in a state where the lower body is blocked by a shelf or a table, and there is a pattern similar to the person's organ in the area of ​​the lower body, the pattern may be misdetected as an organ such as a knee or ankle in some cases. As a result, the posture estimation accuracy may be reduced, and an incorrect action may be output as a recognition result in action recognition using the result of posture estimation.

[0005] The present disclosure is made in consideration of the above-mentioned problems. The present disclosure relates to a technology for preventing incorrect action recognition from being output as a result of action recognition using posture estimation. Summary of the invention

[0006] According to one aspect of the present disclosure, an image processing device includes: one or more memories storing instructions; and one or more processors, which are configured to: specify the position of an organ of a person in an image; obtain posture estimation information related to the posture of the person based on the specified position of the organ; estimate the action of the person based on the obtained posture estimation information; judge the state of the person based on the obtained posture estimation information; and change the information used to indicate the estimated action of the person when the estimated action of the person is a predetermined action and the person is judged to be in a predetermined state.

[0007] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram showing the hardware configuration of the image processing apparatus.

[0009] Figure 2 is a diagram showing a configuration of an image processing apparatus according to a first exemplary embodiment.

[0010] Figure 3 is a diagram illustrating an example of posture estimation and action recognition.

[0011] Figure 4 is a diagram showing an example of the determination result of the correction condition.

[0012] Figure 5 is a diagram showing an example of a motion recognition result to be corrected.

[0013] Figure 6 is a flowchart illustrating processing to be performed by the image processing apparatus according to the first exemplary embodiment.

[0014] Figure 7 is a diagram showing a configuration of an image processing apparatus according to a second exemplary embodiment.

[0015] Figure 8 is a flowchart illustrating processing to be performed by the image processing apparatus according to the second exemplary embodiment.

[0016] Fig. 9 FIG. 1 is a diagram showing the generation of correction region information.

[0017] Fig.10 FIG. 1 is a diagram showing the generation of correction region information.

[0018] Fig.11 is a diagram showing a configuration of an image processing apparatus according to a third exemplary embodiment.

[0019] Fig.12 is a flowchart illustrating processing to be performed by the image processing apparatus according to the third exemplary embodiment.

[0020] FIG. 13A to FIG. 13C is a diagram showing examples of actual scene clusters and ground truth (GT) clusters corresponding to correction conditions. DETAILED DESCRIPTION

[0021] Some exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. The components described in the following exemplary embodiments are examples in the exemplary embodiments of the present disclosure and are not intended to limit the present disclosure thereto.

[0022] In the first exemplary embodiment, a description of an image processing device that estimates the posture of a person in an image acquired by an installed camera (camera) and estimates the action of the person based on the result of the posture estimation will be provided as an example of action recognition. The image processing device determines the state of the person based on the result of the posture estimation, and changes (corrects) information indicating the estimated action of the person based on the determination result.

[0023] Figure 1 2 is a diagram showing a hardware configuration of the image processing apparatus 100 according to the present exemplary embodiment. The image processing apparatus 100 includes an input unit 201, an output unit 202, a network interface (I / F) unit 203, a central processing unit (CPU) 204, a random access memory (RAM) 205, a read-only memory (ROM) 206, a hard disk drive (HDD) 207, and a data bus 208.

[0024] The CPU 204 reads out the control computer program stored in the ROM 206 and the HDD 207, and loads the program to the RAM 205 to perform various types of processing. The RAM 205 is used as a storage area and a work memory, etc., for temporarily storing the program to be executed by the CPU 204. The network I / F unit 203 transmits / receives various types of information between the image processing apparatus 100 and other devices via the Internet. Data received from an external device via the network I / F unit 203 is transmitted to the CPU 204, the RAM 205, and the ROM 206 via the data bus 208, or received from the CPU 204, the RAM 205, and the ROM 206 via the data bus 208. The HDD 207 stores image data, setting parameters, various types of programs, etc. The HDD 207 can receive data from an external device via the network I / F unit 203.

[0025] The input unit 201 is an input device such as a keyboard, a mouse and / or a touch panel. The output unit 202 is a device for outputting information. Examples of the output unit 202 include a display device such as a liquid crystal display and an audio output device such as a speaker. The input unit 201 and the output unit 202 are also connected to the data bus 208.

[0026] Figure 2 is a diagram showing a functional configuration of the image processing apparatus 100 according to the present exemplary embodiment.

[0027] The image processing apparatus 100 includes an image acquisition unit 101 , a posture estimation unit 102 , an action recognition unit 103 , a correction condition determination unit 104 , and a correction unit 105 .

[0028] The image acquisition unit 101 acquires an image. The image acquired by the image acquisition unit 101 may be an image acquired by photographing a network camera or the like or an image recorded in a storage device such as a hard disk or the like. The image acquired by the image acquisition unit 101 may be received via a network such as the Internet or the like. The image acquired by the image acquisition unit 101 is sent to the posture estimation unit 102.

[0029] The posture estimation unit 102 detects organs of the person in the image and specifies the positions of the detected organs. The organs are, for example, parts (joint points) such as the eyes, nose, ears, shoulders, elbows, wrists, hips, knees, and ankles of the person. The posture estimation unit 102 acquires posture estimation information based on the positions of the specified organs. The posture estimation information includes organ position coordinates as position information for indicating the positions of the organs and organ detection likelihoods for indicating the likelihood of the positions of the respective organs. The organ detection likelihood is likelihood information for indicating the likelihood of the positions of the respective organs detected by the posture estimation unit 102. The organ detection likelihood can be restated as information for indicating the reliability, accuracy, or certainty of the positions of the respective organs.

[0030] The posture estimation information includes a character rectangle that surrounds an area of ​​a character in an image and a detection likelihood of the character rectangle. The posture estimation information includes a full body rectangle that surrounds the full body of the character including parts that are not visible in the image and a full body detection likelihood. The character rectangle and the full body rectangle are information that enables identification of an area including the full body of the character, and their shapes are not limited to rectangles. The posture estimation information includes an identifier of the character. The identifier of the character is a scalar value used to indicate the same character between frames. For example, in a case where identifier 1 is assigned to character A in one of the image frames continuously captured by a camera device, identifier 1 is assigned to character A in the next frame to indicate the same character. The information included in the posture estimation information is not limited to the above information, and the posture estimation information may include information other than the above information, or may include optional information from the above information. The posture estimation information according to the present exemplary embodiment includes at least organ position coordinates.

[0031] As an example of pose estimation, in a case where the body below the hips of a person in an image is obscured by, for example, an occluder, the lower end of the person's rectangle is located at the hips, and the lower end of the full-body rectangle is located at the feet estimated from the upper body. As discussed in J. Redmon, "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2015, the pose estimation unit 102 determines the coordinates, width, height, and detection likelihood of the center of the person's rectangle by using a convolutional neural network (CNN) with an image as input. The coordinates, width, and height of the center of the full-body rectangle and the full-body detection likelihood are also determined by replacing the training data in the CNN with the full-body rectangle by the method discussed in J. Redmon, "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2015. As discussed in K. Sun, “Deep High-Resolution Representation Learning for Human PoseEstimation”, CVPR 2019, the pose estimation unit 102 determines the organ position coordinates and the organ detection likelihood for the person rectangle by using a CNN that takes as input an image extracted from the image based on information related to the person rectangle. In addition, as discussed in Y. Zhang, “FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking”, IJCV 2021, the pose estimation unit 102 determines the feature quantity for re-referencing the person between frames by using a CNN that takes as input an image extracted from the image based on information related to the person rectangle. The pose estimation unit 102 assigns an identifier to the person by re-referencing the person between frames.

[0032] Figure 3 301, an organ position coordinate 302, and a person identifier 303 are obtained. Figure 3 In the example shown, the person identifier is “1.” Posture estimation unit 102 sends posture estimation information to action recognition unit 103 and correction condition determination unit 104 .

[0033] The action recognition unit 103 estimates the action of the person with the posture estimation information as input, and generates information indicating the estimated action. The action recognition unit 103 determines the action specified based on the information indicating the estimated action as the result of the action recognition of the person in the image. The information indicating the action includes an action label indicating the action of the person and the action likelihood for the action label. One or more action labels are previously defined corresponding to the action to be identified. The image processing device 100 according to the present exemplary embodiment recognizes the falling action of the person. Therefore, in the present exemplary embodiment, the action label "fall" is defined. The action likelihood is likelihood information indicating the likelihood (reliability) of the estimated action of the person. The action likelihood is represented by a value from 0 to 1, and as the value is closer to 1, the possibility of indicating the action is higher. The action label is illustrative, and other actions can be defined as an action label. Two or more actions can be defined as action labels.

[0034] As discussed in F. Yang, “Make Skeleton-based Action Recognition Model Smaller, Faster and Better”, CVPR 2019, the action recognition unit 103 recognizes the action based on the time series change of the organ position determined from the organ position coordinates of the person in multiple image frames. As an example, the action recognition process is applied to Figure 3 The image shown in the figure is used to generate an action likelihood corresponding to the action label "falling" as a value of 0.98, etc. The action recognition unit 103 recognizes that the action of the person is "falling" based on the generated information. The action recognition unit 103 recognizes the action indicated by the action label corresponding to the action likelihood greater than a predetermined threshold as the action of the person. The action recognition unit 103 sends the information indicating the action to the correction unit 105.

[0035] The correction condition judgment unit 104 takes the posture estimation information as input, judges the state of the person whose posture estimation is performed by the posture estimation unit 102, and generates a judgment result. At this time, the correction condition judgment unit 104 judges whether the person is in a predetermined state. Hereinafter, the condition that the person is in a predetermined state is referred to as the correction condition, and the correction condition judgment unit 104 generates a judgment result related to whether the person satisfies the correction condition as a judgment result. Correction means that the correction unit 105 described below changes the information for indicating the action generated by the action recognition unit 103.

[0036] The judgment result generated by the correction condition judgment unit 104 includes a correction label for indicating the correction condition and a correction likelihood for indicating the degree of correction to be performed. The correction label is a label for indicating the correction condition, in other words, a label for indicating a possible predetermined state of a person. In the present exemplary embodiment, as correction labels, "lower body occlusion" for indicating a state where the lower body of a person is occluded, "cutoff" for indicating a state where a part of a person extends out of an image, and "background misdetection" for indicating a state where a background is misdetected as a person are defined. In addition, in the present exemplary embodiment, as a correction label, "movement" for indicating a state where a person is moving is defined. The correction likelihood is represented by a value from 0 to 1, and as the value is closer to 1, the degree of correction to be performed is indicated to be higher. In other words, the correction likelihood is likelihood information for indicating the likelihood (reliability) of the state of the person judged by the correction condition judgment unit 104.

[0037] The target problem to be solved in the present exemplary embodiment is now described. The image processing device 100 according to the present exemplary embodiment recognizes the falling action of a person, and outputs a recognition result to use the recognition result for, for example, notification to a user. However, in some cases, the position of the person's organs is misdetected because the person is blocked by an object, a part of the person is cut off in the image (extends out of the image), or the background is misdetected as a person. In this case, there is a problem that the person is misidentified as falling. The correction condition judgment unit 104 judges that the state of the person is in a correction condition in which the person is misidentified as falling, and determines the correction likelihood corresponding to the correction condition, thereby enabling correction of the result of the action recognition.

[0038] The correction conditions for determining the correction likelihood corresponding to each correction label are described below. The correction likelihood is calculated based on the posture estimation information acquired from the posture estimation unit 102. The correction likelihood of "lower body occlusion" is calculated to become higher as more of the following conditions are satisfied.

[0039] -The average value of the y coordinates of the left and right hip joints is greater than the y coordinate of the lower end of the character rectangle -The likelihood of full body detection is less than the likelihood of detection of the character rectangle

[0040] - The distance between the y coordinate of the lower end of the person rectangle and the y coordinate of the lower end of the whole body rectangle is large - The likelihood of organ detection of the hip, knee, and ankle is low

[0041] At this time, the calculation expression is as follows.

[0042] [Formula 1]

[0043]

[0044] body score =R body· score-R fullbody· score

[0045]

[0046] lower score =1.0-(average(p lhip· score,p rhip· score)+average(p lknee· score,p rknee· score)+average(p lankle· score,p rankle· score)) / 3

[0047] The correction likelihood of “cut off” is calculated to become higher as more of the following conditions are satisfied.

[0048] -The average value of the y coordinates of the left and right shoulder joints or the left and right hip joints is close to the bottom of the image -The y coordinate of the bottom of the character rectangle is close to the bottom of the image

[0049] -The y coordinate of the bottom end of the full body rectangle is greater than the image height

[0050] At this time, the calculation expression is as follows: The numerical value of 0.95 in the expression is a parameter that changes according to imaging conditions and the like, and the parameter is not limited to a specific value.

[0051] [Formula 2]

[0052]

[0053] The corrected likelihood of “background misdetection” is calculated to become higher as more of the following conditions are satisfied.

[0054] - All organs have low detection likelihood

[0055] -The detection likelihood of the person rectangle and the whole body are low

[0056] At this time, the calculation expression is as follows: The numerical value of 0.2 in the expression is a parameter that changes according to imaging conditions and the like, and the parameter is not limited to a specific value.

[0057] [Formula 3]

[0058]

[0059] Rect score =If (R body· score<0.2 and R fullbody·score<0.2), then 1.0, otherwise 0.0

[0060] The correction likelihood of “movement” is calculated to become higher as more of the following conditions are satisfied.

[0061] -The center coordinates of the character rectangle move a large distance between image frames

[0062] At this time, the calculation expression is as follows.

[0063] [Formula 4]

[0064]

[0065] In the above formulas 1 to 4, S occlusion , S cutoff , S background and S move Indicates the corrected likelihood of lower body occlusion, cut-off, background misdetection, and movement, respectively. position 、body score 、body size and lower score Indicates the likelihood related to lower body occlusion based on hip position, detection likelihood of person rectangle, size of person rectangle and organ detection likelihood respectively. edge , hip edge 、body edge and fullbody edge Indicates the likelihood of cutting off based on the shoulder position, hip position, lower end position of the character rectangle, and lower end position of the whole body rectangle, respectively. score A score indicating the degree to which the condition "the detection likelihood of all joint points is low" is satisfied. score A score indicating the degree to which the condition of "both the detection likelihood and the whole body detection likelihood are low" is satisfied. In addition, the key point num indicates the total number of joint points.

[0066] In addition, "p" is information for indicating the coordinates of the organ position, and the subscript character string indicates the part. The character strings "1shoulder", "rshoulder", "1hip", "rhip", "lknee", "rknee", "lankle", and "rankle" respectively indicate the left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle. In addition, "y" indicates the y coordinate of the joint point, and "score" indicates the detection likelihood of the organ. "R" is rectangle information, the subscript character string "body" indicates the character rectangle, and the subscript character string "fullbody" indicates the full body rectangle. In addition, "x" indicates the center x coordinate of the rectangle, "y" indicates the center y coordinate of the rectangle, "lty" indicates the upper end y coordinate, "rby" indicates the lower end y coordinate, and "score" indicates the detection likelihood of the rectangle. The superscript indicates the number of frames of the image, "t" indicates information related to the current frame, and "t-1" indicates information related to the previous frame. In addition, "image" indicates image information, and "height" indicates the image height.

[0067] The types of correction labels and methods for determining correction likelihood described in this exemplary embodiment are illustrative and not restrictive. For example, other correction conditions may be defined as correction labels. Correction likelihood may be determined by other calculation methods, or an optional number of calculation expressions in the above calculation expressions may be used.

[0068] Figure 4 An example of the judgment result of the correction condition is shown. The correction condition judgment unit 104 performs correction condition judgment on the posture estimation information acquired from the image, thereby determining the correction likelihood of lower body occlusion, cutoff, background misdetection and movement. Figure 4 In the example shown, as a result of the judgment, a correction label corresponding to a correction likelihood greater than a preset predetermined threshold is associated with a rectangle detected as a person in the image. Figure 4 In the correction label, “occlusion” indicates the correction label of lower body occlusion, and “background” indicates the correction label of background misdetection. The correction condition judgment unit 104 sends the judgment result of the correction condition to the correction unit 105.

[0069] The correction unit 105 corrects the information indicating the action based on the information indicating the action obtained from the action recognition unit 103 and the judgment result made by the correction condition judgment unit 104. More specifically, when the action likelihood corresponding to the action label "fall" is greater than a predetermined threshold value and the correction likelihood corresponding to any one of the correction labels is greater than the corresponding predetermined threshold value, the correction unit 105 changes the action likelihood corresponding to the action label "fall" of the character. At this time, the correction unit 105 corrects the action likelihood corresponding to the action label "fall" to a sufficiently small value (for example, 0.0 or 0.1). Correcting the action likelihood in this way prevents the following: as a result of the action recognition identified by the action recognition unit 103, the action is misidentified as "falling". When the correction unit 105 corrects the action likelihood, the correction unit 105 corrects the action likelihood to a value less than the predetermined threshold value at which the action is recognized as "falling".

[0070] Figure 5 An example of an action recognition result to be corrected is shown. The lower body of the person in the image is blocked by a shelf, and the blocked lower body area is shown by a dotted line in the figure. In this exemplary embodiment, it is assumed that the threshold of the action likelihood of "falling" is 0.8, and it is assumed that the threshold of the correction likelihood is 0.8. The value of the threshold is illustrative and is not limited to 0.8. When applying the posture estimation process to Figure 5 When the image shown is viewed, the position of the organs of the upper body is correctly detected, but the position of the organs of the lower body is detected at a position different from the actual position. When performing action recognition based on the result of organ detection, the action likelihood corresponding to the action label "falling" is calculated as 0.98, because the posture is similar to the falling posture with the legs bent. The action likelihood is greater than 0.8, which is a threshold. Therefore, when performing action recognition based on the action likelihood, it is recognized that the person falls. In addition, in the judgment of the correction condition, the correction likelihood of the correction label "lower body occlusion" is calculated as 0.96, because the y coordinate of the hip is greater than the y coordinate of the lower end of the character rectangle and the distance between the y coordinate of the lower end of the character rectangle 501 and the y coordinate of the lower end of the whole body rectangle 502 is large. Since the correction likelihood is greater than 0.8 used as a threshold, the action likelihood is changed to a sufficiently small value (such as 0.0 or 0.1, etc.). The correction unit 105 controls the output of information from the output unit 202 according to the result of action recognition based on the corrected action likelihood.

[0071] The image processing device 100 according to the present exemplary embodiment recognizes that a person has fallen, and notifies the user of the presence of the fallen person through a display or sound, etc. At this time, for example, when a person whose lower body is blocked is identified as a fallen person, incorrect information is notified to the user. However, the correction unit 105 corrects the action likelihood corresponding to the action label "fall", which makes it possible to prevent an incorrect recognition result from being output. In the case where the action likelihood has been corrected, the correction unit 105 can notify the user of the corrected action likelihood, or can omit notification to the user because the fallen person has not been identified.

[0072] The above describes the configuration of the image processing device 100 according to the present exemplary embodiment. The posture estimation information acquired by the posture estimation unit 102 includes at least organ position coordinates. In this case, the correction likelihood is calculated only based on the organ position coordinates. As discussed in Z. Cao, "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", CVPR 2018, an algorithm for directly determining the organ position coordinates from an image can be used to detect the organ position. In other words, any method by which information used for character action recognition and correction condition judgment can be obtained can be applied to the method of obtaining posture estimation information, without being limited to a specific method.

[0073] In addition to storing the action label of falling, the action recognition unit 103 may also store action labels of standing, walking, running, sitting, etc. In the case where multiple action labels are defined, the correction unit 105 may manage the action likelihood of the corresponding action label to be corrected for each correction label when the correction likelihood is greater than a preset threshold.

[0074] In the correction of the action likelihood of the correction unit 105, the highest correction likelihood can be determined among the correction likelihoods of the correction labels of lower body occlusion, cut-off, background misdetection, and movement, and the action likelihood can be weighted using weight = 1-correction likelihood. In addition, the correction unit 105 can apply correction only when the period (number of image frames) during which the correction likelihood of any one of the correction labels of lower body occlusion, cut-off, background misdetection, and movement is greater than a preset threshold is longer than a preset period.

[0075] The correction condition judgment unit 104 according to the present exemplary embodiment calculates the correction likelihood each time the posture estimation information is acquired; however, the configuration is not limited thereto. For example, the correction condition judgment unit 104 may calculate the correction likelihood only when the value of the action likelihood corresponding to the predetermined action (fall) acquired based on the posture estimation information is greater than a predetermined value.

[0076] refer to Figure 6 The flowchart shown in FIG. 1 describes processing performed by the image processing apparatus 100 according to the present exemplary embodiment. When the CPU 204 of the image processing apparatus 100 reads out a control computer program stored in the ROM 206 and the HDD 207, loads the control computer program into the RAM 205, and executes the control computer program, processing is performed. Figure 6 Processing shown.

[0077] In step S601, the image acquisition unit 101 acquires an image from a camera device or an internal or external storage device. The image can be acquired through the Internet, etc. In the case where the image has not been acquired ("No" in step S602), the process proceeds to step S607. In step S607, the process ends. In the case where the image has been acquired ("Yes" in step S602), the process proceeds to step S603. In step S603, the posture estimation unit 102 performs posture estimation processing on the person in the image based on the acquired image to specify the organ position of the person, and acquires posture estimation information. In the case where multiple persons are included in the image, the posture estimation unit 102 performs posture estimation processing on each person. In step S604, the action recognition unit 103 performs action recognition processing based on the posture estimation information, thereby acquiring an action label and the action likelihood corresponding thereto as information for indicating the action.

[0078] In step S605, the correction condition judgment unit 104 judges the correction condition based on the posture estimation information. The correction unit 105 obtains information for indicating an action and a judgment result related to the correction condition. In a case where it is judged that the action of the character is a predetermined action ("falling" in the present exemplary embodiment) and the character is in a predetermined state indicated by the correction label, in step S606, the correction unit 105 changes the action likelihood which is the information for indicating an action. The correction unit 105 outputs the result of action recognition based on the corrected action likelihood by using the output unit 202. The image processing device 100 repeats the above-mentioned processing during image acquisition.

[0079] Through the above-described processing, the image processing apparatus 100 can prevent an incorrect action recognition result from being output.

[0080] A second exemplary embodiment of the present invention will be described. In the second exemplary embodiment, a description of a configuration example will be provided in which a correction candidate region in an image is generated based on posture estimation information and the correction candidate region is output to a user. Descriptions of contents similar to those in the first exemplary embodiment are omitted, and only differences will be described.

[0081] Figure 7is a block diagram showing the configuration of the image processing apparatus 700 according to the present exemplary embodiment. In addition to the configuration of the image processing apparatus 100 according to the first exemplary embodiment, the image processing apparatus 700 further includes a correction region determination unit 106. The correction condition determination unit 104 sends posture estimation information and a determination result of the correction condition to the correction region determination unit 106.

[0082] The correction area determination unit 106 performs an area acquisition process to acquire correction area information based on the posture estimation information and the judgment result related to the correction condition. The correction area information includes information for indicating the area in the image where the action of the person is to be changed (corrected), and a correction object label. The information for indicating the correction area indicates the area in the image, and enables judgment whether the coordinates in the image are included in the area. For example, the information for indicating the correction area may be rectangular information including the center coordinates and the width / height, or polygon information including three or more vertices. The correction object label is at least one correction label for making correction condition judgments on the person in the correction area. The correction area determination unit 106 stores the posture estimation information and the judgment result related to the correction condition for a certain period of time. When data for a certain period of time is collected, the correction area determination unit 106 generates the correction area information. The correction area determination unit 106 sends the correction area information to the correction unit 105.

[0083] The correction unit 105 corrects the information indicating the action based on the correction area information generated by the correction area determination unit 106, the information indicating the action, and the judgment result related to the correction condition. The correction unit 105 calculates the correction likelihood corresponding to the correction object label for the person whose central coordinates of the person rectangle are included in the correction area. When the correction likelihood is greater than a preset threshold, the correction unit 105 corrects the action likelihood of the action label of the person to a sufficiently small value (e.g., 0.0).

[0084] refer to Figure 8 The flowchart shown describes a process for generating correction region information. In step S1001, the correction region determination unit 106 selects one of a plurality of correction labels as a correction target label. In step S1002, the correction region determination unit 106 calculates the correction likelihood corresponding to the correction target label for each of a plurality of characters in the image, and collects and stores posture estimation information related to the characters corresponding to the correction likelihood greater than a preset threshold.

[0085] In step S1003, the correction region determination unit 106 clusters the center coordinates of the person rectangles included in the stored posture estimation information. In step S1004, the correction region determination unit 106 generates a correction region including each cluster. In step S1005, the correction region determination unit 106 generates correction region information by synthesizing the selected correction object label and the generated information for indicating the correction region. Fig. 9 An example of the center coordinate information about the person rectangle included in the posture estimation information stored when the correction label of the lower body occlusion is selected as the correction target label is shown. Fig. 9 In the figure, the marked mark "x" indicates the center coordinates of each character rectangle in the image.

[0086] The image includes a shelf and a table, so that the posture estimation information about the person behind the shelf and the table may be determined as lower body occlusion. The correction area determination unit 106 performs the correction of the posture estimation information by using an existing method or the like. Fig. 9 The multiple labels "x" shown are clustered, thus Fig.10 The area corresponding to the calibration object label "occlusion" shown is designated as the calibration area.

[0087] In the case where there is a correction tag that has not been selected as the correction target tag ("Yes" in step S1006), the process returns to step S1001. In the case where there is no correction tag that has not been selected ("No" in step S1006), the process proceeds to step S1007. The correction area determination unit 106 further selects, for example, the correction tag "cut" as the correction target tag, and performs the processes in and after step S1001 to specify the correction target tag. Fig.10 The calibration area corresponding to the calibration object label "Cut" in the calibration object label.

[0088] In step S1007, the correction region determination unit 106 outputs the generated correction region information by using the output unit 202. In the present exemplary embodiment, the correction region information is output as Fig.10 The display shown is output to the display device. In step S1008, the user specifies the correction area information corresponding to the area to be corrected by using the input unit 201 while viewing the information displayed on the output unit 202. The correction unit 105 calculates the correction likelihood based on the correction object label corresponding to the correction area specified by the user as to be corrected, and corrects the action likelihood based on the value of the correction likelihood.

[0089] With the above configuration, the user can instruct whether to perform correction while visually checking the area to be corrected. As the information output in step S1007, optional information among the plurality of posture estimation information stored in the correction area determination unit 106 can be output as an example of a person whose action is to be corrected.

[0090] A third exemplary embodiment of the present invention will be described. In the third exemplary embodiment, as an example of condition setting in correction condition judgment, a description of a configuration example will be provided in which correction conditions for correction are automatically created based on the distribution of the consistency rate between information for indicating an action and a previously set correction condition. Descriptions of contents similar to those in the first exemplary embodiment are omitted, and only differences are described.

[0091] Fig.11 11 is a block diagram showing a configuration of an image processing apparatus 1100 according to the present exemplary embodiment. The image processing apparatus 1100 further includes a condition generating unit 107 in addition to the configuration of the image processing apparatus 100 according to the first exemplary embodiment.

[0092] The action recognition unit 103 sends the action label and the action likelihood as information for indicating the action to the condition generation unit 107. The condition generation unit 107 generates the correction condition to be applied to the correction condition judgment unit 104 based on the action likelihood and the distribution of the correction likelihood determined based on the previously prepared correction condition. The condition generation unit 107 performs action recognition estimation on the person within a certain period of time (a certain number of image frames) and stores the action likelihood. When collecting the action likelihood within a certain period of time, the condition generation unit 107 creates a distribution map based on the action likelihood and the correction likelihood for the previously prepared correction condition, and clusters the distribution map. The condition generation unit 107 further performs cluster separation judgment by using the cluster center, and generates correction conditions that can each separate the clusters as the correction conditions to be applied in the correction condition judgment unit 104. The condition generation unit 107 sends the generated correction conditions to the correction condition judgment unit 104.

[0093] refer to Fig.12 The flowchart shown describes the process for generating correction conditions. In step S1201, the condition generation unit 107 obtains information for indicating actions collected within a certain period of time. In step S1202, the condition generation unit 107 selects one of the previously prepared correction conditions. In step S1203, the condition generation unit 107 creates a distribution map based on the action likelihood and the correction likelihood of the selected correction condition, and clusters the distribution map. FIG. 13A to FIG. 13CAn example obtained by clustering using a plurality of correction conditions is shown. The data is plotted and clustered in such a manner that the horizontal axis of the graph is set to the correction likelihood corresponding to the selected correction condition and the vertical axis is set to the action likelihood corresponding to the action label "fall".

[0094] Clustering is performed using two types of data, i.e., ground truth (GT) clusters and actual scene clusters.

[0095] The GT cluster is a result obtained by clustering the distribution of the action likelihood based on the posture estimation information with the previously prepared ground truth action label and the correction likelihood corresponding to the selected correction condition. The GT cluster is determined in units of action labels. The GT cluster in this exemplary embodiment is a cluster generated for the action label "falling". For example, the data related to the GT cluster is generated by previously determining the action likelihood and the correction likelihood using an image group including an image in which a falling person exists.

[0096] The actual scene cluster is the result obtained by clustering the distribution of action likelihood and correction likelihood corresponding to an image group different from the image group of the GT cluster. The actual scene cluster is not provided with a ground truth action label, and includes the results of multiple action labels. Thus, the actual scene cluster is clustered into multiple clusters. As a clustering method, for example, k-means can be used. At this time, the action label of each actual scene cluster can be predicted based on the predicted occurrence frequency of the action label and the amount of data in each actual scene cluster.

[0097] FIG. 13A to FIG. 13C is a distribution diagram of action likelihood and correction likelihood created by applying correction conditions A, B, and C. The clusters with hollow white circles are actual scene clusters, and the clusters with solid black circles are GT clusters. For example, in Fig.13A In the clustering results of the actual scene using the correction condition A shown, there is an actual scene cluster group 1301 with low correction likelihood and small frequency (number of plots) and an actual scene cluster group 1302 with high correction likelihood and high frequency. The number of people predicted to take the action of "falling" in the actual scene is small. Therefore, the action label "falling" is assigned to the actual scene cluster group 1301.

[0098] In contrast, for the actual scene cluster group 1302, the action likelihood of "falling" is high and the frequency is also high; however, considering the prediction of a small number of people taking a falling action, the actual scene cluster group 1302 is considered to have posture estimation information with erroneous "falling" estimation. Fig.13AThe GT clusters shown include a GT cluster group 1311 having a low correction likelihood and a small frequency, and a GT cluster group 1312 having a low correction likelihood and a high frequency. The GT clusters are clusters acquired for "falling". Thus, the GT cluster group 1311 is posture estimation information related to a person estimated to have "fallen" based on a high action likelihood, and the GT cluster group 1312 is posture estimation information related to a person estimated to have taken an action other than "falling". The condition generation unit 107 determines a correction condition that enables separation of actual scene clusters having posture estimation information with erroneous action estimation (such as the actual scene cluster group 1302, etc.) and GT clusters having posture estimation information with "falling" estimation (such as the GT cluster group 1311, etc.).

[0099] In step S1204, the condition generation unit 107 determines whether the GT cluster and the actual scene cluster are separable based on the clustering result. The condition generation unit 107 uses, for example, the values ​​of the action likelihood and the correction likelihood as coordinate values, and determines the distance between the center coordinates of the clusters. If the distance is greater than or equal to a certain value, the condition generation unit 107 determines that the clusters have been separated. Fig.13A In the example shown, it is found that under condition A, the actual scene cluster group 1302 having high action likelihood and correction likelihood is separated from the GT cluster group 1311 based on the correction likelihood.

[0100] Found in Fig. 13B Under the correction condition B shown in FIG. 1 , the GT cluster group 1313 with high action likelihood is close to the actual scene cluster group 1303 with high frequency, and the clusters are not separated. Similarly, it is found that Fig. 13C Under the correction condition C shown, the GT cluster group 1314 with high action likelihood is close to the actual scene cluster group 1304 with high frequency, and the clusters are not separated. Based on these results, it is judged that the correction condition A can separate the actual scene cluster and the GT cluster, but the correction conditions B and C cannot separate the actual scene cluster and the GT cluster.

[0101] At this time, in the case where the action label is previously predicted and assigned to the actual scene cluster, the combination of clusters with the same action label can be excluded from the separated object. In addition, based on the distribution of the correction likelihood of the clusters judged to be separable, the threshold of the correction likelihood used in the correction unit 105 can be determined. In the case where the cluster is judged to be separable ("Yes" in step S1205), the processing proceeds to step S1206. In step S1206, the condition generation unit 107 adds the correction condition used to the correction condition to be applied in the correction condition judgment unit 104. In the case where the cluster is judged to be inseparable ("No" in step S1205), the processing proceeds to step S1207. In the case where there is an unprocessed condition ("Yes" in step S1207), the processing returns to step S1202. In the case where there is no unprocessed condition ("No" in step S1207), the processing proceeds to step S1208. In step S1208 , the condition generating unit 107 determines the correction condition to be applied in the correction condition determining unit 104 according to the processing result.

[0102] With the above-described configuration, the image processing apparatus 1100 can correct the information indicating an action (action likelihood) by using an appropriate correction condition.

[0103] The above-described exemplary embodiments may be freely combined and implemented. According to the exemplary embodiments, it is possible to prevent incorrect motion recognition from being output as a result of motion recognition using posture estimation.

[0104] Other embodiments

[0105] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0106] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: a specifying unit configured to specify a position of an organ of a person in the image; an acquisition unit configured to acquire posture estimation information related to the posture of the person based on the position of the organ specified by the specifying unit; an estimating unit configured to estimate the action of the character based on the posture estimation information acquired by the acquiring unit; a first determination unit configured to determine the state of the character based on the posture estimation information acquired by the acquisition unit; as well as A changing unit configured to change information indicating the action of the character estimated by the estimating unit when the action of the character estimated by the estimating unit is a predetermined action and the first judging unit judges that the character is in a predetermined state.

2. The image processing device according to claim 1, in, the estimating unit generates first likelihood information indicating the estimated likelihood of the action of the character as information indicating the action of the character, wherein the first judgment unit generates second likelihood information indicating the likelihood of the judged state of the character, and Wherein, when the likelihood indicated by the first likelihood information is greater than a predetermined threshold and the likelihood indicated by the second likelihood information is greater than a predetermined threshold, the changing unit changes the first likelihood information.

3. The image processing apparatus according to claim 1, wherein: The predetermined action includes a falling action of the character.

4. The image processing apparatus according to claim 1, wherein: The predetermined state includes at least any one of a state in which the lower body of the person is blocked, a state in which the person is cut off in the image, a state in which the background is mistakenly detected as the person, and a state in which the person is moving.

5. The image processing apparatus according to claim 1, wherein: The posture estimation information includes position information indicating the position of the organ specified by the specifying unit.

6. The image processing apparatus according to claim 5, wherein: The posture estimation information includes likelihood information indicating likelihood of the position of the organ specified by the specifying unit.

7. The image processing apparatus according to claim 5, wherein: The posture estimation information includes information indicating a region including a whole body of a person.

8. The image processing device according to claim 1, further comprising an output unit configured to output information, wherein: The changing unit sets the action of the character specified based on the changed information as a recognition result of the action of the character when the information indicating the action of the character is changed, and controls output of the information by the output unit based on the recognition result. 9 . The image processing device according to claim 1 , further comprising a region acquisition unit configured to acquire region information indicating a region where a motion of a person in an image is to be changed based on the plurality of posture estimation information acquired by the acquisition unit.

10. The image processing device according to claim 2, further comprising a second judgment unit configured to judge the predetermined state based on first likelihood information and second likelihood information acquired based on a first image group including a person estimated to take the predetermined action, and based on first likelihood information and second likelihood information acquired based on a second image group different from the first image group, wherein When the action of the person estimated by the estimation unit is a predetermined action and the first judgment unit judges that the person is in the predetermined state judged by the second judgment unit, the first judgment unit changes information indicating the action of the person estimated by the estimation unit.

11. An image processing method, comprising: Specify the location of organs of the person in the image; Based on the position of the designated organ, obtaining posture estimation information related to the posture of the person; estimating a motion of the character based on the acquired posture estimation information; Determining the state of the character based on the acquired posture estimation information; as well as When the estimated motion of the person is a predetermined motion and it is determined that the person is in a predetermined state, information indicating the estimated motion of the person is changed.

12. A non-transitory computer-readable storage medium storing a program for causing a computer to: Specify the location of organs of the person in the image; Based on the position of the designated organ, obtaining posture estimation information related to the posture of the person; estimating a motion of the character based on the acquired posture estimation information; Determining the state of the character based on the acquired posture estimation information; as well as When the estimated motion of the person is a predetermined motion and it is determined that the person is in a predetermined state, information indicating the estimated motion of the person is changed.

13. A computer program product comprising a program for causing a computer to: Specify the location of organs of the person in the image; Based on the position of the designated organ, obtaining posture estimation information related to the posture of the person; estimating a motion of the character based on the acquired posture estimation information; Determining the state of the character based on the acquired posture estimation information; as well as When the estimated motion of the person is a predetermined motion and it is determined that the person is in a predetermined state, information indicating the estimated motion of the person is changed.

Images