Image processing apparatus, method for controlling image processing apparatus, and storage medium

Abstract

To provide an image processing device, a control method of the image processing device, and a program capable of reducing erroneous estimation of an estimation result when estimating a posture of a subject.SOLUTION: The image processing apparatus includes a detection unit (object detection unit 162) configured to detect a plurality of keypoints of a subject, a keypoint sameness determination unit (object detection unit 162) configured to determine whether the plurality of keypoints are of the same subject, an extraction unit (object detection unit 162) configured to extract a detection frame indicating a detection range of the subject, and a determination unit configured to determine whether the subject from which the keypoints are detected and the subject from which the detection frame is extracted are the same subject. The reliability determination unit (object detection unit 162) lowers the reliability of at least one of the detection frame and the key point, or lowers the reliability of the posture determination of the object using the key point determined to be the same, according to the positional relationship between the key point and the detection frame determined to be the same as the key point.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, and a program. [Background technology]

[0002] In the field of computer vision, there is an object detection technology that detects an object in an image and displays a detection frame surrounding the object as a detection result. An example of an application of object detection technology is a technology that detects key points (feature points) such as the joints of a person present in an image and estimates the person's pose based on the detection result. Pose estimation technologies are divided into top-down and bottom-up types. The top-down type detects a person in an image and estimates the person's pose based on the detection of predefined key points for the person. The bottom-up type detects multiple key points in an image and estimates the person's pose by connecting the key points with lines, i.e., stitching them together. The top-down type provides more accurate pose estimation than the bottom-up type, but tends to require higher computational costs for pose estimation. On the other hand, the bottom-up type requires less computational effort for pose estimation than the top-down type, but tends to be prone to false detection of key points and false connections between key points. Furthermore, in bottom-up methods, erroneous detection of keypoints or erroneous connections between keypoints may occur, resulting in an estimated person's posture that is not unnatural, i.e., falls within the range of possible postures. In this case, the estimated person's posture will be erroneously estimated. Non-Patent Document 1 discloses a bottom-up posture estimation technique. Patent Document 1 also discloses an apparatus that performs a detection process to detect multiple types of body parts of a subject in an image and selects one of multiple determination methods for determining the subject's behavior based on the results of the detection process. In the apparatus described in Patent Document 1, when selecting one of the multiple determination methods, the positional relationship of two or more types of body parts among the multiple types of body parts is used to determine the subject's behavior. The subject's behavior is then determined according to the selected determination method. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-68992 [Non-patent literature]

[0004] [Non-Patent Document 1] Alejandro NewelL, Zhiao Huang, Jia Deng, 2017, “Associative Embedding: End-to-End Learning for Joint Detection and Grouping”, ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 30, 2278-2288 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Non-Patent Document 1, i.e., bottom-up pose estimation technology, if a keypoint is misdetected or a keypoint is misconnected, the estimated pose of the person will be erroneous, as described above.

[0006] The present invention has been made in view of the above-mentioned problems, and aims to provide an image processing device, a control method for the image processing device, and a program that can reduce erroneous estimation results when estimating the posture of a subject. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the image processing device of the present invention comprises a detection means for detecting multiple key points for a subject included in an image, a key point identity determination means for determining whether the multiple key points represent the same subject, an extraction means for extracting a detection frame that surrounds parts of the subject's body and indicates the detection range of the subject, and a determination means for determining whether the subject from which the key points have been detected and the subject from which the detection frame has been extracted are the same subject, and the reliability determination means is characterized in that it reduces the reliability of at least one of the detection frame and the key points, or reduces the reliability of the posture determination of the subject using key points that have been determined to be identical, depending at least on the positional relationship between the key points detected by the detection means and determined to represent the same subject, and the detection frame that has been determined to be identical to the key points.

[0008] Moreover, the image processing device of the present invention comprises a detection means for detecting a plurality of key points for a subject included in an image; a connection means for connecting the key points detected by the detection means with straight lines; an extraction means for extracting a detection frame that surrounds a body part of the subject included in the image and is to be determined whose posture or movement is to be determined, and indicates the detection range of the subject; and a judgment means for judging whether the subject from which the key points have been detected and the subject from which the detection frame has been extracted are the same subject, and when the judgment means determines that they are the same subject, the connection means determines, based on at least the positional relationship between the key points detected by the detection means and the detection frame extracted by the extraction means, the key points to be connected from among the key points detected by the detection means. [Effects of the Invention]

[0009] According to the present invention, when estimating the posture of a subject, it is possible to reduce the likelihood of the estimation result being an erroneous estimation. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a block diagram showing an example of a hardware configuration of an imaging apparatus according to a first embodiment. [Figure 2] 10 is a flowchart showing a posture estimation process when a bottom-up method is used. [Figure 3] 10 is a flowchart showing a posture estimation process when a top-down method is used. [Figure 4] 5 is a flowchart showing a posture estimation process executed by the imaging device according to the first embodiment. [Figure 5] This is an image in which key points and detection frames are superimposed on an input image input from the imaging control unit to the object detection unit. [Figure 6] This is an image in which key points and detection frames are superimposed on an input image input from the imaging control unit to the object detection unit. [Figure 7] 10 is a flowchart showing a posture estimation process executed by an imaging device according to a second embodiment. [Figure 8] 10 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a third embodiment. [Figure 9] 10 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a third embodiment. [Figure 10] 13 is a flowchart showing a posture estimation process executed by an imaging device according to a fifth embodiment. [Figure 11] 13 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a fifth embodiment. [Figure 12] 13 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a fifth embodiment. [Figure 13] 13 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a fifth embodiment. [Figure 14]13 is an image in which key points and detection frames are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each component constituting the present invention can be replaced with any configuration that can perform the same function. Also, any component may be added. Furthermore, any two or more configurations (features) of each embodiment can be combined.

[0012] First Embodiment A first embodiment will be described below with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device according to the first embodiment. The imaging device 100 shown in FIG. 1 is, in this embodiment, a digital still camera, a video camera, or the like to which an image processing device is applied, but is not limited thereto. The imaging device 100 includes a lens unit 101, an aperture control unit 105, a zoom control unit 113, a focus control unit 133, an image sensor 141, an image signal processing unit 142, and an imaging control unit 143. The imaging device 100 also includes a monitor display 150, a CPU (Central Processing Unit) 151, an image processing unit 152, an image compression / decompression unit 153, a RAM (Random Access Memory) 154, and a flash memory 155. The imaging device 100 also includes an operation switch 156, an image recording medium 157, a power management unit 158, a battery 159, a position / orientation change acquisition unit 161, an object detection unit 162, and a defocus calculation unit 163. These pieces of hardware included in the imaging device 100 are connected to each other so as to be able to communicate with each other via a bus 160. The lens unit 101 also includes a fixed first group lens 102, an aperture 103, an aperture motor (AM) 104, a zoom lens 111, a zoom motor (ZM) 112, a fixed third group lens 121, a focus lens 131, and a focus motor (FM) 132.

[0013] The CPU 151 is a computer that controls the operation of each hardware component. The aperture control unit 105 drives the aperture 103 via the aperture motor 104. This adjusts the aperture diameter of the aperture 103, thereby adjusting the amount of light during shooting. The zoom control unit 113 drives the zoom lens 111 via the zoom motor 112. This allows the focal length to be changed. The focus control unit 133 determines the drive amount for driving the focus motor 132 based on the amount of deviation (defocus amount) of the lens unit 101 in the focus direction. The focus control unit 133 also drives the focus lens 131 via the focus motor 132. This allows the focus adjustment state to be controlled. AF (autofocus) control is enabled by moving the focus lens 131. Note that the focus lens 131 is a focus adjustment lens, and although it is configured as a single lens in FIG. 1, it is usually configured as multiple lenses. An image of a subject is formed on the image sensor 141 via the lens unit 101. This subject image is converted into an electrical signal by the image sensor 141. The image sensor 141 is a photoelectric conversion element. The image sensor 141 has m (where "m" is an integer) pixels arranged horizontally and n (where "n" is an integer) pixels arranged vertically as light receiving elements. An image formed on the image sensor 141 and photoelectrically converted is arranged as an image signal (image data) by the image signal processing unit 142. In this way, an image is acquired on the imaging surface of the image sensor 141. In this manner, in this embodiment, the image sensor 141 and the like constitute an imaging means that captures an image by photographing a subject.

[0014] Image data is output from the imaging signal processing unit 142. The image data is transmitted to the imaging control unit 143 and temporarily stored in the RAM 154. The image data stored in the RAM 154 is compressed by the image compression / decompression unit 153 and then recorded on the image recording medium 157. In parallel with this recording, the image data stored in the RAM 154 is transmitted to the image processing unit 152. The image processing unit 152 processes the image signal to perform size reduction / enlargement processing on the image data and calculation of similarity between image data. The image data processed to an optimal size by the image processing unit 152 is displayed as an image on the monitor display 150. Furthermore, the monitor display 150 can display a preview image or a through image, and can also superimpose the object detection result of the object detection unit 162 on the image data. Furthermore, in the imaging device 100, the RAM 154 can be used as a ring buffer. This makes it possible to buffer, for example, multiple image data captured within a predetermined period, the detection results of the object detection unit 162 corresponding to each image data, and the position and orientation change of the imaging device 100 acquired by the position and orientation change acquisition unit 161.

[0015] The operation switch 156 is an input interface including, for example, a touch panel or buttons. This allows operations such as selecting various function icons displayed on the monitor display 150. The CPU 151 can determine the accumulation time of the image sensor 141 based on instructions from the operator input through the operation switch 156 or the magnitude of pixel signals of image data temporarily accumulated in the RAM 154. The CPU 151 can also determine a gain setting value when outputting from the image sensor 141 to the image signal processing unit 142. The imaging control unit 143 receives instructions regarding the accumulation time and gain setting value from the CPU 151 and controls the image sensor 141. The object detection unit 162 uses the image signal to determine an area in the image where a specific object exists. This area may be output as rectangular information or as an object area map, an image in which pixel values ​​indicate the "likelihood that the object exists." The focus control unit 133 can perform AF control for a specific object area. The aperture control unit 105 can perform exposure control using the luminance value of the specific object area. The image processing unit 152 can perform gamma correction, white balance processing, and the like based on the subject area.

[0016] The battery 159 is managed by the power management unit 158 ​​and supplies power to each piece of hardware in the imaging device 100. The flash memory 155 stores control programs necessary for the operation of the imaging device 100 and parameters used for the operation of each unit. The control programs include programs for causing a computer to execute each piece of hardware in the imaging device 100, i.e., each unit and each means (a control method for an image processing device). When the imaging device 100 is started by a user operation, i.e., when the imaging device 100 transitions from a power-off state to a power-on state, the control programs and parameters stored in the flash memory 155 are loaded into a portion of the RAM 154. The CPU 151 controls the operation of each piece of hardware in accordance with the control programs and parameters loaded into the RAM 154. The position and orientation change acquisition unit 161 is composed of position and orientation detection sensors such as a gyroscope, an acceleration sensor, and an electronic compass. The position and orientation change acquisition unit 161 measures changes in the position and orientation of the imaging device 100 relative to a scene being photographed. Information on the position and orientation change measured by the position and orientation change acquisition unit 161 is stored in the RAM 154. The defocus calculation unit 163 calculates the defocus amount for any region in the image. The defocus amount may be output at one point, or may be calculated at equal intervals across the entire image and output as a defocus map. The defocus amount is then stored in the RAM 154 and can be referenced by the image processing unit 152.

[0017] In this embodiment, an image acquired by an imaging means is input from the imaging control unit 143 to the object detection unit 162. Under the control of the CPU 151, the object detection unit 162 detects a subject in this input image and estimates the posture of the subject. In this embodiment, a bottom-up method is used for posture estimation. In the bottom-up method, when an input image contains multiple subjects (people), posture estimation is performed for these subjects simultaneously. FIG. 2 is a flowchart showing posture estimation processing when the bottom-up method is used. As shown in FIG. 2, in step S201, the object detection unit 162 simultaneously estimates the posture of all subjects contained in the input image. In this embodiment, a neural network is used for posture estimation. The neural network simultaneously detects the positions of key points of the subjects and outputs the results of tags used to determine whether the subjects are the same person.

[0018] Note that a top-down method is available for pose estimation. In the top-down method, when an input image contains multiple subjects (people), the area of ​​each subject is detected and then the pose of each subject is estimated. FIG. 3 is a flowchart showing the pose estimation process when the top-down method is used. As shown in FIG. 3, in step S301, the object detection unit 162 detects the area of ​​the subject included in the input image. In step S302, the object detection unit 162 estimates the pose of the subject within the area detected in step S301. In step S303, the object detection unit 162 determines whether or not the estimation in step S302 has been completed for the number of people detected in step S301. If it is determined that the estimation has been completed as a result of the determination in step S303, the process ends. On the other hand, if it is determined that the estimation has not been completed as a result of the determination in step S303, the process returns to step S302, and the subsequent steps are executed in order.

[0019] FIG. 4 is a flowchart showing the posture estimation process executed by the imaging device according to the first embodiment. Detailed processing in step S201 of the flowchart shown in FIG. 2 is shown in the flowchart of steps S401 and S402 in FIG. 4. As shown in FIG. 4, in step S401, the object detection unit 162 detects multiple key points of a subject included in an input image from the imaging control unit 143, and tags for each key point, for example, using a neural network (detection process). While the subject is a person (human) in this embodiment, the subject is not limited to this and may be, for example, an animal other than a human. Furthermore, when the subject is a person, at least one of the following body parts is detected (extracted): pupils, ears, top of the head (vertex), neck, shoulders, elbows, wrists, waist, knees, and ankles. These key points are feature points that can contribute to estimating the posture of the person. The key points include position information of which body part of the person they are and a likelihood indicating the accuracy of the position information. The tags are used to identify the same person for each key point. The tag includes a classification (person identification information) indicating which person each keypoint belongs to, and a likelihood indicating the accuracy of the classification. In this manner, in this embodiment, the object detection unit 162 also functions as a detection unit that detects keypoints and tags.

[0020] In step S402, the object detection unit 162 connects the keypoints detected in step S401 with lines (connection step). In this embodiment, the object detection unit 162 also functions as a connection unit that connects keypoints with lines. The object detection unit 162 then estimates the posture of the person based on the connection result of connecting keypoints with lines. In this embodiment, the object detection unit 162 also functions as an estimation unit that estimates the posture of the person whose posture is to be determined. In step S402, the connection is usually determined based on the positions between keypoints and the likelihood of tags. In this embodiment, the connection is made using tag information, but the connection may also be determined by subject segmentation. Note that if the likelihood of classification by tag is equal to or greater than a predetermined threshold and the maximum value is used, the connection between keypoints is confirmed when the keypoints are detected in step S401, and step S402 can effectively be omitted. In such a case, steps S401 and S402 in FIG. 4 may be expressed as one step, like step S201 in FIG.

[0021] Step S403 is executed in parallel with steps S401 and S402. In step S403, object detection unit 162 extracts a detection frame for a person included in the input image from imaging control unit 143, for example, using a neural network (extraction step). The detection frame indicates a detection range for the person by surrounding, in the input image, the body parts of all people whose postures are to be determined, with a rectangle. The range surrounded by this detection frame is preferably large enough to sufficiently identify the body parts. For example, the detection frame preferably surrounds the entire body, entire upper body, entire head, etc. of the person whose posture is to be determined. Furthermore, in step S403, along with the extraction of the detection frame, the center of the detection frame and the vertical and horizontal lengths of the detection frame are also extracted. As such, in this embodiment, object detection unit 162 also functions as extraction means for extracting a detection frame. Furthermore, for example, if there is a face detection frame, a head detection frame, an upper body detection frame, and a whole body detection frame, a determination as to whether these detection frames are for the same person is made based on the degree of overlap between the detection frames and the distance between the detection frames.

[0022] In step S404, the object detection unit 162 determines that the person is running, for example, based on the person's posture estimation result in step S402. In this embodiment, the reliability of the posture is determined by comparing the positional relationship between the key points (key point group) connected in step S402 and the detection frame extracted in step S403.

[0023] In step S405, the object detection unit 162 issues a processing instruction, such as switching the focus range of the imaging device 100, via the CPU 151 in accordance with the determination result in step S404.

[0024] Step S406 is executed. In step S406, the object detection unit 162 determines whether the person whose key points were detected in step S401 and the person whose detection frame was extracted in step S403 are the same person (determination step). This determination is made, for example, based on how the area formed by multiple key points overlaps with the detection frame or the distance between the key points and the detection frame. The comparison can be made by forming the smallest circle or rectangle that includes key points corresponding to the top of the head and neck and comparing it with the detection frame for the head, or by comparing a rectangle connecting key points corresponding to the left and right shoulders and waist with the frame for the upper body. Note that a simple comparison can also be made based on the distance between the key points and the detection frame. If it is determined that the people are the same person, the object detection unit 162 checks whether there is any inconsistency between the key points determined to be the same and the detection frame, as described below.

[0025] FIG. 5 shows an image in which keypoints and detection frames are superimposed on an input image input from the imaging control unit to the object detection unit. In image 500 shown in FIG. 5, an outer solid line 501 indicates the range of image 500. This image 500 includes person A and person B. Person A is located in the lower left of image 500 and shows the upper chest. Person B is located in the center of image 500 and shows the entire body. Detection frame 502 is the detection result of person A's head and is indicated by a dotted line surrounding person A's head. The following keypoints have been detected: keypoint KP1 corresponding to the top of the head, keypoint KP2 corresponding to the neck, keypoint KP3 corresponding to the left shoulder, and keypoint KP4 corresponding to the right shoulder. Keypoint KP5 corresponding to the left hip, keypoint KP6 corresponding to the left knee, and keypoint KP7 corresponding to the left ankle have also been detected. Furthermore, keypoint KP8 corresponding to the right hip, keypoint KP9 corresponding to the right knee, and keypoint KP10 corresponding to the right ankle are also detected. Of keypoints KP1 to KP10, keypoints KP1 to KP4 are keypoints of person A. Also, keypoints KP5 to KP10 are actually keypoints of person B, but have been detected as keypoints of the hips, knees, and feet that are not visible in person A. In this way, multiple keypoints are determined to be the same subject by the object detection unit 162 (keypoint identity determination means) (keypoint identity determination process).

[0026] Here, "identical (sameness)" means that the person is the same person. Key points KP1 to KP10 are connected by dashed lines to form key point group KPG1. Therefore, based on key point group KPG1, the object detection unit 162 detects that person A is in a posture that looks like he is lying down. Note that the key points are not limited to key points KP1 to KP10, and key points other than these may also be detected.

[0027] For example, suppose that the top of the head keypoint KP1 is at ±30 degrees from the neck keypoint KP2 around the vertical axis in image 500, and the center of the head detection frame 502 is within two detection frame 502 units from the bottom edge of image 500. In this case, person A is not bending their body or neck, and their upper body is cut off, so it is highly likely that the waist, knees, and ankles keypoints are not captured. When there is an inconsistency in the relationship between the detection frame 502 and the keypoint detection positions (keypoints KP5 to 10) as described above, the object detection unit 162 (reliability determination means) lowers the reliability of the posture and makes a determination (reliability determination step). When lowering the reliability of the posture, one of the following three processes is selected as appropriate:

[0028] The first process is to reduce the reliability of keypoints KP5 to 10. As a specific example, some of the keypoints included in the keypoint group KPG1, i.e., keypoints KP5 to 10 at the waist, knees, and ankles, may not be used, or the likelihood of detection may be reduced for processing in the next step.

[0029] The second process is a process for lowering the reliability of the detection frame 502. Specifically, for example, the detection frame 502 is not used in the next process and thereafter.

[0030] The third process is a process for reducing the reliability of posture estimation based on the keypoint group KPG1. Specifically, for example, if a posture is initially estimated to be lying down based on the keypoint group KPG1, but the positional relationship between the detection frame 502 and the keypoint KP1 on the top of the head indicates that the posture appears to be standing, the result is not deemed to be a lying down posture. By adding such a process for reducing the reliability (hereinafter referred to as "reliability reduction process"), it is possible to reduce the number of times that the determination in step S404 is an erroneous determination, i.e., that person A is determined to be in a posture that looks like he is lying down. It is also possible to prevent an erroneous process in step S405 based on the determination result in step S404.

[0031] FIG. 6 shows an image in which keypoints and detection frames are superimposed on an input image input from the imaging control unit to the object detection unit. FIG. 6 shows an example in which keypoints for the shoulders and waist have been removed to reduce the amount of calculation. In image 600 shown in FIG. 6, a solid line 601 outside the image 600 indicates the range of the image 600. This image 600 includes person A and person B. Person A is located in the lower left of image 600 and is shown from the waist up. Person B is located slightly to the right of the center of image 600 and is shown entirely in the image. Detection frame 602 is the detection result for the entire upper body of person A and is indicated by a dotted line surrounding the entire upper body of person A. Keypoints detected include keypoint KP1 corresponding to the top of the head and keypoint KP2 corresponding to the neck. Keypoint KP21 corresponding to the left elbow, keypoint KP22 corresponding to the left wrist, keypoint KP23 corresponding to the right elbow, and keypoint KP24 corresponding to the right wrist are also detected. Furthermore, keypoint KP6 corresponding to the left knee, keypoint KP7 corresponding to the left ankle, keypoint KP9 corresponding to the right knee, and keypoint KP10 corresponding to the right ankle are also detected. Of these keypoints, keypoint KP1, keypoint KP2, and keypoints KP21 to KP24 are keypoints of person A. Also, keypoint KP6, keypoint KP7, keypoint KP9, and keypoint KP10 are actually located at the positions of person B, but have been detected as keypoints of the knees and feet that are not visible in person A. As a result, these keypoints form a keypoint group KPG2 connected by dashed lines. For this reason, the object detection unit 162 detects, based on the keypoint group KPG2, that person A is lying down or in a position similar to lying down.

[0032] For example, suppose that the top of the head keypoint KP1 is located at ±30 degrees from the neck keypoint KP2 around the vertical axis in image 500, and the vertical length of the upper body detection frame 602 is at least twice its horizontal length (aspect ratio of 2). In this case, it is highly likely that person A is actually standing, and the knee and ankle keypoints KP6, KP, KP9, and KP10 are not positioned higher than the elbow keypoints KP21 and KP23. Note that in the case of a lying down posture, the ratio of the vertical length to the horizontal length of the upper body detection frame 602 is smaller than the ratio of the vertical length to the horizontal length of the detection frame 602 in the standing-like posture described above. Furthermore, the positional relationship between the top of the head keypoint KP1 and the neck keypoint KP2 also differs between a lying down posture and a standing posture. When there is an inconsistency between the relationship between the detection frame 602 and the keypoint detection positions (keypoints KP6, KP, KP9, KP10) as described above, a determination process is performed in step S404, and a reliability reduction process is performed in step S406. The reliability reduction process reduces the number of times the determination in step S404 is an erroneous determination, that is, a determination that person A is lying down or in a sleeping position. This makes it possible to determine, for example, that person A is in a standing position. Furthermore, although the number of detected keypoints used in posture determination is 10 in both FIGS. 5 and 6, this is not limiting, and posture determination can be performed even if the number is less than 10 or more than 10, for example.

[0033] As described above, in this embodiment, reliability reduction processing can be added based on information such as the position and likelihood of keypoints and the position, size, and likelihood of the detection frame. This can prevent errors in the processing in steps S404 and S405. In addition, in this embodiment, the head orientation is not considered based on the position of the person's eyes and ears, but this is not limited to this. For example, by considering the head orientation, it may be possible to easily determine whether the person is in an impossible posture. Furthermore, depending on the situation, such as a specific event or sport, normal postures may be restricted, and therefore an impossible posture may be determined. Furthermore, if there is an inconsistency between the detection frame and the keypoint detection results, the reliability of the detection frame may be reduced. Whether the keypoint or the detection frame is more reliable depends on the likelihood of the respective detection results, the performance of the detector, the scene, etc.

[0034] Second Embodiment The second embodiment will be described below with reference to FIG. 7. The description will focus on differences from the previously described embodiment, and similar details will not be repeated. FIG. 7 is a flowchart showing a posture estimation process executed by an image capture device according to the second embodiment. The target of the posture estimation process is, for example, image 500 shown in FIG. 5. As described above, for example, assume that the top of the head keypoint KP1 is located at ±30 degrees from the neck keypoint KP2 about the vertical axis in image 500, and the center of the head detection frame 502 is located within two detection frame 502 units from the bottom edge of image 500. In this case, the upper half of person A's body is likely to be cut off, and the waist, knee, and ankle keypoints are likely not captured. For this reason, in this embodiment, as shown in FIG. 7, in step S704 before step S705, the reliability of the waist, knee, and foot keypoints of person A is lowered for determination.

[0035] In the flowchart shown in FIG. 7, steps S701, S702, and S703 are executed. Steps S701, S702, and S703 are respectively the same as steps S401, S402, and S403 in the flowchart shown in FIG. 4. After executing steps S702 and S703, the process proceeds to step S704. In step S704, the object detection unit 162 determines the posture by excluding key points KP5 to KP10 of the waist, knees, and feet. This prevents the determination result in step S705 and the processing result in step S706 from being erroneous. Note that steps S705 and S706 are the same as steps S404 and S405 in the flowchart shown in FIG. 4. In this embodiment, posture determination is performed by excluding some key points, but this is not limited to this. For example, posture determination can be performed by taking into account the reduction in likelihood when reducing the reliability of key points. This makes it possible to prevent the determination result in step S705 and the processing result in step S706 from being incorrect. Furthermore, the same effect as in the case of image 500 shown in Fig. 5 can be obtained for image 600 shown in Fig. 6.

[0036] <Third embodiment> The third embodiment will be described below with reference to FIG. 8. Differences from the previous embodiment will be mainly described, and similar points will not be described again. This embodiment is similar to the first embodiment except for an example related to the reliability of keypoints. FIGS. 8 and 9 each show an image in which keypoints and a detection frame are superimposed on an input image input from an imaging control unit to an object detection unit of an imaging device according to the third embodiment. In image 800 shown in FIG. 8, a solid line 801 outside the image 800 indicates the range of the image 800. This image 800 includes person A and person B. Person A is located on the left side of image 800. Person B is located on the right side of image 800. Person A is located behind person B, and part of their body, i.e., in FIG. 8, the left arm of person A and the right arm of person B, overlap. In this overlapping portion (overlapping portion), the keypoint and posture determination are expected to be lower than usual (although the detection result may also be low, the reliability is controlled based on rules to lower the reliability). For this reason, it is preferable to lower the reliability of the determination based on the keypoints as in the first embodiment, or to lower the reliability of the keypoints as in the second embodiment, thereby preventing erroneous posture determination results and processing based on the posture determination results.

[0037] Note that detection frame 802 is the detection result of the entire upper body of person A, and is indicated by a dotted line surrounding the entire upper body of person A. Detection frame 812 is the detection result of the entire upper body of person B, and is indicated by a dotted line surrounding the entire upper body of person B. Furthermore, the following keypoints have been detected: keypoint KP1 corresponding to the top of the head, keypoint KP2 corresponding to the neck, and keypoint KP3 corresponding to the left shoulder. Additionally, keypoint KP21 corresponding to the left elbow, keypoint KP22 corresponding to the left wrist, keypoint KP23 corresponding to the right elbow, and keypoint KP24 corresponding to the right wrist. Additionally, keypoint KP31 corresponding to the left hip, keypoint KP32 corresponding to the left knee, and keypoint KP33 corresponding to the left ankle have also been detected. Furthermore, keypoint KP34 corresponding to the right hip, keypoint KP35 corresponding to the right knee, and keypoint KP36 corresponding to the right ankle have also been detected. These keypoints are connected by dashed lines to form keypoint group KPG3. 4, the object detection unit 162 compares a detection frame 802 that surrounds the upper body of person A with a detection frame 812 that is adjacent to the detection frame 802 and surrounds the upper body of person B. Then, the reliability of the detection frame with the fewer key points is reduced.

[0038] Furthermore, in image 800′ shown in FIG. 9, even if the positional relationship between person A and person B is the same as in image 800 shown in FIG. 8, some of the key points of person A may not be detected. Even in such a case, in step S404 of the flowchart shown in FIG. 4, the object detection unit 162 compares a detection frame 802 surrounding the upper body of person A with a detection frame 812 surrounding the upper body of person B, who is adjacent to person A and determined not to be person A. If the overlap is equal to or greater than a predetermined amount, the reliability of the posture determination result is reduced. Note that in this embodiment, a state in which person A is located behind person B is illustrated as an example, but a state in which person A is located in front of person B may also be used. Furthermore, the amount of reduction in reliability may be changed (adjusted) depending on the distance between detection frame 802 and detection frame 812 and the degree of overlap. Furthermore, as in image 800′ shown in FIG. 9, a criterion for reducing the reliability of posture determination may be that the number of detected key points is less than a predetermined number.

[0039] <Fourth embodiment> The fourth embodiment will be described below, focusing on differences from the previously described embodiments, and similar aspects will not be described again. In this embodiment, the example of keypoint reliability is the same as in the third embodiment, and the flowchart is the same as in the second embodiment. In this embodiment, as in image 800 shown in FIG. 8 and image 800′ shown in FIG. 9, a detection frame 802 surrounding the upper body of person A and a detection frame 812 surrounding the upper body of person B overlap. If the degree of overlap between the detection frames is a predetermined level, the reliability of keypoints KP3, KP21, and KP22, which are located on the detection frame 812 side adjacent to detection frame 802, among the keypoints included in the keypoint group, is reduced. In this embodiment, the previously described low-reliability keypoints are excluded in step S704 of the flowchart shown in FIG. 7, and posture determination is performed in step S705. In this embodiment, the amount of reliability reduction may be changed or the key points for reducing reliability may be changed depending on the distance or degree of overlap between detection frame 802 and detection frame 812.

[0040] Fifth Embodiment The fifth embodiment will be described below with reference to FIGS. 10 to 14. Differences from the previous embodiments will be mainly described, and similar points will not be described again. In this embodiment, key point connections are determined based on the positional relationship between detection frames of the face, head, upper body, etc. determined to be the same subject and key points such as the vertex of the head and joints. FIG. 10 is a flowchart showing a posture estimation process executed by an image capture device according to the fifth embodiment. In the flowchart shown in FIG. 10, steps S1001 and S1003 are executed in parallel. Steps S1001 and S1003 are the same as steps S401 and S403, respectively, in the flowchart shown in FIG. 4. After steps S1001 and S1003 are executed, the process proceeds to step S1002. In step S1002, the object detection unit 162 uses detection frames determined to be the same subject to connect key points. For example, the object detection unit 162 determines the key point connection range, i.e., the key points to be connected, based on the position and shape of the detection frames of the head and upper body. The connection range can be determined, for example, by determining whether to connect keypoints based on distance information between them.Other methods include creating a cost function based on the likelihood of keypoints, tag IDs, and likelihood information, and connecting the combination that minimizes the cost function.

[0041] In this embodiment, the ranges of the positions of the top of the head, neck, shoulders, waist, and knees, as well as the keypoints to be connected, are determined based on the positions and shapes of the detection frames surrounding the head and the upper body. Then, as a cost function, the cost within the range is set to 0 and the cost outside the range is set to ∞, preventing the connection of keypoints outside the range. This improves the accuracy of connecting keypoints. Even if a keypoint is included within the range, the cost can be changed depending on the position of the keypoint. In this embodiment, the keypoints for the top of the head and neck are within the range of the detection frame surrounding the head. However, taking into account keypoint detection errors, the predetermined range can be set to, for example, 1.3 times the size of the detection frame from the center of the detection frame. If the keypoints for the top of the head and neck are outside the range, the keypoints are not connected. In this embodiment, the positions of the top of the head and neck can be predicted based on the positions and shapes of the detection frame surrounding the head and the detection frame surrounding the upper body. Therefore, the range may be further restricted or the cost function may be modified. In this embodiment, similarly, ranges can be set for the shoulder, waist, and knee key points according to the positions and shapes of the detection frames surrounding the head and the upper body. Furthermore, the orientation of the body can be predicted based on the number and positions of detected pupils. This prediction result can be used to connect key points.

[0042] After step S1002 is executed, the process proceeds to step S1004 and step S1005 in this order. Steps S1004 and S1005 are the same as steps S404 and S405, respectively, in the flowchart shown in FIG.

[0043] 11 to 14 each show an image in which keypoints and detection frames are superimposed on an input image input from the imaging control unit of the image capturing device according to the fifth embodiment to the object detection unit. In image 1100 shown in FIG. 11, an outer solid line 1101 indicates the range of the image 1100. This image 1100 includes person A. Person A is located in the center of the image 1100, standing upright, and his or her entire body is captured. Image 1100 also includes detection frames 1102 and 1103. Detection frame 1102 is indicated by a dotted line and surrounds person A's head. Detection frame 1103 is indicated by a dotted line and surrounds person A's upper body. As shown in FIG. 11, when person A is posing upright, detection frame 1103 forms a vertically long rectangle in image 1100. All detected keypoints are those of person A and are connected by dashed lines to form keypoint group KPG3. In this case, the shoulders (key points KP3 and KP4) are located between near the detection frame 1102 and near the center of the long side of the detection frame 1103. The waist (key points KP31 and KP34) are located near the bottom edge of the detection frame 1103. The knees (key points KP32 and KP35) are located near the bottom edge of the detection frame 1103 or outside the bottom edge.

[0044] In image 1200 shown in FIG. 12, an outer solid line 1201 indicates the range of image 1200. This image 1200 includes person A. Person A is located at the bottom of image 1200, and is depicted with their entire body in a reclined or lying position. Image 1200 also includes detection frame 1202 and detection frame 1203. Detection frame 1202 is indicated by a dotted line and surrounds the head of person A. Detection frame 1203 is indicated by a dotted line and surrounds the upper body of person A. Furthermore, as shown in FIG. 12, when person A is in an upright position, detection frame 1203 forms a horizontally long rectangle in image 1200. In this case, as in FIG. 11, the shoulders are located between near detection frame 1202 and near the center of the long side of detection frame 1203. Furthermore, the waist is located near the right end of detection frame 1203. The knee is located near the right end or outside the right end of the detection frame 1203. In this embodiment, by setting a range of key points that can be assumed from the position of the detection frame and a range of connections between key points, it is possible to reduce such erroneous connections. Note that the number of detection frames is not limited to two, and may be, for example, three or more.

[0045] In image 1300 shown in FIG. 13, an outer solid line 1301 indicates the range of image 1300. This image 1300 includes person A and person B. Person A is located in the center of image 1300, standing and showing the entire upper body. Person B's head is shown behind person A. Image 1300 also includes detection frame 1302 and detection frame 1303. Detection frame 1302 is shown with a dotted line and surrounds the head of person A. Detection frame 1303 is shown with a dotted line and surrounds the upper body of person A. The following key points have been detected: key point KP41 corresponding to the top of the head, key point KP2 corresponding to the neck, key point KP3 corresponding to the left shoulder, and key point KP4 corresponding to the right shoulder. Additionally, keypoint KP21 corresponding to the left elbow, keypoint KP22 corresponding to the left wrist, keypoint KP23 corresponding to the right elbow, and keypoint KP24 corresponding to the right wrist are also detected. Furthermore, keypoint KP31 corresponding to the left hip and keypoint KP34 corresponding to the right hip are also detected. Of these keypoints, keypoint KP41 is a keypoint of person B, and the remaining keypoints are keypoints of person A. Keypoint KP41 is actually a keypoint of person B, but has been detected as a keypoint of person A. As a result, these keypoints form a keypoint group KPG4 connected by dashed lines. By limiting the range of keypoint KP41a to the vicinity of the detection frame, it is possible to reduce the likelihood that keypoint KP41, which is actually a keypoint of person B, will be mistakenly connected with a keypoint of person A, as shown in FIG. 13.

[0046] In image 1400 shown in FIG. 14, an outer solid line 1401 indicates the range of image 1400. This image 1400 includes person A. Person A is located in the center of image 1400 and is running, with the upper half of the body leaning forward. Image 1400 also includes detection frame 1402 and detection frame 1403. Detection frame 1402 is indicated by a dotted line and surrounds the head of person A. Detection frame 1403 is indicated by a dotted line and surrounds the upper half of person A's body. When person A's upper body is leaning forward, the difference (ratio) between the vertical length and horizontal length of detection frame 1403 is smaller than when person A is standing. In this state, within detection frame 1403, the waist (key points KP31 and KP34) is located diagonally from the position of detection frame 1402. Therefore, by setting a waist range or reflecting the position of the keypoint in the cost function, it is possible to reduce erroneous connections between keypoints.

[0047] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and various modifications and variations are possible within the spirit and scope of the present invention. The present invention can also be realized by a process in which a program implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the image capture device 100 is a device incorporating an image processing device, and the posture estimation process is performed within the device, but this is not limited thereto. For example, the image capture device 100 may be communicably connected to an information processing device (server), and the information processing device may be a device incorporating the image processing device. In this case, the information processing device performs the posture estimation process. The information processing device is not particularly limited, and for example, a desktop or notebook personal computer, a tablet terminal, a smartphone, etc. can be used.

[0048] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A detection means for detecting a plurality of key points for a subject included in an image; a keypoint identity determination means for determining whether a plurality of keypoints correspond to the same subject; extraction means for extracting a detection frame that surrounds a body part of a subject and indicates a detection range of the subject; a determination means for determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, The reliability determination means reduces the reliability of at least one of the detection frame and the key point, or reduces the reliability of the posture determination of the subject using the key point determined to be identical, depending on at least the positional relationship between the key point detected by the detection means and determined to be the same subject, and the detection frame determined to be identical to the key point. (Configuration 2) The image processing device described in Configuration 1, characterized in that the reliability determination means reduces the reliability of at least one of the detection frame and the keypoint, depending on the positional relationship between a keypoint determined to be the same subject and a detection frame adjacent to the detection frame determined to be the same as the keypoint but determined to be not the same, or reduces the reliability of the subject pose estimation using the keypoint determined to be the same. (Configuration 3) The image processing device according to configuration 1 or 2, characterized in that the keypoint identity determination means makes a determination using the position and likelihood of the detected keypoint, tag information representing the classification of the subject, and the likelihood of the tag information. (Configuration 4) The image processing device described in any one of configurations 1 to 3, characterized in that the judgment means judges whether the subject from which the key point was detected is the same subject or not based on the degree of overlap or distance between the subject from which the detection frame was extracted, or the degree of overlap or distance between an area formed by multiple key points judged to be the same key point and the detection frame. (Configuration 5) The subject is a human being, 5. The image processing device according to any one of configurations 1 to 4, wherein, when the detection frame includes the upper body of the person, the reliability of at least one of the detection frames determined to be the same subject is reduced, or the reliability of the determination is reduced, depending on the aspect ratio of the detection frame determined to be the same subject and the positional relationship of the key points. (Configuration 6) The subject is a human being, 6. The image processing device according to any one of configurations 1 to 5, wherein, when the detection frame includes the human head, the reliability of at least one of the detection frame and the key point indicating the head is reduced depending on a positional relationship between the detection frame and the key point indicating the head, or the reliability of the determination is reduced. (Configuration 7) An image processing device described in any one of configurations 1 to 6, characterized in that when the extraction means extracts multiple detection frames and an overlapping portion is formed in which a detection frame partially overlaps with a detection frame that is determined not to be of the same subject, the reliability of the keypoint located within the overlapping portion is reduced. (Configuration 8) The image processing device according to any one of configurations 1 to 7, further comprising a connection means for connecting the key points detected by the detection means with straight lines. (Configuration 9) A detection means for detecting a plurality of key points for a subject included in an image; a connecting means for connecting the key points detected by the detecting means with straight lines; an extraction means for extracting a detection frame that is included in the image and surrounds a body part of a subject to be determined whose posture or movement is to be determined, and indicates a detection range of the subject; a determination means for determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, When the determination means determines that the objects are the same subject, the connection means determines which of the key points detected by the detection means are to be connected, based on at least the positional relationship between the key points detected by the detection means and the detection frame extracted by the extraction means. (Configuration 10) The image processing device according to configuration 9, wherein the connection means determines the key points to be connected in accordance with the size of the detection frame. (Configuration 11) The image processing device described in Configuration 9 or 10, characterized in that the judgment means judges whether the subject from which the key point was detected and the subject from which the detection frame was extracted are the same subject based on the degree of overlap between the subject from which the key point was detected and the subject from which the detection frame was extracted, or the distance between the subject from which the key point was detected and the subject from which the detection frame was extracted. (Configuration 12) The image processing device according to any one of configurations 1 to 11, wherein the detection means detects at least one of the subject's pupils, ears, top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles as the key points. (Configuration 13) The image processing device according to any one of configurations 1 to 12, characterized in that the image processing device is an imaging device including an imaging means for photographing a subject and acquiring the image. (Method 1) A method for controlling an image processing device, comprising: a detection step of detecting a plurality of key points for a subject included in an image; a keypoint identity determination step of determining whether a plurality of keypoints correspond to the same subject; an extraction step of extracting a detection frame that surrounds a body part of the subject and indicates a detection range of the subject; a determination step of determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, A control method for an image processing device, characterized in that in the reliability determination process, the reliability of at least one of the detection frame and the keypoints detected in the detection process and determined to be the same subject is reduced depending on the positional relationship between at least the keypoints detected in the detection process and determined to be the same subject, and the detection frame determined to be the same as the keypoints, or the reliability of the posture determination of the subject using the keypoints determined to be the same. (Method 2) A method for controlling an image processing device, comprising: a detection step of detecting a plurality of key points for a subject included in an image; a connecting step of connecting the key points detected in the detecting step with straight lines; an extraction step of extracting a detection frame that is included in the image and surrounds a body part of a subject to be determined whose posture or movement is to be determined, and indicates a detection range of the subject; a determination step of determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, a control method for an image processing device, characterized in that, when the determination in the determination step determines that the objects are the same subject, the connection step determines, from among the key points detected in the detection step, key points to be connected, depending on at least the positional relationship between the key points detected in the detection step and the detection frame extracted in the extraction step. (Program 1) A program for causing a computer to execute each means of the image processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0049] 100 Imaging device 141 Image sensor 162 Object detection unit 502 detection frame KP1~KP10 Key Points

Claims

1. a detection means for detecting a plurality of key points for a subject included in an image; a keypoint identity determination means for determining whether a plurality of keypoints correspond to the same subject; extraction means for extracting a detection frame that surrounds a body part of the subject and indicates a detection range of the subject; a determination means for determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, The reliability determination means reduces the reliability of at least one of the detection frame and the key point, or reduces the reliability of the posture determination of the subject using the key point determined to be identical, depending on at least the positional relationship between the key point detected by the detection means and determined to be the same subject, and the detection frame determined to be identical to the key point.

2. 2. The image processing device according to claim 1, wherein the reliability determination means reduces the reliability of at least one of the detection frame and the keypoint, or reduces the reliability of subject pose estimation using keypoints determined to be identical, depending on a positional relationship between a keypoint determined to be the same subject and a detection frame adjacent to the detection frame determined to be the same as the keypoint but determined to be not the same.

3. 2. The image processing apparatus according to claim 1, wherein the keypoint identity determining means makes a determination using the positions and likelihoods of the detected keypoints, tag information indicating the classification of the subject, and the likelihoods of the tag information.

4. 2. The image processing device according to claim 1, wherein the determination means determines whether the subject from which the key point is detected is the same subject based on the degree of overlap or distance between the subject from which the detection frame is extracted, or based on the degree of overlap or distance between an area formed by multiple key points determined to be the same key point and the detection frame.

5. the subject is a human being, 2. The image processing device according to claim 1, wherein, when the detection frame includes the upper body of the person, the reliability of at least one of the detection frames determined to be the same subject is reduced, or the reliability of the determination is reduced, depending on the aspect ratio of the detection frame determined to be the same subject and the positional relationship of the key points.

6. the subject is a human being, 2. The image processing device according to claim 1, wherein, when the detection frame includes the human head, the reliability of at least one of the detection frame and the key point indicating the head is reduced depending on a positional relationship between the detection frame and the key point indicating the head, or the reliability of the determination is reduced.

7. 2. The image processing device according to claim 1, wherein when a plurality of detection frames are extracted by the extraction means and an overlapping portion is formed in which a detection frame partially overlaps with a detection frame that is determined not to be of the same subject, the reliability of the key point located within the overlapping portion is reduced.

8. 2. The image processing apparatus according to claim 1, further comprising: a connection means for connecting the key points detected by the detection means with straight lines.

9. a detection means for detecting a plurality of key points for a subject included in an image; a connecting means for connecting the key points detected by the detecting means with straight lines; an extraction means for extracting a detection frame that is included in the image and surrounds a body part of a subject to be determined whose posture or movement is to be determined, and indicates a detection range of the subject; a determination means for determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, When the determination means determines that the objects are the same subject, the connection means determines which of the key points detected by the detection means are to be connected, based on at least the positional relationship between the key points detected by the detection means and the detection frame extracted by the extraction means.

10. 10. The image processing apparatus according to claim 9, wherein the connection means determines the key points to be connected in accordance with the size of the detection frame.

11. 10. The image processing device according to claim 9, wherein the determination means determines whether the subject from which the key point is detected and the subject from which the detection frame is extracted are the same subject based on the degree of overlap between the subject from which the key point is detected and the subject from which the detection frame is extracted, or based on the distance between the subject from which the key point is detected and the subject from which the detection frame is extracted.

12. 10. The image processing device according to claim 1, wherein the detection means detects at least one of the subject's eyes, ears, top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles as the key points.

13. 10. The image processing device according to claim 1, wherein the image processing device is an image capturing device that captures an image of a subject and acquires the image.

14. 1. A method for controlling an image processing device, comprising: a detection step of detecting a plurality of key points for a subject included in an image; a keypoint identity determination step of determining whether a plurality of keypoints correspond to the same subject; an extraction step of extracting a detection frame that surrounds a body part of the subject and indicates a detection range of the subject; a determination step of determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, A control method for an image processing device, characterized in that in the reliability determination process, the reliability of at least one of the detection frame and the keypoints detected in the detection process and determined to be the same subject is reduced depending on the positional relationship between at least the keypoints detected in the detection process and determined to be the same subject, and the detection frame determined to be the same as the keypoints, or the reliability of the posture determination of the subject using the keypoints determined to be the same.

15. 1. A method for controlling an image processing device, comprising: a detection step of detecting a plurality of key points for a subject included in an image; a connecting step of connecting the key points detected in the detecting step with straight lines; an extraction step of extracting a detection frame that is included in the image and surrounds a body part of a subject to be determined whose posture or movement is to be determined, and indicates a detection range of the subject; a determination step of determining whether the subject from which the key points are detected and the subject from which the detection frame is extracted are the same subject, a control method for an image processing device, characterized in that, when the determination in the determination step determines that the objects are the same subject, the connection step determines, from among the key points detected in the detection step, key points to be connected, depending on at least the positional relationship between the key points detected in the detection step and the detection frame extracted in the extraction step.

16. 10. A program for causing a computer to execute each means of the image processing apparatus according to claim 1.

Images