Image processing apparatus, imaging apparatus, object size identification method, program and storage medium

Abstract

To enable stably identifying size of an object in an image depending on a shape of the object.SOLUTION: An image processing apparatus comprises: selection means that selects an object which is a detection target; acquisition means that acquires an image; detection means that detects the object selected by the selection means, from the image; and identification means that identifies size in the image of the object detected by the detection means. The identification means changes a region that is identified as the size of the object in the image depending on a shape of the object.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an object size specifying method, a program, and a storage medium, and more particularly to a technique for specifying the size of an object on an image.

Background Art

[0002] Conventionally, in the autofocus control of a camera, a main subject is selected from among a plurality of people, and focus control is performed while following the main subject so as to continue taking a group photo. In particular, in a sports competition using a ball, since a player holding the ball is likely to be the main subject, a method of selecting the main subject based on the detection information of the ball has been disclosed. For example, in Patent Document 1, a technique of detecting a ball in a captured image and selecting a person at a distance close to the detected ball on the screen is adopted.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in a situation where a plurality of people are standing on the near side and the far side in the optical axis (z-axis) direction of the camera, and a ball exists at an intermediate point among them, it is difficult to determine which person the ball is actually close to only based on the information on the screen (xy plane). In such a situation, it is effective to compare the size of the ball on the screen with the size of the human head, and use the comparison result to select the person closer to the ball as the main subject in consideration of the position in the optical axis (z-axis) direction of the camera. For this reason, it is desired to improve not only the detection of the ball but also the measurement accuracy of the size of the ball on the screen.

[0005] In sports competitions using balls, there are two factors that make it difficult to measure the ball size. The first factor is the measurement of the ball size in competitions using oval balls. For example, in competitions such as American football and rugby, oval balls are used. Since the shape of an oval ball changes depending on the viewing angle even if the ball is at the same distance from the camera in the direction of the camera's optical axis (z-axis), the ball size on the screen may increase or decrease depending on the angle of the ball. This can happen not only with balls but also with objects whose shape changes depending on the viewing angle, such as a frisbee.

[0006] Also, the second factor is the measurement of the ball size in a scene where the ball is blocked. For example, in competitions such as American football, rugby, and basketball where players can easily hold the ball, the ball is likely to be hidden, making it difficult to detect the ball itself before measuring its size.

[0007] The present invention has been made in view of the above problems, and an object thereof is to stably specify the size of an object in an image according to the shape of the object.

Means for Solving the Problems

[0008] To achieve the above object, an image processing apparatus of the present invention includes a selection means for selecting an object to be detected, an acquisition means for acquiring an image, a detection means for detecting the object selected by the selection means from the image, and a specifying means for specifying the size of the object in the image detected by the detection means. The specifying means changes the part to be specified as the size of the object in the image according to the shape of the object.

Effects of the Invention

[0009] According to the present invention, the size of an object in an image can be stably specified according to the shape of the object.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0012] <First Embodiment> FIG. 1 is a block diagram showing the functional configuration of an imaging device 10, configured as an image processing device including an object detection unit 11 that detects an object and specifies the size of the object from an image obtained by shooting in the first embodiment of the present invention.

[0013] The imaging device 10 includes an imaging unit 150, an object selection unit 151, an object detection unit 11, a person feature amount detection unit 152, a main subject selection unit 113, and a focus control unit 153. The object detection unit 11 includes an object feature amount detection unit 111 and an object size specification unit 112.

[0014] The user uses the object selection unit 151 to select an object to be detected from a still image or a moving image obtained by the imaging unit 150. FIG. 2 is a diagram showing an example of a user interface displayed on a display unit when the object selection unit 151 is configured by an operation unit such as a display unit and a touch panel or a direction button (not shown). The user can select from the options displayed on the display unit by operating an operation unit such as a touch panel or a direction button. Here, examples of selecting a competition (FIG. 2(a)) and directly selecting an object to be detected (FIG. 2(b)) are shown. When the type of competition is selected, the competition and the object to be detected (for example, a tool such as a ball used in the selected competition) are associated in advance, and the object associated with the selected competition is set as the detection target. In addition, the competition and object to be selected are not limited to those shown in FIG. 2, and the display method of the options is not limited to that shown in FIG. 2.

[0015] Based on the selection by the object selection unit 151, the object feature amount detection unit 111 detects an object (for example, a ball) selected from a still image or a moving image obtained by the imaging unit 150, and obtains the feature amount of the detected object (hereinafter referred to as "object feature amount"). The object size specification unit 112 measures the detected object on the image and specifies the final object size (hereinafter referred to as "object size") based on the obtained value.

[0016] The human feature quantity detection unit 152 (subject detection means) detects a person from a still image or a moving image obtained by the imaging unit 150, and obtains the feature quantity of the detected person (hereinafter referred to as "human feature quantity"). The human feature quantity obtains the position (xy coordinates) and size of the human body part on the image (xy plane) of the human body part such as the head. Note that the human feature quantity may be any information that can determine where the main subject exists on the image when performing focus control described later.

[0017] The main subject selection unit 113 selects, as the main subject, any one of the human body parts detected by the human feature quantity detection unit 152 based on the xy coordinates and size of the detected human body part and the xy coordinates and object size of the detected object. The focus control unit 153 performs focus control so that the main subject is in focus based on the xy coordinates of the human body part of the main subject selected by the main subject selection unit 113.

[0018] FIG. 3 is a flowchart showing the focus control process for the main subject in the present embodiment. First, in S31, an image is acquired by the imaging unit 150. When the imaging unit 150 captures a moving image, a frame image cut out from the moving image is acquired. Next, in S32, a person is detected from the image acquired in S31 by the human feature quantity detection unit 152, and the human feature quantity of the detected person is obtained.

[0019] In S33, an object selected using the object selection unit 151 is detected from the image acquired in S31, and an object size specifying process for specifying the position (xy coordinates) of the detected object on the image (xy plane) and the object size is performed. The details of the object size specifying process will be described later with reference to FIG. 4.

[0020] In S34, the main subject selection unit 113 determines the main subject based on the human feature amount, the xy coordinates and size of the object. In S35, focus control is performed so that the main subject selected in S34 is in focus. Note that the determination of the main subject based on the human feature amount, the xy coordinates and size of the object in S34 will be described later with reference to FIG. 6.

[0021] Next, with reference to FIG. 4, the object size specifying process in S33 will be described. First, in S41, the detection target selected by the object selection unit 151 is acquired. In the object feature amount detection unit 111, the object feature amount detection unit 111 determines the shape of the detected object. If the shape of the object to be detected is a ball such as a soccer ball, a baseball, a tennis ball, or a basketball, the process proceeds to S43. If the shape of the object to be detected is an ellipsoid such as a rugby ball or an American football, the process proceeds to S44. If the shape of the object to be detected is a disc such as a frisbee, the process proceeds to S45.

[0022] In S43, the object feature amount detection unit 111 detects the object to be detected from the image acquired in S31 and obtains the object feature amount of the detected object. The object feature amount is obtained by a neural network and includes the position of the object on the image (xy plane) and the object feature points for specifying the object size described later. Here, since the shape of the object to be detected is a ball, in S46, the object size specifying unit 112 obtains the diameter of the object in the image (XY plane) from the object feature points and specifies the obtained diameter as the object size.

[0023] Also in S44, the object feature amount detection unit 111, in the same manner as in S43, detects the object to be detected from the image acquired in S31 and obtains the object feature amount of the detected object. When the shape of the object to be detected is an ellipsoid, in S47, the object size specifying unit 112 obtains the length of the minor axis of the object based on the object feature points and specifies the obtained length of the minor axis as the object size. The reason for using the length of the minor axis as the object size and the method for obtaining the length of the minor axis will be described later.

[0024] Also in S45, similar to S43, the object feature quantity detection unit 111 detects the object to be detected from the image acquired in S31 and obtains the object feature quantity of the detected object. When the shape of the object to be detected is a disk, in S48, the object size specifying unit 112 obtains the length of the major axis of the object based on the object feature points and specifies the obtained length of the major axis as the object size. The reason for using the length of the major axis as the object size and the method for obtaining the length of the major axis will be described later.

[0025] When the specification of the object size is completed by any of the processes of S46, S47, and S48, the process returns to the process of FIG. 3.

[0026] Here, the reason for specifying the length of the minor axis of the ball having an ellipsoidal shape as the object size will be described. FIGS. 5(a) to (c) show the shapes of the ellipsoidal ball as viewed from different angles. The ball shown in FIGS. 5(a) to (c) has the x-axis, y-axis, and z-axis fixed to the ball. Here, the z-axis direction corresponds to the axis along which the ball extends long, and the x-axis and y-axis directions correspond to the axes in the direction where the length of the ball is short. FIG. 5(a) shows the case where the z-axis extends in the horizontal direction of the screen, FIG. 5(b) shows the case where the z-axis extends from the back to the front of the screen, and FIG. 5(c) shows the case where the z-axis is slightly inclined in the front direction of the screen.

[0027] As shown by the dotted rectangular regions in FIGS. 5(a) to (c), the rectangular region surrounding the ellipsoidal ball changes depending on the angle. Therefore, when the region where the ball exists is detected as a rectangular region and the size of the ball is estimated from the rectangular region, there is a problem that the estimated ball size is not stable.

[0028] Here, the lengths of the minor axes in FIGS. 5(a) to (c) are indicated by 31, 32, and 33. In FIG. 5(a), the ball size in the direction along the y-axis becomes the length of the minor axis. In FIG. 5(b), the length of the minor axis of the ball size is defined in the vertical direction of the screen, but this is not limited to this case. In FIG. 5(c), since the ball is viewed from the direction where the length of the three-dimensional ellipsoidal ball is long, the shape of the ball is approaching a circular shape.

[0029] The point to note here is that when the ellipsoidal sphere is viewed from various angles in this way, the lengths 31, 32, and 33 of the minor axis of the elliptical region obtained by projecting the ellipsoidal sphere onto the screen are almost the same. If the size in the direction orthogonal to the minor axis is defined as the length of the major axis, the length of the major axis will become longer or shorter depending on the angle of the ball. Therefore, paying attention to the fact that the length of the minor axis of the elliptical shape when the ellipsoidal sphere is projected onto the screen hardly changes regardless of the angle, the ball size (object size) is specified by the length of the minor axis. As a result, the object size can be stably specified regardless of the orientation of the ball.

[0030] Next, a method for measuring the length of the minor axis of the ellipsoidal sphere will be described. First, the object feature quantity detection unit 111 detects the object to be detected selected by the object selection unit 151 from the obtained image using a general object detection method using a neural network. FIGS. 5(a) to (c) show a method of detecting the region where the ball of the ellipsoidal sphere shown by the dotted rectangular region exists as a rectangular region. As a method of detecting the rectangular region shown in FIGS. 5(a) to (c), rectangular regions surrounding various objects (here, balls) to be detected in advance are prepared as correct answer data for learning and learned by the neural network. As the correct answer data for learning, rectangular region information surrounding the entire object (ball) is prepared. At this time, if there is an image in which a part of the object is shielded, an image with a large shielding ratio is not used for learning as invalid data, and learning is performed using an image in which the object is not shielded.

[0031] In addition, in the learning, learning is performed so that three maps, namely, a center map for inferring the center position of the ball and two size maps for inferring the vertical size and the horizontal size of the rectangular region surrounding the entire ball, can be inferred. By performing learning in this way, the rectangular region can be estimated from the center position of the ball, the vertical size of the rectangular region surrounding the entire ball, and the horizontal size.

[0032] Next, as a method for detecting object features for measuring the length of the minor axis of the ellipsoid and measuring the length of the minor axis based on the detected object features, two types of methods will be described.

[0033] The first method is a method of detecting the end points in the minor axis direction as object features, measuring the length between the two end points, and specifying the measured length of the minor axis as the object size. In FIG. 5(d), four points, namely, the end points 34 and 35 in the major axis direction and the end points 36 and 37 in the minor axis direction on the ellipse, are indicated by black dots. Among these, in the object feature detection unit 111, the end points in the minor axis direction are detected by learning a center map so as to estimate the center positions of the end points 36 and 37 in the minor axis direction. Then, the object size specifying unit 112 measures the length between the two end points and adopts the measured length of the minor axis as the object size.

[0034] The second method is the same as the detection of the rectangular region described with reference to FIGS. 5(a) to 5(c), but instead of learning the vertical size and the horizontal size of the rectangular region surrounding the entire ball, the minor axis size and the major axis size of the ball are learned. In the case of FIGS. 5(a) to 5(c), rectangular region information surrounding the entire ball was prepared, but four points, namely, the end points 34 and 35 in the major axis direction and the end points 36 and 37 in the minor axis direction shown in FIG. 5(d), are prepared as correct answer data for learning, and the center position, the length of the major axis of the ball, and the length of the minor axis of the ball are learned from these four points. Thereby, the length of the major axis and the length of the minor axis of the ball can be inferred as object features. Here, the angle of one side (long side or short side) of the rectangular region with respect to one side (horizontal axis or vertical axis) of the image is not learned, but by also learning and inferring the angle of the rectangular region, it is possible to estimate a rectangular region as shown by the dotted line in FIG. 5(e). Then, the object size specifying unit 112 adopts the estimated length of the minor axis as the object size based on the shape of the subject to be detected among the length of the major axis and the length of the minor axis.

[0035] In addition, as a method for specifying the length of the minor axis of an object that is an ellipsoid, the above two methods have been described, but the present invention is not limited thereto. For example, in the object feature amount detection unit 111, an elliptical region may be fitted to the ball, and the minor axis may be estimated from the parameters of the estimated elliptical region.

[0036] Also, when detecting a disc-shaped instrument such as a frisbee, it is detected by the same method as the ball of the ellipsoid using a neural network.

[0037] When the shape of the object to be detected is a disc, the circular shape is the largest when looking down at the disc surface from above, and when viewed obliquely, the length of the major axis does not change and it becomes a flattened elliptical shape. At this time, while the length of the major axis of the subject image always remains constant, the length of the minor axis changes. Therefore, for a disc-shaped object such as a frisbee, stable size estimation can be realized by using the length of the major axis.

[0038] Next, the effect on the main subject determination using the object size detected by the above-described method and the autofocus control by stably detecting the object size will be described. In the autofocus control of the camera, one main subject is selected from among a plurality of people and the focus is made to follow. In particular, in a sports competition using a ball, since a player close to the ball is likely to be the main subject, a method of selecting the main subject based on the detection information of the ball has been disclosed.

[0039] At this time, in a scene where a plurality of people stand on the near side and the far side in the optical axis (z-axis) direction of the camera and a ball exists at an intermediate point between them, it is difficult to determine which person will be the main subject only from the information on the positional relationship between the ball and the people on the image (xy plane). In such a situation, it is effective to consider the ball size on the image (xy plane) and the head size of the people on the image (xy plane) and select the person closer to the ball as the main subject in the optical axis (z-axis) direction of the camera. Therefore, it is desirable that not only the position of the ball but also the accuracy of determining the ball size is high.

[0040] An example of a scene where the selection performance of the main subject is improved by improving the determination accuracy of the ball size is shown in FIG. 6. FIG. 6 shows a case where player 61 exists on the back side of the screen and player 61 exists on the front side of the screen in the direction of the optical axis (z-axis) of the camera. FIGS. 6(a) and (c) show the case of viewing the players and the ball from the front where the camera is located, and FIGS. 6(b) and (d) show the case of viewing the players and the ball from above. Also, FIGS. 6(a) and (b) show the case where ball 63 is close to player 61, and FIGS. 6(c) and (d) show the case where ball 64 is close to player 62.

[0041] From the positional relationships of player 61, player 62, ball 63, and ball 64 in the images shown in FIGS. 6(a) and (b), it is not clear whether ball 63 and ball 64 are closer to player 61 or player 62. However, as shown in FIGS. 6(b) and (d), although the actual sizes of balls 63 and 64 are the same, in FIGS. 6(a) and 6(c), the distances in the direction of the optical axis (z-axis) of the camera are different, so the ball sizes appear different. In FIG. 6(a), since ball 63 exists at a position far from the camera, the size of ball 63 in the image appears small. On the other hand, in FIG. 6(c), since ball 64 exists at a position close to the camera, the size of ball 64 in the image appears large. Also, not only the ball, but the head of player 61 exists at a position far from the camera, so the head size of player 61 in the image appears small. On the other hand, the head of player 62 exists at a position close to the camera, so the head size of player 62 in the image appears large.

[0042] Thus, since the size in the image changes according to the distance in the direction of the optical axis (z-axis) of the camera where the ball or the head of the person exists, the distance where the subject exists can be estimated from the size. Here, since it is desired to select the player close to the ball as the main subject, by determining whether the position of the ball is closer to player 61 or 62 in view of the ball size and the head size, the selection of the main subject becomes possible.

[0043] At this time, when the shapes of the balls 63 and 64 are ellipsoidal balls, even if they are at the same distance, the size of the ball changes depending on the angle of the ball, making it difficult to determine which player is closer. Therefore, as described above, for an ellipsoidal ball, by specifying the length of the minor axis as the object size, it is possible to stably determine the ball size, and as a result, it is possible to stably determine the main subject.

[0044] In particular, as shown in FIG. 6, if the selection of the main subject is misjudged when there are people at different positions in the depth direction, the focus will shift greatly back and forth. Therefore, by stably determining the main subject, the quality of autofocus can be improved.

[0045] As described above, according to the first embodiment of the present invention, it is possible to stably specify the object size without being affected by the change in the shape of the object detected due to the shooting angle.

[0046] In the first embodiment, the object size is specified for the purpose of selecting the main subject from one or more detected persons and performing focus control so that the main subject is in focus. However, the object of the present invention is not limited to focus control. When it is necessary to measure the object size for other purposes, the object detection unit 11 may be used to improve the accuracy of specifying the object size.

[0047] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, a method for stably specifying the object size when the object to be detected is frequently shielded during the competition will be described.

[0048] FIG. 7 is a block diagram showing the functional configuration of the imaging device 20 as a configuration including the object detection unit 12 in the second embodiment. The imaging device 20 in the second embodiment includes an imaging unit 150, a competition selection unit 125, an object detection unit 12, a person feature amount detection unit 152, a main subject selection unit 123, and a focus control unit 153. The object detection unit 12 includes an object feature amount detection unit 121, an object size specification unit 122, and an object occlusion determination unit 124. Note that the same components as those in FIG. 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0049] The user uses the competition selection unit 125 to select a competition to be photographed by the imaging unit 150. The competition selection unit 125 may have the same configuration as the object selection unit 151. For example, the competition selection unit 125 provides a user interface such as FIGS. 2(a) and 2(b), and determines a competition according to the selection content by the user. The competition selection unit 125 outputs information on the selected competition to the object detection unit 12.

[0050] Based on the competition selected by the competition selection unit 125, the object feature amount detection unit 121 detects an object (for example, a ball) to be detected used in the selected competition from a still image or a moving image obtained by the imaging unit 150, and detects the object feature amount thereof. Then, based on the competition selected by the competition selection unit 125, the object size specification unit 122 measures the detected object on the image and specifies the object size.

[0051] Next, with reference to FIG. 8, the object size specification process in the focus control process according to the second embodiment will be described. Note that the focus control process is the same as that described with reference to FIG. 3 in the first embodiment, and thus the description thereof is omitted here. In the second embodiment, since the object size specification process performed in S33 is different from the process shown in FIG. 4, the object size specification process will be described below.

[0052] First, in S81, the game selected by the game selection unit 125 is acquired. In S82, it is determined whether the acquired game is a game in which the ratio of the object to be detected being shielded during the game is high. Note that whether the game is a game in which the ratio of the object to be detected being shielded during the game is high may be stored in advance as information corresponding to the game. If the game is a game in which the ratio of the object to be detected being shielded during the game is low, the process proceeds to S84, the object size is specified by the method described with reference to FIG. 4, and the process returns to the process of FIG. 3.

[0053] On the other hand, if the game is a game in which the ratio of the object to be detected being shielded during the game is high, the process proceeds to S83, and the shape of the object used in the selected game is determined. If the shape of the object to be detected is a sphere, the process proceeds to S85, and the sphere ball size specifying process shown in FIG. 10 is performed. If the shape of the object to be detected is an ellipsoid, the process proceeds to S86, and the ellipsoid ball size specifying process shown in FIG. 12 is performed. Further, if the shape of the object to be detected is a disk, the process proceeds to S87, and the disk size specifying process is performed. When the specification of the object size is completed by any of the processes of S85, S86, and S87, the process returns to the process of FIG. 3.

[0054] Next, the sphere ball size specifying process performed in S85 will be described with reference to FIGS. 9 and 10. FIG. 9 shows a state in which player 91 holds ball 92 of the ball in his hand and a part of it is shielded. FIGS. 9(a) and (b) show different methods of preparing learning data and detection methods. As shown in FIG. 9(a), by preparing correct answer data for learning (frame 93) so as to surround the entire ball, the size of the ball can be estimated. On the other hand, since data with a large shielding ratio cannot be used for learning, there is a problem that the detection rate of balls with a high shielding ratio decreases. Therefore, in the present embodiment, as shown in FIG. 9(b), learning is performed by assigning correct answer data for learning (frame 94) only to the unshielded area of the entire ball. By doing so, it becomes possible to detect the ball as long as it is visible at all, and the detection rate of the ball can be improved.

[0055] Figure 10 is a flowchart of the ball size identification process. First, in S101, the object feature quantity detector 121 detects the object feature quantity. The object feature quantity detected here is a rectangular area indicating the area where the ball exists. In the detection of the rectangular area in this embodiment, as described above, when the ball is shielded, data with a rectangular area assigned to the unshielded area of the ball is used as the correct learning data.

[0056] In S103, a shielding determination of the ball (object) is performed. The shielding determination of the ball is performed by the object shielding determination unit 124. In the case of a spherical ball, since the shape of the ball does not change depending on the viewing angle of the ball, it is possible to determine whether a part of the ball is shielded by determining whether the aspect ratio of the rectangular area detected in S101 is close to 1 or far from 1.

[0057] If it is determined in S103 that the ball is shielded, the process proceeds to S104, and the object size identification unit 122 identifies the ball size (object size) by the first identification method. In this embodiment, as the first identification method, the long side of the detected rectangular area of the ball is identified as the ball size (object size). As shown in Fig. 9(b), when a part of the ball is shielded, since the long side of the detected rectangular area is likely to correspond to the diameter of the spherical ball, the length of the long side is adopted as the ball size.

[0058] On the other hand, if it is determined in S103 that the ball is not shielded, the process proceeds to S105, and the object size identification unit 122 identifies the ball size (object size) by the second identification method. In this embodiment, as the second identification method, the average value of the long side and the short side is obtained, or one of the values of the long side or the short side is adopted. This is because when it is determined that the ball is not shielded, that is, when the aspect ratio of the detected rectangular area is approximately 1, the lengths of the long side and the short side are approximately the same. When the ball size determination in S104 or S105 is completed, the process returns to the process of Fig. 8.

[0059] Next, the ellipsoidal ball size identification process performed in S86 will be described with reference to FIGS. 11 and 12. FIG. 11 shows a state where the ellipsoidal ball 95 is held by the player 91 and partially shielded.

[0060] FIG. 12 is a flowchart of the ellipsoidal ball size identification process. First, in S121, the object feature quantity detection unit 121 detects the first object feature quantity. The first object feature quantity refers to a total of four points, namely two points each at the endpoints of the major axis and the minor axis described in the first embodiment.

[0061] Next, in S122, the object feature quantity detection unit 121 calculates a second object feature quantity different from the first object feature quantity. The second object feature quantity is a rectangular area indicating the area where the ball exists. In the detection of the rectangular area in this embodiment, as the correct learning data, data with a rectangular area assigned to the unshielded area of the ball is used. Different from the rectangular area described in the first embodiment, by learning only the unshielded part, the data available for learning can be increased, and the detection rate for the shielded ball can also be improved.

[0062] Next, in S124, a shielding determination of the ball (object) is performed. The shielding determination of the ball is performed by the object shielding determination unit 124. The shielding determination of the ball is performed based on the endpoints of the major axis and the endpoints of the minor axis in the first object feature quantity detected in S121. For example, if even one of the total four points of the endpoints of the major axis and the endpoints of the minor axis cannot be detected, it may be determined that the ball is shielded. Also, as the first object feature quantity, if only two endpoints of the minor axis are detected, and if even one of the two endpoints of the minor axis cannot be detected, it may be determined that the ball is shielded.

[0063] Also, not limited to this method, the shielding state of the ball may be determined using the human feature quantity obtained by the human feature quantity detection unit 152. For example, if the human feature quantity detection unit 152 detects the hands and feet of a person, and the hands and feet of the person are detected near the ball, it may be determined that the ball is in a shielded state.

[0064] When it is determined in S124 that the ball is shielded, the process proceeds to S125, and the object size specifying unit 122 specifies the ball size (object size) by a third specifying method using a rectangular area indicating the area where the ellipsoidal ball, which is the second object feature amount, exists. In the third specifying method, the length of the minor axis of the ball is not measured, and the ball size is specified based on the lengths of the sides of the rectangular area. For example, the average value of the vertical and horizontal lengths may be adopted as the ball size, or a method of comparing the ball size in the previous frame with the vertical and horizontal lengths of the rectangular area and selecting the closer length may be considered, but it is not limited to these.

[0065] On the other hand, when it is determined in S124 that the ball is not shielded, the process proceeds to S125, and the object size specifying unit 122 specifies the ball size (object size) by a fourth specifying method. In the present embodiment, as the fourth specifying method, the length of the minor axis is adopted as the ball size from the endpoints of the minor axis, which is the first object feature amount. When the determination of the ellipsoidal ball size is completed in S125 or S126, the process returns to the process of FIG. 8.

[0066] Note that the disk size specifying process in S87 can be performed as described with reference to FIG. 12. However, in the process in S126, among the first object feature amounts, the length of the major axis is adopted as the disk size (object size) from the endpoints of the major axis.

[0067] Here, the effect of switching the object size specifying method according to the shielding state will be described. When it is determined in S103 and S124 that the ball is shielded, there is a high possibility that a person exists near the ball. It is more likely to correctly perform the main subject determination by detecting the presence of the ball than to correctly estimate the size of the ball. Therefore, by learning only the unshielded part of the ball, the ball detection rate is increased by using the result of learning so that the ball can be detected if it can be seen even a little.

[0068] On the other hand, if it is determined in S103 and S124 that the ball is not blocked, there is a high possibility that the ball exists between the persons, and as described with reference to FIG. 6 of the first embodiment, the estimation accuracy of the ball size is likely to have a great influence on the main subject determination. Therefore, by using the method for specifying the object size described in the first embodiment, the detection accuracy of the object size is increased.

[0069] As described above, according to the second embodiment, by switching the method for estimating the object size according to the shielding state of the object, it is possible to stably specify the object size even when the ratio of the object being shielded during the competition is high.

[0070] <Other Embodiments> Note that the present invention may be applied to a system composed of a plurality of devices or to an apparatus composed of a single device.

[0071] Further, the present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0072] <Summary> The disclosure of the present embodiment includes the following configurations.

[0073] (Item 1) selection means for selecting an object to be detected; acquisition means for acquiring an image; detection means for detecting the object selected by the selection means from the image; specifying means for specifying the size of the object in the image detected by the detection means, wherein the specifying means changes a part to be specified as the size of the object in the image according to the shape of the object. An image processing apparatus characterized by this. (Item 2) The object includes a first object having an ellipsoidal shape, When the first object is detected by the detection means, the specifying means specifies, as the size of the first object, the length of the minor axis of the first object in the image as the size of the part, according to item 1 of the image processing apparatus characterized by the above. (Item 3) The object includes a second object having a disk shape, When the second object is detected by the detection means, the specifying means specifies, as the size of the second object, the length of the major axis of the second object in the image as the size of the part, according to item 1 or 2 of the image processing apparatus characterized by the above. (Item 4) The object includes a third object having a spherical shape, When the third object is detected by the detection means, the specifying means specifies, as the size of the third object, the length of the diameter of the third object in the image as the size of the part, according to any one of items 1 to 3 of the image processing apparatus characterized by the above. (Item 5) The selection means selects a competition and selects, as the detection target, an object pre-associated with the selected competition, When the competition selected by the selection means is a predetermined competition in which the ratio of the object being shielded by other objects is high, the specifying means determines whether the object detected by the detection means is shielded, and changes the method of specifying the size of the object on the image depending on whether it is determined that the object is shielded or not shielded, according to item 1 of the image processing apparatus characterized by the above. (Item 6) The object has an ellipsoidal shape and includes a first object associated with a predetermined competition in which the ratio of being shielded by other objects is high, When the first object is detected by the detection means, the specifying means When it is determined that the first object is shielded, as the part, based on the length of the side of the rectangular area representing the unshielded area of the first object in the image, the size of the first object is specified. When it is determined that the first object is not shielded, as the part, the length of the minor axis of the first object in the image is specified as the size of the first object. The image processing apparatus according to item 5, characterized in that. (Item 7) The object has a disk shape and includes a second object associated with a predetermined competition with a high shielding ratio by the other object. When the second object is detected by the detection means, the specifying means When it is determined that the second object is shielded, as the part, based on the length of the side of the rectangular area representing the unshielded area of the second object in the image, the size of the second object is specified. When it is determined that the second object is not shielded, as the part, the length of the major axis of the second object in the image is specified as the size of the second object. The image processing apparatus according to item 5 or 6, characterized in that. (Item 8) The object has a spherical shape and includes a third object associated with a predetermined competition with a high shielding ratio by the other object. When the third object is detected by the detection means, the specifying means When it is determined that the third object is shielded, as the part, the length of the long side of the rectangular area representing the unshielded area of the third object in the image is specified as the size of the third object. When it is determined that the third object is not shielded, as the part, based on the lengths of the short side and the long side of the rectangular area representing the unshielded area of the third object in the image, the size of the third object is specified. The image processing apparatus according to any one of items 5 to 7, characterized in that. (Item 9) The detection means further acquires the position of the detected object in the image, The image processing apparatus, subject detection means for detecting a predetermined subject included in the image and obtaining the position and size of the detected subject in the image; determination means for determining a main subject among the plurality of subjects based on the positions and sizes of the respective subjects in the image and the position and size of the object in the image when a plurality of the subjects are detected by the subject detection means; The image processing apparatus according to any one of Items 1 to 8, further comprising: (Item 10) The determination means estimates the distance in the depth direction in which the plurality of subjects and the object exist based on the size of each subject in the image and the size of the object in the image, and determines the main subject based on the estimated distance, the positions of the respective subjects in the image, and the position of the object in the image. The image processing apparatus according to Item 9. (Item 11) An image processing apparatus according to any one of Items 1 to 10, and imaging means for capturing an image, The acquisition means acquires an image captured by the imaging means. The imaging apparatus characterized by the above. (Item 12) The image processing apparatus according to Item 9 or 10, and focus control means for performing focus control so as to focus on the main subject determined by the determination means. The imaging apparatus characterized by the above. (Item 13) a selection step of selecting an object to be detected; an acquisition step of acquiring an image; a detection step of detecting the object selected in the selection step from the image; and specifying means for specifying the size of the object in the image detected in the detection step. In the specific process, the method for specifying the size of an object is characterized in that a part specified as the size of the object in the image is changed according to the shape of the object. (Item 14) A program for causing a computer to function as each means of the image processing apparatus according to any one of Items 1 to 10. (Item 15) A computer-readable storage medium storing the program according to Item 14. The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Signs

[0074] 10, 20: Imaging device, 11, 12: Object detection unit, 111, 121: Object feature amount detection unit, 112, 122: Object size specification unit, 113, 123: Main subject selection unit, 124: Object occlusion determination unit, 125: Competition selection unit, 150: Imaging unit, 151: Object selection unit, 152: Person feature amount detection unit, 153: Focus control unit

Claims

1. selection means for selecting an object to be detected; acquisition means for acquiring an image; detection means for detecting the object selected by the selection means from the image; specification means for specifying the size of the object in the image detected by the detection means; and the specification means changes a part to be specified as the size of the object in the image according to the shape of the object, an image processing apparatus characterized by this.

2. The object includes a first object having an ellipsoidal shape, when the first object is detected by the detection means, the specification means specifies, as the size of the first object in the image, the length of the minor axis of the first object in the image as the size of the first object, the image processing apparatus according to claim 1, characterized by this.

3. The object includes a second object having a disk shape, when the second object is detected by the detection means, the specification means specifies, as the size of the second object in the image, the length of the major axis of the second object in the image as the size of the second object, the image processing apparatus according to claim 1, characterized by this.

4. The object includes a third object having a spherical shape, when the third object is detected by the detection means, the specification means specifies, as the size of the third object in the image, the length of the diameter of the third object in the image as the size of the third object, the image processing apparatus according to claim 1, characterized by this.

5. The selection means selects a competition and selects, as the object to be detected, an object previously associated with the selected competition, when the competition selected by the selection means is a predetermined competition in which the ratio of the object being shielded by other objects is high, the specification means determines whether the object detected by the detection means is shielded, and changes the method of specifying the size of the object on the image depending on whether it is determined that the object is shielded or not shielded, the image processing apparatus according to claim 1, characterized by this.

6. The object has an ellipsoidal shape and includes a first object associated with a predetermined competition in which the ratio of being shielded by other objects is high, when the first object is detected by the detection means, the specification means When it is determined that the first object is occluded, as the part, based on the length of the side of the rectangular area representing the unoccluded area of the first object in the image, the size of the first object is specified. When it is determined that the first object is not occluded, as the part, the length of the minor axis of the first object in the image is specified as the size of the first object. The image processing apparatus according to claim 5, characterized in that.

7. The object has a disk shape and includes a second object associated with a predetermined competition in which the ratio of being occluded by the other object is high. When the second object is detected by the detection means, the specifying means. When it is determined that the second object is occluded, as the part, based on the length of the side of the rectangular area representing the unoccluded area of the second object in the image, the size of the second object is specified. When it is determined that the second object is not occluded, as the part, the length of the major axis of the second object in the image is specified as the size of the second object. The image processing apparatus according to claim 5, characterized in that.

8. The object has a spherical shape and includes a third object associated with a predetermined competition in which the ratio of being occluded by the other object is high. When the third object is detected by the detection means, the specifying means. When it is determined that the third object is occluded, as the part, the length of the long side of the rectangular area representing the unoccluded area of the third object in the image is specified as the size of the third object. When it is determined that the third object is not occluded, as the part, based on the lengths of the short side and the long side of the rectangular area representing the unoccluded area of the third object in the image, the size of the third object is specified. The image processing apparatus according to claim 5, characterized in that.

9. The detection means further acquires the position of the detected object in the image. The image processing apparatus. Subject detection means for detecting a predetermined subject included in the image and obtaining the position and size of the detected subject in the image. Determining means for determining a main subject among the plurality of subjects based on the positions and sizes of the respective subjects in the image and the position and size of the object in the image when a plurality of the subjects are detected by the subject detection means The image processing apparatus according to claim 1, further comprising the same.

10. The determining means estimates the distances in the depth direction in which the plurality of subjects and the object exist based on the sizes of the respective subjects in the image and the size of the object in the image, and determines the main subject based on the estimated distances, the positions of the respective subjects in the image, and the position of the object in the image. The image processing apparatus according to claim 9.

11. An image processing apparatus according to any one of claims 1 to 10, Imaging means for capturing an image, The acquisition means acquires an image captured by the imaging means. The imaging apparatus is characterized by this.

12. An image processing apparatus according to claim 9 or 10, Focus control means for performing focus control so as to focus on the main subject determined by the determining means The imaging apparatus is characterized by having the same.

13. A selection step of selecting an object to be detected, An acquisition step of acquiring an image, A detection step of detecting the object selected in the selection step from the image, A specifying step of specifying the size of the object in the image detected in the detection step, In the specifying step, the part specified as the size of the object in the image is changed according to the shape of the object. The object size specifying method is characterized by this.

14. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10.

15. A computer-readable storage medium storing the program according to claim 14.

Images