Information processing apparatus, information processing method, and program

Abstract

To avoid the occurrence of erroneous association in which another object or the like is associated with an object to be tracked.SOLUTION: The information processing apparatus detects a tracking target object from an image, estimates local portions, and selects a local portion having a high degree of association with the tracking target object from among one or more estimated local portions. Then, in accordance with the tracking target object and a local portion selected from the one or more local portions, the information processing apparatus determines whether to associate the tracking target object with the selected local portion.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for detecting and tracking an object from an image. [Background technology]

[0002] Specific object regions are detected and tracked from successive images in a time series. Tracking involves detecting specific object regions from an image and tracking the same object region between successive images in a time series. In imaging devices (cameras), autofocus processing and other processes are performed based on the results of this tracking.

[0003] Patent Document 1 discloses a method for tracking an object to be tracked while associating the entire object with a local portion of the object. For example, if the object to be tracked is a person, the entire object to be tracked is assumed to be the entire human body, and the local portion is assumed to be the face or the like. In Patent Document 1, association is performed based on the positional relationship between the entire object and the local portion in an image, and the amount of positional change between the entire object and the local portion in previous and subsequent images in a time series. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-212581 Summary of the Invention [Problem to be solved by the invention]

[0005] In the association based on the positional relationship between the entire object and local parts disclosed in Patent Document 1, for example, when a local part of an object is not detected, an error may occur, such as associating a local part of an object other than the object being tracked with the entire object being tracked. For example, when an image capture device autofocuses on an associated local part, an incorrect association of the local part may result in the focus being on the head of a different person. In particular, when capturing a sports scene with multiple people crowded together, if the focus is on a person other than the person being tracked, the quality of the captured image may be significantly impaired.

[0006] Therefore, an object of the present invention is to make it possible to avoid the occurrence of erroneous association in which a different object is associated with a tracking target object. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a detection means for detecting a tracking target object from an image, an estimation means for estimating a local part from the image, a selection means for selecting a local part that has a high degree of association with the tracking target object from one or more local parts estimated by the estimation means, and a decision means for deciding whether or not to associate the tracking target object with the selected local part, depending on the tracking target object detected by the detection means and the local part selected from the one or more local parts by the selection means. [Effects of the Invention]

[0008] According to the present invention, it is possible to avoid the occurrence of erroneous association in which a different object is associated with a tracking target object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic configuration diagram of a computer that can realize an information processing device. [Figure 2] FIG. 2 is a functional configuration diagram of the information processing device. [Figure 3]10 is a flowchart showing the flow of information processing. [Figure 4] FIG. 10 is an explanatory diagram of an input image without occlusion and the tracking and local area estimation results. [Figure 5] FIG. 10 is a diagram illustrating an input image in which occlusion occurs and the tracking and local area estimation results. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, in the following embodiments, the same or similar configurations and processing steps are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] First Embodiment The information processing device of this embodiment receives input of successive images in a time series, detects a specific tracking target object from the successive images, detects local regions associated with the tracking target object, and tracks the tracking target object by associating the tracking target object with the local regions. In this embodiment, an example is given in which the tracking target object is a person and the person's face is the local region, but this is not limited to this. For example, the tracking target object may be the person's face and the pupils in the face may be the local region. Furthermore, the tracking target object is not limited to a person but may also be an animal. In this case, the entire object may be the entire body of the animal and the local region may be the animal's head (face). Furthermore, the local region may be any part of the object as long as it is associated with the tracking target object. For example, the tracking target object may be a vehicle carrying a person, and in this case, the local region may be the person or the person's head. Although the person or the person's head is not part of the vehicle, it moves along with the vehicle, which is the object to be tracked. Therefore, if the vehicle is the object to be tracked, the person riding in the vehicle or the person's head can be considered a local part.

[0012] FIG. 1 is a diagram showing a schematic basic configuration of a computer that can realize an information processing device according to this embodiment. The processor 101 is, for example, a CPU, and controls the overall operation of the computer. The storage device 103 includes a computer-readable storage medium, such as an HDD, SSD, or CD-ROM, and stores various programs, data, and the like for a long period of time. The storage device 103 stores an information processing program that implements the processing of each functional unit of the information processing device shown in FIG. 2 and the flowchart of the information processing device shown in FIG. 3, and the information processing program is read into the memory 102. The memory 102 is, for example, a RAM, and temporarily stores various programs and data, including the information processing program according to this embodiment. The processor 101 then executes the information processing program stored in the memory 102 to implement the processing of each functional unit of the information processing device according to this embodiment shown in FIG. 2 and the flowchart of FIG. 3.

[0013] The input IF 104 is an interface for acquiring information from an external device. The output IF 105 is an interface for outputting information to an external device. A bus 106 connects the above-mentioned components and enables the exchange of various data such as images.

[0014] FIG. 2 is a functional block diagram showing each functional unit realized by the information processing device 200 of this embodiment. The images 201 are successive images in time series that are input to the information processing device 200. When the information processing device 200 of this embodiment is mounted on, for example, an imaging device (camera), the images are assumed to be images that constitute each frame of a moving image captured by the imaging device. The successive images in time series may be images captured by the imaging device, or images of a moving image that have been captured in advance and stored in the storage device 103. A tracking unit 202 detects and tracks a tracking target object from a series of images 201 in time series.

[0015] The first estimation unit 203 and the second estimation unit 204 each estimate a local part area that is a candidate for association with the tracking target object from the image 201. In this embodiment, the first estimation unit 203 performs a first estimation process that estimates a local part area with high reliability that is a candidate for association with the tracking target object. On the other hand, the second estimation unit 204 performs a second estimation process that estimates a local part area that is a candidate for association with the tracking target object but with lower reliability than the local part area estimated by the first estimation process. The differences between the first estimation unit 203 and the second estimation unit 204 and specific implementation methods will be described in detail later.

[0016] The associating unit 205 selects one local part area that is most associated with the tracking result of the tracking target object by the tracking unit 202 from among the high-reliability local part areas estimated by the first estimation unit 203 and the low-reliability local part areas estimated by the second estimation unit 204. That is, the associating unit 205 selects a local part that has a high degree of association with the tracking target object from among one or more local parts estimated by the first and second estimation units 203 and 204. Furthermore, the associating unit 205 determines whether or not to associate the selected local part with the tracking target object, that is, whether or not to perform association, depending on the selected local part and the tracking target object. A specific implementation method of the associating unit 205 will be described in detail later. The associating unit 205 then outputs the determination result of whether or not to perform association to the display unit 206.

[0017] The display unit 206 generates display data of images and information and sends the display data to a display device (not shown) connected to the output IF 105. For example, the display unit 206 generates, in addition to images, a graphical user interface (GUI) that allows a user to input various instructions and the like from an operation device (not shown) while viewing the display on the display device, and display data showing the results of information processing by the information processing device 200. In this embodiment, the display data of the GUI includes, for example, display data used when a user sets a tracking target object to be tracked by the tracking unit 202. Furthermore, the display data showing the results of information processing includes display data showing local areas estimated by the first estimation unit 203 and the second estimation unit 204, display data showing the results of association by the associating unit 205, and the like. These display data are then sent to the display device, whereby the display device displays an image according to the display data. This allows the user to input various instructions via the GUI and check images and the results of information processing by the information processing device 200.

[0018] 3 is a flowchart showing the flow of information processing by the information processing device 200 according to the first embodiment. In the following description of the flowchart, the symbol S indicates a processing process (processing step). First, in the process of S301, the tracking unit 202 registers a template of a subject that is to be the tracking target object. The template is registered, for example, by a method in which, when a user selects a subject that is to be the tracking target object from an input image, the selected subject is registered as a template. In this embodiment, an example is given in which the entire body of a person is tracked as the tracking target object, and therefore a full-body image of the person that is the tracking target object is registered as a template. Note that, in this embodiment, an example is given in which the tracking process is performed by template matching using a template registered for tracking, but this is not limited to this, and tracking process using a neural network, for example, may also be performed. Since the tracking process using template matching or a neural network is known process, detailed description thereof will be omitted.

[0019] Next, in the process of S302, the tracking unit 202 performs a tracking process on the tracking target object, i.e., a tracking process on the entire body of a person in this embodiment. For example, the tracking unit 202 acquires a current frame image 201 from a video continuously input in time series, and searches for an area in the current frame image 201 that is similar to the template. If the tracking unit 202 finds multiple areas similar to the template, it sets each of those areas as a tracking candidate and acquires a tracking score for each of those tracking candidates. The tracking score is a numerical value that represents the reliability that the tracking candidate is a tracking target object, i.e., a numerical value that represents the degree of likelihood that the tracking candidate is a tracking target object. A higher numerical value indicates a higher likelihood (higher reliability) that the tracking candidate is a tracking target object. For example, the tracking unit 202 calculates the tracking score based on the degree of match with the tracking target object tracked in the previous image of the previous frame, the image similarity between the tracking target object tracked in the previous image of the previous frame and the template, and the like. The tracking unit 202 then determines the tracking candidate with the highest tracking score among the multiple tracking candidates as the tracking result. In this embodiment, the tracking result is information expressed, for example, by the position and size of a rectangular frame called a bounding box that surrounds the subject of the tracking result on the image, and is passed to the associating unit 205.

[0020] Next, in the process of S303, the first estimation unit 203 performs a first estimation process to estimate, from the input image, a highly reliable local part region that is a candidate for association with the tracking target object. In this embodiment, a highly reliable local part region means that the estimated local part region is a region that can be sufficiently trusted as a local part detection result. In other words, the first estimation process to estimate a highly reliable local part region is a process intended to prevent the inclusion of an estimation result in which a region of another object similar to the local part is erroneously estimated as a local part region.

[0021] The first estimation unit 203 estimates local part regions associated with the tracking target object from the input image and further obtains an estimation score for each estimated local part region. Here, it is assumed that a general, known object estimation method is used to estimate local parts from an image. A general object estimation method, such as an object estimation method using a neural network, is widely known, and a detailed description thereof will be omitted. Of course, the local part estimation method used by the first estimation unit 203 is not limited to an object estimation method using a neural network. Note that the number of local part regions estimated from the input image is not limited to one, and there may be multiple local part regions. The estimation score is a numerical value representing the reliability that the estimated local part region is a local part region associated with the tracking target object, that is, a numerical value representing the degree of likelihood that the estimated local part region is a local part region associated with the tracking target object. The higher the estimation score, the higher the likelihood (reliability) that the estimated local part region is a local part region associated with the tracking target object. The estimation score is a value obtained by an object estimation method using a neural network, for example.

[0022] Next, the first estimation unit 203 compares the estimation score of the estimated local part region with a predetermined first estimation threshold, and determines, based on the comparison result, whether the local part region is a highly reliable local part region associated with the tracking target object. In this embodiment, the first estimation threshold is set to a sufficiently high value that can acquire only highly reliable local part regions that are highly likely to be local parts associated with the tracking target object, and can exclude regions of other objects similar to the local part. The first estimation unit 203 determines only local part regions with an estimation score equal to or greater than the first estimation threshold as highly reliable local part regions, and excludes local part regions with an estimation score less than the first estimation threshold as not being highly reliable.

[0023] Then, the first estimation unit 203 passes the estimation result, which is expressed by the position and size on the image of a rectangular frame (bounding box) surrounding the region of the estimated local part with high reliability, to the associating unit 205. At this time, the first estimation unit 203 also assigns a flag or the like to the estimation result indicating that it was estimated by the first estimation unit 203. This allows the associating unit 205 to identify that the estimation result is derived from the first estimation unit 203.

[0024] For example, if the tracking target object is a person and the local part associated with the person is a person's head, the first estimation unit 203 estimates the area of ​​the person's head associated with the person, which is the tracking target object, from the input image and obtains an estimation score for the estimated area of ​​the person's head. For example, if the estimation score of the area estimated as a person's head is low, the object in that area may be an object similar to a person's head, such as a ball or a tire. In other words, the estimation result of the person's head may include an erroneous estimation result of an object similar to a person's head, such as a ball or a tire. For this reason, the first estimation unit 203 obtains only the person's head with an estimation score equal to or greater than the first estimation threshold, thereby obtaining only the area of ​​the person's head with high reliability, excluding other objects similar to a person's head, such as a ball. Then, the first estimation unit 203 assigns a flag to the estimation result, which is expressed as the position and size on the image of a rectangular frame (bounding box) surrounding the highly reliable estimated area of ​​the human head, indicating that it was estimated by the first estimation unit 203, and passes it to the association unit 205.

[0025] Next, in the process of S304, the second estimation unit 204 performs a second estimation process to estimate, from the input image, local part regions that are candidates for association with the tracking target object but have lower reliability than the local part regions estimated in the first estimation process. In this embodiment, a low-reliability local part region means that the estimated local part region is not as reliable as the high-reliability local part region described above, but is still a possible local part region. The second estimation unit 204 obtains an estimation score similar to that described above for each local part region estimated as being a possible local part, and compares the estimation score with a predetermined estimation threshold. However, the estimation threshold used by the second estimation unit 204 is a different value from the first estimation threshold used by the first estimation unit 203 and is set to a value lower than the first estimation threshold. The second estimation unit 204 compares the estimation score of the estimated local part region with a predetermined second estimation threshold and determines whether the local part region is a low-reliability local part region based on the comparison result. That is, by using a second estimation threshold lower than the first estimation threshold, the second estimation unit 204 acquires low-reliability local part regions that would be excluded by the first estimation unit 203 as estimation results of regions that may be local part regions. In other words, the second estimation process is a process aimed at estimating regions that may be local parts that would be excluded by the first estimation process as not being high-reliability local part regions. Note that the second estimation process also uses a general, known object estimation method, as described above.

[0026] Here, since the second estimation unit 204 uses a second estimation threshold lower than the first estimation threshold, it is expected that the second estimation unit 204 will acquire more local region regions as estimation results, including the local region regions estimated by the first estimation unit 203. Therefore, the second estimation unit 204 eliminates regions of the estimated local region regions that overlap with the highly reliable local region regions estimated by the first estimation unit 203, thereby preventing duplicate output of estimation results that overlap with the local region regions estimated by the first estimation unit 203. For example, the second estimation unit 204 calculates the intersection over union (IoU) between the estimation result of the first estimation unit 203 and determines whether or not there is an overlapping region based on the calculated value. The IoU is, for example, the value obtained by dividing the area of ​​the intersection of two regions by the area of ​​the union of the two regions, in other words, a value indicating the overlap ratio. Since the IoU represents the overlap ratio, the closer the IoU value is to 1, the more the two regions overlap. If the IoU value in the region of the estimated local part is equal to or greater than a certain value, the second estimation unit 204 deletes the estimation result of the estimated local part. Note that the method for determining whether or not the regions are overlapping is not limited to this, and other methods may be used.

[0027] Then, the second estimation unit 204 passes the estimation result, which is expressed by the position and size on the image of a rectangular frame (bounding box) surrounding the region of the local part with low reliability estimated as described above, to the associating unit 205. Furthermore, as described above, the second estimation unit 204 assigns a flag or the like to the estimation result indicating that it was estimated by the second estimation unit 204. This allows the associating unit 205 to identify that the estimation result is derived from the second estimation unit 204.

[0028] Note that, since there may be multiple tracking target objects in one image, there may also be multiple high-reliability local region regions and multiple low-reliability local region regions estimated from one image. Conversely, even if there are multiple tracking target objects in one image, if the estimation scores of the estimated local region regions are all below the first estimation threshold, it may be that no high-reliability local region estimation results are obtained. Similarly, if the estimation scores of the estimated local region regions are all below the second estimation threshold, it may be that no low-reliability local region estimation results are obtained.

[0029] Next, in the process of S305, the associating unit 205 selects one local part area most associated with the tracking result of the tracking target object from among the local parts that are candidates for association with the tracking target object estimated by the first estimation unit 203 and the second estimation unit 204. For this selection process, the associating unit 205 performs an association determination process to determine an association score indicating the degree of association with the tracking target object (reliability of the association) for each of the high-reliability local part area and the low-reliability local part area. Then, the associating unit 205 selects one local part area most associated with the tracking result of the tracking target object based on the association score.

[0030] For example, when there is a past local part associated with the tracking result of the past image of the previous frame, the associating unit 205 sets, for each area of ​​the local part estimated in the current frame, an association score whose value increases as the distance from the past local part associated with the tracking result of the previous frame decreases. That is, the associating unit 205 determines the association of the local part based on the distance between the past local part associated with the tracking target object in the past image of the previous frame captured before the current frame and the local part in the image of the current frame. For example, the associating unit 205 determines the association of each estimated local part so that the association of a first local part whose distance from the past local part is a first distance is higher than the association of a second local part whose distance from the past local part is a second distance longer than the first distance.

[0031] Furthermore, for example, when there is no past local part associated with the tracking result of the previous frame, the associating unit 205 sets, for each area of ​​a local part estimated in the current frame, an association score whose value increases as the distance from the tracking result decreases. In other words, the associating unit 205 determines the association so that the association of a first local part, which is at a first distance from the tracking target object in the image, is higher than the association of a second local part, which is at a second distance from the tracking target object that is longer than the first distance.

[0032] Then, the associating unit 205 selects, from among the local parts estimated as candidates for association by the first estimation unit 203 and the second estimation unit 204, the area of ​​one local part having the highest relevance score as described above, as the area of ​​the local part most associated with the object to be tracked. Note that the method for calculating the relevance score is not limited to the method described above, and a method using a detector that estimates the linear area connecting the joint points of the human body, such as that disclosed in Patent Publication No. 2021-86322, may also be used.

[0033] Next, in the process of S306, associating unit 205 determines whether the local region selected in S305 is a local region estimated by first estimating unit 203 or a local region estimated by second estimating unit 204. If associating unit 205 determines in S306 that the selected local region is a local region estimated by first estimating unit 203, the process proceeds to S307, whereas if associating unit 205 determines that the selected local region is a local region estimated by second estimating unit 204, the process proceeds to S308.

[0034] When the processing proceeds to S307, the associating unit 205 performs processing to associate one local part area selected in S305, i.e., the local part area estimated with high reliability by the first estimation unit 203, with the object to be tracked. On the other hand, when the process proceeds to S308, the associating unit 205 does not perform the process of associating the one local part area selected in S305 with the tracking target object. In other words, when a local part area with low reliability estimated by the second estimation unit 204 is selected in S305, the associating unit 205 does not perform the process of associating the one local part area with the tracking target object.

[0035] After processing S307 or S308, when the process proceeds to S309, display unit 206 displays the result of the association by associating unit 205 on the display device. At this time, display unit 206 displays, for example, a rectangular frame representing the tracking result of the tracking target object and a rectangular frame representing a local part associated with the tracking target object on the input image, each in a different color. Note that if there is no local part area associated by associating unit 205, display unit 206 does not display the rectangular frame representing the local part. Furthermore, for example, if a candidate for association was selected in S305 but it is determined in S306 not to perform the association, display unit 206 may display the rectangular frame corresponding to the local part area in a color different from that used when there is association.

[0036] Thereafter, in the process of S310, the information processing device 200 determines whether to end the tracking process. If it is determined that the tracking process should be continued rather than ended, the information processing device 200 returns to the process of step S302, whereas if it is determined that the tracking process should be ended, the information processing device 200 ends the process of the flowchart shown in FIG. 3. Whether to end or continue the tracking process may be determined based on, for example, separately determined conditions. For example, in a case where the tracking process according to this embodiment is applied to the autofocus function of an imaging device (camera), the start and end of the tracking process may be determined depending on an operation such as whether or not the user half-presses the shutter button.

[0037] The series of processes from S301 to S310 in the above-mentioned flowchart will be described in more detail below with reference to the image examples in Figures 4 and 5. Figures 4 and 5 show an example in which the object to be tracked is a person, and the local part associated with the person is the person's head. For example, suppose the image shown in FIG. 4(a) is input to the tracking unit 202 as the first frame image. Here, if the user selects, for example, the person on the left side of the image in FIG. 4(a), the tracking unit 202 registers the person on the left side as a template in S301. As a result, in S302, the tracking unit 202 performs overall tracking processing on the person corresponding to the registered template. The image shown in FIG. 4(b) shows an example in which a rectangular frame surrounding a person is set as the tracking result 401 of the overall tracking processing performed by the tracking unit 202 in S302.

[0038] In addition, in S303, the first estimation unit 203 estimates a highly reliable local part that is a candidate for association with the person of the tracking target object, and in S304, the second estimation unit 204 estimates a low-reliability local part area that is a candidate for association. The image in FIG. 4(c) shows an example in which, in addition to the tracking result 401 shown in FIG. 4(b), rectangular frames are set to indicate local part estimation results 402, 403, and 404 by the first estimation unit 203 and the second estimation unit 204, respectively. Note that when estimating a local part, another object that resembles the local part's human head may be estimated as the local part. In the image in FIG. 4(c), the estimation result 404 shows a rectangular frame set by estimating a ball that resembles a human head as a candidate for the local part area.

[0039] Next, in S305, the associating unit 205 selects one local part area that is most associated with the tracking result 401 of the tracking unit 202 from among all local part area candidates estimated by the first and second estimation units 203 and 204. As described above, the associating unit 205 selects one local part area based on the relevance score obtained for each local part area estimated from the current frame. Here, the image in FIG. 4(c) is an image of the first frame, and there are no previous local parts associated with the tracking result in the previous frame. Furthermore, of the estimation results 402 to 404 obtained from the image in FIG. 4(c), the estimation result 402 is closest to the tracking result 401. Therefore, the relevance score for the local part of the estimation result 402 is the highest value, and therefore the associating unit 205 selects the local part area of ​​the estimation result 402 as the local part area that is most associated with the person in the tracking result 401.

[0040] Next, in S306, the associating unit 205 determines whether or not to finally associate the local part region of the estimation result 402 selected in S305 with the person of the tracking result 401. That is, if the local part region selected in S305 is a local part region estimated with high reliability by the first estimation unit 203, the associating unit 205 determines to associate the local part region with the tracking result. Note that in the image example of FIG. 4(c), it is assumed that the local part region of the estimation result 402 is a local part region estimated with high reliability by the first estimation unit 203. Therefore, in this case, the associating unit 205 determines to associate the local part region of the estimation result 402 with the tracking result.

[0041] The image in Fig. 4(d) shows an example of an image displayed by the display unit 206 in S307 after the associating unit 205 has decided to associate the area of ​​the local part in the estimation result 402 with the tracking result and performed the association in S307. The image in Fig. 4(d) displays the estimation result 402, which indicates the area of ​​the local part associated with the person in the tracking result 401. The example described above using Figures 4(a) to 4(d) shows a case where the subject of the tracking result is not occluded by other objects, etc., and in this case, the correct local area can be associated with the tracking result 401. Thereafter, in the information processing device 200, it is determined in S310 whether or not to end tracking. However, since the image of the next frame is input here, it is assumed that the tracking is not ended and the process proceeds to S302.

[0042] It is assumed that the image in Fig. 5(a) is an input image of the next frame. In this case, the information processing device 200 performs the processes from S302 onwards on the image of the next frame as shown in Fig. 5(a). Even when the image of the next frame is input, in S302 the tracking unit 202 performs tracking processing using the template registered in S301. It is assumed that a tracking result is obtained for the image in Fig. 5(a).

[0043] The first estimation unit 203 performs the first estimation process on the input image in S303 in the same manner as described above. Then, it is assumed that the first estimation unit 203 estimates the area of ​​a local part indicated by, for example, estimation result 502 from the image of FIG. 5(a). That is, in the example image of FIG. 5(a), the person's head indicated by estimation result 502 is the head of a person different from the person indicated by tracking result 501, and the first estimation unit 203 is unable to estimate the head of the person in tracking result 501. The reason why the first estimation unit 203 was unable to estimate the head of the person in tracking result 501 as a local part is that a part of the person's head in tracking result 501 is hidden by the hand of another person, which lowers the estimation score and makes it below the first estimation threshold.

[0044] If the process of S304 were skipped and the process proceeded to S305, the local area of ​​the estimation result 502 would be associated with the tracking result 501, resulting in an erroneous association. In contrast, in the present embodiment, the process of S304 is performed, making it possible to prevent erroneous association. The reason for this will be explained with reference to the image in Fig. 5(b). In S304, the second estimation unit 204 performs a second estimation process to estimate a region of a local part with low reliability. In the image example of FIG. 5(b), it is assumed that the second estimation unit 204 has obtained estimation results 503 and 504 of a person's head. The second estimation unit 204 uses a second estimation threshold that is lower than the first estimation threshold used by the first estimation unit 203 as a threshold for comparison with the estimation score obtained by the second estimation process. Therefore, the second estimation unit 204 estimates an estimation result 503 of the person's head in the tracking result 501, which the first estimation unit 203 was unable to estimate. Note that in the image example of FIG. 5(b), an erroneous estimation result 504 is also obtained in which the ball is also identified as a person's head, but this will be discussed in more detail later.

[0045] Proceeding to the next process of S305, the associating unit 205 selects one local part that is most associated with the tracking result 501 from among the local parts of all of the estimation results 502, 503, and 504 estimated by the first estimation unit 203 and the second estimation unit 204. Here, in the process performed on the past image of the previous frame described above in Fig. 4, the tracking result is associated with the local part, as described above. Therefore, in S305, the associating unit 205 acquires, for each area of ​​the local part estimated in the current frame, an association score that increases as the distance to the past local part associated with the tracking result in the previous frame decreases.

[0046] Then, the associating unit 205 selects one local part area having the highest relevance score as the local part area most associated with the tracking target object from among the local part values ​​of all estimation results 502 to 504 estimated by the first estimation unit 203 and the second estimation unit 204. That is, in the case of the image of Fig. 5(b), from among the estimation results 502 to 504, the local part area corresponding to estimation result 503 that is closest to estimation result 402 associated with the person in tracking result 401 in the image of the previous frame of Fig. 4(d) is selected.

[0047] Furthermore, in the next step S306, the associating unit 205 determines whether or not to ultimately associate the local portion of the estimation result 503 selected in S305 with the person of the tracking result. The local portion of the estimation result 503 is not the local portion with high reliability estimated by the first estimation unit 203 in S303, but the local portion with low reliability estimated by the second estimation unit 204 in S304. Therefore, the associating unit 205 determines not to associate the local portion of the estimation result 503 selected in S305 with the person of the tracking result 501. That is, in this case, the association result by the associating unit 205 indicates that there is no local portion associated with the person of the tracking result 501.

[0048] The image in Fig. 5(c) shows an example of an image displayed by the display unit 206 in S307 after the associating unit 205 obtained an association result indicating that there is no local part associated with the person in the tracking result 501. In the image in Fig. 5(c), since there is no area of ​​a local part associated with the tracking result 501 of the tracking target object, a rectangular frame indicating an estimated local part is not displayed, and only a rectangular frame representing the person in the tracking result 501 is displayed.

[0049] In S304, as in the example image of Fig. 5(b), there may be an estimation result 504 of not only a person's head but also an object other than a person's head, such as a ball. In this case, unless the estimation result 504 is close to the tracking result 501, it will not be selected as a local part to be associated in S305. Furthermore, even if the estimation result 504 is selected as a local part to be associated in S305, it will be determined in the next step S306 that the association will not be performed, and therefore the estimation result 504 will not be displayed in S309.

[0050] As described above, by performing the process of S304, if the head of the person in the tracking result is occluded, it is possible to prevent the occurrence of an erroneous association with a local part. Thereafter, in S310, the information processing device 200 determines whether or not to end tracking in the same manner as described above. Here, it is assumed that there is no image of the next frame, and therefore tracking is ended.

[0051] As described above, in the first embodiment, it is possible to avoid associating an incorrect local portion with the tracking result of a tracking target object. Furthermore, according to this embodiment, it is possible to eliminate the association result for other objects with which an incorrect association may occur. For example, when this embodiment is applied to an imaging device such as a camera, it is possible to cause autofocus processing to function only for correctly associated local portions. Furthermore, for example, if there is no association result, it is possible to prevent autofocus from being operated on an incorrect subject by performing control to temporarily stop autofocus so as to maintain the focus of the previous frame.

[0052] <Second embodiment> In the first embodiment described above, the second estimation unit 204 estimates a region of a local part with low reliability by setting a second estimation threshold lower than the first estimation threshold. Therefore, the second estimation unit 204 can estimate the head of a person in the tracking result whose estimation score is low due to partial occlusion, as in the example image of FIG. 5(b). However, in an actual use case, for example, as in the example image of FIG. 5(d), the entire head of the person in the tracking result may be occluded. In this case, even if a low second estimation threshold is used, the head of the person in the tracking result cannot be estimated.

[0053] In the second embodiment, the first estimation unit 203 performs the same analysis process on the image of the current frame as in the first embodiment to estimate a local area. Meanwhile, the second estimation unit 204 estimates a local area in the image of the current frame based on the estimation result of the local area in the previous image of the previous frame captured before the current frame. That is, in the second embodiment, the second estimation unit 204 uses the result of the estimation process performed on the previous frame as the second estimation process on the image of the current frame, thereby enabling the second estimation unit 204 to handle cases such as the image example shown in FIG. 5(d). Note that in the information processing device of the second embodiment, the hardware configuration is the same as in FIG. 1, the functional configuration is the same as in FIG. 2, and the information processing flowchart is generally the same as in FIG. 3, so illustrations of these are omitted. The following description will focus on the differences from the information processing device of the first embodiment.

[0054] In the flowchart of FIG. 3 according to the second embodiment, the process in S304 differs from that of the first embodiment. In the second embodiment, in S304, if there is a past local part associated with the tracking result in the previous frame, the second estimation unit 204 predicts the position of the local part in the current frame from the position of the past local part in the previous frame. Then, the second estimation unit 204 acquires an estimation result in which the predicted position of the local part is the position of the local part region obtained by the second estimation process. Note that, for example, if the images 201 input to the information processing device 200 are consecutive images with a high frame rate, the predicted position of the local part in the current frame may simply be the same as the position of the past local part estimated in the previous frame. On the other hand, if the input images 201 are consecutive images with a low frame rate, the predicted position of the local part in the current frame may be estimated from the position data of the past local part estimated in the previous frame using a Bayes filter such as a Kalman filter or a particle filter. In the following description, an example will be given in which estimation is performed only on past local parts associated with the tracking result in the previous frame, but similar estimation may be performed on all past local parts estimated in the previous frame.

[0055] The processing in the second embodiment will be described below using the example images in Fig. 5(d) to Fig. 5(f). In the second embodiment, the processing for the image of the first frame is the same as that in the first embodiment described above, and therefore the description thereof will be omitted. The images in Fig. 5(d) and Fig. 5(e) are, for example, images of the second frame, and show an example in which the entire head of a person, which is the tracking target object, is occluded. Furthermore, the image in Fig. 5(f) is assumed to be a past image of the previous frame relative to the current frame shown in Fig. 5(d) and Fig. 5(e).

[0056] In the example image of Fig. 5(d), the head of the person that is the tracking target object is occluded by the body of another person, and therefore the first estimation process of S303 only obtains an estimation result 505 of the head of the other person. Since the person in this estimation result 505 is a different person from the person in tracking result 501, if estimation result 505 is associated with tracking result 501, this will be an erroneous association.

[0057] Here, it is assumed that in the previous frame, as shown in the image of FIG. 5(f), the person's head shown in estimation result 507 was associated with tracking result 501 as a result of the association performed in S307. That is, it is assumed that in the previous frame, the people were not intertwined with each other, and the local part was correctly associated with tracking result 501 as shown in estimation result 507. In this case, in S304, the second estimation unit 204 generates estimation result 506 corresponding to the position of the person's head in tracking result 501, as shown in the image of FIG. 5(e), based on the position of estimation result 507 in the previous frame, as an estimation result of the local part.

[0058] Next, in S305, the associating unit 205 selects the estimation result that is most closely associated with the tracking result 501. In the case of the example image of FIG. 5(e), the estimation result 506 is selected. Next, in S306, the associating unit 205 performs a process of determining whether the estimation result 506 selected in S305 was estimated by the first estimation process or the second estimation process. In the case of the image example of Fig. 5(e), the estimation result 506 is a region of a local part estimated by the second estimation unit 204, and therefore the associating unit 205 determines not to associate the estimation result 506 with the tracking result 501.

[0059] This avoids erroneous association of the head estimation result 505 of a different person with the person in the tracking result 501. The subsequent processing is the same as in the first embodiment, and therefore description thereof will be omitted. As described above, according to the second embodiment, even in a case where the entire local portion of the object to be tracked is occluded, it is possible to avoid erroneous association of the local portion with the object to be tracked.

[0060] <Third embodiment> In the second embodiment, the occurrence of erroneous association in a situation where an entire local portion of a tracking target object is occluded is avoided by utilizing the result of association in the previous frame. However, for example, if the image in FIG. 5(d) is the image of the first frame, the result of association in the previous frame cannot be utilized. Of course, this is not limited to the first frame, and the same applies to the second and subsequent frames when no association was performed in the previous frame.

[0061] Therefore, the second estimation unit 204 of the third embodiment performs occlusion detection processing to estimate the position of a local part that is occluded by an occluding object in an input image, thereby estimating the position of the local part even in cases where the association result of the previous frame cannot be utilized. Then, when the association unit 205 selects a local part estimated by the occlusion detection processing in the second estimation unit 204 as a candidate local part associated with the tracking target object, it determines not to associate the selected local part with the tracking target object. Note that in the information processing device of the third embodiment, the hardware configuration is the same as in FIG. 1 , the functional configuration is the same as in FIG. 2 , and the information processing flowchart is also generally the same as in FIG. 3 , so illustrations of these are omitted. The following description will focus on the differences from the information processing devices of the first and second embodiments.

[0062] 3 according to the third embodiment, the process in S304 differs from that in the first and second embodiments. In the third embodiment, in S304, the second estimation unit 204 executes occlusion detection processing to estimate the position of a local area occluded by another person, object, or the like.

[0063] In the third embodiment, the occlusion detection process in the second estimation unit 204 is realized, for example, by using an inference device that has been trained in advance to be able to reliably detect local areas occluded by an occluding object or the like. The inference device is trained to directly detect the area of ​​the occluded local area from a real-life image in which the local area is actually occluded, such as the image in FIG. 5(d), or a composite image in which an occluding object is combined with a real-life image in which the local area is not occluded. Alternatively, the inference device may be trained in advance to detect areas on lines connecting joint points of the human body, as in the technology disclosed in Japanese Patent Laid-Open Publication No. 2021-86322, and estimate the position of the head (position of the local area) from the detection results of the areas on lines connecting joint points of the human body.

[0064] The third embodiment will also be described using the image examples of FIGS. 5(d) and 5(e) described above. 5(d) is input as the first input image to the second estimation unit 204. Note that the processing up to S303 is the same as that in the second embodiment described above, and therefore description thereof will be omitted. In the third embodiment, it is also assumed that a tracking result 501 is obtained by the overall tracking processing in S302, and an estimation result 505 is obtained by the first estimation processing in S303.

[0065] In the image of FIG. 5(d), the head of the person in the tracking result 501 is occluded by the body of another person, and therefore, in S303, only an estimation result 505 of the other person's head is obtained. If a local part were to be associated with the tracking result 501 in this state, the estimation result 505 of the other person's head would be associated, resulting in an erroneous association. For this reason, in S304, the second estimation unit 204 performs occlusion detection processing of the local part within the rectangular frame area of ​​the tracking result 501 using the estimator for occlusion detection processing described above, and estimates the position of the local part based on the occlusion detection result. As a result, the second estimation unit 204 can obtain an estimation result 506 of the local part as shown in the image of FIG. 5(e). The subsequent processing is the same as in the second embodiment, and therefore a description thereof will be omitted.

[0066] As described above, according to the third embodiment, even in cases where the entire local area of ​​the object to be tracked is occluded and the association result of the previous frame cannot be utilized, it is possible to avoid associating an incorrect local area with the object to be tracked.

[0067] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of specific embodiments for implementing the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0068] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) a detection means for detecting a tracking target object from an image; an estimation means for estimating a local area from the image; a selection means for selecting a local part having a high degree of association with the tracking target object from among the one or more local parts estimated by the estimation means; a determining means for determining whether or not to associate the tracking target object with the selected local portion, in accordance with the tracking target object detected by the detecting means and the local portion selected from the one or more local portions by the selecting means; An information processing device comprising: (Configuration 2) The determining means determining that the local portion selected by the selection means is associated with the tracking target object when the reliability of the local portion selected by the selection means is equal to or greater than a threshold; If the reliability of the local portion selected by the selection means is less than the threshold, it is determined not to associate the local portion with the tracking target object. 2. The information processing device according to configuration 1, (Configuration 3) The estimation means a first estimation means for estimating the local area from the image; a second estimation means for estimating a local region having a reliability lower than the reliability of the local region estimated by the first estimation means from the image; and the selection means selects a local part having the highest degree of association with the tracking target object from among the one or more local parts estimated by the first estimation means and the second estimation means; The determining means determining, when the local portion selected by the selection means is the local portion estimated by the first estimation means, to associate the selected local portion with the tracking target object; If the local part selected by the selection means is the local part estimated by the second estimation means, it is determined that the selected local part is not associated with the tracking target object. 2. The information processing device according to configuration 1, (Configuration 4) 4. The information processing device according to configuration 2 or 3, wherein the reliability is a degree indicating the likelihood of the local area estimated by the estimation means. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein when the determining means determines that the object to be tracked is not associated with the local area selected by the selecting means, no local area is associated with the object to be tracked in the image. (Configuration 6) an association degree determining means for determining an association degree of the local part based on a distance between a past local part associated with the tracking target object in a past image captured before the image and the local part in the image; The information processing device according to any one of configurations 1 to 5, wherein the relevance determination means determines the relevance of each of the one or more local sites such that the relevance of a first local site, the distance from which the past local site is located is a first distance, is higher than the relevance of a second local site, the distance from which the past local site is located is a second distance that is longer than the first distance. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, further comprising: a relevance determining means for determining the relevance of a first local portion, of the one or more local portions, the first local portion being a first distance from the tracking target object, in accordance with a distance between the tracking target object and the local portion in the image, such that the relevance of a first local portion, the first local portion being a first distance from the tracking target object, is higher than the relevance of a second local portion, the second local portion being a second distance longer than the first distance from the tracking target object. (Configuration 8) The estimation means a first estimation means for estimating a local site by performing an analysis process on the image; a second estimation means for estimating a local area in the image based on an estimation result of a local area in a previous image captured before the image; and The information processing device according to configuration 1, wherein the determining means determines not to associate the tracking target object with the selected local part when the local part estimated by the second estimating means is selected by the selecting means. (Configuration 9) the estimation means includes occlusion detection means for estimating a position of a local area occluded by an occluding object in the image, The information processing device according to any one of configurations 1 to 8, wherein the determining means determines not to associate the tracking target object with the selected local portion when the local portion estimated by the occlusion detecting means is selected by the selecting means. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, further comprising a display means for displaying a result of association between the object to be tracked and the selected local region. (Configuration 11) 11. The information processing device according to any one of configurations 1 to 10, further comprising a control means for controlling the focus of the imaging means, which focuses on the local area, so as to temporarily stop autofocus processing and maintain the focus that was set on the local area in a previous image, when the determination means determines not to associate the tracking target object with the selected local area. (Method 1) a detection step of detecting a tracking target object from the image; an estimation step of estimating a local site from the image; a selection step of selecting a local part having a high degree of association with the tracking target object from among the one or more local parts estimated by the estimation step; a determination step of determining whether or not to associate the tracking target object with the selected local portion, in accordance with the tracking target object detected in the detection step and the local portion selected from the one or more local portions in the selection step; An information processing method comprising: (Program 1) 12. A program that causes a computer to function as the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0069] 200: Information processing device, 202: Tracking unit, 203: First estimation unit, 204: Second estimation unit unit, 205: association unit, 206: display unit

Claims

1. a detection means for detecting a tracking target object from an image; an estimation means for estimating a local area from the image; a selection means for selecting a local part having a high degree of association with the tracking target object from among the one or more local parts estimated by the estimation means; a determining means for determining whether or not to associate the tracking target object with the selected local portion, in accordance with the tracking target object detected by the detecting means and the local portion selected from the one or more local portions by the selecting means; An information processing device comprising:

2. The determining means determining that the local portion selected by the selection means is associated with the tracking target object when the reliability of the local portion selected by the selection means is equal to or greater than a threshold; If the reliability of the local portion selected by the selection means is less than the threshold, it is determined not to associate the local portion with the tracking target object.

2. The information processing apparatus according to claim 1, wherein:

3. The estimation means a first estimation means for estimating the local area from the image; a second estimation means for estimating a local region from the image with a reliability lower than the reliability of the local region estimated by the first estimation means; and the selection means selects a local part having the highest degree of association with the tracking target object from among the one or more local parts estimated by the first estimation means and the second estimation means; The determining means determining, when the local portion selected by the selection means is the local portion estimated by the first estimation means, to associate the selected local portion with the tracking target object; If the local part selected by the selection means is the local part estimated by the second estimation means, it is determined that the selected local part is not associated with the tracking target object.

2. The information processing apparatus according to claim 1, wherein:

4. 4. The information processing apparatus according to claim 2, wherein the reliability is a degree indicating the likelihood of the local area estimated by the estimation means.

5. 2. The information processing device according to claim 1, wherein when the determining means determines that the object to be tracked is not to be associated with the local area selected by the selecting means, no local area is associated with the object to be tracked in the image.

6. an association degree determining means for determining an association degree of the local part based on a distance between a past local part associated with the tracking target object in a past image captured before the image and the local part in the image; 2. The information processing device according to claim 1, wherein the relevance determination means determines the relevance of each of the one or more local sites so that the relevance of a first local site, the distance from which the past local site is located is a first distance, is higher than the relevance of a second local site, the distance from which the past local site is located is a second distance that is longer than the first distance.

7. 2. The information processing device according to claim 1, further comprising: a relevance determining means for determining the relevance of a first local portion, of the one or more local portions, that is a first distance from the tracking target object, in accordance with a distance between the tracking target object and the local portion in the image, such that the relevance of the first local portion is higher than the relevance of a second local portion, that is a second distance from the tracking target object that is longer than the first distance.

8. The estimation means a first estimation means for estimating a local site by performing an analysis process on the image; a second estimation means for estimating a local area in the image based on an estimation result of a local area in a previous image captured before the image; and 2. The information processing device according to claim 1, wherein, when the local part estimated by the second estimation means is selected by the selection means, the determination means determines not to associate the tracking target object with the selected local part.

9. the estimation means includes occlusion detection means for estimating a position of a local area occluded by an occluding object in the image, 2. The information processing device according to claim 1, wherein, when a local portion estimated by the occlusion detection means is selected by the selection means, the determination means determines not to associate the tracking target object with the selected local portion.

10. The information processing apparatus according to claim 1 , further comprising a display means for displaying a result of the association between the object to be tracked and the selected local region.

11. 2. The information processing device according to claim 1, further comprising a control means for controlling the focus of the imaging means, which focuses on the local area, so as to temporarily stop autofocus processing and maintain the focus that was set on the local area in a past image, when the determination means determines not to associate the tracking target object with the selected local area.

12. a detection step of detecting a tracking target object from the image; an estimation step of estimating a local site from the image; a selection step of selecting a local part having a high degree of association with the tracking target object from among the one or more local parts estimated by the estimation step; a determination step of determining whether or not to associate the tracking target object with the selected local portion, in accordance with the tracking target object detected in the detection step and the local portion selected from the one or more local portions in the selection step; An information processing method comprising:

13. Computer, a detection means for detecting a tracking target object from an image; an estimation means for estimating a local area from the image; a selection means for selecting a local part having a high degree of association with the tracking target object from among the one or more local parts estimated by the estimation means; a determining means for determining whether or not to associate the tracking target object with the selected local portion, in accordance with the tracking target object detected by the detecting means and the local portion selected from the one or more local portions by the selecting means; A program that causes the device to function as an information processing device having the above.

Images