Information processing apparatus, information processing method and program

Abstract

To reduce an error in association of a specific part to a subject.SOLUTION: A subject is detected from a first image and a second image following the first image in time series. Partial regions showing a specific part associated with the subject are detected from the first image and second image, respectively. A state of change in number of detected partial regions between the first image and second image is acquired. According to the state of change, it is controlled whether to associate the specific part extracted from the second image with the subject.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] The area of ​​a specific subject is detected and tracked from a series of images. Tracking involves detecting the area of ​​a desired subject from an image and tracking the same subject area between successive images. Based on the tracking results, the autofocus process of the camera capturing the images is carried out.

[0003] Patent Document 1 discloses a method for tracking a subject to be tracked while associating the whole body with parts. For example, in the case of a person as the subject, the whole body and parts of the subject are considered to be the whole and the face as parts. In Patent Document 1, the association of the whole body and parts is performed based on the positional relationship (for example, the proximity) between the body part and parts of the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-152578 Summary of the Invention [Problem to be solved by the invention]

[0005] When associating a whole with parts, a method based on the positional relationship between the whole and parts has a problem in that the association between the whole and parts is likely to fail if similar objects intersect at the same position on the image. For example, the technology described in Patent Document 1 has a problem in that, when tracking a person, if the head of another person intersects with the head of the person being tracked, an incorrect association is made.

[0006] The present invention aims to reduce errors in associating specific parts with subjects. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an information processing device according to one embodiment includes the following configuration: a first detection means for detecting a subject from each of a first image and a second image that chronologically follows the first image, a second detection means for detecting partial regions that indicate specific parts of the subject from each of the first image and the second image, an acquisition means for acquiring a change in the number of the detected partial regions between the first image and the second image, and a first control means for controlling, according to the change, whether to associate the specific parts of the partial regions extracted in the second image with the subject. [Effects of the Invention]

[0008] To reduce errors in associating a specific part with a subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment. [Figure 2] 10 is a flowchart showing an example of the overall processing by the information processing device. [Figure 3] 10 is a flowchart showing an example of processing by a status acquisition unit. [Figure 4] 10 is a flowchart showing an example of processing by a threshold setting unit. [Figure 5] 10 is a flowchart showing an example of processing by an association unit. [Figure 6] 10 is a flowchart showing an example of processing by a display control unit. [Figure 7] FIG. 10 is a diagram for explaining a state in which the number of local detections changes. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] An information processing device according to embodiment 1 will be described below. The information processing device according to this embodiment uses time-series consecutive images as input, tracks each subject detected from the image, tracks the entire subject to be tracked, and detects a specific part of the subject and associates it with the subject. Hereinafter, when simply referred to as a "specific part," this refers to a specific part detected by the information processing device according to this embodiment, related to the subject to be processed. In addition, below, such a specific part may be referred to as a "local part" or a "local part."

[0012] In this embodiment, an example will be described in which a human body (the entire human body) is used as the subject to be processed, and the head of the human body is used as the specific part of the subject; however, the present invention is not limited to this as long as similar processing can be performed. For example, the subject and its specific part may be the head and eyes of a human body, an animal (the entire body) and the animal's face, or a vehicle and its license plate. Furthermore, the specific part does not have to be a part of the subject, as long as it is an object that is expected to be captured in association with the subject. For example, a vehicle such as a vehicle or an animal and the head of a person riding in the vehicle may be used as the subject and its specific part. Although the person's head is not part of the vehicle, it moves in association with the vehicle being tracked, and therefore can be considered a specific part of the subject.

[0013] FIG. 1 is a diagram showing an example of the configuration of an information processing device 100 according to this embodiment. As an example of a hardware configuration, the information processing device 100 includes a CPU 101, a computer bus 102, a first memory 103, a second memory 104, an input unit 105, a display unit 106, and a communication unit 107. The CPU 101 controls the entire information processing device 100. The first memory 103 and the second memory 104 are memories that store a control program that executes processing according to this embodiment, as well as various data. In FIG. 1, the first memory 103 is illustrated as mainly storing the control program, and the second memory 104 is illustrated as mainly storing various data, but this is not particularly limited as long as similar data can be stored throughout the information processing device 100.

[0014] The input unit 105 is configured with a keyboard, a touch panel, or the like, and accepts input from the user. The display unit 106 is configured with a display device such as a liquid crystal display, and can display processing results to the user. The communication unit 107 can communicate with external devices to exchange data. The computer bus 102 connects the various functional units of the information processing device 100. The information processing device 100 according to this embodiment may be implemented as, for example, a computer having a program that executes the various processes described below.

[0015] In this embodiment, the first memory 103 stores a program for executing each process described as being executed by the information processing device 100. The local feature calculation unit 118 shown in FIG. 1 will be described in the second embodiment.

[0016] The tracking unit 110 tracks a subject in an image. The tracking of the subject by the tracking unit 110 can be performed using any commonly used technology for tracking subjects in an image, and tracking may be performed, for example, by template matching or a machine learning model that has been trained in advance to track subjects in an image. The following description will be given assuming that the tracking unit 110 according to this embodiment detects and tracks subjects by template matching.

[0017] The local detection unit 111 detects a partial region (local region) indicating a specific part from an image. The local detection unit 111 according to this embodiment detects a local region from each of a first image and a second image that chronologically follows the first image. The detection of a specific part by the local detection unit 111 can be performed by a general process of detecting an object in an image. Here, the local detection unit 111 detects the local part using a machine learning model that has been trained in advance to detect a specific part in an image. Hereinafter, the first image may be referred to as (an image of) the previous frame, and the second image may be referred to as (an image of) the current frame.

[0018] The status acquisition unit 112 acquires the status of change in the number of local regions detected by the local detection unit 111 between the first image and the second image. The status acquisition unit 112 acquires information on, for example, whether the number of local regions detected between the first image and the second image is increasing, decreasing, or unchanged. Hereinafter, such a status of change in the number of local regions between the first image and the second image may be simply referred to as the "change status." Specific examples of processing by the status acquisition unit 112 will be described later.

[0019] The associating unit 113 associates the local region (specific portion) detected by the local detection unit 111 with the subject to be tracked. Multiple local regions can be detected from an image, but the most suitable local region is selected based on predetermined conditions and associated with the subject to be tracked. Hereinafter, this process of "associating a local region with a subject" will be considered to be the same as the process of associating a specific portion corresponding to such a local region with the subject. In this embodiment, the process of associating the local region with the subject by the associating unit 113 is controlled according to the change state acquired by the state acquisition unit 112. In particular, the threshold set by the threshold setting unit 115, which will be described later, is controlled according to the change state, and the associating unit 113 associates the local region with the subject using the threshold controlled in this manner.

[0020] For example, when the number of detected local regions in the second image is smaller than in the first image, the associating unit 113 may suppress association of the local regions detected in the second image with the subject. As will be described in more detail later with reference to FIG. 5, when the number of detected local regions decreases over time, it is considered that the local regions are hidden. From this perspective, such processing by the associating unit makes it possible to suppress association when there is an increased possibility of erroneous association. Therefore, it is possible to reduce a negative impression on the user due to erroneous association (for example, by displaying a bounding box for the wrong object), or to reduce adverse effects on subsequent processing such as tracking processing due to erroneous association. Details of the processing by the associating unit 113 will be described later with reference to FIG. 5.

[0021] The suppression of association is not particularly limited as long as it is a process that makes it more difficult to associate local regions with a subject when fewer local regions are detected in the second image than in the first image. For example, the associating unit 113 may associate local regions with a subject without using a judgment threshold 132 (described later) when the number of detected local regions in the second image is the same as or greater than that in the first image, and may set an additional condition for association (for example, use a judgment threshold 132 (described later)) when fewer local regions are detected in the second image than in the first image. Furthermore, for example, the process of suppressing association may be a process of increasing the threshold used for association, or a process of additionally displaying a display indicating that association is suppressed.

[0022] When using conventional technology to associate an entire subject with a specific part based solely on their positional relationship, even if the specific part does not correspond to the subject, it can be erroneously associated if the specific part is located correctly. In a scene where people cross paths, the specific parts (e.g., heads) of two people may overlap, resulting in specific parts that should not be associated appearing in the same location. Figure 7 illustrates a situation in which the number of local detections decreases due to overlapping specific parts of the subject. Figure 7(A) illustrates a state in which the number of local detections is 2 for an input image 701. An overall tracking frame 702 represents the overall tracking region 124, which indicates the subject to be tracked. Local detection frames 703 and 704 each represent two local regions detected as local part regions and stored in the local region candidate 127. Figure 7(B) shows an input image 705, which is the next frame of Figure 7(A). In the input image 705, the number of local detections has decreased from 2 in the previous frame to 1 due to the overlapping of people. The local detection frame 707 in FIG. 7(B) is an area in which a local part is detected in the frame in FIG. 7(B). The overall tracking frame 706 is a frame representing the overall tracking area 124, which indicates the subject to be tracked, just like in FIG. 7(A). In the example of FIG. 7(B), a local part of another person overlaps the position of a local part associated with the subject that is the original tracking target. In such cases, conventional methods have had the problem of being unable to correctly associate the subject with a specific part. On the other hand, by controlling whether or not to associate a specific part with the subject depending on the state of change in the number of detected partial areas, it is possible to reduce the above-mentioned association errors.

[0023] The judgment index setting unit 114 sets a judgment index. The judgment index according to this embodiment is an evaluation value used for evaluation when associating a subject in an image with a local part. In this embodiment, the description will be made assuming that a first-rank tracking score, which will be described later, is used as the judgment index.

[0024] The threshold setting unit 115 sets a threshold for the judgment index 131. For example, when the number of detected local regions is decreasing based on the change state, the threshold setting unit 115 according to this embodiment may set a threshold (as the judgment threshold 132) based on equation (1) described later. The processing by the threshold setting unit 115 will be described later with reference to FIG. 4.

[0025] 2 is a flowchart showing an example of the overall information processing performed by the information processing device 100 according to the first embodiment. In S201, the information processing device 100 performs initial settings of data related to various processes. For example, as the initial settings, the tracking unit 110 registers a template of the subject to be tracked in the template 120 in the memory 104. This processing can be performed, for example, by accepting a user's selection of the subject on the screen. In this example, the subject is the entire person, and a full-body image of the person is registered as a template to be tracked.

[0026] Furthermore, for example, the state acquisition unit 112 performs initial setting of a memory for storing the state of local detection (detection of local areas) as an initial setting. The state acquisition unit 112 according to this embodiment manages the state of change in the number of detected local areas by an ID (indicated by, for example, 0 to 1), and can set the initial value of the ID to 0.

[0027] Furthermore, for example, the local detection unit 111 initializes a memory for storing the number of detected local regions. Here, the local detection unit 111 stores the number of local regions detected from the current frame as current frame local region number 129 in the memory 104, and the number of local regions detected from the previous frame as previous frame local region number 130 in the memory 104, and can set the initial values ​​of each to 0.

[0028] In S202, the tracking unit 110 performs a process of tracking the subject in the image to be processed. Here, the tracking unit 110 acquires one frame of image to be processed, searches the image for an area similar to the template 120, and outputs a tracking area and a tracking score. The tracking score according to this embodiment is an evaluation value (a numerical value representing the reliability of the tracking result) that evaluates the tracking accuracy, and the higher the numerical value, the more reliable the tracking result. Here, the tracking unit 110 calculates multiple tracking area candidates on the image, and the one with the highest tracking score among them is determined to be the tracking area (tracking result) of the subject. The tracking score for the candidate can be calculated, for example, based on the degree of match with the tracking area in the previous frame, or the image similarity between the tracking area and the template. Here, the tracking unit 110 stores the respective scores of the candidate with the first and second highest tracking scores in the memory 104 as a first-place tracking score 122 and a second-place tracking score 123, and stores the tracking area of ​​the first-place candidate in an entire tracking area 124 in the memory 104. In this embodiment, each area including the tracking area is represented by the position and size of a bounding box (rectangular area) on the image. Note that the tracking unit 110 stores the first-place tracking score and the second-place tracking score in the previous frame before performing the above tracking process and storing the first-place tracking score 122 and the second-place tracking score 123. The first-place tracking score and the second-place tracking score in the previous frame are stored in the memory 104 as a previous frame first-place tracking score 125 and a previous frame second-place tracking score 126.

[0029] In S203, the local detection unit 111 detects local regions from the image. Here, the local detection unit 111 detects local region areas associated with the subject to be tracked from the input image and outputs the local regions and local region detection scores. Here, a human head is detected as the local region, so a rectangular region surrounding the human head is output as the local region. The local region detection score is a numerical value that represents the reliability of the detection result, and the higher the numerical value, the more reliable the detection result. Note that, for example, when a region whose local region detection score exceeds a predetermined threshold (which can be set arbitrarily) is detected as a local region, there may be multiple local regions detected from a single image. In the example of FIG. 1, the local detection unit 111 stores a local region candidate 127 and a local region candidate detection score 128 in the memory 104. The local region candidate 127 and the local region candidate detection score 128 are assumed to be an array that stores multiple detection results. The number of local regions detected in the current frame is stored as the current frame local region detection number 129 in the memory 104.

[0030] In S204, the judgment index setting unit 114 stores the judgment index 131 in the memory 104. The judgment index is used in the association process by the association unit 113, which will be described later. The judgment index setting unit 114 according to this embodiment uses the first-place tracking score 122 as the judgment index 131. Note that, although details will be described later in a second embodiment, something other than the tracking score may be used as the judgment index 131.

[0031] In S205, the status acquisition unit 112 acquires the change status of the frame to be processed from the previous frame. In this embodiment, the status acquisition unit 112 stores an ID (numeric value) that identifies the change status in the memory 104 as the local detection status 121. Here, the status acquisition unit 112 sets the local detection status 121 to "1" if the number of local regions detected in the current frame has decreased compared to the previous frame, and sets the local detection status 121 to "0" if the number of detected local regions has increased. Details of the processing performed in S205 will be described later with reference to FIG. 3.

[0032] In S206, the threshold setting unit 115 sets a threshold for the judgment index 131. Here, the threshold setting unit 115 stores a judgment threshold 132 as a threshold in the memory 104. Details of the processing by the threshold setting unit 115 will be described later.

[0033] In S207, the associating unit 113 associates the subject with a local region. Here, the associating unit 113 can select one of multiple candidates included in the local region candidate 127 and associate it with the subject. Information indicating the local region associated with the subject is stored in the memory 104 as a local site region 133. If there is no local region associated with the subject, information indicating that there is no associated local region is stored as the local site region 133. Furthermore, the associating unit 113 stores the reason for the association as a corresponding value in the association status 137. Details of these processes by the associating unit 113 will be described later.

[0034] In S208, the display control unit 117 displays the tracking results on the display unit 106. For example, the display control unit 117 can display frames representing the overall tracking region 124 and the local region region 133 on the input image in different colors. If the local region region 133 is empty (no associated local region exists), the frame corresponding to the local region region 133 may not be displayed. Furthermore, if a local region candidate is located near the subject to be tracked (for example, within a predetermined range centered on the subject), but the candidate has not been associated with the subject by the associating unit 113 based on the determination result of the state acquiring unit 112, the display control unit 117 may display the region frame corresponding to the candidate in a color different from that used when it has been determined that there is an association. Details of the processing by the display control unit 117 will be described later.

[0035] In S209, the information processing device 100 determines whether to end the tracking process. If tracking is to continue, the process returns to S202; if not, the process in FIG. 2 ends. The conditions for whether to end the tracking process can be set arbitrarily. For example, if the first-place tracking score 122 becomes lower than a predetermined threshold, the information processing device 100 may terminate the tracking process, assuming that the tracking target has been lost. Furthermore, for example, assuming that the information processing device 100 is used in the autofocus function of a camera, the information processing device 100 may determine whether to start or end tracking depending on whether a predetermined operation, such as whether or not the user half-presses the shutter button, is performed.

[0036] Next, details of the processing performed by the status acquisition unit 112 will be described. Fig. 3 is a flowchart showing an example of the processing in S205. In S301, the status acquisition unit 112 branches the processing depending on the current local detection status 121. Here, if the local detection status is 0, the status acquisition unit 112 advances the processing to S302, and if it is 1, the status acquisition unit 112 advances the processing to S304.

[0037] In S302, the status acquisition unit 112 determines whether the number of local region detections in the current frame (local detection count) has decreased compared to the previous frame. The status acquisition unit 112 can make this determination by comparing the current frame local region detection count 129 with the previous frame local region detection count 130 in the memory 104. If the local detection count has decreased from the previous frame, the process proceeds to S303, and if the local detection count is the same as or has increased compared to the previous frame, the process in FIG. 3 ends.

[0038] In S303, the state acquisition unit 112 sets the local detection state 121 in the memory 104 to 1. If the number of local detections has decreased compared to the previous frame, this suggests that the decrease in the number of local detections may be due to the overlapping of the positions of multiple local regions. When this state occurs, there is a possibility that local parts associated with the subject to be tracked are hidden, so the threshold used by the association unit 113 is controlled. Also, in S303, the state acquisition unit 112 sets a number of elapsed frames 134 in the memory 104 to count the number of elapsed frames after changing the local detection state, and initializes this to 0. The state acquisition unit 112 also stores the previous frame first-place tracking score 125 and the previous frame second-place tracking score 126, which are currently in the memory 104, as a first reference index 135 and a second reference index 136, respectively. The first reference index 135 is used to record the value of the tracking score immediately before the number of local detections decreased. In this embodiment, the judgment index setting unit 114 uses the tracking score as a judgment index for determining whether to perform association, and therefore the tracking score is stored here as the reference index. When an evaluation value other than the tracking score is used as the judgment index 131, the judgment index used by the judgment index setting unit 114 is stored in the first reference index 135.

[0039] In S304, when the local detection state 121 is set to 1 in S301, the state acquisition unit 112 adds one to the number of elapsed frames 134 in the memory 104.

[0040] In S305, the status acquisition unit 112 determines whether the number of local detections has increased. This determination can be made by comparing the number of local area detections in the current frame 129 with the number of local area detections in the previous frame 130 in the memory 104. In S305, if the number of local detections is the same as or has decreased compared to the previous frame, the process proceeds to S306, and if the number of local detections has increased from the previous frame, the process proceeds to S307.

[0041] In S306, it is determined whether the number of elapsed frames 134 in memory 104 exceeds a separately set maximum value for the number of elapsed frames (a predetermined number of frames). If the number of elapsed frames 134 exceeds the predetermined number of frames, the process proceeds to S307; if not, the process in FIG. 3 ends.

[0042] In S307, the state acquisition unit 112 sets the local detection state 121 in the memory 104 to 0. Therefore, if the number of local detections increases or if the number of elapsed frames 134 exceeds a predetermined maximum value, the local detection state 121 is set to 0 in S307. The fact that the number of local detections decreases and then increases again suggests that the desired local region may have reappeared after being hidden by another object. Furthermore, if the number of elapsed frames exceeds a predetermined maximum value, this suggests that there may be less likelihood that the desired local region will reappear. From this perspective, in either of the above cases, the local detection state 121 is set to 0, which is the initial value, in S307. In the processing by the associating unit 113, which will be described later, a process of associating the subject with the local region is performed based on the local detection state 121 set by the state acquisition unit 112 as described above. The state acquisition unit 112 sets the local detection state based on an increase or decrease in the number of detected local regions, and the associating unit 113 associates the subject with the local regions based on this local detection state, thereby making it possible to appropriately control whether or not to associate a specific part with the subject depending on the state of change in the number of detected local regions.This makes it possible to reduce the mistake of associating the wrong specific part with the subject.

[0043] Note that a decrease in the number of local detections can occur for reasons other than obscuring the local region of the tracking target subject, such as when the head of a subject that is not the tracking target is framed out. In such cases, it is conceivable that association errors may not be detected using simple conventional techniques. From this perspective, although the local detection state is set to 1 when the number of detected local regions is decreasing in FIG. 3 , the local detection state may also be set to 1 when the number of local regions becomes 1. According to this processing, only when the number of local detections decreases from multiple to 1, it is possible to carefully check using the index 131 whether the detected local region is associated with the tracking target, thereby simplifying the overall processing accordingly.

[0044] Next, a description will be given of the processing performed by the threshold setting unit 115. The threshold setting unit 115 sets a threshold (decision threshold 132) for the decision index 131 according to the state of change in the number of local regions. Here, in this embodiment, the threshold setting unit 115 uses the tracking score as the decision index 131, and therefore the decision threshold 132 becomes the threshold for the tracking score.

[0045] 4 is a flowchart showing an example of the processing in S206. In S401, the threshold setting unit 115 determines whether the local detection state 121 is 0. If the local detection state 121 is 0, the processing ends without setting the judgment threshold 132 as a threshold (using the initial value as the threshold); otherwise, the processing proceeds to S402. Here, if the local detection state 121 is 0, the judgment threshold 132 is not used, so it does not need to be set.

[0046] In S402, the threshold setting unit 115 sets the decision threshold 132 based on the first reference index 135 and the number of elapsed frames 134 in the memory 104. The threshold setting unit 115 can calculate the decision threshold 132 based on, for example, the following equation (1). Th=S1×α f Formula (1)

[0047] Here, Th is the judgment threshold 132, S1 is the first reference index 135, and f is the number of elapsed frames 134. Also, α is a predetermined coefficient in the range of 0 to 1. If α is 1, the first reference index 135 is always the judgment threshold 132 regardless of the number of elapsed frames 134. On the other hand, if α is less than 1, the judgment threshold 132 decreases as the number of elapsed frames increases.

[0048] The first reference index 135 stores the tracking score when the local detection state 121 is 0, that is, when a local part associated with the subject to be tracked is not hidden. Here, the more time passes after the local detection state 121 becomes 1, the more likely the tracking score will decrease due to changes in the posture of the human body of the subject to be tracked, and so the judgment threshold is lowered as the number of elapsed frames increases.

[0049] Note that, although the description has been given assuming that the judgment threshold 132 decreases in accordance with the number of elapsed frames (when α is less than 1), the method of setting the judgment threshold 132 by the threshold setting unit 115 is not particularly limited to this. For example, the judgment threshold 132 may be set by multiplying the first reference index 135 by a predetermined coefficient, and maintained regardless of the number of elapsed frames. Also, the judgment threshold when the local detection state 121 is 0 may be set in advance, and maintained at that value.

[0050] The threshold setting unit 115 may also use the second reference index 136 to calculate the decision threshold 132 based on the following equation (2): where S2 is the second reference index 136. Th=max(S1)×α f ,S2)

[0051] The second reference index 136 stores the tracking score for the second-highest tracking candidate in the previous frame at the time when the local detection state 121 becomes 1, i.e., the tracking score for the tracking candidate that was not deemed to be a tracking target. By using Equation 2, there is an effect that the judgment threshold 132 is set so that it does not fall below the tracking score of a tracking candidate that was determined not to be a tracking target at least in the previous frame.

[0052] Although the coefficient α is assumed to be a fixed value here, the coefficient α may be set based on, for example, the first reference index 135 and the second reference index 136. For example, the threshold setting unit 115 may set the judgment threshold 132 to the same value as the second reference index when the number of elapsed frames reaches a predetermined value (N). For example, when calculating the judgment threshold 132 based on equation (2), the threshold setting unit 115 may use α calculated based on the following equation (3). α=10^((log(S1) / S2)) / N) Equation (3)

[0053] Equation (3) is S1×α N =S2 is solved for α. By setting α in this way, after the local detection state 121 becomes 1, the judgment threshold 132 is lowered from the first reference index 135 to the second reference index 136 in the Nth frame, and thereafter the judgment threshold is fixed to the second reference index. This has the effect of preventing the judgment threshold from becoming lower than the tracking score of a tracking candidate that was not previously determined to be a tracking target.

[0054] Next, the processing by associating unit 113 will be described. The area of ​​the subject to be tracked is stored in memory 104 as overall tracking area 124, and zero or more candidate areas for local parts are stored as local area candidate 127. Associating unit 113 selects a local area to be associated with a local part accompanying the subject to be tracked from the candidates included in local area candidate 127, and stores the selected local area in local part area 133 in memory 104. Associating unit 113 also stores a numerical value (which will be described later, but is shown in four stages from 0 to 3 here) indicating the reason for determining the association in memory 104 as association status 137.

[0055] FIG. 5 is a flowchart showing an example of the processing in S207. In S501, the associating unit 113 selects a local region from among the local region candidates. The processing in S501 is processing for selecting a local region associated with a local part associated with a subject to be tracked from among the candidates included in the local region candidates 127, and can be performed using any known technology for selecting from among the candidate regions. For example, the associating unit 113 may select the local region candidate that is closest to the overall tracking region 124 on the image, or may select the local region candidate based on its positional relationship with the overall tracking region 124 (e.g., whether it is located in the up, down, left, or right direction). Furthermore, when the local part is a human head, taking into account that the head is usually photographed so that it is positioned above the center of the human body on the image, the associating unit 113 may add a condition that the local part be located above the overall tracking region 124 on the image to the selection conditions in S501. Furthermore, from the viewpoint that it is considered that the position of the local part will not move significantly from the previous frame when the frame rate is relatively fast, the associating unit 113 may select in S501, from among the local part candidates, one that is closest to the local part area 133 in the previous frame. Furthermore, a calculation unit (not shown) may calculate the depth distance from the image capture device to the subject, and the local area for which the difference between the depth distance to the entire tracking area and the depth distance of the local area is smallest may be selected in S501.

[0056] In S502, the associating unit 113 determines whether or not the candidate selected in S501 exists. If the selected candidate does not exist, the process proceeds to S503, and if the selected candidate exists, the process proceeds to S504.

[0057] In S503, associating unit 113 does not associate a local region with the subject, and ends the processing in Fig. 5. In this example, associating unit 113 sets association state 137 in memory 104 to 0. Here, association state 137 being 0 means that no local region associated with the subject to be tracked has been detected, and this indicates that no association has been determined. Processing according to the association state will be described later with reference to Fig. 6.

[0058] In S504, the associating unit 113 determines whether the local detection status 121 in the memory 104 is 0. If it is 0, the process proceeds to S505; otherwise, the process proceeds to S506. In S505, the associating unit 113 associates the local region selected in S501 with the subject and stores information indicating that no decrease in the detected local region has occurred, and the process of FIG. 5 ends. Here, the associating unit 113 stores information indicating that the local region selected in S501 is associated with the subject and information indicating that no decrease in the detected local region has occurred, by setting the association status 137 to 1. This "setting the association status 137 to 1 (or 2, described later)" may be information indicating that the local region corresponding to the selection in S501 is associated with the subject, or may be information indicating that information indicating such a local region is associated with the subject. Here, when the association state 137 is 1, there is a local part associated with the subject to be tracked, and since the local detection state is 0, there is no decrease in the number of local detections, which means that there is a high possibility that the local part to be associated with the subject to be tracked is not hidden in the current frame.

[0059] In S506, the associating unit 113 determines whether the judgment index 131 in the memory 104 exceeds the judgment threshold 132. If the judgment index 131 exceeds the judgment threshold 132, the process proceeds to S507, and if not, the process proceeds to S508.

[0060] In S507, the associating unit 113 associates the local region selected in S501 with the subject and stores information indicating that a decrease in the detected local region has occurred, thereby completing the processing of FIG. 5 . Here, the associating unit 113 stores information indicating that the local region selected in S501 is associated with the subject and information indicating that a decrease in the detected local region has occurred by setting the association status 137 to 2. Here, when the association status 137 is 2, a decrease in the number of local detections is observed, and it is possible that a local part to be associated with the subject to be tracked is occluded in the current frame. However, since the judgment index is equal to or greater than the threshold, it is highly likely that the local part is not occluded. Note that in S507, the local detection status 121 may be set to its initial value of 0. This is because it is determined that the desired local part is not occluded and this state can be considered the initial state. If it is considered necessary to check the judgment index in subsequent frames as well, the local detection status 121 may remain at 1.

[0061] In S508, the associating unit 113 stores information indicating that the local region selected in S501 is not associated with the subject, and ends the processing in Fig. 5. Here, the associating unit 113 stores information indicating that the local region selected in S501 is not associated with the subject by setting the association status 137 to 3. Here, the association status 137 being 3 means that a decrease in the number of local detections has been observed, there is a possibility that a local part that should be associated with the subject to be tracked has been hidden in the current frame, and there is a high possibility that it is hidden because the judgment index is equal to or less than the threshold.

[0062] Note that when association state 137 is set to 3 in this way, the reliability of the local part associated with the subject to be tracked using the conventional method is low. Therefore, when association state 137 is 3, association unit 113 determines that there is no local part to be associated, and does not perform processing to associate the entire subject with the local part. For example, when autofocus processing is being performed to focus on a local part associated with the subject being tracked, if association state 137 is 3, processing is performed to determine that the focus target has been lost.

[0063] Furthermore, the information processing device 100 may perform autofocus processing (preset to correspond to the value of the association information 137) based on the reason for determination recorded in the association state 137. For example, various processes, such as restarting from the initial tracking settings or waiting for several frames while maintaining the focus position for the desired local part to reappear, may be associated in advance with the association state 137, and such processes may be performed according to the association state 137. Furthermore, as will be described later, when displaying a local part associated with the subject being tracked, processes such as stopping the display of the local part region or changing the display expression may be performed based on the association state 137.

[0064] In addition, in the explanation of the associating unit 113 of this embodiment, the threshold setting unit 115 sets the judgment threshold 132 in S206 of Fig. 2 and performs a comparison with the judgment index 131 in S506 of Fig. 5, but these processes may be omitted. In that case, the process may always proceed to S508 without performing a comparison in S506. In this case, whenever a decrease in the number of local detections is observed, the association status 137 is set to 3, and it is determined that there is no association.

[0065] Next, processing by the display control unit 117 will be described. The display control unit 117 displays the tracking results on the display unit 106. The display control unit 117 according to this embodiment can display the local region associated with the subject in a format based on the association state 137. For example, the display control unit 117 displays, on the input image, a frame representing the overall tracking region 124 and a frame representing the local part region 133, each in a different color.

[0066] An example of changing the display mode of such a local area in accordance with the association state 137 will be described below. Note that this process is just an example, and the display mode is not limited to this. FIG. 6 is a flowchart showing an example of the process in S208. In S601, the display control unit 117 copies the input image to the display image 138. In S602, the display control unit 117 determines whether the contents of the overall tracking area 124 are empty (whether there are any local areas stored as the overall tracking area 124). If the overall tracking area 124 is empty, the process proceeds to S609, where the input image is displayed as is as the display image 138, and the process ends. If the overall tracking area 124 is not empty, the process proceeds to S603.

[0067] In S603, the display control unit 117 controls so that a frame corresponding to the overall tracking area 124 in the memory 104 is drawn in red and superimposed on the display image 138, and the process proceeds to S609. The overall tracking area 124 is a bounding box that represents the area of ​​the subject to be tracked. In S604, the display control unit 117 branches the process depending on the value stored in the association status 137. Here, if the association status 137 is 0, the process proceeds to S605; if it is 1, the process proceeds to S606; if it is 2, the process proceeds to S607; and if it is 3, the process proceeds to S608.

[0068] In S605, the display control unit 117 performs control so that information indicating that there is no local part associated with the subject to be tracked is superimposed on the display image 138, and the process proceeds to S609. Here, as information indicating that there is no local part associated with the subject to be tracked, text indicating this may be superimposed on the display image, or control may be performed so that no additional superimposed display is performed.

[0069] In S606, the display control unit 117 controls the drawing of a frame corresponding to the local part region 133 in green on the display image 138, and the process proceeds to S609. This indicates that the number of detected local parts has not decreased in the current frame, and therefore the local parts have been associated using the conventional association method.

[0070] In S607, the cover control unit 117 controls so that a frame corresponding to the local part area 133 is drawn in yellow and superimposed on the display image 138, and the process proceeds to S609. This indicates that although the number of detected local parts has decreased and there is a possibility that the desired local part is hidden, the tracking score, which is a judgment index, is equal to or greater than the judgment threshold, and therefore the subject to be tracked is visible and not hidden, and it has been determined that the association of the local parts is valid. In other words, it has been determined that the decrease in the number of local parts in the current frame is due to the head of a person other than the subject being hidden.

[0071] In S608, the cover control unit 117 controls the drawing of a frame corresponding to the local part region 133 as a gray dashed line superimposed on the display image 138, and proceeds to S609. This indicates that the number of local parts detected in the current frame has decreased, and the tracking score, which is a judgment index, is below the judgment threshold, so that the local part association performed by the conventional association method has been determined to be invalid. In this case, it is possible to stop displaying the local part region.

[0072] In S609, the cover control unit 117 displays the display image 138 on the display unit 106, and then ends the process. Here, the cover control unit 117 displays the display image 138 in a manner controlled in either S602 or S605 to S608. Note that, in FIG. 6, the description has been given of each frame being displayed in a different color, but this is an example of displaying each frame in a different manner, and is not particularly limited to this. For example, the type of frame line may be differentiated using a straight line, a dashed line, or a dashed-dotted line, or the inside of the frame may be differentiated by hatching, etc.

[0073] According to this processing, it is possible to appropriately control whether or not to associate a specific part with a subject, depending on the state of change in the number of detected local areas. Therefore, it is possible to reduce the mistake of associating an incorrect specific part with a subject. Note that the entire tracking area 124, the local part area 133, or the association status 137 may be transmitted to an external device (for example, to an external device that operates based on the tracking result by the information processing device 100) via the communication unit 107 in FIG. 1.

[0074] [Embodiment 2] In the first embodiment, a tracking score is used as the judgment index 131. In the second embodiment, a similarity of image features of local site regions is used as the judgment index 131. In particular, here, a similarity between a feature amount of a local region associated with a subject in a first image and a feature amount of a local region in a second image is used.

[0075] The local feature calculation unit 118 according to this embodiment can calculate image features of two or more given image regions and calculate the similarity therebetween. Methods for calculating the similarity between two images using image features are widely known, and detailed explanations of the processing will be omitted here. For example, the local feature calculation unit 118 can calculate the similarity used in a method for calculating the similarity between face images and performing personal authentication using face images. In this embodiment, some of the processing described as being performed in the first embodiment is changed.

[0076] The information processing device 100 according to this embodiment can basically perform the same processing as that of embodiment 1. In the following, processing performed by the information processing device 100 according to embodiment 2, which differs from that of embodiment 1, will be described, and overlapping descriptions will be omitted.

[0077] In S204 according to this embodiment, the judgment index setting unit 114 stores the image features of each candidate stored in the local region candidate 117 in the judgment index 131. The other processes shown in Fig. 2 are the same as those in the first embodiment, and therefore will not be described here.

[0078] In S303 according to this embodiment, the state acquisition unit 112 calculates image features of the local region area in the previous frame as a reference index, and stores this as a first reference index 135. The state acquisition unit 112 may also store image features of a candidate that was not determined to be a local region associated with the subject to be tracked in the previous frame as a second reference index 136. The other processes shown in Fig. 3 are the same as those in the first embodiment, and therefore will not be described here.

[0079] In S402 of Fig. 4, the threshold setting unit 115 sets, as the judgment threshold 132, a threshold for the similarity between the first reference index 135 and the image features of the candidates stored in the local region candidate 127. This threshold may be determined in advance as a predetermined value. When face recognition is used as the local feature calculation unit 118, a similarity threshold for determining that the images are the same person may be provided, and this may be set as the judgment threshold 132. Furthermore, the threshold setting unit 115 may use test data to examine changes in the similarity of the images of the local region being tracked, determine an appropriate threshold (according to conditions desired by the user), and set this value as the judgment threshold 132. The other processes shown in Fig. 4 are the same as those in the first embodiment, and therefore description thereof will be omitted here.

[0080] In S506 of FIG. 5, the associating unit 113 compares the image feature stored in the first reference index 135 with the image feature corresponding to the candidate selected in S501 from among the image features stored in the judgment index 131, and calculates the similarity. Next, the associating unit 113 uses the calculated similarity as the judgment index 131 and compares it with the judgment threshold 132. At this time, the similarity between the second reference index 136 and the judgment index 131 may also be calculated, and if the similarity with the second reference index is higher than the similarity with the first reference index, the process may proceed to S508. This has the effect of preventing association if the second reference index 136 is an image feature of a region that was not determined to be the desired local region in the previous frame in which the local detection state 121 became 1, and if the second reference index 136 is more similar to the image feature of the region, the process may not be repeated. The other processing steps shown in FIG. 5 are the same as those in the first embodiment, and therefore will not be described here.

[0081] According to this configuration, by using the image features as the determination index 131, it is possible to advantageously perform the association process based on the image similarity of the local site region and the local detection state 121.

[0082] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) a first detection means for detecting a subject from each of a first image and a second image that chronologically follows the first image; a second detection means for detecting a partial area indicating a specific part of the subject from each of the first image and the second image; an acquisition means for acquiring a change in the number of detected partial regions between the first image and the second image; a first control means for controlling whether or not to associate a specific part of the partial region extracted in the second image with the subject, depending on the state of the change; An information processing device comprising: (Item 2) a calculation means for calculating an evaluation value for determining association from the partial region in the second image; and a setting means for setting a threshold value for the evaluation value in accordance with the state of the change. The information processing device described in item 1, characterized in that the first control means controls whether or not to associate a specific part of the partial region extracted in the second image with the subject based on the evaluation value and the threshold value. (Item 3) Item 2. The information processing device according to item 2, characterized in that the first control means performs control so as not to associate specific parts of the partial regions extracted in the second image with the subject when the number of the detected partial regions in the second image is reduced compared to that in the first image and the evaluation value is less than the threshold value. (Item 4) 4. The information processing device according to item 2 or 3, wherein the evaluation value is a score that evaluates the tracking accuracy of the subject in the second image. (Item 5) 5. The information processing device according to any one of items 2 to 4, characterized in that the threshold value is set based on the evaluation value in the partial region having the highest evaluation value calculated by the calculation means in the first image. (Item 6) 6. The information processing device according to item 5, characterized in that the threshold is set further based on the evaluation value in the partial region having the second highest evaluation value calculated by the calculation means in the first image. (Item 7) 7. The information processing device according to item 6, wherein the threshold value is set further based on the number of frames elapsed in the second image relative to the first image. (Item 8) 3. The information processing device according to item 2, wherein the evaluation value is a similarity between a feature of the partial region associated with the subject in the first image and a feature of the partial region in the second image. (Item 9) The information processing device described in any one of items 1 to 8, characterized in that, when the number of detected partial regions in the second image is reduced compared to that in the first image, the first control means suppresses associating specific parts of the partial regions extracted in the second image with the subject. (Item 10) Item 9. The information processing device according to item 9, characterized in that the first control means sets an additional condition for associating specific parts of the partial regions extracted in the second image with the subject when the number of the detected partial regions in the second image is reduced compared to that in the first image. (Item 11) Item 1. The information processing device according to item 1, further comprising a second control means for performing display control of the partial area in the second image in accordance with the association control content by the first control means. (Item 12) Item 11. The information processing device according to item 11, wherein the second control means displays a frame indicating the partial region when the partial region is associated with the subject in the second image, and does not display a frame indicating the partial region when the partial region is not associated with the subject. (Item 13) detecting a subject from each of a first image and a second image that chronologically follows the first image; detecting a partial area representing a specific part of the subject from each of the first image and the second image; acquiring a change in the number of detected partial regions between the first image and the second image; a step of controlling whether or not to associate a specific part of the partial region extracted in the second image with the subject, depending on the state of the change; An information processing method comprising: (Item 14) A program for causing a computer to function as each of the means of the information processing device described in any one of items 1 to 12.

[0083] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0084] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0085] 101: CPU, 102: computer bus, 103: first memory, 104: second memory, 105: input unit, 106: display unit, 107: communication unit

Claims

1. a first detection means for detecting a subject from each of a first image and a second image that chronologically follows the first image; a second detection means for detecting a partial area indicating a specific part of the subject from each of the first image and the second image; an acquisition means for acquiring a change in the number of detected partial regions between the first image and the second image; a first control means for controlling whether or not to associate a specific part of the partial region extracted from the second image with the subject, depending on the state of the change; An information processing device comprising:

2. a calculation means for calculating an evaluation value for determining association from the partial region in the second image; and a setting means for setting a threshold value for the evaluation value in accordance with the state of the change.

2. The information processing device according to claim 1, wherein the first control means controls whether or not to associate a specific part of the partial region extracted in the second image with the subject, based on the evaluation value and the threshold value.

3. 3. The information processing device according to claim 2, characterized in that the first control means performs control so as not to associate specific parts of the partial regions extracted in the second image with the subject when the number of detected partial regions in the second image is reduced compared to that in the first image and the evaluation value is less than the threshold value.

4. The information processing device according to claim 2 , wherein the evaluation value is a score that evaluates the tracking accuracy of the subject in the second image.

5. 3. The information processing apparatus according to claim 2, wherein the threshold value is set based on the evaluation value calculated by the calculation means in the first image for the partial region having the highest evaluation value.

6. 6. The information processing device according to claim 5, wherein the threshold value is set further based on the evaluation value of the partial region having the second highest evaluation value, calculated by the calculation means in the first image.

7. The information processing apparatus according to claim 6 , wherein the threshold value is set further based on the number of frames elapsed in the second image relative to the first image.

8. 3. The information processing apparatus according to claim 2, wherein the evaluation value is a similarity between a feature amount of the partial region associated with the subject in the first image and a feature amount of a partial region in the second image.

9. 2. The information processing device according to claim 1, characterized in that the first control means suppresses associating specific parts of the partial regions extracted in the second image with the subject when the number of the partial regions detected in the second image is reduced compared to that in the first image.

10. 10. The information processing device according to claim 9, wherein the first control means sets an additional condition for associating specific parts of the partial regions extracted in the second image with the subject when the number of the partial regions detected in the second image is reduced compared to that in the first image.

11. The information processing apparatus according to claim 1 , further comprising: second control means for performing display control of the partial area in the second image in accordance with the content of association control by the first control means.

12. 12. The information processing device according to claim 11, wherein the second control means displays a frame indicating the partial region when the partial region is associated with the subject in the second image, and does not display a frame indicating the partial region when the partial region is not associated with the subject.

13. detecting a subject from each of a first image and a second image that chronologically follows the first image; detecting a partial area representing a specific part of the subject from each of the first image and the second image; acquiring a change in the number of detected partial regions between the first image and the second image; a step of controlling whether or not to associate a specific part of the partial region extracted in the second image with the subject, depending on the state of the change; An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images