Information processing device, information processing method, and program

Abstract

To reduce an error in association of a specific part to a subject.SOLUTION: A subject is detected from a first image and a second image following the first image in time series. Partial regions showing a specific part associated with the subject are detected from the first image and second image, respectively. A state of change in number of detected partial regions between the first image and second image is acquired. According to the state of change, it is controlled whether to associate the specific part extracted from the second image with the subject.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Detection and tracking of a region of a specific subject from consecutive images have been performed. Tracking means detecting a region of a desired subject from an image and tracking the same subject region between consecutive images. Based on the result of tracking, autofocus processing of a camera that is taking an image is performed.

[0003] Patent Document 1 discloses a method of tracking while associating the whole of a subject to be tracked with its parts. The whole and parts of the subject are, for example, when a person is the subject, the whole body of the human body is the whole, and the face part is the part. In Patent Document 1, the association between the whole and the parts is performed based on the positional relationship (for example, proximity of distance) between the body part and the part of the subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In performing the association between the whole and the parts, the method based on the positional relationship between the whole and the parts has a problem that the association between the whole and the parts is likely to fail when similar objects cross at the same position on the image. For example, in the technique described in Patent Document 1, when the head of another person crosses the head of the person being tracked, an incorrect association is made.

[0006] The present invention aims to reduce errors in associating specific body parts with a subject. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an information processing apparatus according to one embodiment comprises the following configuration: a first image and a second image that follows the first image in chronological order. Each from , corresponding to the target being tracked subject region A first detection means for detecting the first image and the second image Each mosquito Ra, special A second detection means for detecting a partial region indicating a specific part, and the first image The number of the first subregions detected in and 、 The second image In Detected Second Number of the aforementioned subregions and, A means of obtaining, The system includes an association means for associating the subregion with the subject region, wherein the association means, when the number of the second subregions is less than the number of the first subregions, In the second image above detection The partial region As shown A specific part of the subject region Associated with do not have . [Effects of the Invention]

[0008] Reduce errors in associating specific body parts with a subject. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example of the configuration of the information processing device according to Embodiment 1. [Figure 2] A flowchart illustrating an example of the overall processing performed by an information processing device. [Figure 3] A flowchart showing an example of processing by the state acquisition unit. [Figure 4] A flowchart illustrating an example of processing by the threshold setting unit. [Figure 5] A flowchart illustrating an example of processing performed by the association unit. [Figure 6] A flowchart illustrating an example of processing by the display control unit. [Figure 7] A diagram for explaining a state in which the number of local detections changes.

Embodiments of the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] [Embodiment 1] Hereinafter, an information processing apparatus according to Embodiment 1 will be described. The information processing apparatus according to the present embodiment takes as input a series of consecutive images, tracks the subject detected from each image, tracks the entire subject to be tracked, and can detect a specific part related to the subject and associate it with the subject. Hereinafter, when simply referred to as a "specific part", it refers to a specific part detected by the information processing apparatus according to the present embodiment related to the subject to be processed. Further, hereinafter, when referring to such a specific part, it may be referred to as a "local" or "local part".

[0012] In this embodiment, an example will be described in which a human body (the whole body) is used as a subject to be processed, and the head of the human body is used as a specific part related to the subject. However, it is not particularly limited to this as long as the same processing can be executed. For example, as the subject and its specific part, the head and pupils of the human body may be used, the animal (the whole body) and the face of the animal may be used, or a vehicle and a license plate may be used. Further, the specific part does not necessarily have to be a part of the subject as long as it is an object that is assumed to be imaged along with the subject. For example, as the subject and its specific part, a vehicle such as a car or an animal and the head of a person riding on the vehicle may be used. Although the head of a person is not a part of the vehicle, since it moves along with the vehicle that is performing the follow-up, it can be regarded as a specific part related to the subject.

[0013] FIG. 1 is a diagram showing an example of the configuration of an information processing apparatus 100 according to this embodiment. As an example of the hardware configuration, the information processing apparatus 100 includes a CPU 101, a computer bus 102, a first memory 103, a second memory 104, an input unit 105, a display unit 106, and a communication unit 107. The CPU 101 controls the entire information processing apparatus 100. The first memory 103 and the second memory 104 are memories that store a control program for executing the processing according to this embodiment and various types of data. In FIG. 1, the first memory 103 is mainly shown as storing the control program, and the second memory 104 is mainly shown as storing various types of data. However, it is not particularly limited to this as long as the same data can be stored in the entire information processing apparatus 100.

[0014] The input unit 105 is composed of a keyboard or a touch panel, etc., and receives input from the user. The display unit 106 is composed of a display device such as a liquid crystal display, and can display the processing result to the user. The communication unit 107 can communicate with an external device to exchange data. The computer bus 102 connects each functional unit of the information processing apparatus 100. The information processing apparatus 100 according to this embodiment may be implemented as a computer including a program for executing each processing described below, for example.

[0015] In this embodiment, the first memory 103 stores programs for executing each process described as being performed by the information processing device 100. The local feature calculation unit 118 shown in Figure 1 will be described in Embodiment 2.

[0016] The tracking unit 110 tracks the subject in the image. The tracking unit 110 can perform subject tracking using any commonly used image subject tracking technique, for example, by template matching or by a machine learning model that has been pre-trained to track the subject in the image. In this embodiment, the tracking unit 110 will be described below assuming that it detects and tracks the subject by template matching.

[0017] The local detection unit 111 detects a partial region (local region) that indicates a specific part from an image. In this embodiment, the local detection unit 111 detects local regions from both the first image and the second image that chronologically follows the first image. Detection of a specific part by the local detection unit 111 can be performed by general object detection processing in images. Here, the local detection unit 111 performs local region detection using a machine learning model that has been pre-trained to detect a specific part in an image. In the following, the first image may be referred to as the previous frame (image), and the second image as the current frame (image).

[0018] The state acquisition unit 112 acquires the state of change in the number of local regions detected by the local detection unit 111 between the first image and the second image. For example, the state acquisition unit 112 acquires information on whether the number of detected local regions between the first image and the second image is increasing, decreasing, or unchanged. Hereinafter, such a state of change in the number of local regions between the first image and the second image may simply be referred to as the "change state". Specific examples of processing by the state acquisition unit 112 will be described later.

[0019] The association unit 113 associates the local region (specific area) detected by the local detection unit 111 with the subject being tracked. Multiple local regions may be detected in the image, but the association unit selects the most suitable local region based on predetermined conditions and associates it with the subject being tracked. Hereinafter, this process of "associating a local region with a subject" refers to the same process as associating a specific area corresponding to such a local region with the subject. In this embodiment, the association unit 113's process of associating a local region with the subject is controlled according to the change state acquired by the state acquisition unit 112. In particular, the threshold set by the threshold setting unit 115, which will be described later, is controlled according to the change state, and the association unit 113 uses the threshold controlled in this way to associate the local region with the subject.

[0020] The association unit 113 may, for example, suppress the association between the local area detected in the second image and the subject if the detected local area in the second image is smaller than that in the first image. As will be described in more detail with reference to Figure 5, if the detected local area decreases over time, it is considered that the local area is being obscured. From this perspective, this type of association unit processing makes it possible to suppress the association when there is a high possibility of incorrect association. Therefore, it is possible to reduce the negative impression on the user caused by incorrect association (for example, by displaying a bounding box on the wrong object) or to reduce the negative impact of incorrect association on subsequent processes such as tracking. Details of the processing by the association unit 113 will be described later with reference to Figure 5.

[0021] The term "suppressing association" is not particularly limited to any process that makes it more difficult to associate local regions with a subject when the number of detected local regions in the second image is less than that in the first image. For example, the association unit 113 may associate local regions with a subject without using the judgment threshold 132 described later if the number of detected local regions in the second image is the same as or greater than that in the first image, and may set additional conditions for association (for example, using the judgment threshold 132 described later) if the number of detected local regions in the second image is less than that in the first image. Alternatively, the process of suppressing association may be to raise the threshold used for association, or to add a display indicating that association is being suppressed.

[0022] When using conventional techniques to associate the entire subject with specific parts based solely on their positional relationship, incorrect associations can occur even if the specific part does not correspond to the subject, as long as its position matches. In scenes where people cross paths, specific parts of two people (e.g., heads) may overlap positionally, resulting in the appearance of specific parts that should not be associated with the same location. Figure 7 illustrates a situation where the number of local detections decreases due to the overlapping of specific parts of a subject. Figure 7(A) illustrates a state where the number of local detections is 2 for the input image 701. The overall tracking frame 702 is a frame representing the overall tracking area 124, which indicates the subject being tracked. Local detection frames 703 and 704 are frame displays of two local areas that have been detected as local part areas and stored in the local area candidate 127, respectively. Figure 7(B) shows the input image 705, which is the next frame after Figure 7(A). In the input image 705, the number of local detections has decreased from 2 in the previous frame to 1 due to the overlapping of people. The local detection frame 707 in Figure 7(B) is the region where a local area was detected in the frame of Figure 7(B). The overall tracking frame 706 is the same as in Figure 7(A) and represents the overall tracking region 124 that indicates the subject to be tracked. In the example in Figure 7(B), another person's local area overlaps with the position of a local area associated with the original subject to be tracked. In such cases, conventional methods have the problem of not being able to correctly associate the subject with a specific area. On the other hand, by controlling whether or not to associate a specific area with the subject according to the change in the number of detected sub-regions, it is possible to reduce such association errors.

[0023] The judgment index setting unit 114 sets the judgment index. The judgment index according to this embodiment is an evaluation value used for evaluation when associating a subject with a local area in an image. In this embodiment, the first tracking score, which will be described later, is used as the judgment index.

[0024] The threshold setting unit 115 sets a threshold for the judgment index 131. In this embodiment, the threshold setting unit 115 may, for example, set a threshold (as the judgment threshold 132) based on equation (1) described later, if the detected local area is decreasing based on the change state. The processing by the threshold setting unit 115 will be described later with reference to Figure 4.

[0025] Figure 2 is a flowchart showing an example of the overall information processing performed by the information processing device 100 according to Embodiment 1. In S201, the information processing device 100 performs initial setup of data related to various processes. For example, the tracking unit 110 registers a template of the subject to be tracked in the template 120 in the memory 104 as an initial setup. This process can be performed, for example, by accepting a selection of a subject on the screen by the user. In this example, the subject is a person, and a full-body image of the person is registered as the template to be tracked.

[0026] For example, the state acquisition unit 112 performs initial setup of the memory for storing the state of local detection (detection of local regions) as an initial setting. In this embodiment, the state acquisition unit 112 manages the state of change in the number of detected local regions by an ID (for example, represented by 0 to 1), and the initial value of the ID can be set to 0.

[0027] For example, the local detection unit 111 initializes memory for storing the number of local regions detected. Here, the local detection unit 111 stores the number of local regions detected from the current frame in memory 104 as the number of local regions of the current frame 129, and the number of local regions detected from the previous frame in memory 104 as the number of local regions of the previous frame 130, and sets the initial values ​​of each to 0.

[0028] In S202, the tracking unit 110 performs subject tracking on the image to be processed. Here, the tracking unit 110 acquires a single frame image to be processed, searches the image for regions similar to the template 120, and outputs a tracking region and a tracking score. The tracking score according to this embodiment is an evaluation value (a numerical value representing the reliability of the tracking result) that evaluates the tracking accuracy, and the higher the value, the more likely the tracking result is. Here, the tracking unit 110 calculates multiple candidate tracking regions on the image, and the one with the highest tracking score among them is selected as the subject's tracking region (tracking result). The tracking score for a candidate can be calculated, for example, by the degree of agreement with the tracking region in the previous frame, or by the image similarity between the tracking region and the template. Here, the tracking unit 110 stores the scores of the first-ranked candidate and the second-ranked candidate in memory 104 as the first-ranked tracking score 122 and the second-ranked tracking score 123, respectively, and stores the tracking area of ​​the first-ranked candidate in the overall tracking area 124 in memory 104. In this embodiment, each area including the tracking area is represented by the position and size of the bounding box (rectangular area) on the image. Before performing the above tracking process and storing the first-ranked tracking score 122 and the second-ranked tracking score 123, the tracking unit 110 stores the first-ranked tracking score and the second-ranked tracking score from the previous frame. The first-ranked tracking score and the second-ranked tracking score from the previous frame are stored in memory 104 as the previous frame's first-ranked tracking score 125 and the previous frame's second-ranked tracking score 126.

[0029] In S203, the local detection unit 111 detects local areas from the image. Here, the local detection unit 111 detects the region of local areas associated with the subject to be tracked from the input image and outputs the local area and the local area detection score. In this case, since the head of the human body is detected as a local area, a rectangular region surrounding the head of the human body is output as the local area. The local area detection score is a numerical value that represents the reliability of the detection result, and the higher the number, the more likely the detection result is. Note that, for example, if a region is detected as a local area when the local area detection score exceeds a predetermined threshold (which can be set arbitrarily), there may be multiple local areas detected from a single image. In the example in Figure 1, the local detection unit 111 stores local area candidates 127 and local area candidate detection scores 128 in the memory 104. The local area candidates 127 and local area candidate detection scores 128 are assumed to be arrays that store multiple detection results. The number of local areas detected in the current frame is stored in the memory 104 as the current frame local area detection count 129.

[0030] In S204, the judgment index setting unit 114 stores the judgment index 131 in the memory 104. The judgment index is used in the association processing by the association unit 113, which will be described later. In this embodiment, the judgment index setting unit 114 uses the first tracking score 122 as the judgment index 131. As will be described in detail in Embodiment 2, a judgment index other than the tracking score may be used as the judgment index 131.

[0031] In S205, the state acquisition unit 112 acquires the change state of the frame to be processed from the previous frame. In this embodiment, the state acquisition unit 112 stores an ID (numerical value) that identifies the change state as a local detection state 121 in the memory 104. Here, the state acquisition unit 112 sets the local detection state 121 to "1" if the number of local regions detected in the current frame has decreased compared to the previous frame, and sets the local detection state 121 to "0" if the number of detected local regions has increased. Details of the processing performed in S205 will be described later with reference to Figure 3.

[0032] In S206, the threshold setting unit 115 sets a threshold for the judgment index 131. Here, the threshold setting unit 115 stores the judgment threshold 132 as the threshold in the memory 104. Details of the processing by the threshold setting unit 115 will be described later.

[0033] In S207, the association unit 113 associates the subject with a local region. Here, the association unit 113 can select one of several candidates included in the local region candidate 127 and associate it with the subject. Information indicating the local region associated with the subject is stored in the memory 104 as a local region area 133. If there is no local region to associate with the subject, information indicating that there is no associated local region is stored in the local region area 133. The association unit 113 also stores the reason for the association as a corresponding value in the association status 137. Details of these processes by the association unit 113 will be described later.

[0034] In S208, the display control unit 117 displays the tracking results on the display unit 106. For example, the display control unit 117 can display frames representing the overall tracking area 124 and the local area area 133 on the input image in different colors. If the local area area 133 is empty (no associated local area exists), the frame corresponding to the local area area 133 may not be displayed. Furthermore, if a candidate local area is near the tracking subject (for example, within a predetermined range centered on the subject), but the association unit 113 does not associate the candidate with the subject based on the determination result of the state acquisition unit 112, the display control unit 117 may display the area frame corresponding to the candidate in a different color than when an association is determined. Details of the processing by the display control unit 117 will be described later.

[0035] In S209, the information processing device 100 determines whether to terminate the tracking process. If tracking is to continue, the process returns to S202; otherwise, the process shown in Figure 2 terminates. The conditions for terminating the tracking process can be arbitrarily set. For example, the information processing device 100 may terminate the tracking process if the first tracking score 122 falls below a predetermined threshold, indicating that the tracking target has been lost. Alternatively, for example, assuming use in the autofocus function of a camera, the information processing device 100 may determine the start and end of tracking depending on whether the user has performed a predetermined operation, such as half-pressing or not pressing the shutter button.

[0036] Next, we will explain the details of the processing performed by the state acquisition unit 112. Figure 3 is a flowchart of an example of the processing in S205. In S301, the state acquisition unit 112 branches the processing according to the current local detection state 121. Here, if the local detection state is 0, the state acquisition unit 112 proceeds to S302, and if it is 1, it proceeds to S304.

[0037] In S302, the state acquisition unit 112 determines whether the number of local region detections in the current frame (local detection count) has decreased compared to the previous frame. The state acquisition unit 112 can make this determination by comparing the number of local region detections in the current frame (129) with the number of local region detections in the previous frame (130) in the memory 104. If the number of local detections has decreased from the previous frame, the process proceeds to S303. If the number of local detections is the same as or has increased compared to the previous frame, the process in Figure 3 ends.

[0038] In S303, the state acquisition unit 112 sets the local detection state 121 in memory 104 to 1. If the number of local detections has decreased compared to the previous frame, it suggests that the decrease may be due to the overlapping positions of multiple local regions. In this state, there is a possibility that local parts associated with the subject being tracked are hidden, so the threshold used by the association unit 113 is controlled. Also in S303, the state acquisition unit 112 sets up an elapsed frame count 134 in memory 104 to count the number of elapsed frames since the local detection state was changed, and initializes it to 0. At this point, the state acquisition unit 112 stores the previous frame's first-rank tracking score 125 in memory 104 as the first reference index 135, and the previous frame's second-rank tracking score 126 as the second reference index 136. The first reference index 135 is used to record what the tracking score was in the state immediately before the number of local detections decreased. In this embodiment, the judgment indicator setting unit 114 uses the tracking score as a judgment indicator for determining whether or not to perform an association, so the tracking score is stored here as a reference indicator. If an evaluation value other than the tracking score is used as the judgment indicator 131, the judgment indicator used by the judgment indicator setting unit 114 is stored in the first reference indicator 135.

[0039] In S304, if the local detection state 121 is set to 1 in S301, the state acquisition unit 112 adds 1 to the elapsed frame count 134 in the memory 104.

[0040] In S305, the state acquisition unit 112 determines whether the number of local detections has increased. This determination can be made by comparing the number of local area detections in the current frame (129) with the number of local area detections in the previous frame (130) in the memory 104. In S305, if the number of local detections is the same as or less than that of the previous frame, the process proceeds to S306; if the number of local detections has increased from the previous frame, the process proceeds to S307.

[0041] In S306, it is determined whether the number of elapsed frames 134 in memory 104 exceeds the maximum number of elapsed frames (predetermined number of frames) which is set separately. If the number of elapsed frames 134 exceeds the predetermined number of frames, the process proceeds to S307; otherwise, the process shown in Figure 3 is terminated.

[0042] In S307, the state acquisition unit 112 sets the local detection state 121 in the memory 104 to 0. Therefore, if the number of local detections increases, or if the number of elapsed frames 134 exceeds a predetermined maximum value, the local detection state 121 is set to 0 in S307. If the number of local detections decreases and then increases again, it suggests that the desired local area may have reappeared after being hidden by another object. Also, if the number of elapsed frames exceeds a predetermined maximum value, it suggests that the expectation of the desired local area reappearing may be decreasing. From this perspective, in either of the above cases, the local detection state 121 is set to its initial value of 0 in S307. In the processing by the association unit 113, which will be described later, the association processing between the subject and the local area is performed based on the local detection state 121 set by the state acquisition unit 112 as described above. The state acquisition unit 112 sets a local detection state based on an increase or decrease in the number of detected local regions, and the association unit 113 associates the subject with the local regions based on such a local detection state. This makes it possible to appropriately control whether or not to associate a specific part with the subject according to the change in the number of detected local regions. Therefore, it is possible to reduce the error of associating the wrong specific part with the subject.

[0043] Furthermore, a decrease in the number of local detections can occur for reasons other than the obscuration of a local area of ​​the subject being tracked, such as when the head of a subject that is not being tracked goes out of frame. In such cases, it is possible that even simple conventional techniques may not misinterpret the association. From this perspective, although Figure 3 explains the case where the local detection state is set to 1 when the number of detected local areas decreases, the local detection state may be set to 1 when the number of local areas becomes 1. With such processing, only when the number of local detections decreases from multiple to 1, it becomes possible to carefully confirm whether the detected local area is associated with the subject being tracked using the judgment index 131, thereby simplifying the overall processing.

[0044] Next, the processing by the threshold setting unit 115 will be described. The threshold setting unit 115 sets a threshold (judgment threshold 132) for the judgment index 131 according to the state of change in the number of local regions. In this embodiment, since the tracking score is used as the judgment index 131, the judgment threshold 132 becomes the threshold for the tracking score.

[0045] Figure 4 is a flowchart showing an example of the processing in S206. In S401, the threshold setting unit 115 determines whether the local detection state 121 is 0. If the local detection state 121 is 0, the judgment threshold 132 is not set as the threshold (the initial value is used as the threshold) and the process ends; otherwise, the process proceeds to S402. Here, if the local detection state 121 is 0, the judgment threshold 132 is not used and does not need to be set.

[0046] In S402, the threshold setting unit 115 sets a judgment threshold 132 based on the first reference index 135 in the memory 104 and the number of elapsed frames 134. The threshold setting unit 115 can calculate the judgment threshold 132 based on, for example, the following formula (1). Th = S1 × α f Formula (1)

[0047] Here, Th is the judgment threshold of 132, S1 is the first reference index of 135, and f is the number of elapsed frames of 134. α is a predetermined coefficient between 0 and 1. If α is 1, the first reference index of 135 is always the judgment threshold of 132, regardless of the number of elapsed frames of 134. On the other hand, if α is less than 1, the judgment threshold of 132 decreases as the number of elapsed frames increases.

[0048] The first reference index 135 stores the tracking score when the local detection state 121 is 0, that is, when local parts associated with the subject being tracked are not hidden. Here, the tracking score may decrease as time passes after the local detection state 121 becomes 1, due to changes in the posture of the subject being tracked, etc., so the judgment threshold is lowered as the number of elapsed frames increases.

[0049] Here, we have explained that the judgment threshold 132 decreases according to the number of elapsed frames (when α is less than 1), but the method of setting the judgment threshold 132 by the threshold setting unit 115 is not limited to this. For example, the judgment threshold 132 may be set by multiplying the first reference index 135 by a predetermined coefficient and maintaining that value regardless of the number of elapsed frames. Alternatively, a judgment threshold for when the local detection state 121 is 0 may be set in advance and that value may be maintained at all times.

[0050] The threshold setting unit 115 may also use the second reference index 136 to calculate the judgment threshold 132 based on the following equation (2). Here, S2 is the second reference index 136. Th = max(S1) × α f ,S2)

[0051] The second reference index 136 stores the tracking score for the second-ranked tracking candidate in the frame prior to the local detection state 121 becoming 1, i.e., the tracking score for the tracking candidate that was not considered a target for tracking. Using Equation 2 has the effect of setting the judgment threshold 132 so that it does not fall below the tracking score of a tracking candidate that was determined not to be a target for tracking in at least the previous frame.

[0052] Furthermore, although the coefficient α is assumed to be a fixed value here, the coefficient α may be set based on, for example, a first reference index 135 and a second reference index 136. For example, the threshold setting unit 115 may set the judgment threshold 132 to the same value as the second reference index when the number of elapsed frames reaches a predetermined value (N). For example, when the threshold setting unit 115 calculates the judgment threshold 132 based on equation (2), it may use α calculated based on the following equation (3). α=10^((log(S1) / S2)) / N) Equation (3)

[0053] Equation (3) is S1 × α N This is the equation obtained by solving S2 for α. By setting α in this way, after the local detection state 121 becomes 1, the judgment threshold 132 decreases from the first reference index 135 to the second reference index 136 in N frames, and thereafter the judgment threshold is fixed at the second reference index. This has the effect of preventing the judgment threshold from becoming lower than the tracking score of tracking candidates that were not previously judged as tracking targets.

[0054] Next, the processing by the association unit 113 will be explained. The area of ​​the subject to be tracked is stored in the memory 104 as the overall tracking area 124, and zero or more candidate areas for local parts are stored as local area candidates 127. The association unit 113 selects a local area from the candidates included in the local area candidates 127 that is associated with a local part attached to the subject to be tracked, and stores it in the local part area 133 in the memory 104. The association unit 113 also stores a numerical value (which will be shown later, but here it will be shown in four stages from 0 to 3) representing the reason for determining the association as the association status 137 in the memory 104.

[0055] Figure 5 is a flowchart showing an example of the processing in S207. In S501, the association unit 113 selects a local region from among the local region candidates. The processing in S501 is the process of selecting a local region from the candidates included in the local region candidates 127 that is associated with a local part attached to the subject being tracked, and can be performed using any known technique for selecting from among candidate regions. For example, the association unit 113 may select the local region candidate that is closest in distance on the image to the overall tracking region 124, or it may select one based on its positional relationship with the overall tracking region 124 (for example, in which direction it is located: up, down, left, or right). Furthermore, if the local part is the head of a human body, taking into account that the head is usually positioned above the center of the human body in the image, the association unit 113 may add the condition that it is above the overall tracking region 124 in the image to the selection conditions in S501. Furthermore, from the perspective that the position of a local area does not move significantly from the previous frame when the frame rate is relatively fast, the association unit 113 may select a local area candidate in S501 that is close in distance to the local area region 133 in the previous frame. Alternatively, a calculation unit (not shown) may calculate the depth distance from the imager to the subject, and the local area that minimizes the difference between the depth distance to the overall tracking area and the depth distance to the local area may be selected in S501.

[0056] In S502, the association unit 113 determines whether or not the candidate selected in S501 exists. If no candidate is selected, the process proceeds to S503; if a candidate is selected, the process proceeds to S504.

[0057] In S503, the association unit 113 does not associate a local area with the subject and terminates the process shown in Figure 5. In this example, the association unit 113 sets the association state 137 in memory 104 to 0. Here, an association state of 137 of 0 means that no local area associated with the subject being tracked was detected, and thus it is determined that there is no association. The processing according to the association state will be described later with reference to Figure 6.

[0058] In S504, the association unit 113 determines whether the local detection state 121 in memory 104 is 0. If it is 0, the process proceeds to S505; otherwise, the process proceeds to S506. In S505, the association unit 113 associates the local region selected in S501 with the subject and stores information indicating that there has been no decrease in the detected local region, thus terminating the process shown in Figure 5. Here, the association unit 113 stores information indicating that the local region selected in S501 is associated with the subject by setting the association state 137 to 1, and information indicating that there has been no decrease in the detected local region. Setting the association state 137 to 1 (or 2, as described later) may also be information indicating that the local region corresponding to the selection in S501 is associated with the subject, or information indicating such a local region may be stored separately, and information indicating that such a local region is associated with the subject may be stored separately. In this case, if association status 137 is 1, there is a local area associated with the subject being tracked, and since the local detection status is 0, there has been no decrease in the number of local detections, which means that there is a high probability that the local area that should be associated with the subject being tracked is not hidden in the current frame.

[0059] In S506, the association unit 113 determines whether the judgment index 131 in the memory 104 exceeds the judgment threshold 132. If the judgment index 131 exceeds the judgment threshold 132, the process proceeds to S507; otherwise, the process proceeds to S508.

[0060] In S507, the association unit 113 associates the local region selected in S501 with the subject and stores information indicating that a decrease in the detected local region has occurred, thus ending the process shown in Figure 5. Here, the association unit 113 sets the association state 137 to 2, thereby storing information indicating that the local region selected in S501 is associated with the subject, and information indicating that a decrease in the detected local region has occurred. Here, when the association state 137 is 2, a decrease in the number of local detections is observed, meaning that the local part that should be associated with the subject being tracked may be hidden in the current frame, but since the judgment index is above the threshold, it is highly likely that it is not hidden. Note that in S507, the local detection state 121 may also be set to its initial value of 0. This is because the desired local part is not hidden, and it is judged that this state can be considered the initial state. If it is thought that confirmation by the judgment index is necessary in subsequent frames as well, the local detection state 121 may be left at 1.

[0061] In S508, the association unit 113 stores information indicating that the local region selected in S501 is not to be associated with the subject, and terminates the processing shown in Figure 5. Here, the association unit 113 stores information indicating that the local region selected in S501 is not to be associated with the subject by setting the association state 137 to 3. Here, an association state 137 of 3 means that a decrease in the number of local detections has been observed, there is a possibility that the local part that should be associated with the subject being tracked is hidden in the current frame, and since the judgment index is below the threshold, it is highly likely that it is hidden.

[0062] Furthermore, when the association status 137 is set to 3, the local parts associated with the subject being tracked using the conventional method are unreliable. Therefore, when the association status 137 is 3, the association unit 113 determines that there are no local parts to associate and does not perform the process of associating the entire subject with the local parts. For example, when autofocus processing is performed to focus on a local part attached to the subject being tracked, if the association status 137 is 3, the system determines that the focus target has been lost.

[0063] The information processing device 100 may also perform autofocus processing based on the reason for judgment recorded in the association state 137 (pre-set to correspond to the value of the association information 137). For example, various processes such as restarting from the initial tracking settings or waiting for several frames for the desired local area to reappear while maintaining the focus position may be pre-associated with the association state 137, and such processing may be performed according to the association state 137. Furthermore, as will be described later, when displaying local areas associated with the subject being tracked, processing such as stopping the display of the local area region or changing the display representation may be performed based on the association state 137.

[0064] Furthermore, in the description of the association unit 113 in this embodiment, it was explained that the threshold setting unit 115 sets the judgment threshold 132 in S206 of Figure 2, and a comparison judgment is performed with the judgment index 131 in S506 of Figure 5. However, these processes may be omitted. In that case, the process can always proceed to S508 without performing a comparison judgment in S506. In this case, if a decrease in the number of local detections is observed, the association state 137 will always be set to 3, and it will be determined that there is no association.

[0065] Next, the processing by the display control unit 117 will be described. The display control unit 117 displays the tracking results on the display unit 106. In this embodiment, the display control unit 117 can display local areas associated with the subject in a manner based on the association state 137. For example, the display control unit 117 may, as one example, display a frame representing the overall tracking area 124 and a frame representing the local area 133 on the input image, each in a different color.

[0066] The following describes an example of changing the display mode of such a local area according to the association state 137. Note that this process is just one example, and the display mode is not limited to this. Figure 6 is a flowchart of an example of the process in S208. In S601, the display control unit 117 copies the input image to the display image 138. In S602, the display control unit 117 determines whether the contents of the overall tracking area 124 are empty (whether there is a local area stored as the overall tracking area 124). If the overall tracking area 124 is empty, the process proceeds to S609, and the input image is displayed as is as the display image 138, and the process ends. If the overall tracking area 124 is not empty, the process proceeds to S603.

[0067] In S603, the display control unit 117 controls the drawing of the frame corresponding to the overall tracking area 124 in memory 104 on the display image 138 in red, and proceeds to S609. The overall tracking area 124 is a bounding box representing the area of ​​the subject being tracked. In S604, the display control unit 117 branches the processing according to the value stored in the association state 137. Here, if the association state 137 is 0, the processing proceeds to S605; if it is 1, the processing proceeds to S606; if it is 2, the processing proceeds to S607; and if it is 3, the processing proceeds to S608.

[0068] In S605, the display control unit 117 controls the display image 138 to superimpose information indicating that there were no local parts associated with the subject being tracked, and proceeds to S609. Here, as information indicating that there were no local parts associated with the subject being tracked, text indicating that fact may be superimposed on the display image, or control may be performed so that no additional superimposition is performed.

[0069] In S606, the display control unit 117 controls the frame corresponding to the local area region 133 to be superimposed in green on the display image 138, and proceeds to S609. This indicates that the local areas have been associated using the conventional association method, as the number of detected local areas has not decreased in the current frame.

[0070] In S607, the cover control unit 117 controls the drawing of the frame corresponding to the local area region 133 in yellow on the display image 138, and proceeds to S609. This indicates that although the number of detected local areas has decreased and the desired local area may be hidden, the tracking score, which is a judgment indicator, is above the judgment threshold, so it is determined that the subject being tracked is not hidden and the association of local areas is valid. In other words, it is determined that the decrease in the number of local areas in the current frame is due to the head of a person other than the subject being hidden.

[0071] In S608, the cover control unit 117 controls the drawing of a frame corresponding to the local area region 133 on the display image 138 with a gray dashed line, and proceeds to S609. This indicates that the number of detected local areas has decreased in the current frame, and the tracking score, which is a judgment indicator, is below the judgment threshold, so it has been determined that the association of local areas made by the conventional association method is invalid. In this case, the display of the local area region may be stopped.

[0072] In S609, the cover control unit 117 displays the display image 138 on the display unit 106 and terminates processing. Here, the cover control unit 117 displays the display image 138 in a manner controlled in S602 and in any of S605 to S608. In Figure 6, the explanation describes each frame being displayed in a different color, but this is just one example of how each frame can be displayed in a different manner, and is not particularly limited to this. For example, the type of frame line may be differentiated by a straight line, dashed line, or dotted line, or differentiation may be achieved by applying hatching within the frame.

[0073] This processing makes it possible to appropriately control whether or not to associate a specific part with the subject, depending on the state of change in the number of detected local areas. Therefore, it is possible to reduce the error of associating the wrong specific part with the subject. In addition, the overall tracking area 124, the local area area 133, or the association state 137 may be transmitted to an external device (for example, to an external device that operates based on the tracking results of the information processing device 100) via the communication unit 107 in Figure 1.

[0074] [Embodiment 2] In Embodiment 1, the tracking score was used as the judgment index 131. In Embodiment 2, the similarity of image features of local regions is used as the judgment index 131. In particular, the similarity between the feature quantities of the local region associated with the subject in the first image and the feature quantities of the local region in the second image is used.

[0075] The local feature calculation unit 118 in this embodiment can calculate image features of two or more given image regions and calculate their similarity. Methods for calculating the similarity of two images using image features are widely known, and a detailed explanation of such processes is omitted here. For example, the local feature calculation unit 118 can calculate the similarity used in methods for performing personal authentication using facial images by calculating the similarity of facial images. In this embodiment, some of the processes described as being performed in Embodiment 1 are modified.

[0076] The information processing device 100 according to this embodiment is basically capable of performing the same processing as that of Embodiment 1. In the following, we will describe the processing performed by the information processing device 100 according to Embodiment 2, which differs from that of Embodiment 1, and will omit redundant explanations.

[0077] In S204 of this embodiment, the judgment index setting unit 114 stores the image features of each candidate stored in the local area candidate 117 as the judgment index 131. Other processes shown in Figure 2 are the same as in Embodiment 1, so their explanation is omitted here.

[0078] In S303 of this embodiment, the state acquisition unit 112 calculates the image features of the local area region in the previous frame as a reference index and stores it as the first reference index 135. The state acquisition unit 112 may also store candidate image features that were not determined to be local areas associated with the subject being tracked in the previous frame as the second reference index 136. Other processes shown in Figure 3 are the same as in Embodiment 1 and are therefore not described here.

[0079] In S402 of Figure 4, the threshold setting unit 115 sets a threshold for the similarity between the first reference index 135 and the candidate image features stored in the local region candidate 127 as the judgment threshold 132. This threshold may be predetermined as a fixed value. When face recognition is used as the local feature calculation unit 118, a similarity threshold for determining that they are the same person may be provided, and this can be set as the judgment threshold 132. Alternatively, the threshold setting unit 115 may examine the change in the similarity of the images of the local region being tracked using test data, determine an appropriate threshold (according to user-desired conditions), and set that value as the judgment threshold 132. Other processes shown in Figure 4 are the same as in Embodiment 1, so their explanation is omitted here.

[0080] In S506 of Figure 5, the association unit 113 compares the image features stored in the first reference index 135 with the image features corresponding to the candidates selected in S501 from among the image features stored in the judgment index 131, and calculates the similarity. Next, the association unit 113 uses the calculated similarity as the judgment index 131 and compares it with the judgment threshold 132. At this time, the similarity between the second reference index 136 and the judgment index 131 may also be calculated, and if the similarity with the second reference index is higher than the similarity with the first reference index, the process may proceed to S508. This has the effect of preventing association if the second reference index 136 contains image features of a region that was not determined to be the desired local area in the previous frame when the local detection state 121 became 1, and the image features are more similar to those in that region. Other processes shown in Figure 5 are the same as in Embodiment 1, so their explanation is omitted here.

[0081] This configuration has the effect of allowing association processing to be suitably performed based on the similarity of images of local area regions and the local detection state 121 by using image features as the judgment index 131.

[0082] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A first detection means for detecting a subject from a first image and a second image that chronologically follows the first image, A second detection means for detecting a partial region indicating a specific part of the subject from the first image and the second image, respectively, Acquisition means for acquiring the state of change in the number of detected subregions between the first image and the second image, A first control means that controls whether or not to associate a specific part of the partial region extracted in the second image with the subject, depending on the state of the change, An information processing device equipped with the following features. (Item 2) A calculation means for calculating an evaluation value for determining association from the partial region in the second image, The system further includes a setting means for setting a threshold value for the evaluation value according to the state of change, The information processing apparatus according to item 1, characterized in that the first control means controls whether or not to associate a specific part of the partial region extracted in the second image with the subject, based on the evaluation value and the threshold. (Item 3) The information processing apparatus according to item 2, characterized in that the first control means controls the system so as not to associate a specific part of the partial region extracted in the second image with the subject when the number of detected partial regions in the second image is less than that in the first image and the evaluation value is less than the threshold. (Item 4) The information processing device according to item 2 or 3, characterized in that the evaluation value is a score that evaluates the tracking accuracy of the subject in the second image. (Item 5) The information processing device according to any one of items 2 to 4, characterized in that the threshold is set based on the evaluation value in the partial region with the highest evaluation value, which is calculated by the calculation means in the first image. (Item 6) The information processing apparatus according to item 5, characterized in that the threshold is set based on the evaluation value in the subregion with the second highest evaluation value, which is calculated by the calculation means in the first image. (Item 7) The information processing apparatus according to item 6, characterized in that the threshold is set based on the number of elapsed frames in the second image relative to the first image. (Item 8) The information processing apparatus according to item 2, characterized in that the evaluation value is the similarity between the feature quantity of the subregion associated with the subject in the first image and the feature quantity of the subregion in the second image. (Item 9) The information processing device according to any one of items 1 to 8, characterized in that the first control means suppresses associating a specific part of the partial region extracted in the second image with the subject when the number of partial regions detected in the second image is less than that in the first image. (Item 10) The information processing apparatus according to item 9, wherein the first control means provides additional conditions for associating a specific part of the partial region extracted in the second image with the subject if the number of partial regions detected in the second image is less than that in the first image. (Item 11) The information processing apparatus according to item 1, further comprising a second control means that performs display control indicating the partial region in the second image according to the association control content by the first control means. (Item 12) The information processing apparatus according to item 11, wherein the second control means displays a frame indicating the partial region in the second image when the partial region is associated with the subject, and does not display a frame indicating the partial region when the partial region is not associated with the subject. (Item 13) A step of detecting a subject from a first image and a second image that chronologically follows the first image, A step of detecting a partial region indicating a specific part of the subject from the first image and the second image, respectively. A step of obtaining the state of change in the number of detected subregions between the first image and the second image, Depending on the state of the change, a step is performed to control whether or not to associate a specific part of the partial region extracted in the second image with the subject, An information processing method comprising: (Item 14) A program to cause a computer to function as one of the means of an information processing device described in any one of items 1 through 12.

[0083] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0084] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0085] 101: CPU, 102: Computer bus, 103: First memory, 104: Second memory, 105: Input unit, 106: Display unit, 107: Communication unit

Claims

1. A first detection means for detecting a subject region corresponding to the tracking target from a first image and a second image that chronologically follows the first image, A second detection means for detecting a partial region indicating a specific part from the first image and the second image, An acquisition means for acquiring the number of first subregions detected in the first image and the number of second subregions detected in the second image, The system includes a means for associating the aforementioned partial region with the aforementioned subject region, The association means is an information processing device that, when the number of second subregions is less than the number of first subregions, does not associate a specific part shown in the subregion detected in the second image with the subject region.

2. A calculation means for calculating an evaluation value for determining association from the second sub-region, The system further includes setting means for setting a threshold value for the evaluation value in accordance with the change in the number of the second subregion and the number of the first subregion, The information processing apparatus according to claim 1, characterized in that it controls whether or not to associate a specific part of the second subregion with the subject region based on the evaluation value and the threshold value.

3. The information processing apparatus according to claim 2, characterized in that when the number of the second subregions is less than the number of the first subregions and the evaluation value is less than the threshold, control is performed so as not to associate a specific part of the second subregion with the subject region.

4. The information processing apparatus according to claim 2, characterized in that the evaluation value is a score that evaluates the tracking accuracy of the subject area in the second image.

5. The information processing apparatus according to claim 2, characterized in that the threshold is set based on the evaluation value in the subregion with the highest evaluation value, which is calculated by the calculation means in the first image.

6. The information processing apparatus according to claim 5, characterized in that the threshold is set based on the evaluation value in the subregion with the second highest evaluation value, which is calculated by the calculation means in the first image.

7. The information processing apparatus according to claim 6, characterized in that the threshold is set based on the number of elapsed frames in the second image relative to the first image.

8. The information processing apparatus according to claim 2, characterized in that the evaluation value is the similarity between the feature quantity of the first subregion associated with the subject region in the first image and the feature quantity of the second subregion.

9. The information processing apparatus according to claim 1, characterized in that when the number of the second subregions is less than the number of the first subregions, it suppresses associating a specific part of the second subregion with the subject region.

10. The information processing apparatus according to claim 9, characterized in that when the number of the second subregions is less than the number of the first subregions, additional conditions are provided for associating a specific part of the second subregion with the subject region.

11. The information processing apparatus according to claim 1, characterized in that the display control indicating the second sub-region is performed according to the content of the association control.

12. The information processing apparatus according to claim 11, characterized in that, in the second image, if the second partial region is associated with the subject region, a frame indicating the second partial region is displayed, and if the second partial region is not associated with the subject region, a frame indicating the second partial region is not displayed.

13. A step of detecting a subject region corresponding to the tracking target from a first image and a second image that chronologically follows the first image, A step of detecting a partial region indicating a specific part from the first image and the second image, A step of obtaining the number of first subregions detected in the first image and the second subregions detected in the second image, The process includes relating the aforementioned partial region to the aforementioned subject region, An information processing method wherein, if the number of the second subregions is less than the number of the first subregions, a specific part shown in the subregion detected in the second image is not associated with the subject region.

14. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 12.

Images