Image processing device, image processing method, and program

Abstract

To allow for stable object tracking even when an attribute of an object in an imaged scene is unknown.SOLUTION: With respect to an image sequence including geometric data of an object, a first group of points corresponding to geometric data included in a first frame and a second group of points corresponding to geometric data included in a second frame adjacent to the first frame are associated with each other on the basis of a distance between the groups of points. On the basis of a result from associating them, IDs are given to groups of points corresponding to geometric data included in individual frames to track the object.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to object tracking technology.

Background Art

[0002] In a situation where multiple subjects (objects) simultaneously exist in an imaging scene and each object moves independently, proximity between objects often causes tracking errors. Patent Document 1 discloses a technique for detecting a person in each frame of an image sequence (a group of continuously captured imaging images) of a scene in which multiple people move, and switching a method for updating a tracking model according to whether the people are in a proximity state or a non-proximity state. According to this technique, even when a shielding relationship occurs between objects, it is possible to stably track the objects.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique of Patent Document 1 above, objects are detected in each frame in advance to identify their positions and numbers, and proximity states are determined based on the distances between objects within the frame, enabling the tracking of each object. That is, in the case of this method, it is necessary to determine in advance the attributes of the objects to be tracked (such as humans, dogs, or specific parts such as heads and feet) and design in advance a model suitable for their tracking. Therefore, it cannot be applied to an image sequence of an arbitrary imaging scene where the attributes of the objects are not clear.

Means for Solving the Problems

[0005] The image processing apparatus according to this disclosure includes: acquisition means for acquiring sequence data including shape data of objects in a plurality of consecutive frames; matching means for matching a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame consecutive to the first frame, based on the distance between the point clouds; and tracking means for assigning an ID to the point cloud corresponding to the shape data contained in each frame based on the result of the matching, and tracking the object across the plurality of consecutive frames. The matching means performs the matching using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different. It is characterized by the following: [Effects of the Invention]

[0006] According to the technology disclosed herein, stable object tracking is possible even when the attributes of objects in the imaging scene are unknown. [Brief explanation of the drawing]

[0007] [Figure 1] (a) and (b) are diagrams illustrating ideal object tracking. [Figure 2] Hardware configuration diagram of an image processing device. [Figure 3] Software configuration diagram of an image processing device. [Figure 4] A flowchart illustrating the object tracking process. [Figure 5] A flowchart detailing the process of mapping point clouds between adjacent frames. [Figure 6] (a) is a schematic representation of the point cloud between adjacent frames, and (b) is a diagram showing the distance between the centroids of the point clouds shown in (a). [Figure 7] A diagram showing an example of the results of the linking process. [Figure 8] A flowchart showing the details of the object tracking process according to Embodiment 1. [Figure 9] (a) and (b) are diagrams illustrating the determination of the shape attributes of an object. [Figure 10](a) to (c) are diagrams illustrating specific examples of cases where the shape attribute determination results are duplicated. [Figure 11] (a) to (c) are diagrams illustrating how the point cloud is tracked. [Figure 12] (a) to (c) are diagrams illustrating asymmetric distance. [Figure 13] A diagram showing how objects of similar size come into contact and then separate. [Figure 14] A flowchart showing the details of the object tracking process according to Embodiment 2. [Modes for carrying out the invention]

[0008] Hereinafter, embodiments for implementing this embodiment will be described with reference to drawings and other materials. Note that the following embodiments are not intended to limit the technology of this disclosure, and not all configurations described in the following embodiments are necessarily essential for solving the problem.

[0009] [Embodiment 1] First, we will explain ideal object tracking with a concrete example. Figure 1(a) shows three frames extracted from a scene in which a person is juggling a ball, corresponding to specific times (t-1, t, t+1). In this case, a binary image (silhouette image) is given from each frame, in which the foreground region corresponding to the silhouettes of the person and the ball is represented by black pixels (pixel value "1") and the rest of the region by white pixels (pixel value "0"), and this is given as shape data representing the two-dimensional shape of the object. Here, the center of the black pixels in the silhouette image is considered a point, and the point cloud, which is a collection of points, is treated as representing the two-dimensional shape of the object. In the frame at time t-1, the silhouette of the person and the silhouette of the ball are separated, so there are two point clouds, PC_0 and PC_1, in that frame. On the other hand, in the frame at time t, the silhouette of the person and the silhouette of the ball overlap, so there is only one point cloud, PC_0. Note that here, the centers of all pixels constituting the foreground region are considered as point clouds, but it is also acceptable to consider only the centers of pixels in the contour portion of the foreground region as point clouds.

[0010] In this embodiment, an image sequence (hereinafter referred to as "sequence data") consisting of a plurality of consecutive frames, in which shape data representing the two-dimensional shape of an object as described above is included in each frame, is the processing target. Then, as shown in FIG. 1(b), the object is to correctly assign an object ID to the point group representing the object shape in each frame of the input sequence data to achieve highly accurate object tracking.

[0011] Note that the shape data included in each frame is not limited to representing the two-dimensional shape of an object, and may be volume data representing a three-dimensional shape. For example, in the case of volume data composed of voxels, the set of the center points of each voxel may be regarded as the point group representing the object shape. Also, only the point group distributed on the object surface among such point groups may be used. Further, when the shape data is mesh data representing the surface of an object by a set of polygons in which point groups distributed on the surface are connected topologically, the mesh surface may be regarded as a set of an infinite number of point groups.

[0012] <Hardware Configuration> FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 100. The image processing apparatus 100, which is an information processing apparatus, includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, an operation unit 215, a communication I / F 216, and a bus 217.

[0013] The CPU 211 controls the entire image processing apparatus 100 by using computer programs and data stored in the ROM 212 or the RAM 213, thereby realizing each function of the image processing apparatus 100. Note that the image processing apparatus 100 may have one or more dedicated hardware different from the CPU 211 or a GPU (Graphics Processing Unit). And at least a part of the processing by the CPU 211 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0014] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is composed of, for example, a hard disk drive, and stores various data such as an input image sequence.

[0015] The operation unit 215 is composed of, for example, a keyboard or a mouse, and receives an operation by the user and inputs various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display device 104 and an operation control unit that controls the operation unit 215. The communication I / F 216 is used for communication with a device outside the image processing apparatus 100. For example, when the image processing apparatus 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 216. When the image processing apparatus 100 has a function of wireless communication with an external device, the communication I / F 216 includes an antenna.

[0016]

[0017] The bus 217 connects each part of the image processing apparatus 100 and transmits information.

[0017] <Software Configuration> Figure 3 shows an example of the software configuration of the image processing device 100. As shown in Figure 3, the image processing device 100 has the following functional units: a data input unit 301, a mapping unit 302, a tracking unit 303, and a data output unit 304. The tracking unit 303 also has an attribute determination unit 303a and an ID assignment unit 303b. Figure 4 is a flowchart showing the object tracking flow according to this embodiment. The operation of each functional unit of the image processing device 100 will be described below with reference to the flowchart in Figure 4. In the following description, the symbol "S" means step.

[0018] In S401, the data input unit 301 receives sequence data to be tracked from an external device.

[0019] In S402, the mapping unit 302 analyzes the point clouds representing object shapes based on the shape data contained in each frame of the frame group that constitutes the input sequence data, and performs a process to map point clouds between adjacent frames. The correspondence between point clouds between adjacent frames is determined by direction and type. The direction has two forms: "forward direction," which is the direction in which time progresses, and "reverse direction," which is the direction in which time goes backward. There are also three types: The first is "one-to-one correspondence," where point clouds that are in a correspondence relationship between adjacent frames have a one-to-one relationship. The second is "joint correspondence," where one point cloud in the frame of interest is in a correspondence relationship with multiple point clouds in the previous frame. The third is "separated correspondence," where one point cloud in the frame of interest is in a correspondence relationship with multiple point clouds in the next frame. The determined correspondence between point clouds is output, for example, in the form of a graph composed of nodes and edges, and is stored in RAM 213. Details of the mapping process will be described later.

[0020] In S403, the tracking unit 303 performs a process to track each object across multiple consecutive frames that constitute the sequence data, based on the correspondence between point clouds determined by the correspondence unit 302. In this tracking process, first, the attribute determination unit 303a determines whether the point cloud corresponding to the shape data of each frame represents the shape of a single object or the combined shape of multiple objects. Here, the point cloud representing the shape of a single object is called a "single point cloud," and the point cloud representing the combined shape of multiple objects that corresponds to a group of multiple objects is called a "non-single point cloud." Following this determination of shape attributes, the ID assignment unit 303b assigns an object ID to the point cloud of each frame and tracks the object based on the correspondence between point clouds obtained in S402 and the result of the shape attribute determination above. At this time, using the characteristics of the object being tracked, it selects a point cloud with similar characteristics from the point cloud of the next frame. In this embodiment, the total number of points constituting the point cloud is used as the characteristics of the object. This is equivalent to using the number of pixels in the foreground region and its contours in a silhouette image as its features. Furthermore, if the shape data is volume data representing the three-dimensional shape of an object, this is equivalent to using the object's volume (or the object's surface area in the case of a point cloud of a surface) as its features. Details of the tracking process will be described later.

[0021] In S404, the data output unit 305 outputs the results of the tracking process performed by the tracking unit 303 as tracking data.

[0022] In this embodiment, the above-described functional units are explained as being realized by the CPU 211 executing a predetermined program, but some of them may be realized by hardware.

[0023] <Matching process> Figure 5 is a flowchart detailing the process of associating point clouds between adjacent frames in S402 described above. The following explanation will be given in detail with reference to the flowchart in Figure 5.

[0024] In S501, point clouds that are close together are linked to adjacent frames within the sequence data. This linking is performed in both the forward direction (time advancing) and the reverse direction (time reversing). For example, in the forward direction, between frames at time t and t+1, the system finds the point cloud in the frame at time t+1 that is the smallest in distance from the point cloud in the frame at time t+1, and that distance is below a threshold, and this is designated as the "forward correspondence". In the reverse direction, the system finds the point cloud in the frame at time t+1 that is the smallest in distance from the point cloud in the frame at time t, and that distance is below a threshold, and this is designated as the "reverse correspondence". Let's explain this using a concrete example. Figure 6(a) is a schematic representation of point clouds between adjacent frames. Currently, the frame at time t contains a large circular point group 601 and a small circular point group 602, shown by solid lines. The frame at time t+1 contains a single point group 603, shown by a dashed line, where the large and small circles are in contact. Figure 6(b) shows the distance between the centroids of the point groups shown in Figure 6(a), with black circles representing the centroids of each point group. The double arrow 604 represents the distance between the centroids of point group 601 at time t and point group 603 at time t+1, and the double arrow 605 represents the distance between the centroids of point group 602 at time t and point group 603 at time t+1. At this time, point group 601 has a forward correspondence with point group 603, and point group 603 has an inverse correspondence with point group 601. Similarly, point group 602 has a forward correspondence with point group 603, and point group 603 has an inverse correspondence with point group 602. Here, if the pre-set threshold is smaller than the distance 605, it will be determined that point cloud 602 and point cloud 603 do not correspond to each other. Therefore, in this embodiment, it is necessary to set a threshold that allows a distance of the magnitude indicated by the double arrow 605. Figure 7 is a graph structure representing an example of the result of linking when sequence data of a scene in which three objects exist in the imaging space is input. The sequence data contains frames for four time points, t0, t1, t2, and t3, and point clouds are linked for each pair: t0 and t1, t1 and t2, and t2 and t3. In the graph of Figure 7, the vertical axis represents the difference in the position of point clouds within the same frame, and the horizontal axis represents the time axis.The arrows connecting point clouds across frames indicate the direction and type of linking. For example, point cloud 1 in frame at time t0 is connected to point cloud 3 in frame at time t1 by a double-headed arrow, and to point cloud 4 by a single-headed arrow pointing to itself. This means that point cloud 1 corresponds forward to point cloud 3, point cloud 3 corresponds inversely to point cloud 1, and furthermore, point cloud 4 corresponds inversely to point cloud 1. This process is performed for all frames in the sequence data in this step.

[0025] In S502, the process to proceed to the next step is determined based on the number of point clouds in the next frame that were linked in S501 to the point cloud of interest in the frame of interest within the group of frames that make up the sequence data. The frames of interest are set sequentially from the first frame of the sequence data. If the number of point clouds linked to the point cloud of interest in the set frame of interest (the number of arrows connecting the point clouds in Figure 7 above) is 0, the process proceeds to S507; if it is 1, the process proceeds to S503; and if it is 2 or more, the process proceeds to S506.

[0026] In S503, the next step in the process is determined based on the number of inverse correspondences associated with the point cluster of interest. If there is one inverse correspondence associated with the point cluster of interest, the process proceeds to S504; if there are two, the process proceeds to S505.

[0027] In S504, the transition states of the point group of interest are determined in a "one-to-one correspondence" for both the forward and reverse directions.

[0028] In S505, the transition state of the point group of interest is determined to be "joined" in the forward direction and "separated" in the reverse direction.

[0029] In S506, the transition state of the point group of interest is determined to be "separated correspondence" in the forward direction and "joined correspondence" in the reverse direction.

[0030] In S507, it is determined whether all point clouds in the frame of interest have been processed. If there are any unprocessed point clouds, the next point cloud to be processed is selected from them and processing continues. If all point clouds have been processed, the process proceeds to S508.

[0031] In S508, it is determined whether all frames constituting the sequence data have been processed. If there are any unprocessed frames, the next frame to focus on is selected from among them, and processing continues. If all frames have been processed, this process ends.

[0032] The above describes the content of the mapping process. Now, referring to Figure 7 mentioned above, we will show a specific example of the result of the mapping process, using the case where the frame of interest is the frame at time t0.

[0033] ≪When the point group of interest is point group 1≫ The number of arrows connecting the point cloud in the next frame, t1, to point cloud 3 is two: a double arrow to point cloud 3 and a single arrow to point cloud 4. Therefore, the determination in S502 leads to proceeding to S506. In S506, it is determined that the forward direction corresponds to "separation" and the reverse direction corresponds to "combination". Looking at the forward direction, point cloud 1 is separated into point cloud 3 and point cloud 4 in the next frame (frame t1), so the determination result of "separation" is accurate. Also, regarding the reverse direction, point cloud 3 and point cloud 4 in frame t1 are combined into point cloud 1 in frame t0, so the determination result of "combination" is also accurate.

[0034] ≪When the point group of interest is point group 2≫ The number of arrows connecting the point cloud in the next frame, t1, to point cloud 5 is one double-headed arrow, so the decision in S502 leads to proceeding to S503. Then, the number of inverse correspondences is only one, from point cloud 5 to point cloud 2, so the decision in S503 leads to proceeding to S504. Then, in S504, a "one-to-one correspondence" is determined for both the forward and reverse directions. Point cloud 2 has a one-to-one correspondence with point cloud 5 in the next frame (frame t1), so the "one-to-one correspondence" is accurate.

[0035] <Tracking process> Figure 8 is a flowchart detailing the process of tracking an object across multiple consecutive frames in S403 described above. The following explanation will be detailed with reference to the flowchart in Figure 8.

[0036] In S801, the attribute determination unit 303a performs a shape attribute determination to determine whether the point clouds of all frames in the sequence data are single point clouds or multi-single point clouds. This shape attribute determination consists of a primary process and a secondary process.

[0037] <<Primary Processing>> In the first stage of processing, point clouds determined to be connected in either the forward or reverse direction in relation to the preceding and succeeding frames in the aforementioned mapping process are classified as "non-single point clouds," and point clouds determined to be separated are classified as "single point clouds." Figure 9(a) shows the results of the first stage of processing for each point cloud included in the four frames shown in Figure 7. Point clouds classified as single point clouds are assigned the symbol "S," and point clouds classified as non-single point clouds are assigned the symbol "M."

[0038] ≪Secondary Processing≫ Next, in the secondary processing, for point clouds that were determined to have a one-to-one correspondence with the preceding and succeeding frames in the aforementioned mapping process, the results of their initial determination are propagated to the point clouds of the preceding and succeeding frames that are associated with them. As a result, as shown in Figure 9(b), all point clouds are assigned a determination result of whether they are single point clouds or not.

[0039] It should be noted here that point clouds that are not linked to any point clouds in the preceding or following frames (point cloud 6 in frame t1 of Figure 7) are deleted without being determined to be either a standalone point cloud or a non-standalone point cloud. This is equivalent to determining that such a point cloud is noise, and prevents object IDs from being generated and assigned to the corresponding point clouds, even if false shapes of objects that do not actually exist are extracted due to errors in generating the shape data.

[0040] ≪Duplicate judgment results≫ In addition, if a single point cloud and a non-single point cloud overlap in the primary and secondary processing, the determination of the non-single point cloud takes precedence in both cases. Figures 10(a) to (c) illustrate specific examples of cases where the shape attribute determination results overlap. Figure 10(a) shows the state before the shape attribute determination, (b) shows the result of the primary processing, and (c) shows the result of the secondary processing. As shown by the dashed box in Figure 10(b), in the primary processing, point cloud 1 of the frame at time t1 is determined to be a separate correspondence in relation to point cloud 0 of the frame at time t0, and is therefore determined to be a single point cloud (S). However, since it is determined to be a one-to-one correspondence in relation to point cloud 3 of the frame at time t2, which was determined to be a non-single point cloud, it is rewritten to a non-single point cloud (M) in the secondary processing, as shown by the dashed box in Figure 10(c).

[0041] Returning to the explanation of the flow in Figure 8, the processing from S802 onward is executed by the ID assignment unit 303b.

[0042] In S802, an object ID is assigned to the point cloud of interest at the starting frame in a predetermined tracking direction. At this time, the number of points constituting the point cloud of interest, which indicates the size of the object being tracked, is stored in RAM213 as characteristic information of the object being tracked. By setting a lower limit on the number of points that can be stored as characteristic information, it is possible to prevent point clouds (noise) that are too small to be objects in the imaging scene from being selected as the point cloud of interest. Note that the starting frame when tracking in the forward direction is the first frame of the sequence data, and the starting frame when tracking in the reverse direction is the last frame of the sequence data. Either the forward or reverse direction can be performed first, and after performing it in one time axis direction, the remaining time axis direction can be performed next. The object ID assigned to the point cloud of interest (hereinafter simply referred to as "ID") can be any information that can uniquely identify the object.

[0043] In S803, the next step in the process is determined based on whether the point group of interest, to which an ID has been assigned, is a single point group or not. If the point group of interest is a single point group, the process proceeds to S804; otherwise, it proceeds to S805.

[0044] In S804, the feature information of the object being tracked is updated. Specifically, the average number of points in the single point cloud of interest for the same ID up to the present time is calculated, and the obtained average value is stored in RAM213 as a new feature of the object being tracked. In this way, by updating the information that indicates the object's features only when the point cloud of interest is a single point cloud, it is possible to avoid distortion of the original features of the object being tracked due to contact with other objects. If the shape data is mesh data representing the three-dimensional shape of the object, for example, the sum of the polygon areas (surface area) can be used as the object's feature information.

[0045] In S805, it is determined whether there is a next frame for the frame to which the point group of interest belongs. If the time axis direction of the processing target is forward, the next frame for the target frame at time t is the frame at time t+1. If the time axis direction of the processing target is reverse, the next frame for the target frame at time t is the frame at time t-1. If a next frame exists, the process proceeds to S806; otherwise, it proceeds to S810.

[0046] In S806, based on the feature information of an object, a point cloud similar to the point cloud of interest is selected from the point clouds of the next frame. In this embodiment, the point cloud with the number of points closest to the number of points held as feature information is selected from the point clouds of the next frame. Let's explain this with a specific example. Suppose the number of points in the feature information is 100. If there are two point clouds in the next frame, one consisting of 95 points and the other consisting of 102 points, the latter point cloud will be selected. Now, suppose there are two point clouds in the next frame: point cloud A consisting of 98 points and point cloud B consisting of 102 points. In this case, the difference between the number of points held ("100") and the number of points is 2 in both cases, so it is not possible to determine which to select. Therefore, in such cases, the ratio of each value to the number of points held is calculated as the size evaluation value, and the one whose calculated value is closest to 1 is selected. In the example above, point cloud A has a size evaluation value of approximately 100 ÷ 99 ≈ 1.0204, and point cloud B has a size evaluation value of approximately 0.9804. Therefore, point cloud B, which has a size evaluation value closer to 1, is selected. Alternatively, a threshold can be set to prevent the selection of point clouds that are too small as objects (noise), and the above selection can be made from point clouds with a number of points exceeding the threshold. In this embodiment, point clouds similar to the point cloud of interest are selected from the next frame based on the size of the point cloud. This makes it possible to avoid the IDs of multiple objects that are clearly different in size, such as a ball and a person, from being swapped, even if they come into contact at one time and separate at another time.

[0047] In S807, the process to proceed to the next step is determined by whether or not a point cloud similar to the point cloud of interest can be selected from the point cloud of the next frame. If a similar point cloud can be selected, the process proceeds to S808; otherwise, it proceeds to S810.

[0048] In S808, the point cloud of the next frame selected in S806 is assigned the same ID as the ID assigned to the point cloud of interest. In the following S809, an update process is performed to set the point cloud of the next frame with the same ID as the point cloud of interest. After the update is complete, the process returns to S803 and the same process is repeated.

[0049] In S810, it is determined whether the above processing has been completed for all point clouds included in the starting frame. If there are any unprocessed point clouds in the starting frame, the process returns to S802 to determine the next point cloud of interest and continues. On the other hand, if all point clouds in the starting frame have been processed, the process proceeds to S811.

[0050] In S811, it is determined whether point cloud tracking has been completed in both the forward and reverse directions. If there are any unprocessed directions, the process proceeds to S812, where the time axis direction of the processing target is changed to the opposite direction, and the processing from S802 onwards is restarted.

[0051] The above describes the tracking process according to this embodiment. Now, referring to Figures 11(a) to (c), we will explain how the point cloud is tracked in chronological order.

[0052] In Figure 11(a), point cloud 1101 is the first point cloud of interest in the frame at time t0, which is the starting frame, when the time axis direction of the first object to be processed is forward. First, point cloud 1101 is assigned the ID "0" (S802). Then, point cloud 1101 is not a single point cloud (No in S803), and there is a next frame (Yes in S805). Therefore, point cloud 1102, which has the closest object features, is selected from the next frame at time t1 (S806), and the same ID "0" is assigned to it (S808). The same process is carried out up to the final frame at time t3, and the same ID "0" is assigned to point cloud 1103 in the frame at time t2 and point cloud 1104 in the frame at time t3.

[0053] Figure 11(b) shows the results of a similar tracking process starting from the second point cloud 1105 in the frame at time t0, which serves as the starting frame. The same ID "1" assigned to point cloud 1105 is assigned to the corresponding point clouds 1106, 1107, and 1108 in the remaining frames. At this point, tracking of all point clouds in the starting frame in the forward direction is complete. Looking at Figure 11(b), we can see that point clouds 1009 and 1110, which have not been assigned an ID, remain in the frames at time t1 and time t3, respectively. This is because, although there are originally three objects in the imaging space, in the starting frame in the forward direction, two of the three objects are merged, and only two point clouds representing the shape of the objects are obtained. Considering the possibility of such a situation, in this embodiment, after tracking in the forward direction is completed, tracking in the reverse direction is performed next.

[0054] Figure 11(c) shows the result of performing a similar trace in reverse, starting with the frame at time t3 as the starting frame. By performing a reverse trace starting from point cloud 1109 in the frame at time t3, point clouds 1109 and 1110, which were not assigned an ID in the forward trace, are also assigned a common ID "2". In this way, an ID is assigned to each point cloud within the multiple frames that make up the sequence data without any omissions.

[0055] <Variation> Simply performing tracking processes in both forward and reverse directions, starting from the first and last frames of the sequence data, may result in some point clouds remaining without assigned IDs. Therefore, it is also possible to perform further tracking processes in both forward and reverse directions, starting from the point clouds of intermediate frames. This reduces the risk of point clouds remaining without assigned IDs.

[0056] As described above, according to this embodiment, in cases where multiple objects present in the imaging scene repeatedly come into contact with and separate from each other, it is possible to accurately track each object without having to identify in advance whether it is a person or an object.

[0057] [Embodiment 2] In Embodiment 1, the distance between the centroids of the point clouds was used to determine the correspondence between point clouds in adjacent frames. This distance between centroids is called the "symmetric distance" because the distance from point cloud A to point cloud B is equal to the distance from point cloud B to point cloud A. Next, Embodiment 2 describes a method in which the distance between point clouds is defined by an "asymmetric distance," where the distance from point cloud A to point cloud B is different to the distance from point cloud B to point cloud A, and the correspondence between point clouds is determined using this asymmetric distance. Note that the explanation of content common to Embodiment 1 will be omitted, and the following explanation will focus on the differences.

[0058] <About asymmetrical distance> First, let's explain the fundamental concept of asymmetric distance in this embodiment. Let X be the set of points in point cloud A, and Y be the set of points in point cloud B. The distance d of point cloud B to point cloud A is given by the following equation (1).

[0059]

number

[0060] In equation (1) above, the distance d can be found by determining the distance from a point x, which is an element of X, to the nearest element of Y, and then obtaining the maximum value of this distance over the entire X. In Figure 12(a), the point groups 1201 and 1202, shown by solid lines, belong to the frame at time t, and the point group 1203, shown by a dashed line, belongs to the frame at time t+1. In this case, the distance d from point group 1201 to point group 1203 (point group 1201 → point group 1203) and the distance d from point group 1203 to point group 1201 (point group 1203 → point group 1201) are represented by arrows 1204 and 1205, respectively, in Figure 12(b). Furthermore, the distance d(1202→1203) between point cloud 1202 and point cloud 1203 is represented by arrow 1207, and the distance d(1203→1202) between point cloud 1203 and point cloud 1202 is represented by arrow 1206. Figures 12(b) and (c) show that the value differs depending on the direction in which the distance is calculated.

[0061] The advantage of using asymmetric distance as described above is that the correspondence between large and small objects remains stable when multiple objects transition from a state of contact to a state of separation. In Embodiment 1, it was necessary to set a large threshold for correspondence so that the point cloud 1202 in the frame at time t and the point cloud 1203 in the frame at time t+1 were linked. However, setting a large threshold can lead to incorrect correspondences and increase the risk of tracking errors. By using asymmetric distance, it is possible to obtain a graph that accurately represents the correspondence without breaking the correspondence relationship, even if the threshold is small.

[0062] <Tracking process> Figure 14 is a flowchart detailing the process of tracking an object over multiple consecutive frames according to this embodiment. The following explanation will focus on the differences between this embodiment and the flowchart in Figure 8 of Embodiment 1, referring to the flowchart in Figure 14.

[0063] In S1401, similar to S801, shape attribute determination is performed for each point cloud within each frame. In the following S1402, similar to S803, the process to proceed to the next step is determined based on whether the point cloud of interest in the frame being tracked is a single point cloud or not. If the point cloud of interest is a single point cloud, the process proceeds to S1403; otherwise, it proceeds to S1404.

[0064] In S1403, the feature information of the object being tracked, corresponding to the point cloud of interest, is updated. In Embodiment 1, the total number of points in the point cloud, which indicates the size of the object, was used as feature information, but in this embodiment, information indicating the position of the object is also used. Specifically, information indicating how far the current position is from the position when the point cloud of interest was last (most recently) a single point cloud (travel distance) is used as feature information. If the point cloud of interest merges with another point cloud and then separates, the distance between the position of the point cloud of interest before merging and the positions of each candidate point cloud is calculated, and the point cloud with the shorter distance (specifically, the one with a larger value obtained by dividing the size evaluation value by the distance) is selected. By selecting the point cloud in the next frame while taking the position of the object into consideration in this way, stable tracking becomes possible even if multiple objects of similar size come into contact and then separate, as shown in Figure 13. For example, in the specific example used in the explanation of S806 above (point clouds A and B with only a small difference in size evaluation values), suppose the distance from the position when the point cloud of interest was last a single point cloud was 0.5m for point cloud A and 0.8m for point cloud B. Currently, the size estimation value of point cloud A is 1.0204 and the size estimation value of point cloud B is 0.9804. Dividing each size estimation value by the distance gives us 2.0408 for point cloud A and 1.2255 for point cloud B. In other words, point cloud A has a larger value after division, so point cloud A is selected.

[0065] In S1404, similar to S805, it is determined whether there is a next frame for the target frame to which the group of points of interest belongs. If a next frame exists, the process proceeds to S1405; otherwise, it proceeds to S1409.

[0066] In S1405, based on the object's feature information, a point cloud similar to the point cloud of interest is selected from the point cloud of the next frame. In this embodiment, in addition to information indicating the size of the point cloud, a point cloud similar to the point cloud of interest is selected from the next frame based on the position of the most recent single point cloud.

[0067] In S1406, similar to S807, the next step in the process is determined by whether or not a point cloud similar to the point cloud of interest can be selected from the next frame. If a similar point cloud can be selected, the process proceeds to S1407; otherwise, it proceeds to S1409.

[0068] In S1407, the count value of a counter (hereinafter referred to as the "tracking counter") that counts the number of times the corresponding point cloud was selected from the next frame is incremented (+1). In the following S1408, an update process is performed to set the point cloud in the next frame selected in S1405 as the next point cloud of interest. After the update is complete, the process returns to S1402 and the same process is repeated.

[0069] In S1409, it is determined whether the tracking counter value (i.e., the number of consecutive tracking attempts) is above a threshold. If it is above the threshold, the process proceeds to S1410; otherwise, it proceeds to S1411. In S1410, a common ID is assigned to the point cloud of each frame that was successfully tracked consecutively. This process ensures that only objects whose number of tracking attempts is above the threshold are assigned an ID, while objects whose number of tracking attempts is below the threshold are not assigned an ID.

[0070] In S1411, similar to S810, it is determined whether the above processing has been completed for all point clouds included in the starting frame. If there are any unprocessed point clouds in the starting frame, the process returns to S1402 to determine the next point cloud of interest and continues. On the other hand, if all point clouds in the starting frame have been processed, the process proceeds to S1412.

[0071] S1412 and S1413 correspond to S811 and S812, respectively. That is, it is determined whether or not point cloud tracking is complete in both the forward and reverse directions (S1412), and if there are any unprocessed directions, processing is started in the opposite direction (S1413).

[0072] The above describes the tracking process according to this embodiment.

[0073] <Example 1> Even if the number of times tracking is successful exceeds a threshold, if the object being tracked moves out of (disappears from) the image space midway through, it may be judged as noise and an ID may not be assigned. For example, suppose the threshold is set to "3" on the premise that an ID will be assigned if tracking is successful for 3 consecutive frames or more. In this case, if an object that should be in the center of the field suddenly disappears in the 4th frame, it is highly likely that the object is a false object that does not actually exist, so an ID will not be assigned even if tracking is successful more times than the threshold. This cancellation of ID assignment is determined, for example, by whether the position of the point cloud at the frame where tracking ends is more than a certain distance from the edge of the image space. Specifically, if the distance from the centroid of the point cloud at the time tracking ends to the edge of the captured image is greater than or equal to the threshold, the ID assignment will be canceled. Note that point clouds for which ID assignment has been canceled will not be used as a new starting point for tracking, and tracking will be terminated.

[0074] <Modification 2> Even if the object's position is included as feature information for the object being tracked, accurate tracking becomes difficult in cases where the object's position changes frame by frame (for example, when two people are holding hands and spinning around). To address such cases, it is conceivable to use the color information (pixel values) from the image captured in the last frame where the object was a single point cloud as feature information. Specifically, the image region corresponding to the point cloud in the captured image is divided into multiple parts (for example, the upper half and the lower half), the median value of the pixel values ​​is calculated for each divided region, and the color difference between the divided regions is used as the selection criterion. This is intended to use the colors of the upper and lower body clothing (uniforms) as features when the object in the captured scene is a human, but it can also be applied to objects other than humans. The use of color information described above is just one example; the image region corresponding to the point cloud can also be divided into the left half and the right half, or further divided into smaller parts. In addition, text information such as the number on the back of the uniform can be obtained through image analysis and combined.

[0075] As described above, this embodiment makes it possible to suppress the issuance of IDs for shape data of objects that do not actually exist but are mixed into the sequence data. In other words, it has the effect of reducing noise in the sequence data. As a result, it becomes possible to achieve more accurate tracking.

[0076] (Other examples) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0077] 100 Image Processing Devices 301 Data Entry Section 302 Correspondence section 303 Tracking Unit

Claims

1. Acquisition means for acquiring sequence data including shape data of objects in multiple consecutive frames, A correspondence means that associates a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame continuous with the first frame, using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different. Based on the results of the aforementioned correspondence, a determination means is provided to determine whether the point cloud corresponding to the shape data of the frames included in multiple frames corresponds to a single object or to a group of multiple objects. A tracking means that tracks the object across multiple consecutive frames by assigning an ID to the point cloud corresponding to the shape data contained in each frame based on the results of the correspondence and the results of the determination, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the matching means matches the second point cloud that is closest to the first point cloud with the first point cloud.

3. The image processing apparatus according to claim 2, characterized in that the matching means does not match the second point cloud to the first point cloud if the distance of the closest second point cloud to the first point cloud is greater than a threshold.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the tracking means tracks the object by sequentially performing a process for a plurality of consecutive frames, starting from a point cloud of interest included in a starting frame in a predetermined time axis direction, and selecting a point cloud similar to the point cloud of interest from among the point clouds included in the next frame based on feature information indicating the characteristics of the object.

5. The aforementioned tracking means is During the aforementioned tracking process, if a point cloud representing the individual shape of the object appears, the feature information of the object is updated. The image processing apparatus according to any one of claims 1 to 4.

6. The aforementioned correspondence means is, In the sequence data, for both the forward direction (time moving forward) and the reverse direction (time moving backward), the point clouds that are closest to each other in adjacent frames within the multiple consecutive frames are linked together. If the number of point clouds associated with the point cloud of interest in the frame of interest among the aforementioned consecutive frames is one, and the number of point clouds associated with the point cloud of interest in the reverse direction is one, then a one-to-one correspondence is determined for both the forward and reverse directions. If the number of point clouds associated with the point cloud of interest in the frame of interest is two or more, the correspondence is determined to be separated in the forward direction and joined in the reverse direction. If the number of point clouds associated with the point cloud of interest in the frame of interest among the aforementioned consecutive frames is two or more, and the number of point clouds associated with the point cloud of interest in the reverse direction is two or more, then the forward direction is determined to be a combined correspondence, and the reverse direction is determined to be a separated correspondence. The determination means determines that a point group determined to be a combined correspondence in either the forward or reverse direction is not a point group representing a single shape, and determines that a point group determined to be a separated correspondence is a point group representing a single shape. The image processing apparatus according to any one of claims 1 to 5.

7. The image processing apparatus according to any one of claims 1 to 6, characterized in that, if the determination means determines that it is a point cloud representing a single shape and determines that it is not a single shape, the determination means prioritizes the determination that it is not a single shape.

8. The image processing apparatus according to any one of claims 1 to 7, further characterized in that the determination means propagates the result of the determination concerning itself to the point clouds in the preceding and succeeding frames associated with the point cloud that has been determined to be a paired pair.

9. The image processing apparatus according to any one of claims 1 to 8, characterized in that, as a result of performing a process to propagate the result of the determination, if the determination result that it is a point cloud representing a single shape and the determination result that it is not a single shape overlap, the determination result that it is not a single shape is given priority.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that the determination means deletes point clouds that are not associated with other point clouds after the process of propagating the result of the determination has been performed.

11. The image processing apparatus according to claim 4 or 5, characterized in that the aforementioned feature information is information indicating the size of the object.

12. The image processing apparatus according to claim 4 or 5, characterized in that the characteristic information is information indicating the volume of the object.

13. The image processing apparatus according to claim 4 or 5, characterized in that the aforementioned feature information is information indicating the surface area of ​​the object.

14. The image processing apparatus according to any one of claims 11 to 13, characterized in that the feature information is information on the number of points constituting a point cloud corresponding to the shape data.

15. The image processing apparatus according to claim 5, characterized in that the feature information is information indicating the position of the point cloud that was last determined to represent the individual shape of the object in the tracking.

16. The image processing apparatus according to claim 5, characterized in that the feature information is color information in the captured image of the frame to which the point cloud that was last determined to represent the individual shape of the object in the tracking belongs.

17. The image processing apparatus according to any one of claims 1 to 16, characterized in that the tracking means performs the tracking in both the forward direction (time advancing) and the reverse direction (time reversing) of the sequence data.

18. The image processing apparatus according to any one of claims 1 to 17, characterized in that the tracking means does not assign the ID if the number of frames that can be tracked is less than or equal to a threshold.

19. The image processing apparatus according to any one of claims 1 to 18, characterized in that the tracking means does not assign an ID if the position of the point cloud at the time the tracking is completed is more than a certain distance from the edge of the imaging space.

20. A step of obtaining sequence data containing shape data of objects in multiple consecutive frames, The steps include: associating a first point cloud corresponding to the shape data of a first frame among the plurality of frames with a second point cloud corresponding to the shape data of a second frame continuous with the first frame, using an asymmetric distance in which the distance from the first point cloud to the second point cloud and the distance from the second point cloud to the first point cloud are different; Based on the results of the aforementioned correspondence, the step of determining whether the point cloud corresponding to the shape data of the frames included in multiple frames corresponds to a single object or to a group of multiple objects, Based on the results of the correspondence and the results of the determination, an ID is assigned to the point cloud corresponding to the shape data contained in each frame, and the object is tracked across a plurality of consecutive frames. An image processing method characterized by including [a certain element].

21. A program for causing a computer to function as an image processing device according to any one of claims 1 to 19.

Images