Correlation method, device, and program

Abstract

This association device: detects tracklets that are a series of areas that show objects detected from time-series multi-viewpoint images; calculates consistent distances between the tracklets on the basis of consistent distances between the areas in the images captured at the same time and sets a symbol Φ that indicates postponing the calculation of the consistent distances as consistence distances that do not overlap in time between the tracklets; merges, to the same cluster, a first tracklet and a second tracklet of which a consistent distance is equal to or smaller than a threshold value and uses the consistent distance except for Φ among the consistent distances between a third tracklet and each of the first and second tracklets, and updates the consistence distance between the merged cluster and the third tracklet; and associates, as the same object, an object indicated by the tracklet belonging to the same cluster in a step in which a tracklet, which can be merged, is not left.

Description

[Technical Field]

[0001] The disclosed technology relates to an association method, an association device, and an association program. [Background technology]

[0002] Conventionally, the three-dimensional position of an object in a world coordinate system is calculated using triangulation from the two-dimensional position of the object in multi-view images captured from multiple different viewpoints. In this case, a process is required to associate the same object between the images included in the multi-view images.

[0003] For example, a device has been proposed that determines whether the same person is included in multiple images. This device extracts a face region from first image data as a template and calculates a line (epipolar line) that connects the focal point of the lens of the first image capture device and the object onto a second image. This device also compares the template with a partial region of the second image on the calculated epipolar line, and determines whether the same person is included in the second image data based on the comparison result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-233040 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are cases where it is not possible to accurately match the same object between images, such as when the multi-view images include images in which the object is not detected or images in which the object is falsely detected.

[0006] In one aspect, the disclosed technology aims to improve the accuracy of matching the same object between images included in multi-view images. [Means for solving the problem]

[0007] In one aspect, the disclosed technology detects tracklets, which are a series of regions representing an object detected from each of time-series images captured by multiple cameras that capture the object from multiple different viewpoints. The disclosed technology also calculates the distance between the tracklets based on the distance between the regions in images captured at the same time, and sets a symbol indicating that distance calculation is to be postponed as the distance between tracklets that do not overlap in time. The disclosed technology then merges a first tracklet and a second tracklet whose distance is equal to or less than a threshold into the same cluster. At the same time, the disclosed technology updates the distance between the merged cluster and the third tracklet using the distance between each of the first and second tracklets and a third tracklet other than the symbol. The disclosed technology then associates objects represented by tracklets belonging to the same cluster as the same object when there are no more tracklets that can be merged. [Effects of the Invention]

[0008] As one aspect, it has an effect of improving the accuracy of matching the same object between images included in the multi-viewpoint image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a schematic diagram showing a connection between an association device and a camera according to the present embodiment. [Figure 2] 10A and 10B are diagrams for explaining the area of ​​an object detected from each image. [Figure 3] FIG. 1 is a diagram for explaining a general method for associating objects in multi-viewpoint images. [Figure 4] FIG. 1 is a diagram for explaining a general method for associating objects in multi-viewpoint images. [Figure 5]FIG. 10 is a diagram for explaining an outline of object association using spatiotemporal information. [Figure 6] FIG. 10 is a diagram for explaining a problem in associating objects using spatiotemporal information. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of tracklets detected from a multi-viewpoint image. [Figure 8] FIG. 10 is a diagram showing an example of a distance matrix in the reference method. [Figure 9] FIG. 2 is a functional block diagram of the association device according to the present embodiment. [Figure 10] FIG. 4 is a diagram illustrating an example of a distance matrix in the present embodiment. [Figure 11] FIG. 10 is a diagram for explaining the processing of an update unit. [Figure 12] FIG. 10 is a diagram for explaining the processing of an update unit. [Figure 13] FIG. 2 is a block diagram showing a schematic configuration of a computer that functions as an association device. [Figure 14] 10 is a flowchart illustrating an example of association processing according to the present embodiment. [Figure 15] 10 is a flowchart illustrating an example of an update process according to the present embodiment. [Figure 16] FIG. 10 is a diagram for explaining a comparative method 1. [Figure 17] FIG. 10 is a diagram for explaining a comparative method 2. [Figure 18] FIG. 10 is a diagram for explaining a comparative method 3. [Figure 19] FIG. 10 is an image diagram showing an example of a correlation result according to the present embodiment. [Figure 20] FIG. 10 is a diagram illustrating the application of the association device according to the present embodiment to a scoring system for gymnastics competitions. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0011] As shown in Fig. 1, the association device 10 according to this embodiment is connected to each of a plurality of cameras 30n that capture images of an object 90 (in the example of Fig. 1, the object is a person) at viewpoint n, each from a different direction. In the example of Fig. 1, n = 0, 1, 2, and a camera 300 that captures images from viewpoint 0, a camera 301 that captures images from viewpoint 1, and a camera 302 that captures images from viewpoint 2 are connected to the association device 10. Note that the number of cameras 30n connected to the association device 10 is not limited to the example of Fig. 1, and may be two, four or more.

[0012] The camera 30n is installed at an angle and position such that the object 90 falls within the photographing range. The images photographed by the camera 30n are sequentially input to the association device 10. A synchronization signal is sent to each camera 30n, and the images photographed by each camera 30n are synchronized.

[0013] Here, a general method and problems involved in associating objects between images contained in a plurality of images taken from a plurality of different viewpoints (hereinafter referred to as "multi-viewpoint images") will be described.

[0014] For example, as shown in FIG. 2, in each image 40n included in the multi-view image, the area of ​​the object 90 is detected by a two-dimensional bounding box (hereinafter referred to as "2D-BBOX") 42n. The 2D-BBOX 42n is detected using a detection model that is generated in advance by machine learning in order to detect the area of ​​the object 90 from the image 40n. In the following, the coordinate of a predetermined position of the 2D-BBOX 42n is defined as "x". The predetermined position may be, for example, the center of the 2D-BBOX 42n, the midpoint of the base, or one of the corners (e.g., the upper left corner). In the example of FIG. 2, the center of the 2D-BBOX 42n (the black circle in FIG. 2) is defined as x.

[0015] For example, as shown in Fig. 3, it is assumed that 2D-B BOXes 420 No. 1 and 2 are detected from an image 400 captured by a camera 300, and 2D-B BOXes 421 No. 3 and 4 are detected from an image 401 captured by a camera 301. Furthermore, for example, the center of 2D-B BOX 420 No. 1 is "x1", and the center of 2D-B BOX 421 No. 4 is "x4". In this case, the distance d indicating the degree of consistency between x1 and x4 (hereinafter referred to as "consistency distance") is cross (x1, x4) can be calculated using the following formula (1).

[0016]

number

[0017] In addition, L c0 (x4) is the epipolar line obtained by projecting the line connecting the focal point of the camera 301 and x4 onto the image 400, and L c1 (x1) is the epipolar line obtained by projecting the line connecting the focal point of the camera 300 and x1 onto the image 401. c0 (x4)) is x1 and L c0 (x4) (hereinafter referred to as "epipolar distance"), that is, the distance from x1 to L on the image 400. c0 is the length of the perpendicular to (x4). Similarly, d(x4,L c1 (x1)) is x4 and L c1 (x1), that is, the epipolar distance from x4 to L on image 401. c1 is the length of the perpendicular to (x1).

[0018] If two 2D-B BOXes 42n indicate the same object 90, the consistency distance between the two 2D-B BOXes 42n will be small, and if they indicate different objects 90, the consistency distance between the two 2D-B BOXes 42n will be large. Therefore, for example, as shown in the upper diagram of Figure 4, a distance matrix is ​​created in which 2D-B BOXes 42n numbers are associated with rows and columns, and each cell stores the consistency distance between the 2D-B BOX 42n indicated by the row of that cell and the 2D-B BOX 42n indicated by the column. Then, as shown in the lower diagram of Figure 4, 2D-B BOXes 42n with a close consistency distance are clustered, and 2D-B BOXes 42n in the same cluster are associated as 2D-B BOXes 42n indicating the same object 90.

[0019] However, when multiple objects 90 are close to each other, the objects 90 may overlap in the image 40n, causing the detection result of the 2D-BBOX 42n to contain noise or result in no detection. In such cases, matching only between images taken at the same time, i.e., matching between a single frame, may not be able to accurately match the same objects 90. Furthermore, because the number of objects 90 detected from the image 40n is unknown, it is desirable to be able to perform matching using a non-parametric clustering method.

[0020] Therefore, a method of associating 2D-B BOXes 42n using spatiotemporal information, i.e., information on multiple frames, can be considered. Specifically, as shown in the upper diagram of Fig. 5, for an image 40n captured by a camera 30n (n = 0, 1 in the example of Fig. 5), 2D-B BOXes 42n are detected and tracked in multiple frames (frames k-1, k, and k+1 in the example of Fig. 5). This allows tracklets, which are a series of 2D-B BOXes 42n, to be detected. Then, as shown in the lower diagram of Fig. 5, tracklets are associated between images.

[0021] A reference method that is considered to be a method using spatiotemporal information will be described in more detail. For example, suppose that a sliding window of a predetermined number of frames, where key frame is k, is applied, and two tracklets 0 and 1 are detected from image 400, and tracklet 2 is detected from image 401, as shown in FIG. 6. In the example of FIG. 6, k=0, and the number of frames in the sliding window is 6. In this case, the reference method calculates, for each frame in the sliding window, the consistency distance between 2D-BBOXes 42n that are detected from frames that correspond to the same time information between tracklets, as shown in A, B, etc. in FIG. 6. Then, the reference method calculates the consistency distance between tracklets by averaging the calculated consistency distances for each frame, and associates the tracklets by clustering tracklets with similar consistency distances.

[0022] However, if the 2D-BBOX 42n is not detected due to overlapping of the objects 90 or the like, and there is no temporal overlap of the tracklets within the sliding window, the reference method cannot calculate the consistency distance between the tracklets. If there are tracklets between which the consistency distance cannot be calculated, it may not be possible to properly perform the process of associating the tracklets by clustering as described above.

[0023] For example, as shown in FIG. 7, suppose that tracklets 0 to 4 are detected from images 400, 401, 402, and 403 by applying a sliding window of frames 0 to 8 with a key frame of 0. Also, suppose that tracklets 1, 2, 3, and 4 indicate the same object (person) 90. Since tracklets 0 and 2, tracklets 2 and 3, tracklets 2 and 4, and tracklets 3 and 4 do not overlap in time, the consistency distance cannot be calculated using the reference method, as shown in FIG. 8. Note that FIG. 8 is a distance matrix, in which the numbers written along the rows and columns are tracklet numbers, and each square stores the consistency distance between the tracklet indicated by the row and the tracklet indicated by the column of that square. In the example of FIG. 7, as shown in the diagonally shaded squares in FIG. 8, there are tracklets between which the consistency distance cannot be calculated, so tracklets 1, 2, 3, and 4 cannot be merged into the same cluster, and tracklets indicating the same object 90 cannot be properly associated.

[0024] Therefore, the association device 10 according to this embodiment solves the above problem by propagating the calculated consistency distance between tracklets under spatiotemporal constraints for consistency distances between tracklets that cannot be calculated. The association device 10 according to this embodiment will be described in detail below.

[0025] 9, the association device 10 functionally includes a detection unit 12, a calculation unit 14, an update unit 16, and an output unit 18. A camera parameter DB (Database) 20 is stored in a predetermined storage area of ​​the association device 10. The camera parameter DB 20 stores internal parameters and external parameters of each camera 30n.

[0026] The detection unit 12 acquires time-series multi-viewpoint images captured by a plurality of cameras 30n. In the time-series multi-viewpoint images, an image 40n captured by the camera 30n includes a plurality of frames (images) in time series. A frame captured at time t among the images 40n is designated as a frame 40n(t). t = 0, 1, . . . , T, where T is time information for the final frame. Each frame 40n(t) is provided with information about a 2D-B BOX 42n detected as the region of the object 90. The information about the 2D-B BOX 42n includes coordinate values ​​of a predetermined point (e.g., the center) of the 2D-B BOX 42n, a width, and a height. Note that when acquiring multi-viewpoint images to which information about the 2D-B BOX 42n is not provided, the detection unit 12 may use the above detection model to detect the 2D-B BOX 42n of the object 90 and acquire the information about the 2D-B BOX 42n.

[0027] The detection unit 12 detects tracklets, which are a series of 2D-B BOXes 42n, by tracking, in the time direction, the 2D-B BOXes 42n detected from each frame 40n(t) within a sliding window of a predetermined number of frames from the key frame. Figure 7 is a diagram showing an example of tracklets detected by the detection unit 12.

[0028] The calculation unit 14 calculates the consistency distance between tracklets based on the consistency distance of the 2D-BBOX 42n between frames 40n(t) captured at the same time. Specifically, the calculation unit 14 calculates or sets the consistency distance between tracklets using a method depending on the temporal overlapping relationship of the tracklets and whether the cameras 30n that captured the images are the same or different. There are the following three cases:

[0029] Case 1: There is a time overlap between two tracklets, but they were taken by different cameras 30n. Case 2: There is no temporal overlap between the two tracklets Case 3: There is a temporal overlap between two tracklets, and they were taken by the same camera 30n.

[0030] In the example of Figure 7, tracklets 0 and 3, tracklets 0 and 4, tracklets 1 and 2, tracklets 1 and 3, and tracklets 1 and 4 correspond to case 1. Furthermore, tracklets 0 and 2, tracklets 2 and 3, tracklets 2 and 4, and tracklets 3 and 4 correspond to case 2. Furthermore, tracklets 0 and 1 correspond to case 3.

[0031] In Case 1, the calculation unit 14 calculates the consistency distance between tracklets in the same manner as in the reference method described above. Specifically, the calculation unit 14 calculates, for each of the temporally overlapping frames, a value based on the epipolar distance between a predetermined position of the 2D-B BOX 42n and the epipolar line as the consistency distance between the 2D-B BOX 42n. Then, the calculation unit 14 calculates the average of the calculated consistency distances between the 2D-B BOX 42n as the consistency distance between the tracklets.

[0032] More specifically, the predetermined position of the 2D-BBOX 42n detected from the frame 40i(t) is expressed as x t,i ∈ real number R 2 and x in frame 40j(t) t,i The epipolar line of L cj (x t,i In this case, the calculation unit 14 calculates x t,i and x t,j The consistency distance d cross (x t,i ,x t,j ) is calculated.

[0033]

number

[0034] In addition, w t,i and h t,iare the width and height of the 2D-BBOX 42n detected from the frame 40i(t). By normalizing the epipolar distance by the size of the 2D-BBOX 42n, the influence of the projection dispersion from three dimensions to two dimensions caused by the distance from the object 90 to the camera 30n is suppressed. The calculation unit 14 calculates d using the information of the 2D-BBOX 42n assigned to each image 40n(t) and the camera parameters of each camera 30n stored in the camera parameter DB 20. cross (x t,i ,x t,j ) is calculated.

[0035] The calculation unit 14 calculates the i-th tracklet T k,i and the j-th tracklet T k,j The consistency distance d k cross (T k,i ,T k,j ) is calculated using the following formula (3).

[0036]

number

[0037] Ψ i and Ψ j is Tracklet T k,i ,T k,j In the example of FIG. 7, Ψ0={0,1,2,3,4}, Ψ1={2,3,4,5,6,7,8}, Ψ2={6,7,8}, Ψ3={1,2}, and Ψ4={3,4}.

[0038] In case 2, the calculation unit 14 sets a symbol indicating that calculation of the consistency distance is postponed as the consistency distance between tracklets. For example, the calculation unit 14 sets the consistency distance between tracklets corresponding to case 2 as d k cross (T k,i ,T k,j )=φ.

[0039] In case 3, the calculation unit 14 sets a value larger than the upper limit value assumed for the consistency distance between tracklets as the consistency distance between tracklets. For example, the calculation unit 14 sets the consistency distance between tracklets corresponding to case 3 as d k cross (T k,i ,T k,j )=inf. In case 3, the objects 90 represented by the two tracklets are not the same, that is, if the object 90 is a person, the two tracklets represent different people. Therefore, "inf", which represents infinity, is set as the consistency distance between the tracklets. Note that the distance set in case 3 is not limited to "inf". For example, when a consistency distance normalized to 0 to 1 is used, a value greater than 1 (for example, 2) may be set as the consistency distance between the tracklets in case 3.

[0040] The calculation unit 14 creates a distance matrix and stores the consistency distances between tracklets calculated for each case. An example of the distance matrix is ​​shown in Fig. 10. Compared with the example of the reference method shown in Fig. 8, in this embodiment, "φ" is set for the squares for which it was not possible to calculate the consistency distance between tracklets in the reference method (the diagonally shaded squares in Fig. 8).

[0041] The updating unit 16 merges the first tracklet and the second tracklet whose consistency distance is equal to or less than a threshold into the same cluster. Then, the updating unit 16 updates the consistency distance between the merged cluster and the third tracklet using a consistency distance other than "φ" among the consistency distances between each of the first tracklet and the second tracklet and the third tracklet. Furthermore, when updating the consistency distance, if the consistency distance between each of the first tracklet and the second tracklet and the third tracklet is other than "φ", the updating unit 16 updates using the largest consistency distance. Furthermore, if "φ" is set as the consistency distance between each of the first tracklet and the second tracklet and the third tracklet, the updating unit 16 updates using "φ". The updating unit 16 repeats the above process until there are no more tracklets that can be merged.

[0042] Specific explanations will be given with reference to FIGS. 11 and 12. Here, the threshold value TH is set to 0.5. The minimum value of the consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1," which is less than or equal to TH. Therefore, the updating unit 16 merges tracklets 1 and 2 to form cluster 0. Since the consistency distance between tracklets 0 and 1 is "inf" and the consistency distance between tracklets 0 and 2 is "φ," the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 to "inf." Furthermore, the consistency distance between tracklets 3 and 1 is "0.2," and the consistency distance between tracklets 3 and 2 is "φ," so the updating unit 16 updates the consistency distance between tracklet 3 and cluster 0 to "0.2." Furthermore, the consistency distance between tracklets 4 and 1 is "0.3," and the consistency distance between tracklets 4 and 2 is "φ," so the updating unit 16 updates the consistency distance between tracklet 4 and cluster 0 to "0.3."

[0043] The smallest consistency distance between tracklets or clusters after the update is the consistency distance between cluster 0 and tracklet 3, which is "0.2", which is less than or equal to TH, so the updating unit 16 merges tracklet 3 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "inf", and the consistency distance between tracklets 0 and 3 is "0.4", so the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 after the merge to "inf". Furthermore, the consistency distance between tracklet 4 and cluster 0 before the merge is "0.3", and the consistency distance between tracklets 4 and 3 is "φ", so the updating unit 16 updates the consistency distance between tracklet 4 and cluster 0 after the merge to "0.3".

[0044] The smallest consistency distance between tracklets after the latest update is the consistency distance between cluster 0 and tracklet 4, which is "0.3", which is less than or equal to TH, so the updating unit 16 merges tracklet 4 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "inf", and the consistency distance between tracklets 0 and 4 is "0.6", so the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 after the merge to "inf". At this stage, there are no more tracklets that can be merged, so the updating unit 16 ends the update process.

[0045] Based on the update result of the update unit 16, the output unit 18 outputs an association result in which objects 90 indicated by tracklets belonging to the same cluster are associated as tracklets indicating the same object 90. In the examples of Fig. 11 and Fig. 12, the output unit 18 outputs an association result in which tracklets 1, 2, 3, and 4 belonging to cluster 0 are associated as tracklets indicating the same object 90.

[0046] The association device 10 may be realized by, for example, a computer 50 shown in Fig. 13. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage device 53. The computer 50 also includes an input / output device 54 such as an input device and a display device, and an R / W (Read / Write) device 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage device 53, input / output device 54, R / W device 55, and communication I / F 56 are connected to one another via a bus 57.

[0047] The storage device 53 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 53 serving as a storage medium stores an association program 60 for causing the computer 50 to function as the association device 10. The association program 60 includes detection process control instructions 62, calculation process control instructions 64, update process control instructions 66, and output process control instructions 68. The storage device 53 also includes an information storage area 70 in which information constituting the camera parameter DB 20 is stored.

[0048] The CPU 51 reads the association program 60 from the storage device 53, loads it in the memory 52, and sequentially executes the control instructions included in the association program 60. The CPU 51 executes the detection process control instruction 62 to operate as the detection unit 12 shown in FIG. 9. The CPU 51 executes the calculation process control instruction 64 to operate as the calculation unit 14 shown in FIG. 9. The CPU 51 executes the update process control instruction 66 to operate as the update unit 16 shown in FIG. 9. The CPU 51 executes the output process control instruction 68 to operate as the output unit 18 shown in FIG. 9. The CPU 51 reads information from the information storage area 70 and loads the camera parameter DB 20 in the memory 52. ​​As a result, the computer 50 that has executed the association program 60 functions as the association device 10. The CPU 51 that executes the program is hardware.

[0049] The functions realized by the association program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0050] Next, the operation of the association device 10 according to this embodiment will be described. When time-series multi-viewpoint images are input to the association device 10 and an instruction is given to perform processing for associating objects 90, the association device 10 executes the association processing shown in Fig. 14. Note that the association processing is an example of an association method of the disclosed technology.

[0051] In step S10, the detection unit 12 acquires time-series multi-viewpoint images to which information about 2D-BBOX 42n indicating the area of ​​the object 90 has been added. Next, in step S12, the detection unit 12 detects tracklets, which are a series of 2D-BBOX 42n, by tracking, in the time direction, the 2D-BBOX 42n detected from each frame 40n(t) within a sliding window of a predetermined number of frames from the key frame.

[0052] Next, in step S14, the calculation unit 14 calculates the tracklet T k,i and T k,j Next, in step S16, the calculation unit 14 selects the tracklet T k,i and T k,j If there is a time overlap, the process proceeds to step S18, and if there is no time overlap, the process proceeds to step S24.

[0053] In step S18, the calculation unit 14 calculates the tracklet T k,i and T k,j It is determined whether the cameras 40n that captured the detected images 40n are different. If the cameras 40n are different, the process proceeds to step S20, and if they are the same, the process proceeds to step S22.

[0054] In step S20, since the case corresponds to the above-mentioned case 1, the calculation unit 14 calculates the consistency distance d between tracklets by the above-mentioned equations (2) and (3). k cross (T k,i ,T k,j ) and stores it in the corresponding cell of the distance matrix. In step S22, since the case corresponds to the above-mentioned case 3, the calculation unit 14 calculates d k cross (T k,i ,T k,j ) and stores "inf" in the corresponding cell of the distance matrix. In step S24, since the above case 2 applies, the calculation unit 14 stores d k cross (T k,i ,T k,j ) and store "φ" in the corresponding cell of the distance matrix.

[0055] Next, in step S26, the calculation unit 14 determines whether or not all tracklet pairs have been selected from the tracklets detected in step S12. If there are any unselected pairs, the process returns to step S14, and if all pairs have been selected, the process proceeds to step S30. In step S30, an update process is performed.

[0056] The update process will now be described with reference to Fig. 15. In step S32, the update unit 16 determines whether the minimum value of the consistency distance between clusters or tracklets in the distance matrix is ​​equal to or less than a threshold value TH. If the minimum value of the consistency distance is equal to or less than the threshold value, the process proceeds to step S34.

[0057] In step S34, the update unit 16 merges the clusters or tracklets with the smallest consistency distance into the same cluster. Next, in step S36, the update unit 16 determines whether the consistency distances between other tracklets and the cluster or tracklet merged into the cluster in step S34 are all "φ". If all are "φ", the process proceeds to step S38, and if any other consistency distance is included, the process proceeds to step S40.

[0058] In step S38, the update unit 16 updates the consistency distance between the cluster merged in step S34 and other tracklets with "φ". Meanwhile, in step 40, the update unit 16 updates the consistency distance between the cluster merged in step S34 and other tracklets with the maximum consistency distance other than "φ". Then, the process returns to step S32.

[0059] If the update unit 16 determines in step S32 that the minimum value of the consistency distance between clusters or tracklets is greater than the threshold value TH, the process returns to the association process (FIG. 14).

[0060] Next, in step S50, based on the result of the update process in step S30, the matching result is output in which the objects 90 indicated by the tracklets belonging to the same cluster are matched as tracklets indicating the same object 90, and the matching process is terminated.

[0061] As described above, the association device according to this embodiment detects tracklets, which are a series of regions representing an object detected from each of time-series images captured by multiple cameras that capture the object from multiple different viewpoints. The association device also calculates a consistency distance between tracklets based on the consistency distance between regions in images captured at the same time. At this time, the association device sets a symbol indicating that calculation of the consistency distance is to be postponed as the consistency distance between tracklets that do not temporally overlap. The association device then merges a first tracklet and a second tracklet whose consistency distance is equal to or less than a threshold into the same cluster. At this time, the association device updates the consistency distance between the merged cluster and the third tracklet using the consistency distance between each of the first and second tracklets and the third tracklet, other than the symbol. When there are no more mergeable tracklets, the association device associates objects represented by tracklets belonging to the same cluster as the same object. As a result, even if there are tracklets that do not overlap in time, the calculable consistency distance is propagated to the consistency distance between those tracklets, thereby improving the accuracy of matching the same object between images included in a multi-view image.

[0062] Furthermore, the association device according to this embodiment updates the consistency distance between the merged cluster and other tracklets using the maximum consistency distance between each of the first tracklet and the second tracklet and the third tracklet, thereby enabling appropriate merging of tracklets or clusters.

[0063] Furthermore, the correlation device according to this embodiment sets a value greater than the upper limit of the consistency distance between tracklets, for example, infinity, as the consistency distance between temporally overlapping tracklets detected from images captured by the same camera, thereby making it possible to easily identify the relationship between tracklets representing different objects.

[0064] Here, in order to explain the effectiveness of the above embodiment, a method (hereinafter referred to as a "comparison method") that does not employ the method of the above embodiment will be described. Note that in each of the following comparison methods, as in the above embodiment, tracklets 0 to 4 are detected as shown in Fig. 7, and tracklets 1 to 4 indicate the same object, with only tracklet 0 indicating a different object. In addition, the threshold TH for merging tracklets is set to TH = 0.5.

[0065] First, comparative method 1 will be described with reference to Fig. 16. In comparative method 1, if the consistency distance between tracklets cannot be calculated, it is set to "0". In this case, if tracklets with the smallest consistency distance equal to or less than a threshold are merged, tracklets 0, 2, 3, and 4 will be merged into the same cluster, as shown in Fig. 16. In other words, tracklet 0 and tracklets 2, 3, and 4, which indicate different objects, will be associated as tracklets indicating the same object, and an appropriate association result will not be obtained.

[0066] Next, comparison method 2 will be described with reference to FIG. 17. In comparison method 2, if the consistency distance between tracklets cannot be calculated, it sets it to "1.0." The smallest consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1," which is less than TH. Therefore, comparison method 2 merges tracklets 1 and 2 to form cluster 0. Since the consistency distance between tracklets 0 and 1 is "inf" and the consistency distance between tracklets 0 and 2 is "1.0," comparison method 2 updates the consistency distance between tracklet 0 and cluster 0 to "inf." Furthermore, since the consistency distance between tracklets 3 and 1 is "0.2" and the consistency distance between tracklets 3 and 2 is "1.0," comparison method 2 updates the consistency distance between tracklet 3 and cluster 0 to "1.0." Furthermore, since the consistency distance between tracklet 4 and 1 is "0.3" and the consistency distance between tracklet 4 and 2 is "1.0", comparative method 2 updates the consistency distance between tracklet 4 and cluster 0 to "1.0".

[0067] The smallest consistency distance between tracklets after the update is the consistency distance between tracklet 0 and tracklet 3, which is "0.4", which is less than TH, so comparison method 2 merges tracklets 0 and 3 to form cluster 1. Since the consistency distance between cluster 0 and tracklet 0 is "inf" and the consistency distance between cluster 0 and 3 is "0.4", comparison method 2 updates the consistency distance between clusters 0 and 1 to the maximum value, "inf". Furthermore, since the consistency distance between tracklet 4 and tracklet 0 is "0.6" and the consistency distance between tracklets 4 and 3 is "1.0", comparison method 2 updates the consistency distance between tracklet 4 and cluster 1 to "1.0".

[0068] At this stage, there is no consistency distance equal to or less than TH, and therefore comparison method 2 terminates the update process. Therefore, tracklets 1 and 2 are merged into the same cluster, and are merged into the same cluster as tracklets 0 and 3. That is, tracklets 0 and 3, which indicate different objects, are associated as tracklets indicating the same object, and tracklet 3, which indicates the same object, cannot be associated with tracklets 1 and 2, and therefore an appropriate association result cannot be obtained.

[0069] Next, comparative method 3 will be described with reference to Fig. 18. In comparative method 3, when the consistency distance between tracklets cannot be calculated, it sets it to "1.0" and updates the consistency distance with the cluster after merging with the minimum value of the consistency distance between the tracklets to be merged.

[0070] The minimum consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1," which is less than or equal to TH. Therefore, comparison method 3 merges tracklets 1 and 2 to form cluster 0. The consistency distance between tracklets 0 and 1 is "inf," and the consistency distance between tracklets 0 and 2 is "1.0," so comparison method 3 updates the consistency distance between tracklet 0 and cluster 0 to the minimum value, "1.0." Furthermore, the consistency distance between tracklets 3 and 1 is "0.2," and the consistency distance between tracklets 3 and 2 is "1.0," so comparison method 3 updates the consistency distance between tracklet 3 and cluster 0 to "0.2." Furthermore, the consistency distance between tracklets 4 and 1 is "0.3," and the consistency distance between tracklets 4 and 2 is "1.0," so comparison method 3 updates the consistency distance between tracklet 4 and cluster 0 to "0.3."

[0071] The smallest consistency distance between tracklets after the update is the consistency distance between cluster 0 and tracklet 3, which is "0.2", which is less than TH, so comparison method 3 merges tracklet 3 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "1.0", and the consistency distance between tracklets 0 and 3 is "0.4", so comparison method 3 updates the consistency distance between cluster 0 after the merge and tracklet 0 to "0.4". Furthermore, the consistency distance between tracklet 4 and cluster 0 before the merge is "0.3", and the consistency distance between tracklets 4 and 3 is "1.0", so comparison method 3 updates the consistency distance between cluster 0 after the merge and tracklet 4 to "0.3".

[0072] The smallest consistency distance between tracklets after the latest update is the consistency distance between cluster 0 and tracklet 4, which is "0.3", which is less than TH, so comparison method 3 merges tracklet 4 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "0.4", and the consistency distance between tracklets 0 and 4 is "0.6", so comparison method 3 updates the consistency distance between cluster 0 and tracklet 0 after the merge to "0.4".

[0073] Furthermore, the smallest consistency distance between tracklets after the latest update is the consistency distance between cluster 0 and tracklet 0 of 0.4, which is less than TH, so comparative method 3 merges tracklet 0 into cluster 0. As a result, all tracklets, including tracklet 0 indicating a different object, are merged into the same cluster, and appropriate matching results cannot be obtained.

[0074] In this embodiment, "φ" is set between tracklets that do not overlap in time, i.e., between tracklets for which a consistency distance cannot be calculated. Furthermore, in this embodiment, when tracklets or clusters are merged, the consistency distance between the merged cluster and other tracklets is updated with the maximum consistency distance between the tracklet or cluster to be merged and other tracklets. In this embodiment, these points are different from the above comparison methods 1 to 3, and as a result, an appropriate association result can be obtained.

[0075] Next, an example of a matching result obtained by the matching device according to the above embodiment will be described. FIG. 19 is an illustration of a multi-viewpoint image captured by four cameras. In FIG. 19, the four image sets at the top left, top right, bottom left, and bottom right are image sets captured at different times. In addition, in FIG. 19, 2D-BBOXes representing the same object are represented by the same line type, while 2D-BBOXes representing different objects are represented by different line types. Even when the relative distance between different objects in an image is small, as in the example of FIG. 19, according to this embodiment, the same object can be correctly matched. That is, this embodiment is effective when multiple objects exist in the same frame. Furthermore, since this embodiment is a process of merging the same object into clusters using a non-parametric clustering method, it functions effectively even when the number of objects present in the same frame is variable.

[0076] The association device according to the above embodiment can be applied to, for example, a scoring system for gymnastics. Here, an outline of the processing of the scoring system for gymnastics will be described with reference to FIG.

[0077] When multi-perspective images are input, the scoring system detects a person's area from each image included in the multi-perspective images. The scoring system then determines whether the person indicated by the detected area is a player or a non-player, based on whether the person's location is within the competition area, and identifies the area representing the player. The scoring system tracks the player by matching areas representing the same player across multiple frames from a single viewpoint in the time-series multi-perspective images, and matches the tracked player across multiple viewpoints, i.e., across images. The scoring system recognizes the player's two-dimensional skeletal information from each of the tracked images using a recognition model, etc. The scoring system estimates three-dimensional skeletal information from the two-dimensional skeletal information using camera parameters. The scoring system then performs post-processing, such as smoothing, on the time-series three-dimensional skeletal information to estimate the phases (breaks) of the performance, and recognizes the technique.

[0078] In the processing of the scoring system described above, the matching device according to the embodiment described above can be applied to the tracking processing. Specifically, the process of tracking a player by matching areas showing the same player between multiple frames from a single viewpoint corresponds to the detection of tracklets in the embodiment described above. Furthermore, the process of matching tracked players between images corresponds to the matching between tracklets in the embodiment described above.

[0079] The disclosed technology is not limited to gymnasts as the target object, but can be applied to various people as the target object, such as athletes of other sports, ordinary pedestrians, etc. Furthermore, it can also be applied to animals, vehicles, etc. as the target object other than people.

[0080] In the above embodiment, the association program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory. [Explanation of symbols]

[0081] 10. Corresponding device 12 Detector 14 Calculation section 16 Update section 18 Output section 20 Camera parameter DB 30n camera 40n images 50 Computers 51 CPU 52 memory 53 Storage device 54 Input / Output Devices 55 R / W device 56 Communication I / F 57 Bus 59 Storage medium 60 Mapping Program 62 Detection Process Control Instructions 64 Calculation Process Control Instructions 66 Update Process Control Instructions 68 Output Process Control Instructions 70 Information storage area 90 Objects

Claims

1. Detecting tracklets, which are a series of regions representing an object detected from each of time-series images captured by each of a plurality of cameras that capture the object from a plurality of different viewpoints; calculating the distance between the tracklets based on the distance between the regions between the images captured at the same time, and setting a symbol indicating that calculation of the distance is to be postponed as the distance between tracklets that do not overlap in time; merging the first tracklet and the second tracklet whose distance is equal to or less than a threshold into the same cluster, and updating the distance between the merged cluster and the third tracklet by using a distance other than the symbol among the distances between the first tracklet and the second tracklet and a third tracklet; When there are no more tracklets that can be merged, the objects indicated by tracklets belonging to the same cluster are associated as the same object. The method includes the steps of:

2. The association method according to claim 1 , wherein the process of updating the distances includes updating the distances using the largest distance between the first tracklet and a third tracklet and the distances between the second tracklet and a third tracklet.

3. 2. The correspondence method according to claim 1, wherein the process of updating the distance includes, when the symbol is set as the distance between each of the first tracklet and the second tracklet and a third tracklet, updating the distance using the symbol.

4. The correspondence method according to any one of claims 1 to 3, wherein the distance between the tracklets detected from the time series images taken by different cameras and overlapping in time is a distance that becomes smaller as the possibility that the objects indicated by the tracklets are the same object increases.

5. The correspondence method according to claim 4, wherein the distance between the tracklets is the average of the distances between the epipolar line on the second image corresponding to the position of the region in the first image and the position of the region in the second image, in the first image and the second image taken at the same time within the tracklet.

6. The method for assembling according to any one of claims 1 to 3, wherein the process of calculating the distance between the tracklets is performed by setting a value greater than an upper limit value of the distance between tracklets as the distance between tracklets that are detected from the time series images taken by the same camera and that overlap in time.

7. a detection unit that detects tracklets, which are a series of regions representing an object, detected from each of time-series images captured by each of a plurality of cameras that capture the object from a plurality of different viewpoints; a calculation unit that calculates the distance between the tracklets based on the distance between the regions in the images captured at the same time, and sets a symbol indicating that calculation of the distance is to be postponed as a distance between tracklets that do not overlap in time; an updating unit that merges the first tracklet and the second tracklet, the distance of which is equal to or less than a threshold, into the same cluster, and updates the distance between the cluster after merging and the third tracklet by using a distance other than the symbol among the distances between the first tracklet and the second tracklet and a third tracklet; an output unit that, when there are no more mergeable tracklets, associates objects indicated by tracklets belonging to the same cluster as the same object; A mapping device including:

8. Detecting tracklets, which are a series of regions representing an object detected from each of time-series images captured by each of a plurality of cameras that capture the object from a plurality of different viewpoints; calculating the distance between the tracklets based on the distance between the regions between the images captured at the same time, and setting a symbol indicating that calculation of the distance is to be postponed as the distance between tracklets that do not overlap in time; merging the first tracklet and the second tracklet whose distance is equal to or less than a threshold into the same cluster, and updating the distance between the merged cluster and the third tracklet by using a distance other than the symbol among the distances between the first tracklet and the second tracklet and a third tracklet; When there are no more tracklets that can be merged, the objects indicated by tracklets belonging to the same cluster are associated as the same object. A correspondence program for causing a computer to execute a process including the above.

Images