Association method, device, and program
Patent Information
- Application Number
- JP2024530250
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-06-30
- Filing Date
- 2022-06-30
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2042-06-30
Abstract
Description
Correlation method, device, and program
[0001] The disclosed technology relates to an association method, an association device, and an association program.
[0002] Conventionally, the three-dimensional position of an object in a world coordinate system is calculated using triangulation from the two-dimensional position of the object in multi-viewpoint images obtained by capturing the object from multiple different viewpoints. In this case, a process is required to associate the same object between the images included in the multi-viewpoint images.
[0003] For example, a device has been proposed that determines whether the same person is included in multiple images. This device extracts a face region from first image data as a template and calculates a line (epipolar line) that connects the focal point of the lens of the first image capture device and the object onto a second image. This device also compares the template with a partial region of the second image on the calculated epipolar line, and determines whether the same person is included in the second image data based on the comparison result.
[0004] JP 2011-233040 A
[0005] However, there are cases where it is not possible to accurately match the same object between images, such as when the multi-view images include images in which the object is not detected or images in which the object is falsely detected.
[0006] In one aspect, the disclosed technology aims to improve the accuracy of matching the same object between images included in multi-view images.
[0007] In one aspect, the disclosed technology detects tracklets, which are a series of regions representing an object detected from each of time-series images captured by multiple cameras that capture the object from multiple different viewpoints. The disclosed technology also calculates the distance between the tracklets based on the distance between the regions in images captured at the same time, and sets a symbol indicating that distance calculation is to be postponed as the distance between tracklets that do not overlap in time. The disclosed technology then merges a first tracklet and a second tracklet whose distance is equal to or less than a threshold into the same cluster. At the same time, the disclosed technology updates the distance between the merged cluster and the third tracklet using the distance between each of the first and second tracklets and a third tracklet other than the symbol. The disclosed technology then associates objects represented by tracklets belonging to the same cluster as the same object when there are no more tracklets that can be merged.
[0008] As one aspect, it has an effect of improving the accuracy of matching the same object between images included in the multi-view image.
[0009] 1 is a schematic diagram showing a connection between a matching device and a camera according to the present embodiment. FIG. 1 is a diagram for explaining object regions detected from each image. FIG. 2 is a diagram for explaining a general method for matching objects in multi-viewpoint images. FIG. 3 is a diagram for explaining a general method for matching objects in multi-viewpoint images. FIG. 4 is a diagram for explaining an overview of matching objects using spatio-temporal information. FIG. 5 is a diagram for explaining problems with matching objects using spatio-temporal information. FIG. 6 is a schematic diagram showing an example of tracklets detected from multi-viewpoint images. FIG. 7 is a diagram for explaining an example of a distance matrix in a reference method. FIG. 8 is a functional block diagram of a matching device according to the present embodiment. FIG. 9 is a diagram for explaining an example of a distance matrix in the present embodiment. FIG. 10 is a diagram for explaining processing by an update unit. FIG. 11 is a diagram for explaining processing by the update unit. FIG. 12 is a block diagram showing a schematic configuration of a computer functioning as a matching device. FIG. 13 is a flowchart showing an example of matching processing according to the present embodiment. FIG. 14 is a flowchart showing an example of updating processing according to the present embodiment. FIG. 15 is a diagram for explaining comparison method 1. FIG. 16 is a diagram for explaining comparison method 2. FIG. 17 is a diagram for explaining comparison method 3. FIG. 18 is an image diagram showing an example of a matching result according to the present embodiment. FIG. 19 is a diagram for explaining application of the matching device according to the present embodiment to a scoring system for gymnastics.
[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.
[0011] As shown in Fig. 1, the association device 10 according to this embodiment is connected to each of a plurality of cameras 30n that capture images of an object 90 (in the example of Fig. 1, the object is a person) at viewpoint n, each from a different direction. In the example of Fig. 1, n = 0, 1, 2, and a camera 300 that captures images from viewpoint 0, a camera 301 that captures images from viewpoint 1, and a camera 302 that captures images from viewpoint 2 are connected to the association device 10. Note that the number of cameras 30n connected to the association device 10 is not limited to the example of Fig. 1, and may be two, four or more.
[0012] The camera 30n is installed at an angle and position such that the object 90 falls within the photographing range. The images photographed by the camera 30n are sequentially input to the association device 10. A synchronization signal is sent to each camera 30n, and the images photographed by each camera 30n are synchronized.
[0013] Here, a general method and problems involved in associating objects between images contained in a plurality of images taken from a plurality of different viewpoints (hereinafter referred to as "multi-viewpoint images") will be described.
[0014] For example, as shown in FIG. 2, in each image 40n included in the multi-view image, the area of the object 90 is detected by a two-dimensional bounding box (hereinafter referred to as "2D-BBOX") 42n. The 2D-BBOX 42n is detected using a detection model generated in advance by machine learning to detect the area of the object 90 from the image 40n. In the following, the coordinate of a predetermined position in the 2D-BBOX 42n is referred to as "x." The predetermined position may be, for example, the center of the 2D-BBOX 42n, the midpoint of the base, or one of the corners (e.g., the upper left corner). In the example of FIG. 2, the center of the 2D-BBOX 42n (the black circle in FIG. 2) is referred to as x.
[0015] For example, as shown in FIG. 3, it is assumed that 2D-BBOXes 420 No. 1 and No. 2 are detected from an image 400 taken by a camera 300, and 2D-BBOXes 421 No. 3 and No. 4 are detected from an image 401 taken by a camera 301. Furthermore, for example, if the center of 2D-BBOX No. 1 420 is determined by "x １ ", and the center of No. 4 2D-BBOX421 is "x ４ " In this case, x １ and x ４ and the distance indicating the degree of consistency (hereinafter referred to as the "consistency distance") d ｃｒｏｓｓ (x １ , x ４ ) can be calculated by the following formula (1).
[0016]
[0017] In addition, L ｃ０ (x ４ ) is the focal point of the camera 301 and x ４The epipolar line L is the line connecting the ｃ１ (x １ ) is the focal point of the camera 300 and x １ The line connecting the two points is the epipolar line projected onto the image 401. １ , L ｃ０ (x ４ )) is x １ and L ｃ０ (x ４ ) on the image 400 (hereinafter referred to as the “epipolar distance”). １ From L ｃ０ (x ４ ) is the length of the perpendicular to the point. ４ , L ｃ１ (x １ )) is x ４ and L ｃ１ (x １ ) on the image 401, that is, the epipolar distance ４ From L ｃ１ (x １ ) is the length of the perpendicular to
[0018] When two 2D-BBOXes 42n represent the same object 90, the consistency distance between the two 2D-BBOXes 42n is small, and when they represent different objects 90, the consistency distance between the two 2D-BBOXes 42n is large. Therefore, for example, as shown in the upper diagram of Figure 4, a distance matrix is created in which 2D-BBOX 42n numbers are associated with rows and columns, and each cell stores the consistency distance between the 2D-BBOX 42n represented by the row of that cell and the 2D-BBOX 42n represented by the column. Then, as shown in the lower diagram of Figure 4, 2D-BBOXes 42n with close consistency distances are clustered, and 2D-BBOXes 42n in the same cluster are associated as 2D-BBOXes 42n representing the same object 90.
[0019] However, when multiple objects 90 are close to each other, the objects 90 may overlap in the image 40n, and the detection results of the 2D-BBOX 42n may contain noise or may not be detected. In such cases, matching only between images captured at the same time, i.e., matching between single frames, may not be able to accurately match the same objects 90. Furthermore, because the number of objects 90 detected from the image 40n is unknown, it is desirable to be able to perform matching using a non-parametric clustering technique.
[0020] Therefore, a method of associating 2D-BBOXes 42n using spatiotemporal information, i.e., information from multiple frames, can be considered. Specifically, as shown in the upper diagram of Fig. 5, for an image 40n captured by a camera 30n (n = 0, 1 in the example of Fig. 5), 2D-BBOXes 42n are detected and tracked in multiple frames (frames k-1, k, and k+1 in the example of Fig. 5). This allows tracklets, which are a series of 2D-BBOXes 42n, to be detected. Then, as shown in the lower diagram of Fig. 5, tracklets are associated between images.
[0021] A reference method that is considered to be a method using spatiotemporal information will now be described in more detail. For example, suppose that a sliding window of a predetermined number of frames, with key frame k, is applied, and two tracklets 0 and 1 are detected from image 400, and tracklet 2 is detected from image 401, as shown in FIG. 6. In the example of FIG. 6, k = 0, and the number of frames in the sliding window is 6. In this case, the reference method calculates, for each frame in the sliding window, the consistency distance between 2D-BBOXes 42n that are detected from frames that correspond to the same time information between tracklets, as shown by A, B, etc. in FIG. 6. The reference method then calculates the consistency distance between tracklets by averaging the calculated consistency distances for each frame, and associates the tracklets by clustering tracklets with similar consistency distances.
[0022] However, if the 2D-BBOX 42n is not detected due to overlapping of the objects 90 or the like, and there is no temporal overlap of the tracklets within the sliding window, the reference method cannot calculate the consistency distance between the tracklets. If there are tracklets between which the consistency distance cannot be calculated, it may not be possible to appropriately perform the process of associating the tracklets by clustering as described above.
[0023] For example, as shown in FIG. 7 , suppose that a sliding window of frames 0 to 8, with key frame 0, is applied, and tracklets 0 to 4 are detected from images 400, 401, 402, and 403. Also, suppose that tracklets 1, 2, 3, and 4 indicate the same object (person) 90. Since tracklets 0 and 2, tracklets 2 and 3, tracklets 2 and 4, and tracklets 3 and 4 do not overlap in time, the consistency distance cannot be calculated using the reference method, as shown in FIG. 8 . Note that FIG. 8 is a distance matrix, in which the numbers written alongside the rows and columns are tracklet numbers, and each square stores the consistency distance between the tracklet indicated by the row and the tracklet indicated by the column of that square. In the example of FIG. 7 , as shown in the diagonally shaded squares in FIG. 8 , there are tracklets between which the consistency distance cannot be calculated. Therefore, tracklets 1, 2, 3, and 4 cannot be merged into the same cluster, and tracklets indicating the same object 90 cannot be properly associated.
[0024] Therefore, the association device 10 according to this embodiment solves the above problem by propagating the calculated consistency distance between tracklets under spatiotemporal constraints for consistency distances between tracklets that cannot be calculated. The association device 10 according to this embodiment will be described in detail below.
[0025] 9, the association device 10 functionally includes a detection unit 12, a calculation unit 14, an update unit 16, and an output unit 18. A camera parameter DB (Database) 20 is stored in a predetermined storage area of the association device 10. The camera parameter DB 20 stores internal parameters and external parameters of each camera 30n.
[0026] The detection unit 12 acquires time-series multi-viewpoint images captured by multiple cameras 30n. In the time-series multi-viewpoint images, an image 40n captured by the camera 30n includes multiple frames (images) in time series. Of the images 40n, a frame captured at time t is designated as frame 40n(t). t = 0, 1, ..., T, where T is the time information for the final frame. Each frame 40n(t) is also provided with information about a 2D-BBOX 42n detected as the region of the object 90. The information about the 2D-BBOX 42n includes the coordinates, width, and height of a specific point (e.g., the center) of the 2D-BBOX 42n. When acquiring multi-viewpoint images without information about the 2D-BBOX 42n, the detection unit 12 may use the detection model described above to detect the 2D-BBOX 42n of the object 90 and acquire the information about it.
[0027] The detection unit 12 detects tracklets, which are a series of 2D-BBOXes 42n, by tracking, in the time direction, the 2D-BBOXes 42n detected from each frame 40n(t) within a sliding window of a predetermined number of frames from the key frame. Figure 7 is a diagram showing an example of tracklets detected by the detection unit 12.
[0028] The calculation unit 14 calculates the consistency distance between tracklets based on the consistency distance of the 2D-BBOX 42n between frames 40n(t) captured at the same time. Specifically, the calculation unit 14 calculates or sets the consistency distance between tracklets using a method according to the temporal overlapping relationship between the tracklets and whether the cameras 30n that captured the images are the same or different. There are the following three cases:
[0029] Case 1: There is a time overlap between the two tracklets, and the cameras 30n that took the images are different. Case 2: There is no time overlap between the two tracklets. Case 3: There is a time overlap between the two tracklets, and the cameras 30n that took the images are the same.
[0030] 7 , tracklets 0 and 3, tracklets 0 and 4, tracklets 1 and 2, tracklets 1 and 3, and tracklets 1 and 4 correspond to case 1. Furthermore, tracklets 0 and 2, tracklets 2 and 3, tracklets 2 and 4, and tracklets 3 and 4 correspond to case 2. Furthermore, tracklets 0 and 1 correspond to case 3.
[0031] In Case 1, the calculation unit 14 calculates the consistency distance between tracklets in the same manner as in the reference method described above. Specifically, the calculation unit 14 calculates, for each of temporally overlapping frames, a value based on the epipolar distance between a predetermined position of the 2D-BBOX 42 n and the epipolar line as the consistency distance between the 2D-BBOXes 42 n. Then, the calculation unit 14 calculates the average of the calculated consistency distances between the 2D-BBOXes 42 n as the consistency distance between the tracklets.
[0032] More specifically, the predetermined position of the 2D-BBOX 42n detected from the frame 40i(t) is defined as x ｔ，ｉ ∈ real number R ２ and x in frame 40j(t) ｔ，ｉ The epipolar line of L ｃｊ (x ｔ，ｉ In this case, the calculation unit 14 calculates x ｔ，ｉ and x ｔ，ｊ The consistency distance d ｃｒｏｓｓ (x ｔ，ｉ , x ｔ，ｊ ) is calculated.
[0033]
[0034] Furthermore, w ｔ，ｉ and h ｔ，ｉare the width and height of the 2D-BBOX 42n detected from the frame 40i(t). By normalizing the epipolar distance by the size of the 2D-BBOX 42n, the influence of the projection dispersion from three dimensions to two dimensions caused by the distance from the object 90 to the camera 30n is suppressed. The calculation unit 14 calculates d using the information of the 2D-BBOX 42n assigned to each image 40n(t) and the camera parameters of each camera 30n stored in the camera parameter DB 20. ｃｒｏｓｓ (x ｔ，ｉ , x ｔ，ｊ ) is calculated.
[0035] The calculation unit 14 calculates the i-th tracklet T ｋ，ｉ and the j-th tracklet T ｋ，ｊ The consistency distance d ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) is calculated by the following formula (3).
[0036]
[0037] Ψ ｉ and Ψ ｊ is Tracklet T ｋ，ｉ , T ｋ，ｊ In the example of FIG. 7, Ψ ０ ={0,1,2,3,4},Ψ １ = {2, 3, 4, 5, 6, 7, 8}, Ψ ２ = {6, 7, 8}, Ψ ３ = {1, 2}, Ψ ４ = {3, 4}.
[0038] In case 2, the calculation unit 14 sets a symbol indicating that calculation of the consistency distance is postponed as the consistency distance between tracklets. For example, the calculation unit 14 sets the consistency distance between tracklets corresponding to case 2 as d ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) = φ.
[0039] In case 3, the calculation unit 14 sets a value larger than the upper limit value assumed for the consistency distance between tracklets as the consistency distance between tracklets. For example, the calculation unit 14 sets the consistency distance between tracklets corresponding to case 3 as d ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) = inf. In case 3, the objects 90 represented by the two tracklets are not the same, that is, if the object 90 is a person, the two tracklets represent different people. Therefore, "inf", which represents infinity, is set as the consistency distance between the tracklets. Note that the distance set in case 3 is not limited to "inf". For example, when a consistency distance normalized to 0 to 1 is used, a value greater than 1 (for example, 2) may be set as the consistency distance between the tracklets in case 3.
[0040] The calculation unit 14 creates a distance matrix and stores the consistency distances between tracklets calculated for each case. An example of the distance matrix is shown in Fig. 10. Compared with the example of the reference method in Fig. 8, in this embodiment, "φ" is set for squares for which it was not possible to calculate the consistency distance between tracklets in the reference method (diagonally shaded squares in Fig. 8).
[0041] The updating unit 16 merges the first tracklet and the second tracklet whose consistency distance is equal to or less than a threshold into the same cluster. Then, the updating unit 16 updates the consistency distance between the merged cluster and the third tracklet using a consistency distance other than "φ" among the consistency distances between each of the first tracklet and the second tracklet and the third tracklet. Furthermore, when updating the consistency distance, if the consistency distance between each of the first tracklet and the second tracklet and the third tracklet is other than "φ", the updating unit 16 updates using the largest consistency distance. Furthermore, if "φ" is set as the consistency distance between each of the first tracklet and the second tracklet and the third tracklet, the updating unit 16 updates using "φ". The updating unit 16 repeats the above process until no mergeable tracklets remain.
[0042] Specific explanations will be given with reference to FIGS. 11 and 12 . Here, the threshold value TH is set to 0.5. The minimum value of the consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1," and is less than TH. Therefore, the updating unit 16 merges tracklets 1 and 2 to form cluster 0. Since the consistency distance between tracklets 0 and 1 is "inf" and the consistency distance between tracklets 0 and 2 is "φ," the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 to "inf." Furthermore, the consistency distance between tracklets 3 and 1 is "0.2," and the consistency distance between tracklets 3 and 2 is "φ," so the updating unit 16 updates the consistency distance between tracklet 3 and cluster 0 to "0.2." Furthermore, the consistency distance between tracklets 4 and 1 is "0.3," and the consistency distance between tracklets 4 and 2 is "φ," so the updating unit 16 updates the consistency distance between tracklet 4 and cluster 0 to "0.3."
[0043] The smallest consistency distance between tracklets or clusters after the update is the consistency distance between cluster 0 and tracklet 3, which is "0.2", which is less than or equal to TH, so the updating unit 16 merges tracklet 3 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "inf", and the consistency distance between tracklets 0 and 3 is "0.4", so the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 after the merge to "inf". Furthermore, the consistency distance between tracklet 4 and cluster 0 before the merge is "0.3", and the consistency distance between tracklets 4 and 3 is "φ", so the updating unit 16 updates the consistency distance between tracklet 4 and cluster 0 after the merge to "0.3".
[0044] The smallest consistency distance between tracklets after the latest update is the consistency distance "0.3" between cluster 0 and tracklet 4, which is less than or equal to TH, so the updating unit 16 merges tracklet 4 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "inf", and the consistency distance between tracklets 0 and 4 is "0.6", so the updating unit 16 updates the consistency distance between tracklet 0 and cluster 0 after the merge to "inf". At this stage, there are no more tracklets that can be merged, so the updating unit 16 ends the update process.
[0045] Based on the update result of the update unit 16, the output unit 18 outputs an association result in which objects 90 indicated by tracklets belonging to the same cluster are associated as tracklets indicating the same object 90. In the examples of Fig. 11 and Fig. 12, the output unit 18 outputs an association result in which tracklets 1, 2, 3, and 4 belonging to cluster 0 are associated as tracklets indicating the same object 90.
[0046] The association device 10 may be realized by, for example, a computer 50 shown in FIG. 13 . The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage device 53. The computer 50 also includes an input / output device 54 such as an input device and a display device, and an R / W (Read / Write) device 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage device 53, input / output device 54, R / W device 55, and communication I / F 56 are connected to one another via a bus 57.
[0047] The storage device 53 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 53, which serves as a storage medium, stores an association program 60 for causing the computer 50 to function as the association device 10. The association program 60 includes detection process control instructions 62, calculation process control instructions 64, update process control instructions 66, and output process control instructions 68. The storage device 53 also includes an information storage area 70 in which information constituting the camera parameter DB 20 is stored.
[0048] The CPU 51 reads the association program 60 from the storage device 53, loads it into the memory 52, and sequentially executes the control instructions contained in the association program 60. The CPU 51 operates as the detection unit 12 shown in FIG. 9 by executing the detection process control instruction 62. The CPU 51 also operates as the calculation unit 14 shown in FIG. 9 by executing the calculation process control instruction 64. The CPU 51 also operates as the update unit 16 shown in FIG. 9 by executing the update process control instruction 66. The CPU 51 also operates as the output unit 18 shown in FIG. 9 by executing the output process control instruction 68. The CPU 51 also reads information from the information storage area 70 and loads the camera parameter DB 20 into the memory 52. As a result, the computer 50 that has executed the association program 60 functions as the association device 10. The CPU 51 that executes the program is hardware.
[0049] The functions realized by the association program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.
[0050] Next, the operation of the association device 10 according to this embodiment will be described. When time-series multi-viewpoint images are input to the association device 10 and an instruction is given to perform processing for associating objects 90, the association device 10 executes the association processing shown in Fig. 14. Note that the association processing is an example of an association method of the disclosed technology.
[0051] In step S10, the detection unit 12 acquires time-series multi-viewpoint images to which information about 2D-BBOX 42n indicating the area of the object 90 has been added. Next, in step S12, the detection unit 12 detects tracklets, which are a series of 2D-BBOX 42n, by tracking, in the time direction, the 2D-BBOX 42n detected from each frame 40n(t) within a sliding window of a predetermined number of frames from the key frame.
[0052] Next, in step S14, the calculation unit 14 calculates the tracklet T ｋ，ｉ and T ｋ，ｊ Next, in step S16, the calculation unit 14 selects the tracklet T ｋ，ｉ and T ｋ，ｊ If there is a time overlap, the process proceeds to step S18, and if there is no time overlap, the process proceeds to step S24.
[0053] In step S18, the calculation unit 14 calculates the tracklet T ｋ，ｉ and T ｋ，ｊ It is determined whether the cameras 40n that captured the detected images 40n are different. If the cameras 40n are different, the process proceeds to step S20. If the cameras 40n are the same, the process proceeds to step S22.
[0054] In step S20, since the case corresponds to the above-mentioned case 1, the calculation unit 14 calculates the consistency distance d between tracklets by the above-mentioned equations (2) and (3). ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) and stores it in the corresponding cell of the distance matrix. In step S22, since the case corresponds to the above-mentioned case 3, the calculation unit 14 calculates d ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) and stores "inf" in the corresponding cell of the distance matrix. In step S24, since the case corresponds to the above case 2, the calculation unit 14 calculates d ｋ ｃｒｏｓｓ (T ｋ，ｉ , T ｋ，ｊ ) and store "φ" in the corresponding cell of the distance matrix.
[0055] Next, in step S26, the calculation unit 14 determines whether or not all pairs of tracklets have been selected from the tracklets detected in step S12. If there are any pairs that have not been selected, the process returns to step S14. If all pairs have been selected, the process proceeds to step S30. In step S30, an update process is performed.
[0056] The update process will now be described with reference to Fig. 15. In step S32, the update unit 16 determines whether the minimum value of the consistency distance between clusters or tracklets in the distance matrix is equal to or less than a threshold TH. If the minimum value of the consistency distance is equal to or less than the threshold, the process proceeds to step S34.
[0057] In step S34, the update unit 16 merges clusters or tracklets with the smallest consistency distance into the same cluster. Next, in step S36, the update unit 16 determines whether the consistency distances between other tracklets and the cluster or tracklet merged into the cluster in step S34 are all "φ". If all are "φ", the process proceeds to step S38, and if any is other than "φ", the process proceeds to step S40.
[0058] In step S38, the update unit 16 updates the consistency distance between the cluster merged in step S34 and other tracklets with "φ". On the other hand, in step 40, the update unit 16 updates the consistency distance between the cluster merged in step S34 and other tracklets with the maximum consistency distance other than "φ". Then, the process returns to step S32.
[0059] If the update unit 16 determines in step S32 that the minimum value of the consistency distance between clusters or tracklets is greater than the threshold TH, the process returns to the association process (FIG. 14).
[0060] Next, in step S50, based on the result of the update process in step S30, the correspondence result in which the objects 90 indicated by the tracklets belonging to the same cluster are corresponded as tracklets indicating the same object 90 is output, and the correspondence process ends.
[0061] As described above, the association device according to this embodiment detects tracklets, which are a series of regions representing objects detected from each of time-series images captured by multiple cameras that capture the object from multiple different viewpoints. The association device also calculates a consistency distance between tracklets based on the consistency distance between regions in images captured at the same time. At this time, the association device sets a symbol indicating that calculation of the consistency distance will be postponed as the consistency distance between tracklets that do not temporally overlap. The association device then merges a first tracklet and a second tracklet whose consistency distance is equal to or less than a threshold into the same cluster. At this time, the association device updates the consistency distance between the merged cluster and the third tracklet using the consistency distance between each of the first and second tracklets and the third tracklet, other than the symbol. When there are no more tracklets that can be merged, the association device associates objects represented by tracklets belonging to the same cluster as the same object. As a result, even if there are tracklets that do not overlap in time, the calculable consistency distance is propagated to the consistency distance between those tracklets, thereby improving the accuracy of matching the same object between images included in a multi-view image.
[0062] Furthermore, the association device according to this embodiment updates the consistency distance between the merged cluster and other tracklets using the largest consistency distance between each of the first tracklet and the second tracklet and the third tracklet, thereby enabling appropriate merging of tracklets or clusters.
[0063] Furthermore, the correlation device according to this embodiment sets a value greater than the upper limit of the consistency distance between tracklets, for example, infinity, as the consistency distance between temporally overlapping tracklets detected from images captured by the same camera, thereby making it possible to easily identify the relationship between tracklets representing different objects.
[0064] Here, in order to explain the effectiveness of the above embodiment, a method (hereinafter referred to as a "comparison method") that does not employ the method of the above embodiment will be described. Note that in each of the following comparison methods, as in the above embodiment, tracklets 0 to 4 are detected as shown in Fig. 7, and tracklets 1 to 4 indicate the same object, with only tracklet 0 indicating a different object. In addition, the threshold TH for merging tracklets is set to TH = 0.5.
[0065] First, comparison method 1 will be described with reference to Fig. 16. In comparison method 1, if the consistency distance between tracklets cannot be calculated, it is set to "0". In this case, if tracklets with the smallest consistency distance equal to or less than a threshold are merged, tracklets 0, 2, 3, and 4 will be merged into the same cluster, as shown in Fig. 16. In other words, tracklet 0 and tracklets 2, 3, and 4, which indicate different objects, will be associated as tracklets indicating the same object, and an appropriate association result will not be obtained.
[0066] Next, comparison method 2 will be described with reference to FIG. 17 . In comparison method 2, if the consistency distance between tracklets cannot be calculated, it sets the consistency distance to "1.0". The smallest consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1", and is less than or equal to TH. Therefore, comparison method 2 merges tracklets 1 and 2 to form cluster 0. Since the consistency distance between tracklets 0 and 1 is "inf" and the consistency distance between tracklets 0 and 2 is "1.0", comparison method 2 updates the consistency distance between tracklet 0 and cluster 0 to "inf". Furthermore, since the consistency distance between tracklets 3 and 1 is "0.2" and the consistency distance between tracklets 3 and 2 is "1.0", comparison method 2 updates the consistency distance between tracklet 3 and cluster 0 to "1.0". Furthermore, since the consistency distance between tracklet 4 and 1 is "0.3" and the consistency distance between tracklet 4 and 2 is "1.0", comparison method 2 updates the consistency distance between tracklet 4 and cluster 0 to "1.0".
[0067] The smallest consistency distance between tracklets after the update is the consistency distance "0.4" between tracklet 0 and tracklet 3, which is less than or equal to TH, so comparison method 2 merges tracklets 0 and 3 to form cluster 1. Since the consistency distance between cluster 0 and tracklet 0 is "inf" and the consistency distance between clusters 0 and 3 is "0.4", comparison method 2 updates the consistency distance between clusters 0 and 1 to the maximum value, "inf". Furthermore, since the consistency distance between tracklet 4 and tracklet 0 is "0.6" and the consistency distance between tracklets 4 and 3 is "1.0", comparison method 2 updates the consistency distance between tracklet 4 and cluster 1 to "1.0".
[0068] At this stage, there is no consistency distance less than or equal to TH, so comparison method 2 ends the update process. Therefore, tracklets 1 and 2 are merged into the same cluster, and tracklets 0 and 3 are merged into the same cluster. That is, tracklets 0 and 3, which indicate different objects, are matched as tracklets indicating the same object, and tracklet 3, which indicates the same object, cannot be matched with tracklets 1 and 2, and an appropriate matching result cannot be obtained.
[0069] Next, comparison method 3 will be described with reference to Fig. 18. In comparison method 3, when the consistency distance between tracklets cannot be calculated, it sets the consistency distance to "1.0" and updates the consistency distance with the cluster after merging with the minimum value of the consistency distance between the tracklets to be merged.
[0070] The minimum consistency distance between tracklets is the consistency distance between tracklets 1 and 2, which is "0.1", and is less than or equal to TH, so comparison method 3 merges tracklets 1 and 2 to form cluster 0. The consistency distance between tracklets 0 and 1 is "inf", and the consistency distance between tracklets 0 and 2 is "1.0", so comparison method 3 updates the consistency distance between tracklet 0 and cluster 0 to the minimum value, "1.0". Furthermore, the consistency distance between tracklets 3 and 1 is "0.2", and the consistency distance between tracklets 3 and 2 is "1.0", so comparison method 3 updates the consistency distance between tracklet 3 and cluster 0 to "0.2". Furthermore, the consistency distance between tracklets 4 and 1 is "0.3", and the consistency distance between tracklets 4 and 2 is "1.0", so comparison method 3 updates the consistency distance between tracklet 4 and cluster 0 to "0.3".
[0071] The smallest consistency distance between tracklets after the update is the consistency distance between cluster 0 and tracklet 3, which is "0.2", which is less than or equal to TH, so comparison method 3 merges tracklet 3 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "1.0", and the consistency distance between tracklets 0 and 3 is "0.4", so comparison method 3 updates the consistency distance between cluster 0 after the merge and tracklet 0 to "0.4". Furthermore, the consistency distance between tracklet 4 and cluster 0 before the merge is "0.3", and the consistency distance between tracklets 4 and 3 is "1.0", so comparison method 3 updates the consistency distance between cluster 0 after the merge and tracklet 4 to "0.3".
[0072] The smallest consistency distance between tracklets after the latest update is the consistency distance between cluster 0 and tracklet 4, which is "0.3", which is less than or equal to TH, so comparison method 3 merges tracklet 4 into cluster 0. The consistency distance between tracklet 0 and cluster 0 before the merge is "0.4", and the consistency distance between tracklets 0 and 4 is "0.6", so comparison method 3 updates the consistency distance between cluster 0 and tracklet 0 after the merge to "0.4".
[0073] Furthermore, the smallest consistency distance between tracklets after the latest update is the consistency distance between cluster 0 and tracklet 0 of "0.4", which is less than TH, so comparison method 3 merges tracklet 0 into cluster 0. As a result, all tracklets, including tracklet 0 indicating a different object, are merged into the same cluster, and an appropriate matching result cannot be obtained.
[0074] In this embodiment, "φ" is set between tracklets that do not overlap in time, that is, between tracklets for which a consistency distance cannot be calculated. Furthermore, in this embodiment, when tracklets or clusters are merged, the consistency distance between the merged cluster and other tracklets is updated with the maximum consistency distance between the tracklet or cluster to be merged and the other tracklets. This embodiment differs from the above comparison methods 1 to 3 in these points, and as a result, an appropriate association result can be obtained.
[0075] Next, an example of the matching results obtained by the matching device according to the above embodiment will be described. FIG. 19 is an illustration of multi-viewpoint images captured by four cameras. In FIG. 19, the four image sets at the top left, top right, bottom left, and bottom right are image sets captured at different times. Also, in FIG. 19, 2D-BBOXes representing the same object are represented by the same line type, while 2D-BBOXes representing different objects are represented by different line types. Even when the relative distance between different objects within an image is small, as in the example of FIG. 19, this embodiment can correctly match the same object. In other words, this embodiment is effective when multiple objects exist within the same frame. Furthermore, since this embodiment uses a nonparametric clustering method to merge identical objects into clusters, it functions effectively even when the number of objects present within the same frame is variable.
[0076] The association device according to the above embodiment can be applied to, for example, a scoring system for gymnastics. Here, an outline of the processing of the scoring system for gymnastics will be described with reference to FIG.
[0077] When multi-view images are input, the scoring system detects a person's area from each image included in the multi-view images. The scoring system then determines whether the person indicated by the detected area is a player or a non-player, based on whether the person's location is within the competition area or not, and identifies the area representing the player. The scoring system tracks the player by matching areas representing the same player across multiple frames from a single viewpoint in the time-series multi-view images, and matches the tracked player across multiple viewpoints, i.e., across images. The scoring system recognizes the player's two-dimensional skeletal information from each of the tracked images using a recognition model or the like. The scoring system estimates three-dimensional skeletal information from the two-dimensional skeletal information using camera parameters. The scoring system then performs post-processing, such as smoothing, on the time-series three-dimensional skeletal information to estimate the phases (breaks) of the performance and recognizes the technique.
[0078] In the processing of the scoring system described above, the matching device according to the embodiment described above can be applied to the tracking processing. Specifically, the process of tracking a player by matching areas showing the same player between multiple frames from a single viewpoint corresponds to the detection of tracklets in the embodiment described above. Furthermore, the process of matching tracked players between images corresponds to the matching between tracklets in the embodiment described above.
[0079] The disclosed technology is not limited to gymnasts as the target object, but can be applied to various people as the target object, such as athletes of other sports, ordinary pedestrians, etc. Furthermore, it can also be applied to animals, vehicles, etc. as the target object other than people.
[0080] In the above embodiment, the association program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.
[0081] REFERENCE SIGNS LIST 10 Correlation device 12 Detection unit 14 Calculation unit 16 Update unit 18 Output unit 20 Camera parameter DB 30n Camera 40n Image 50 Computer 51 CPU 52 Memory 53 Storage device 54 Input / output device 55 R / W device 56 Communication I / F 57 Bus 59 Storage medium 60 Correlation program 62 Detection process control command 64 Calculation process control command 66 Update process control command 68 Output process control command 70 Information storage area 90 Object
Claims
1. Detecting tracklets, which are a series of regions representing an object detected from each of time-series images captured by each of a plurality of cameras that capture the object from a plurality of different viewpoints; calculating a distance between the tracklets based on a distance between the regions between images captured at the same time, and setting a symbol indicating that distance calculation is to be postponed as a distance between tracklets that do not overlap in time; merging the first tracklet and the second tracklet whose distance is equal to or less than a threshold into the same cluster, and updating the distance between the merged cluster and the third tracklet by using a distance between each of the first tracklet and the second tracklet and a third tracklet other than the symbol; When there are no more tracklets that can be merged, the objects indicated by the tracklets that belong to the same cluster are matched as the same object. The method includes the steps of:
2. The method according to claim 1 , wherein the process of updating the distances includes updating the distances by using a maximum distance between the first tracklet and a third tracklet and between the second tracklet and the first tracklet and a maximum distance between the second tracklet and a third tracklet.
3. The method according to claim 1 , wherein the process of updating the distance includes, when the symbol is set as the distance between the first tracklet and the third tracklet, updating the distance using the symbol.
4. The method for matching according to any one of claims 1 to 3, wherein the distance between the tracklets detected from the time series of images taken by different cameras and overlapping in time is a distance that becomes smaller as the likelihood that the objects indicated by the tracklets are the same object increases.
5. The method of claim 4, wherein the distance between the tracklets is an average of distances between an epipolar line on the second image corresponding to the position of the region in the first image and a position of the region in the second image, in a first image and a second image taken at the same time within the tracklet.
6. The method for matching according to any one of claims 1 to 3, wherein the process of calculating the distance between the tracklets is performed by setting a value greater than an upper limit value of the distance between tracklets as the distance between tracklets that are detected from the time series images taken by the same camera and overlap in time.
7. a detection unit that detects tracklets, which are a series of regions representing an object detected from each of time-series images captured by each of a plurality of cameras that capture an object from a plurality of different viewpoints; a calculation unit that calculates the distance between the tracklets based on the distance between the regions between images captured at the same time, and sets a symbol indicating that calculation of the distance is to be postponed as a distance between tracklets that do not overlap in time; an update unit that merges the first tracklet and the second tracklet, the distance of which is equal to or less than a threshold, into the same cluster, and updates a distance between the cluster after the merge and the third tracklet by using a distance between each of the first tracklet and the second tracklet and a third tracklet other than the symbol; an output unit that, when there are no more mergeable tracklets, associates objects indicated by tracklets belonging to the same cluster as the same object; An association device comprising:
8. Detecting tracklets, which are a series of regions representing an object detected from each of time-series images captured by each of a plurality of cameras that capture the object from a plurality of different viewpoints; calculating a distance between the tracklets based on a distance between the regions between images captured at the same time, and setting a symbol indicating that distance calculation is to be postponed as a distance between tracklets that do not overlap in time; merging the first tracklet and the second tracklet whose distance is equal to or less than a threshold into the same cluster, and updating the distance between the merged cluster and the third tracklet by using a distance between each of the first tracklet and the second tracklet and a third tracklet other than the symbol; When there are no more tracklets that can be merged, the objects indicated by the tracklets that belong to the same cluster are matched as the same object. A correspondence program for causing a computer to execute a process including the steps of: