Trajectory generation method and information processing apparatus

Abstract

To accurately generate trajectory information.SOLUTION: The information processing apparatus acquires an analysis result of an image frame for each camera that has captured an image frame including a person. The information processing apparatus generates a fragmentary trajectory of the person included in the image frame based on the acquired analysis result. The information processing device calculates a feature amount related to the appearance of the person based on the plurality of image frames associated with the generated fragmentary trajectory. The information processing apparatus generates a plurality of representative clusters by clustering the plurality of identified fragmentary trajectories using the calculated similarity of the representative values of the feature amounts of the plurality of image frames. The information processing apparatus generates a plurality of sub-clusters by clustering each of a plurality of trajectories clustered in each of the generated representative clusters by using representative values of a plurality of postures of a person. The information processing apparatus sets a plurality of trajectories corresponding to the sub-clusters as fragmentary trajectories of the same person based on the result of the plurality of generated sub-clusters.SELECTED DRAWING: Figure 3B

Description

[Technical Field]

[0001] The present invention relates to a trajectory generation program and the like. [Background technology]

[0002] There is a multi-camera tracking system that uses multiple cameras to track a target object, person, or the like in a specified application area. In a conventional multi-camera tracking system, trajectory information is generated by performing object detection, intra-camera tracking, camera calibration, and inter-camera tracking. This specification describes the case of tracking a person, but the system can also be applied to objects other than people.

[0003] The object detection process detects the target person, extracts key points, and classifies body parts. The intra-camera tracking process tracks multiple people using a single camera. The camera calibration process predicts the person's position and maps it to a map. The inter-camera tracking process extracts ReID (Re-Identification) features and generates multiple trajectory information by integrating each ReID feature.

[0004] Fig. 21 is a diagram illustrating an example of trajectory information. In the example shown in Fig. 21, multiple cameras 1-1, 1-2, 1-3, and 1-4 are installed. For example, using conventional technology, trajectory information 11a of person 10a is generated, trajectory information 11b of person 10b is generated, and trajectory information 11c of person 10c is generated.

[0005] In addition, to improve tracking accuracy, there is also a conventional technique that combines anchor-guided clustering and spatiotemporal consistent ID reassignment in addition to a multi-camera tracking system.

[0006] For example, in anchor-induced clustering, an anchor is selected as the basis for clustering, and other data points are clustered based on the anchor. During clustering, a decision is made as to whether or not a data point should be assigned to the same cluster as the anchor based on the distance between the anchor and the data point and a threshold. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2024-008869 [Patent Document 2] U.S. Patent No. 11,024,043 [Patent Document 3] U.S. Patent Application Publication No. 2021 / 0240851 [Patent Document 4] Japanese Patent Application Publication No. 2019-185615 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the above-mentioned conventional technology has a problem in that it is not possible to generate trajectory information with high accuracy.

[0009] According to one aspect, the present invention aims to provide a trajectory generation program, a trajectory generation method, and an information processing device that can generate trajectory information with high accuracy. [Means for solving the problem]

[0010] In the first proposal, a computer performs the following processes. The computer acquires analysis results of image frames captured by each camera that includes a person. The computer generates fragmentary trajectories of the person included in the image frames based on the acquired analysis results. The computer calculates feature amounts related to the person's appearance based on the multiple image frames associated with the generated fragmentary trajectories. The computer generates multiple representative clusters by clustering the identified fragmentary trajectories using similarities in representative values ​​of the calculated feature amounts of the multiple image frames. The computer generates multiple sub-clusters by clustering each of the clustered trajectories in each of the generated representative clusters using representative values ​​of multiple postures of the person. Based on the results of the generated sub-clusters, the computer sets the multiple trajectories corresponding to the sub-clusters as fragmentary trajectories of the same person. [Effects of the Invention]

[0011] Trajectory information can be generated with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining the problems of the conventional technology. [Figure 2] FIG. 2 is a diagram illustrating an information processing system according to this embodiment. [Figure 3A] FIG. 3A is a diagram illustrating the pre-processing executed by the information processing device according to the present embodiment. [Figure 3B] FIG. 3B is a diagram illustrating an example of the global optimization process. [Figure 4] FIG. 4 is a diagram for explaining ID transfer. [Figure 5] FIG. 5 is a flowchart showing the procedure of the ID transfer error removal process. [Figure 6] FIG. 6 is a diagram for explaining the first outlier removal process and the second outlier removal process. [Figure 7]FIG. 7 is a diagram illustrating the relationship between Z scores and outliers. [Figure 8] FIG. 8 is a diagram showing the processing result of the ID transfer error removal processing. [Figure 9] FIG. 9 is a flowchart showing the procedure of the optimal hierarchical clustering process. [Figure 10] FIG. 10 is a flowchart showing the procedure of the attitude-oriented grouping process. [Figure 11] FIG. 11 is a diagram (1) for explaining posture classification. [Figure 12] FIG. 12 is a diagram (2) for explaining posture classification. [Figure 13] FIG. 13 is a diagram for explaining sub-clustering. [Figure 14] FIG. 14 is a flowchart showing the procedure of the error suppression process. [Figure 15] FIG. 15 is a diagram for explaining the error suppression process. [Figure 16] FIG. 16 shows tracklets before and after error suppression processing is performed. [Figure 17] FIG. 17 is a functional block diagram illustrating the configuration of an information processing device according to this embodiment. [Figure 18] FIG. 18 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 19] FIG. 19 is a flowchart showing the procedure of the pre-processing. [Figure 20] FIG. 20 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 21] FIG. 21 is a diagram illustrating an example of the trajectory information. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the trajectory generation program, the trajectory generation method, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0014] Before describing this embodiment, the problems with the conventional technology will be described in more detail. FIG. 1 is a diagram for explaining the problems with the conventional technology. In the following description, a device that executes processing related to the conventional technology will be referred to as a "conventional device." Furthermore, an image captured by a camera will be referred to as a "frame." The camera generates frames in a time series.

[0015] In intra-camera tracking, multiple people are tracked using one camera, so multiple tracklets are generated for each camera. A tracklet is a fragment of a person's trajectory information.

[0016] For example, the conventional device generates tracklet 11-1 for person 10a based on frames from camera 1-1. The conventional device generates tracklet 11-2 for person 10a based on frames from camera 1-2. It is assumed that the conventional device mistakenly merges the tracklet for person 10b into the tracklet for person 10a.

[0017] The conventional device generates tracklet 11-3 for person 10a based on frames from camera 1-3, and generates tracklet 11-4 for person 10a based on frames from camera 1-4.

[0018] In intra-camera tracking, each camera focuses on a specific person and generates a tracklet for that person, and the tracklets for the same person generated based on frames from each camera are not integrated.

[0019] Conventional devices perform inter-camera tracking after intra-camera tracking. In inter-camera tracking, tracklets are integrated in a single process based on the average feature values ​​(position, etc.) of tracklets created separately for each camera. However, if integration fails due to insufficient information, it is not possible to generate complete trajectory information for a person.

[0020] 1, the conventional device integrates tracklets 11-1 and 11-2 to generate tracklet 12 for person 10a. Note that the tracklet for person 10b remains connected to tracklet 12. Furthermore, the conventional device fails to integrate tracklets 11-3 and 11-4 due to insufficient information or the like.

[0021] As explained in Fig. 1, in the conventional technology, tracklets that appear on the same camera and overlap in time remain connected. Furthermore, in the conventional technology, each tracklet is integrated by a single process based on the average feature amount related to the tracklet, so tracklets that could not be integrated due to insufficient information or the like are not given the opportunity to be integrated.

[0022] There is a need to solve the above-mentioned problems of the conventional technology and to integrate the tracklets of each person to generate trajectory information of each person with high accuracy. For example, in Fig. 1, the appropriate trajectory information of person 10a is tracklet 13a. Also, the appropriate trajectory information of person 10b is tracklet 13b.

[0023] In the anchor-guided clustering described in the prior art, an anchor-specific threshold is set for each data set, so it depends on the data set and cannot generate trajectory information efficiently.

[0024] The problems with the conventional technology have been described above.

[0025] Next, an example of an information processing system according to this embodiment will be described. Fig. 2 is a diagram showing the information processing system according to this embodiment. As shown in Fig. 2, the information processing system 30 has cameras 31-1, 31-2, 31-3, ..., 31-n and an information processing device 100. The cameras 31-1 to 31-n (n is an arbitrary natural number) and the information processing device 100 are connected to each other via a network 32.

[0026] The network 32 may be any of a variety of communication networks, whether wired or wireless, such as an intranet used within a retail store. The network 32 may not be a single network, but may be configured, for example, by connecting an intranet and the Internet via a network device such as a gateway or other device (not shown). The expression "inside" a retail store is not limited to indoors, but may also refer to facilities outside the premises of the retail store.

[0027] Cameras 31-1 to 31-n are surveillance cameras installed in each sales area or cash register area in a retail store or the like. In the following description, cameras 31-1 to 31-n may be collectively referred to as camera 31. Video data captured by camera 31 is transmitted to information processing device 100 via network 32. The video data includes frames in time series.

[0028] The information processing device 100 is a device that generates trajectory information of a person based on time-series frames. An example of processing by the information processing device 100 will be described below. For example, the information processing device 100 generates trajectory information of a person by performing global optimization after performing preprocessing.

[0029] An example of pre-processing performed by the information processing device 100 will be described. Fig. 3A is a diagram for explaining the pre-processing performed by the information processing device according to this embodiment. As shown in Fig. 3A, the information processing device 100 sequentially performs person detection processing, intra-camera tracking processing, and inter-camera tracking processing as pre-processing.

[0030] The person detection process executed by the information processing device 100 will be described. The information processing device 100 acquires time-series frames from the camera 31. The frames correspond to "image frames." The information processing device 100 extracts people from the frames using an existing detection algorithm such as YOLO (YOU Only Look Once). The extracted people are represented by bounding boxes (BBOX) that enclose their areas in a rectangular shape on the frames. For example, FIG. 3A shows a bounding box 10a-1 for person 10a and a bounding box 10b-1 for person 10b. The information processing device 100 also extracts key points of people 10a and 10b. The information processing device 100 calculates position information corresponding to the key points of the people using parameters of the camera 31 that captured the frames.

[0031] The information processing device 100 repeatedly executes the above process for frames in a time series. The detection result of the person detection process is generated for each camera 31.

[0032] The intra-camera tracking process executed by the information processing device 100 will be described. The information processing device 100 generates a tracklet for each camera based on the results of the person detection process (time-series person BBOX). For example, the information processing device 100 repeatedly executes a process for identifying the same person between frames based on the similarity of the person's BBOX between multiple frames, and generates a tracklet. To identify the same person, an existing tracking algorithm such as TLD (Tracking Learning Detection) or KCF (Kernelized Correlation Filters) can be used. The information processing device 100 generates a tracklet by connecting position information calculated from each frame of the same person. An ID is assigned to the tracklet, and information on the BBOX extracted from each frame is associated with it.

[0033] For example, the information processing device 100 generates tracklet 13-1 for person 10b based on the result of person detection processing for frames captured by camera 31-1. The information processing device 100 generates tracklet 13-2 for person 10b based on the result of person detection processing for frames captured by camera 31-2. The information processing device 100 generates tracklet 13-3 for person 10a based on the result of person detection processing for frames captured by camera 31-3. The information processing device 100 generates tracklet 13-4 for person 10a based on the result of person detection processing for frames captured by camera 31-4.

[0034] The following describes the intra-camera tracking process executed by the information processing device 100. The information processing device 100 executes multi-layer camera calibration for a person's head, waist, legs, etc. The information processing device 100 uses the execution results of the camera calibration to estimate the position on the floor of each tracklet generated by the intra-camera tracking process. The information processing device 100 extracts ReID features from each tracklet generated by the intra-camera tracking process.

[0035] For example, when extracting the ReID feature of one tracklet, the information processing device 100 extracts the ReID feature of the tracklet by inputting the BBOX of a frame associated with the tracklet into a learning model such as a neural network. When a tracklet is generated by integrating tracklets of multiple frames, this tracklet has the ReID feature of the tracklet of each frame.

[0036] The information processing device 100 executes multi-level DBSCAN (Density-Based Spatial Clustering of Applications with Noise) based on the ReID feature extracted from each tracklet. The information processing device 100 integrates tracklets classified into the same cluster. For example, in the example shown in FIG. 3A, tracklet 13-3 and tracklet 13-4 are integrated to generate tracklet 14. Note that tracklets 13-1 and 13-2 remain unintegrated.

[0037] When the tracklets include tracklets for each frame, the information processing device 100 may calculate a representative value of the ReID feature of each tracklet and treat this representative value as the ReID feature of the tracklet.

[0038] The above has described the pre-processing (person detection processing, intra-camera tracking processing, and inter-camera tracking processing) executed by the information processing device 100. An ID for identifying each tracklet is set to each tracklet generated by the pre-processing. In addition, information on the BBOX extracted from each frame is associated with each tracklet.

[0039] Next, a description will be given of the global optimization processing executed by the information processing device 100. For example, the information processing device 100 executes ID-transfer error removal, optimized hierarchical clustering, pose-oriented cluster-level grouping, and error suppression and refinements as the global optimization processing.

[0040] 3B is a diagram showing an example of global optimization processing. The information processing device 100 performs ID transfer error elimination processing on multiple tracklets 9 generated by preprocessing. After performing the ID transfer error elimination processing, the information processing device 100 performs optimal hierarchical clustering processing, attitude-oriented grouping processing, and error suppression processing. The information processing device 100 repeatedly performs the optimal hierarchical clustering processing, attitude-oriented grouping processing, and error suppression processing until the tracklets stabilize (until the number of tracklets no longer changes).

[0041] The ID transfer error removal process, optimal hierarchical clustering process, attitude-oriented grouping process, and error suppression process shown in FIG. 3B will be described in order.

[0042] First, an example of ID transfer error removal processing executed by information processing device 100 will be described. FIG. 4 is a diagram for explaining ID transfer. ID transfer occurs when tracklets of multiple people are erroneously integrated into one tracklet. For example, in the example shown in FIG. 4, tracklet 15 is generated by integrating the tracklet of person 10a and the tracklet of person 10b. For example, one ID "ID=1" is set in tracklet 15.

[0043] When performing ID transfer error removal processing, the information processing device 100 removes ID transfer errors based on Z scores in two stages. FIG. 5 is a flowchart showing the processing procedure of the ID transfer error removal processing. As shown in FIG. 5, the information processing device 100 calculates a ReID similarity matrix S based on the tracklet. NN is calculated (step S10).

[0044] The information processing device 100 calculates the ReID similarity matrix S NN By performing the first outlier removal process on the matrix S m(Step S11). The information processing device 100 executes a second outlier removal process to separate tracklets corresponding to outliers from the target tracklet (Step S12).

[0045] 6 is a diagram for explaining the first outlier removal process and the second outlier removal process. For example, the tracklet 15 is a tracklet obtained by integrating the tracklets from the first frame to the Nth frame.

[0046] ReID similarity matrix S NN The information in the first row of the ReID similarity matrix S NN The elements in the first row and first column of the ReID similarity matrix S are the similarity between the ReID feature of the first frame of the tracklet 15 and the ReID feature of the first frame of the tracklet 15. The similarity is a cosine similarity or the like. NN The elements in the first row and second column of the ReID similarity matrix S NN The element in the first row and Nth column of the above expression is the similarity between the ReID feature of the first frame of the tracklet 15 and the ReID feature of the Nth frame of the tracklet 15.

[0047] ReID similarity matrix S NN The information in the second row of the table is information about the second frame of the tracklet 15. The information about the second frame, like the information about the first frame, is the similarity of the ReID features of other frames. The ReID similarity matrix S NN The information on the Nth line is information about the Nth frame of the tracklet 15. The information about the second frame is the similarity of the ReID feature amounts of other frames, similar to the information about the first frame.

[0048] where the ReID similarity matrix S NNThe first outlier removal process will be described using the information in the first row of the following table. The information processing device 100 calculates the ReID similarity matrix S NN The information processing device 100 calculates the mean value μ and standard deviation σ of each element in the first row of the equation (1).

[0049]

number

[0050] In equation (1), x is a data point, e.g., the ReID similarity matrix S NN The information processing device identifies an element whose Z-score value is "-2 or less" or "2 or more" as an outlier.

[0051] Figure 7 is a diagram illustrating the relationship between Z scores and outliers. The horizontal axis of graph G1 corresponds to the Z score. For example, if the Z score is within the range of "-1 or greater and 1 or less," the corresponding data point (the ReID feature of the corresponding element) is a normal value. If the Z score is "greater than -2 and less than -1" or "greater than 1 and less than 2," the corresponding data point is a reasonably normal value. On the other hand, if the Z score is "-2 or less" or "2 or greater," the corresponding data point is an outlier.

[0052] As shown in Figure 7, outliers represent extremely fluctuating information within a data set, which can lead to misleading analytical decisions.

[0053] The information processing device 100 calculates the ReID similarity matrix S NN After excluding the outliers, the information processing device 100 calculates the ReID similarity matrix S NN The average value of each element in the first row of the matrix S m Set it as element m1 of the first row of

[0054] The information processing device 100 calculates the ReID similarity matrix S NN By repeating the above process for the information in the 2nd to Nth rows of the matrix S m Elements m2~m in the first row of N Set as.

[0055] Next, Matrix S m The second outlier removal process will be described using the matrix S m Elements m1~m N The information processing device 100 calculates the mean value μ and the standard deviation σ based on the formula (1). N The information processing device 100 calculates the Z-scores of the elements m1 to m N Among these, elements with a Z score of "-2 or less" or "2 or more" are identified as outliers.

[0056] The information processing device 100 performs a process of separating the tracklets corresponding to the elements identified as outliers from the tracklets 15. For example, the information processing device 100 performs a process of separating the tracklets corresponding to the elements m1 to m N If the element m3 is an outlier, the tracklet corresponding to the third frame is separated from the tracklet 15.

[0057] Fig. 8 is a diagram showing the processing results of the ID transfer error removal process. The tracklet 15 shown in Fig. 8 is the tracklet before the ID transfer error removal process is executed. When the information processing device 100 executes the above process on the tracklet 15, tracklets 15-1 and 15-2 that are outliers are identified.

[0058] The information processing device 100 separates the tracklets 15-1 and 15-2 from the tracklet 15. The information processing device 100 assigns new IDs to the tracklets 15-1 and 15-2.

[0059] The ID transfer error removal process executed by information processing device 100 has been described above.

[0060] Next, an example of the optimal hierarchical clustering process executed by the information processing device 100 will be described. Fig. 9 is a flowchart showing the processing procedure of the optimal hierarchical clustering process. As shown in Fig. 9, the information processing device 100 acquires a plurality of tracklets generated in the previous process (ID transfer error removal process) (step S20).

[0061] The information processing device 100 calculates the average value of the ReID feature of the tracklets for each ID (step S21). For example, in step S21, a tracklet for a certain ID is an integration of tracklets for each frame, and the average value of the ReID feature of the tracklets for each frame is calculated as the average value of the ReID feature of the tracklets for a certain ID. Here, the explanation will be given using the average value of the ReID feature, but a representative value such as a median may be used instead of the average value.

[0062] The information processing device 100 sets a minimum cluster threshold (step S22). The information processing device 100 clusters each tracklet based on the cluster threshold (step S23). For example, in step S22, if the similarity between the average value of the ReID feature of the tracklet with ID "1" and the average value of the ReID feature of the tracklet with ID "2" is equal to or greater than the cluster threshold, the information processing device 100 classifies the tracklet with ID "1" and the tracklet with ID "2" into the same cluster.

[0063] The information processing device 100 sets a label for each cluster (step S24). The information processing device 100 calculates the average value of the silhouette scores (step S25). The information processing device 100 associates the average value of the silhouette scores with the clustering result and stores them (step S26).

[0064] The information processing device 100 adds a predetermined value to the clustering threshold and updates the cluster threshold (step S27). If the cluster threshold has not reached the maximum value (step S28, No), the information processing device 100 proceeds to step S23.

[0065] On the other hand, if the cluster threshold reaches the maximum value (Yes at step S28), the information processing apparatus 100 generates a cluster with the maximum average silhouette score based on the list (step S29).

[0066] Here, an example of the silhouette score calculated by the information processing device 100 will be described. A clustering result with a high silhouette score indicates a better clustering result than a clustering result with a low silhouette score. For example, the silhouette score can be said to be a value indicating the quality of a cluster.

[0067] For example, the information processing device 100 calculates the average value s of the silhouette scores. avg,K is calculated based on equation (2).

[0068]

number

[0069] s included in Eq. (2) i is the score of data point i, and is given by equation (3). The data points for the silhouette score calculation represent tracklets.

[0070]

number

[0071] b included in equation (3) i is the cluster C that is closest to the cluster to which data point i (tracklet) belongs. k This is a value that indicates how far the distance is between the two points, and is expressed by equation (4). iThe larger the value of , the more likely the cluster to which data point i (tracklet) belongs is to be compared with other clusters C k It means being away from.

[0072]

number

[0073] d(i,j) in Equation (4) is the relationship between data point i and other clusters C k indicates the distance between the tracklet j in the cluster C and the tracklet j in the cluster C. k Each tracklet in is selected in turn.

[0074] a included in equation (3) i is the cluster C to which data point i (tracklet) belongs i is a value that indicates how closely tracklets are clustered together, and is expressed by equation (5). i The larger the value of, the more likely it is that the cluster C i This means that the tracklets are more densely packed.

[0075]

number

[0076] d(i,j) in equation (5) is the cluster C to which data point i belongs. i In the equation, the distance between data point i and other tracklets j is shown. i is fixed and j is the distance between data point i and other tracklets j. i Each tracklet in is selected in turn.

[0077] The information processing device 100 calculates the average silhouette score s for all clusters K based on the formula (2). avg,K For example, the information processing device 100 calculates the average silhouette score s for each cluster. avg,K The silhouette score for the cluster result may be calculated by further calculating the average value of

[0078] The optimal hierarchical clustering process executed by the information processing device 100 has been described above.

[0079] Next, a description will be given of an example of posture-oriented grouping processing executed by the information processing device 100. Fig. 10 is a flowchart showing the processing procedure of the posture-oriented grouping processing. As shown in Fig. 10, the information processing device 100 executes posture classification for each tracklet (step S30).

[0080] The information processing device 100 calculates a similarity score at the posture level (step S31). The information processing device 100 performs matching for a pair of two tracklets (step S32). The information processing device 100 performs sub-clustering (step S33).

[0081] Here, the pose classification described in step 30 of FIG. 10 will be described in more detail. For example, one of the clusters obtained as a result of the above-described optimal hierarchical clustering process includes multiple tracklets. Furthermore, each tracklet is formed by integrating multiple tracklets for each frame, and each tracklet is associated with an image of a person cut out using a bounding box or the like. For example, one tracklet is associated with one image of a person. In the following description, the image of a person associated with a tracklet will be referred to as a "person image."

[0082] 11 and 12 are diagrams for explaining posture classification. First, Fig. 11 will be explained. The information processing device 100 calculates a "human body orientation score" for each person image, and determines whether the person image is facing a "front direction," a "back direction," or a "side direction" based on the human body orientation score.

[0083] When the person image 20-1 faces forward, the information processing device 100 assigns the label "FRONT" to the person image 20-1. When the person image 20-2 faces backward, the information processing device 100 assigns the label "BACK" to the person image 20-2. When the person image 20-3 faces sideways, the information processing device 100 assigns the label "SIDE" to the person image 20-3.

[0084] We now move on to the description of Fig. 12. The information processing device 100 also performs the process described in Fig. 11 on other person images, thereby assigning one of the labels "FRONT," "BACK," and "SIDE" to the person images and classifying the person images to which the same label has been assigned.

[0085] A certain tracklet integrates tracklets for each frame, and each tracklet for each frame is associated with a person image. That is, one tracklet is associated with multiple person images. In the example shown in Fig. 12, multiple person images for one tracklet are shown as a person image group 20.

[0086] The information processing device 100 calculates a person score for each person image in the person image group 20, and assigns the label "FRONT" to person images 21-1, 21-2, and 21-3. The information processing device 100 assigns the label "BACK" to person images 22-1, 22-2, and 22-3. The information processing device 100 assigns the label "SIDE" to person images 23-1, 23-2, and 23-3.

[0087] The information processing device 100 may calculate the human body orientation score in any manner. For example, the information processing device 100 inputs a human image into a trained neural network or the like, and estimates the human's posture. The information processing device compares the estimated posture with a frontal posture template, a rearward posture template, and a sideward posture template, and calculates a frontal orientation likelihood score, a rearward posture likelihood score, and a sideward posture likelihood score as the human body orientation score. If the frontal orientation likelihood score is greater than the other scores, the information processing device 100 determines that the human image is oriented in a frontal posture.

[0088] The posture classification performed by the information processing device 100 has been described above.

[0089] Next, the calculation of the similarity score at the posture level, the matching of two tracklets as a pair, and the sub-clustering described in steps S31, S32, and S33 of FIG. 10 will be described in more detail.

[0090] First, the similarity score between frames (two tracklets to be compared) is calculated by the first score (S score1 ), Second Score (S score2 ), Third Score (S score3 The information processing device 100 calculates the first score based on the formula (6). nm is a matrix (n rows and m columns) that indicates the similarity of ReID features between frames.

[0091]

number

[0092] The information processing device 100 calculates the second score based on the formula (7). max is defined as in equation (8).

[0093]

number

number

[0094] The information processing device 100 calculates the third score based on the formula (9). i >th" is S nm is the number of elements whose values ​​are greater than a preset threshold th.

[0095]

number

[0096] Here, the information processing device 100 selects two tracklets to be used for calculating the similarity score from all tracklets included in the same cluster. For example, one of the tracklets is designated as “Trj p " and the other tracklet is "Trj qp The information processing device 100 assigns a certain tracklet "Trj1" to a sub-cluster (SubC1) in advance.

[0097] For example, Trj p is a tracklet that integrates tracklets for each frame, and the person images in the tracklet for each frame are classified (labeled) as either "FRONT", "BACK", or "SIDE" by the above-mentioned pose classification. In the following explanation, the pose classification result of the person images in the tracklet is also applied to the tracklet. For example, if the classification result of the person image in a certain tracklet is "FRONT", the classification result of the certain tracklet is also "FRONT". Trj qp The same is true for .

[0098] The information processing device 100 is p Among the tracklets for each frame included in p,FRONT ", Trj qp Among the tracklets for each frame included inqp,FRONT " and calculate the first score (FRONT), second score (FRONT), and third score (FRONT) between the frames.

[0099] The information processing device 100 is p Among the tracklets for each frame included in p,BACK ", Trj qp Among the tracklets for each frame included in qp,BACK " and calculate the first score (BACK), second score (BACK), and third score (BACK) between the frames.

[0100] The information processing device 100 is p Among the tracklets for each frame included in p,SIDE ", Trj qp Among the tracklets for each frame included in qp,SIDE Calculate the first score (SIDE), second score (SIDE), and third score (SIDE) between the frames.

[0101] The information processing device 100 calculates the average value (first score average value) of the first score (FRONT), the first score (BACK), and the first score (SIDE). The information processing device 100 calculates the average value (second score average value) of the second score (FRONT), the second score (BACK), and the second score (SIDE). The information processing device 100 calculates the average value (third score average value) of the third score (FRONT), the third score (BACK), and the third score (SIDE).

[0102] When the conditions that the first score average value is equal to or greater than the threshold value Th1, the second score average value is equal to or greater than the threshold value Th2, and the third score average value is equal to or greater than the threshold value Th3 are satisfied, the information processing device 100 calculates Trj based on the formula (10). p,SubCq Calculate the score.

[0103]

number

[0104] The information processing device 100 is pq By repeatedly executing the above process while changing the value of q from 1 to Y (q=1 to Y), Trj for each q p,SubCq The information processing device 100 calculates Trj p,SubCq (q=1 to Y) Based on the maximum value, Trj pq For example, the information processing device may identify a sub-cluster of Trj p,SubC2 When the maximum value is reached, Trj p is assigned to the sub-cluster (SubC2).

[0105] On the other hand, if the information processing device 100 does not satisfy the conditions that the first score average value is greater than or equal to the threshold value Th1, the second score average value is greater than or equal to the threshold value Th2, and the third score average value is greater than or equal to the threshold value Th3, it generates a new sub-cluster and increments the above-mentioned "Y".

[0106] The information processing device 100 is p By repeatedly executing the above process while changing the value of p from 2 to X (p=2 to X), all tracklets included in the same cluster are classified into any of the sub-clusters.

[0107] The information processing device 100 repeats the above process for other clusters, so that the tracklets classified into one cluster are classified into multiple sub-clusters.

[0108] Fig. 13 is a diagram for explaining sub-clustering. Each circle in Fig. 13 represents a tracklet. For example, it is assumed that the tracklets are classified into clusters C1, C2, and C3 by the optimal hierarchical clustering process. Clusters C1, C2, and C3 correspond to "representative clusters."

[0109] By the information processing device 100 performing the above process, each tracklet of the cluster C1 is divided into a subcluster SubC 1-1 ,SubC 1-2 Each tracklet in cluster C2 is separated into subcluster SubC 2-1 ,SubC 2-2 ,SubC 2-3 Each tracklet in cluster C3 is separated into subcluster SubC 3-1 ,SubC 3-2 ,SubC 3-3 is separated into

[0110] The posture-oriented grouping process executed by the information processing device 100 has been described above.

[0111] Next, an error suppression process executed by the information processing device 100 will be described. Fig. 14 is a flowchart showing the processing procedure of the error suppression process. As shown in Fig. 14, the information processing device 100 selects a plurality of tracklets that appear in a common camera (step S40).

[0112] The information processing device 100 divides the selected tracklet into a plurality of element tracklets (step S41). The information processing device 100 compares the ReID features of each element tracklet based on a common camera area (step S42). The information processing device 100 reintegrates the element tracklets based on the comparison result of the ReID features (step S43).

[0113] Fig. 15 is a diagram for explaining the error suppression processing. In the example shown in Fig. 15, a tracklet (Long_tri1) and a tracklet (Long_tri2) are used for explanation. Furthermore, for the tracklet (Long_tri1) and the tracklet (Long_tri2), an area A1 is assumed to be an area photographed by a common camera 31.

[0114] The information processing device 100 divides the tracklet in region A1 of the tracklet (Long_tri1) into element tracklets T_Trk1, T_Trk2, T_Trk3, and T_Trk4. The information processing device 100 divides the tracklet in region A1 of the tracklet (Long_tri2) into element tracklets C_Trk1, C_Trk2, and C_Trk3. For example, the information processing device 100 divides the tracklet into a plurality of element tracklets using the above-described ID transfer error removal process.

[0115] In Fig. 15, region A2 is an overlapping region in time. The element tracklets included in region A2 are element tracklets T_Trk3, T_Trk4, and element tracklet C_Trk1. The information processing device 100 compares the ReID features of the element tracklets T_Trk3 and T_Trk4 with the ReID features of the element tracklets T_Trk1 and T_Trk2 to calculate a similarity (similarity SA). The information processing device 100 compares the ReID feature of the element tracklet C_Trk1 with the ReID features of the element tracklets T_Trk1 and T_Trk2 to calculate a similarity (similarity SB).

[0116] The information processing device 100 determines the one with the greater similarity as the integration destination of the element tracklets T_Trk1 and T_Trk2. For example, if similarity SA>similarity SB, the information processing device 100 determines that the integration destination of the element tracklets T_Trk1 and T_Trk2 is element tracklets T_Trk3 and T_Trk4.

[0117] The information processing device 100 also compares the ReID feature amounts of the element tracklets T_Trk3 and T_Trk4 with the ReID feature amounts of the element tracklets C_Trk2 and C_Trk3 to calculate a similarity (similarity SC).The information processing device 100 compares the ReID feature amount of the element tracklet C_Trk1 with the ReID feature amounts of the element tracklets C_Trk2 and C_Trk3 to calculate a similarity (similarity SD).

[0118] The information processing device 100 determines the tracklet with the greater similarity as the tracklet to be combined with the element tracklets C_Trk2 and C_Trk3. For example, if similarity SC>similarity SD, the information processing device 100 determines the tracklets to be combined with the element tracklets T_Trk3 and T_Trk4.

[0119] Based on the above determination result, the information processing device 100 integrates the element tracklets C_Trk2 and C_Trk3 into element tracklets T_Trk3 and T_Trk4, thereby generating element tracklets T_Trk1, T_Trk2, T_Trk3, T_Trk4, C_Trk2, and C_Trk3 as new tracklets.

[0120] 16 is a diagram showing tracklets before and after the error suppression process is performed. For example, before the error suppression process is performed, tracklets 16 and 17 are generated by intra-camera tracking. Note that tracklet 16 is a combination of the tracklet of person 10a and the tracklet of person 10b. Also, tracklet 17 is a tracklet of person 10a, but tracklet 17 has been separated from tracklet 16.

[0121] In contrast to this, after the error suppression process is performed, the tracklet 18 of the person 10a and the tracklet 19 of the person 10b are generated with high accuracy.

[0122] The error suppression process executed by the information processing device 100 has been described above.

[0123] As described above, the information processing device 100 performs an ID transfer error elimination process on the multiple tracklets generated by preprocessing. After performing the ID transfer error elimination process, the information processing device 100 performs an optimal hierarchical clustering process, an attitude-oriented grouping process, and an error suppression process. The information processing device 100 repeatedly performs the optimal hierarchical clustering process, the attitude-oriented grouping process, and the error suppression process until the tracklets are stabilized. This enables trajectory information to be generated with high accuracy.

[0124] As described in the optimal hierarchical clustering process, the information processing device 100 calculates silhouette scores while changing the cluster threshold and generates multiple clusters (representative clusters) with the largest average silhouette score. This makes it possible to generate optimal clusters without adjusting a threshold specific to each data set.

[0125] As described in the error suppression processing, the information processing device 100 selects multiple tracklets that appear in a common camera, divides tracklets in regions where the selected tracklets overlap in time into multiple tracklets, and recombines each tracklet using the similarity of the ReID feature of each tracklet. This makes it possible to solve the problem of the conventional technology that tracklets that appear in the same camera and overlap in time remain connected.

[0126] As explained in the ID transfer error removal process, the information processing device 100 performs a process of dividing a target tracklet into multiple tracklets, identifying outliers in the feature quantities of each tracklet, and separating the tracklets corresponding to the identified outliers from the original tracklet, thereby eliminating ID transfer errors that may be generated in the preprocessing.

[0127] Next, a configuration example of the above-mentioned information processing device 100 will be described. Fig. 17 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 17, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0128] The communication unit 110 executes data communication with the camera 31, external devices, etc. via a network.

[0129] The input unit 120 inputs various types of information to the control unit 150 .

[0130] The display unit 130 displays information output from the control unit 150 .

[0131] The storage unit 140 includes a video DB 141 and a trajectory DB 142. The storage unit 140 is a memory or the like.

[0132] The video DB 141 stores data of frames in time series captured by the camera 31. The video DB 141 can store position information such as a person's BBOX and coordinates for identifying a person, in association with each frame.

[0133] The trajectory DB 142 stores various information related to tracklets. For example, in the trajectory DB 142, an ID for identifying the tracklet is assigned to each tracklet, and when a tracklet is integrated with multiple tracklets, information about the tracklet for each frame (position information, ReID feature amount), information about the BBOX extracted from each frame, etc. are associated with each tracklet.

[0134] The control unit 150 includes an acquisition unit 151, a preprocessing unit 152, and a global optimization processing unit 153. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0135] The acquisition unit 151 acquires video data (frames in time series) from the camera 31. The acquisition unit 151 stores the acquired video data in the video DB 141.

[0136] The preprocessing unit 152 acquires frames in time series from the video DB 141 and performs preprocessing. As preprocessing, the preprocessing unit 152 sequentially performs person detection processing, intra-camera tracking processing, and inter-camera tracking processing, and registers information on the generated tracklets in the trajectory DB 142.

[0137] For example, the preprocessing unit 152 analyzes time-series frames captured by the camera 31 and generates fragmentary tracklets of people included in each frame based on the analysis results. The preprocessing unit 152 also inputs information about the fragmentary tracklets into a neural network or the like to calculate features (ReID features) related to the appearance of the people.

[0138] Other explanations regarding the person detection processing, intra-camera tracking processing, and inter-camera tracking processing executed by the preprocessing unit 152 are the same as those explained with reference to FIG. 3A and the like.

[0139] The global optimization processor 153 performs a global optimization process based on the execution result of the preprocessing unit 152, thereby generating trajectory information that connects fragmentary tracklets of the same person. As the global optimization process, the global optimization processor 153 performs an ID transfer error elimination process, an optimal hierarchical clustering process, a posture-oriented grouping process, and an error suppression process. The global optimization processor 153 repeatedly performs the optimal hierarchical clustering process, the posture-oriented grouping process, and the error suppression process until the number of tracklets does not change. Note that the global optimization processor 153 may repeatedly perform the above processes a predetermined number of times. The global optimization processor 153 may register the tracklets that are the final processing results in the trajectory DB 142, or may output them to an external device.

[0140] For example, the global optimization processor 153 generates a plurality of representative clusters by clustering the identified fragmented trajectories using the similarity of representative values ​​of feature amounts (ReID feature amounts) of a plurality of frames. The global optimization processor 153 generates a plurality of sub-clusters by clustering each of the clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of person's postures. Based on the results of the generated sub-clusters, the global optimization processor 153 sets a plurality of trajectories corresponding to the sub-clusters as fragmented trajectories of the same person.

[0141] Other explanations regarding the ID transfer error elimination process, optimal hierarchical clustering process, attitude-oriented grouping process, and error suppression process executed by the global optimization processor 153 are the same as those explained with reference to FIG. 3B and the like.

[0142] Furthermore, the information processing device 100 generates a movement trajectory of the person from the entrance to the exit of the facility using the set fragmentary trajectories of the same person, and displays the generated movement trajectory of the person on a display device. For example, the information processing device 100 identifies each of the entrances and exits that the person passed through among multiple entrances and exits in a store, and draws the trajectory of the person in the passage area from the identified entrance to the exit on a floor map. More specifically, for example, the information processing device 100 identifies the entrance and exit by tracking the movement trajectory of the person in the store. Then, the information processing device 100 generates the movement trajectory of the person by connecting each of the multiple fragmentary trajectories.

[0143] The expressions "entrance" and "exit" of a facility include areas set aside for entering and exiting each floor in a one-story or multi-story facility. For example, the information processing device 100 performs the above-described process on time-series camera images of a person, thereby mapping the person's movement trajectory onto a two-dimensional floor map. For example, the information processing device 100 determines from the movement trajectory that the person passed through the entrance of the facility, looked at several shelves, paid at the cash register, and then passed through the exit of the facility.

[0144] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 18 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 18, the acquisition unit 151 of the information processing device 100 acquires frames in time series from the camera 31 and registers them in the video DB 141 (step S101).

[0145] The preprocessing unit 152 of the information processing device 100 executes preprocessing (step S102). The global optimization unit 153 of the information processing device 100 prepares data for integrating tracklets (step S103).

[0146] The global optimization processor 153 executes an ID transfer error removal process (step S104), and an optimal hierarchical clustering process (step S105).

[0147] The global optimization processor 153 executes an attitude-oriented grouping process (step S106), and an error suppression process (step S107).

[0148] If the number of tracklets is not stable (step S108, No), the global optimization processor 153 proceeds to step S105. On the other hand, if the number of tracklets is stable (step S108, Yes), the global optimization processor 153 outputs the final tracklet information (step S109).

[0149] Next, a description will be given of the processing procedure of the pre-processing shown in step S102 of Fig. 18. Fig. 19 is a flowchart showing the processing procedure of the pre-processing. The pre-processing unit 152 of the information processing device 100 detects a person (step S201). The pre-processing unit 152 extracts key points (step S202).

[0150] The pre-processing unit 152 calculates position information of a person (step S203). The pre-processing unit 152 executes intra-camera tracking processing (step S204). The pre-processing unit 152 executes inter-camera tracking processing (step S205).

[0151] The preprocessing unit 152 extracts ReID features of the tracklets (step S206). The preprocessing unit 152 executes multi-level DBSCAN (step S207). The preprocessing unit 152 merges tracklets classified into the same cluster (step S208).

[0152] The ID transfer error removal process shown in step S103 of Fig. 18 corresponds to the process shown in Fig. 5. The optimal hierarchical clustering process shown in step S105 of Fig. 18 corresponds to the process shown in Fig. 9. The attitude-oriented grouping process shown in step S106 of Fig. 18 corresponds to the process shown in Fig. 10. The error suppression process shown in step S107 of Fig. 18 corresponds to the process shown in Fig. 14.

[0153] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 performs an ID transfer error elimination process on a plurality of tracklets generated by preprocessing. After performing the ID transfer error elimination process, the information processing device 100 performs an optimal hierarchical clustering process, an attitude-oriented grouping process, and an error suppression process. The information processing device 100 repeatedly performs the optimal hierarchical clustering process, the attitude-oriented grouping process, and the error suppression process until the tracklets are stabilized. This enables trajectory information to be generated with high accuracy.

[0154] As described in the optimal hierarchical clustering process, the information processing device 100 calculates silhouette scores while changing the cluster threshold and generates multiple clusters (representative clusters) with the largest average silhouette score. This makes it possible to generate optimal clusters without adjusting a threshold specific to each data set.

[0155] As described in the error suppression processing, the information processing device 100 selects multiple tracklets that appear in a common camera, divides tracklets in regions where the selected tracklets overlap in time into multiple tracklets, and recombines each tracklet using the similarity of the ReID feature of each tracklet. This makes it possible to solve the problem of the conventional technology that tracklets that appear in the same camera and overlap in time remain connected.

[0156] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 20 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0157] 20, computer 200 has CPU 201 that executes various types of arithmetic processing, input device 202 that accepts data input from a user, and display 203. Computer 200 also has communication device 204 that exchanges data with camera 31, external devices, etc. via a wired or wireless network, and interface device 205. Computer 200 also has RAM 206 that temporarily stores various types of information, and hard disk drive 207. Each of devices 201 to 207 is connected to bus 208.

[0158] The hard disk drive 207 stores an acquisition program 207a, a pre-processing program 207b, and a global optimization processing program 207c. The CPU 201 reads out each of the programs 207a to 207c and loads them into the RAM 206.

[0159] The acquisition program 207a functions as the acquisition process 206a. The pre-processing program 207b functions as the pre-processing process 206b. The global optimization processing program 207c functions as the global optimization processing process 206c.

[0160] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the pre-processing process 206b corresponds to the processing of the pre-processing unit 152. The processing of the global optimization processing process 206c corresponds to the processing of the global optimization processing unit 153.

[0161] It should be noted that each of the programs 207a to 207c does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207c.

[0162] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0163] (Supplementary Note 1) Acquire an analysis result of an image frame including a person for each camera that captured the image frame, generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified plurality of fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. A trajectory generation program that causes a computer to execute a process.

[0164] (Appendix 2) A trajectory generation program according to appendix 1, characterized in that it causes a computer to repeatedly execute the process of generating the plurality of representative clusters, the process of generating the plurality of sub-clusters, and the process of setting them as fragmentary trajectories of the same person until the number of fragmentary trajectories of the same person no longer changes.

[0165] (Supplementary Note 3) The fragmentary trajectory has a plurality of sub-trajectories, 2. The trajectory generation program according to claim 1, further causing a computer to execute a process of identifying an outlier feature from a plurality of feature values ​​calculated from image frames corresponding to the plurality of partial trajectories, and separating the partial trajectories corresponding to the image frames from which the identified feature values ​​are calculated, from the fragmentary trajectories.

[0166] (Appendix 4) The trajectory generation program according to appendix 1, characterized in that the program further causes a computer to execute the following process: identify a target trajectory from the fragmentary trajectories of the same person set by the setting process, the target trajectory being a trajectory of a portion photographed by the same camera and overlapping in time; separate the target trajectory into a plurality of partial trajectories; and re-integrate the plurality of partial trajectories based on a plurality of feature amounts calculated from image frames corresponding to the plurality of partial trajectories.

[0167] (Appendix 5) The trajectory generation program according to appendix 1, characterized in that the process of generating the plurality of representative clusters generates a plurality of representative clusters by clustering the plurality of fragmentary trajectories using a cluster threshold that is a criterion for whether or not to classify the trajectories into the same representative cluster, and further calculates a silhouette score that indicates the quality of the plurality of generated representative clusters.

[0168] (Appendix 6) The trajectory generation program according to appendix 5, characterized in that the process of generating the plurality of representative clusters calculates the silhouette score while changing the cluster threshold, and generates the plurality of representative clusters with the maximum silhouette score.

[0169] (Appendix 7) A trajectory generation program as described in Appendix 1, characterized in that the computer is further made to execute a process of generating a movement trajectory of a person from the entrance of a facility to the exit of the facility using the set fragmentary trajectory of the same person, and displaying the generated movement trajectory of the person on a display device.

[0170] (Appendix 8) Acquire an analysis result of an image frame including a person for each camera that captured the image frame, generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified plurality of fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. A trajectory generation method characterized in that processing is executed by a computer.

[0171] (Appendix 9) The trajectory generation method described in Appendix 8, characterized in that a computer repeatedly executes the process of generating the plurality of representative clusters, the process of generating the plurality of sub-clusters, and the process of setting them as fragmentary trajectories of the same person until the number of fragmentary trajectories of the same person no longer changes.

[0172] (Supplementary Note 10) The fragmentary trajectory has a plurality of sub-trajectories; The trajectory generation method according to claim 8, further comprising the computer executing a process of identifying an outlier feature from a plurality of feature values ​​calculated from image frames corresponding to the plurality of partial trajectories, and separating the partial trajectories corresponding to the image frames from which the identified feature values ​​are calculated, from the fragmentary trajectories.

[0173] (Appendix 11) The trajectory generation method according to appendix 8, characterized in that the computer further executes the process of identifying a target trajectory from the fragmentary trajectories of the same person set by the setting process, which is a trajectory of a portion photographed by the same camera and which overlaps in time, separating the target trajectory into a plurality of partial trajectories, and reintegrating the plurality of partial trajectories based on a plurality of feature amounts calculated from image frames corresponding to the plurality of partial trajectories.

[0174] (Appendix 12) The trajectory generation method according to appendix 8, characterized in that the process of generating the plurality of representative clusters generates a plurality of representative clusters by clustering the plurality of fragmentary trajectories using a cluster threshold that is a criterion for whether or not to classify the trajectories into the same representative cluster, and further calculates a silhouette score that indicates the quality of the plurality of generated representative clusters.

[0175] (Appendix 13) The trajectory generation method according to appendix 12, characterized in that the process of generating the plurality of representative clusters calculates the silhouette score while changing the cluster threshold, and generates the plurality of representative clusters with the maximum silhouette score.

[0176] (Appendix 14) The trajectory generation method described in Appendix 8, characterized in that the computer further executes a process of generating a movement trajectory of the person from the entrance to the facility to the exit of the facility using the set fragmentary trajectory of the same person, and displaying the generated movement trajectory of the person on a display device.

[0177] (Appendix 15) Acquire an analysis result of an image frame including a person for each camera that captured the image frame, generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified plurality of fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. An information processing device having a control unit that executes processing.

[0178] (Appendix 16) The information processing device described in Appendix 15, characterized in that the control unit repeatedly executes the process of generating the plurality of representative clusters, the process of generating the plurality of sub-clusters, and the process of setting them as fragmentary trajectories of the same person until the number of fragmentary trajectories of the same person no longer changes.

[0179] (Supplementary Note 17) The fragmentary trajectory has a plurality of sub-trajectories, The information processing device according to claim 15, wherein the control unit further executes a process of identifying an outlier feature from a plurality of feature values ​​calculated from image frames corresponding to the plurality of partial trajectories, and separating the partial trajectories corresponding to the image frames from which the identified feature values ​​are calculated, from the fragmentary trajectories.

[0180] (Appendix 18) The information processing device described in Appendix 15 is characterized in that the control unit further performs a process of identifying a processing target trajectory from the fragmentary trajectories of the same person set by the setting process, which is a trajectory of a portion photographed by the same camera and which is a trajectory that overlaps in time, separating the processing target trajectory into a plurality of partial trajectories, and re-integrating the plurality of partial trajectories based on a plurality of feature amounts calculated respectively from image frames corresponding to the plurality of partial trajectories.

[0181] (Appendix 19) The information processing device described in Appendix 15, characterized in that the process of generating the plurality of representative clusters generates the plurality of representative clusters by clustering the plurality of fragmentary trajectories using a cluster threshold that serves as a criterion for whether or not to classify them into the same representative cluster, and further calculates a silhouette score that indicates the quality of the plurality of generated representative clusters.

[0182] (Appendix 20) The information processing device described in Appendix 19, characterized in that the process of generating the plurality of representative clusters calculates the silhouette score while changing the cluster threshold, and generates the plurality of representative clusters with the maximum silhouette score.

[0183] (Appendix 21) An information processing device as described in Appendix 15, characterized in that the computer is further made to execute a process of generating a movement trajectory of a person from the entrance of a facility to the exit of the facility using the set fragmentary trajectory of the same person, and displaying the generated movement trajectory of the person on a display device. [Explanation of symbols]

[0184] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video DB 142 Trajectory DB 150 control section 151 Acquisition Department 152 Preprocessing section 153 Global optimization processing unit

Claims

1. Acquire an analysis result of an image frame including a person for each camera that captured the image frame; generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. A trajectory generation program that causes a computer to execute a process.

2. The trajectory generation program according to claim 1, characterized in that the program causes a computer to repeatedly execute the process of generating the plurality of representative clusters, the process of generating the plurality of sub-clusters, and the process of setting the representative clusters as fragmentary trajectories of the same person until the number of fragmentary trajectories of the same person no longer changes.

3. the fragmentary trajectory has a plurality of sub-trajectories; 2. The trajectory generation program according to claim 1, further comprising the step of: identifying an outlier feature from a plurality of feature amounts calculated from image frames corresponding to the plurality of partial trajectories; and separating the partial trajectories corresponding to the image frames from which the identified feature amounts were calculated from the fragmentary trajectories.

4. The trajectory generation program according to claim 1, further comprising the steps of: identifying a target trajectory from the fragmentary trajectories of the same person set by the setting process, the target trajectory being a trajectory of a portion photographed by the same camera and overlapping in time; separating the target trajectory into a plurality of partial trajectories; and reintegrating the plurality of partial trajectories based on a plurality of feature amounts calculated from image frames corresponding to the plurality of partial trajectories.

5. The trajectory generation program according to claim 1, characterized in that the process of generating the plurality of representative clusters generates the plurality of representative clusters by clustering the plurality of fragmentary trajectories using a cluster threshold that serves as a criterion for whether or not to classify the trajectories into the same representative cluster, and further calculates a silhouette score that indicates the quality of the plurality of generated representative clusters.

6. 6. The trajectory generation program according to claim 5, wherein the process of generating the plurality of representative clusters calculates the silhouette score while changing the cluster threshold, and generates the plurality of representative clusters with the maximum silhouette score.

7. The trajectory generation program according to claim 1, further comprising a process of generating a movement trajectory of the person from the entrance of a facility to the exit of the facility using the set fragmentary trajectories of the same person, and displaying the generated movement trajectory of the person on a display device.

8. Acquire an analysis result of an image frame including a person for each camera that captured the image frame; generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. A trajectory generation method characterized in that processing is executed by a computer.

9. Acquire an analysis result of an image frame including a person for each camera that captured the image frame; generating a fragmentary trajectory of the person included in the image frame based on the acquired analysis result; Calculating a feature amount relating to the appearance of the person based on a plurality of image frames associated with the generated fragmentary trajectory; generating a plurality of representative clusters by clustering the identified fragmentary trajectories using the calculated similarity of the representative values ​​of the feature amounts of the plurality of image frames; generating a plurality of sub-clusters by clustering each of the plurality of clustered trajectories in each of the generated representative clusters using representative values ​​of a plurality of postures of the person; Based on the results of the generated sub-clusters, the trajectories corresponding to the sub-clusters are set as fragmentary trajectories of the same person. An information processing device having a control unit that executes processing.

Images