Program, tracking method, and tracking device

Abstract

To provide a program capable of tracking multiple targets in multiple frames constituting an image.SOLUTION: A program is provided that causes a computer to execute the processing of: displaying a user interface for setting weights on a screen of a display device; tracking a person or object using key points of the person or object appearing in an image based on the set weights; and in order to set the weights, setting a first index indicating how much importance is placed on the position of the target to be tracked, and setting a second index indicating how much importance is placed on the orientation of the target to be tracked.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates to a program, a tracking method, and a tracking device. [Background technology]

[0002] Person tracking technology is a technology that detects people from image frames (hereinafter also referred to as frames) that make up video captured by a surveillance camera or the like, and tracks the detected people in the video. In person tracking technology, for example, each detected person is identified by facial recognition or the like, and an identification number is assigned, and the person with the assigned identification number is tracked in the video.

[0003] Patent Document 1 discloses a posture estimation device that estimates a three-dimensional posture based on two-dimensional joint positions. The device of Patent Document 1 calculates feature amounts for position candidates of a tracking target from an input image, and estimates the position of the tracking target based on similarity weights obtained by comparing the feature amounts with template data. The device of Patent Document 1 sets position candidates of the tracking target based on the similarity weights and three-dimensional motion model data. The device of Patent Document 1 tracks the position of the tracking target by repeating the process of estimating the position of the tracking target and setting position candidates of the tracking target multiple times. The device of Patent Document 1 also estimates the three-dimensional posture of the posture estimation target by referring to estimated position information of the tracking target and three-dimensional motion model data.

[0004] Patent Document 2 discloses an image processing device that identifies people from an image. The device in Patent Document 2 compares people in an input image with registered people based on the pose similarity between the pose of the person in the input image and the pose of the person in the reference image, the feature amount of the input image, and the feature amount of the reference image for each person.

[0005] Non-Patent Document 1 discloses a technique for pose tracking of multiple people in a video. The method in Non-Patent Document 1 samples a pair of pose estimates from different frames of the video and performs a binary classification to determine whether one pose follows another pose in time. Furthermore, the method in Non-Patent Document 1 improves the pose estimation method using a parameter-free keypoint refinement technique.

[0006] Non-Patent Document 2 discloses a related technology for estimating the skeletons of multiple people captured in a two-dimensional image. The technology in Non-Patent Document 2 uses a technique called Part Affinity Fields to estimate the skeletons of multiple people captured in a two-dimensional image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-092876 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-097549 [Non-patent literature]

[0008] [Non-Patent Document 1] Michael Snower, Asim Kadav, Farley Lai, Hans Peter Graf, “15 Keypoints Is All You Need”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp.6738-6748 [Non-patent document 2] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp.7291-7299. Summary of the Invention [Problem to be solved by the invention]

[0009] The method in Patent Document 1 can estimate the 3D posture of a single person from information about the 2D joint positions, but cannot estimate the 3D postures of multiple people. Furthermore, the method in Patent Document 1 cannot determine whether people appearing in different frames are the same person based on the estimated 3D postures, and therefore cannot track people across frames.

[0010] In the method of Patent Document 2, people are registered based on the similarity between the estimated posture and the feature amount of a reference image for each posture of each person that is registered in advance. Therefore, in the method of Patent Document 2, it is not possible to track people based on their posture unless a reference image for each posture of each person is stored in a database.

[0011] In the method of Non-Patent Document 1, posture tracking is performed using deep learning, so tracking accuracy depends on the learning data. Therefore, in the method of Non-Patent Document 1, if conditions such as the degree of crowding, angle of view, distance between the camera and person, and frame rate differ from the learned conditions, it is not possible to continue tracking based on the posture of the tracking target.

[0012] An object of the present disclosure is to provide a program, a tracking method, and a tracking device that can track multiple tracking targets in multiple frames that make up a video. [Means for solving the problem]

[0013] A program according to one embodiment of the present disclosure causes a computer to execute the following processes: displaying a user interface for setting weights on a screen of a display device; tracking a person or object appearing in an image using key points of the person or object based on the set weights; and, in order to set the weights, setting a first indicator that indicates how much importance is to be placed on the position of the tracked object; and setting a second indicator that indicates how much importance is to be placed on the orientation of the tracked object.

[0014] In a tracking method according to one embodiment of the present disclosure, a computer displays a user interface for setting weights on the screen of a display device, and tracks the person or object using key points of the person or object appearing in an image based on the set weights. To set the weights, a first indicator is set to indicate how much importance is placed on the position of the tracked object, and a second indicator is set to indicate how much importance is placed on the orientation of the tracked object.

[0015] A tracking device according to one embodiment of the present disclosure includes a means for displaying a user interface for setting weights on a screen of a display device, a means for tracking a person or object using key points of the person or object appearing in an image based on the set weights, and a means for setting a first indicator representing how much importance is to be placed on the position of the tracked object, and a means for setting a second indicator representing how much importance is to be placed on the orientation of the tracked object, in order to set the weights. [Effects of the Invention]

[0016] According to the present disclosure, it is possible to provide a program, a tracking method, and a tracking device that can track multiple tracking targets in multiple frames that make up a video. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of a tracking system according to a first embodiment. [Figure 2]FIG. 2 is a conceptual diagram for explaining an example of key points extracted by a tracking device of the tracking system according to the first embodiment. [Figure 3] FIG. 2 is a conceptual diagram for explaining a tracking process performed by a tracking device of the tracking system according to the first embodiment. [Figure 4] 4 is a table showing an example of scores used by the tracking device of the tracking system according to the first embodiment to track a tracking target. [Figure 5] 4 is a flowchart for explaining an example of an outline of the operation of the tracking system according to the first embodiment. [Figure 6] 4 is a flowchart illustrating an example of a tracking process performed by a tracking device of the tracking system according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the configuration of a tracking system according to a second embodiment. [Figure 8] FIG. 10 is a conceptual diagram for explaining an example of a skeleton line extracted by a tracking device of a tracking system according to a second embodiment. [Figure 9] 10 is a flowchart illustrating an example of a tracking process performed by a tracking device of a tracking system according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a tracking system according to a third embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a terminal device of a tracking system according to a third embodiment. [Figure 12] FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment to track a tracking target. [Figure 13] FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment to track a tracking target. [Figure 14]FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment to track a tracking target. [Figure 15] FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting the designation of key points used by a tracking device of a tracking system according to a third embodiment to generate posture information. [Figure 16] FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting the designation of key points used by a tracking device of a tracking system according to a third embodiment to generate posture information. [Figure 17] FIG. 10 is a conceptual diagram showing an example of displaying, on the screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment to track a tracking target. [Figure 18] 11 is a flowchart showing an example of a process in which a tracking device of a tracking system according to a third embodiment accepts settings via a terminal device. [Figure 19] FIG. 10 is a block diagram showing an example of the configuration of a tracking device according to a fourth embodiment. [Figure 20] FIG. 2 is a block diagram showing an example of a hardware configuration for realizing the tracking device according to each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. However, the embodiments described below are limited in a manner that is technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In all drawings used to describe the following embodiments, the same reference numerals are used for similar parts unless otherwise specified. Furthermore, in the following embodiments, repeated explanations of similar configurations and operations may be omitted.

[0019] (First embodiment) First, a tracking system according to a first embodiment will be described with reference to the drawings. The tracking system of this embodiment detects a tracking target such as a person from image frames (also called frames) constituting a video captured by a surveillance camera or the like, and tracks the detected tracking target between frames. Note that there are no particular limitations on the tracking target of the tracking system of this embodiment. For example, the tracking system of this embodiment may track not only people, but also animals such as dogs and cats, moving objects such as automobiles, bicycles, and robots, and any other object. Below, an example of tracking a person in a video will be described.

[0020] (composition) Fig. 1 is a block diagram showing an example of the configuration of a tracking system 1 of this embodiment. The tracking system 1 includes a tracking device 10, a monitoring camera 110, and a terminal device 120. Although Fig. 1 shows only one monitoring camera 110 and one terminal device 120, there may be multiple monitoring cameras 110 and multiple terminal devices 120.

[0021] The surveillance camera 110 is placed in a position where it can capture the area to be monitored. The surveillance camera 110 has the functions of a general surveillance camera. The surveillance camera 110 may be a camera that is sensitive in the visible range, or an infrared camera that is sensitive in the infrared range. For example, the surveillance camera 110 may be placed on a street where there are people or indoors. There are no particular limitations on the method of connection between the surveillance camera 110 and the tracking device 10. For example, the surveillance camera 110 may be connected to the tracking device 10 via a network such as the Internet or an intranet. Alternatively, the surveillance camera 110 may be connected to the tracking device 10 by a cable or the like.

[0022] The surveillance camera 110 captures images of the monitored area at set intervals and generates video data. The surveillance camera 110 outputs the generated video data to the tracking device 10. The video data is made up of multiple frames captured at set intervals. For example, the surveillance camera 110 may output video data made up of multiple frames to the tracking device 10, or may output each of the multiple frames to the tracking device 10 in the chronological order in which they were captured. There are no particular limitations on the timing at which the surveillance camera 110 outputs data to the tracking device 10.

[0023] The tracking device 10 includes an image acquisition unit 11, a memory unit 12, a detection unit 13, an extraction unit 15, a posture information generation unit 16, a tracking unit 17, and a tracking information output unit 18. For example, the tracking device 10 is placed on a server or a cloud. For example, the tracking device 10 may be provided as an application installed on the terminal device 120.

[0024] In this embodiment, the tracking device 10 tracks the tracking target between two verification target frames (hereinafter referred to as verification frames). The verification frame that precedes the target in chronological order is referred to as the preceding frame, and the verification frame that follows the target is referred to as the following frame. The tracking device 10 tracks the target between frames by matching the target included in the preceding frame with the target included in the following frame. The preceding frame and the following frame may be consecutive frames, or may be separated by several frames.

[0025] The video acquisition unit 11 acquires video data to be processed from the surveillance camera 110. The video acquisition unit 11 stores the acquired video data in the storage unit 12. There are no particular limitations on the timing at which the tracking device 10 acquires data from the surveillance camera 110. For example, the video acquisition unit 11 may acquire video data consisting of multiple frames from the surveillance camera 110, or may acquire each of the multiple frames from the surveillance camera 110 in the order in which they were captured. Note that the video acquisition unit 11 may acquire not only video data generated by the surveillance camera 110, but also video data stored in an external storage device, server, or the like (not shown).

[0026] The storage unit 12 stores the video data generated by the surveillance camera 110. The frames constituting the video data stored in the storage unit 12 are acquired by the tracking unit 17 and used to track the tracking target.

[0027] The detection unit 13 acquires a verification frame from the storage unit 12. The detection unit 13 detects tracking targets from the acquired verification frame. The detection unit 13 assigns IDs (Identifiers) to all tracking targets detected from the verification frame. In the following, it is assumed that tracking targets detected from the preceding frame have been assigned formal IDs. The detection unit 13 assigns temporary IDs to tracking targets detected from the subsequent frame.

[0028] For example, the detection unit 13 detects the tracking target from the verification frame using a detection technique such as background subtraction. For example, the detection unit 13 may detect the tracking target from the verification frame using a detection technique (for example, a detection algorithm) that uses a feature such as a motion vector. The tracking target detected by the detection unit 13 is a person or a moving object (also called a moving body). For example, if the tracking target is a person, the detection unit 13 detects the tracking target from the verification frame using a face detection technique. For example, the detection unit 13 may detect the tracking target from the verification frame using a human body detection technique or an object detection technique. For example, the detection unit 13 may detect an object that is not a moving object but whose feature such as shape, pattern, or color changes at a certain position.

[0029] The extraction unit 15 extracts multiple key points from the tracking target detected from the verification frame. For example, if the tracking target is a person, the extraction unit 15 extracts the positions of the person's head, joints, limbs, etc. included in the verification frame as key points. For example, the extraction unit 15 detects the skeletal structure of the person included in the verification frame and extracts key points based on the detected skeletal structure. For example, the extraction unit 15 detects the skeletal structure of the person included in the verification frame based on features of the person's joints, etc., using a skeletal estimation technique using machine learning. For example, the extraction unit 15 detects the skeletal structure of the person included in the verification frame using the skeletal estimation technique disclosed in Non-Patent Document 2 (Non-Patent Document 2: Z. Cao et al., The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7291-7299).

[0030] For example, the extraction unit 15 assigns a number from 1 to n (n is a natural number) to each extracted keypoint, such as 0 for the right shoulder and 1 for the right elbow. For example, if the kth keypoint of a person detected from the verification frame is not extracted, the keypoint is deemed undetected (k is a natural number between 1 and n).

[0031] FIG. 2 is a conceptual diagram illustrating keypoints when the tracking target is a person. FIG. 2 is a diagram of a person viewed from the front. In the example of FIG. 2, 14 keypoints are set for one person. HD is a keypoint set for the head. N is a keypoint set for the neck. RS and LS are keypoints set for the right and left shoulders, respectively. RE and LE are keypoints set for the right and left elbows, respectively. RH and LH are keypoints set for the right and left hands, respectively. RW and LW are keypoints set for the right and left hips, respectively. RK and LK are keypoints set for the right and left knees, respectively. RF and LF are keypoints set for the right and left feet, respectively. Note that the number of keypoints set for one person is not limited to 14. Furthermore, the positions of each keypoint are not limited to the example of FIG. 2. For example, face detection may also be used, and keypoints may be set for the eyes, eyebrows, nose, mouth, etc. according to the face detection.

[0032] The posture information generating unit 16 generates posture information of all the tracked objects detected from the verification frame based on the key points extracted by the extracting unit 15. The posture information is position information of each key point of each tracked object in the verification frame. When tracking a tracked object between two verification frames, the posture information f of the tracked object detected from the previous frame is used. p is expressed by the following equation 1, and the posture information of the person detected from the subsequent frame f s is expressed by the following equation 2.

number

number

[0033] The tracking unit 17 tracks the tracking target between frames using the posture information generated for the tracking target detected from the previous frame and the posture information generated for the tracking target detected from the previous frame. The tracking unit 17 tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of at least two frames. The tracking unit 17 tracks the tracking target by assigning the ID of the tracking target detected from the previous frame to the tracking target detected from the subsequent frame that is identified as the tracking target detected from the previous frame. Note that if a tracking target corresponding to the tracking target detected from the subsequent frame is not detected in the previous frame, the temporary ID assigned to the tracking target detected from the subsequent frame may be made the official ID, or a new ID may be assigned as the official ID.

[0034] For example, the tracker 17 calculates the position of the keypoint of the tracked target using coordinate information in the frame. The tracker 17 calculates the distance in a specific direction between the position of a reference keypoint and the head keypoint as the orientation of the tracked target. For example, the tracker 17 calculates the distance from the neck keypoint to the head keypoint in the horizontal direction (x direction) of the screen as the orientation of the tracked target. The tracker 17 calculates the distances related to the positions and orientations of all tracked targets detected in the previous frame and all tracked targets detected in the subsequent frame in a brute-force manner. The tracker 17 calculates the sum of the distances related to the positions and the distances related to the orientations calculated between all tracked targets detected in the previous frame and all tracked targets detected in the subsequent frame as a score. The tracker 17 tracks the tracked targets by assigning the same ID to the tracked target with the smallest score among the pair of tracked targets detected in the previous frame and the tracked target detected in the subsequent frame.

[0035] Positional distance D p is the weighted average of the absolute value of the difference in coordinate values ​​of each keypoint extracted from the tracked object being compared in the previous and subsequent frames. The weight for the position of each keypoint is w k Then, the tracking unit 17 calculates the distance D p Calculate.

number

[0036] Orientation distance D d is the weighted average of the absolute value of the difference in the x-coordinate of each keypoint extracted from the tracked object being compared in the previous and subsequent frames relative to the reference point. The neck keypoint is taken as the reference point, and the reference point in the previous frame is taken as x p_neck and the reference point of the subsequent frame is expressed as x s_neck and the weight for each keypoint position is w k Then, the tracking unit 17 calculates the distance D in terms of the direction using the following equation 4: d Calculate.

number

[0037] Positional distance D p and the distance D d The sum of these is the score S. The tracking unit 17 calculates the score S using the following formula 5.

number

[0038] 3A is a conceptual diagram for explaining an example of key point extraction, an example of key point (skeleton line) extraction used for tracking, and an example of ID allocation, all by the tracking unit 17. In FIG. 3, the upper diagram corresponds to the preceding frame, and the lower diagram corresponds to the following frame.

[0039] Figure 3(A) shows an example of extracting keypoints from a tracked object contained in a verification frame. Figure 3(A) shows the outline of the tracked object and the line segments connecting the keypoints extracted from the tracked object. In Figure 3(A), two people are included in the preceding and succeeding frames. The two people extracted from the preceding frame are assigned IDs P_ID4 and P_ID8, respectively. The two people extracted from the succeeding frame are assigned IDs S_ID1 and S_ID2, respectively. The IDs assigned to the two people extracted from the succeeding frame are temporary IDs.

[0040] 3B is a diagram showing only the line segments (also called skeleton lines) connecting the key points used for tracking the tracking target among the key points extracted from the tracking target. For example, the key points used for tracking may be set in advance or may be set for each verification.

[0041] FIG. 4 is a table summarizing scores calculated by the tracker 17 for the example of FIG. 3. The score between the tracked object with S_ID1 detected from the subsequent frame and P_ID4 detected from the preceding frame is 0.2. The score between the tracked object with S_ID1 detected from the subsequent frame and P_ID8 detected from the preceding frame is 1.5. The score between the tracked object with S_ID2 detected from the subsequent frame and P_ID4 detected from the preceding frame is 1.3. The score between the tracked object with S_ID2 detected from the subsequent frame and P_ID8 detected from the preceding frame is 0.3. That is, the tracked object with the smallest score relative to the tracked object with S_ID1 is P_ID4. Also, the tracked object with the smallest score relative to the tracked object with S_ID2 is P_ID8. The tracker 17 assigns an ID of P_ID4 to the tracked object with S_ID1, and an ID of P_ID8 to the tracked object with S_ID2.

[0042] Figure 3(C) shows a situation in which the same ID is assigned to the same tracking target detected in the preceding and following frames based on the score values ​​in Figure 4. In this way, a tracking target assigned the same ID in the preceding and following frames is further referenced in the following frame.

[0043] The tracking information output unit 18 outputs tracking information including the tracking result by the tracking unit 17 to the terminal device 120. For example, the tracking information output unit 18 outputs an image in which key points and a skeleton line are superimposed on the tracking target detected from the verification frame as tracking information. For example, the tracking information output unit 18 outputs an image in which key points and a skeleton line are displayed at the position of the tracking target detected from the verification frame as tracking information. For example, the image output from the tracking information output unit 18 is displayed on the display unit of the terminal device 120.

[0044] The terminal device 120 acquires tracking information for each of multiple frames constituting the video data from the tracking device 10. The terminal device 120 displays an image including the acquired tracking information on a screen. For example, the terminal device 120 displays the image including the tracking information on a screen according to preset display conditions. For example, the preset display conditions are conditions for displaying images including tracking information corresponding to a predetermined number of consecutive frames including a preset frame number in chronological order. For example, the preset display conditions are conditions for displaying images including tracking information corresponding to multiple frames generated during a predetermined time period including a preset time in chronological order. Note that the display conditions are not limited to the examples given here, as long as they are set in advance.

[0045] (operation) Next, an example of the operation of the tracking device 10 will be described with reference to the drawings. Below, an overview of the processing by the tracking device 10 and details of the tracking processing by the tracking unit 17 of the tracking device 10 will be described.

[0046] Fig. 5 is a flowchart for explaining the operation of the tracking device 10. In Fig. 5, first, the tracking device 10 acquires a verification frame (step S11). The tracking device 10 may acquire a verification frame that has been stored in advance, or may acquire a newly input verification frame.

[0047] If a tracking target is detected from the verification frame (Yes in step S12), the tracking device 10 assigns an ID to the detected tracking target (step S13). At this time, the ID assigned to the tracking target by the tracking device 10 is a temporary ID. On the other hand, if a tracking target is not detected from the verification frame (No in step S12), the process proceeds to step S18.

[0048] After step S13, the tracking device 10 extracts key points from the detected tracking target (step S14). If multiple tracking targets are detected, the tracking device 10 extracts key points for each of the detected tracking targets.

[0049] Next, the tracking device 10 generates posture information for each tracking target (step S15). The posture information is information obtained by integrating position information of key points extracted for each tracking target for each tracking target. When multiple tracking targets are detected, the tracking device 10 generates posture information for each detected tracking target.

[0050] If there is a preceding frame (Yes in step S16), the tracking device 10 executes tracking processing (step S17). On the other hand, if there is no preceding frame (No in step S16), the tracking device 10 proceeds to step S18. Details of the tracking processing will be explained later using the flowchart in FIG.

[0051] If there are any further subsequent frames (Yes in step S18), the process returns to step S11. On the other hand, if there are no further subsequent frames (No in step S18), the process according to the flowchart in FIG.

[0052] Fig. 6 is a flowchart for explaining the tracking process by the tracking unit 17 of the tracking device 10. In Fig. 6, first, the tracking unit 17 calculates the distances related to the positions and orientations between the tracking targets for the preceding and subsequent frames (step S171).

[0053] Next, the tracking unit 17 calculates a score between the tracking targets from the distances between the tracking targets regarding their positions and orientations (step S172). For example, the tracking unit 17 calculates the score as the sum of the distance between the tracking targets regarding their positions and the distance between the tracking targets regarding their orientations.

[0054] Next, the tracking unit 17 selects an optimal combination of tracking targets according to the scores between the tracking targets (step S173). For example, the tracking unit 17 selects a combination of tracking targets that has the smallest score from the preceding and succeeding frames.

[0055] Next, the tracking unit 17 assigns an ID to the tracking target detected from the subsequent frame according to the selected combination (step S174). For example, the tracking unit 17 assigns the same ID to the combination of tracking targets that has the smallest score in the preceding and subsequent frames.

[0056] As described above, the tracking device of the tracking system of this embodiment includes a detection unit, an extraction unit, a posture information generation unit, and a tracking unit. The detection unit detects a tracking target from at least two frames constituting video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0057] The tracking device of this embodiment tracks a tracking target based on the position and orientation of the tracking target's posture information. If the tracking target is tracked based only on its position, there is a possibility that the identification numbers of different tracking targets will be swapped when multiple tracking targets pass each other. The tracking device of this embodiment tracks a tracking target based not only on its position but also on its orientation, reducing the possibility that the identification numbers of different tracking targets will be swapped when multiple tracking targets pass each other. Therefore, the tracking device of this embodiment makes it possible to track multiple tracking targets across multiple frames based on their postures. In other words, the tracking device of this embodiment can track multiple tracking targets based on their postures in multiple frames constituting a video.

[0058] Furthermore, the tracking device of this embodiment can track a tracking target based on its posture, even if reference images for each posture of each tracking target are not stored in a database. Furthermore, the tracking device of this embodiment does not reduce tracking accuracy even if conditions such as the degree of congestion, angle of view, distance between the camera and the tracking target, and frame rate differ from the learned conditions. In other words, this embodiment can track a tracking target in frames constituting a video with high accuracy. The tracking device of this embodiment can be applied to, for example, monitoring the movement of people in towns, public facilities, stores, etc.

[0059] In one aspect of this embodiment, the tracking unit calculates a score according to the distance related to the position and orientation of the tracking target detected from each of at least two frames based on the posture information. The tracking unit tracks the tracking target based on the calculated score. According to this aspect, by tracking the tracking target based on the score according to the distance related to the position and orientation of the tracking target, it is possible to continuously track multiple tracking targets between frames constituting a video.

[0060] In one aspect of the present embodiment, the tracking unit tracks a pair of tracking targets detected from at least two frames, with the pair having the smallest score, as the same tracking target. According to this aspect, by identifying the pair having the smallest score as the same tracking target, it is possible to more continuously track the tracking target between frames constituting the video.

[0061] In one aspect of this embodiment, the tracking unit calculates, for the tracked object detected from each of the at least two frames, a weighted average of absolute values ​​of differences in coordinate values ​​of keypoints as the position distance. The tracking unit calculates, for the tracked object detected from each of the at least two frames, a weighted average of absolute values ​​of differences in coordinate values ​​relative to a reference point in a specific direction of the keypoints as the orientation distance. The tracking unit calculates, for the tracked object detected from each of the at least two frames, a sum of the position distance and the orientation distance as the score. According to this aspect, weights for position and orientation are clearly defined, and the tracked object can be appropriately tracked between frames according to the weights.

[0062] In one aspect of this embodiment, the tracking device includes a tracking information output unit that outputs tracking information related to tracking of the tracked object. The tracking information is, for example, an image in which key points are displayed at the positions of the tracked object detected from the verification frame. According to this aspect, by displaying an image in which the tracking information is superimposed on the tracked object on the screen of a display device, it becomes easier to visually grasp the posture of the tracked object.

[0063] (Second embodiment) Next, a tracking system according to a second embodiment will be described with reference to the drawings. The tracking system of this embodiment differs from the first embodiment in that the distances between tracked objects in terms of their positions and orientations are normalized by the size of the tracked objects in the frame.

[0064] (composition) Fig. 7 is a block diagram showing an example of the configuration of a tracking system 2 of this embodiment. The tracking system 2 includes a tracking device 20, a monitoring camera 210, and a terminal device 220. Although Fig. 7 shows only one monitoring camera 210 and one terminal device 220, there may be multiple monitoring cameras 210 and multiple terminal devices 220. Each of the monitoring camera 210 and the terminal device 220 is similar to each of the monitoring camera 110 and the terminal device 120 of the first embodiment, and therefore detailed description thereof will be omitted.

[0065] The tracking device 20 has an image acquisition unit 21, a memory unit 22, a detection unit 23, an extraction unit 25, a posture information generation unit 26, a tracking unit 27, and a tracking information output unit 28. For example, the tracking device 20 is placed on a server or a cloud. For example, the tracking device 20 may be provided as an application installed on a terminal device 220. Each of the image acquisition unit 21, the memory unit 22, the detection unit 23, the extraction unit 25, the posture information generation unit 26, and the tracking information output unit 28 is similar to the corresponding configuration in the first embodiment, and therefore detailed description thereof will be omitted.

[0066] The tracking unit 27 tracks the tracking target between frames using the posture information generated for the tracking target detected from the previous frame and the posture information generated for the tracking target detected from the previous frame. The tracking unit 27 tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of at least two frames. The tracking unit 27 tracks the tracking target by assigning the ID of the tracking target detected from the previous frame to the tracking target detected from the subsequent frame that is identified as the tracking target detected from the previous frame. Note that if a tracking target corresponding to the tracking target detected from the subsequent frame is not detected in the previous frame, the temporary ID assigned to the tracking target detected from the subsequent frame may be made the official ID, or a new ID may be assigned as the official ID.

[0067] For example, the tracking unit 27 calculates the distances related to the positions and orientations normalized by the size of the tracked objects for all tracked objects detected in the previous frame and all tracked objects detected in the subsequent frame in a brute-force manner. The tracking unit 27 calculates a normalized score by summing the distances related to the positions and orientations normalized by the size of the tracked objects calculated for all tracked objects detected in the previous frame and all tracked objects detected in the subsequent frame. The tracking unit 27 tracks the tracked objects by assigning the same ID to the tracked objects with the smallest normalized score among pairs of tracked objects detected in the previous frame and the subsequent frame. For example, if the tracked person in the frame is walking upright, the size can be estimated by surrounding the person with a frame such as a rectangle. However, if the tracked person in the frame is sitting or frequently changing direction, it is difficult to estimate the size simply by surrounding the person with a frame such as a rectangle. In such cases, the size can be estimated based on the skeletal structure of the tracked person, as described below.

[0068] FIG. 8 is a conceptual diagram for explaining skeletal lines used by the tracking unit 27 when estimating the size of a tracking target (person). Skeleton lines are line segments connecting specific key points. FIG. 8 is a diagram of a person viewed from the front. In the example of FIG. 8, 14 key points are set for one person, and 15 skeletal lines are set. L1 is a line segment connecting HD and N. L21 is a line segment connecting N and RS, and L22 is a line segment connecting N and LS. L31 is a line segment connecting RS and RE, and L32 is a line segment connecting LS and LE. L41 is a line segment connecting RE and RH, and L42 is a line segment connecting LE and LH. L51 is a line segment connecting N and RW, and L52 is a line segment connecting N and LW. L61 is a line segment connecting RW and RK, and L62 is a line segment connecting LW and LK. L71 is a line segment connecting RK and RF, and L42 is a line segment connecting LK and LF. The number of key points set for one person is not limited to 14. The number of skeleton lines set for one person is not limited to 13. The positions of the key points and skeleton lines are not limited to the example in FIG. 8.

[0069] The tracking unit 27 calculates the person's standing height (called height pixel count) based on the skeleton lines corresponding to the person in the verification frame. The height pixel count corresponds to the person's height in the verification frame (the person's total body length in two frames). The tracking unit 27 calculates the height pixel count (number of pixels) from the length of each skeleton line in the frame.

[0070] For example, the tracking unit 27 estimates the height pixel number using the length of the skeleton line from the head (HD) to the feet (RF, LF). For example, the tracking unit 27 estimates the height pixel number by using the sum H of the lengths of L1, L51, L61, and L71 in the verification frame among the skeleton lines extracted from the person in the verification frame. R For example, the tracking unit 27 calculates the sum H of the lengths L1, L52, L62, and L72 of the skeleton lines extracted from the person in the verification frame. L For example, the tracking unit 27 calculates the sum H of the lengths L1, L51, L61, and L71 in the verification frame. R and the sum of the lengths of L1, L52, L62, and L72 in the validation frame, H L For example, in order to calculate a more accurate height pixel count, the tracking unit 27 may calculate the height pixel count after correcting each skeleton line with a correction coefficient for correcting the inclination, posture, etc. of each skeleton line.

[0071] For example, the tracking unit 27 may estimate the number of height pixels using the length of each skeletal line based on the relationship between the length of each skeletal line and the height of an average person. For example, the length of the skeletal line (L1) connecting the head (HD) and neck (N) is approximately 20 percent of the height. For example, the length of the skeletal line connecting the elbows (RE, LE) and hands (RH, LH) is approximately 25 percent of the height. In this way, by storing the ratio of the length of each skeletal line to the height in a memory unit (not shown), the number of height pixels corresponding to the height of a person detected from the verification frame can be estimated based on the length of each skeletal line of the person. Furthermore, the ratio of the length of each skeletal line to the height of an average person tends to vary depending on the person's age. Therefore, the ratio of the length of each skeletal line to the height of an average person may be stored in the memory unit for each age of the person. For example, if the ratio of the length of each skeletal line to the height of an average person is stored in the memory unit, when an upright person is detected from the verification frame, the person's approximate age can be estimated based on the length of each skeletal line. The above-described method for estimating the number of height pixels based on the length of the skeleton line is merely an example, and does not limit the method for estimating the number of height pixels by the tracking unit 27.

[0072] The tracking unit 27 calculates the distance D p and the distance D d Here, for the person being compared, the height detected from the previous frame is normalized as H p , and the height detected from the subsequent frame is H s The tracking unit 27 calculates the normalized distance ND p is calculated using the following formula 6, and the normalized distance ND d is calculated using the following formula 7.

number

number

number

[0073] (operation) Next, an example of the operation of the tracking device 20 will be described with reference to the drawings. The outline of the processing by the tracking device 20 is the same as that of the first embodiment, so a description thereof will be omitted. Below, the tracking processing by the tracking unit 27 of the tracking device 20 will be described in detail.

[0074] Fig. 9 is a flowchart for explaining the tracking process by the tracking unit 27 of the tracking device 20. In Fig. 9, first, the tracking unit 27 estimates the number of pixels of the height of the tracked object based on the skeleton line of the detected object detected from the verification frame (step S271).

[0075] Next, the tracking unit 27 calculates the normalized distance between the tracked objects in the preceding and succeeding frames in terms of position and orientation (step S272). The normalized distance is the distance between the position and orientation normalized by the estimated height pixel count.

[0076] Next, the tracking unit 27 calculates a normalized score between the tracking targets from the normalized distances related to the positions and orientations between the tracking targets (step S273). For example, the tracking unit 17 calculates the normalized score as the sum of the normalized distance related to the positions and the normalized distance related to the orientations between the tracking targets.

[0077] Next, the tracking unit 27 selects an optimal combination of tracking targets according to the normalized score between the tracking targets (step S274). For example, the tracking unit 27 selects a combination of tracking targets that has the smallest normalized score from the preceding and succeeding frames.

[0078] Next, the tracking unit 27 assigns an ID to the tracking target detected from the subsequent frame according to the selected combination (step S275). For example, the tracking unit 27 assigns the same ID to the combination of tracking targets in the preceding and subsequent frames that results in the smallest normalized score.

[0079] As described above, the tracking device of the tracking system of this embodiment includes a detection unit, an extraction unit, a posture information generation unit, and a tracking unit. The detection unit detects a tracking target from at least two frames constituting video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0080] Furthermore, in this embodiment, the tracking unit estimates the height pixel count of the tracking target based on a skeleton line connecting any of the multiple key points, normalizes the score by the estimated height pixel count, and tracks the tracking target detected from each of the at least two frames according to the normalized score.

[0081] In this embodiment, the score is normalized according to the size of the tracking target in the frame. Therefore, according to this embodiment, a tracking target that appears large due to its position relative to the surveillance camera is not overestimated, and tracking bias in the position within the frame can be reduced. Therefore, according to this embodiment, tracking with higher accuracy is possible across multiple frames that make up the video. Furthermore, according to this embodiment, tracking can be performed regardless of the tracking target's posture, so tracking of the tracking target can be continued even if there is a large change in posture between frames.

[0082] In one aspect of this embodiment, the tracking device includes a tracking information output unit that outputs tracking information related to tracking of the tracked object. The tracking information is, for example, an image in which a skeleton line is displayed at the position of the tracked object detected from the verification frame. According to this aspect, by displaying an image in which the tracking information is superimposed on the tracked object on the screen of a display device, it becomes easier to visually grasp the posture of the tracked object.

[0083] (Third embodiment) Next, a tracking system according to a third embodiment will be described with reference to the drawings. The tracking system of this embodiment differs from the first and second embodiments in that it displays a user interface for setting weights for positions and orientations and for setting key points.

[0084] (composition) Fig. 10 is a block diagram showing an example of the configuration of a tracking system 3 of this embodiment. The tracking system 3 includes a tracking device 30, a monitoring camera 310, and a terminal device 320. Although Fig. 10 shows only one monitoring camera 310 and one terminal device 320, there may be multiple monitoring cameras 310 and multiple terminal devices 320. The monitoring camera 310 is similar to the monitoring camera 110 of the first embodiment, and therefore a detailed description thereof will be omitted.

[0085] The tracking device 30 includes an image acquisition unit 31, a memory unit 32, a detection unit 33, an extraction unit 35, an attitude information generation unit 36, a tracking unit 37, a tracking information output unit 38, and a setting acquisition unit 39. For example, the tracking device 30 is placed on a server or a cloud. For example, the tracking device 30 may be provided as an application installed on a terminal device 320. The image acquisition unit 31, the memory unit 32, the detection unit 33, the extraction unit 35, the attitude information generation unit 36, the tracking unit 37, and the tracking information output unit 38 are each similar to the corresponding configurations in the first embodiment, and therefore detailed description thereof will be omitted.

[0086] Fig. 11 is a block diagram showing an example of the configuration of a terminal device 320, etc. The terminal device 320 has a tracking information acquisition unit 321, a tracking information storage unit 322, a display unit 323, and an input unit 324. Fig. 11 also shows the tracking device 10, an input device 327, and a display device 330 connected to the terminal device 320.

[0087] The tracking information acquisition unit 321 acquires tracking information for each of a plurality of frames that make up the video data from the tracking device 30. The tracking information acquisition unit 321 stores the tracking information for each frame in the tracking information storage unit 322.

[0088] The tracking information storage unit 322 stores the tracking information acquired from the tracking device 30. The tracking information stored in the tracking information storage unit 322 is displayed as a GUI (Graphical User Interface) on the screen of the display unit 323 in response to, for example, a user operation.

[0089] The display unit 323 is connected to a display device 330 having a screen. The display unit 323 acquires tracking information from the tracking information storage unit 322. The display unit 323 displays display information including the acquired tracking information on the screen of the display device 330. Note that the terminal device 320 may include the function of the display device 330.

[0090] For example, the display unit 323 accepts an operation by the user via the input unit 324, and displays display information corresponding to the accepted operation on the screen of the display device 330. For example, the display unit 323 displays display information corresponding to a frame having a frame number specified by the user on the screen of the display device 330. For example, the display unit 323 displays display information corresponding to each of a series of multiple frames including the frame having the frame number specified by the user on the screen of the display device 330 in chronological order.

[0091] For example, the display unit 323 may display at least one piece of display information on the screen of the display device 330 in accordance with preset display conditions. For example, the preset display conditions are conditions that cause a plurality of pieces of display information corresponding to a predetermined number of consecutive frames including a preset frame number to be displayed in chronological order. For example, the preset display conditions are conditions that cause a plurality of pieces of display information corresponding to a plurality of frames generated during a predetermined time period including a preset time to be displayed in chronological order. Note that the display conditions are not limited to the examples given here, as long as they are set in advance.

[0092] The input unit 324 is connected to an input device 327 that accepts operations by a user. For example, the input device 327 is realized by a keyboard, a touch panel, a mouse, or the like. The input unit 324 outputs the content of the user's operation input via the input device 327 to the tracking device 30. Furthermore, when the input unit 324 accepts designation of video data, frames, display information, or the like from the user, it outputs an instruction to the display unit 323 to display the designated image on the screen.

[0093] The setting acquisition unit 39 acquires settings input using the terminal device 320. The setting acquisition unit 39 acquires weight settings related to position and orientation, key point settings, etc. The setting acquisition unit 39 reflects the acquired settings in the functions of the tracking device 30.

[0094] 12 is a conceptual diagram illustrating an example of display information displayed on the screen of display device 330. A weight setting area 340 and an image display area 350 are set on the screen of display device 330. A first operation image 341 for setting a weight related to a position and a second operation image 342 for setting a weight related to an orientation are displayed in setting area 340. A tracking image for each frame constituting the video captured by surveillance camera 310 is displayed in image display area 350. Note that display areas other than weight setting area 340 and image display area 350 may be set on the screen of display device 330. Furthermore, the display positions of weight setting area 340 and image display area 350 on the screen can be changed as desired.

[0095] First operation image 341 displays a scroll bar for setting a position weight. The position weight is an index value indicating how much importance is given to the position of a tracked target when comparing tracked targets detected in a preceding frame and a succeeding frame. The position weight is set in the range of 0 to 1. A minimum value (left end) and a maximum value (right end) of the position weight are set in the scroll bar displayed in first operation image 341. Moving knob 361 on the scroll bar left or right changes the position weight. In the example of FIG. 12, the position weight is set to 0.8. Note that first operation image 341 may display a vertical scroll bar instead of a horizontal scroll bar. Also, first operation image 341 may display a spin button, a combo box, or the like for setting the position weight instead of a scroll bar. Also, first operation image 341 may display an element other than a scroll bar for setting the position weight.

[0096] Second operation image 342 displays a scroll bar for setting the weight related to the orientation. The weight related to the orientation is an index value indicating how much importance is given to the orientation of the tracked target when comparing the tracked target detected from each of the preceding and subsequent frames. The weight related to the orientation is set in the range of 0 to 1. The minimum value (left end) and maximum value (right end) of the weight related to the orientation are set in the scroll bar displayed in second operation image 342. Moving knob 362 on the scroll bar left or right changes the weight related to the orientation. In the example of FIG. 12, the weight related to the orientation is set to 0.2. Note that second operation image 342 may display a vertical scroll bar instead of a horizontal scroll bar. Also, second operation image 342 may display a spin button, a combo box, or the like for setting the weight related to the orientation instead of a scroll bar. Also, second operation image 342 may display an element other than a scroll bar or the like for setting the weight related to the orientation.

[0097] In the example of FIG. 12, frames including six people assigned IDs 11 to 16 as tracking targets are displayed in the image display area 350. FIG. 12 shows an example in which an image corresponding to a subsequent frame is displayed in the image display area 350. The previous frame and the subsequent frame may be displayed side by side in the image display area 350. Furthermore, the previous frame and the subsequent frame may be displayed in the image display area 350 so as to be switched between in response to the selection of a button (not shown), etc.

[0098] In the example of FIG. 12, tracking information associated with a person detected from a frame is displayed. The tracking information is displayed by associating multiple key points extracted from the person detected from the frame with line segments (skeleton lines) connecting these key points. For example, whether or not to display tracking information in the image display area 350 may be switched in response to a user operation via the terminal device 320. In the example of FIG. 12, six people are walking in the same direction. As such, when there are many tracking targets moving in the same direction, it is better to prioritize position over orientation in order to track the tracking targets between frames with high accuracy. When there are many tracking targets moving in the same direction, if the weights related to position and orientation are set equal, the weight related to orientation may be overestimated, which may result in a decrease in tracking accuracy. Therefore, when there are many tracking targets moving in the same direction, setting the weight related to position to a large value and the weight related to orientation to a small value can reduce the decrease in tracking accuracy.

[0099] FIG. 13 is a conceptual diagram illustrating another example of display information displayed on the screen of the display device 330. In the example of FIG. 13, the weight related to position is set to 0.2, and the weight related to orientation is set to 0.8. In the example of FIG. 13, six people are walking while passing each other. When there are many tracking targets moving while passing each other, it is better to place more importance on orientation than position in order to track the tracking targets with high accuracy between frames. When there are many tracking targets moving while passing each other, if the weight related to orientation and the weight related to position are set to be the same, the weight related to position may be overestimated, which may result in a decrease in tracking accuracy. Therefore, when there are many tracking targets moving while passing each other, the weight related to orientation may be set to be large and the weight related to position may be set to be small, thereby reducing the decrease in tracking accuracy.

[0100] Fig. 14 is a conceptual diagram for explaining yet another example of display information displayed on the screen of display device 330. In the example of Fig. 14, third operation image 343 for setting weights related to position and orientation, and fourth operation image 344 for setting weights related to position and orientation according to the scene are displayed in weight setting area 340. Note that third operation image 343 and fourth operation image 344 do not have to be displayed simultaneously in weight setting area 340.

[0101] Third operation image 343 displays a scroll bar for setting the weights for the position and orientation. The scroll bar displayed in first operation image 341 is set to the maximum value (left end) of the weight for the position and the maximum value (right end) of the weight for the orientation. When the weight for the position is set to the maximum value (left end), the weight for the orientation is set to the minimum value. On the other hand, when the weight for the orientation is set to the maximum value (right end), the weight for the position is set to the minimum value. Moving knob 363 on the scroll bar left or right changes the weights for the position and orientation collectively. Note that third operation image 343 may display a vertical scroll bar instead of a horizontal scroll bar. Also, third operation image 343 may display spin buttons, combo boxes, or the like for setting the weights for the position and orientation instead of scroll bars. Also, third operation image 343 may display elements other than scroll bars, etc., for setting the weights for the position and orientation. The weights for the position and the weights for the orientation often have a complementary relationship depending on the scene. Therefore, in a scene where the weighting related to the position is important, it is better to reduce the weighting related to the orientation. Conversely, in a scene where the weighting related to the orientation is important, it is better to reduce the weighting related to the position. In the example of Fig. 14, the weighting related to the position and orientation can be set collectively according to the situation of the tracking target in the frame displayed in the image display area 350, so the settings of the weighting related to the position and orientation can be changed appropriately according to the scene.

[0102] The fourth operation image 344 displays checkboxes for setting weights for the position and orientation according to the scene. FIG. 14 shows an example in which weights are set according to a "passing" scene in response to operation of the pointer 365 via the terminal device 320. In the example of FIG. 14, when a scene is selected in the fourth operation image 344, the setting of the third operation image 343 is simultaneously changed. For example, in a scene in which many people pass each other, it is preferable to emphasize orientation, taking into account the direction of their faces, so as to reduce the likelihood of ID confusion between the tracking targets passing each other. For example, when a "passing" scene is selected, the position weight is set to 0.2 and the direction weight is set to 0.8. For example, in a scene in which many people are moving in the same direction, it is sufficient to emphasize position, regardless of the direction of their faces. For example, when a "same direction" scene is selected, the position weight is set to 0.8 and the direction weight is set to 0.2. By selecting a scene according to the situation of the tracking target within the frame displayed in the image display area 350, the position and orientation weight settings can be intuitively changed.

[0103] FIG. 15 is a conceptual diagram illustrating another example of display information displayed on the screen of the display device 330. A key point designation area 370 and a key point designation area 380 are set on the screen of the display device 330. An individual designation image 371 and a collective designation image 372 are displayed in the key point designation area 370. An image in which the key points designated in the key point designation area 370 are associated with a human body is displayed in the key point designation area 380. For example, key points are designated in accordance with the selection of each key point in the individual designation image 371 or the selection of a body part in the collective designation image 372. In the example of FIG. 15, all key points designated in the individual designation image 371 are displayed in the key point designation area 380. The selected key points are displayed in the key point designation area 380 in a blacked-out state. Note that display areas other than the key point designation area 370 and the key point designation area 380 may be set on the screen of the display device 330. Furthermore, the display positions of the key point designation area 370 and the key point designation area 380 on the screen can be changed as desired.

[0104] FIG. 16 is a conceptual diagram illustrating yet another example of display information displayed on the screen of the display device 330. In the example of FIG. 16, "trunk" is selected in the collective designation image 372 in response to operation of the pointer 365 via the terminal device 320. When "trunk" is selected in the collective designation image 372, the head (HD), neck (N), right hip (RW), and left hip (LW) are designated collectively. In the example of FIG. 16, the key points of "trunk" designated in the collective designation image 372 are displayed in the key point designation area 380. The selected key points are displayed in the key point designation area 370 in a black-filled state. For example, the arms and legs change more between frames than the trunk, so if the weights are too large, tracking accuracy may decrease. Therefore, the weights of the arms and legs may be set to be smaller by default than the weight of the trunk.

[0105] For example, when "upper body" is selected, the head (HD), neck (N), right shoulder (RS), left shoulder (LS), right elbow (RE), left elbow (LE), right hand (RH), and left hand (LH) are specified all at once. For example, when "lower body" is selected, the right hip (RW), left hip (LW), right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once. For example, when "right half of body" is selected, the right shoulder (RS), right elbow (RE), right hand (RH), right knee (RK), and right foot (RF) are specified all at once. For example, when "left half of body" is selected, the left shoulder (LS), left elbow (LE), left hand (LH), left knee (LK), and left foot (LF) are specified all at once. For example, when "limbs" is selected, the right elbow (RE), left elbow (LE), right hand (HR), left hand (LH), right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once. For example, when "arms" is selected, the right elbow (RE), left elbow (LE), right hand (RH), and left hand (LH) are specified all at once. For example, when "feet" is selected, the right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once.

[0106] For example, the weight of a selected keypoint is set to 1, and the weight of a non-selected keypoint is set to 0. For example, if the upper body is selected, the weight of the keypoints included in the upper body is set to 1. For example, if the upper body is selected, the weight of the keypoints included in the upper body may be set to 1, and the weight of the keypoints included in the lower body may be set to 0.5.

[0107] The key points selected collectively when selected in the collective designation image 372 as described above are merely an example, and different combinations may be used. For example, instead of selecting key points collectively by body part, sets of key points appropriate for a scene or situation may be prepared in advance, and the set of key points may be configured to be intuitively selected. For example, a model trained on key points selected by a skilled user according to a scene or situation may be used to estimate appropriate key points appropriate for a scene or situation. For example, questions for setting key points may be prepared, and key points may be set according to the answers to the questions. By configuring the system to allow selection of a set of pre-prepared key points, even a user who is not skilled in individually selecting key points according to a scene or situation can select appropriate key points just like a skilled user.

[0108] FIG. 17 shows an example in which tracking information is displayed in association with a person detected from a frame when "trunk" is selected and the head (HD), neck (N), right hip (RW), and left hip (LW) are specified all at once, as in FIG. 16. The tracking information is displayed by associating four key points (HD, N, RW, LW) extracted from the person detected from the frame with the line segments (skeleton lines) connecting these key points. When the tracking target moves in the same direction as in FIG. 17, it is sufficient to grasp the position of the tracking target, so tracking can be performed with emphasis on the key points of the trunk, which moves relatively little. For example, the display information in FIGS. 15 to 16 and the display information in FIG. 17 can be switched by pressing a button (not shown) displayed on the screen of display device 330.

[0109] (operation) Next, an example of the operation of the tracking device 30 will be described with reference to the drawings. The outline of the processing by the tracking device 30 is the same as that of the first embodiment, and therefore will be omitted. Below, the details of the setting processing in the tracking unit 37 of the tracking device 30 will be described. For example, this is inserted somewhere between steps S13 and S14 in FIG. 5. The setting processing is executed in accordance with the specification of key points and the adjustment of weights of positions and orientations.

[0110] 18, first, the tracking device 30 determines whether or not a key point (KP) has been designated (step S31). If a key point has been designated (Yes in step S31), the tracking device 30 sets the designated key point as an extraction target (step S32). On the other hand, if a key point has not been designated (No in step S31), the tracking device 30 proceeds to step S33.

[0111] Next, if the weights of the position and orientation have been adjusted (Yes in step S33), the tracking device 30 sets the weights of the position and orientation according to the adjustment (step S34). After step S34, the process proceeds to the subsequent process in the flowchart of Fig. 5. On the other hand, if the weights of the position and orientation have not been adjusted (No in step S33), the process proceeds to the subsequent process in the flowchart of Fig. 5 without readjusting the weights of the position and orientation.

[0112] As described above, the tracking system of this embodiment includes a surveillance camera, a tracking device, and a terminal device. The surveillance camera captures an image of a surveillance target area and generates video data. The terminal device is connected to a display device having a screen that displays display information generated by the tracking device. The tracking device includes a video acquisition unit, a memory unit, a detection unit, an extraction unit, a posture information generation unit, a tracking unit, a tracking information output unit, and a setting acquisition unit. The video acquisition unit acquires video data from the surveillance camera. The memory unit stores the acquired video data. The detection unit detects a tracking target from at least two frames constituting the video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames. The tracking information output unit outputs tracking information related to tracking of the tracking target to the terminal device. The setting acquisition unit acquires settings input using the terminal device. The setting acquisition unit acquires weight settings related to position and orientation, key point settings, etc. The setting acquisition unit reflects the acquired settings in the functions of the tracking device.

[0113] In this embodiment, the terminal device sets an image display area and a weight setting area on the screen of a display device. The image display area displays a tracking image in which key points are associated with a tracking target detected from frames constituting the video data. The weight setting area displays an operation image for setting a position weight and an orientation weight. The terminal device outputs the position weight and orientation weight set in the weight setting area to the tracking device. The tracking device acquires the position weight and orientation weight selected in the weight setting area from the terminal device. The tracking device uses the acquired position weight and orientation weight to calculate a score according to the distance related to the position and orientation of the tracking target detected from each of at least two frames constituting the video data. The tracking device tracks the tracking target based on the calculated score.

[0114] In this embodiment, the weights for the position and orientation can be adjusted arbitrarily in response to a user operation, and therefore, according to this embodiment, it is possible to track a tracking target with high accuracy based on the weights according to a user request.

[0115] In one aspect of this embodiment, the terminal device displays an operation image in a weight setting area for setting position weights and orientation weights according to a scene. The terminal device outputs the position weights and orientation weights according to the scene set in the weight setting area to the tracking device. According to this aspect, the position and orientation weights can be arbitrarily adjusted according to the scene. Therefore, according to this embodiment, it is possible to achieve high-precision tracking of the tracked object suited to the scene.

[0116] In one aspect of this embodiment, the terminal device sets a keypoint designation area on the screen of the display device, in which a designation image for designating keypoints to be used in generating posture information of the tracked object is displayed. The terminal device outputs keypoints selected in the keypoint area to the tracking device. The tracking device acquires the keypoints selected in the keypoint selection area from the terminal device. The tracking device generates posture information for the acquired keypoints. In this aspect, the keypoints used in generating posture information can be arbitrarily adjusted in response to a user operation. Therefore, according to this embodiment, highly accurate tracking of the tracked object can be achieved using posture information according to the user's request.

[0117] (Fourth embodiment) Next, a tracking device according to a fourth embodiment will be described with reference to the drawings. The tracking device of this embodiment has a simplified configuration of the tracking devices of the first to third embodiments. FIG. 19 is a block diagram showing an example of the configuration of a tracking device 40 of this embodiment. The tracking device 40 includes a detection unit 43, an extraction unit 45, a posture information generation unit 46, and a tracking unit 47.

[0118] The detection unit 43 detects a tracking target from at least two frames constituting the video data. The extraction unit 45 extracts at least one key point from the detected tracking target. The posture information generation unit 46 generates posture information of the tracking target based on the at least one key point. The tracking unit 47 tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0119] As described above, the tracking device of this embodiment tracks a target based on the position and orientation of the target's posture information, thereby tracking a plurality of targets based on their postures in frames constituting a video.

[0120] (Hardware) Here, a hardware configuration for executing the processing of a tracking device, a terminal device, etc. (hereinafter referred to as a tracking device, etc.) according to each embodiment of the present disclosure will be described using an information processing device 90 in Fig. 20 as an example. Note that the information processing device 90 in Fig. 20 is an example configuration for executing the processing of the tracking device, etc. according to each embodiment, and does not limit the scope of the present disclosure.

[0121] As shown in Fig. 20, an information processing device 90 includes a processor 91, a main storage device 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In Fig. 20, interface is abbreviated as I / F (Interface). The processor 91, the main storage device 92, the auxiliary storage device 93, the input / output interface 95, and the communication interface 96 are connected to each other via a bus 98 so as to be able to communicate data with each other. The processor 91, the main storage device 92, the auxiliary storage device 93, and the input / output interface 95 are also connected to a network such as the Internet or an intranet via the communication interface 96.

[0122] The processor 91 loads a program stored in the auxiliary storage device 93 or the like into the main storage device 92 and executes the loaded program. In this embodiment, a software program installed in the information processing device 90 may be used. The processor 91 executes processing by the tracking device or the like according to this embodiment.

[0123] The main memory device 92 has an area where a program is loaded. The main memory device 92 may be a volatile memory such as a dynamic random access memory (DRAM). Alternatively, a non-volatile memory such as a magnetoresistive random access memory (MRAM) may be configured as the main memory device 92 or added.

[0124] The auxiliary storage device 93 stores various data. The auxiliary storage device 93 is configured with a local disk such as a hard disk or flash memory. Note that it is also possible to configure the main storage device 92 to store various data, and omit the auxiliary storage device 93.

[0125] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices. The communication interface 96 is an interface for connecting to external systems or devices via a network such as the Internet or an intranet based on standards and specifications. The input / output interface 95 and the communication interface 96 may be a common interface for connecting to external devices.

[0126] The information processing device 90 may be configured to be connected to input devices such as a keyboard, mouse, or touch panel as needed. These input devices are used to input information and settings. When a touch panel is used as the input device, the display screen of the display device may also serve as an interface for the input device. Data communication between the processor 91 and the input devices may be mediated by an input / output interface 95.

[0127] The information processing device 90 may also be equipped with a display device for displaying information. When a display device is equipped, the information processing device 90 preferably includes a display control device (not shown) for controlling the display of the display device. The display device may be connected to the information processing device 90 via the input / output interface 95.

[0128] The information processing device 90 may also be equipped with a drive device. The drive device acts as an intermediary between the processor 91 and a recording medium (program recording medium) for reading data and programs from the recording medium, writing the processing results of the information processing device 90 to the recording medium, etc. The drive device may be connected to the information processing device 90 via an input / output interface 95.

[0129] The above is an example of a hardware configuration for enabling a tracking device, etc., according to each embodiment of the present invention. Note that the hardware configuration in FIG. 20 is an example of a hardware configuration for executing arithmetic processing of a tracking device, etc., according to each embodiment, and does not limit the scope of the present invention. Furthermore, a program that causes a computer to execute processing related to a tracking device, etc., according to each embodiment is also within the scope of the present invention. Furthermore, a program recording medium on which a program according to each embodiment is recorded is also within the scope of the present invention. The recording medium can be realized, for example, as an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may also be realized as a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card, a magnetic recording medium such as a flexible disk, or other recording medium. When a program executed by a processor is recorded on a recording medium, the recording medium corresponds to a program recording medium.

[0130] The components of the tracking device and the like of each embodiment can be combined in any manner. Furthermore, the components of the tracking device and the like of each embodiment can be realized by software or by circuitry.

[0131] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0132] 1, 2, 3 Tracking System 10, 20, 30, 40 Tracking Device 11, 21, 31 Video acquisition section 12, 22, 32 storage section 13, 23, 33, 43 Detector 15, 25, 35, 45 Extraction part 16, 26, 36, 46 Posture information generation section 17, 27, 37, 47 Tracking Section 18, 28, 38 Tracking information output section 39 Setting acquisition unit 110, 210, 310 surveillance cameras 120, 220, 320 terminal equipment 321 Tracking Information Acquisition Unit 322 Tracking information storage unit 323 Display section 324 Input Section 327 Input Devices 330 Display equipment

Claims

1. On the computer, A process of displaying a user interface for setting weights on a screen of a display device; a process of tracking a person or an object in an image using key points of the person or the object based on the set weights; To set the weights, A process of setting a first index indicating how much importance is attached to the position of the tracking target; a process of setting a second index indicating how much importance is attached to the orientation of the tracked object; a process of displaying a means for selecting a scene; a process of setting at least one of the first index and the second index according to a selected scene; A program that executes the following.

2. The program according to claim 1 , causing a computer to execute a process of, when one of the first index or the second index is changed, changing the other index in accordance with the changed index.

3. The program according to claim 1, wherein when tracking the person or object in a process of tracking the person, the computer executes a process of tracking the person using key points acquired based on the skeleton of the person.

4. The computer A user interface for setting weights is displayed on the screen of a display device, Tracking the person or object in the image using key points of the person or object based on the set weights; To set the weights, setting a first index indicating how much importance is attached to the position of the tracking target; setting a second index indicating how much importance is placed on the orientation of the tracked object; Displaying a means for selecting a scene, A tracking method for setting at least one of the first index and the second index according to a selected scene.

5. a means for displaying a user interface for setting weights on a screen of a display device; a means for tracking a person or an object appearing in an image using key points of the person or the object based on the set weights; To set the weights, means for setting a first index indicating how much importance is attached to the position of the tracking target; means for setting a second index indicating how much importance is to be attached to the orientation of the tracked object; means for displaying means for selecting a scene; means for setting at least one of the first index and the second index in accordance with a selected scene.

Images