Program, tracking method, and tracking device

Abstract

To provide a program capable of tracking multiple targets in multiple frames constituting an image.SOLUTION: A program is provided that causes a computer to execute the processing of: displaying a user interface for setting weights on a screen of a display device; tracking a person or object using key points of the person or object appearing in an image based on the set weights; and in order to set the weights, setting a first index indicating how much importance is placed on the position of the target to be tracked, and setting a second index indicating how much importance is placed on the orientation of the target to be tracked.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a program, a tracking method, and a tracking device. [Background technology]

[0002] Person tracking technology is a technology that detects people from image frames (hereinafter also referred to as frames) that make up video captured by a surveillance camera or the like, and tracks the detected people in the video. In person tracking technology, for example, each detected person is identified by face recognition or the like, an identification number is assigned, and the person to which the identification number is assigned is tracked in the video.

[0003] Patent Document 1 discloses a posture estimation device that estimates a three-dimensional posture based on two-dimensional joint positions. The device of Patent Document 1 calculates feature amounts of position candidates of a tracking target from an input image, and estimates the position of the tracking target based on a similarity weight obtained by comparing the feature amounts with template data. The device of Patent Document 1 sets position candidates of the tracking target based on the similarity weight and three-dimensional motion model data. The device of Patent Document 1 tracks the position of the tracking target by repeating the estimation of the position of the tracking target and the setting of the position candidates of the tracking target multiple times. The device of Patent Document 1 also estimates the three-dimensional posture of the posture estimation target by referring to the estimated information of the position of the tracking target and the three-dimensional motion model data.

[0004] Patent Document 2 discloses an image processing device that identifies a person from an image. The device in Patent Document 2 compares a person appearing in an input image with a registered person based on the pose similarity between the pose of the person appearing in an input image and the pose of the person appearing in a reference image, the feature amount of the input image, and the feature amount of the reference image for each person.

[0005] Non-Patent Document 1 discloses a technique for pose tracking of multiple people in a video. The method in Non-Patent Document 1 samples a pair of pose estimates from different frames of the video and performs a binary classification to determine whether one pose follows another pose in time. Furthermore, the method in Non-Patent Document 1 improves the pose estimation method using a parameter-free keypoint refinement technique.

[0006] Non-Patent Document 2 discloses a related technology for estimating the skeletons of multiple people captured in a two-dimensional image. The technology in Non-Patent Document 2 uses a method called Part Affinity Fields to estimate the skeletons of multiple people captured in a two-dimensional image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2013-092876 A [Patent Document 2] JP 2017-097549 A [Non-patent literature]

[0008] [Non-Patent Document 1] Michael Snower, Asim Kadav, Farley Lai, Hans Peter Graf, “15 Keypoints Is All You Need”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp.6738-6748 [Non-Patent Document 2] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp.7291-7299. Summary of the Invention [Problem to be solved by the invention]

[0009] The method of Patent Document 1 can estimate the 3D posture of one person from information on the 2D joint positions, but cannot estimate the 3D postures of multiple people. Furthermore, the method of Patent Document 1 cannot determine whether people appearing in different frames are the same person based on the estimated 3D postures, and cannot track people across frames.

[0010] In the method of Patent Document 2, a person is registered based on the similarity between the estimated posture and the feature amount of a reference image for each posture of the person that is registered in advance. Therefore, in the method of Patent Document 2, unless a reference image for each posture of the person is stored in a database, it is not possible to track a person based on the posture.

[0011] In the method of Non-Patent Document 1, since posture tracking is performed using deep learning, the tracking accuracy depends on the learning data. Therefore, in the method of Non-Patent Document 1, when conditions such as the degree of crowding, the angle of view, the distance between the camera and the person, and the frame rate are different from the learned conditions, tracking cannot be continued based on the posture of the tracking target.

[0012] An object of the present disclosure is to provide a program, a tracking method, and a tracking device capable of tracking multiple tracking targets in multiple frames that make up a video. [Means for solving the problem]

[0013] A program of one embodiment of the present disclosure causes a computer to execute the following processes: displaying a user interface for setting weights on a screen of a display device; tracking a person or object appearing in an image using key points of the person or object based on the set weights; and, in order to set the weights, setting a first indicator indicating how much importance is to be placed on the position of the tracked target, and setting a second indicator indicating how much importance is to be placed on the orientation of the tracked target.

[0014] In a tracking method of one embodiment of the present disclosure, a computer displays a user interface for setting weights on a screen of a display device, tracks the person or object using key points of the person or object appearing in an image based on the set weights, and, in order to set the weights, sets a first indicator indicating how much importance is to be placed on the position of the tracked target, and sets a second indicator indicating how much importance is to be placed on the orientation of the tracked target.

[0015] A tracking device of one embodiment of the present disclosure comprises: means for displaying a user interface for setting weights on a screen of a display device; means for tracking a person or object using key points of the person or object appearing in an image based on the set weight; and means for setting a first indicator representing how much importance is to be placed on the position of the tracked object, and means for setting a second indicator representing how much importance is to be placed on the orientation of the tracked object. Effect of the Invention

[0016] According to the present disclosure, it is possible to provide a program, a tracking method, and a tracking device capable of tracking multiple tracking targets in multiple frames constituting a video. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of a configuration of a tracking system according to a first embodiment. [Diagram 2]FIG. 2 is a conceptual diagram for explaining an example of key points extracted by a tracking device of the tracking system according to the first embodiment. [Diagram 3] FIG. 2 is a conceptual diagram for explaining a tracking process by a tracking device of the tracking system according to the first embodiment. [Figure 4] 4 is a table showing an example of scores used by the tracking device of the tracking system according to the first embodiment to track a tracking target. [Diagram 5] 4 is a flowchart for explaining an example of an outline of the operation of the tracking system according to the first embodiment. [Figure 6] 4 is a flowchart for explaining an example of a tracking process by a tracking device of the tracking system according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of a tracking system according to a second embodiment. [Figure 8] 13 is a conceptual diagram for explaining an example of a skeleton line extracted by a tracking device of a tracking system according to a second embodiment. FIG. [Figure 9] 13 is a flowchart for explaining an example of a tracking process by a tracking device of a tracking system according to a second embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of a tracking system according to a third embodiment. [Figure 11] FIG. 13 is a block diagram showing an example of the configuration of a terminal device of a tracking system according to a third embodiment. [Figure 12] FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment for tracking a tracking target. [Figure 13] FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment for tracking a tracking target. [Figure 14]FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment for tracking a tracking target. [Figure 15] FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting designation of key points used by a tracking device of a tracking system according to a third embodiment for generating posture information. [Figure 16] FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting designation of key points used by a tracking device of a tracking system according to a third embodiment for generating posture information. [Figure 17] FIG. 13 is a conceptual diagram showing an example of displaying, on a screen of a display device, display information including an image for adjusting weights of a position and an orientation used by a tracking device of a tracking system according to a third embodiment for tracking a tracking target. [Figure 18] 13 is a flowchart showing an example of a process in which the tracking device of the tracking system according to the third embodiment accepts settings via the terminal device. [Figure 19] FIG. 13 is a block diagram showing an example of the configuration of a tracking device according to a fourth embodiment. [Figure 20] FIG. 2 is a block diagram showing an example of a hardware configuration for implementing the tracking device according to each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the embodiments for carrying out the present invention will be described with reference to the drawings. However, the embodiments described below are limited in a manner that is technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In addition, in all the drawings used to explain the following embodiments, the same reference numerals are used for similar parts unless otherwise specified. In addition, in the following embodiments, repeated explanations of similar configurations and operations may be omitted.

[0019] (First embodiment) First, a tracking system according to a first embodiment will be described with reference to the drawings. The tracking system of this embodiment detects a tracking target such as a person from image frames (also called frames) constituting a video captured by a surveillance camera or the like, and tracks the detected tracking target between frames. There is no particular limitation on the tracking target of the tracking system of this embodiment. For example, the tracking system of this embodiment may track not only people, but also animals such as dogs and cats, moving objects such as automobiles, bicycles, and robots, and arbitrary objects. An example of tracking a person in a video will be described below.

[0020] (composition) Fig. 1 is a block diagram showing an example of the configuration of a tracking system 1 according to this embodiment. The tracking system 1 includes a tracking device 10, a surveillance camera 110, and a terminal device 120. Although Fig. 1 shows only one surveillance camera 110 and one terminal device 120, there may be multiple surveillance cameras 110 and multiple terminal devices 120.

[0021] The surveillance camera 110 is placed in a position where it can capture the monitored area. The surveillance camera 110 has the functions of a general surveillance camera. The surveillance camera 110 may be a camera sensitive to the visible range, or an infrared camera sensitive to the infrared range. For example, the surveillance camera 110 is placed on a street or indoors where there are people. There is no particular limitation on the method of connection between the surveillance camera 110 and the tracking device 10. For example, the surveillance camera 110 is connected to the tracking device 10 via a network such as the Internet or an intranet. The surveillance camera 110 may also be connected to the tracking device 10 by a cable or the like.

[0022] The surveillance camera 110 captures images of the monitored area at set shooting intervals and generates video data. The surveillance camera 110 outputs the generated video data to the tracking device 10. The video data is composed of a plurality of frames captured at the set shooting intervals. For example, the surveillance camera 110 may output video data composed of a plurality of frames to the tracking device 10, or may output each of the plurality of frames to the tracking device 10 in the chronological order in which they were captured. There is no particular limitation on the timing at which the surveillance camera 110 outputs data to the tracking device 10.

[0023] The tracking device 10 includes an image acquisition unit 11, a storage unit 12, a detection unit 13, an extraction unit 15, a posture information generation unit 16, a tracking unit 17, and a tracking information output unit 18. For example, the tracking device 10 is disposed on a server or a cloud. For example, the tracking device 10 may be provided as an application installed on the terminal device 120.

[0024] In this embodiment, the tracking device 10 tracks the tracking target between two verification target frames (hereinafter referred to as verification frames). The verification frame that precedes in chronological order is referred to as the preceding frame, and the verification frame that follows is referred to as the following frame. The tracking device 10 tracks the tracking target between frames by matching the tracking target included in the preceding frame with the tracking target included in the following frame. The preceding frame and the following frame may be consecutive frames, or may be separated by several frames.

[0025] The video acquisition unit 11 acquires video data to be processed from the surveillance camera 110. The video acquisition unit 11 stores the acquired video data in the storage unit 12. There is no particular limitation on the timing at which the tracking device 10 acquires data from the surveillance camera 110. For example, the video acquisition unit 11 may acquire video data composed of a plurality of frames from the surveillance camera 110, or may acquire each of the plurality of frames from the surveillance camera 110 in the order in which they were shot. Note that the video acquisition unit 11 may acquire not only video data generated by the surveillance camera 110, but also video data stored in an external storage device, server, or the like (not shown).

[0026] The storage unit 12 stores the video data generated by the surveillance camera 110. Frames constituting the video data stored in the storage unit 12 are acquired by the tracking unit 17 and used to track the tracking target.

[0027] The detection unit 13 acquires a verification frame from the storage unit 12. The detection unit 13 detects tracking targets from the acquired verification frame. The detection unit 13 assigns IDs (Identifiers) to all tracking targets detected from the verification frame. In the following, it is assumed that a formal ID has been assigned to a tracking target detected from a preceding frame. The detection unit 13 assigns a provisional ID to a tracking target detected from a subsequent frame.

[0028] For example, the detection unit 13 detects the tracking target from the verification frame by a detection technique such as a background subtraction method. For example, the detection unit 13 may detect the tracking target from the verification frame by a detection technique (for example, a detection algorithm) using a feature amount such as a motion vector. The tracking target detected by the detection unit 13 is a person or a moving object (also called a moving body). For example, when the tracking target is a person, the detection unit 13 detects the tracking target from the verification frame by using a face detection technique. For example, the detection unit 13 may detect the tracking target from the verification frame by using a human body detection technique or an object detection technique. For example, the detection unit 13 may detect an object that is not a moving object but whose feature amount such as a shape, pattern, or color changes at a certain position.

[0029] The extraction unit 15 extracts a plurality of key points from the tracking target detected from the verification frame. For example, when the tracking target is a person, the extraction unit 15 extracts the positions of the head, joints, limbs, etc. of the person included in the verification frame as key points. For example, the extraction unit 15 detects the skeletal structure of the person included in the verification frame and extracts key points based on the detected skeletal structure. For example, the extraction unit 15 detects the skeletal structure of the person based on the features of the person's joints, etc. included in the verification frame using a skeletal estimation technique using machine learning. For example, the extraction unit 15 detects the skeletal structure of the person included in the verification frame using the skeletal estimation technique disclosed in Non-Patent Document 2 (Non-Patent Document 2: Z. Cao et al., The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp.7291-7299).

[0030] For example, the extraction unit 15 assigns a number from 1 to n (n is a natural number) to each extracted keypoint, such as 0 for the right shoulder, 1 for the right elbow, etc. For example, if the k-th keypoint of a person detected from the verification frame is not extracted, the keypoint is deemed undetected (k is a natural number between 1 and n).

[0031] FIG. 2 is a conceptual diagram for explaining key points when the tracking target is a person. FIG. 2 is a diagram of a person viewed from the front. In the example of FIG. 2, 14 key points are set for one person. HD is a key point set for the head. N is a key point set for the neck. RS and LS are key points set for the right shoulder and the left shoulder, respectively. RE and LE are key points set for the right elbow and the left elbow, respectively. RH and LH are key points set for the right hand and the left hand, respectively. RW and LW are key points set for the right hip and the left hip, respectively. RK and LK are key points set for the right knee and the left knee, respectively. RF and LF are key points set for the right foot and the left foot, respectively. Note that the number of key points set for one person is not limited to 14. In addition, the position of each key point is not limited to the example of FIG. 2. For example, face detection may be used in combination, and key points may be set for the eyes, eyebrows, nose, mouth, etc. according to the detection of the face.

[0032] The posture information generating unit 16 generates posture information of all the tracked objects detected from the verification frame based on the key points extracted by the extracting unit 15. The posture information is the position information of each key point of each tracked object in the verification frame. When tracking a tracked object between two verification frames, the posture information f of the tracked object detected from the previous frame is used. p is expressed by the following equation 1, and the posture information of the person detected from the subsequent frame f s is expressed by the following equation 2.

number

number

[0033] The tracking unit 17 tracks the target between frames using the posture information generated for the target detected from the preceding frame and the posture information generated for the target detected from the preceding frame. The tracking unit 17 tracks the target based on the position and orientation of the posture information of the target detected from at least two frames. The tracking unit 17 tracks the target by assigning the ID of the target detected from the preceding frame to the target identified as the target detected from the preceding frame among the targets detected from the subsequent frame. Note that, if a target corresponding to the target detected from the subsequent frame is not detected from the preceding frame, the temporary ID assigned to the target detected from the subsequent frame may be set as the official ID, or a new ID may be assigned as the official ID.

[0034] For example, the tracking unit 17 calculates the position of the key point of the tracking target by using coordinate information in the frame. The tracking unit 17 calculates the distance in a specific direction between the position of a reference key point and the head key point as the orientation of the tracking target. For example, the tracking unit 17 calculates the distance (distance in the x direction) from the neck key point to the head key point in the horizontal direction (x direction) of the screen as the orientation of the tracking target. The tracking unit 17 calculates the distances related to the positions and orientations of all the tracking targets detected from the preceding frame and all the tracking targets detected from the subsequent frame in a brute force manner. The tracking unit 17 calculates the sum of the distances related to the positions and the distances related to the orientations calculated between all the tracking targets detected from the preceding frame and all the tracking targets detected from the subsequent frame as a score. The tracking unit 17 tracks the tracking targets by assigning the same ID to the tracking target with the smallest score among the pair of the tracking target detected from the preceding frame and the tracking target detected from the subsequent frame.

[0035] Distance D in relation to position p is the weighted average of the absolute difference in coordinate values ​​of each keypoint extracted from the tracked object being compared in the previous and subsequent frames. The weight for the position of each keypoint is w k Then, the tracking unit 17 calculates the distance D p Calculate.

number

[0036] Orientation distance D d is the weighted average of the absolute difference of the x-coordinates of each keypoint extracted from the tracked object being compared in the previous and subsequent frames relative to the reference point. The neck keypoint is taken as the reference point, and the reference point in the previous frame is taken as x p_neck and the reference point of the subsequent frame is x s_neck and the weights for each keypoint position are expressed as w k Then, the tracking unit 17 calculates the distance D d Calculate.

number

[0037] Distance D in relation to position p and the distance D d The sum of these is the score S. The tracking unit 17 calculates the score S using the following formula 5.

number

[0038] 3 is a conceptual diagram for explaining an example (A) of extraction of key points, an example (B) of extraction of key points (skeleton lines) used for tracking, and an example (C) of ID allocation by the tracking unit 17. In Fig. 3, the upper diagram corresponds to the preceding frame, and the lower diagram corresponds to the following frame.

[0039] FIG. 3A shows an example of extracting key points from a tracked target included in a verification frame. FIG. 3A shows the contour of the tracked target and line segments connecting key points extracted from the tracked target. In FIG. 3A, two people are included in the preceding and succeeding frames. The two people extracted from the preceding frame are assigned IDs P_ID4 and P_ID8, respectively. The two people extracted from the succeeding frame are assigned IDs S_ID1 and S_ID2, respectively. The IDs assigned to the two people extracted from the succeeding frame are temporary IDs.

[0040] 3B is a diagram showing only line segments (also called skeleton lines) connecting key points used for tracking the target among key points extracted from the target. For example, the key points used for tracking may be set in advance or may be set for each verification.

[0041] FIG. 4 is a table summarizing scores calculated by the tracking unit 17 with respect to the example of FIG. 3. The score between the tracking target of S_ID1 detected from the subsequent frame and P_ID4 detected from the preceding frame is 0.2. The score between the tracking target of S_ID1 detected from the subsequent frame and P_ID8 detected from the preceding frame is 1.5. The score between the tracking target of S_ID2 detected from the subsequent frame and P_ID4 detected from the preceding frame is 1.3. The score between the tracking target of S_ID2 detected from the subsequent frame and P_ID8 detected from the preceding frame is 0.3. That is, the tracking target with the smallest score with respect to the tracking target of S_ID1 is P_ID4. Also, the tracking target with the smallest score with respect to the tracking target of S_ID2 is P_ID8. The tracking unit 17 assigns an ID of P_ID4 to the tracking target of S_ID1, and an ID of P_ID8 to the tracking target of S_ID2.

[0042] Fig. 3C shows a situation in which the same ID is assigned to the same tracking target detected in the preceding frame and the following frame based on the score value in Fig. 4. In this way, the tracking target assigned the same ID in the preceding frame and the following frame is further referred to in the following frame.

[0043] The tracking information output unit 18 outputs tracking information including the tracking result by the tracking unit 17 to the terminal device 120. For example, the tracking information output unit 18 outputs an image in which key points and a skeleton line are superimposed on the tracking target detected from the verification frame as tracking information. For example, the tracking information output unit 18 outputs an image in which key points and a skeleton line are displayed at the position of the tracking target detected from the verification frame as tracking information. For example, the image output from the tracking information output unit 18 is displayed on the display unit of the terminal device 120.

[0044] The terminal device 120 acquires tracking information for each of a plurality of frames constituting the video data from the tracking device 10. The terminal device 120 displays an image including the acquired tracking information on a screen. For example, the terminal device 120 displays an image including tracking information on a screen according to a preset display condition. For example, the preset display condition is a condition that images including tracking information corresponding to a predetermined number of consecutive frames including a preset frame number are displayed in chronological order. For example, the preset display condition is a condition that images including tracking information corresponding to a plurality of frames generated in a predetermined time period including a preset time are displayed in chronological order. Note that the display condition is not limited to the example given here as long as it is preset.

[0045] (operation) Next, an example of the operation of the tracking device 10 will be described with reference to the drawings. In the following, an overview of the processing by the tracking device 10 and details of the tracking processing by the tracking unit 17 of the tracking device 10 will be described.

[0046] Fig. 5 is a flowchart for explaining the operation of the tracking device 10. In Fig. 5, first, the tracking device 10 acquires a verification frame (step S11). The tracking device 10 may acquire a verification frame that has been stored in advance, or may acquire a newly input verification frame.

[0047] When the tracking target is detected from the verification frame (Yes in step S12), the tracking device 10 assigns an ID to the detected tracking target (step S13). At this time, the ID that the tracking device 10 assigns to the tracking target is a temporary ID. On the other hand, when the tracking target is not detected from the verification frame (No in step S12), the process proceeds to step S18.

[0048] After step S13, the tracking device 10 extracts key points from the detected tracking target (step S14). If multiple tracking targets are detected, the tracking device 10 extracts key points for each of the detected tracking targets.

[0049] Next, the tracking device 10 generates posture information for each tracking target (step S15). The posture information is information obtained by integrating position information of key points extracted for each tracking target for each tracking target. When multiple tracking targets are detected, the tracking device 10 generates posture information for each detected tracking target.

[0050] Here, if there is a preceding frame (Yes in step S16), the tracking device 10 executes the tracking process (step S17). On the other hand, if there is no preceding frame (No in step S16), the tracking device 10 proceeds to step S18. Details of the tracking process will be described later with reference to the flowchart in FIG.

[0051] If there is a further subsequent frame (Yes in step S18), the process returns to step S11. On the other hand, if there is no further subsequent frame (No in step S18), the process according to the flowchart in FIG.

[0052] Fig. 6 is a flowchart for explaining the tracking process by the tracking unit 17 of the tracking device 10. In Fig. 6, first, the tracking unit 17 calculates the distances related to the positions and orientations between the tracking targets for the preceding and subsequent frames (step S171).

[0053] Next, the tracking unit 17 calculates a score between the tracking targets from the distances between the tracking targets in terms of their positions and orientations (step S172). For example, the tracking unit 17 calculates the sum of the distance between the tracking targets in terms of their positions and the distance in terms of their orientations as the score.

[0054] Next, the tracking unit 17 selects an optimal combination of tracking targets according to the scores between the tracking targets (step S173). For example, the tracking unit 17 selects a combination of tracking targets that has the smallest score from the preceding frame and the following frame.

[0055] Next, the tracking unit 17 assigns an ID to the tracking target detected from the subsequent frame according to the selected combination (step S174). For example, the tracking unit 17 assigns the same ID to the combination of tracking targets that has the smallest score in the preceding and subsequent frames.

[0056] As described above, the tracking device of the tracking system of this embodiment includes a detection unit, an extraction unit, a posture information generation unit, and a tracking unit. The detection unit detects a tracking target from at least two frames constituting video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0057] The tracking device of this embodiment tracks the tracking target based on the position and orientation of the posture information of the tracking target. If the tracking target is tracked only by its position, there is a possibility that the identification numbers of different tracking targets will be swapped when multiple tracking targets pass each other. Since the tracking device of this embodiment tracks the tracking target based not only on the position of the tracking target but also on the orientation of the tracking target, there is a low possibility that the identification numbers of different tracking targets will be swapped when multiple tracking targets pass each other. Therefore, according to the tracking device of this embodiment, it is possible to track multiple tracking targets across multiple frames based on the posture of the tracking target. That is, according to the tracking device of this embodiment, multiple tracking targets can be tracked based on their posture in multiple frames constituting a video.

[0058] Moreover, according to the tracking device of this embodiment, even if a reference image for each posture of each tracking target is not stored in a database, the tracking accuracy is not reduced even if conditions such as the degree of congestion, the angle of view, the distance between the camera and the tracking target, and the frame rate are different from the learned conditions. That is, according to this embodiment, the tracking target in the frames constituting the video can be tracked with high accuracy. The tracking device of this embodiment can be applied to, for example, monitoring the movement of people in the city, public facilities, stores, etc.

[0059] In one aspect of the present embodiment, the tracking unit calculates a score according to a distance related to a position and an orientation of the tracking target detected from each of at least two frames based on the posture information. The tracking unit tracks the tracking target based on the calculated score. According to this aspect, by tracking the tracking target based on a score according to a distance related to the position and orientation of the tracking target, it is possible to continuously track multiple tracking targets between frames constituting a video.

[0060] In one aspect of the present embodiment, the tracking unit tracks a pair of tracking targets detected from at least two frames that have the smallest score as the same tracking target. According to this aspect, by identifying the pair of tracking targets that have the smallest score as the same tracking target, it is possible to more continuously track the tracking target between frames that make up the video.

[0061] In one aspect of this embodiment, the tracking unit calculates, for the tracked object detected from each of the at least two frames, a weighted average of absolute values ​​of differences in coordinate values ​​of key points as a distance related to position. The tracking unit calculates, for the tracked object detected from each of the at least two frames, a weighted average of absolute values ​​of differences in relative coordinate values ​​in a specific direction with respect to a reference point of the key points as a distance related to orientation. The tracking unit calculates, for the tracked object detected from each of the at least two frames, a sum of the distance related to position and the distance related to orientation as a score. According to this aspect, weights related to position and orientation are clearly defined, and the tracked object can be appropriately tracked between frames according to the weighting.

[0062] In one aspect of the present embodiment, the tracking device includes a tracking information output unit that outputs tracking information related to tracking of the tracking target. The tracking information is, for example, an image in which key points are displayed at the position of the tracking target detected from the verification frame. According to this aspect, the posture of the tracking target can be visually grasped by displaying an image in which the tracking information is superimposed on the tracking target on the screen of a display device.

[0063] Second Embodiment Next, a tracking system according to a second embodiment will be described with reference to the drawings. The tracking system of this embodiment differs from the first embodiment in that the distances between the tracking targets in terms of their positions and orientations are normalized by the size of the tracking targets in the frame.

[0064] (composition) Fig. 7 is a block diagram showing an example of the configuration of a tracking system 2 of this embodiment. The tracking system 2 includes a tracking device 20, a monitoring camera 210, and a terminal device 220. Although Fig. 7 shows only one monitoring camera 210 and one terminal device 220, there may be a plurality of monitoring cameras 210 and terminal devices 220. Each of the monitoring camera 210 and the terminal device 220 is similar to each of the monitoring camera 110 and the terminal device 120 of the first embodiment, and therefore a detailed description thereof will be omitted.

[0065] The tracking device 20 has an image acquisition unit 21, a storage unit 22, a detection unit 23, an extraction unit 25, a posture information generation unit 26, a tracking unit 27, and a tracking information output unit 28. For example, the tracking device 20 is disposed on a server or a cloud. For example, the tracking device 20 may be provided as an application installed on a terminal device 220. Each of the image acquisition unit 21, the storage unit 22, the detection unit 23, the extraction unit 25, the posture information generation unit 26, and the tracking information output unit 28 is similar to the corresponding configuration in the first embodiment, and therefore detailed description thereof will be omitted.

[0066] The tracking unit 27 tracks the target between frames using the posture information generated for the target detected from the preceding frame and the posture information generated for the target detected from the preceding frame. The tracking unit 27 tracks the target based on the position and orientation of the posture information of the target detected from at least two frames. The tracking unit 27 tracks the target by assigning the ID of the target detected from the preceding frame to the target identified as the target detected from the preceding frame among the targets detected from the subsequent frame. Note that, if a target corresponding to the target detected from the subsequent frame is not detected from the preceding frame, the temporary ID assigned to the target detected from the subsequent frame may be set as the official ID, or a new ID may be assigned as the official ID.

[0067] For example, the tracking unit 27 calculates the distances related to the positions and orientations normalized by the size of the tracking targets for all tracking targets detected from the preceding frame and all tracking targets detected from the subsequent frame in a brute force calculation. The tracking unit 27 calculates the sum of the distances related to the positions and orientations normalized by the size of the tracking targets, calculated for all tracking targets detected from the preceding frame and all tracking targets detected from the subsequent frame, as a normalized score. The tracking unit 27 tracks the tracking targets by assigning the same ID to the tracking targets having the smallest normalized score among the pairs of tracking targets detected from the preceding frame and tracking targets detected from the subsequent frame. For example, if the tracking target person in the frame is walking upright, the size can be estimated by surrounding the person with a frame such as a rectangle. However, if the tracking target person in the frame is sitting or frequently changing direction, it is difficult to estimate the size by simply surrounding the person with a frame such as a rectangle. In such a case, the size can be estimated based on the skeleton of the tracking target person as described below.

[0068] FIG. 8 is a conceptual diagram for explaining a skeleton line used when the tracking unit 27 estimates the size of a tracking target (person). A skeleton line is a line segment connecting specific key points. FIG. 8 is a diagram of a person viewed from the front. In the example of FIG. 8, 14 key points are set for one person, and 15 skeleton lines are set. L1 is a line segment connecting HD and N. L21 is a line segment connecting N and RS, and L22 is a line segment connecting N and LS. L31 is a line segment connecting RS and RE, and L32 is a line segment connecting LS and LE. L41 is a line segment connecting RE and RH, and L42 is a line segment connecting LE and LH. L51 is a line segment connecting N and RW, and L52 is a line segment connecting N and LW. L61 is a line segment connecting RW and RK, and L62 is a line segment connecting LW and LK. L71 is a line segment connecting RK and RF, and L42 is a line segment connecting LK and LF. The number of key points set for one person is not limited to 14. The number of skeleton lines set for one person is not limited to 13. The positions of the key points and skeleton lines are not limited to the example in FIG.

[0069] The tracking unit 27 calculates the height of the person when standing upright (called height pixel number) based on the skeleton lines corresponding to the person in the verification frame. The height pixel number corresponds to the height of the person in the verification frame (the total body length of the person in two frames). The tracking unit 27 obtains the height pixel number (number of pixels) from the length of each skeleton line in the frame.

[0070] For example, the tracking unit 27 estimates the height pixel number using the length of the skeleton line from the head (HD) to the feet (RF, LF). For example, the tracking unit 27 estimates the height pixel number by using the sum H of the lengths of L1, L51, L61, and L71 in the verification frame among the skeleton lines extracted from the person in the verification frame. R For example, the tracking unit 27 calculates the sum H of the lengths L1, L52, L62, and L72 of the skeleton lines extracted from the person in the verification frame. L For example, the tracking unit 27 calculates the sum H of the lengths L1, L51, L61, and L71 in the verification frame. R and the sum of the lengths of L1, L52, L62, and L72 in the validation frame, H L For example, in order to calculate a more accurate height pixel count, the tracking unit 27 may calculate the height pixel count after correcting each skeleton line with a correction coefficient for correcting the inclination, posture, etc. of each skeleton line.

[0071] For example, the tracking unit 27 may estimate the height pixel number using the length of each skeletal line based on the relationship between the length of each skeletal line of an average person and the height. For example, the length of the skeletal line (L1) connecting the head (HD) and neck (N) is about 20 percent of the height. For example, the length of the skeletal line connecting the elbows (RE, LE) and the hands (RH, LH) is about 25 percent of the height. In this way, if the ratio of the length of each skeletal line to the height is stored in a storage unit (not shown), the height pixel number corresponding to the height of the person can be estimated based on the length of each skeletal line of the person detected from the verification frame. In addition, the ratio of the length of each skeletal line of the average person to the height tends to differ depending on the age. Therefore, the ratio of the length of each skeletal line of the average person to the height may be stored in the storage unit for each age of the person. For example, if the ratio of the length of each skeletal line of the average person to the height is stored in the storage unit, when an upright person is detected from the verification frame, the rough age of the person can be estimated based on the length of each skeletal line of the person. The above-mentioned method of estimating the number of height pixels based on the length of the skeleton line is merely an example, and does not limit the method of estimating the number of height pixels by the tracking unit 27.

[0072] The tracking unit 27 calculates the distance D p and the distance D d Here, for the person being compared, the height detected from the previous frame is normalized as H p , the height detected from the subsequent frame is H s The tracking unit 27 calculates a normalized distance ND p is calculated using the following formula 6, and the normalized distance ND d is calculated using the following formula 7.

number

number

number

[0073] (operation) Next, an example of the operation of the tracking device 20 will be described with reference to the drawings. The outline of the process by the tracking device 20 is the same as that of the first embodiment, and therefore will not be described. In the following, the tracking process by the tracking unit 27 of the tracking device 20 will be described in detail.

[0074] Fig. 9 is a flowchart for explaining the tracking process by the tracking unit 27 of the tracking device 20. In Fig. 9, first, the tracking unit 27 estimates the number of pixels of the height of the tracked object based on the skeleton line of the detected object detected from the verification frame (step S271).

[0075] Next, the tracking unit 27 calculates the normalized distance between the tracked objects in terms of position and orientation for the preceding and succeeding frames (step S272). The normalized distance is the distance in terms of position and orientation normalized by the estimated height pixel count.

[0076] Next, the tracking unit 27 calculates a normalized score between the tracking targets from the normalized distances between the tracking targets with respect to the positions and orientations (step S273). For example, the tracking unit 17 calculates the sum of the normalized distance between the tracking targets with respect to the positions and the normalized distance with respect to the orientations as the normalized score.

[0077] Next, the tracking unit 27 selects an optimal combination of tracking targets according to the normalized score between the tracking targets (step S274). For example, the tracking unit 27 selects a combination of tracking targets that has a minimum normalized score from the preceding frame and the following frame.

[0078] Next, the tracking unit 27 assigns an ID to the tracking target detected from the subsequent frame according to the selected combination (step S275). For example, the tracking unit 27 assigns the same ID to the combination of tracking targets in the preceding and subsequent frames that results in the smallest normalized score.

[0079] As described above, the tracking device of the tracking system of this embodiment includes a detection unit, an extraction unit, a posture information generation unit, and a tracking unit. The detection unit detects a tracking target from at least two frames constituting video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0080] Furthermore, in this embodiment, the tracking unit estimates the height pixel number of the tracking target based on a skeleton line connecting any of the multiple key points. The tracking unit normalizes a score by the estimated height pixel number, and tracks the tracking target detected from each of the at least two frames according to the normalized score.

[0081] In this embodiment, the score is normalized according to the size of the tracking target in the frame. Therefore, according to this embodiment, the tracking target that appears large due to its positional relationship with the surveillance camera is not overestimated, and tracking bias in the position within the frame can be reduced. Therefore, according to this embodiment, tracking with higher accuracy is possible across multiple frames that make up the video. Furthermore, according to this embodiment, tracking can be performed regardless of the posture of the tracking target, so that tracking of the tracking target can be continued even if the posture changes significantly between frames.

[0082] In one aspect of the present embodiment, the tracking device includes a tracking information output unit that outputs tracking information related to tracking of the tracking target. The tracking information is, for example, an image in which a skeleton line is displayed at the position of the tracking target detected from the verification frame. According to this aspect, an image in which the tracking information is superimposed on the tracking target is displayed on the screen of a display device, making it easier to visually grasp the posture of the tracking target.

[0083] (Third embodiment) Next, a tracking system according to a third embodiment will be described with reference to the drawings. The tracking system of this embodiment is different from the first and second embodiments in that a user interface for setting weights of position and orientation and setting key points is displayed.

[0084] (composition) Fig. 10 is a block diagram showing an example of the configuration of a tracking system 3 of this embodiment. The tracking system 3 includes a tracking device 30, a surveillance camera 310, and a terminal device 320. Although Fig. 10 shows only one surveillance camera 310 and one terminal device 320, there may be multiple surveillance cameras 310 and multiple terminal devices 320. The surveillance camera 310 is similar to the surveillance camera 110 of the first embodiment, and therefore a detailed description thereof will be omitted.

[0085] The tracking device 30 includes an image acquisition unit 31, a storage unit 32, a detection unit 33, an extraction unit 35, a posture information generation unit 36, a tracking unit 37, a tracking information output unit 38, and a setting acquisition unit 39. For example, the tracking device 30 is disposed on a server or a cloud. For example, the tracking device 30 may be provided as an application installed on a terminal device 320. Each of the image acquisition unit 31, the storage unit 32, the detection unit 33, the extraction unit 35, the posture information generation unit 36, the tracking unit 37, and the tracking information output unit 38 is similar to the corresponding configuration in the first embodiment, and therefore detailed description thereof will be omitted.

[0086] Fig. 11 is a block diagram showing an example of the configuration of a terminal device 320 etc. The terminal device 320 has a tracking information acquisition unit 321, a tracking information storage unit 322, a display unit 323, and an input unit 324. Fig. 11 also illustrates the tracking device 10, an input device 327, and a display device 330 connected to the terminal device 320.

[0087] The tracking information acquisition unit 321 acquires tracking information for each of a plurality of frames constituting the video data from the tracking device 30. The tracking information acquisition unit 321 stores the tracking information for each frame in the tracking information storage unit 322.

[0088] The tracking information storage unit 322 stores the tracking information acquired from the tracking device 30. The tracking information stored in the tracking information storage unit 322 is displayed as a GUI (Graphical User Interface) on the screen of the display unit 323 in response to, for example, a user operation.

[0089] The display unit 323 is connected to a display device 330 having a screen. The display unit 323 acquires tracking information from the tracking information storage unit 322. The display unit 323 displays display information including the acquired tracking information on the screen of the display device 330. Note that the terminal device 320 may include the function of the display device 330.

[0090] For example, the display unit 323 accepts an operation by a user via the input unit 324, and displays display information corresponding to the accepted operation content on the screen of the display device 330. For example, the display unit 323 displays display information corresponding to a frame having a frame number specified by the user on the screen of the display device 330. For example, the display unit 323 displays display information corresponding to each of a series of multiple frames including the frame having the frame number specified by the user on the screen of the display device 330 in chronological order.

[0091] For example, the display unit 323 may display at least one piece of display information on the screen of the display device 330 according to a preset display condition. For example, the preset display condition is a condition that a plurality of pieces of display information corresponding to a predetermined number of consecutive frames including a preset frame number is displayed in chronological order. For example, the preset display condition is a condition that a plurality of pieces of display information corresponding to a plurality of frames generated in a predetermined time period including a preset time is displayed in chronological order. Note that the display condition is not limited to the example given here as long as it is preset.

[0092] The input unit 324 is connected to an input device 327 that accepts operations by a user. For example, the input device 327 is realized by a keyboard, a touch panel, a mouse, or the like. The input unit 324 outputs the contents of the user's operation input via the input device 327 to the tracking device 30. Furthermore, when the input unit 324 accepts designation of video data, frames, display information, or the like from a user, it outputs an instruction to the display unit 323 to display the designated image on the screen.

[0093] The setting acquisition unit 39 acquires settings input using the terminal device 320. The setting acquisition unit 39 acquires weight settings related to position and orientation, key point settings, etc. The setting acquisition unit 39 reflects the acquired settings in the functions of the tracking device 30.

[0094] FIG. 12 is a conceptual diagram for explaining an example of display information displayed on the screen of the display device 330. A weight setting area 340 and an image display area 350 are set on the screen of the display device 330. A first operation image 341 for setting a weight related to a position and a second operation image 342 for setting a weight related to a direction are displayed in the setting area 340. A tracking image for each frame constituting the video captured by the surveillance camera 310 is displayed in the image display area 350. Note that a display area other than the weight setting area 340 and the image display area 350 may be set on the screen of the display device 330. Also, the display positions of the weight setting area 340 and the image display area 350 on the screen can be changed arbitrarily.

[0095] A scroll bar for setting a weight for a position is displayed in the first operation image 341. The weight for a position is an index value indicating how much importance is attached to the position of a tracking target detected from each of a preceding frame and a subsequent frame when comparing the tracking targets. The weight for a position is set in a range of 0 to 1. A minimum value (left end) and a maximum value (right end) of the weight for a position are set in the scroll bar displayed in the first operation image 341. When the knob 361 on the scroll bar is moved left and right, the weight for a position is changed. In the example of FIG. 12, the weight for a position is set to 0.8. Note that a vertical scroll bar may be displayed in the first operation image 341 instead of a horizontal scroll bar. Also, a spin button or a combo box for setting a weight for a position may be displayed in the first operation image 341 instead of a scroll bar. Also, an element other than a scroll bar may be displayed in the first operation image 341 in order to set a weight for a position.

[0096] A scroll bar for setting a weight for the orientation is displayed in the second operation image 342. The weight for the orientation is an index value indicating how much importance is attached to the orientation of the tracking target detected from each of the preceding and subsequent frames when comparing the tracking targets. The weight for the orientation is set in a range of 0 to 1. A minimum value (left end) and a maximum value (right end) of the weight for the orientation are set in the scroll bar displayed in the second operation image 342. When the knob 362 on the scroll bar is moved left and right, the weight for the orientation is changed. In the example of FIG. 12, the weight for the orientation is set to 0.2. Note that a vertical scroll bar may be displayed in the second operation image 342 instead of a horizontal scroll bar. Also, a spin button or a combo box for setting the weight for the orientation may be displayed in the second operation image 342 instead of a scroll bar. Also, an element other than a scroll bar or the like may be displayed in the second operation image 342 in order to set the weight for the orientation.

[0097] In the example of Fig. 12, frames including six people assigned IDs 11 to 16 as tracking targets are displayed in the image display area 350. Fig. 12 shows an example in which an image corresponding to a subsequent frame is displayed in the image display area 350. The previous frame and the subsequent frame may be displayed side by side in the image display area 350. Also, the previous frame and the subsequent frame may be displayed in the image display area 350 so as to be switched between in response to the selection of a button (not shown) or the like.

[0098] In the example of FIG. 12, tracking information associated with a person detected from a frame is displayed. The tracking information is displayed by associating a plurality of key points extracted from a person detected from a frame with a line segment (skeleton line) connecting the key points. For example, it may be possible to switch whether or not to display tracking information in the image display area 350 in response to a user's operation via the terminal device 320. In the example of FIG. 12, six people are walking in the same direction. In this way, when there are many tracking targets moving in the same direction, it is better to emphasize the position rather than the orientation in order to track the tracking targets between frames with high accuracy. When there are many tracking targets moving in the same direction, if the weights related to the position and the weights related to the orientation are set to be the same, the weights related to the orientation may be overestimated, and the tracking accuracy may decrease. Therefore, when there are many tracking targets moving in the same direction, the weights related to the position may be set to be large, and the weights related to the orientation may be set to be low, thereby reducing the decrease in tracking accuracy.

[0099] FIG. 13 is a conceptual diagram for explaining another example of display information displayed on the screen of the display device 330. In the example of FIG. 13, the weighting on the position is set to 0.2, and the weighting on the orientation is set to 0.8. In the example of FIG. 13, six people are walking so as to pass each other. In this way, when there are many tracking targets moving so as to pass each other, it is better to emphasize the orientation rather than the position in order to track the tracking targets with high accuracy between frames. When there are many tracking targets moving so as to pass each other, if the weighting on the orientation and the weighting on the position are set to be the same, the weighting on the position may be overestimated, and the tracking accuracy may decrease. Therefore, when there are many tracking targets moving so as to pass each other, the weighting on the orientation is set to be large, and the weighting on the position is set to be low, thereby reducing the decrease in tracking accuracy.

[0100] Fig. 14 is a conceptual diagram for explaining yet another example of display information displayed on the screen of display device 330. In the example of Fig. 14, third operation image 343 for setting weights related to position and orientation, and fourth operation image 344 for setting weights related to position and orientation according to a scene are displayed in weight setting area 340. Note that third operation image 343 and fourth operation image 344 do not have to be displayed in weight setting area 340 at the same time.

[0101] A scroll bar for setting the weights for the position and the orientation is displayed in the third operation image 343. The maximum value (left end) of the weight for the position and the maximum value (right end) of the weight for the orientation are set in the scroll bar displayed in the first operation image 341. When the weight for the position is set to the maximum value (left end), the weight for the orientation is set to the minimum value. On the other hand, when the weight for the orientation is set to the maximum value (right end), the weight for the position is set to the minimum value. When the knob 363 on the scroll bar is moved left and right, the weights for the position and the orientation are changed all at once. Note that a vertical scroll bar may be displayed in place of a horizontal scroll bar in the third operation image 343. Also, a spin button or a combo box for setting the weights for the position and the orientation may be displayed in place of a scroll bar in the third operation image 343. Also, an element other than the scroll bar may be displayed in the third operation image 343 in order to set the weights for the position and the orientation. The weights for the position and the orientation are often in a complementary relationship depending on the scene. Therefore, in a scene where the weighting on the position is important, it is better to make the weighting on the orientation smaller. Conversely, in a scene where the weighting on the orientation is important, it is better to make the weighting on the position smaller. In the example of Fig. 14, the weighting on the position and orientation can be set collectively according to the state of the tracking target in the frame displayed in the image display area 350, so that the setting of the weighting on the position and orientation can be appropriately changed according to the scene.

[0102] In the fourth operation image 344, a check box for setting weights for the position and orientation according to the scene is displayed. FIG. 14 shows an example in which weights according to the "passing" scene are set in response to the operation of the pointer 365 via the terminal device 320. In the example of FIG. 14, when any scene is selected in the fourth operation image 344, the setting of the third operation image 343 is changed at the same time. For example, in a scene in which many people pass each other, it is preferable to emphasize the orientation in consideration of the face orientation so that IDs are less likely to be mixed up between the tracking targets that pass each other. For example, when a "passing" scene is selected, the position weight is set to 0.2 and the orientation weight is set to 0.8. For example, in a scene in which many people move in the same direction, the position may be emphasized regardless of the face orientation. For example, when a "same orientation" scene is selected, the position weight is set to 0.8 and the orientation weight is set to 0.2. By selecting a scene according to the situation of the tracking target in the frame displayed in the image display area 350, the setting of weights for the position and orientation can be intuitively changed.

[0103] FIG. 15 is a conceptual diagram for explaining another example of display information displayed on the screen of the display device 330. A key point designation area 370 and a key point designation area 380 are set on the screen of the display device 330. An individual designation image 371 and a collective designation image 372 are displayed on the key point designation area 370. An image in which the key points designated in the key point designation area 370 are associated with a human body is displayed on the key point designation area 380. For example, key points are designated according to the selection of each key point in the individual designation image 371 or the selection of a body part in the collective designation image 372. In the example of FIG. 15, all key points designated in the individual designation image 371 are displayed in the key point designation area 380. In the key point designation area 380, the selected key points are displayed in a state of being filled in black. Note that display areas other than the key point designation area 370 and the key point designation area 380 may be set on the screen of the display device 330. Also, the display positions of the key point designation area 370 and the key point designation area 380 on the screen can be changed arbitrarily.

[0104] FIG. 16 is a conceptual diagram for explaining yet another example of display information displayed on the screen of the display device 330. In the example of FIG. 16, the "trunk" is selected in the collective designation image 372 in response to the operation of the pointer 365 via the terminal device 320. When the "trunk" is selected in the collective designation image 372, the head (HD), neck (N), right hip (RW), and left hip (LW) are designated collectively. In the example of FIG. 16, the key points of the "trunk" designated in the collective designation image 372 are displayed in the key point designation area 380. In the key point designation area 370, the selected key points are displayed in a state of being filled in black. For example, the two hands and both feet have a larger change between frames than the trunk, so if the weight is too large, the tracking accuracy may decrease. Therefore, the weights of the two hands and both feet may be set to be smaller by default than the weight of the trunk.

[0105] For example, when "upper body" is selected, the head (HD), neck (N), right shoulder (RS), left shoulder (LS), right elbow (RE), left elbow (LE), right hand (RH), and left hand (LH) are specified all at once. For example, when "lower body" is selected, the right hip (RW), left hip (LW), right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once. For example, when "right body" is selected, the right shoulder (RS), right elbow (RE), right hand (RH), right knee (RK), and right foot (RF) are specified all at once. For example, when "left body" is selected, the left shoulder (LS), left elbow (LE), left hand (LH), left knee (LK), and left foot (LF) are specified all at once. For example, when "hands and feet" is selected, the right elbow (RE), left elbow (LE), right hand (HR), left hand (LH), right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once. For example, when "arms" is selected, the right elbow (RE), left elbow (LE), right hand (RH), and left hand (LH) are specified all at once. For example, when "feet" is selected, the right knee (RK), left knee (LK), right foot (RF), and left foot (LF) are specified all at once.

[0106] For example, the weight of a selected keypoint is set to 1, and the weight of a non-selected keypoint is set to 0. For example, when the upper body is selected, the weight of a keypoint included in the upper body is set to 1. For example, when the upper body is selected, the weight of a keypoint included in the upper body may be set to 1, and the weight of a keypoint included in the lower body may be set to 0.5.

[0107] The key points selected collectively when selected in the collective designation image 372 as described above are merely an example, and may be a combination different from the above. For example, instead of selecting key points collectively according to body parts, a set of appropriate key points according to a scene or situation may be prepared in advance, and the set of key points may be configured to be intuitively selected. For example, a model that has learned key points selected by a skilled user according to a scene or situation may be used to estimate appropriate key points according to a scene or situation. For example, questions for setting key points may be prepared, and key points may be set according to answers to the questions. If a set of prepared key points is configured to be selectable, even a user who does not have the skill to individually select key points according to a scene or situation can select appropriate key points in the same way as a skilled user.

[0108] FIG. 17 is an example in which tracking information is displayed in association with a person detected from a frame in a state in which "trunk" is selected and the head (HD), neck (N), right hip (RW), and left hip (LW) are specified collectively as in FIG. 16. The tracking information is displayed in association with four key points (HD, N, RW, LW) extracted from the person detected from the frame and the line segments (skeleton lines) connecting these key points. When the tracking target moves in the same direction as in FIG. 17, it is sufficient to grasp the position of the tracking target, so that the key points of the trunk, which moves relatively little, may be focused on and tracked. For example, the display information in FIGS. 15 to 16 and the display information in FIG. 17 may be configured to be switched by pressing a button (not shown) displayed on the screen of the display device 330.

[0109] (operation) Next, an example of the operation of the tracking device 30 will be described with reference to the drawings. The outline of the processing by the tracking device 30 is the same as that of the first embodiment, and therefore will be omitted. Below, the details of the setting processing in the tracking unit 37 of the tracking device 30 will be described. For example, it is inserted somewhere between steps S13 and S14 in FIG. 5. The setting processing is executed according to the designation of key points and the adjustment of weights of position and orientation.

[0110] 18, first, the tracking device 30 judges whether or not a key point (KP) is specified (step S31). If a key point is specified (Yes in step S31), the tracking device 30 sets the specified key point as an extraction target (step S32). On the other hand, if a key point is not specified (No in step S31), the tracking device 30 proceeds to step S33.

[0111] Next, if the weights of the position and orientation have been adjusted (Yes in step S33), the tracking device 30 sets the weights of the position and orientation according to the adjustment (step S34). After step S34, the process proceeds to the subsequent process in the flowchart of Fig. 5. If the weights of the position and orientation have not been adjusted (No in step S33), the process proceeds to the subsequent process in the flowchart of Fig. 5 without readjusting the weights of the position and orientation.

[0112] As described above, the tracking system of the present embodiment includes a surveillance camera, a tracking device, and a terminal device. The surveillance camera captures an image of a surveillance target range and generates video data. The terminal device is connected to a display device having a screen that displays display information generated by the tracking device. The tracking device includes a video acquisition unit, a storage unit, a detection unit, an extraction unit, a posture information generation unit, a tracking unit, a tracking information output unit, and a setting acquisition unit. The video acquisition unit acquires video data from the surveillance camera. The storage unit stores the acquired video data. The detection unit detects a tracking target from at least two frames that constitute the video data. The extraction unit extracts at least one key point from the detected tracking target. The posture information generation unit generates posture information of the tracking target based on the at least one key point. The tracking unit tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames. The tracking information output unit outputs tracking information related to tracking of the tracking target to the terminal device. The setting acquisition unit acquires settings inputted using the terminal device. The setting acquisition unit acquires weight settings related to position and orientation, key point settings, and the like. The setting acquisition unit reflects the acquired settings in the functions of the tracking device.

[0113] In this embodiment, the terminal device sets an image display area and a weight setting area on the screen of a display device. In the image display area, a tracking image in which key points are associated with a tracking target detected from a frame constituting the video data is displayed. In the weight setting area, an operation image for setting a weight related to a position and a weight related to an orientation is displayed. The terminal device outputs the weight related to a position and the weight related to an orientation set in the weight setting area to the tracking device. The tracking device acquires the weight related to a position and the weight related to an orientation selected in the weight setting area from the terminal device. The tracking device calculates a score according to a distance related to a position and an orientation related to the tracking target detected from each of at least two frames constituting the video data, using the acquired weight related to a position and the weight related to an orientation. The tracking device tracks the tracking target based on the calculated score.

[0114] In this embodiment, the weights related to the position and orientation can be arbitrarily adjusted in response to a user operation, and therefore, according to this embodiment, it is possible to realize highly accurate tracking of the tracking target based on the weights according to the user's request.

[0115] In one aspect of this embodiment, the terminal device displays an operation image in a weight setting area for setting a weight related to a position and a weight related to a direction according to a scene. The terminal device outputs the weight related to a position and a weight related to a direction according to a scene set in the weight setting area to the tracking device. According to this aspect, the weight related to a position and a weight related to a direction can be arbitrarily adjusted according to a scene. Therefore, according to this embodiment, it is possible to realize tracking of a tracking target with high accuracy suitable for a scene.

[0116] In one aspect of this embodiment, the terminal device sets a key point designation area on the screen of the display device, in which a designation image for designating key points used to generate posture information of the tracked object is displayed. The terminal device outputs key points selected in the key point area to the tracking device. The tracking device acquires the key points selected in the key point selection area from the terminal device. The tracking device generates posture information for the acquired key points. In this aspect, the key points used to generate posture information can be arbitrarily adjusted in response to a user's operation. Therefore, according to this embodiment, it is possible to achieve highly accurate tracking of the tracked object using posture information according to a user's request.

[0117] (Fourth embodiment) Next, a tracking device according to a fourth embodiment will be described with reference to the drawings. The tracking device of this embodiment has a simplified configuration of the tracking devices of the first to third embodiments. FIG. 19 is a block diagram showing an example of the configuration of a tracking device 40 of this embodiment. The tracking device 40 includes a detection unit 43, an extraction unit 45, a posture information generation unit 46, and a tracking unit 47.

[0118] The detection unit 43 detects a tracking target from at least two frames constituting the video data. The extraction unit 45 extracts at least one key point from the detected tracking target. The posture information generation unit 46 generates posture information of the tracking target based on the at least one key point. The tracking unit 47 tracks the tracking target based on the position and orientation of the posture information of the tracking target detected from each of the at least two frames.

[0119] As described above, the tracking device of this embodiment tracks a tracking target based on the position and orientation of the posture information of the tracking target, thereby being able to track a plurality of tracking targets based on their postures in frames constituting a video.

[0120] (Hardware) Here, a hardware configuration for executing the processing of a tracking device or a terminal device (hereinafter referred to as a tracking device, etc.) according to each embodiment of the present disclosure will be described by taking an information processing device 90 in Fig. 20 as an example. Note that the information processing device 90 in Fig. 20 is an example of a configuration for executing the processing of the tracking device, etc. according to each embodiment, and does not limit the scope of the present disclosure.

[0121] As shown in Fig. 20, an information processing device 90 includes a processor 91, a main storage device 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In Fig. 20, the interface is abbreviated as I / F (Interface). The processor 91, the main storage device 92, the auxiliary storage device 93, the input / output interface 95, and the communication interface 96 are connected to each other via a bus 98 so as to be able to communicate data with each other. In addition, the processor 91, the main storage device 92, the auxiliary storage device 93, and the input / output interface 95 are connected to a network such as the Internet or an intranet via the communication interface 96.

[0122] The processor 91 loads a program stored in the auxiliary storage device 93 or the like into the main storage device 92 and executes the loaded program. In this embodiment, a software program installed in the information processing device 90 may be used. The processor 91 executes processing by the tracking device or the like according to this embodiment.

[0123] The main storage device 92 has an area in which a program is loaded. The main storage device 92 may be a volatile memory such as a dynamic random access memory (DRAM). A non-volatile memory such as a magnetoresistive random access memory (MRAM) may also be configured or added as the main storage device 92.

[0124] The auxiliary storage device 93 stores various data. The auxiliary storage device 93 is configured with a local disk such as a hard disk or a flash memory. Note that it is also possible to configure the main storage device 92 to store various data, and omit the auxiliary storage device 93.

[0125] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices. The communication interface 96 is an interface for connecting to an external system or device through a network such as the Internet or an intranet based on standards and specifications. The input / output interface 95 and the communication interface 96 may be a common interface for connecting to an external device.

[0126] The information processing device 90 may be configured to be connected to input devices such as a keyboard, a mouse, and a touch panel, as necessary. These input devices are used to input information and settings. When a touch panel is used as the input device, the display screen of the display device may be configured to also function as an interface for the input device. Data communication between the processor 91 and the input devices may be mediated by an input / output interface 95.

[0127] The information processing device 90 may be equipped with a display device for displaying information. When the display device is equipped, the information processing device 90 is preferably equipped with a display control device (not shown) for controlling the display of the display device. The display device may be connected to the information processing device 90 via the input / output interface 95.

[0128] The information processing device 90 may also be equipped with a drive device. The drive device mediates between the processor 91 and a recording medium (program recording medium) for reading data and programs from the recording medium, writing the processing results of the information processing device 90 to the recording medium, and the like. The drive device may be connected to the information processing device 90 via an input / output interface 95.

[0129] The above is an example of a hardware configuration for enabling the tracking device and the like according to each embodiment of the present invention. The hardware configuration in FIG. 20 is an example of a hardware configuration for executing the arithmetic processing of the tracking device and the like according to each embodiment, and does not limit the scope of the present invention. In addition, a program for causing a computer to execute processing related to the tracking device and the like according to each embodiment is also included in the scope of the present invention. Furthermore, a program recording medium on which the program according to each embodiment is recorded is also included in the scope of the present invention. The recording medium can be realized, for example, by an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). In addition, the recording medium may be realized by a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card, a magnetic recording medium such as a flexible disk, or other recording medium. When the program executed by the processor is recorded on a recording medium, the recording medium corresponds to a program recording medium.

[0130] The components of the tracking device and the like of each embodiment can be combined in any manner. Furthermore, the components of the tracking device and the like of each embodiment can be realized by software or by a circuit.

[0131] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0132] 1, 2, 3 Tracking System 10, 20, 30, 40 Tracking Device 11, 21, 31 Video acquisition section 12, 22, 32 storage section 13, 23, 33, 43 Detector 15, 25, 35, 45 Extraction part 16, 26, 36, 46 Attitude information generation section 17, 27, 37, 47 Tracking Section 18, 28, 38 Tracking information output section 39 Setting acquisition unit 110, 210, 310 Surveillance Cameras 120, 220, 320 terminal equipment 321 Tracking Information Acquisition Department 322 Tracking information storage unit 323 Display section 324 Input section 327 Input Devices 330 Display equipment

Claims

1. On the computer, A process of displaying a user interface for setting weights on a screen of a display device; a process of tracking a person or object in an image using key points of the person or object based on the set weights; To set the weights, A process of setting a first index representing how much importance is attached to the position of the tracking target; and a program for executing a process of setting a second index that indicates how much importance is to be attached to the orientation of the tracked object.

2. A process of displaying a means for allowing a user to select a scene; and setting at least one of the first index and the second index in accordance with a selected scene.

3. The program according to claim 1 , causing a computer to execute a process of, when one of the first index or the second index is changed, changing the other index in accordance with the changed index.

4. The program according to claim 1 , wherein when a person is tracked in a process of tracking the person or object, the program causes a computer to execute a process of tracking the person using key points acquired based on a skeleton of the person.

5. The computer A user interface for setting weights is displayed on a screen of a display device; Tracking the person or object in the image using key points of the person or object based on the set weights; To set the weights, A first index is set to indicate how much importance is attached to the position of the tracking target; A tracking method in which a second index is set that indicates how much importance is attached to the orientation of the tracked object.

6. a means for displaying a user interface for setting the weights on a screen of a display device; a means for tracking a person or object appearing in an image using key points of the person or object based on the set weights; To set the weights, A means for setting a first index representing how much importance is attached to the position of the tracking target; and means for setting a second index indicating how much importance is to be attached to the orientation of the tracked object.

Images