Object tracking method, device and storage medium

Abstract

The present disclosure provides an object tracking method, device and storage medium. The present disclosure improves the accuracy of tracking a person by jointly detecting a face and a body part having a fixed position relationship with the face.

Description

Technical Field

[0001] The present disclosure relates to detection and tracking of objects, and more particularly, to detection and tracking of people in a sequence of image frames. Background Art

[0002] In recent years, with the development of target detection technology, target tracking technology based on target detection has also attracted more and more attention, especially the application of human tracking technology in videos (image frame sequences) shot by surveillance cameras. In video tracking, it is necessary to detect the person as the tracked object in each image frame, and then combine the detection results of the person in each frame to determine the tracking trajectory of the person.

[0003] Human tracking technology can be applied in the following scenarios:

[0004] 1) Count the number of pedestrians. Use a camera to shoot a video of a certain location and estimate the flow of people at the location by counting the number of pedestrian tracks in the video.

[0005] 2) Human identification: Track the person in the video and use face recognition technology to determine the identity of the person being tracked.

[0006] 3) Human behavior analysis: Track the people in the video and determine their various behaviors by analyzing their movement trajectories.

[0007] In addition to the above scenarios, human tracking technology can also be widely used in other scenarios, which are not listed here. In the above human tracking technology, it is necessary to detect people in each frame, and the commonly used detection method is face detection. However, when the visible state of the face in the video frame changes, such as turning the head, turning around, the face is blocked by a water cup when drinking water, wearing a mask to block the face, etc., it is impossible to detect the person based on face detection, and it is easy to have the problem of target tracking loss or tracking error. If face detection is not used as the detection of people, but body detection is used as the detection of people, when people are in a crowded or blocked situation, it is also impossible to detect people based on body detection, and it is also easy to have the problem of target tracking loss or tracking error.

[0008] US Patent No. 8,929,598 B2 discloses a tracking technology for a person, which first uses face detection as the detection of the person, and if the tracking based on face detection fails, then uses body (or part of the body) detection as the detection of the person, and continues to track the person based on body detection. However, in the tracking technology disclosed in US Patent No. 8,929,598 B2, the detection of the body is often inaccurate. Specifically, when tracking based on face detection fails in the current frame, when performing body detection, if the detection area of ​​the body is estimated based on the detection area of ​​the face in the current frame, the estimated detection area of ​​the body is also inaccurate due to the inaccurate detection area of ​​the face, and the tracking based on body detection may ultimately fail; if the detection area of ​​the body in the current frame is estimated based on the motion information of the body in the previous frames, and tracking is successfully performed based on face detection in the previous frames, the motion information of the body in the previous frames is not updated in real time, that is, the motion information of the body in the previous frames cannot truly reflect the detection area of ​​the body in the current frame, which also leads to inaccurate detection area of ​​the body, and the tracking based on body detection ultimately fails. Summary of the invention

[0009] The present disclosure is proposed in view of the technical problems in the prior art, and aims to provide an improved object tracking technology.

[0010] The present disclosure proposes an improved object tracking technology, which realizes the detection of a person by jointly detecting the face and the body (or part of the body) that has a specific positional relationship with the face, and then realizes the tracking of the person, thereby avoiding tracking failure.

[0011] According to one aspect of the present disclosure, there is provided an object tracking method for an image frame sequence, wherein the image frame sequence comprises a plurality of image frames, each image frame comprising at least one object; the object tracking method comprises: determining a region of interest of a face-body part pair in a current frame based on face tracking results and tracking results of body parts having a certain positional relationship with the face stored in a created trajectory; detecting the face and body parts within the determined region of interest of the face-body part pair to obtain a detected face-body part pair; associating the detected face-body part pair with the trajectory, and when the association is successful, updating the trajectory using the detected face-body part pair.

[0012] According to another aspect of the present disclosure, there is provided an object tracking device for an image frame sequence, wherein the image frame sequence includes a plurality of image frames, each image frame includes at least one object; the object tracking device includes: a region of interest determination unit, which is constructed to determine the region of interest of a face-body part pair in a current frame based on face tracking results and tracking results of body parts having a certain positional relationship with the face stored in a created trajectory; a detection unit, which is constructed to detect the face and body parts within the determined region of interest of the face-body part pair to obtain a detected face-body part pair; an association unit, which is constructed to associate the detected face-body part pair with the trajectory; and an update unit, which is constructed to update the trajectory using the detected face-body part pair when the association is successful.

[0013] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a computer, the computer performs the above-mentioned object tracking method for an image frame sequence.

[0014] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description of the embodiments, serve to explain the principles of the present disclosure.

[0016] Figure 1 A flow chart of a known object tracking technique is shown.

[0017] Figure 2 Schematic diagram of the hardware structure for implementing the object tracking technology disclosed in the present invention.

[0018] Figure 3 FIG. 1 is a flowchart of the steps of the object tracking method according to the first exemplary embodiment of the present disclosure.

[0019] Figure 4 are examples of face detection frames and head and shoulders detection frames.

[0020] Figure 5 It is a flowchart for implementing step S102.

[0021] Figure 6(a) to Figure 6(e) is an example of a region of interest for determining a face-head-shoulder pair.

[0022] Figure 7(a) to Figure 7(c) is an example of face-head-shoulder pair detection.

[0023] Figure 8 It is a flowchart for implementing step S104.

[0024] Figure 9(a) to Figure 9(d) This is an example of face-head-shoulder pair detection when a person turns around.

[0025] Figure 10(a) to Figure 10(b) This is an example of face-head-shoulder pair detection when wearing a mask.

[0026] Figure 11(a) to Figure 11(c) This is an example of face-head-shoulder pair detection when multiple people are moving crosswise.

[0027] Fig.12 FIG. 1 is a schematic diagram showing the structure of an object tracking device according to a second exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Exemplary possible embodiments related to object tracking are described herein. In the following description, for the purpose of illustration, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent that the present disclosure can be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily blocking, obscuring, or obscuring the present disclosure.

[0029] Figure 1 The flowchart of the human body tracking process disclosed in the known U.S. patent US8,929,598 B2 is shown. First, the process of creating a track is performed, and face detection is performed from the first frame of the video image. Based on the face detection, the detection of the person is realized, and a track is created for the person (face) to be tracked. Here, the information included in the track includes but is not limited to: an ID for uniquely representing the track, a face template for face detection and a head and shoulder template for detecting body parts (here taking the head and shoulders as an example), and the position information of the tracked person in the current frame (i.e., the tracking result in the current frame). In addition to the above information, the created track can also reserve a storage area for storing the face tracking results and head and shoulder tracking results in several frames that have been tracked during subsequent frame-by-frame tracking.

[0030] After creating a track for the person being tracked, face detection can be performed in the real-time captured video frame, and the person can be detected based on face detection, so as to achieve the purpose of tracking the person. Taking the current frame as the i-th frame as an example, full-image face detection is performed in the i-th frame. Specifically, first, the region of interest of each face in the i-th frame is estimated based on the face motion information in the iN-th frame to the i-1-th frame; then, in the estimated region of interest of the face, face detection is performed using a face detector. After face detection, the detected face is associated with each track using a target association algorithm to determine whether there is a track associated with the detected face, that is, whether the detected face matches the face template in a certain track. If the detected face is successfully associated with a certain track, the position information of the detected face is used as the tracking result of the associated track in the i-th frame, and the detected face is used to update the face template in the associated track, and the face tracking result of the current frame is stored. If the detected face is not successfully associated with the track, it is necessary to further track it using head and shoulder detection.

[0031] In tracking based on head and shoulder detection, first, the region of interest of the head and shoulders is estimated based on the face region detected by the previous face detection. Then, the head and shoulder detector is used to perform head and shoulder detection in the region of interest of the head and shoulders. After the head and shoulder detection, the detected head and shoulders are associated with each track respectively using the target association algorithm to determine whether there is a track associated with the detected head and shoulders, that is, whether the detected head and shoulders match the head and shoulder template in a certain track. If the detected head and shoulders are successfully associated with a certain track, the position information of the detected head and shoulders is used as the tracking result of the associated track in the i-th frame, and the detected head and shoulders are used to update the head and shoulder template in the associated track, and the face tracking result of the current frame is stored. If the detected head and shoulders are not successfully associated with the track, the person represented by the face and head and shoulders is not the person being tracked.

[0032] Figure 1 The tracking technology shown is to perform tracking based on head and shoulder detection after tracking based on face detection fails. Due to changes in the visible state of the face, such as turning the head, turning around, wearing a mask, etc., the detected face area deviates from the actual face area or is smaller in area. In this case, the inaccurate face area is used to estimate the head and shoulder area of ​​interest, and the head and shoulder area of ​​interest is also inaccurate. Therefore, the accuracy of the tracking results based on head and shoulder detection is not high, and it is easy to have problems such as tracking loss or tracking target errors.

[0033] In view of this, the present disclosure proposes an improved object tracking technology, based on the joint detection of a face and a body part having a certain positional relationship with the face, and the detection result of the joint detection is associated with the trajectory to achieve tracking of the person, thereby improving the success rate of tracking and reducing the possibility of tracking loss or tracking target error. Here, the face information and body part information required for joint detection include but are not limited to: the positional relationship between the face and the body part, the detection of the face and the body part by the detector, the apparent features of the face and the body part (for example, the features on the face (eyes, nose, mouth) and the texture features of the clothes on the body part) and the motion information of the face and the body part. In addition, the body part that is jointly detected with the face is a body part with a relatively fixed positional relationship with the face, and even when the person moves (turning the head, turning around, walking, etc.), the positional relationship with the face will not change greatly, such as the head and shoulders, the upper body trunk, etc. For the convenience of description, the subsequent embodiments are explained by taking the joint detection and tracking of the face-head and shoulder pair as an example. It should be understood that the scheme of the present disclosure is not limited to the joint detection and tracking of the face-head and shoulder pair.

[0034] Various exemplary embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. It should be understood that the present disclosure is not limited to the various exemplary embodiments described below. In addition, as a solution to the problems of the present disclosure, it is not necessary to include a combination of features described in all exemplary embodiments.

[0035] Figure 2 The hardware environment for executing the object tracking method in the present disclosure is shown, which includes: a processor unit 10 , an internal memory unit 11 , a network interface unit 12 , an input unit 13 , an external memory 14 , and a bus unit 15 .

[0036] The processor unit 10 may be a CPU or a GPU. The internal memory unit 11 includes a random access memory (RAM) and a read-only memory (ROM). The RAM may be used as a main memory, a work area, etc. of the processor unit 10. The ROM may be used to store the control program of the processor unit 10. In addition, it may be used to store files or other data to be used when running the control program. The network interface unit 12 may be connected to a network and implement network communication. The input unit 13 controls input from devices such as a keyboard and a mouse. The external memory 14 stores startup programs and various applications, etc. The bus unit 15 is used to connect the units in the optimization device of the multi-layer neural network model.

[0037] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0038] <First Exemplary Embodiment>

[0039] Figure 3 The following is a flowchart of the steps of the object tracking method for an image frame sequence according to the first exemplary embodiment of the present disclosure. In the first embodiment, the CPU 10 executes the program stored in the ROM and / or the external memory 14 by using the RAM as the working memory. Figure 3 The object tracking process is shown. Note that in the context of the description, "image" refers to an image that can be in any suitable form, such as a video image in a video, etc., and "image" can be used interchangeably with "image frame" and "frame".

[0040] Step S101: performing face-head-shoulder pair detection in an image frame, and creating a track for a person to be tracked according to the detection result.

[0041] This step is the initial step of the tracking process, in which face-head-shoulder pairs are detected in the area of ​​the entire first video frame, and tracks are created for the detected face-head-shoulder pairs. The "first video frame" here can be the first frame received from the outside world after the object tracking program is initialized, or it can be the current frame in which a new person to be tracked appears. There may be multiple people in the entire first video frame, so in this step S101, the face-head-shoulder pairs of each person are detected separately, and tracks are created for each face-head-shoulder pair, thereby achieving tracking of multiple people. Of course, it is also possible to detect the face-head-shoulder pairs of a specific person according to the user's specification, and create tracks for the face-head-shoulder pairs, thereby achieving tracking of the specific person. The object tracking method disclosed in the present invention does not limit the number of tracked objects.

[0042] Each track represents the tracking information of a person to be tracked, and the contents of the track include but are not limited to: an ID for uniquely identifying the track, a face template and a head and shoulder template of the person to be tracked, and face tracking results and head and shoulder tracking results of the person to be tracked in the past M frames.

[0043] Where ID is the unique identification number of the track.

[0044] The face template and the head-shoulder template represent the face information and the head-shoulder information of the person being tracked. The face information and the head-shoulder information in the face template and the head-shoulder template are reliable information. In the subsequent tracking process, it can be determined whether the face and the head-shoulder detected in real time are associated with the current track based on the face template and the head-shoulder template. In the subsequent tracking process, in the case of each successful tracking, the face template and the head-shoulder template are updated using the face information and the head-shoulder information detected in the current frame of successful tracking, so that the face template and the head-shoulder template contained in the track are always the latest information.

[0045] Among them, the face tracking results and head and shoulder tracking results in the past M frames have not been generated when the track is initially created. Starting from the first frame of the track creation, after each successful tracking for a frame, the face tracking results and head and shoulder tracking results of the current frame are stored as information in the track, and after tracking more than M frames, the latest tracking results of the current frame overwrite the tracking results of the M frames before the current frame, so that the tracking results of the M frames closest to the current frame are always stored in the track. Here, M can be set according to an empirical value or an experimental value, for example, M=100.

[0046] In the first embodiment, when performing face-head-shoulder pair detection, an AdaBoost-based face detector can be used to detect the face, and an AdaBoost-based head-shoulder detector can be used to detect the head and shoulders. The first embodiment does not limit the choice of detector, and various detectors known in the industry can be applied to the method of the first embodiment.

[0047] Figure 4 Examples of a face detection frame detected by a face detector and a head and shoulder detection frame detected by a head and shoulder detector are shown. Since the positional relationship between the face and the head and shoulders is relatively fixed, the position and size relationship between the face detection frame and the head and shoulder detection frame can be set in advance.

[0048] Position relationship: IoM = OverlapArea / MinArea Formula (1)

[0049] Among them, IoM (Interaction of Minimum) represents the minimum overlap ratio of the face detection frame and the head and shoulder detection frame, and the value of IoM is not less than 0.9; OverlapArea represents the area of ​​the overlapping area between the face detection frame and the head and shoulder detection frame; MinArea represents the area of ​​the smaller detection frame between the face detection frame and the head and shoulder detection frame.

[0050] Size relationship: Size_ratio=Face_size / Omega_size Formula (2)

[0051] Among them, Face_size represents the side length of the face detection box; Omega_size represents the side length of the head and shoulder detection box; Size_ratio ranges from 0.3 to 0.6.

[0052] It should be noted that the positional relationship and size relationship between the face detection frame and the head and shoulders detection frame shown above are optional conditions for implementing the first embodiment of the present invention, but the first embodiment of the present invention is not limited to the above relationship, and the positional relationship and size relationship between the two can be limited according to experience or experimenters. In addition, the first embodiment of the present invention is described by taking the joint detection of the face and the head and shoulders as an example. If the joint detection of the face and other body parts, such as the upper body, is adopted, the positional relationship and size relationship between the face detection frame and the upper body detection frame need to be adaptively changed.

[0053] Step S102: In frame-by-frame tracking (assuming the current frame is the i-th frame), the face estimation area and the head and shoulders estimation area in the i-th frame are estimated based on the face tracking results and the head and shoulders tracking results stored in the trajectory, and the region of interest of the face-head and shoulders pair is determined based on the face estimation area and the head and shoulders estimation area.

[0054] Note that step S102 is executed after step S101, but not necessarily immediately after step S101. After the trajectory is created in step S101, the tracking process of the subsequent steps can be executed according to the real-time incoming video frames until the i-th frame arrives.

[0055] In step S102, the face estimation region and the head and shoulder estimation region of the current i-th frame may be estimated based on the motion estimation method according to the face tracking results and the head and shoulder tracking results of the i-1-th frame to the iM-th frame.

[0056] Figure 5 A flow chart of estimating the region of interest of the face-head-shoulder pair in the i-th frame in step S102 is shown, and is described in detail as follows.

[0057] Step S102-1: Obtain the face estimation area in the i-th frame through the face tracking results from the i-1th frame to the iMth frame.

[0058] Step S102-2: Determine the region of interest of the face in the i-th frame according to the obtained estimated face region.

[0059] by Figure 6(a) to Figure 6(e) Taking the case shown in the figure as an example, according to the face tracking results from the i-1th frame to the iMth frame, based on the face motion estimation, the position and size of the face estimation area are estimated (Figure 6(a)), and then the region of interest (RoI, Region of Interesting) of the face is determined (Figure 6(b)). An optional method for determining the region of interest of the face is:

[0060] Size_RoIface=w1*face_Size formula (3)

[0061] Wherein, face_Size represents the size of the face estimation region estimated according to motion estimation, and the region concentrically enlarged by w1 (for example, w1=3.0) times is used as the region of interest of the face.

[0062] Step S102-3: Obtain the head and shoulder estimation area in the i-th frame through the head and shoulder tracking results from the i-1th frame to the iMth frame.

[0063] Step S102 - 4 : Determine the head and shoulder region of interest of the i-th frame according to the obtained head and shoulder estimation region.

[0064] Similar to the estimation of the region of interest of the face in step S102-1 and step S102-2, based on the head and shoulder motion estimation of the first M frames, the position and size of the estimated region of the head and shoulders in the i-th frame are obtained (Figure 6(c)), and then the region of interest of the head and shoulders is determined (Figure 6(d)). An optional method for determining the region of interest of the head and shoulders is:

[0065] Size_RoIOmega=w2*Omega_Size formula (4)

[0066] Omega_Size represents the size of the head and shoulder estimation region estimated according to motion estimation, and the region concentrically enlarged by w2 (eg, w2=1.8) times is used as the region of interest of the head and shoulder.

[0067] Step S102-5: Combine the region of interest of the face and the region of interest of the head and shoulders to obtain the final region of interest of the face-head and shoulders pair.

[0068] In this step S102-5, the minimum rectangular area including the area of ​​interest of the face and the area of ​​interest of the head and shoulders can be used as the final area of ​​interest for face-head and shoulder pair detection and tracking. Taking the area of ​​interest of the face-head and shoulder pair including the coordinate axis shown in Figure 6(e) as an example, the final area of ​​interest of the face-head and shoulder pair is a rectangle, and the positions of the four sides of the rectangle on the coordinate axis are: Left = MIN(Left(RoIface), Left(RoIOmega)); Top = MIN(Top(RoIface), Top(RoIOmega)); Right = MAX(Right(RoIface), Right(RoIOmega)); Bottom = MAX(Bottom(RoIface), Bottom(RoIOmega)).

[0069] Note that in the first embodiment, the region of interest of the face and the region of interest of the head and shoulders are used as the basis to determine the region of interest of the face-head and shoulders pair, that is, the joint region of interest. However, the present disclosure is not limited to other ways of determining the joint region of interest. For example, based only on the region of interest of the face, the region of interest of the face is directly used as the joint region of interest, or the region of interest of the face is concentrically expanded to a certain range as the joint region of interest; for another example, based only on the region of interest of the head and shoulders, the region of interest of the head and shoulders is directly used as the joint region of interest, or the region of interest of the head and shoulders is concentrically expanded to a certain range as the joint region of interest. The present disclosure does not limit the specific method of determining the joint region of interest, and different algorithms can be adopted in different business scenarios based on empirical values ​​or experimental values.

[0070] Step S103: Detecting face-head-shoulder pairs in the face-head-shoulder pair region of interest of the i-th frame.

[0071] In this step S103, a partial image is cropped from the i-th frame, and in the cropped partial image, face detection and head and shoulder detection are performed therefrom using an AdaBoost detector according to the region of interest of the face-head and shoulder pair determined in step S102, and a face detection frame and a head and shoulder detection frame are determined. In addition to using a detector to detect the face and the head and shoulders, this embodiment is not limited to other detection methods, for example, using a pre-set face template and head and shoulder template, and detecting the face and the head and shoulders through a template matching method.

[0072] Taking the region of interest of the face-head-shoulder pair determined in Figure 6(e) as an example, Figure 7(a) to Figure 7(c) In the schematic diagram of the detection step shown, first, a local image including a human body is cropped from the i-th frame of the video image, and then, in the region of interest of the determined face-head-shoulders pair, a detector is used to determine a face detection frame and a head-shoulders detection frame as the detected face-head-shoulders pair.

[0073] Step S104: Associating the detected face-head-shoulder pair with the trajectory.

[0074] In the method of the first embodiment, if there is only one track in the i-th frame (i.e., only one tracked person) to be associated with the detected face-head-shoulder pair, the detected face-head-shoulder pair can be associated with the track in step S104; if there are multiple tracks in the i-th frame (i.e., multiple tracked persons) to be associated with the detected face-head-shoulder pair, then in step S103, a face-head-shoulder pair is detected for each track, and then each detected face-head-shoulder pair is associated with each track.

[0075] Figure 8A flowchart of the associated steps of step S104 is shown, and the specific description is as follows.

[0076] Step S104-1: Determine the correlation between the detected face in the face-head-shoulder pair and each track.

[0077] Here, an optional method for calculating the association between the face and each track is:

[0078] Sface=w3*distanceRatio_face+w4*sizeRatio_face+w5*colorSimilarity_face formula (5)

[0079] Wherein, distanceRatio_face represents the ratio of the difference between the face detected in step S101 and the face prediction result of the trajectory to be associated to the side length of the face frame of the face template in the trajectory to be associated, the difference refers to the distance between the center point of the detected face frame and the center point of the face estimation area estimated according to the face tracking result stored in the trajectory, that is, the distance between the center point of the face detection frame in FIG7(c) and the center point of the face estimation area in FIG6(b); sizeRatio_face=MIN(detected face size, face size of trajectory) / MAX(detected face size, facesize of trajectory), which represents the ratio of the smaller value of the side length of the face detection frame in the i-th frame and the side length of the face estimation frame estimated according to the face tracking result stored in the trajectory to be associated to the larger value of the two; colorSimilarity_face represents the similarity between the face of the face detection frame in the i-th frame and the color of the face template stored in the trajectory to be associated. w3, w4 and w5 are constants, for example, w3=0.5, w4=0.5, w5=0.8.

[0080] Step S104-2: Determine the correlation between the head and shoulders in the detected face-head and shoulders pair and each track.

[0081] Similar to step S104-1, this step S104-2 also determines the correlation between the detected head and shoulders and each track. Here, an optional method for calculating the correlation between the detected head and shoulders and each track is:

[0082] SOmega=w3*distanceRatio_Omega+w4*sizeRatio_Omega+w5*colorSimilarity_Omega formula (6)

[0083] The parameters in the above formula have similar meanings to the parameters in the formula for calculating the degree of association between the detected face and each track in step S104 - 1 , and will not be described in detail here.

[0084] Step S104-3: Determine the degree of association between the face-head-shoulder pair and each track according to the degree of association between the face and each track and the degree of association between the head-shoulder pair and each track.

[0085] Here, an optional method for calculating the correlation between the face-head-shoulder pair and the trajectory is:

[0086] Score_trajectory_pair=WOmega*SOmega+Wface*Sface formula (7)

[0087] Wherein, WOmega and Wface represent the weight values ​​of the correlation between the head and shoulders and the track and the correlation between the face and the track calculated according to formula (6) and formula (5), respectively, for example, WOmega = 0.5, Wface = 0.5. Of course, the method of the first embodiment is not limited thereto. In the case where the visible range of the face may change, WOmega may be set to a weight value greater than Wface; or, in the case where the shoulder may be blocked (for example, in the case of dense crowds), WOmega may be set to a weight value less than Wface.

[0088] The association process of step S104 is described below by taking an example. Assume that there are three tracks in the i-th frame, namely track 1, track 2 and track 3. According to the method described in step S102 and step S103, face and head-shoulder estimation are performed based on the face tracking results stored in track 1, track 2 and track 3, respectively, and then the face estimation area and the head-shoulder estimation area are determined, and the area of ​​interest of the face-head-shoulder pair is determined according to the face estimation area and the head-shoulder estimation area, and then three face-head-shoulder pairs are detected from the area of ​​interest of the face-head-shoulder pair. In step S104, the association degree between the face detected in face-head-shoulder pair A and track 1, track 2 and track 3 is calculated respectively, and the association degree between the head and shoulders detected in face-head-shoulder pair A and track 1, track 2 and track 3 is calculated, and then the association degree between face-head-shoulder pair A and track 1, track 2 and track 3 is calculated by weighted summation method. According to the above method, the correlation between the face-head-shoulder pair B and track 1, track 2 and track 3 and the correlation between the face-head-shoulder pair C and track 1, track 2 and track 3 can be calculated respectively.

[0089] In the actual object tracking process, the number of tracks may be more, and a data pool can be created to store the calculated multiple associations. Table 1 shows the data pool created by taking the above three detected face-head-shoulder pairs and three tracks as an example.

[0090]

[0091]

[0092] Table 1

[0093] Step S105: Determine whether the track is successfully associated with the detected face-head-shoulder pair, if so, execute step S106; otherwise, the object tracking fails.

[0094] In this step S105, it is determined whether each track is successfully associated with a face-head-shoulder pair detected in the current frame. For the track that is successfully associated with the corresponding face-head-shoulder pair, the subsequent step S106 is executed; for the track that is not successfully associated with the face-head-shoulder pair, it indicates that the tracking of the track has failed.

[0095] This step S105 can determine the associated face-head-shoulder pairs and trajectories according to the data pool shown in Table 1, which is specifically described as:

[0096] a) Referring to Table 1, the face-head-shoulder pair with the highest correlation is associated with the corresponding track, that is, face-head-shoulder pair B is associated with track 1.

[0097] b) Remove the correlation between track 1 and other face-head-shoulder pairs, and remove the correlation between face-head-shoulder pair B and other tracks to avoid repeated correlation. At this time, the data in Table 1 is updated as shown in Table 2.

[0098] c) Repeat step a) and step b) in the updated table 2 until the association of face-head-shoulder pairs is performed for each pair.

[0099] Face-Head and Shoulders A Face-Head-Shoulders B Face-Head-Shoulders C Track 1 - 0.9 - Track 2 0.8 - 0.4 Track 3 0.2 - 0.7

[0100] Table 2

[0101] Step S106: using the detection result of the successfully associated face-head-shoulder pair to update the information in the associated track, and using the detection result of the successfully associated face-head-shoulder pair as the tracking result.

[0102] In this step S106, when the face-head-shoulder pair and the trajectory are successfully associated, for example, face-head-shoulder pair A is associated with trajectory 1 with reference to Table 1, the information of face-head-shoulder pair A can be used to update the information in trajectory 1. Specifically, the information in the face detection frame (feature information of the face) and the information in the head and shoulders detection frame (feature information of the head and shoulders) of the face-head-shoulder pair A in the i-th frame can be updated to the face template and head and shoulders template in trajectory 1, and the face tracking result (position and size of the face) and head and shoulders tracking information (position and size of the head and shoulders) of the face-head and shoulder pair A can replace the face tracking result and head and shoulders tracking result of the iM-th frame.

[0103] The effects of the first exemplary embodiment of the present disclosure will be described below in comparison with the human body tracking technology disclosed in US Pat. No. 8,929,598 B2.

[0104] The human body tracking technology disclosed in US Pat. No. 8,929,598 B2 performs head and shoulder tracking after face tracking fails. This will result in the following problems.

[0105] Question 1: Assume that the reason for the failure of face tracking is that the visible state of the face has changed. Assume that in the nth frame, the front of the face is fully visible, and the face detector can detect the face normally and track the person based on the detected face. In the n+10th frame, the person turns to the left, and the face detector can only detect the right half of the face; in the n+20th frame, the person turns to the back, and the face detector cannot detect the face at all; in the n+30th frame, the person turns to the right, and the face detector can only detect the left half of the face. If the tracking based on face detection fails in the n+10th frame or the n+20th frame, on the one hand, if the region of interest of the head and shoulders is still estimated based on the face detected in the n+10th frame or the n+20th frame, the size of the estimated region of interest of the head and shoulders is too small or offset because the detected face is incomplete or no face is detected, which makes the tracking based on head and shoulder detection fail. On the other hand, if the head and shoulders region of interest is not estimated based on the face detected in the n+10th frame or the n+20th frame, but the head and shoulders motion information of the previous frames (for example, the previous 5 frames) is used to estimate the head and shoulders region of interest in, for example, the n+10th frame, then if face detection and tracking are successfully performed in the first 5 frames (that is, head and shoulders detection and tracking are not used), the tracking results of the head and shoulders in the first 5 frames are not updated. If the head and shoulders region of interest in the n+10th frame is estimated using the head and shoulders motion information in the first 5 frames that has not been updated in real time, the estimation result of the head and shoulders region of interest is inaccurate.

[0106] In the solution of the first embodiment, the face and head and shoulders are jointly detected to determine the area of ​​interest, and tracking is performed based on the joint detection result. Figure 9(a) to Figure 9(d) For example, if the person turns to the left (as shown in Figure 9(b)) or turns to the back (as shown in Figure 9(c)), the face detector cannot accurately detect the face. At this time, the correlation between the detected face and the trajectory is very low, or even 0. However, since the joint detection of the face-head-shoulder pair is performed in each frame, even if the face cannot be accurately detected, tracking can still be achieved based on the detection results of the face-head-shoulder pair because the head and shoulders can be accurately detected. Figure 9(a) to Figure 9(d)The description is made using an example of a person turning around. If the situation in which a person's face is blocked by a mask as shown in FIG. 10( a) and FIG. 10( b) is taken as an example, based on the solution of the first embodiment, when the face is blocked, face detection and head and shoulders detection will not be performed only in the region of interest of the blocked face. Instead, face detection and head and shoulders detection will be performed in the region of interest of the face-head and shoulders pair after the region of interest of the face and the region of interest of the head and shoulders are merged, thereby avoiding the problem of tracking failure.

[0107] Question 2: In the technology of US Pat. No. 8,929,598 B2, when multiple people are moving crosswise, such as Figure 11(a) to Figure 11(c) In the case shown, when one person is walking and the other is stationary, when the two people pass each other, it is easy to make mistakes when associating the detected faces with the trajectories because the apparent features of the faces of the two people (e.g., skin texture, color) are similar.

[0108] In the solution of the first embodiment, since the face-head-shoulder pair is jointly associated with the trajectory, the head and shoulders have more discriminative features (such as clothes, etc.), so different heads and shoulders can be distinguished more accurately, reducing the possibility of errors when associating the face-head-shoulder pair with the trajectory.

[0109] <Second Exemplary Embodiment>

[0110] The second exemplary embodiment of the present disclosure describes an object tracking device under the same inventive concept as the first exemplary embodiment. Fig.12 As shown, the object tracking device includes an area of ​​interest determination unit 1001 , a detection unit 1002 , an association unit 1003 and an update unit 1004 .

[0111] The region of interest determination unit 1001 determines the region of interest of the face-body part pair in the current frame based on the face tracking results stored in the created trajectory and the tracking results of the body parts having a certain positional relationship with the face. The detection unit 1002 detects the face and the body parts in the determined region of interest of the face-body part pair to obtain the detected face-body part pair. The association unit 1003 associates the detected face-body part pair with the trajectory. When the association is successful, the update unit 1004 updates the trajectory using the detected face-body part pair, thereby realizing the tracking process of the object.

[0112] Preferably, the object tracking device also includes a trajectory creation unit 1000, which initially creates a trajectory based on the detection results of the face and body parts, and the trajectory includes: an identification number for uniquely identifying the trajectory; a face template including the detection results of the face and a body part template including the detection results of the body parts; when performing object tracking for each image frame, the update unit 1004 updates the face tracking results and body part tracking results when the tracking is successful to the trajectory.

[0113] Preferably, the region of interest determination unit 1001 estimates the face estimation region and the body part estimation region in the current frame based on motion estimation according to the face tracking results and the body part tracking results stored in the trajectory, determines the region of interest of the face according to the face estimation region, and determines the region of interest of the body parts according to the body part estimation region, and combines the region of interest of the face and the region of interest of the body parts to obtain the region of interest of the face-body part pair.

[0114] Preferably, the association unit 1003 calculates, for each detected face-body part pair, the degree of association between the face in the face-body part pair and each trajectory, and calculates the degree of association between the body part in the face-body part pair and each trajectory, and determines the degree of association between each face-body part pair and each trajectory based on the calculated degree of association between the face and each trajectory and the calculated degree of association between the body part and each trajectory; and repeats the following process until all determined degrees of association are processed: associating the face-body part pair corresponding to the maximum degree of association with the trajectory, and removing the degree of association between the associated face-body part pair and other trajectories and removing the degree of association between the associated trajectory and other face-body part pairs.

[0115] Preferably, the association unit 1003 calculates the association between the face and the trajectory based on the following information: the distance between the face in the face-body part pair detected in the current frame and the face in the current frame estimated based on the face tracking result stored in the trajectory, the size difference between the detection frame of the face in the detected face-body part pair and the estimated frame of the face in the current frame, and the similarity between the color of the face in the detected face-body part pair and the color of the face template of the current trajectory. The association unit 1003 calculates the association between the body part and the trajectory based on the following information: the distance between the body part in the face-body part pair detected in the current frame and the body part in the current frame estimated based on the body part tracking result stored in the trajectory, the size difference between the detection frame of the body part in the detected face-body part pair and the estimated frame of the body part in the current frame, and the similarity between the color of the body part in the detected face-body part pair and the color of the body part template of the current trajectory.

[0116] Other embodiments

[0117] The embodiments of the present disclosure may also be implemented by a computer of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-mentioned embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-mentioned embodiments, and by a method executed by a computer of a system or device, by, for example, reading and executing computer-readable instructions from a storage medium to perform the functions of one or more of the above-mentioned embodiments and / or controlling one or more circuits to perform the functions of one or more of the above-mentioned embodiments. The computer may include one or more processors (e.g., central processing units (CPUs), microprocessing units (MPUs)), and may include a network of independent computers or independent processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer from, for example, a network or a storage medium. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD) (registered trademark)), a flash memory device, a memory card, and the like.

[0118] The embodiments of the present disclosure may also be implemented by the following method, that is, software (program) that performs the functions of the above embodiments is provided to a system or device through a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0119] While the present disclosure has been described with reference to exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims should be accorded the broadest interpretation so as to encompass all modifications, equivalent structures and functions.

Claims

1. An object tracking method for an image frame sequence, wherein the image frame sequence comprises a plurality of image frames, each image frame comprising at least one object; The object tracking method comprises: Determine the region of interest of the face-body part pair in the current frame based on face motion estimation and head-shoulder motion estimation according to the face tracking results and body part pair tracking results stored in the created trajectory; Detecting the face and body parts within the determined region of interest of the face-body part pair to obtain a detected face-body part pair; Associating the detected face-body part pair with the track, and when the association is successful, updating the track with the detected face-body part pair, The region of interest of the face-body part pair in the current frame is determined, specifically including: According to the face tracking results and body part tracking results stored in the trajectory, the face estimation area and the body part estimation area in the current frame are estimated based on the face motion estimation and the head and shoulder motion estimation; Determine a region of interest of the face according to the estimated face region, and determine a region of interest of the body part according to the estimated body part region; The region of interest of the face and the region of interest of the body part are combined to obtain the region of interest of the face-body part pair.

2. The object tracking method according to claim 1, wherein: The method further comprises: Initially, a track is created based on the detection results of the face and body parts, wherein the track includes: an identification number for uniquely identifying the track; a face template including the detection results of the face and a body part template including the detection results of the body parts; When object tracking is performed for each image frame, the face tracking result and the body part tracking result when tracking is successful are updated into the track.

3. The object tracking method according to claim 1, wherein: Associating the detected face-body part pairs with the trajectories, specifically comprising: For each detected face-body part pair, calculating the correlation between the face in the face-body part pair and each track, and calculating the correlation between the body part in the face-body part pair and each track; Determining the degree of association between each face-body part pair and each trajectory according to the calculated degree of association between the face and each trajectory and the calculated degree of association between the body part and each trajectory; Repeat the following process until all determined associations have been processed: Associating the face-body part pair corresponding to the maximum association with the trajectory; Remove the associations of the associated face-body part pairs with other tracks and remove the associations of the associated tracks with other face-body part pairs.

4. The object tracking method according to claim 3, wherein: The association between the face and the track is calculated based on the following information: the distance between a face in a face-body part pair detected in the current frame and a face in the current frame estimated based on face tracking results stored in the trajectory, the size difference between a detection box of the face in the detected face-body part pair and an estimated box of the face in the current frame, and the similarity between the color of the face in the detected face-body part pair and the color of a face template in the current trajectory; The association between body parts and trajectories is calculated based on the following information: The distance between the body part in the face-body part pair detected in the current frame and the body part in the current frame estimated based on the body part tracking results stored in the trajectory, the size difference between the detection box of the body part in the detected face-body part pair and the estimated box of the body part in the current frame, and the similarity between the color of the body part in the detected face-body part pair and the color of the body part template of the current trajectory.

5. An object tracking device for an image frame sequence, wherein the image frame sequence comprises a plurality of image frames, each image frame comprising at least one object; The object tracking device comprises: an area of ​​interest determination unit, which is configured to determine an area of ​​interest of a face-body part pair in a current frame based on face motion estimation and head-shoulder motion estimation according to the face tracking results and body part pair tracking results stored in the created trajectory; A detection unit, which is configured to detect a face and a body part within a region of interest of a determined face-body part pair to obtain a detected face-body part pair; an associating unit configured to associate the detected face-body part pairs with the trajectories; an updating unit configured to update the trajectory using the detected face-body part pair when the association is successful, The region of interest determination unit estimates the face estimation region and the body part estimation region in the current frame based on the face tracking results and the body part tracking results stored in the trajectory, based on the face motion estimation and the head and shoulder motion estimation, determines the region of interest of the face based on the face estimation region, and determines the region of interest of the body part based on the body part estimation region; and The region of interest of the face and the region of interest of the body part are combined to obtain the region of interest of the face-body part pair.

6. The object tracking device according to claim 5, wherein: The device also includes: A trajectory creation unit is configured to initially create a trajectory based on the detection results of the face and body parts, wherein the trajectory includes: an identification number for uniquely identifying the trajectory; a face template including the detection results of the face and a body part template including the detection results of the body parts; When object tracking is performed for each image frame, the updating unit updates the face tracking result and the body part tracking result when tracking is successful into the track.

7. The object tracking device according to claim 5, wherein: The association unit calculates, for each detected face-body part pair, a degree of association between the face in the face-body part pair and each track, and calculates a degree of association between the body part in the face-body part pair and each track; determines a degree of association between each face-body part pair and each track based on the calculated degrees of association between the face and each track and the calculated degrees of association between the body part and each track; and Repeat the following process until all determined associations have been processed: Associating the face-body part pair corresponding to the maximum association with the trajectory; Remove the associations of the associated face-body part pairs with other tracks and remove the associations of the associated tracks with other face-body part pairs.

8. The object tracking device according to claim 7, wherein: The association unit calculates the association degree between the face and the trajectory according to the following information: The distance between the face in the face-body part pair detected in the current frame and the face in the current frame estimated based on the face tracking results stored in the trajectory, the size difference between the detection frame of the face in the detected face-body part pair and the estimated frame of the face in the current frame, and the similarity between the color of the face in the detected face-body part pair and the color of the face template of the current trajectory; and The association between body parts and trajectories is calculated based on the following information: The distance between the body part in the face-body part pair detected in the current frame and the body part in the current frame estimated based on the body part tracking results stored in the trajectory, the size difference between the detection box of the body part in the detected face-body part pair and the estimated box of the body part in the current frame, and the similarity between the color of the body part in the detected face-body part pair and the color of the body part template of the current trajectory.

9. A non-transitory computer-readable storage medium storing instructions, which, when executed by a computer, cause the computer to perform the object tracking method for an image frame sequence according to claim 1.

Images