Target tracking method and target tracking device

Abstract

This invention provides a target tracking method and a target tracking device. The method includes: acquiring a face region from an image frame to determine a human body region; extracting human body features from the human body region and determining whether multiple historical target trajectories match the human body features; when one of the multiple historical target trajectories matches the human body features, updating the age of the human body features to a preset value; when none of the multiple historical target trajectories matches the human body features, adding a new target trajectory corresponding to the human body features to the multiple historical target trajectories. This results in better tracking performance.

Description

Technical Field

[0001] This invention relates to an image processing technique, and more particularly to a target tracking method and a target tracking device. Background Technology

[0002] The main task of multi-object tracking (MOT) is to detect moving objects in an image sequence after it has been acquired. Then, MOT assigns an ID to each object and tries to ensure that the ID of the same object does not change across different frames.

[0003] While existing technologies include MOT algorithms such as optical flow and kernel correlation filter (KCF) which offer faster tracking speeds, the tracking performance of these technologies is generally poor.

[0004] In addition, with the development of deep learning technology, there are now several deep learning algorithms proposed for MOT, but most of these methods are slow in tracking speed and therefore cannot be applied in industry. Summary of the Invention

[0005] In view of this, the present invention provides a target tracking method and a target tracking device, which can be used to solve the above-mentioned technical problems.

[0006] This invention provides a target tracking method, comprising: acquiring a first face region of an image frame and determining a first human body region accordingly; extracting a first human body feature from the first human body region and determining whether multiple historical target trajectories match the first human body feature; when one of the multiple historical target trajectories matches the first human body feature, updating the first age of the first human body feature to a preset value; when none of the multiple historical target trajectories matches the first human body feature, adding a target trajectory to the multiple historical target trajectories corresponding to the first human body feature.

[0007] This invention provides a target tracking device, including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute: acquiring a first face region of an image frame and determining a first human body region accordingly; extracting a first human body feature from the first human body region and determining whether multiple historical target trajectories match the first human body feature; when one of the multiple historical target trajectories matches the first human body feature, updating the first age of the first human body feature to a preset value; when none of the multiple historical target trajectories matches the first human body feature, adding a new target trajectory corresponding to the first human body feature to the multiple historical target trajectories.

[0008] This invention provides a target tracking device, including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute: a first execution thread identifies a first face region in an image frame and determines a first human body region based on the first face region; a second execution thread extracts a first human body feature from the first human body region; a third execution thread determines whether multiple historical target trajectories match the first human body feature; when one of the multiple historical target trajectories matches the first human body feature, the first age of the first human body feature is updated to a preset value; and when none of the multiple historical target trajectories matches the first human body feature, a new target trajectory corresponding to the first human body feature is added to the multiple historical target trajectories. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a target tracking device illustrated according to an embodiment of the present invention.

[0010] Figure 2 This is a flowchart illustrating a target tracking method based on an embodiment of the present invention.

[0011] Figure 3 This is an application scenario diagram illustrated according to an embodiment of the present invention.

[0012] Figure 4 This is a schematic diagram illustrating how a segmentation region is determined in a training image based on a segmentation template, according to an embodiment of the present invention.

[0013] Figure 5 This is a schematic diagram illustrating occlusion processing in a weak occlusion scenario, based on an embodiment of the present invention.

[0014] The symbols in the attached diagram are briefly explained as follows:

[0015] 100: Target tracking device; 102: Storage circuit; 104: Processor; 300, 500: Image frame; 310: Face region; 320: Converted face region; 330, 520: First human body region; 410: Segmentation template; 411: Outer contour; 413: Inner contour; 420: Training image; 430: Segmentation region; 1-7: Converted face region; 531: Overlapping region; 531a: First region; 532: Non-overlapping region; 532a: Second region; DD: Specific distance; D1: First specified distance; D2: Second specified distance; w o Width; S210~S250: Steps. Detailed Implementation

[0016] One embodiment of this application implements multi-target tracking based on deep sorting technology. Deep sorting is a multi-target tracking method based on deep learning, which is widely used in industry due to its high speed, strong model customizability, good performance, and simple implementation. Application areas of deep sorting include face tracking, pedestrian tracking, video analysis, video surveillance, autonomous driving, etc. Generally speaking, the deep sorting described in this embodiment may include the following steps: (1) target detection; (2) feature extraction; (3) state estimation; (4) trajectory processing; (5) matching.

[0017] In the target detection step, a trained deep learning target detection model is used to detect the positions of all targets (such as faces or bodies) in the current image frame. In the feature extraction step, the region of interest (ROI) containing the detected target is cropped based on the detected target position. Then, a trained deep learning feature extraction model is used to extract depth features for each ROI; these depth features are also called deep appearance features.

[0018] In the state estimation step, a Kalman filter can be used to estimate the state information (such as velocity, position, etc.) of the tracked target at the current moment.

[0019] In the trajectory processing step, a trajectory can be recorded for each tracked target, and each trajectory is matched with a corresponding tracker. Each tracker is used to record information such as the target's tracking ID, age, historical depth characteristics, and location.

[0020] For new targets that do not match existing tracks (historical target tracks), the new track generated by this new target needs to be added to the existing tracks, and a corresponding tracker and tracking identifier need to be configured for the new track. In addition, the age of this new track needs to be initialized to a preset value (e.g., 1), and the depth features, location, and other information of this new target need to be saved.

[0021] If a target in an image frame (hereinafter referred to as image frame A) is determined to correspond to a historical target trajectory, the age of this historical target trajectory can be updated to a preset value (e.g., 1). On the other hand, if any target in image frame A is determined not to correspond to a historical target trajectory, the age of this historical target trajectory is updated (incremented by 1). If the age of a historical target trajectory is greater than an upper limit value (e.g., 100), the historical target trajectory will be removed from the set of existing trajectories (historical target trajectories).

[0022] The matching process mainly includes state matching, depth feature matching, and comprehensive matching. In the state matching section, a Kalman filter is used to generate the estimated current position of each historical target trajectory based on multiple historical positions within those trajectories. The Mahalanobis distance between the current position of target B in image frame A and the estimated positions of each historical target trajectory is also estimated. Subsequently, preliminary matching results are obtained through Hungarian matching, and then qualified matches are filtered based on a set distance threshold (e.g., which historical target trajectory might target B match).

[0023] In the deep feature matching step, the cosine distance between the depth features of target B and the depth features of each historical target trajectory can be estimated. Then, preliminary matching results can be obtained through Hungarian matching. Finally, qualified matches are filtered based on a set distance threshold (e.g., which historical target trajectory might target B match).

[0024] In the comprehensive matching step, the results of state matching and deep feature matching can be combined to form the comprehensive matching result. For example, a weighted operation can be performed on the results of state matching and deep feature matching to generate the comprehensive matching result.

[0025] For details related to depth sorting, please refer to the literature “Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP). pp. 3645-3649. IEEE (2017)”, which will not be repeated here.

[0026] Generally speaking, there are two ways to implement deep feature matching through deep sorting technology: using facial features as deep features (hereinafter referred to as method 1) and using human body features as deep features (hereinafter referred to as method 2).

[0027] In Method 1, a pre-trained face detection model is used to detect the location of faces in the image frame, and a face feature extraction model is used to extract the facial features at that location as depth features. In Method 2, a model capable of simultaneously detecting faces and bodies is used to detect the locations of faces and bodies in the image frame, and a pre-trained body feature extraction model is used to extract the corresponding body features as depth features. After obtaining the depth features from Methods 1 and 2 respectively, the relevant state estimation, trajectory processing, and matching steps can be performed. Details of these steps can be found in the previous explanation and will not be repeated here.

[0028] However, in Method 1, because the face occupies a small area in the image frame, its feature information is less obvious. Furthermore, facial features are easily affected by changes in facial angle and movement, making them relatively unstable. Under these circumstances, using facial features as depth features for the aforementioned matching operation will result in a low matching success rate and may easily lead to the loss of the tracking target.

[0029] Furthermore, Method 2 requires the creation of a separate dataset labeled with face and body location information, which consumes significant time and manpower. In other words, training a model capable of simultaneously detecting both faces and bodies is more challenging. Moreover, due to the substantial differences in face and body pose / size among different individuals, simultaneously detecting both faces and bodies necessitates designing deeper and more complex networks / models compared to face detection to achieve better detection results. However, such networks / models generate greater computational demands, leading to slower performance and lower frame rates on devices with the same specifications. A decreased frame rate consequently affects matching performance, and consequently, tracking performance.

[0030] In view of this, the present invention proposes a new target tracking method for application in other embodiments. This new target tracking method has the characteristics of better tracking effect, faster speed and easy implementation, which will be further described below.

[0031] Please refer to Figure 1 This is a schematic diagram of a target tracking device according to an embodiment of the present invention. In different embodiments, the target tracking device 100 may be implemented as various intelligent devices and / or computer devices, but is not limited thereto.

[0032] like Figure 1As shown, the target tracking device 100 may include a storage circuit 102 and a processor 104. The storage circuit 102 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record multiple program codes or modules.

[0033] Processor 104 is coupled to storage circuit 102 and may be a general purpose processor, special purpose processor, conventional processor, digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, controller, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), any other type of integrated circuit, state machine, processor based on advanced reduced instruction set machine (ARM), and the like.

[0034] In an embodiment of the present invention, the processor 104 may access the modules and program code recorded in the storage circuit 102 to implement the target tracking method proposed in the present invention, the details of which are described below.

[0035] Please refer to Figure 2 This is a flowchart illustrating a target tracking method according to an embodiment of the present invention. The method in this embodiment can be derived from... Figure 1 The target tracking device 100 performs the following steps: Figure 1 Component description shown Figure 2 Details of each step. Furthermore, the processor 104 of this invention can also execute these steps separately using different execution threads. Figure 2 One or more steps are used to improve processing efficiency and achieve full decoupling and independence between the steps.

[0036] First, in step S210, the processor 104 (through the first execution thread) acquires an image frame and identifies a first face region within that image frame. In one embodiment, the processor 104 may input the image frame into a face detection model, which can identify one or more face regions in the image frame based on the face detection model. In embodiments of the present invention, the aforementioned face detection model can be any existing model capable of identifying image regions corresponding to faces in an image frame, but is not limited to this.

[0037] Subsequently, in step S220, the processor 104 (through the first execution thread) transforms a first converted face region based on the first face region to determine a first human body region. To make the concept of this invention easier to understand, the following will further elaborate... Figure 3 The content is explained, including Figure 3 This is an application scenario diagram illustrated according to embodiments of the present invention. Figure 3 In the image frame 300, if there is only one face region 310, the processor 104 may regard this face region 310 as the first face region (hereinafter referred to as the first face region 310), but it may not be limited to this.

[0038] Subsequently, the processor 104 can convert the first face region 310 into a converted face region 320, thereby further generating a first human body region 330. In one embodiment, the processor 104 can define a region as the first human body region 330 in the image frame 300 based on the first face region 310, but it is not limited to this.

[0039] In one embodiment, the width and height of the first face region 310 can be w1 and h1 (width-to-height ratio w1:h1), the coordinates of the center point of the first face region 310 in the image frame 300 can be represented as (cx, cy), the coordinates of the upper left corner of the first face region 310 in the image frame 300 can be represented as (l, t), and the coordinates of the lower right corner of the first face region 310 in the image frame 300 can be represented as (r, b). In this case, the width w1 = rl, the height h1 = bt, the coordinates of the center point cx = l + w1 / 2, and the coordinates of the center point cy = t + h1 / 2 can be calculated. In one embodiment, based on the data w1, h1 and the center point (cx, cy) of the first face region 310, a square first transformed face region 320 with a side length s of (w1 + h1) / 2 and a center point of (cx, cy) is transformed. In another embodiment, the processor 104 may also convert a first transformed face region 320 with other side lengths, other center points, or other shapes based on the data of the first face region 310, but the present invention may not be limited to this.

[0040] After converting the first converted face region 320, the processor 104 can determine the first human body region 330 based on the converted face region 320, a preset image aspect ratio w2:h2, and multiple expansion parameters. For example... Figure 3 As shown, the first human body region 330 may include a first transformed face region 320. That is, the first transformed face region 320 may be located in the first human body region 330.

[0041] In one embodiment, the above-mentioned expansion parameters may include m and n (each of which is an integer), which are used to expand the first human body region 330 upward by m pixels, expand it to the left and right by n pixels each, and expand it downward by [(2n+s)×h2 / w2-ms] pixels when the aspect ratio of the first human body region 330 does not match the preset image aspect ratio w2:h2, so as to make the aspect ratio of the first human body region 330 equal to the preset image aspect ratio w2:h2, thereby avoiding distortion.

[0042] Specifically, when the coordinates of the upper left corner of the first human body region 330 in the image frame 300 are (bl, bt), and the coordinates of the lower right corner of the first human body region 330 in the image frame 300 are (br, bb), then the coordinates of the first human body region 330 can be expressed as follows:

[0043] (bl, bt, br, bb) = (cx-s / 2-n, cy-s / 2-m, cx+s / 2+n, cy-s / 2-m+(2n+s)×h2 / w2)

[0044] The coordinates of the first human body region 330 indicate the location of the first human body region 330 and the extent of the region.

[0045] In one embodiment, when the preset image aspect ratio is 1:2, i.e., w2:h2 = 1:2, the coordinates of the first human body region 330 can be expressed as:

[0046] (bl, bt, br, bb) = (cx-3×s / 2, cy-3×s / 2, cy-3×s / 2, cy+3×s / 2+3×s)

[0047] After determining the first human body region 330, the processor 104 can execute step S230 to extract the first human body feature from the first human body region 330 (through the second execution thread) and (through the third execution thread) determine whether there is a first historical target trajectory that matches the first human body feature among multiple historical target trajectories.

[0048] In one embodiment, the processor 104 can input the first human body region 330 into a human body feature extraction model trained through a training process, and based on this human body feature extraction model, the processor 104 can extract first human body features from the first human body region 330.

[0049] In one embodiment, the aforementioned human feature extraction model is, for example, a person re-identification (ReID) model. In embodiments of the present invention, to improve the performance of the aforementioned human feature extraction model, the present invention further proposes a mechanism for enhancing the data in the training dataset, the details of which will be described below.

[0050] In one embodiment, the training of the aforementioned human feature extraction model is achieved by obtaining a training image dataset through another processor or server (not shown, not processor 104). This training image dataset may include multiple training images, each with the same preset size. In one embodiment, the training image dataset may be, for example, Market1501 or Mars, and the preset size of each training image may be, for example, 128x256, but is not limited to this.

[0051] Subsequently, the processor or server can determine a segmentation template based on the aforementioned preset dimensions. To further facilitate understanding of the concept of this invention, the following supplementary information is provided. Figure 4 For further explanation, please refer to [link / reference]. Figure 4 This is a schematic diagram illustrating, according to an embodiment of the present invention, how a segmentation region is determined in a training image based on a segmentation template.

[0052] exist Figure 4 In this context, assuming the preset size of each training image in the considered training image dataset is w×h (its aspect ratio w:h is equal to the preset image aspect ratio w2:h2), the processor or server can determine the segmentation template 410 accordingly. For example... Figure 4 As shown, the segmentation template 410 may include an outer contour 411 and an inner contour 413 located within the outer contour 411. The preset size of the outer contour 411 is w×h, while the size of the inner contour 413 is another preset size in proportion, such as (2w / 3)×(2h / 3).

[0053] In one embodiment, the relative positions between the outer contour 411 and the inner contour 413 can be as follows: Figure 4 The examples provided are merely illustrative and not intended to limit the possible implementations of the invention. In other embodiments, the designer may adjust the relative positions between the outer contour 411 and the inner contour 413 as needed, and is not limited to these embodiments. Figure 4 The state shown is as described.

[0054] Subsequently, the processor or server can randomly segment each training image in the training image dataset based on the segmentation template 410, and train the aforementioned human feature extraction model based on each segmented training image. In one embodiment, each training image is randomly segmented in an equal ratio (i.e., aspect ratio w2:h2) between the outer contour 411 and the inner contour 413 of the segmentation template 410. The size of each segmented training image can be randomly distributed between the outer contour 411 and the inner contour 413, but the aspect ratio of each segmented training image must remain w2:h2.

[0055] exist Figure 4 In this context, assuming training image 420 is one of the aforementioned training images, the processor or server can determine the segmentation region 430 in training image 420 based on segmentation template 410. For example... Figure 4 As shown, the segmentation region 430 in the training image 420 is located between the outer contour 411 and the inner contour 413 of the segmentation template 410. That is, the segmentation region 430 needs to cover the range of the corresponding inner contour 413, but not exceed the range of the corresponding outer contour 411.

[0056] In one embodiment, the positions of the segmented regions in each training image may be different, but the aspect ratio of the segmented regions in each training image needs to be maintained at w2:h2.

[0057] After determining the corresponding segmentation regions in each training image, the processor or server can train the human feature extraction model based on the segmentation regions in each training image. Thus, after completing the training of the human feature extraction model, the processor 104 can use this human feature extraction model to identify a first human region 330 with an aspect ratio equal to w2:h2 from image frame 300, and extract the first human feature.

[0058] In one embodiment, the image frame 300 further includes a second face region. The processor 104 can generate a second transformed face region corresponding to the second face region based on the method of generating the first transformed face region 320, and determine the second human body region corresponding to the second face region based on the method of determining the first human body region 330. If the image frame 300 includes multiple targets (human bodies), it is necessary to determine whether the first human body region 330 is occluded by other targets before extracting the first human body feature from the first human body region 330. Specifically, this is determined by judging whether the first human body region 330 includes other targets, such as the second transformed face region, to determine whether the first human body region 330 is occluded.

[0059] exist Figure 3 In this scenario, since the first human body region 330 does not include other targets, such as the second transformed face region, the processor 104 can successfully and accurately extract the first human body feature from the first human body region 330. However, in other embodiments, when the first human body region 330 includes other targets, such as the second transformed face region, it means that the first human body region 330 may be occluded by other targets, thus preventing the processor 104 from successfully and accurately extracting the first human body feature. In this case, the present invention also proposes a corresponding occlusion processing mechanism to avoid affecting the extraction of the first human body feature due to the above-mentioned occlusion situation. Related details will be provided later. Figure 5 Further explanation is needed.

[0060] Back Figure 3The first human body region 330 shown does not include other targets. After extracting the first human body feature from the first human body region 330, the processor 104 can then determine whether there is a first historical target trajectory matching the first human body feature among the plurality of historical target trajectories. In this embodiment, the plurality of historical target trajectories are, for example, trajectories corresponding to tracked human bodies, and each historical target trajectory may also have a corresponding tracker. In one embodiment, each tracker may record information such as the identifier (i.e., tracking identifier), age, historical depth features of the human body, location, and human body region (hereinafter referred to as historical human body region) of the corresponding human body. Related concepts can be referred to the description in the previous embodiments, and will not be repeated here.

[0061] In this embodiment, assuming that the target corresponding to each historical target trajectory is a human body, the historical depth features of the target corresponding to each historical target trajectory can be called historical human body features, but are not limited to this. In this case, the processor 104 can generate an estimated position of each historical target trajectory based on multiple historical positions of each historical target trajectory. In one embodiment, the processor 104 can use a Kalman filter to generate the estimated position of each historical target trajectory based on the aforementioned historical positions of each historical target trajectory.

[0062] Subsequently, the processor 104 can determine the first matching result based on the first distance between the position of the first human body region 330 and the estimated positions of each historical target trajectory. Related details can be found in the previously mentioned state matching section and will not be repeated here.

[0063] Additionally, the processor 104 can estimate a second distance between the first human body feature and the historical human body features of each historical target trajectory, and determine the second matching result accordingly. Details can be found in the previously mentioned section on deep feature matching, and will not be repeated here.

[0064] The processor 104 can then generate a combined matching result based on the first and second matching results. Details regarding this can be found in the previously mentioned section on combined matching, and will not be repeated here.

[0065] After obtaining the comprehensive matching result, the processor 104 can determine whether the comprehensive matching result indicates that the first human feature corresponds to one of the aforementioned historical target trajectories. In response to determining that the comprehensive matching result indicates that the first human feature corresponds to one of the aforementioned historical target trajectories, the processor 104 can determine that one of the aforementioned historical target trajectories is a first historical target trajectory matched with the first human feature, and can accordingly (through the third execution thread) execute step S240. Furthermore, in response to determining that the comprehensive matching result indicates that the first human feature does not correspond to any of the aforementioned historical target trajectories, the processor 104 can determine that there is no historical target trajectory matched with the first human feature, and can accordingly (through the third execution thread) execute step S250.

[0066] In step S240, the processor 104 can update the first age of the first historical target trajectory to a preset value (e.g., 1). Additionally, in step S250, the processor 104 can add a reference target trajectory corresponding to the first human feature to the plurality of historical target trajectories, set the reference age of the reference target trajectory to a preset value (e.g., 1), and configure a corresponding tracker for the reference target trajectory. Details of steps S240 and S250 can be found in the previously mentioned trajectory processing section and will not be repeated here.

[0067] As previously mentioned, in other embodiments, when other targets, such as a second transformed face region, exist in the first human body region 330, the processor 104 can further determine whether the occlusion relationship is indeed established, in order to perform corresponding occlusion processing. In one embodiment, in response to determining that other targets, such as a second transformed face region, exist in the first human body region 330, the processor 104 can further determine whether the other target is located below the first transformed face region 320 and whether the side length of the other target (e.g., the second transformed face region) is greater than the side length of the first transformed face region 320, to determine whether the other target actually occludes the first human body region 330. When the other target is located below the first transformed face region 320 and the side length of the other target (e.g., the second transformed face region) is greater than the side length of the first transformed face region 320, it means that the other target is closer to the image acquisition device. Combining the conclusion that the first human body region 330 includes the other target, it can be determined that the other target actually occludes the first human body region 330, the occlusion relationship is indeed established, and the processor 104 performs corresponding occlusion processing accordingly.

[0068] Specifically, the processor 104 obtains a straight-line distance (hereinafter referred to as a specific distance) between the first transformed face region 320 and the other target (e.g., the second transformed face region), and determines whether this specific distance is greater than a threshold value. If the specific distance is greater than the threshold value, it means that the second transformed face region is far from the first transformed face region (which can be referred to as a weak occlusion scenario). Accordingly, the processor 104 can adjust the first human body region 330 by occluding a first region within the second transformed face region. The threshold value is, for example, 1.5 times the side length s of the first transformed face region 320, but is not limited to this. In other embodiments, the designer can select the desired threshold value as needed.

[0069] Please refer to Figure 5 This is a schematic diagram illustrating occlusion processing in a weak occlusion scenario, according to an embodiment of the present invention. Figure 5In this embodiment, it is assumed that the processor 104 has converted multiple transformed face regions 1 to 7 (each of which is a square) in the image frame 500 according to the teachings of the previous embodiment. For ease of explanation, transformed face region 1 will be understood as the first transformed face region under consideration. In this case, the processor 104 may determine the corresponding first human body region 520 for transformed face region 1 according to the previous teachings.

[0070] exist Figure 5 In this scenario, it can be seen that the transformed face region 2 is partially included in the first human body region 520. The transformed face region 2 is located below the transformed face region 1, and the side length of the transformed face region 2 is greater than the side length of the transformed face region 1. Therefore, it can be determined that the occlusion relationship between the transformed face region 2 (other target) and the first human body region 520 is valid. Next, the processor 104 can obtain a specific distance DD between the transformed face region 2 and the transformed face region 1, and determine whether the specific distance DD is greater than a threshold value.

[0071] exist Figure 5 If the processor 104 determines that a specific distance DD is greater than the threshold value, the processor 104 can adjust the first human body region 520 by converting the region 531 that overlaps with the face region 1 and the first human body region 520.

[0072] In one embodiment, the processor 104 may obtain the overlapping region 531 and the non-overlapping region 532 between the converted face region 1 and the first human body region 520.

[0073] In one embodiment, the processor 104 may define a first region 531a based on the overlapping region 531, wherein the first region 531a may include the overlapping region 531. In one embodiment, the processor 104 may vertically expand a square region (the largest rectangle or square that can be framed) within the overlapping region 531 (hereinafter referred to as the first image range) to a preset limit, and use the expanded first image range as the first region 531a.

[0074] For example, the processor 104 may expand the first image range of the overlapping region 531 upward by 0.5 times the side length of the face region 1 (denoted as s1) and downward to the bottom of the first human body region 520 to define the first region 531a. However, this is only an example to make the reader understand the practice of the present invention and is not intended to limit the present invention.

[0075] Subsequently, the processor 104 may define a second region 532a in the non-overlapping region 532 based on the first region 531a, and cover the first region 531a with the image content of the second region 532a, wherein the size of the second region 532a is set to be no smaller than the size of the first region 531a.

[0076] Specifically, such as Figure 5 As shown, the processor 104 can vertically move the first region 531a by a first specified distance D1 and horizontally move it by a second specified distance D2, and use the vertically and horizontally moved first region 531a as the second region 532a. In one embodiment, the first specified distance D1 is, for example, 0.25 times s1, and the second specified distance D2 is, for example, equal to the width w of the overlapping region 531. o However, the present invention is not limited thereto.

[0077] Additionally, the processor 104 can determine the horizontal movement direction of the image range of the first region 531a based on the relative position between the transformed face regions 1 and 2. Figure 5 In this embodiment, since the transformed face region 2 is located to the upper right of the transformed face region 1, the processor 104 can move the image range of the first region 531a upwards by a first specified distance D1, and then move it horizontally to the right by a second specified distance D2 to determine the second region 532a. In other embodiments, when the transformed face region 2 is located to the upper left of the transformed face region 1, the processor 104 can move the image range of the first region 531a upwards by a first specified distance D1, and then move it horizontally to the left by a second specified distance D2 to determine the second region 532a, but the present invention is not limited to this.

[0078] After determining the second region 532a, the processor 104 can overlay the image content of the second region 532a onto the first region 531a to replace the first region 531a. This allows the portion of the first human body region 520 obscured by the transformed face region 2 to no longer affect the extraction operation of the first human body feature of the first human body region 520, but the invention is not limited to this.

[0079] In other embodiments, if a specific distance DD between the transformed face region 1 and the transformed face region 2 is less than a threshold value, this indicates that the transformed face region 1 and the transformed face region 2 are relatively close (which can be referred to as a strong occlusion scenario). Accordingly, the processor 104 can directly perform a state matching operation based on the first human body region 520 to generate a first matching result. Related details can be found in the state matching content described in the previous embodiments, and will not be repeated here.

[0080] In one embodiment, the processor 104 may perform an intersection over union (IoU) matching operation based on the first human body region 520 and the aforementioned historical human body regions to generate a second matching result corresponding to the first human body region 520. Details of the aforementioned IoU matching operation can be found in "Erik Bochinski, Volker Eiselein and Thomas Sikora.: High-Speed ​​Tracking-by-Detection Without Using Image Information. In: IEEE AVSS2017, August 2017, Lecce, ITALY.", and will not be repeated here.

[0081] Subsequently, the processor 104 may determine a specific target trajectory corresponding to the first human body region 520 based on the first matching result and the second matching result. In one embodiment, the processor 104 may first generate a corresponding comprehensive matching result based on the first matching result and the second matching result, and then determine the corresponding target trajectory for the first human body region 520 as the aforementioned specific target trajectory. Related details can be found in the previous comprehensive matching section and will not be repeated here.

[0082] Furthermore, since the present invention can execute different steps through different execution threads, it can further improve processing efficiency and increase the number of image frames that the processor can process per second, i.e., the frame rate. For example, at the t-th time point (t is the time index value), the processor 104 can execute a first operation related to the i-th image frame (i is the image index value) through the first execution thread, such as finding the face region / determining the human body region.

[0083] At time point t+1, processor 104 can perform a second operation related to the i-th image frame through the second execution thread, such as extracting human features. Simultaneously, processor 104 can perform a first operation related to the i+1-th image frame through the first execution thread, such as identifying face regions / determining human body regions.

[0084] At time point t+2, processor 104 can perform a third operation related to the i-th image frame via a third execution thread, such as state estimation, trajectory processing, and matching. Simultaneously, processor 104 can perform a second operation related to the (i+1)-th image frame via a second execution thread, such as extracting human features, and perform a first operation related to the (i+2)-th image frame via a first execution thread, such as identifying face regions / determining human body regions. Therefore, this invention can improve both the matching success rate and the frame rate, thereby enhancing the tracking effect.

[0085] In summary, embodiments of the present invention can determine the corresponding human body region in addition to the face region in the image frame, and then extract human body features from this human body region. Subsequently, the present invention can perform subsequent state estimation, trajectory processing, and matching (e.g., state matching / depth feature matching) based on the extracted human body features.

[0086] Because this invention uses relatively obvious and stable human body features for matching, the success rate of feature matching is higher, resulting in better tracking performance. Furthermore, since this invention determines the corresponding human body region based on the face region, it eliminates the need to train a dedicated human detection model or a model capable of simultaneously detecting both faces and bodies. Therefore, this invention is also simple and fast. In some embodiments, this invention also provides corresponding occlusion processing methods for weak / strong occlusion scenarios, thereby further improving tracking performance.

[0087] Furthermore, since the present invention can execute different steps through different execution threads, it can improve the frame rate and the matching success rate at the same time, thereby improving the tracking effect.

[0088] The above description is only a preferred embodiment of the present invention, but it is not intended to limit the scope of the present invention. Any person skilled in the art can make further improvements and changes on this basis without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims of this application.

Claims

1. A target tracking method, characterized in that, include: Obtain the first face region of the image frame; The first human body region in the image frame is determined based on the first face region. Extract the first human body feature from the first human body region and determine whether the trajectories of multiple historical targets match the first human body feature; When one of the multiple historical target trajectories is a first historical target trajectory that matches the first human body feature, the first age of the first historical target trajectory is updated to a preset value. as well as When none of the multiple historical target trajectories match the first human feature, a new target trajectory corresponding to the first human feature is added to the multiple historical target trajectories. The step of determining the first human body region based on the first face region includes: Convert the first face region into a transformed face region; The first human body region is determined based on the transformed face region; and When the aspect ratio of the first human body region does not match the aspect ratio of the image segmentation, the first human body region is expanded based on the expansion parameter so that the aspect ratio of the first human body region is equal to the aspect ratio of the image segmentation.

2. The target tracking method as described in claim 1, wherein the transformed face region is obtained based on the angular coordinates and center point of the first face region.

3. The target tracking method as described in claim 1, wherein after determining the first human body region, the target tracking method further includes: When the first human body region is occluded by other targets, obtain the specific distance between the other targets and the first transformed human face region; as well as When the specific distance is greater than the threshold, the first human body region is adjusted by occluding the first region of the first human body region with other targets.

4. The target tracking method as described in claim 3, wherein the step of adjusting the first human body region through the first region includes: The first region is panned to select a second region where the first human body region is not obscured, and the image content of the second region is used to cover the first region, wherein the size of the second region is not smaller than the size of the first region.

5. The target tracking method as described in claim 3, wherein when the specific distance is not greater than the threshold value, the target tracking method further includes: A state matching operation is performed based on the first human body region to generate a first matching result; A crosslink set matching operation is performed based on the first human body region and the historical human body region to generate a second matching result corresponding to the first human body region; as well as Based on the first matching result and the second matching result, a specific target trajectory corresponding to the first human body region is determined.

6. The target tracking method as described in claim 1, wherein the step of extracting the first human feature from the first human body region includes: The first human body region is input into the human body feature extraction model, whereby the human body feature extraction model extracts the first human body features based on the first human body region.

7. The target tracking method as described in claim 1 further includes an enhanced face detection model to provide an image segmentation aspect ratio for determining the first human body region, wherein, The steps to enhance the face detection model include: Obtain a training image dataset, wherein the training image dataset includes multiple training images, and each training image has the same preset size; The segmented region is determined based on the preset size, wherein the segmented region includes an outer contour and an inner contour, and the outer contour has the preset size, and the inner contour is a proportionally scaled-down version of the outer contour; and Based on the inner contour and the outer contour, the segmentation region is randomly determined in each training image.

8. A target tracking device, characterized in that, include: Storage circuit, storing program code; The processor, coupled to the storage circuit, accesses the program code to: Obtain the first face region of the image frame; The first human body region in the image frame is determined based on the first face region. Extract the first human body feature from the first human body region and determine whether the trajectories of multiple historical targets match the first human body feature; When one of the multiple historical target trajectories is a first historical target trajectory that matches the first human body feature, the first age of the first historical target trajectory is updated to a preset value. as well as When none of the multiple historical target trajectories match the first human feature, a new target trajectory corresponding to the first human feature is added to the multiple historical target trajectories. The processor is configured to execute: Convert the first face region into a transformed face region; The first human body region is determined based on the transformed face region; and When the aspect ratio of the first human body region does not match the aspect ratio of the image segmentation, the first human body region is expanded based on the expansion parameter so that the aspect ratio of the first human body region is equal to the aspect ratio of the image segmentation.

9. The target tracking device of claim 8, wherein the transformed face region is obtained based on the angular coordinates and center point of the first face region.

10. The target tracking device of claim 8, wherein after determining the first human body region, the processor is further configured to perform: When the first human body region is occluded by other targets, obtain a specific distance between the other targets and the first transformed face region; and When the specific distance is greater than the threshold, the first human body region is adjusted by occluding the first region of the first human body region with other targets.

11. The target tracking device of claim 10, wherein the processor is configured to perform: The first region is panned to select a second region where the first human body region is not obscured, and the image content of the second region is used to cover the first region, wherein the size of the second region is not smaller than the size of the first region.

12. The target tracking device of claim 10, wherein when the specific distance is not greater than the threshold value, the processor is further configured to perform: A state matching operation is performed based on the first human body region to generate a first matching result; A crosslink set matching operation is performed based on the first human body region and the historical human body region to generate a second matching result corresponding to the first human body region; as well as Based on the first matching result and the second matching result, a specific target trajectory corresponding to the first human body region is determined.

13. The target tracking device of claim 8, wherein the processor is configured to perform: The first human body region is input into the human body feature extraction model, whereby the human body feature extraction model extracts the first human body features based on the first human body region.

14. The target tracking apparatus of claim 8, wherein the processor is further configured to execute an enhanced face detection model to provide an image segmentation aspect ratio for determining the first human body region, wherein the processor is configured to execute: Obtain a training image dataset, wherein the training image dataset includes multiple training images, and each training image has the same preset size; The segmented region is determined based on the preset size, wherein the segmented region includes an outer contour and an inner contour, and the outer contour has the preset size, and the inner contour is a proportionally scaled-down version of the outer contour; and Based on the inner contour and the outer contour, the segmentation region is randomly determined in each training image.

15. A target tracking device, characterized in that, include: Storage circuit, storing program code; The processor, coupled to the storage circuit, accesses the program code to: The first execution thread finds the first face region in the image frame and determines the first human body region in the image frame based on the first face region. The first human feature is extracted from the first human body region by the second execution thread; Executed via a third execution thread: Determine whether the trajectories of multiple historical targets match the first human feature; When one of the multiple historical target trajectories is a first historical target trajectory that matches the first human body feature, the first age of the first historical target trajectory is updated to a preset value. as well as When none of the multiple historical target trajectories match the first human feature, a new target trajectory corresponding to the first human feature is added to the multiple historical target trajectories. The processor is configured to execute: Convert the first face region into a transformed face region; The first human body region is determined based on the transformed face region; and When the aspect ratio of the first human body region does not match the aspect ratio of the image segmentation, the first human body region is expanded based on the expansion parameter so that the aspect ratio of the first human body region is equal to the aspect ratio of the image segmentation.

Images