Method and system for tracking footstep positions of figures in video sequence in real time

Abstract

The invention provides a method and a system for tracking the footstep position of a person in a video sequence in real time. The method comprises the following steps: acquiring a current image; inputting the current image into a target detection network to obtain a target detection frame of the current image; respectively inputting the target detection frame into the first semantic segmentation model and the second semantic segmentation model to obtain a foot mask and a figure mask of the current image; determining a first foot centroid position based on the foot mask, and determining a first human centroid position based on the human mask; when the current image is the image after the first frame of image in the acquired multiple frames of images, predicting a second foot mass center position of the current image and a second human mass center position of the current image based on a Kalman filtering algorithm; when the second foot centroid position does not meet the first preset condition and the second human centroid position meets the second preset condition, the second human centroid position is mapped to be a third foot centroid position based on the specific mapping relation, and the third foot centroid position serves as the human foot position.

Description

Technical Field

[0001] This application belongs to the field of image processing, and more specifically, relates to methods and systems for real-time tracking of the position of human footsteps, as well as electronic devices and computationally readable storage media. Background Technology

[0002] Human foot tracking technology is of great significance in many application areas, such as security monitoring, virtual reality, and human-computer interaction. Over the past few decades, various methods have been proposed to achieve human foot tracking, including traditional computer vision techniques.

[0003] Traditional computer vision techniques typically include feature extraction-based methods and motion model-based methods. Feature extraction-based methods usually use techniques such as edge detection and optical flow to extract features from images and then use these features to track the position of a person's feet. Motion model-based methods, on the other hand, track objects based on their motion characteristics, such as speed and acceleration.

[0004] In addition, some methods for human foot tracking based on pose estimation, deep learning trackers and other technologies have been proposed.

[0005] However, these technologies require simple image backgrounds and small movements of the person, and rely on keying techniques to extract the person from the green screen for subsequent work. This greatly limits the application scope of foot tracking technology, and there is a lack of technology to solve the problem of real-time tracking of people's movements in complex environments. Summary of the Invention

[0006] This disclosure provides a method and system for real-time tracking of a person's footsteps, as well as an electronic device and a computationally readable storage medium, to at least solve the problems in the aforementioned related technologies, or may not solve any of the aforementioned problems.

[0007] According to a first aspect of the embodiments of this disclosure, a method for real-time tracking of a person's foot position is provided. The method may include: acquiring a current image; inputting the current image into a target detection network trained on a person's foot dataset to obtain target detection boxes of the current image; inputting the target detection boxes of the current image into a first semantic segmentation model trained on a foot segmentation dataset and a second semantic segmentation model trained on a person segmentation dataset, respectively, to obtain a foot mask and a person mask of the current image; determining a first foot centroid position of the current image based on the foot mask, and determining a first person centroid position of the current image based on the person mask. Centroid position; when the current image is an image after the first frame in the acquired multi-frame images, the second foot centroid position and the second human centroid position of the current image are predicted based on the Kalman filter algorithm; when the predicted second foot centroid position of the current image does not meet the first preset condition and the predicted second human centroid position of the current image meets the second preset condition, based on the specific mapping relationship between the human centroid position and the foot centroid position, the predicted second human centroid position of the current image is mapped to the third foot centroid position of the current image, so as to use the mapped third foot centroid position of the current image as the human foot position of the current image.

[0008] Optionally, the method may further include: when the current image is the first frame of the acquired multi-frame images, taking the determined first foot centroid position of the current image as the position of the person's foot in the current image.

[0009] Optionally, the method may further include: when the predicted centroid position of the second foot in the current image meets a first preset condition, the predicted centroid position of the second foot in the current image is taken as the position of the person's foot in the current image.

[0010] Optionally, the method may further include: when the predicted second foot centroid position of the current image does not meet the first preset condition and the predicted second human centroid position of the current image does not meet the second preset condition, the current image is re-input into a target detection network trained on a human foot dataset to obtain a target detection box of the current image, and the re-obtained target detection box of the current image is input into a second semantic segmentation model trained on a human segmentation dataset to obtain a human mask of the current image, and the second human centroid position of the current image is predicted based on the re-obtained human mask of the current image.

[0011] Optionally, when the current image is an image following the first frame in a multi-frame image set, predicting the second foot centroid position and the second human centroid position of the current image based on the Kalman filter algorithm may include: predicting the second foot centroid position of the current image based on the foot centroid position of the previous frame of the current image using the Kalman filter algorithm, and correcting the predicted second foot centroid position of the current image based on the determined first foot centroid position of the current image; predicting the second human centroid position of the current image based on the human centroid position of the previous frame of the current image, and correcting the predicted second foot centroid position of the current image based on the determined first foot centroid position of the current image; and predicting the second human centroid position of the current image based on the human centroid position of the previous frame of the current image using the Kalman filter algorithm, and correcting the predicted second foot centroid position of the current image based on the determined first foot centroid position of the current image. The determined first human centroid position of the current image is used to correct and predict the second human centroid position of the current image. Specifically, when the previous frame of the current image is the first frame of the acquired multi-frame image, the foot centroid position of the previous frame of the current image is the first foot centroid position, and the human centroid position of the previous frame of the current image is the first human centroid position. Furthermore, when the previous frame of the current image is not the first frame of the acquired multi-frame image, the foot centroid position of the previous frame of the current image is the second foot centroid position, and the human centroid position of the previous frame of the current image is the second human centroid position.

[0012] Optionally, correcting the predicted second foot centroid position of the current image based on the determined first foot centroid position of the current image may include: determining the foot prediction error of the current image based on the foot prediction error of the previous frame of the current image; determining the first Kalman gain of the current image based on the foot prediction error of the current image; and correcting the predicted second foot centroid position of the current image based on the first Kalman gain of the current image and the determined first foot centroid position of the current image.

[0013] Optionally, correcting the predicted second person's center position in the current image based on the determined first person's center position in the current image may include: determining the person's prediction error in the current image based on the person's prediction error in the previous frame of the current image; determining the second Kalman gain of the current image based on the person's prediction error in the current image; and correcting the predicted second person's center position in the current image based on the second Kalman gain of the current image and the determined first person's center position in the current image.

[0014] Optionally, the method may further include: determining a foot correction error of the current image based on the foot prediction error of the current image and a first Kalman gain of the current image, so as to update the foot prediction error of the current image to the foot correction error of the current image.

[0015] Optionally, the method may further include: determining a person correction error for the current image based on the person prediction error of the current image and the second Kalman gain of the current image, so as to update the person prediction error of the current image to the person correction error of the current image.

[0016] According to a second aspect of this disclosure, a system for real-time tracking of a person's foot position is provided. The system may include: an image acquisition module configured to acquire a current image; an object detection module configured to input the current image into an object detection network trained on a human foot dataset to obtain object detection boxes in the current image; a semantic segmentation module configured to input the object detection boxes of the current image into a first semantic segmentation model trained on a foot segmentation dataset and a second semantic segmentation model trained on a human segmentation dataset, respectively, to obtain a foot mask and a human mask in the current image; and a centroid position determination module configured to determine the first foot centroid position in the current image based on the foot mask, and to determine the... The first centroid position of the current image; the centroid position prediction module is configured to predict the second foot centroid position and the second human centroid position of the current image based on the Kalman filter algorithm when the current image is an image after the first frame of the acquired multi-frame images; the mapping module is configured to map the predicted second human centroid position of the current image to the third foot centroid position of the current image based on the specific mapping relationship between the human centroid position and the foot centroid position when the predicted second foot centroid position of the current image does not meet the first preset condition and the predicted second human centroid position of the current image meets the second preset condition, so as to use the mapped third foot centroid position of the current image as the human foot position of the current image.

[0017] According to a third aspect of the embodiments of this disclosure, an electronic device is provided, which may include: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by at least one of the processors, cause at least one of the processors to perform the method of real-time tracking of a person's footsteps.

[0018] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium storing instructions is provided, characterized in that, when the instructions are executed by at least one processor, they cause at least one of the processors to perform the method for real-time tracking of a person's footsteps.

[0019] According to embodiments of this disclosure, real-time tracking of a person's feet can be achieved, solving the problems of inaccurate tracking caused by foot position jumps due to occlusion or excessively fast movement of the person, and the significant interference from external factors due to complex image backgrounds. Furthermore, this method ensures high accuracy and adaptability of real-time automatic tracking of the target person's feet with minimal speed performance overhead, making it effectively applicable to virtual reality scenarios.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a method for real-time tracking of a person's footsteps according to an exemplary embodiment of the present disclosure;

[0022] Figure 2 This is an example flowchart illustrating the training of an object detection network based on a human foot dataset according to an example embodiment of the present disclosure;

[0023] Figure 3 This is an example illustration showing a person mask obtained by a second semantic segmentation model according to an example embodiment of the present disclosure;

[0024] Figure 4 This is a schematic block diagram illustrating a system for real-time tracking of a person's footsteps according to an exemplary embodiment of the present disclosure;

[0025] Figure 5 This is a schematic block diagram illustrating an electronic device according to an example embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two. The phrases "A and / or B" and "A and / or B" in this disclosure indicate the following three parallel cases: only A; only B; both A and B.

[0029] This disclosure proposes a technique for real-time tracking of human feet in video image sequence frames based on object detection and semantic segmentation combined with Kalman filtering algorithm. It can solve the problems of tracking loss caused by position jumps when human feet are occluded in image sequences, unstable tracking when moving too fast, and large external interference during target tracking.

[0030] Figure 1 This is a flowchart illustrating a method for real-time tracking of a person's footsteps according to an example embodiment of the present disclosure.

[0031] In step 110, the current image can be obtained.

[0032] In step 120, the current image can be input into a target detection network trained on a human foot dataset to obtain the target detection bounding box of the current image.

[0033] Figure 2 This is an example flowchart illustrating a training of an object detection network based on a human foot dataset according to an example embodiment of the present disclosure.

[0034] In step 210, data collection can begin. First, a dataset of human feet needs to be collected, containing a sufficient number and diversity of images to ensure the network's robustness and generalization performance. Data can be obtained through self-photography, downloading from the internet, crowdsourcing, etc. Currently, there is no publicly available dataset of human feet, making direct training impossible. Therefore, a custom-made dataset of human feet can be created. For example, a custom-made dataset of human feet could include images of 24 different colors, sizes, and angles, totaling 1440 images. Of these, 1300 images can be used as the training dataset, and 140 images as the validation dataset.

[0035] In step 220, data preprocessing can be performed.

[0036] Before training a network, the data needs to be preprocessed. For example, for object detection tasks, the data can be labeled, that is, a bounding box can be assigned to each person in the image, along with a category label for that bounding box, such as "person". For multi-class cases, different category labels can be assigned to people of different colors, sizes, and angles.

[0037] Furthermore, data augmentation can be performed to increase its diversity and quantity. Data augmentation techniques include mirroring, rotation, scaling, translation, random cropping, and color dithering. These augmentation operations can yield more data while improving the network's robustness.

[0038] In step 230, network design can be performed.

[0039] For object detection tasks, common network architectures can be used, such as Faster R-CNN, YOLO, and SSD. Among them, Faster R-CNN is an object detection network based on a Region Proposal Network (RPN), which can achieve high detection accuracy and fast detection speed.

[0040] In network design, hyperparameters such as learning rate, optimizer, and batch size can be set, as well as network structure parameters such as kernel size, number of convolutional layers, and number of fully connected layers. These parameters can be adjusted based on factors such as dataset size, sample distribution, and task difficulty.

[0041] In step 240, network training can be performed. Network training may include three steps: forward propagation, back propagation, and parameter update. During forward propagation, the network performs operations such as feature extraction, object classification, and location regression on the input image to generate bounding boxes and class probability predictions. During back propagation, the network calculates the gradient based on the error between the prediction results and the true labels, and updates the network parameters according to the optimizer's algorithm. The network training process can be performed on hardware platforms such as CPUs and GPUs to accelerate computation.

[0042] In step 250, network evaluation can be performed. After network training is complete, it needs to be evaluated to verify its detection accuracy and generalization performance on the validation set. Standard metrics such as mean average precision (mAP), precision, and recall can be used to evaluate and compare the network's detection results. If the network's detection accuracy is not ideal, it can be optimized by adjusting the network structure, modifying the data augmentation strategy, or changing the optimizer.

[0043] As an example, a YOLOv5-based object detection network can be used. The object detection network utilizes YOLOv5 to detect objects, generating a set of bounding boxes. This data serves as input for semantic segmentation, reducing the complexity of the input data, increasing the speed of semantic segmentation, and effectively improving segmentation accuracy.

[0044] The YOLOv5 object detection algorithm uses Masaic data augmentation at the input stage. Masaic data augmentation stitches together four images by random scaling, cropping, and arrangement into a single image. This enriches the dataset, improves network training speed, and reduces model memory requirements. Simultaneously, it embeds adaptive anchor box calculation and adaptive image scaling. Adaptive anchor boxes are calculated adaptively based on the dataset name during each training iteration. Adaptive image scaling addresses the issue of varying aspect ratios and redundant information in black borders affecting algorithm inference speed. The processed input image is then passed to a baseline network. The Foucs (downsampling layer) structure in the baseline network crops the input image using a slice-concat-LeakyReLU operation to obtain deep features. In the Neck network, a CSP (Cross-Stage Local Network) structure can be added to enhance the network's feature fusion capabilities. The detection head uses GIoU_Loss as the Bounding_Boxd loss function. GIoU_Loss adds a way to measure the intersection scale, which solves the problem of non-overlapping bounding boxes and alleviates the situation where the predicted box and the ground truth box do not intersect or cannot distinguish the relative positional relationship between the predicted box and the ground truth box. Finally, the accurate target detection box is output.

[0045] Object detection technology can detect people in an image, while semantic segmentation technology can further segment the feet / people to obtain more accurate information about the location of the feet.

[0046] In steps 131 and 132, the target detection bounding box of the current image can be input into the first semantic segmentation model trained on the foot segmentation dataset and the second semantic segmentation model trained on the person segmentation dataset, respectively, to obtain the foot mask and person mask of the current image.

[0047] Figure 3 This is an example diagram illustrating a person mask obtained by a second semantic segmentation model according to an example embodiment of the present disclosure.

[0048] In this disclosure, a real-time semantic segmentation model can be trained using a foot segmentation dataset and a person segmentation dataset, respectively. A foot segmentation dataset can be collected, which may include foot images and their corresponding segmentation annotations. Similarly, a person segmentation dataset can be collected, which may include person images and their corresponding segmentation annotations. As an example only, the Cocoa public dataset can be used as the person segmentation dataset. The segmentation annotations are images of the same size as the original images, with each pixel labeled with a different color to indicate which semantic category that pixel belongs to.

[0049] When training on foot segmentation and person segmentation datasets, it's crucial to select a suitable semantic segmentation model. Commonly used models include PP_LiteSeg, UNet, DeepLabV3, and FCN. The appropriate model should be chosen based on the specific task requirements and hardware resource limitations. Furthermore, for foot segmentation and person segmentation datasets, common data augmentation methods such as random rotation, flipping, scaling, and cropping can be employed. It's also essential to preserve the key features of the feet / person to improve model accuracy.

[0050] After preparing the dataset and selecting the model, the model can be trained using the training dataset from the foot segmentation dataset / person segmentation dataset. During training, common loss functions such as cross-entropy loss and Dice loss can be used. Additionally, hyperparameters such as learning rate and batch size need to be set during training. During training, the model needs to be evaluated using a validation dataset to identify and adjust any issues promptly. Evaluation metrics include mIoU (mean intersection-over-union), accuracy, and recall, which can be used to assess the model's performance.

[0051] After training and evaluation, the model can be optimized based on its performance. For example, deeper network structures can be used, pre-trained models can be fine-tuned, and data augmentation strategies can be implemented. Furthermore, advanced techniques such as transfer learning and adaptive learning rates can be employed to further improve model performance.

[0052] Here, the first semantic segmentation model and the second semantic segmentation model can be the same type of real-time semantic segmentation model, or they can be different types of real-time semantic segmentation models.

[0053] As an example only, for both the first and second semantic segmentation models, a real-time semantic segmentation model based on PP_LiteSeg can be used for semantic segmentation. Here, a foot segmentation dataset can be created. The original images used in the foot segmentation dataset can be the same as those in the aforementioned human figure foot dataset, and the number of training and validation images is also the same as those used for object detection, with a one-to-one correspondence. During the segmentation process, the target object is set to black, and the background is set to white to obtain a binary image with RGB three channels (e.g., ...). Figure 3 As shown, single-channel data is extracted using the binary image's property of being either 0 or 1, thus obtaining single-channel target segmentation data. This aims to reduce the impact of external factors on accurate target tracking, simplify the tracking data, and reduce computational complexity. The semantic segmentation network PP_LiteSeg is a lightweight semantic segmentation network that performs four downsampling operations in the encoding part. Each layer (stage) introduces an STDC (Short-Term Dense Connection) module. In the decoding part, combining the U-Net idea of ​​gradually reducing the number of channels and increasing the feature space size through upsampling, a flexible lightweight decoder FLD is proposed, balancing the computational complexity of encoding and decoding and improving network computational efficiency. To better integrate contextual information, a Simple Pyramid Pooling (SPPM) module is proposed. Furthermore, a unified attention fusion module (UAFM) that integrates multi-layer features is proposed in the decoder FLD to improve segmentation accuracy.

[0054] In the UAFM module, you can first process the higher-level features F. high Upsampling is performed to obtain high-level features F up :

[0055] F up =Upsample(F high (1)

[0056] Simultaneously, the downsampling phase and F up Low-level feature F with the same number of channels low As input, the attention module is used to calculate the weight α:

[0057] α = Attention(F) up ,F low (2)

[0058] Finally, the upsampled and downsampled features are fused to obtain a new feature map:

[0059] F out=F up ·α+F low ·(1-α) (3)

[0060] FLD consists of two UAFMs and a segmentation head. After the two UAFMs are fused, 1 / 8 of the downsampled features are obtained. In the segmentation head, Conv (convolution)-BN (batch sample normalization)-ReLU (activation function) operations are performed to reduce the number of channels of the 1 / 8 downsampled features to the number of classes. Upsampling operation is used to expand the feature size to the size of the input image, and argmax (channel extraction) operation is used to predict the label of each pixel. Then, post-processing is performed on each pixel to indicate whether it is the background or the target person / foot. Finally, the target mask is output.

[0061] In step 141, the position of the first foot centroid of the current image can be determined based on the foot mask of the current image.

[0062] In this disclosure, a foot mask is obtained after semantic segmentation of the target person detected in the current image. f Considering the uncertainty of the character's left and right foot movements, the connections between feet, and the accuracy and robustness of the foot tracking algorithm, the foot mask is divided into two parts: the left foot mask. fL and right foot mask fR And calculate the corresponding foot center of mass respectively. The calculation of the centroid involves using the mask as the calculation region, which is simplified by setting μ(x,y)=1.

[0063]

[0064]

[0065] in, and Indicates the coordinates of the left or right foot.

[0066] In step 142, the first centroid position of the person in the current image can be determined based on the person mask in the current image. The method for determining the first centroid position of the person in the current image is similar to the method for determining the first centroid position of the feet in the current image; simply replace the foot mask with the person mask, which will not be described in detail here.

[0067] In step 151, it can be determined whether the current image is the first frame of the acquired multi-frame images. When the current image is the first frame of the acquired multi-frame images (yes in step 151), in step 199, the determined first foot centroid position of the current image can be used as the position of the person's foot in the current image.

[0068] When the current image is an image following the first frame in a multi-frame image set (No in step 151), in step 161, the centroid position of the second foot in the current image can be predicted based on the Kalman filter algorithm. That is, the left and right sides of the foot from the previous frame can be masked. Center of mass Used as initial values ​​for the Kalman filter algorithm to predict the left and right occlusion centroids of the feet in the next frame. Location.

[0069] According to an example embodiment of this disclosure, the second foot centroid position of the left / right foot in the current image can be predicted based on the foot centroid position of the previous frame image using a Kalman filter algorithm.

[0070]

[0071] in, It can indicate the position of the centroid of the foot in the previous frame image. The predicted second foot centroid position of the current image can be represented by 'a', which can be a preset parameter and can be adjusted. When the previous frame of the current image is the first frame of the acquired multi-frame image, the foot centroid position of the previous frame of the current image is the first foot centroid position. When the previous frame of the current image is not the first frame of the acquired multi-frame image, the foot centroid position of the previous frame of the current image is the second foot centroid position.

[0072] Furthermore, based on the determined first foot centroid position of the current image, the predicted second foot centroid position of the current image can be corrected to obtain the optimal estimated centroid of the left / right foot.

[0073] According to an example embodiment of this disclosure, the foot prediction error can be based on the previous frame image of the current image. Determine the foot prediction error of the current image.

[0074]

[0075] When the current image is the second frame in the multi-frame image set, the foot prediction error of the previous frame of the current image can be 1. The first Kalman gain K[t] of the current image can be determined based on the foot prediction error of the current image, and used as the weight of the correction part.

[0076]

[0077] The predicted second foot centroid position of the current image can be corrected based on the first Kalman gain of the current image and the determined first foot centroid position of the current image.

[0078]

[0079] in, σ 2 These can be preset parameters and can be adjusted, where Z[t] is the determined position of the first foot centroid of the current image.

[0080] Furthermore, the foot correction error of the current image can be determined based on the foot prediction error of the current image and the first Kalman gain of the current image. The foot prediction error of the current image is updated to the foot correction error of the current image:

[0081]

[0082] In other words, the position of the second foot centroid can be predicted and corrected separately for the left and right feet. The foot correction error of the current image can be used to determine the foot prediction error of the next frame, and this process is repeated iteratively.

[0083] Similarly, in step 152, it can be determined whether the current image is the first frame image in the acquired multi-frame images. When the current image is an image after the first frame image in the acquired multi-frame images ("No" in step 152), in step 162, the second person's heart position of the current image can be predicted based on the Kalman filter algorithm. That is, based on the Kalman filter algorithm, the second person's heart position of the current image can be predicted according to the person's heart position of the previous frame image of the current image, and the predicted second person's heart position of the current image can be corrected according to the determined first person's heart position of the current image. Here, when the previous frame image of the current image is the first frame image in the acquired multi-frame images, the person's heart position of the previous frame image of the current image is the first person's heart position; when the previous frame image of the current image is not the first frame image in the acquired multi-frame images, the person's heart position of the previous frame image of the current image is the second person's heart position. When correcting the predicted second person's heart position in the current image based on the determined first person's heart position, the prediction error of the current image can be determined based on the person prediction error of the previous frame. The second Kalman gain of the current image is then determined based on this error, and the predicted second person's heart position is corrected based on the second Kalman gain and the determined first person's heart position. When the current image is the second frame in a multi-frame image set, the person prediction error of the previous frame can be 1. Furthermore, the person correction error of the current image can be determined based on the person prediction error and the second Kalman gain, updating the current image's prediction error to the current image's correction error. The current image's correction error can be used to determine the person prediction error for the next frame, and this process is iterated. Here, the method for predicting (and correcting) the centroid position of the second person in the current image based on the Kalman filter algorithm is similar to the method for predicting (and correcting) the centroid position of the second foot in the current image based on the Kalman filter algorithm, so it will not be described again here.

[0084] When the predicted second foot centroid position of the current image does not meet the first preset condition ("No" in step 171) and the predicted second human centroid position of the current image meets the second preset condition ("Yes" in step 172), in step 180, based on the specific mapping relationship between the human centroid position and the foot centroid position, the predicted second human centroid position of the current image can be mapped to the third foot centroid position of the current image, so that the mapped third foot centroid position of the current image can be used as the human foot position of the current image (step 199).

[0085] Here, the predicted position of the second foot centroid of the current image not satisfying the first preset condition can be any one or a combination of the following:

[0086] ① The span of the center of gravity of the left foot is too large.

[0087]

[0088] ② The span of the right foot's center of gravity is too large.

[0089]

[0090] ③ The relative distance between the centers of mass of the left and right feet is too large.

[0091]

[0092] in, or This refers to the centroid position of the left or right foot in the current image. or This refers to the position of the centroid of the left or right foot in the previous frame. or This refers to the centroid position of the left or right foot in the next frame. If the centroid deviation during foot tracking exceeds the aforementioned constraints, tracking is considered a failure, requiring a re-evaluation. Therefore, the search area is expanded to the target person corresponding to the foot, and after determining the person's position, the centroid of the left and right feet is re-estimated. and The accurate foot position is obtained. That is, when the predicted second foot centroid position of the current image does not meet the first preset condition and the predicted second human centroid position of the current image meets the second preset condition, based on the specific mapping relationship between the human centroid position and the foot centroid position, the predicted second human centroid position of the current image is mapped to the third foot centroid position of the current image, so as to use the mapped third foot centroid position of the current image as the foot position of the person in the current image.

[0093] Here, the second human body center position predicted in the current image satisfies the second preset condition if the difference between the predicted second human body center position and the true value of the human body center position (the determined first human body center position in the current image) is within a preset range.

[0094] Furthermore, when the predicted second foot centroid position of the current image meets the first preset condition ("yes" in step 171), the predicted second foot centroid position of the current image can be used as the position of the person's foot in the current image.

[0095] Furthermore, when the predicted second foot centroid position of the current image does not meet the first preset condition ("No" in step 171) and the predicted second human centroid position of the current image does not meet the second preset condition ("No" in step 172), in step 190, the current image can be re-input into the target detection network trained based on the human foot dataset to obtain the target detection box of the current image, and in step 132, the re-obtained target detection box of the current image is input into the second semantic segmentation model trained based on the human segmentation dataset to obtain the human mask of the current image, and the second human centroid position of the current image is predicted based on the re-obtained human mask of the current image (steps 142 to 162).

[0096] As described above, the detection and tracking of the target person and their feet are performed in parallel (i.e., steps 131-161 and steps 132-162 are performed in parallel). When an error occurs in foot tracking, it can be corrected in real time using the centroid tracking data of the person, and the correct tracking data is fed back to ensure that subsequent tracking proceeds normally. Compared with the method of initializing the target object tracking box after foot tracking failure, this method is faster, has better real-time performance, higher accuracy, and better robustness.

[0097] However, those skilled in the art should understand that this disclosure is not limited thereto. For example, steps 131 to 171 can be performed first, and steps 132 to 172 can only be performed if the foot position cannot be accurately tracked (in step 171 it is "No"). In this way, the amount of computation can be greatly reduced, and the burden on the processor can be alleviated. Optionally, a specific time difference can be set between steps 131 and 132, and steps 132 and subsequent steps can begin after step 131 has started and the specific time difference has elapsed.

[0098] This disclosure integrates object detection (e.g., the YOLOv5 network, which boasts fast inference speed, good performance, and fewer weight files), semantic segmentation (e.g., the lightweight real-time semantic segmentation network PP_LiteSeg, which balances accuracy and speed with good compatibility), and Kalman filtering. Furthermore, it proposes a dual-detection, dual-prediction model during the Kalman filtering process (i.e., the detection and tracking of the target person and their feet are performed in parallel). The Kalman filtering algorithm is a computer-friendly algorithm capable of real-time signal processing. It cleverly integrates observation and estimation datasets for closed-loop feedback management, accelerating algorithm implementation while ensuring accurate estimation. In the overall algorithm's timeliness test, it achieves an output time of approximately 0.018 seconds per image, guaranteeing real-time foot tracking and accurate location of the person.

[0099] Foot tracking technology can be applied to XR (Extended Reality) technology, greatly enhancing the realism of people in virtual scenes. Foot tracking can accurately capture a person's position after movement within a scene, achieving precise localization and depth tracking without adding any external sensors. This allows people in virtual scenes to move freely within the shooting range without restriction. Foot tracking technology can also be applied to human behavior analysis, predicting a person's next movement. This technology can also be extended to accurately locate the coordinates of a target person in three-dimensional space, potentially advancing fields such as autonomous driving, intelligent video surveillance, human-computer interaction, and robot visual navigation.

[0100] Figure 4 This is a schematic block diagram illustrating a system 400 for real-time tracking of a person's footsteps according to an example embodiment of the present disclosure.

[0101] System 400 may include an image acquisition module 410, an object detection module 420, a semantic segmentation module 430, a centroid position determination module 440, a centroid position prediction module 450, and a mapping module 460. System 400 can be installed in various electronic devices to track the position of a person's footsteps in real time, or the various modules in system 400 can implement their respective functions separately, or system 400 can be distributed on the internet or in the cloud for user use.

[0102] According to an example embodiment of this disclosure, when the current image is the first frame of a multi-frame image, the centroid position determination module 440 can use the determined first foot centroid position of the current image as the position of the person's foot in the current image.

[0103] According to an example embodiment of this disclosure, when the predicted centroid position of the second foot in the current image meets the first preset condition, the centroid position prediction module 450 can use the predicted centroid position of the second foot in the current image as the position of the person's foot in the current image.

[0104] According to an example embodiment of this disclosure, when the predicted second foot centroid position of the current image does not meet the first preset condition and the predicted second human centroid position of the current image does not meet the second preset condition, the re-detection module (not shown) can re-input the current image into a target detection network trained on a human foot dataset to obtain the target detection box of the current image, and input the re-obtained target detection box of the current image into a second semantic segmentation model trained on a human segmentation dataset to obtain the human mask of the current image, and predict the second human centroid position of the current image based on the re-obtained human mask of the current image.

[0105] According to an example embodiment of this disclosure, the centroid position prediction module 450 can predict the second foot centroid position of the current image based on the foot centroid position of the previous frame image of the current image using a Kalman filter algorithm, and correct the predicted second foot centroid position of the current image based on the determined first foot centroid position of the current image.

[0106] According to an example embodiment of this disclosure, the centroid position prediction module 450 can predict the second centroid position of the current image based on the centroid position of the person in the previous frame of the current image using a Kalman filter algorithm, and correct the predicted second centroid position of the current image based on the determined first centroid position of the person in the current image.

[0107] When the preceding frame of the current image is the first frame of the acquired multi-frame image, the foot centroid position of the preceding frame of the current image is the first foot centroid position, and the human centroid position of the preceding frame of the current image is the first human centroid position. When the preceding frame of the current image is not the first frame of the acquired multi-frame image, the foot centroid position of the preceding frame of the current image is the second foot centroid position, and the human centroid position of the preceding frame of the current image is the second human centroid position.

[0108] According to an example embodiment of this disclosure, the centroid position prediction module 450 can determine the foot prediction error of the current image based on the foot prediction error of the previous frame image of the current image, determine the first Kalman gain of the current image based on the foot prediction error of the current image, and correct the predicted second foot centroid position of the current image based on the first Kalman gain of the current image and the determined first foot centroid position of the current image.

[0109] According to an example embodiment of this disclosure, the centroid position prediction module 450 can determine the person prediction error of the current image based on the person prediction error of the previous frame image of the current image, determine the second Kalman gain of the current image based on the person prediction error of the current image, and correct the predicted second person centroid position of the current image based on the second Kalman gain of the current image and the determined first person centroid position of the current image.

[0110] According to an example embodiment of this disclosure, a correction error determination module (not shown) can determine a foot correction error for the current image based on the foot prediction error and a first Kalman gain of the current image, so as to update the foot prediction error of the current image to the foot correction error of the current image. The correction error determination module can also determine a person correction error for the current image based on the person prediction error and a second Kalman gain of the current image, so as to update the person prediction error of the current image to the person correction error of the current image.

[0111] Various modules according to exemplary embodiments of this disclosure may be integrated into a single module to perform their functions, or at least one of the various modules may be divided into more smaller modules.

[0112] Figure 5 This is a schematic block diagram illustrating an electronic device 500 according to an example embodiment of the present disclosure.

[0113] According to an example embodiment of this disclosure, an electronic device 500 may also be provided, which may include a processor 510 and a memory 520 storing computer-executable instructions. When executed by the processor 510, the computer-executable instructions cause the processor 510 to perform the method for real-time tracking of a person's footsteps as described above according to an example embodiment of this disclosure.

[0114] According to embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform a method according to an exemplary embodiment of the present disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0115] According to embodiments of the present disclosure, a computer program product may also be provided, wherein instructions in the computer program product may be executed by at least one processor in an electronic device to perform a method according to exemplary embodiments of the present disclosure.

[0116] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0117] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the claims.

Claims

1. A method for tracking a human foot position in real time, comprising: obtaining a current image; inputting the current image into a target detection network trained based on a human foot dataset to obtain a target detection box of the current image; inputting the target detection box of the current image into a first semantic segmentation model trained based on a foot segmentation dataset and a second semantic segmentation model trained based on a human segmentation dataset respectively to obtain a foot mask and a human mask of the current image respectively; determining a first foot center position of the current image based on the foot mask of the current image and determining a first human center position of the current image based on the human mask of the current image; when the current image is an image after a first image in a plurality of obtained images, predicting a second foot center position of the current image and a second human center position of the current image based on a Kalman filtering algorithm; when the predicted second foot center position of the current image does not satisfy a first preset condition and the predicted second human center position of the current image satisfies a second preset condition, mapping the predicted second human center position of the current image to a third foot center position of the current image based on a specific mapping relationship between the human center position and the foot center position, and taking the mapped third foot center position of the current image as the human foot position of the current image.

2. The method of claim 1, further comprising: when the current image is the first image in the plurality of obtained images, taking the determined first foot center position of the current image as the human foot position of the current image.

3. The method of claim 1, further comprising: when the predicted second foot center position of the current image satisfies the first preset condition, taking the predicted second foot center position of the current image as the human foot position of the current image.

4. The method of claim 1, further comprising: when the predicted second foot center position of the current image does not satisfy the first preset condition and the predicted second human center position of the current image does not satisfy the second preset condition, re-inputting the current image into the target detection network trained based on the human foot dataset to obtain a target detection box of the current image, inputting the re-obtained target detection box of the current image into the second semantic segmentation model trained based on the human segmentation dataset to obtain a human mask of the current image, and predicting a second human center position of the current image based on the re-obtained human mask of the current image.

5. The method of claim 1, when the current image is an image after the first image in the plurality of obtained images, predicting a second foot center position of the current image and a second human center position of the current image based on a Kalman filtering algorithm, comprising: predicting the second foot center position of the current image according to the foot center position of the previous frame image of the current image based on a Kalman filtering algorithm, and correcting the predicted second foot center position of the current image according to the determined first foot center position of the current image, predicting the second person center position of the current image according to the person center position of the previous frame image of the current image based on a Kalman filtering algorithm, and correcting the predicted second person center position of the current image according to the determined first person center position of the current image, wherein, when the previous frame image of the current image is the first frame image in the plurality of acquired frame images, the foot center position of the previous frame image of the current image is the first foot center position, the person center position of the previous frame image of the current image is the first person center position, and wherein, when the previous frame image of the current image is not the first frame image in the plurality of acquired frame images, the foot center position of the previous frame image of the current image is the second foot center position, and the person center position of the previous frame image of the current image is the second person center position.

6. The method of claim 5, wherein, correcting the predicted second foot center position of the current image according to the determined first foot center position of the current image comprises: determining a foot prediction error of the current image based on the foot prediction error of the previous frame image of the current image; determining a first Kalman gain of the current image according to the foot prediction error of the current image; and correcting the predicted second foot center position of the current image according to the first Kalman gain of the current image and the determined first foot center position of the current image, correcting the predicted second person center position of the current image according to the determined first person center position of the current image comprises: determining a person prediction error of the current image based on the person prediction error of the previous frame image of the current image; determining a second Kalman gain of the current image according to the person prediction error of the current image; and correcting the predicted second person center position of the current image according to the second Kalman gain of the current image and the determined first person center position of the current image.

7. The method of claim 6, further comprising: determining a foot correction error of the current image according to the foot prediction error of the current image and the first Kalman gain of the current image, to update the foot prediction error of the current image to the foot correction error of the current image, and determining a person correction error of the current image according to the person prediction error of the current image and the second Kalman gain of the current image, to update the person prediction error of the current image to the person correction error of the current image.

8. A system for tracking a person's foot position in real time, comprising: an image acquisition module configured to acquire a current image, The target detection module is configured to input the current image into a target detection network trained based on a person foot dataset to obtain a target detection frame of the current image. The semantic segmentation module is configured to input the target detection frame of the current image into a first semantic segmentation model trained based on a foot segmentation dataset and a second semantic segmentation model trained based on a person segmentation dataset respectively to obtain a foot mask and a person mask of the current image respectively. The center position determination module is configured to determine a first foot center position of the current image based on the foot mask of the current image and determine a first person center position of the current image based on the person mask of the current image. The center position prediction module is configured to predict a second foot center position of the current image and a second person center position of the current image based on a Kalman filtering algorithm when the current image is an image after a first frame image in the obtained multiple frames of images. The mapping module is configured to map the predicted second person center position of the current image to a third foot center position of the current image based on a specific mapping relationship between the person center position and the foot center position when the predicted second foot center position of the current image does not satisfy a first preset condition and the predicted second person center position of the current image satisfies a second preset condition, so as to take the mapped third foot center position of the current image as the person foot position of the current image.

9. An electronic device comprising: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 7.

10. A computer readable storage medium storing instructions, characterized in that, The instructions, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 7.

Images