Object tracking method, device, storage medium and product based on fttr
By constructing a distributed feature storage architecture using FTTR technology, the problem of object tracking accuracy in complex scenarios is solved, achieving efficient object tracking and cross-camera trajectory correlation, thus improving the system's tracking performance in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZTE CORP
- Filing Date
- 2025-05-07
- Publication Date
- 2026-08-04
AI Technical Summary
In complex scenarios, existing object tracking methods suffer from inaccurate trajectory association, frequent ID switching, low efficiency in cross-scene feature comparison, and lack of cross-modal interaction capabilities, resulting in poor object tracking accuracy.
By employing FTTR technology and deploying fiber optic networks in each room to build a distributed feature storage architecture, combined with the computing power of edge nodes and central nodes, efficient feature matching and trajectory updates are achieved, reducing latency and improving real-time performance and accuracy.
In complex scenarios such as differences in camera perspectives, rapid movement, or occlusion, the accuracy and stability of object tracking are improved, mismatches and trajectory interruptions are reduced, and the robustness of the system is enhanced.
Smart Images
Figure CN120107624B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of visual tracking technology, and in particular to an object tracking method, device, storage medium and product based on FTTR. Background Technology
[0002] With the advancement of computer vision, the application scenarios of visual tracking are gradually increasing. However, in complex scenarios, there are still problems such as inaccurate trajectory association and frequent ID switching caused by cross-camera, occlusion, rapid movement or detection noise, which result in poor object tracking accuracy. Summary of the Invention
[0003] This application provides an object tracking method, device, storage medium, and product based on FTTR, which can solve the technical problem of poor accuracy in object tracking under complex scenarios. The technical solution is as follows.
[0004] On the one hand, an object tracking method based on FTTR is provided, the method comprising: Perform object detection on the target video frame to obtain at least one detection box; In response to the fact that the at least one detection box contains a target detection box that does not match the target predicted position, the target detection box is matched with the first detection boxes with known tracking trajectories stored in the database to obtain a detection box matching result. The target predicted position is the predicted position of each tracking trajectory in the target video frame. The first detection box is a detection box with a confidence score higher than a score threshold. The confidence score is used to indicate the probability that the detected object is contained at the position marked by the detection box. The tracking trajectory of each detected object is updated based on the detection box matching results.
[0005] On the other hand, an electronic device is provided, comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described FTTR-based object tracking method.
[0006] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described FTTR-based object tracing method.
[0007] On the other hand, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to execute to implement the FTTR-based object tracking method provided in the various optional implementations described above.
[0008] The object tracking method based on FTTR provided in this application stores high-confidence historical detection boxes for each tracking trajectory in a database. If at least one detection box contains a target detection box whose predicted position is not matched to any tracking trajectory, the target detection box is matched with the high-confidence historical detection boxes of each tracking trajectory in the database. Based on the detection box matching results, the tracking trajectory corresponding to the target detection box is determined. In this method, storing high-confidence historical detection boxes in the database reduces the interference of detection noise on detection box matching and improves matching accuracy. Furthermore, by performing feature matching between historical detection boxes and target detection boxes, cross-frame association can be achieved, enabling trajectory tracking in complex scenarios such as differences in camera viewpoints, rapid movement, or trajectory interruptions caused by object occlusion, thereby improving the tracking effect of detected objects in complex scenarios.
[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0011] Figure 1 A flowchart of an FTTR-based object tracking method provided in an exemplary embodiment of this application is shown; Figure 2 A flowchart of an object tracking method provided in another exemplary embodiment of this application is shown; Figure 3 A schematic diagram of an object tracking method provided in an exemplary embodiment of this application is shown; Figure 4 An architecture diagram of an FTTR-based object tracing method provided in an exemplary embodiment of this application is shown; Figure 5 This application shows a structural block diagram of an electronic device according to an exemplary embodiment; Figure 6 A structural block diagram of an electronic device is shown in another exemplary embodiment of this application. Detailed Implementation
[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods consistent with some aspects of this application as detailed in the appended claims.
[0013] In complex scenarios, existing object tracking methods still suffer from issues such as identity shifting in occluded environments, insufficient handling of low-quality bounding boxes, low efficiency in cross-scene feature comparison, and lack of cross-modal interaction capabilities. Specifically, regarding identity shifting in occluded environments, current object tracking methods rely on a single bounding box confidence threshold for matching. When target occlusion causes a sharp drop in detection scores, low-quality bounding boxes are easily discarded incorrectly or confused with adjacent targets, leading to frequent identity switching (e.g., the same person being misidentified as a new individual after occlusion), making it difficult to guarantee identity continuity in cross-room tracking. Furthermore, for low-quality bounding boxes... The existing object tracking methods suffer from several problems. First, insufficient quality bounding box processing. They cannot effectively correlate low-scoring bounding boxes of blurred or partially occluded targets with existing trajectories, leading to low utilization of low-quality bounding box information and causing trajectory interruptions or redundant trajectory generation. Second, in cross-scene feature comparison, especially in multi-target scenarios, existing feature libraries incur significant computational overhead and poor real-time performance when performing cross-room matching. Third, the lack of cross-modal interaction capabilities means that existing systems rely on structured queries (such as timestamps and region coordinates) and cannot directly retrieve data using natural language descriptions, limiting the flexibility of user interaction. To address these issues, this application provides an FTTR (Fiber To The Room) based object tracking method, which can solve one or more of these problems and improve the accuracy of object tracking in complex scenarios.
[0014] FTTR is an all-optical networking solution that deploys optical fiber directly to each room, providing ultra-high-speed, low-latency, and highly stable whole-house network coverage. In this embodiment, based on the high bandwidth and low latency transmission capabilities of FTTR, a distributed feature storage architecture is constructed. This involves deploying visual sensing devices and lightweight feature extraction models at the edge nodes of each room (or area), storing the locally extracted feature vectors in real time through a distributed storage system, and establishing a global index library at the central node. This architecture can reduce the latency of multi-room data synchronization, improve real-time performance, reduce the storage and computing pressure on the central node, and support concurrent processing of multiple visual sensing devices.
[0015] Based on FTTR technology, visual sensors can transmit acquired video frame data to a central node via a local area network or wireless network, enabling the central node to perform corresponding processing, such as executing the FTTR-based object tracking method provided in the embodiments of this application. In another possible implementation, to reduce data transmission latency and improve data processing efficiency, some calculations in the method provided in this application can be performed on edge devices, such as on the visual sensor or a local server. For example, the edge device can perform object detection tasks and classify detection boxes into different types, sending each detection box and its corresponding type to the central node for subsequent operations, thereby reducing transmission latency and lowering the computational burden on the central node. Furthermore, to support large-scale data storage, model inference, and historical data analysis, and to reduce the burden on the central node, this architecture can also be combined with a cloud computing platform for large-scale model inference and user query processing, etc.
[0016] The aforementioned architecture features high-efficiency computing and storage capabilities, low-latency communication networks, real-time visual processing and object tracking algorithms, and intelligent query capabilities across modal large models, enabling it to provide accurate multi-room visual monitoring and object tracking services.
[0017] Based on this architecture, taking the FTTR-based object tracing method provided in this application as an example, which is executed by the central node, Figure 1 A flowchart illustrating an exemplary embodiment of this application shows an object tracking method based on FTTR, which can be executed by an electronic device that can be implemented as a central node in the above architecture, such as... Figure 1 As shown, the FTTR-based object tracking method may include the following steps.
[0018] Step 110: Perform object detection on the target video frame to obtain at least one detection box.
[0019] The target video frame can be any one of the continuous video frames acquired by the visual sensing device, or any one of the video frames acquired according to the target period. The visual sensing device can be a camera or other visual sensing device deployed in the FTTR scenario, such as a depth camera, an infrared camera, or a 360-degree panoramic camera. In one possible application scenario, the visual sensing device can be installed in various rooms indoors to acquire indoor image data in real time and synchronize the acquired indoor image data to electronic devices in real time. Correspondingly, the electronic devices acquire the indoor image data synchronized by each visual sensing device.
[0020] For a target video frame, the electronic device can perform object detection based on a visual detection model to obtain at least one detection box if the target video frame contains a detected object. This visual detection model can be a machine learning model trained on a CNN (Convolutional Neural Network). Illustratively, the visual detection model can be determined based on the actual usage scenario, the computing power provided by the hardware device, and the inference latency requirements; this application does not impose any limitations on this. To run or train the model trained on the CNN, the electronic device can deploy a deep learning architecture, such as TensorFlow, PyTorch, etc. Illustratively, the visual detection model can be implemented as at least one of the following: YOLO (You Only Look Once), Fast R-CNN (Fast Region-based Convolutional Neural Network), Faster R-CNN (Faster Region-based Convolutional Neural Network), or SSD (Single Shot MultiBox). The detector (single-shot multi-frame detector) includes location information and a confidence score. The location information is the labeled coordinates of the detection box in the video frame. The shape of the detection box can be a preset shape, such as a rectangle or an ellipse, or it can be a shape adapted to the detected object. This application does not impose any restrictions on this. The confidence score can be generated by the visual detection model when performing object detection. It is used to indicate the probability that the detected object is contained at the labeled position of the detection box. The higher the confidence score of the detection box, the higher the probability that the detection box contains the detected object. Conversely, the lower the confidence score, the lower the probability that the detection box contains the detected object.
[0021] The object of detection can be a human body, or other living organisms, or other mobile devices (such as vehicles, robots, etc.), and this application does not impose any restrictions on this.
[0022] Step 120: In response to at least one detection box containing a target detection box that does not match the target predicted position, feature matching is performed between the target detection box and each first detection box with known tracking trajectories stored in the database to obtain a detection box matching result. The target predicted position is the predicted position of each tracking trajectory in the target video frame. The first detection box is a detection box with a confidence score higher than a score threshold. The confidence score is used to indicate the probability that the detected object is contained at the position marked by the detection box.
[0023] In this embodiment, the predicted position of each tracking trajectory in the target video frame can be predicted based on the position information and motion state of the detected object tracked in the previous frame. In one possible implementation, the electronic device can use a motion model to predict the predicted position of each tracking trajectory in the current frame, i.e., the predicted trajectory of the current frame. The motion model can be implemented as a Kalman filter (KF). The electronic device can input the position information and motion state of the detected object in the previous frame into the motion model to obtain the predicted position of the tracking trajectory corresponding to the detected object in the current frame. The motion state can include motion speed and motion direction, etc.
[0024] For each detection box, the electronic device can match the detection box with the predicted target location. During the matching process, the electronic device can calculate the matching degree between the detection box and the predicted target location and determine whether a match has been made with the predicted target location based on a preset matching degree threshold. For target detection boxes that have not been matched with the predicted target location, the electronic device can perform feature matching with each high-scoring box (i.e., a detection box with a confidence score higher than the score threshold) stored in the database that has a known tracking path to obtain the detection matching result.
[0025] The database can be implemented as a relational database, vector database, key-value store database, or distributed storage system, etc., to facilitate the storage and management of detection box data, object feature vectors, and tracking trajectory information.
[0026] Step 130: Update the tracking trajectory of each detected object based on the detection box matching results.
[0027] If the detection box matching result indicates that the target detection box successfully matches the first detection box of a certain tracking trajectory in the database, the electronic device can determine that the target detection box is a continuation of the tracking trajectory in the target video frame, and thus can update the tracking trajectory based on the target detection box.
[0028] If the detection box matching result indicates that the target detection box has not been successfully matched with the first detection box of each tracking trajectory in the database, it means that the detected object in the target detection box has not appeared before, or that the detected object has few features and cannot be accurately matched. The electronic device can further determine the tracking trajectory based on the type of the target detection box.
[0029] In summary, the FTTR-based object tracking method provided in this application stores high-confidence historical detection boxes for each tracking trajectory in a database. If at least one detection box contains a target detection box whose predicted position is not matched to any tracking trajectory, feature matching is performed between this target detection box and the high-confidence historical detection boxes for each tracking trajectory in the database. Based on the detection box matching results, the tracking trajectory corresponding to the target detection box is determined. In this method, storing high-confidence historical detection boxes in the database reduces the interference of detection noise on detection box matching and improves matching accuracy. Furthermore, feature matching between historical detection boxes and target detection boxes enables cross-frame association, allowing trajectory tracking in complex scenarios such as differences in camera viewpoints, rapid movement, or trajectory interruptions caused by object occlusion, thereby improving the tracking effect of detected objects in complex scenarios.
[0030] In one possible implementation, the electronic device can classify detection boxes into high-scoring and low-scoring boxes based on their confidence scores. High-scoring boxes are the first detection boxes with confidence scores above a certain threshold, and low-scoring boxes are the second detection boxes with confidence scores below a certain threshold. This threshold can be set based on actual needs, and this application does not impose any restrictions on it. In this case, the electronic device can employ different matching strategies for high-scoring and low-scoring boxes. Figure 2 A flowchart of an object tracking method provided in another exemplary embodiment of this application is shown. This method can be executed by an electronic device, which can be implemented as a central node in the above architecture, such as... Figure 2 As shown, the method may include the following steps.
[0031] Step 210: Perform object detection on the target video frame to obtain at least one detection box.
[0032] In one possible implementation, when performing object detection on a target video frame, the electronic device can also determine the type of the detected object contained in the detection box through a visual detection model, and generate a corresponding type label for each detection box. For example, the type label can indicate whether the detected object is a human body, an animal, or a mobile device, so that after determining the tracking trajectory corresponding to the detection box and extracting the object feature vector, it can be associated and stored, thereby facilitating efficient retrieval or cross-video semantic analysis based on the detected object type.
[0033] Step 220: In response to determining a third detection box from a first detection box of at least one detection box, extract a third target tracking trajectory corresponding to a first predicted position, wherein the third detection box is a detection box that successfully matches the first predicted position in the target predicted position.
[0034] In this embodiment, the electronic device can classify detection boxes into high-scoring boxes and low-scoring boxes based on their confidence scores. High-scoring boxes are first detection boxes with confidence scores higher than a threshold, and low-scoring boxes are second detection boxes with confidence scores lower than the threshold. When matching at least one detection box with a target predicted location, the electronic device can employ a hierarchical dynamic matching mechanism. This involves first matching the first detection box within the at least one detection box with the target predicted location, and then matching the second detection box with a predicted location within the target predicted location that did not successfully match the first detection box.
[0035] Specifically, for any first detection box in at least one detection box, during matching, the electronic device can calculate the matching degree between the first detection box and each predicted position indicated by the target predicted position, and determine the matching result of the first detection box according to the matching judgment threshold set for the first detection box. The matching result of the first detection box includes the matching result between the first detection box and each predicted position, such as the matching score or matching distance, so as to determine the tracking trajectory corresponding to the first detection box.
[0036] In one possible implementation, the electronic device can perform Intersection over Union (IOU) matching between the first detection box and each predicted location. IOU matching measures the degree of overlap between two bounding boxes. Schematably, the formula for calculating IOU matching can be expressed as: Where A∩B represents the intersection area of the two bounding boxes, and A∪B represents the union area of the two bounding boxes.
[0037] Electronic devices can determine the matching result by setting an intersection-union ratio (IU) threshold. If the IU between the first detection box and the bounding box corresponding to the predicted position is greater than the IU threshold, the corresponding matching result is determined to be a successful match. Conversely, if the IU between the first detection box and the bounding box corresponding to the predicted position is less than the IU threshold, the corresponding matching result is determined to be a failed match. The value of the IU threshold can be set based on actual needs. The higher the value of the IU threshold, the higher the matching accuracy.
[0038] In another possible implementation, the electronic device can also match the first detection box with each predicted position using Mahalanobis distance. The Mahalanobis distance between the first detection box and the predicted position is calculated. The smaller the Mahalanobis distance, the higher the matching probability between the detection box and the predicted position. The larger the Mahalanobis distance, the lower the matching probability between the detection box and the predicted position. The electronic device can determine the matching result by setting a Mahalanobis distance threshold corresponding to the first detection box.
[0039] In practical applications, electronic devices can select any feasible matching method to match the first detection box with the predicted position, and this application does not impose any restrictions on this.
[0040] For at least one detection box that successfully matches any predicted position in the target predicted position, taking the third detection box as an example, if the number of predicted positions that successfully match the third detection box is 1, the electronic device can determine the predicted position as the first predicted position corresponding to the third detection box.
[0041] If there are multiple predicted positions that successfully match the third detection box, the electronic device can determine the one with the highest matching degree (e.g., the highest intersection-union ratio or the smallest Mahalanobis distance) among the multiple predicted positions as the first predicted position corresponding to the third detection box. Alternatively, it can feed back the multiple predicted positions to relevant personnel for confirmation.
[0042] In this embodiment of the application, the electronic device also needs to maintain the object feature vector in the database so as to facilitate subsequent tracking trajectory determination or object retrieval based on the object feature vector; therefore, in response to determining a third detection box from a first detection box of at least one detection box, the third object feature vector of the detected object contained in the third detection box is extracted, and the third detection box is a detection box that successfully matches the first predicted position in the target predicted position; The feature vector of the third object is associated with the tracking trajectory of the third target corresponding to the first predicted position and stored in the database.
[0043] In this database, a limited number of object feature vectors are dynamically stored for each tracking trajectory. Therefore, the object feature vectors for each tracking trajectory in the database are dynamically updated. Taking the storage of object feature vectors for the third target tracking trajectory as an example, in response to the number of object feature vectors for the third target tracking trajectory in the database reaching the number threshold, the object feature vector with the longest storage time corresponding to the third target tracking trajectory is removed from the database, and then the third object feature vector is associated with the third target tracking trajectory and stored in the database.
[0044] In one possible implementation, the electronic device can maintain a feature vector queue or feature vector table with limited capacity for each tracking trajectory, and dynamically update it through a sliding window mechanism: for a feature vector queue or feature vector table, when a new object feature vector is added, the object feature vector with the earliest storage time is removed; while recording the object feature vector of each tracking trajectory, the database can also record the update time, and the trajectory image of the tracking trajectory in each video frame, or the storage address of the trajectory image corresponding to each video frame. Illustratively, Table 1 shows the field information of the feature vector queue or feature vector table stored in the database provided in an exemplary embodiment of this application.
[0045] Table 1
[0046] As shown in Table 1, the feature vector queue or feature vector table of the tracking trajectory may contain information such as the tracking trajectory identifier, the object feature vector corresponding to the tracking trajectory, and the storage path of the trajectory image corresponding to the tracking trajectory. Furthermore, it may also contain the update time information of each object feature vector.
[0047] Due to differences in the installation location and shooting angle of visual sensing devices, the object features of the same detected object collected by different visual sensing devices may be the same or different. To avoid data redundancy when storing object feature vectors, electronic devices can selectively store object feature vectors. Taking the association storage process of a third feature vector and a third target tracking trajectory as an example, in this process, the electronic device can calculate the similarity between the third object feature vector and the object feature vectors already stored in the database for the third target tracking trajectory. If the similarity between the third object feature vector and each object feature vector is less than the corresponding similarity threshold, the third object feature vector is stored in the feature queue or object feature vector set corresponding to the third target tracking trajectory. If the database contains object feature vectors with a similarity higher than the similarity threshold, no association storage operation is performed. The similarity calculation method can include, but is not limited to, cosine similarity, Euclidean distance, Mahalanobis distance, structural similarity index, and perceptual hashing, etc., and this application does not impose any restrictions on this.
[0048] Step 230: Update the position information of the detected object in the target video frame corresponding to the third target tracking trajectory according to the third detection box.
[0049] After determining the first predicted position that matches the third detection box, since the position indicated by the third detection box is more accurate than the first predicted position, the electronic device can continue the third target tracking trajectory corresponding to the first predicted position matched by the third detection box. That is, the position information indicated by the third detection box in the target video frame is determined as the position information of the detected object corresponding to the third target tracking trajectory in the target video frame. In one possible implementation, the electronic device can obtain the tracking identifier of the third target tracking trajectory and assign the tracking identifier to the third detection box, thereby associating the third detection box with the third target tracking trajectory to achieve the continuation of the third target tracking trajectory and ensure the continuity of the tracking trajectory.
[0050] Furthermore, in order to improve the accuracy of tracking trajectory prediction, after the third detection box successfully matches the first predicted position, the electronic device can update the Kalman filter based on the third detection box and the first predicted position, so that the predicted position of the Kalman filter is as close as possible to the actual position indicated by the detection box.
[0051] Step 240: Match the second detection box in at least one detection box with the predicted position of the remaining tracking trajectory in the target video frame to obtain the matching result of the second detection box; the second detection box is the detection box with a confidence score lower than the score threshold, and the remaining tracking trajectory is the tracking trajectory in the database that has not been successfully matched with the first detection box in at least one detection box.
[0052] When matching any second detection box (i.e., low-scoring box) in at least one detection box, the electronic device can calculate the matching degree between the second detection box and each predicted position in the target video frame of each tracking trajectory that did not match the first detection box, such as performing IOU matching or Mahalanobis distance matching, and determine the matching result of the second detection box according to the matching judgment threshold set for the second detection box. The matching result of the second detection box includes the matching result of the second detection box with the predicted position of each remaining tracking trajectory.
[0053] The matching threshold corresponding to the second detection box can be the same as or different from the matching threshold corresponding to the first detection box. For example, since the feature information in the second detection box is relatively small, the value of the matching threshold corresponding to the second detection box can be set to be smaller than the matching threshold of the first detection box. The matching process between the second detection box and the predicted position of the remaining tracking trajectory in the target video frame can refer to the matching process between the first detection box and each predicted position in step 220, which will not be repeated here.
[0054] Step 250: In response to the matching result indication of the second detection box, the fourth detection box in the second detection box is successfully matched with the second predicted position in the target predicted position, and the fourth target tracking trajectory corresponding to the second predicted position is extracted.
[0055] After determining the second predicted position that matches the fourth detection box, it can be determined that the fourth detection box may contain the fourth target tracking trajectory corresponding to the second predicted position. Therefore, the fourth target tracking trajectory corresponding to the second predicted position can be extracted to update the fourth target tracking trajectory based on the second detection box.
[0056] Step 260: Update the position information of the detected object in the target video frame corresponding to the fourth target tracking trajectory according to the fourth detection box.
[0057] Since the position indicated by the fourth detection box is more accurate than the second predicted position, the electronic device can continue the fourth target tracking trajectory corresponding to the second predicted position matched by the fourth detection box. That is, the position information indicated by the fourth detection box in the target video frame is determined as the position information of the detected object corresponding to the fourth target tracking trajectory in the target video frame. In one possible implementation, the electronic device can obtain the tracking identifier of the fourth target tracking trajectory and assign the tracking identifier to the fourth detection box, thereby associating the fourth detection box with the fourth target tracking trajectory to achieve the continuation of the fourth target tracking trajectory and ensure the continuity of the tracking trajectory.
[0058] Furthermore, if the fourth detection box successfully matches the second predicted position, the electronic device can also update the Kalman filter based on the fourth detection box and the second predicted position, so that the predicted position of the Kalman filter is as close as possible to the actual position indicated by the detection box.
[0059] Step 270: In response to at least one detection box containing a target detection box that does not match the target prediction position, the target detection box is matched with the first detection boxes with known tracking trajectories stored in the database to obtain the detection box matching result, and the target prediction position is the predicted position of each tracking trajectory in the target video frame.
[0060] Specifically, when performing feature matching between the target detection box and each first detection box in the database, the electronic device can perform feature matching through similarity calculation. This process can be implemented as follows: Extract the first object feature vector of the detected object contained within the target detection box; Calculate the similarity between the feature vector of the first object and the feature vectors of each second object, wherein the feature vectors of the second objects are the feature vectors of the detected objects contained in each first detection box in the database; The detection box matching result is obtained based on the similarity between the feature vector of the first object and the feature vectors of each second object.
[0061] In this embodiment, the electronic device can extract the object feature vector of the detected object within the detection box through a feature extraction model. This feature extraction model can be implemented as a ReID (Re-identification) model, or it can be implemented as other feature extraction models, such as visual feature models, multimodal fusion models, etc. In this embodiment, the feature extraction model is deployed in edge nodes distributed in various rooms (or areas). The object feature vectors stored in the database are feature vectors of detected objects with known tracking trajectories extracted by the feature extraction model.
[0062] When matching the feature vector of the first object with the feature vectors of each second object in the database, the electronic device can calculate the similarity between the vectors using a vector similarity calculation algorithm to determine the vector matching result. The vector similarity calculation algorithm can be one of the following: cosine similarity, Euclidean distance, Mahalanobis distance, structural similarity index, and perceptual hash, etc. The electronic device can determine whether the vectors match based on a pre-set similarity threshold. If the similarity between the object feature vectors is greater than the similarity threshold, the detection box matching result indicates that the two object feature vectors match. If the similarity between the object feature vectors is less than or equal to the similarity threshold, the detection box matching result indicates that the two object feature vectors do not match.
[0063] Step 280: Update the tracking trajectory of each detected object based on the detection box matching results.
[0064] The detection box matching result can indicate whether the target detection box matched successfully or failed to match.
[0065] In the case of a successful match, in response to the detection box matching result indicating that the target detection box is successfully matched with the first target detection box in the database, the first target tracking trajectory corresponding to the first target detection box is extracted; Update the position information of the detected object in the target video frame corresponding to the first target tracking trajectory based on the target detection box.
[0066] In other words, if a detection box in at least one detection box fails to match the predicted position of each tracking trajectory in the target video frame, but successfully matches the first detection box of each tracking trajectory stored in the database, it indicates that the detection object corresponding to the detection box may be a detection object with a detection interruption. Taking the target detection box as an example, the first target tracking trajectory is updated based on the target detection box, and the position information indicated by the target detection box is determined as the position information of the detection object corresponding to the first target tracking trajectory in the target video frame. This can achieve the continuation of the first target tracking trajectory, realize object tracking across video frames, and effectively avoid the problem of detection object loss caused by cross-camera, occlusion, fast movement, etc.
[0067] In one possible implementation, to further improve the accuracy of tracking trajectory matching based on object feature vectors, the electronic device can perform feature matching between the first detection box in the target detection box and the first detection boxes with known tracking trajectories stored in the database, so as to update the tracking trajectory of each detected object according to the matching result. That is, for high-scoring boxes, if they do not match the predicted position of each tracking trajectory in the target video frame, they are further matched with the first detection boxes with known tracking trajectories in the database to determine the tracking trajectory corresponding to the high-scoring box.
[0068] In the event of a failed match, in response to the detection box matching result indicating that the target detection box does not match any of the first detection boxes in the database and that the target detection box is a first detection box, a second target tracking trajectory corresponding to the new detection object is created based on the target detection box.
[0069] In other words, if a high-resolution bounding box in at least one detection box fails to match the predicted position of each tracking trajectory in the target video frame, and also fails to match the first detection box of each tracking trajectory stored in the database, it indicates that the object in the detection box may be a newly appearing object. In this case, the electronic device can create a new tracking trajectory corresponding to the object and assign a corresponding tracking trajectory identifier.
[0070] Furthermore, for the tracking trajectory newly created based on the high-resolution bounding box, the electronic device can store the corresponding object features to facilitate subsequent object feature vector matching or object querying. This process can be implemented as follows: After creating a second target tracking trajectory corresponding to the new detected object based on the target detection box, the feature vector of the first object corresponding to the target detection box is associated with the second target tracking trajectory and stored in the database.
[0071] Furthermore, for tracking trajectories that do not match detection boxes, the electronic device can retain N video frames. This means matching the predicted position of the tracking trajectory with detection boxes in N video frames. If the fifth target tracking trajectory fails to match with detection boxes in N consecutive target video frames, the fifth target tracking trajectory is deleted. N is a positive integer. In other words, if the fifth target tracking trajectory still does not match a detection box within N video frames, it is determined that the detected object has left the detection area, and the fifth target tracking trajectory is deleted; subsequent matching based on the fifth target tracking trajectory will not be performed. Conversely, if the fifth target tracking trajectory matches a corresponding detection box within N video frames, the Kalman filter is updated, and the trajectory identifier of the fifth target tracking trajectory is assigned to that detection box, thus continuing the fifth target tracking trajectory.
[0072] In one possible implementation, the electronic device can also provide an object query service based on object feature vectors stored in a database. This process can be implemented as follows: In response to the query information of the target detection object, a feature vector query is performed in the database based on the query feature vector of the query information to obtain the object query result of the target detection object; the object query result includes at least one of the following: image information of the target detection object, activity time information, and tracking trajectory information; Provide the user with the results of the object query.
[0073] Users can input query information based on the interactive interface provided by the electronic device to query the image or trajectory information of a specified object. This interactive interface can be implemented as a user interface, which can be the interface of a smart furniture mobile application or web platform. Users input query information through this user interface, and the system returns the object query results through this interface.
[0074] The query information can be text or image information. Electronic devices can use a cross-modal large model to map the query information and object feature vectors into a unified space. By calculating the similarity between the query feature vector and the object feature vector in the database, the corresponding query object is determined, and the query result is returned to the user. When the query information is text, text-based image search can be implemented. The query result can include the image or image path of the detected object, the appearance time of the detected object, and further, the tracking trajectory of the detected object. This cross-modal large model can be implemented as an LLaVA (Large Language and Vision Assistant) model, or as a BLIP (Bootstrapped Language-Image Pre-training) model, or a Kosmos model, etc., to retrieve relevant image or trajectory information based on text descriptions, thus realizing the "text-based image search" function.
[0075] For example, if the query is "search for people wearing red clothes", the query will be input into a cross-modal big data model for processing. The cross-modal big data model converts the query into a high-order vector, namely the query feature vector, which contains the semantic information of the query. The similarity between the query feature vector and the object feature vectors of each tracking trajectory stored in the database is calculated. The electronic device finds the most matching image and its related information (such as tracking trajectory and timestamp) based on the similarity score, and feeds these query results back to the user to help the user quickly locate and obtain the information they need.
[0076] In summary, the FTTR-based object tracking method provided in this application stores high-confidence historical detection boxes for each tracking trajectory in a database. If at least one detection box contains a target detection box whose predicted position is not matched to any tracking trajectory, feature matching is performed between this target detection box and the high-confidence historical detection boxes for each tracking trajectory in the database. Based on the detection box matching results, the tracking trajectory corresponding to the target detection box is determined. In this method, storing high-confidence historical detection boxes in the database reduces the interference of detection noise on detection box matching and improves matching accuracy. Furthermore, feature matching between historical detection boxes and target detection boxes enables cross-frame association, allowing trajectory tracking in complex scenarios such as differences in camera viewpoints, rapid movement, or trajectory interruptions caused by object occlusion, thereby improving the tracking effect of detected objects in complex scenarios.
[0077] Furthermore, by stratifying the detection boxes according to their confidence scores and designing a two-stage matching strategy, the information of the detection boxes can be fully utilized. Specifically, by filtering detection boxes with confidence scores higher than the score threshold and matching them with the predicted positions, false matches can be effectively reduced. The tracking trajectory is further updated by combining the matching results, which enhances the robustness of the system under interference such as occlusion and noise. By matching and associating low-scoring boxes with the remaining tracking trajectories, the interruption of the tracking trajectory can be reduced, or the generation of redundant trajectories can be reduced, and the frequent switching of object identity can be reduced.
[0078] Taking the matching of object feature vectors in high-resolution bounding boxes at unmatched target prediction locations as an example, Figure 3 The diagram illustrates an exemplary embodiment of the object tracking method provided in this application, such as... Figure 3 As shown, the process may include the following steps.
[0079] S301, a vision sensor that captures video frames.
[0080] S302, Object Detection.
[0081] S303, based on the confidence score of the detection box, divide the detection boxes into high-scoring boxes and low-scoring boxes.
[0082] S304 uses a Kalman filter to predict the predicted position of each tracking trajectory in the current frame.
[0083] S305 uses high-resolution bounding boxes to perform IOU matching with the predicted locations of the tracking trajectory.
[0084] S306, determine if the match is successful. If yes, execute S307; otherwise, execute 311.
[0085] S307, update the Kalman filter.
[0086] S308 extracts the feature vector of the object within the detection box through a feature extraction model.
[0087] S309, store the extracted object feature vector.
[0088] S310 outputs the tracking trajectory of the current frame.
[0089] S311 uses the predicted locations of the low-resolution bounding box and the remaining tracking trajectory for IOU matching.
[0090] S312, determine if the match is successful. If yes, execute S313; otherwise, execute S314.
[0091] S313, update the Kalman filter.
[0092] S314: For the tracking trajectory that does not match the detection box, match it with the detection boxes of N consecutive video frames.
[0093] S315. For high-resolution bounding boxes that do not match the tracking trajectory, the feature vectors of the objects within the detection box are extracted using a feature extraction model and matched with the object feature vectors already stored in the database.
[0094] S316: Determine if a match is successful. If yes, execute S317; otherwise, execute S318.
[0095] S317: Extract the tracking trajectory identifier from the database and assign it to the current detection box.
[0096] S318, create a new tracking track.
[0097] S319, update the Kalman filter.
[0098] Taking human body testing as an example, Figure 4 An architectural diagram of an exemplary embodiment of this application provides an object tracing method based on FTTR, as shown below. Figure 4As shown, in the smart home scenario of FTTR, cameras or other types of visual sensors are installed in each room to acquire indoor image data in real time. The video frames from the acquired indoor image data are input into a visual detection model in the CV (Computer Vision) model library for human detection, resulting in human detection boxes. Based on the detection box type (high-resolution or low-resolution), the predicted position of each tracking trajectory in the current frame by the motion model, the object feature vector extracted by the feature extraction model, and the known high-resolution boxes of each tracking trajectory stored in the database, human tracking is performed to determine the tracking trajectory corresponding to the human in the current video frame. Furthermore, if the human detection box is a high-resolution box, the object feature vector of the human detection box is associated with the tracking trajectory and stored in the database. The above process of human tracking and object feature vector storage can be referenced as follows: Figure 1 or Figure 2 The relevant details of the illustrated embodiments will not be repeated here.
[0099] When making a query, users can input the query information into the cross-modal big model for cross-modal search. The cross-modal big model performs feature vector queries in the database based on the object query information, and can obtain the corresponding human body query results, which are then fed back to the user.
[0100] Figure 5 This diagram illustrates a structural block diagram of an electronic device 500 according to an exemplary embodiment of this application. This electronic device can be implemented as the central node in the above-described scheme of this application. The electronic device 500 includes a Central Processing Unit (CPU) 501, a system memory 504 including Random Access Memory (RAM) 502 and Read-Only Memory (ROM) 503, and a system bus 505 connecting the system memory 504 and the CPU 501. The electronic device 500 also includes a mass storage device 506 for storing an operating system 509, application programs 510, and other program modules 511. The system memory 504 and the mass storage device 506 can be collectively referred to as memory.
[0101] According to various embodiments of this application, the electronic device 500 can also be connected to a remote computer on a network, such as the Internet. That is, the electronic device 500 can be connected to a network 508 via a network interface unit 507 connected to the system bus 505, or the network interface unit 507 can be used to connect to other types of networks or remote computer systems (not shown).
[0102] The memory also includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 501 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the FTTR-based object tracking method shown in the above embodiments.
[0103] Figure 6 A structural block diagram of an electronic device 600 is shown in another exemplary embodiment of this application. The electronic device 600 can be implemented as the central node described above. For example, the electronic device can be an Android terminal device; typically, the electronic device 600 includes a processor 601 and a memory 602. The memory 602 may include one or more computer-readable storage media for storing at least one instruction, which is executed by the processor 601 to implement all or part of the steps in the FTTR-based object tracking method shown in the method embodiments of this application.
[0104] In some embodiments, the electronic device 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608. In some embodiments, the electronic device 600 also includes one or more sensors 609. These one or more sensors 609 include, but are not limited to, an accelerometer 610, a gyroscope 611, a pressure sensor 612, an optical sensor 613, and a proximity sensor 614. Those skilled in the art will understand that... Figure 6 The structure shown does not constitute a limitation on the electronic device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0105] In addition, to ensure real-time image processing and data storage, the electronic device in this application embodiment has the support of a central processing unit or a graphics processing unit (GPU) to accelerate visual detection, target tracking and feature extraction based on deep learning models (such as YOLO, Fast R-CNN, etc.); the memory is used to store tracking data and extracted feature vectors, and can be implemented as a database or a distributed storage system to facilitate the storage and management of a large amount of detection box data, feature vectors and tracking trajectory information.
[0106] In one exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement all or part of the steps in the FTTR-based object tracing method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.
[0107] In one exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the above-described actions. Figure 1 or Figure 2 All or part of the steps of the embodiments shown in any of the embodiments.
[0108] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0109] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. An object tracking method based on FTTR, characterized in that, The method includes: Object detection is performed on the target video frame to obtain at least one detection box; the target video frame is a video frame acquired by visual sensing devices deployed at the edge nodes of each region; the at least one detection box is divided into high-scoring boxes and low-scoring boxes based on the confidence score of the detection box, the high-scoring box is the first detection box whose confidence score is higher than the score threshold, and the low-scoring box is the second detection box whose confidence score is lower than the score threshold, the confidence score is used to indicate the probability that the detected object is contained at the position marked by the detection box; In response to the fact that the at least one detection box contains a target detection box that does not match the target prediction position, the target detection box is matched with the first detection boxes with known tracking trajectories stored in the database to obtain a detection box matching result, wherein the target prediction position is the predicted position of each tracking trajectory in the target video frame; The tracking trajectory of each detected object is updated based on the detection box matching results; The method further includes: In response to determining a third detection box from the first detection box of the at least one detection box, a third target tracking trajectory corresponding to the first predicted position is extracted, wherein the third detection box is a detection box that successfully matches the first predicted position in the target predicted position; Update the position information of the detected object in the target video frame corresponding to the third target tracking trajectory according to the third detection box; The second detection box in the at least one detection box is matched with the predicted position of the remaining tracking trajectory in the target video frame to obtain the matching result of the second detection box; the remaining tracking trajectory is the tracking trajectory in the database that has not been successfully matched with the first detection box in the at least one detection box.
2. The method according to claim 1, characterized in that, The step of performing feature matching between the target detection box and each first detection box with a known tracking trajectory stored in the database to obtain the detection box matching result includes: Extract the first object feature vector of the detected object contained within the target detection box; Calculate the similarity between the first object feature vector and each second object feature vector, where the second object feature vector is the feature vector of each detected object contained in the first detection box in the database; The detection box matching result is obtained based on the similarity between the feature vector of the first object and the feature vectors of each of the second objects.
3. The method according to claim 1 or 2, characterized in that, The step of updating the tracking trajectory of each detected object based on the detection box matching result includes: In response to the detection box matching result indicating that the target detection box is successfully matched with the target first detection box in the database, the first target tracking trajectory corresponding to the target first detection box is extracted; Update the position information of the detected object corresponding to the first target tracking trajectory in the target video frame according to the target detection box.
4. The method according to claim 1 or 2, characterized in that, The step of updating the tracking trajectory of each detected object based on the detection box matching result includes: In response to the detection box matching result indicating that the target detection box does not match any of the first detection boxes in the database, and that the target detection box is a first detection box, a second target tracking trajectory corresponding to the new detection object is created based on the target detection box.
5. The method according to claim 4, characterized in that, After creating a second target tracking trajectory corresponding to the new detected object based on the target detection box, the method further includes: The first object feature vector corresponding to the target detection box is associated with the second target tracking trajectory and stored in the database.
6. The method according to claim 1, characterized in that, After performing object detection on the target video frame to obtain at least one detection box, the method further includes: In response to determining a third detection box from the first detection box of the at least one detection box, a third object feature vector of the detected object contained in the third detection box is extracted; the third detection box is a detection box that successfully matches the first predicted position in the target predicted position; The feature vector of the third object is associated with the tracking trajectory of the third target corresponding to the first predicted position and stored in the database.
7. The method according to claim 6, characterized in that, The step of associating and storing the feature vector of the third object with the tracking trajectory of the third target corresponding to the first predicted position in the database includes: In response to the number of object feature vectors of the third target tracking trajectory in the database reaching a threshold, the object feature vector with the longest storage time corresponding to the third target tracking trajectory is removed from the database, and then the third object feature vector is associated with the third target tracking trajectory and stored in the database.
8. The method according to claim 1, characterized in that, The method further includes: In response to the matching result of the second detection box indicating that the fourth detection box in the second detection box is successfully matched with the second predicted position in the target predicted position, the fourth target tracking trajectory corresponding to the second predicted position is extracted; The position information of the detected object corresponding to the fourth target tracking trajectory in the target video frame is updated according to the fourth detection box.
9. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the fifth target tracking trajectory in each tracking trajectory fails to match the detection boxes of N consecutive target video frames, the fifth target tracking trajectory is deleted, where N is a positive integer.
10. The method according to claim 1, characterized in that, The method further includes: In response to the query information of the target detection object, a feature vector query is performed in the database based on the query feature vector of the query information to obtain the object query result of the target detection object; the object query result includes at least one of the following: image information, activity time information and tracking trajectory information of the target detection object; The query results for the object are then fed back to the user.
11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to implement the FTTR-based object tracking method as described in any one of claims 1 to 10.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the FTTR-based object tracking method as described in any one of claims 1 to 10.
13. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the FTTR-based object tracking method as described in any one of claims 1 to 10.