A target tracking method and device of a smart device and the smart device

Abstract

The application discloses a target tracking method and device of a smart device and the smart device, relates to the technical field of smart devices, and the posture rotation information of the smart device is collected in real time in the process that the smart device continuously collects images, according to which, when the body of the smart device rotates greatly, the depth consistency penalty value and the appearance consistency penalty value are calculated based on the depth measured values and the appearance features of the current frame target and the historical frame target, and the target detection result of the current frame is analyzed by fusing the depth consistency penalty value and the appearance consistency penalty value, and the target tracking result of the current frame is output, so that the target loss and the false matching problem under the violent motion of the smart device are solved.

Description

Technical Field

[0001] This application relates to the field of smart devices, and more particularly to a target tracking method, apparatus, and smart device for a smart device. Background Technology

[0002] Intelligent following devices and intelligent wearable devices (such as smart glasses) have shown broad application prospects in many fields such as industrial inspection, medical assistance, navigation and guidance, and social interaction. Among them, target detection and tracking are the core visual functions that are indispensable for such devices, and their real-time performance and accuracy directly affect the user experience.

[0003] In practical applications, when intelligent following devices and smart glasses perform target tracking tasks, they will rotate their own posture as the target moves. For example, the body of an intelligent following robot rotates, or the smart glasses rotate due to the head rotation of the wearer. This kind of rotation will directly cause the camera's field of view to shift rapidly, resulting in drastic changes in the camera's field of view between adjacent video frames. Consequently, the displacement of the target on the image plane can reach hundreds of pixels.

[0004] Traditional target tracking algorithms (such as SORT, DeepSORT, ByteTrack, etc.) are generally designed based on a constant velocity motion model or linear motion assumption. The core premise of this assumption is that the displacement of the target between adjacent frames is small and the motion state is stable. When scenarios such as rapid head rotation occur, the above assumptions are severely broken, directly causing the motion prediction module of the algorithm to fail, making it impossible to accurately predict the target's motion trajectory, and ultimately affecting the real-time performance and accuracy of tracking. Summary of the Invention

[0005] This application provides a target tracking method, apparatus, and smart device for intelligent devices, in order to solve the problem of inaccurate target tracking and easy loss of target due to drastic changes in camera perspective caused by the rotation of the device itself during the target tracking process.

[0006] In a first aspect, this application provides a target tracking method for a smart device, comprising: During the continuous image acquisition process of the smart device, the real-time attitude and rotation information of the smart device is obtained. When the attitude rotation information satisfies the high-speed rotation condition, the measured depth value and appearance features of the target in the current frame are obtained based on the target image of the current frame; Based on the measured depth values ​​and appearance features of the target in the current frame and the targets in historical frames, the depth consistency value and appearance consistency value are calculated accordingly. The depth consistency value and appearance consistency value are then fused to analyze the target detection results of the current frame and output the target tracking results of the current frame.

[0007] In an optional implementation of the first aspect, obtaining the measured depth value and appearance features of the target in the current frame based on the target image of the current frame includes: A lightweight monocular depth estimation network is used to perform pixel-level depth processing on the target image of the current frame to obtain a depth map; Target detection is performed on the depth map to obtain target detection boxes; Extract the depth value of the target region in the target detection box to obtain the measured depth value of the target in the current frame; Image features are extracted from the target detection box to obtain the appearance features of the target in the current frame.

[0008] In one optional implementation of the first aspect, the step of calculating depth consistency value and appearance consistency value based on the measured depth values ​​and appearance features of the target in the current frame and the target in historical frames, and then analyzing the target detection results of the current frame by fusing the depth consistency value and appearance consistency value, and outputting the target tracking results of the current frame, includes: Obtain the appearance features of targets in historical frames, and calculate the similarity between the appearance features of targets in the current frame and the appearance features of targets in historical frames to obtain the appearance consistency cost. Obtain historical target trajectory information, and predict the depth value of the target in the current frame based on the historical target trajectory information; The consistency value of the measured depth and the predicted depth of the target in the current frame is calculated to obtain the depth consistency cost. The total association value is obtained by weighted fusion of the appearance consistency value and the depth consistency value. Based on the total correlation value analysis, the matching degree between the target detection result of the current frame and the historical target tracking trajectory is analyzed, and the target tracking result of the current frame is output.

[0009] In an alternative embodiment of the first aspect, the attitude rotation information includes the rotational angular velocity of the smart device, and the method further includes: The weighting coefficients corresponding to the appearance consistency value and the depth consistency value are dynamically adjusted based on the real-time rotational angular velocity. The weighting coefficient corresponding to the appearance consistency value is negatively correlated with the rotational angular velocity, and the weighting coefficient corresponding to the depth consistency value is positively correlated with the rotational angular velocity.

[0010] In an optional implementation of the first aspect, the step of predicting the depth prediction value of the target in the current frame based on the historical target trajectory information includes: Based on the depth time series data and motion state of historical target trajectories, the initial predicted depth of the target in the current frame is deduced. An adaptive temporal depth filtering method is used to weight and fuse the measured depth value of the current frame with the initial predicted depth to obtain the predicted depth value of the current frame.

[0011] In one alternative embodiment of the first aspect, the filter gain in the adaptive temporal depth filtering is dynamically adjusted based on the depth measurement noise error and the equipment rotation error.

[0012] In an alternative embodiment of the first aspect, the method further includes: When the attitude rotation information meets the low-speed rotation condition, the system switches to normal tracking mode, where the normal tracking mode does not integrate the device's attitude rotation information and the target's depth information for target tracking.

[0013] Secondly, this application provides a target tracking device, comprising: The rotation acquisition module is used to acquire real-time attitude rotation information of the smart device during the continuous image acquisition process. The depth acquisition module is used to acquire the measured depth value and appearance features of the target in the current frame based on the target image in the current frame, provided that the attitude rotation information meets the high-speed rotation condition. The tracking module is used to calculate the depth consistency value and appearance consistency value based on the measured depth value and appearance features of the target in the current frame and the target in historical frames. It then integrates the depth consistency value and appearance consistency value to analyze the target detection results of the current frame and outputs the target tracking results of the current frame.

[0014] Thirdly, this application provides a smart device, including a smart device body, wherein the smart device body is provided with an image acquisition unit, a rotation detection unit and a target tracking device, wherein the image acquisition unit is used to acquire images, the rotation detection unit is used to detect the real-time attitude rotation information of the smart device, the image acquisition unit and the rotation detection unit are communicatively connected to the target tracking device, and the target tracking device is used to execute the target tracking method of the smart device provided in the first aspect.

[0015] In one possible implementation of the third aspect, the smart device is smart glasses, a smart follow device, or a smart helmet.

[0016] In one possible implementation of the third aspect, the image acquisition unit is a monocular camera.

[0017] In one possible implementation of the third aspect, the rotation detection unit is an IMU sensor.

[0018] The target tracking method, apparatus, and smart device provided in this application have at least the following beneficial effects: The image acquisition unit continuously acquires images, and the rotation detection unit collects the attitude and rotation information of the smart device in real time. Based on this, when the smart device rotates significantly, the depth consistency value and appearance consistency value are calculated based on the measured depth value and appearance features of the target in the current frame and the target in historical frames. The depth consistency value and appearance consistency value are then combined to analyze the target detection results of the current frame and output the target tracking results of the current frame to solve the problem of target loss and mismatch under the violent movement of the smart device. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A flowchart illustrating the target tracking method for the intelligent device provided in this application; Figure 2 This is a schematic diagram of the structure of smart glasses provided in an embodiment of this application. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] As mentioned in the background section, intelligent tracking devices and smart glasses, when performing target tracking tasks, will rotate their own posture as the target moves, resulting in rapid changes in the camera's perspective. This characteristic poses a challenge to traditional target tracking algorithms. 1. Excessive inter-frame shift

[0023] When the device itself rotates rapidly or moves rapidly with a person's head, the perspective changes drastically between adjacent video frames, and the target's displacement on the image plane can reach hundreds of pixels. Traditional tracking algorithms (such as SORT, DeepSORT, ByteTrack, etc.) are usually based on constant velocity motion models or linear motion assumptions, assuming that the target displacement between adjacent frames is small. When the head rotates rapidly, this assumption is severely violated, causing motion prediction to fail.

[0024] 2. Degradation of appearance features

[0025] Rapid changes in perspective can cause significant changes in the appearance of a target. Current tracking algorithms rely on appearance features extracted by deep learning to identify targets. When appearance features degrade due to drastic changes in perspective, the accuracy of feature matching drops sharply.

[0026] 3. Limited computing resources

[0027] As edge devices, smart glasses are subject to strict limitations in terms of computing resources and storage space.

[0028] To address at least one of the above technical problems, this application provides a target tracking method, apparatus, and intelligent device for intelligent devices. The intelligent device is equipped with an image acquisition unit and a rotation detection unit. During the process of the intelligent device following a moving target, the intelligent device continuously acquires images through the image acquisition unit and acquires the attitude rotation information of the intelligent device in real time through the rotation detection unit. Accordingly, when the intelligent device rotates significantly, the target depth information in the target image is combined with motion perception to compensate for target tracking, thereby solving the problems of target loss and mismatch under the vigorous movement of the intelligent device.

[0029] The target tracking method of the intelligent device according to the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the target tracking method of the smart device in this application embodiment includes: S10 acquires real-time attitude and rotation information of the smart device during the continuous image acquisition process.

[0031] S20: When the attitude rotation information meets the high-speed rotation condition, the measured depth value and appearance features of the target in the current frame are obtained based on the target image of the current frame.

[0032] S30: Based on the measured depth values ​​and appearance features of the target in the current frame and the target in historical frames, calculate the depth consistency value and appearance consistency value, and analyze the target detection results of the current frame by fusing the depth consistency value and appearance consistency value, and output the target tracking results of the current frame.

[0033] In this embodiment, the smart device is equipped with an image acquisition unit and a rotation detection unit. The image acquisition unit can be a monocular camera or a binocular camera. The rotation detection unit can be an IMU sensor.

[0034] During the tracking process, the smart device continuously acquires images through its built-in image acquisition unit and detects the real-time attitude and rotation information of the smart device, such as the device's rotational angular velocity, through its built-in rotation detection unit.

[0035] Then, the intelligent device makes a decision on the target tracking mode based on the attitude rotation information. When the attitude rotation information meets the low-speed rotation condition, the normal tracking mode is adopted. In the normal tracking mode, the attitude rotation information of the device and the depth information of the target are not fused for target tracking compensation. Only the traditional target detection algorithm is used for tracking, such as using yolov8n with a nano backbone (which can be replaced with any suitable mainstream lightweight target detection structure according to the embedded resources). Based on the target detection results of multiple consecutive frames, bytetrack is used to compensate for the target motion trajectory and give the target object identification and corrected position information.

[0036] If the attitude rotation information meets the high-speed rotation conditions, the depth compensation tracking mode is adopted. In this mode, the measured depth value and appearance features of the target in the current frame are first obtained, and then target tracking compensation is performed based on the measured depth value and appearance features.

[0037] Alternatively, assume the rotational angular velocity at time t is ω_t = [ω_x, ω_y, ω_z]. T (Unit: rad / s), Calculate the magnitude of angular velocity: .

[0038] like If the value is less than the threshold τ_ω, then the attitude rotation information is determined to meet the low-speed rotation condition. If the value is greater than or equal to τ_ω, then the attitude rotation information is determined to meet the high-speed rotation condition.

[0039] Optionally, the measured depth value and appearance features of the target in the current frame are obtained based on the target image in the current frame, including: A lightweight monocular depth estimation network is used to perform pixel-level depth processing on the target image of the current frame to obtain a depth map; Perform object detection on the depth map to obtain object detection boxes; Extract the depth value of the target region within the target detection box to obtain the measured depth value of the target in the current frame; Image features are extracted from the target detection box to obtain the appearance features of the target in the current frame.

[0040] The image feature extraction can employ existing image feature extraction algorithms / models, and this application does not limit this. Appearance features can include, but are not limited to, color features, texture features, image spectrum, image histogram, etc.

[0041] The lightweight monocular depth estimation network can adopt a network architecture with a UNet-like structure as the backbone. This application does not limit this. When a smart device is equipped with a binocular camera, a binocular depth estimation network can also be used to acquire depth information.

[0042] In this embodiment, the intelligent device can use the YOLO series of lightweight detection algorithms to perform target detection on the target image of the current frame, obtain a target detection box, and further extract the depth value of the central region of the target detection box as the measured depth value of the target in the current frame, so as to improve the accuracy and robustness of the depth information. Then, target tracking compensation is performed based on the measured depth value and appearance features of the target in the current frame.

[0043] Optionally, based on the measured depth values ​​and appearance features of the target in the current frame and the targets in historical frames, the depth consistency value and appearance consistency value are calculated accordingly. These two values ​​are then combined to analyze the target detection results of the current frame, and the target tracking results of the current frame are output. This can include: Obtain the appearance features of targets in historical frames, and calculate the similarity between the appearance features of targets in the current frame and the appearance features of targets in historical frames to obtain the appearance consistency cost. Obtain historical target trajectory information, and predict the depth value of the target in the current frame based on the historical target trajectory information; The consistency value of the measured depth and the predicted depth of the target in the current frame is calculated to obtain the depth consistency cost. The total associated value is obtained by weighted fusion of the value of appearance consistency and the value of depth consistency. The matching degree between the target detection result of the current frame and the historical target tracking trajectory is analyzed based on the total correlation value, and the target tracking result of the current frame is output.

[0044] Specifically, the embodiments of this application construct a total association cost function. Used to match the target detection bounding box in the current frame with the historical target tracking trajectory:

[0045] in, , , All are weighted coefficients. This represents the matching degree between the target detection box i in the current frame and the target detection box j in the historical frames, also referred to as the "total association cost" in this application. This represents the similarity between the appearance features of the target detection box i in the current frame and the appearance features of the target detection box j in a historical frame, also referred to in this application as the "appearance consistency cost". The depth consistency index represents the consistency between the measured depth value of the target detection box i in the current frame and the depth prediction value of the current frame obtained based on the depth information of the target detection box j in the historical frames. In this application, it is also referred to as the "depth consistency cost".

[0046] Optionally, in this embodiment of the application, the weighting coefficients corresponding to the appearance consistency value and the depth consistency value are dynamically adjusted based on the real-time rotational angular velocity; wherein, the weighting coefficient corresponding to the appearance consistency value is negatively correlated with the rotational angular velocity, and the weighting coefficient corresponding to the depth consistency value is positively correlated with the rotational angular velocity.

[0047] That is, when the equipment rotates violently... When increasing, decrease ,improve .

[0048] Understandably, when a device rotates violently, image blurring reduces the credibility of appearance features. By reducing... This reduces the influence of appearance factors on the matching results and can prevent false matches. Depth information is less affected by motion blur and has strong geometric constraints; as motion intensifies, the system automatically improves its performance. This makes the confidence of target association mainly depend on depth consistency, thereby locking the target in the event of violent shaking and ultimately improving the robustness of target tracking.

[0049] in, Used to measure the matching degree between the target detection bounding box in the current frame and the historical target tracking trajectory. The larger the value, the higher the confidence level of the current object detection result (i.e., the object detection box in the current frame). The smaller the value, the lower the confidence level of the current target detection result.

[0050] As an example, If the target detection box is greater than or equal to the preset confidence threshold, output the current target detection box; otherwise, discard the current target detection box and determine that the target tracking has failed.

[0051] Optionally, ,in, This represents the appearance features of the target detection box i in the current frame t (or the appearance features of the target at time t). This represents the appearance features of the target detection box j in the previous historical frame t-1 (or the appearance features of the target at time t-1).

[0052] The appearance features of the target in each time frame are extracted using a lightweight ReID network (such as OSNet-AIN). As an example, the appearance features are represented as a 256-dimensional appearance feature vector.

[0053] Optionally, ,in, This represents the measured depth of the target detection box i detected in the target image at time frame t. This represents the predicted depth value of the target detection box at time frame t, i.e., the predicted depth value of the target at time frame t. It is predicted based on the depth information and / or motion state of the target detection box j in historical frames (e.g., time frame t-1 and previous historical frames). To prevent division by zero constant, as an example, =0.01.

[0054] Optionally, predicting the depth of the target in the current frame based on historical target trajectory information includes the following steps: Based on the depth time series data and motion state of historical target trajectories, the initial predicted depth of the target in the current frame is deduced. An adaptive temporal depth filtering method is used to weight and fuse the measured depth value of the current frame with the initial predicted depth to obtain the predicted depth value of the current frame.

[0055] Optionally, the filtering gain of the above adaptive temporal depth filter is dynamically adjusted based on the depth measurement noise error and the equipment rotation error.

[0056] In this embodiment, considering the inter-frame noise in monocular depth estimation, direct use for matching would cause jitter. Therefore, an adaptive temporal depth filter is introduced to calculate the depth prediction value of the next frame based on the historical trajectory state. .

[0057] in, This represents the initial predicted depth of the target detection box j in the current frame t, obtained based on the depth information of the target detection boxes in historical frames (including time frame t-1 and previous historical frames). This represents the filter gain.

[0058] in, , This represents the depth measurement noise error, which can be obtained by measuring the variance / standard deviation of the measured depth values ​​within a preset sliding time window of a first length. This quantifies the random error and inter-frame jitter of the depth values ​​output by the lightweight monocular depth estimation network. The device rotation error can be obtained by measuring the variance / standard deviation of the device rotation angular velocity within a preset second-length sliding time window, thereby quantifying the degree of disturbance to the target motion state caused by the rapid rotation of the intelligent device.

[0059] As an example, , Where N and M are both positive integers. , Let represent the measured depth value of the target in the i-th frame and the standard deviation of the measured depth values ​​of the target in the most recent N frames, respectively. , represents the device rotation angular velocity detected in the i-th frame and the standard deviation of the rotation angular velocity in the most recent M frames, respectively.

[0060] This application embodiment also provides a target tracking device, the target tracking device comprising: The rotation acquisition module is used to acquire real-time attitude rotation information of the smart device during the continuous image acquisition process. The depth acquisition module is used to acquire the measured depth value and appearance features of the target in the current frame based on the target image in the current frame, provided that the attitude rotation information meets the high-speed rotation conditions. The tracking module is used to calculate the depth consistency value and appearance consistency value based on the measured depth value and appearance features of the target in the current frame and the target in historical frames. It then integrates the depth consistency value and appearance consistency value to analyze the target detection results of the current frame and outputs the target tracking results of the current frame.

[0061] It should be noted that the functions implemented by each module in the target tracking device of this application correspond one-to-one with the steps of the target tracking method of the intelligent device provided in the above embodiments, and will not be repeated here.

[0062] Thirdly, this application also provides an intelligent device, including an intelligent device body, on which an image acquisition unit, a rotation detection unit, and a target tracking device are provided. The image acquisition unit is used to acquire images, the rotation detection unit is used to detect the real-time attitude rotation information of the intelligent device, and the image acquisition unit, the rotation detection unit, and the target tracking device are connected to the target tracking device, which is used to execute the target tracking method of the intelligent device provided in the above embodiments.

[0063] Alternatively, the smart device may be smart glasses, such as Figure 2 As shown, the smart glasses 1 include an image acquisition unit 11, a rotation detection unit 12, and a target tracking device that is communicatively connected to the image acquisition unit 11 and the rotation detection unit 12.

[0064] In other embodiments, the smart device may also be a smart following device, a smart helmet, or other smart devices that integrate a visual perception system and have target detection and tracking functions.

[0065] Optionally, the image acquisition unit can be a monocular camera or a binocular camera.

[0066] Optionally, the rotation detection unit is an IMU sensor.

[0067] The target tracking device may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the target tracking method of the smart device provided in the above embodiments.

[0068] The processor can be a microcontroller unit (MCU) or a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0069] In some embodiments, the memory can be an internal storage unit of the smart device, such as the hard drive or RAM. In other embodiments, the memory can be an external storage device of the smart device, such as a plug-in hard drive, SmartMediaCard (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory can include both internal and external storage units of the smart device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0070] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Modules described in the embodiments of this application can be implemented in software or hardware. The names of modules do not, in some cases, constitute a limitation on the unit itself.

[0071] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A target tracking method for an intelligent device, characterized in that, include: During the continuous image acquisition process of the smart device, the real-time attitude and rotation information of the smart device is obtained. When the attitude rotation information satisfies the high-speed rotation condition, the measured depth value and appearance features of the target in the current frame are obtained based on the target image of the current frame; Based on the measured depth values ​​and appearance features of the target in the current frame and the target in historical frames, the depth consistency value and appearance consistency value are calculated accordingly. The depth consistency value and appearance consistency value are then combined to analyze the target detection results of the current frame and output the target tracking results of the current frame. The depth consistency cost refers to the consistency index between the measured depth value of the target in the current frame and the predicted depth value of the current frame based on historical target trajectory information, and the appearance consistency cost refers to the similarity between the appearance features of the target in the current frame and the appearance features of the target in historical frames. The method involves calculating depth consistency value and appearance consistency value based on the measured depth values ​​and appearance features of the target in the current frame and the targets in historical frames. These two values ​​are then combined to analyze the target detection results of the current frame, outputting the target tracking results for the current frame, including: Obtain the appearance features of targets in historical frames, and calculate the similarity between the appearance features of targets in the current frame and the appearance features of targets in historical frames to obtain the appearance consistency cost. Obtain historical target trajectory information, and predict the depth value of the target in the current frame based on the historical target trajectory information; The consistency value of the measured depth and the predicted depth of the target in the current frame is calculated to obtain the depth consistency cost. The appearance consistency value and the depth consistency value are weighted and fused to obtain the total correlation value. The attitude rotation information includes rotation angular velocity. The weighting coefficients corresponding to the appearance consistency value and the depth consistency value are dynamically adjusted based on the real-time rotation angular velocity. The weighting coefficient corresponding to the appearance consistency value is negatively correlated with the rotation angular velocity, and the weighting coefficient corresponding to the depth consistency value is positively correlated with the rotation angular velocity. Based on the total correlation cost analysis, the matching degree between the target detection result of the current frame and the historical target tracking trajectory is analyzed, and the target tracking result of the current frame is output. The process of predicting the depth prediction value of the target in the current frame based on the historical target trajectory information includes: Based on the depth time series data and motion state of historical target trajectories, the initial predicted depth of the target in the current frame is deduced. An adaptive temporal depth filtering method is used to weight and fuse the measured depth value of the current frame with the initial predicted depth to obtain the predicted depth value of the current frame.

2. The target tracking method for intelligent devices as described in claim 1, characterized in that, The process of obtaining the measured depth value and appearance features of the target in the current frame based on the target image of the current frame includes: A lightweight monocular depth estimation network is used to perform pixel-level depth processing on the target image of the current frame to obtain a depth map; Target detection is performed on the depth map to obtain target detection boxes; Extract the depth value of the target region in the target detection box to obtain the measured depth value of the target in the current frame; Image features are extracted from the target detection box to obtain the appearance features of the target in the current frame.

3. The target tracking method for intelligent devices as described in claim 1, characterized in that, The filter gain in the adaptive temporal depth filter is dynamically adjusted based on the depth measurement noise error and the equipment rotation error.

4. The target tracking method of the intelligent device as described in any one of claims 1 to 3, characterized in that, The method further includes: When the attitude rotation information meets the low-speed rotation condition, the system switches to normal tracking mode, where the normal tracking mode does not integrate the device's attitude rotation information and the target's depth information for target tracking.

5. A target tracking device, characterized in that, include: The rotation acquisition module is used to acquire real-time attitude rotation information of the smart device during the continuous image acquisition process. The depth acquisition module is used to acquire the measured depth value and appearance features of the target in the current frame based on the target image in the current frame, provided that the attitude rotation information meets the high-speed rotation condition. The tracking module is used to calculate the depth consistency value and appearance consistency value based on the measured depth values ​​and appearance features of the target in the current frame and the targets in historical frames. It then integrates the depth consistency value and appearance consistency value to analyze the target detection results of the current frame and outputs the target tracking results of the current frame. The depth consistency value refers to the consistency index between the measured depth value of the target in the current frame and the predicted depth value of the current frame based on the target trajectory information in historical frames. The appearance consistency value refers to the similarity between the appearance features of the target in the current frame and the appearance features of the targets in historical frames. The method involves calculating depth consistency value and appearance consistency value based on the measured depth values ​​and appearance features of the target in the current frame and the targets in historical frames. These two values ​​are then combined to analyze the target detection results of the current frame, outputting the target tracking results for the current frame, including: Obtain the appearance features of targets in historical frames, and calculate the similarity between the appearance features of targets in the current frame and the appearance features of targets in historical frames to obtain the appearance consistency cost. Obtain historical target trajectory information, and predict the depth value of the target in the current frame based on the historical target trajectory information; The consistency value of the measured depth and the predicted depth of the target in the current frame is calculated to obtain the depth consistency cost. The appearance consistency value and the depth consistency value are weighted and fused to obtain the total correlation value. The attitude rotation information includes rotation angular velocity. The weighting coefficients corresponding to the appearance consistency value and the depth consistency value are dynamically adjusted based on the real-time rotation angular velocity. The weighting coefficient corresponding to the appearance consistency value is negatively correlated with the rotation angular velocity, and the weighting coefficient corresponding to the depth consistency value is positively correlated with the rotation angular velocity. Based on the total correlation cost analysis, the matching degree between the target detection result of the current frame and the historical target tracking trajectory is analyzed, and the target tracking result of the current frame is output. The process of predicting the depth prediction value of the target in the current frame based on the historical target trajectory information includes: Based on the depth time series data and motion state of historical target trajectories, the initial predicted depth of the target in the current frame is deduced. An adaptive temporal depth filtering method is used to weight and fuse the measured depth value of the current frame with the initial predicted depth to obtain the predicted depth value of the current frame.

6. A smart device, characterized in that, The device includes a smart device body, which is equipped with an image acquisition unit, a rotation detection unit, and a target tracking device. The image acquisition unit is used to acquire images, the rotation detection unit is used to detect the real-time attitude rotation information of the smart device, the image acquisition unit and the rotation detection unit are connected to the target tracking device, and the target tracking device is used to execute the target tracking method of the smart device as described in any one of claims 1 to 4.

7. The intelligent device as described in claim 6, characterized in that, The smart device may be smart glasses, a smart following device, or a smart helmet; And / or, the image acquisition unit is a monocular camera; And / or, the rotation detection unit is an IMU sensor.

Images