Dynamic mark point detection tracking method and system based on improved CSRT

Abstract

The invention discloses a dynamic mark point detection tracking method and system based on improved CSRT, and belongs to the technical field of bionic visual tactile sensors, and the method comprises the steps: determining a mark point, and obtaining a first frame image containing the mark point; acquiring an initial position of the mark point in the first frame image based on a morphological feature spot detection method; acquiring a CSRT tracker of the mark point based on the initial position, and acquiring motion information of the mark point through the CSRT tracker; obtaining vector modulus length of inter-frame displacement of the mark points based on the motion information, comparing the vector modulus length with a preset modulus length threshold value, and determining the mark points corresponding to the vector modulus length outside the modulus length threshold value as moving points; acquiring a region covering the moving points as a dynamic attention region based on the spatial distribution of the moving points; and performing displacement tracking of the contact object based on the dynamic region of interest. According to the morphological feature-based spot detection method, accurate initial positioning of the mark points is realized, and a reliable starting point is provided for subsequent tracking.

Description

Technical Field

[0001] This application belongs to the field of biomimetic visual-tactile sensor technology, specifically relating to a dynamic marker detection and tracking method and system based on an improved CSRT. Background Technology

[0002] Tactile perception is crucial for robots to interact with their physical environment and perform dexterous maneuvers. Biomimetic visual-tactile sensors capture the displacement of densely packed markers within an elastic gel layer using a camera, converting tactile signals into visual signals to obtain high-resolution contact information. However, this technology faces significant challenges: the tiny size and dense distribution of markers within the sensor impose extremely high requirements on detection accuracy; and the variable lighting conditions in the robot's operating environment lead to unstable performance and low accuracy of traditional optical flow methods. Existing methods struggle to balance accuracy, robustness, and real-time performance. Summary of the Invention

[0003] Purpose of the invention: This application develops a dynamic marker detection and tracking method and system based on improved CSRT, aiming to solve the technical problems of low detection accuracy and poor real-time performance of existing bionic visual-tactile sensors.

[0004] Technical Solution: In a first aspect, embodiments of this application provide a dynamic marker detection and tracking method based on improved CSRT, applied to a bionic visual-tactile sensor, comprising: Identify the marker points and acquire the first frame image containing the marker points; A blob detection method based on morphological features is used to obtain the initial position of the marker points in the first frame image; The CSRT tracker of the marker point is obtained based on the initial position, and the motion information of the marker point is obtained through the CSRT tracker; Based on the motion information, the vector magnitude of the inter-frame displacement of the marker point is obtained and compared with a preset magnitude threshold. The marker point corresponding to the vector magnitude outside the magnitude threshold is determined as the moving point. Based on the spatial distribution of the moving points, the area covering the moving points is identified as the dynamic area of ​​interest. Displacement tracking of the contacting object is performed based on the dynamic region of interest.

[0005] In some embodiments, the step of the blob detection method based on morphological features to obtain the initial position of the marker point in the first frame image includes: The marker points in the first frame image are identified based on the aforementioned blob detection method; The initial position of the marker point is determined by the center coordinates and radius of the marker point in the first frame image.

[0006] In some embodiments, the step of identifying marker points in the first frame image based on the blob detection method includes: Based on multi-scale threshold segmentation, connected component analysis, and geometric feature filtering, points with regular shapes and significant contrast with the background are identified as the marker points in the first frame image.

[0007] In some embodiments, the step of acquiring the motion information of the marker point through the CSRT tracker includes: The position and bounding box of the marker point are updated based on the CSRT tracker; The motion trajectory of the marked point is obtained based on the bounding box.

[0008] In some embodiments, the dynamic attention region is the smallest bounding rectangle covering the moving point.

[0009] In some embodiments, before performing displacement tracking of the contacting object based on the dynamic region of interest, the method further includes extending the boundary of the dynamic region of interest to fully encompass the moving point.

[0010] In some embodiments, the method further includes adjusting the dynamic attention region based on the motion information of the marker points in each frame.

[0011] Secondly, embodiments of this application also provide a dynamic marker detection and tracking system based on an improved CSRT, applied to a bionic visual-tactile sensor, comprising: An image acquisition module is used to determine marker points and acquire the first frame image containing the marker points; A position determination module is used to obtain the initial position of the marker point in the first frame image using a blob detection method based on morphological features; A motion information module is used to acquire the CSRT tracker of the marker point based on the initial position, and to acquire the motion information of the marker point through the CSRT tracker; The moving point determination module is used to obtain the vector magnitude of the inter-frame displacement of the marker point based on the motion information, and compare it with a preset magnitude threshold to determine the marker point corresponding to the vector magnitude outside the magnitude threshold as the moving point; A dynamic attention module is used to obtain the area covering the mobile point as a dynamic attention area based on the spatial distribution of the mobile point. A displacement tracking module is used to track the displacement of a contacting object based on the dynamic area of ​​interest.

[0012] Beneficial Effects: Compared with existing technologies, the present application provides a dynamic marker detection and tracking method based on improved CSRT, including determining marker points and acquiring a first frame image containing the marker points; obtaining the initial position of the marker points in the first frame image using a blob detection method based on morphological features; acquiring a CSRT tracker for the marker points based on the initial positions, and obtaining the motion information of the marker points through the CSRT tracker; obtaining the vector magnitude of the inter-frame displacement of the marker points based on the motion information, and comparing it with a preset magnitude threshold to determine the marker points corresponding to vector magnitudes outside the magnitude threshold as moving points; obtaining the region covering the moving points as the dynamic interest region based on the spatial distribution of the moving points; and performing displacement tracking of contacting objects based on the dynamic interest region. The blob detection method based on morphological features in this application achieves accurate initial positioning of the marker points, providing a reliable starting point for subsequent tracking. Furthermore, a target CSRT tracker is constructed based on the initial detection results, utilizing its channel and spatial reliability mechanism to achieve stable tracking of marker point deformation, scale changes, and partial occlusion. Furthermore, a dynamic ROI mechanism based on motion analysis was introduced. By identifying moving markers in real time and generating adaptive regions of interest, computational resources are focused on the deformed region, significantly improving the system's real-time performance. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating the steps of the dynamic marker detection and tracking method based on improved CSRT provided in this application embodiment; Figure 2 A flowchart illustrating the steps of obtaining the initial position of a marker point in the first frame image in the dynamic marker point detection and tracking method based on improved CSRT provided in this application embodiment; Figure 3 A flowchart illustrating the steps of acquiring motion information of marker points using a CSRT tracker in the dynamic marker detection and tracking method based on improved CSRT provided in this application embodiment; Figure 4 A module connection diagram of a dynamic marker detection and tracking system based on an improved CSRT provided in this application embodiment; Figure 5 A flowchart illustrating the dynamic marker detection and tracking method based on improved CSRT provided in this application embodiment; Figure 6 A schematic diagram of the GelStereo visual-tactile sensor; Figure 7 A schematic diagram comparing the marker point detection of different algorithms; Figure 8 This is a schematic diagram of marker tracking based on optical flow algorithm; Figure 9 This is a schematic diagram of marker tracking based on the CSRT algorithm; Figure 10 This is a schematic diagram of the improved CSRT marker tracking based on the fusion of speckle detection and dynamic region of interest optimization strategy according to the embodiments of this application; Figure 11 This is a schematic diagram of an experiment based on a GelStereo type visual-tactile sensor for object contact position detection and displacement tracking, according to an embodiment of this application. Attached reference numerals: 10, Image acquisition module; 20, Position determination module; 30, Motion information module; 40, Moving point determination module; 50, Dynamic tracking module; 60, Displacement tracking module. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0016] Tactile perception is crucial for robots to interact with their physical environment and perform dexterous maneuvers. Biomimetic tactile sensors capture the displacement of densely packed markers within an elastic gel layer using a camera, converting tactile signals into visual signals to obtain high-resolution contact information. However, this technology faces significant challenges: the tiny size and dense distribution of markers within the sensor impose extremely high requirements on detection accuracy; and the variable lighting conditions in robot operating environments lead to unstable performance and low accuracy of traditional optical flow methods. Existing methods struggle to balance accuracy, robustness, and real-time performance. To investigate a dynamic marker tracking method that can adapt to complex lighting conditions and efficiently handle densely packed small targets, enabling high-precision real-time tactile perception in robots, this invention proposes a dynamic marker detection and tracking method based on an improved CSRT (Channel and Spatial Reliability Tracker).

[0017] In view of this, embodiments of this application provide a dynamic marker detection and tracking method based on an improved CSRT, including determining markers and acquiring a first frame image containing the markers; obtaining the initial position of the markers in the first frame image using a blob detection method based on morphological features; obtaining a CSRT tracker for the markers based on the initial position, and obtaining the motion information of the markers through the CSRT tracker; obtaining the vector magnitude of the inter-frame displacement of the markers based on the motion information, and comparing it with a preset magnitude threshold to determine the markers corresponding to vector magnitudes outside the magnitude threshold as moving points; obtaining the region covering the moving points as a dynamic region of interest based on the spatial distribution of the moving points; and performing displacement tracking of contacting objects based on the dynamic region of interest. The blob detection method based on morphological features in this application achieves accurate initial positioning of the markers, providing a reliable starting point for subsequent tracking. Secondly, a target CSRT tracker is constructed based on the initial detection results, utilizing its channel and spatial reliability mechanism to achieve stable tracking of marker deformation, scale changes, and partial occlusion. Thirdly, a dynamic ROI mechanism based on motion analysis is introduced, which focuses computational resources on the deformed region by real-time identification of moving markers and generation of adaptive regions of interest, significantly improving the system's real-time performance.

[0018] In some embodiments, please refer to Figure 1 , Figure 5 and Figure 6 , Figure 1 This is a flowchart illustrating the steps of the dynamic marker detection and tracking method based on improved CSRT provided in an embodiment of this application. Figure 5 This is a flowchart illustrating the dynamic marker detection and tracking method based on improved CSRT provided in an embodiment of this application. Figure 6 This is a schematic diagram of the GelStereo visual-tactile sensor. The dynamic marker detection and tracking method based on the improved CSRT provided in this application embodiment is specifically implemented through steps 100 to 600: Step 100: Determine the marker points and acquire the first frame image containing the marker points.

[0019] Step 200: Obtain the initial position of the marker points in the first frame image using a morphological feature-based blob detection method.

[0020] In some embodiments, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of obtaining the initial position of a marker point in the first frame image in the dynamic marker point detection and tracking method based on improved CSRT provided in this application embodiment. The method for obtaining the initial position of a marker point in the first frame image in this application is specifically implemented through steps 210 to 220: Step 210: Identify marker points in the first frame image based on the blob detection method.

[0021] Step 220: Determine the initial position of the marker point by using the center coordinates and radius of the marker point in the first frame image.

[0022] Specifically, during the system initialization phase, a lightweight speckle detection algorithm is used to perform a full-image scan of the first frame image captured by the sensor. This algorithm accurately identifies marker points with regular shapes and significant contrast with the background in the first frame image through multi-scale threshold segmentation, connected component analysis, and geometric feature filtering, and outputs the center coordinates and radius of the marker points, providing a high-precision, low-computational-cost initial position for subsequent tracking tasks.

[0023] Step 300: Obtain the CSRT tracker for the marker points based on the initial position, and obtain the motion information of the marker points through the CSRT tracker.

[0024] In some embodiments, please refer to Figure 3 , Figure 3 This is a flowchart illustrating the steps of obtaining motion information of marker points using a CSRT tracker in the dynamic marker point detection and tracking method based on improved CSRT provided in this application. Specifically, the method for obtaining motion information of marker points using a CSRT tracker in this application is implemented through steps 310 to 320: Step 310: Update the position and bounding box of the marker point based on the CSRT tracker.

[0025] Step 320: Obtain the motion trajectory of the marked points based on the bounding box.

[0026] Specifically, based on the initial positions provided by blob detection, a CSRT tracker is created for each marker point, forming a multi-target tracking system. In subsequent frame sequences, leveraging the advantages of the CSRT algorithm in channel and spatial reliability, the position and bounding box of each marker point are continuously updated, and its motion trajectory is recorded. This step effectively utilizes the strong robustness of the CSRT algorithm to target deformation, scale changes, and partial occlusion, ensuring tracking accuracy under complex contact conditions.

[0027] Step 400: Obtain the vector magnitude of the inter-frame displacement of the marker point based on motion information, and compare it with the preset magnitude threshold to determine the marker point corresponding to the vector magnitude outside the magnitude threshold as the moving point.

[0028] Step 500: Based on the spatial distribution of the mobile points, obtain the area covering the mobile points as the dynamic interest area.

[0029] In some embodiments, the dynamic interest region is the smallest bounding rectangle covering the moving point.

[0030] In some embodiments, before performing displacement tracking of the contacting object based on the dynamic region of interest, the method further includes extending the boundary of the dynamic region of interest to fully encompass the moving point.

[0031] In some embodiments, the dynamic attention region is adjusted based on the motion information of each frame marker point.

[0032] Understandably, to overcome the bottleneck of high computational complexity in the CSRT algorithm, this application introduces a dynamic ROI optimization strategy based on motion analysis. This strategy analyzes the motion trajectories of all marker points in real time, calculates their inter-frame displacement vector magnitudes, and compares them with preset thresholds to accurately select a set of "moving points" that have undergone significant displacement due to contact forces. Subsequently, based on the spatial distribution of these moving points, the algorithm automatically calculates and generates a minimum bounding rectangle region that completely covers this set as a dynamic ROI, and appropriately expands it around its boundaries to ensure that the moving target is fully contained. This ROI region is not fixed but adaptively adjusted and updated in each frame according to the motion state of the marker point group. Through this mechanism, the algorithm strictly constrains the two most computationally demanding operations—full-image blob detection and CSRT tracker updates—to be performed within this dynamic ROI region, thereby completely avoiding invalid computation on a vast background area where no deformation has occurred. This method essentially intelligently allocates limited computational resources, focusing on the most critical tactile perception area, ultimately effectively resolving the contradiction between high accuracy and high real-time performance.

[0033] Step 600: Track the displacement of the contacting object based on the dynamic region of interest.

[0034] Understandably, embodiments of this application provide a dynamic marker detection and tracking method based on an improved CSRT, including determining markers and acquiring a first frame image containing the markers; obtaining the initial position of the markers in the first frame image using a blob detection method based on morphological features; acquiring a CSRT tracker for the markers based on the initial position, and obtaining the motion information of the markers through the CSRT tracker; obtaining the vector magnitude of the inter-frame displacement of the markers based on the motion information, and comparing it with a preset magnitude threshold to determine the markers corresponding to vector magnitudes outside the magnitude threshold as moving points; obtaining the region covering the moving points as a dynamic region of interest based on the spatial distribution of the moving points; and performing displacement tracking of contacting objects based on the dynamic region of interest. The blob detection method based on morphological features in this application achieves accurate initial positioning of the markers, providing a reliable starting point for subsequent tracking. Secondly, a target CSRT tracker is constructed based on the initial detection results, utilizing its channel and spatial reliability mechanism to achieve stable tracking of marker deformation, scale changes, and partial occlusion. Thirdly, a dynamic ROI mechanism based on motion analysis is introduced, which, by real-time identification of moving markers and generation of adaptive regions of interest, focuses computational resources on deformed regions, significantly improving the system's real-time performance. Specifically, this application significantly improves the detection robustness of marker points under different lighting conditions by integrating lightweight speckle detection with a CSRT tracker; it introduces a dynamic ROI mechanism based on motion analysis to adaptively focus on deformed regions, increasing the computation frame rate from 8.34 FPS to 22.17 FPS; and finally, while maintaining a tracking persistence rate of 99.41%, it achieves an effective balance between detection accuracy, tracking stability and system real-time performance.

[0035] For example, to verify the effectiveness of the dynamic marker detection and tracking method based on the improved CSRT proposed in this application, a system performance test experiment was conducted based on the GelStereo bionic visual-tactile sensor platform. The experimental platform configuration included a computer system with an Intel Core i5-14700 processor, an NVIDIA GeForce RTX 4080 graphics card, and 32GB of memory. The sensor had 529 regularly arranged markers with a spatial resolution of 0.9mm.

[0036] The experiment was divided into two stages: marker detection performance testing and tracking performance comparison testing. In the marker detection experiment, three typical lighting environments—weak light, normal light, and strong light—were constructed by adjusting the light source intensity. Marker recognition was performed using both the traditional optical flow algorithm and the algorithm proposed in this application. The experimental results are as follows: Figure 11 As shown, the algorithm in this application can maintain stable detection performance under three lighting conditions and the marker points are completely identified, while the optical flow algorithm has a large number of missed detections in low light environment.

[0037] In the performance tracking test, please refer to Figure 7 , Figure 7 This diagram illustrates the comparison of marker point detection using different algorithms. This application compares the tracking performance of the optical flow algorithm, the original CSRT algorithm, and the algorithm presented in this application. Figure 8 The tracking results of the optical flow algorithm are shown, and there is a clear tracking interruption phenomenon. Figure 9 This shows that while the CSRT algorithm can maintain high tracking accuracy, its processing speed is relatively slow. Figure 10 The tracking performance of the algorithm presented in this application is shown, which significantly improves the processing speed while maintaining a high tracking sustain rate. The quantitative analysis results are shown in Table 2. While maintaining a tracking sustain rate of 99.41%, the algorithm of this application increases the frame rate from 8.34 FPS of the original CSRT algorithm to 22.17 FPS, an improvement of 2.66 times.

[0038] Furthermore, the practicality of the algorithm was verified through experiments tracking the displacement of contact objects. Figure 11 As shown, when the object moves from the upper left to the lower right, more than 85% of the marker points show a consistent displacement direction, accurately reflecting the object's trajectory, thus proving the effectiveness and reliability of the algorithm in the robot's tactile perception task.

[0039] Accordingly, embodiments of this application also provide a dynamic marker detection and tracking system based on improved CSRT. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a module connection diagram of the dynamic marker detection and tracking system based on improved CSRT provided in this application embodiment. The dynamic marker detection and tracking system based on improved CSRT provided in this application embodiment includes: Image acquisition module 10 is used to determine the marker points and acquire the first frame image containing the marker points; The position determination module 20 is used to obtain the initial position of the marker point in the first frame image by a blob detection method based on morphological features; Motion information module 30 is used to acquire the CSRT tracker of the marker point based on the initial position, and to acquire the motion information of the marker point through the CSRT tracker; The moving point determination module 40 is used to obtain the vector magnitude of the inter-frame displacement of the marker point based on motion information, and compare it with a preset magnitude threshold to determine the marker point corresponding to the vector magnitude outside the magnitude threshold as the moving point. Dynamic attention module 50 is used to obtain the area covering the mobile point as the dynamic attention area based on the spatial distribution of the mobile point. The displacement tracking module 60 is used to track the displacement of a contacting object based on a dynamic region of interest.

[0040] In some embodiments, the location determination module 20 is specifically used for: Identify marker points in the first frame image based on a blob detection method; The initial position of the marker point is determined by the center coordinates and radius of the marker point in the first frame image.

[0041] In some embodiments, the location determination module 20 is specifically used for: Based on multi-scale threshold segmentation, connected component analysis, and geometric feature filtering, points with regular shapes and significant contrast with the background are identified as marker points in the first frame image.

[0042] In some embodiments, the motion information module 30 is specifically used for: Update the position and bounding box of the marker point based on the CSRT tracker; Obtain the motion trajectory of the marked points based on the bounding box.

[0043] This application has provided a detailed description of a dynamic marker detection and tracking method and system based on an improved CSRT, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dynamic marker detection and tracking method based on improved CSRT, characterized in that, Applications in biomimetic visual and tactile sensors include: Identify the marker points and acquire the first frame image containing the marker points; A blob detection method based on morphological features is used to obtain the initial position of the marker points in the first frame image; The CSRT tracker of the marker point is obtained based on the initial position, and the motion information of the marker point is obtained through the CSRT tracker; Based on the motion information, the vector magnitude of the inter-frame displacement of the marker point is obtained and compared with a preset magnitude threshold. The marker point corresponding to the vector magnitude outside the magnitude threshold is determined as the moving point. Based on the spatial distribution of the moving points, the area covering the moving points is identified as the dynamic area of ​​interest. Displacement tracking of the contacting object is performed based on the dynamic region of interest.

2. The dynamic marker detection and tracking method based on improved CSRT according to claim 1, characterized in that, The step of the blob detection method based on morphological features to obtain the initial position of the marker point in the first frame image includes: The marker points in the first frame image are identified based on the aforementioned blob detection method; The initial position of the marker point is determined by the center coordinates and radius of the marker point in the first frame image.

3. The dynamic marker detection and tracking method based on improved CSRT according to claim 2, characterized in that, The step of identifying marker points in the first frame image based on the blob detection method includes: Based on multi-scale threshold segmentation, connected component analysis, and geometric feature filtering, points with regular shapes and significant contrast with the background are identified as the marker points in the first frame image.

4. The dynamic marker detection and tracking method based on improved CSRT according to claim 1, characterized in that, The step of acquiring the motion information of the marker point through the CSRT tracker includes: The position and bounding box of the marker point are updated based on the CSRT tracker; The motion trajectory of the marked point is obtained based on the bounding box.

5. The dynamic marker detection and tracking method based on improved CSRT according to claim 1, characterized in that, The dynamic attention area is the smallest bounding rectangle covering the moving point.

6. The dynamic marker detection and tracking method based on improved CSRT according to claim 1, characterized in that, Before performing displacement tracking of the contacting object based on the dynamic region of interest, the method further includes extending the boundary of the dynamic region of interest to fully encompass the moving point.

7. The dynamic marker detection and tracking method based on improved CSRT according to claim 1, characterized in that, The method further includes adjusting the dynamic attention region based on the motion information of the marker points in each frame.

8. A dynamic marker detection and tracking system based on improved CSRT, characterized in that, Applications in biomimetic visual and tactile sensors include: Image acquisition module (10), the image acquisition module (10) is used to determine the marker points and acquire the first frame image containing the marker points; The position determination module (20) is used to obtain the initial position of the marker point in the first frame image by a blob detection method based on morphological features; Motion information module (30), the motion information module (30) is used to obtain the CSRT tracker of the marker point based on the initial position, and to obtain the motion information of the marker point through the CSRT tracker; The moving point determination module (40) is used to obtain the vector magnitude of the inter-frame displacement of the marker point based on the motion information, and compare it with a preset magnitude threshold to determine the marker point corresponding to the vector magnitude outside the magnitude threshold as the moving point; The dynamic attention module (50) is used to obtain the area covering the mobile point as the dynamic attention area based on the spatial distribution of the mobile point; The displacement tracking module (60) is used to track the displacement of the contacting object based on the dynamic interest area.

Images