A method and system for continuous measurement of target line-of-sight angle of a quadruped robot

Abstract

The application discloses a target line-of-sight angle continuous measurement method and system of a quadruped robot, relates to the technical field of visual measurement, and solves the technical problem that in the prior art, the imaging is unstable and is easily disturbed by shielding during the running of the quadruped robot due to high-frequency vibration and posture disturbance, the two-dimensional observation result is caused to drift or be lost for a short time, and the accuracy of the line-of-sight angle measurement is low; the target template region and a target template feature vector are obtained through initial image data, a prediction result and a prediction evaluation index are generated based on real-time image data, the prediction result is updated by using the prediction evaluation index, the final prediction result after correction is finally solved into line-of-sight angle measurement data, observation drift is inhibited from the source by monitoring and triggering correction through the evaluation index, the calculation overhead is reduced by adopting a hierarchical strategy, target misassociation is avoided by combining consistency identification, and the accuracy of the line-of-sight angle measurement is improved by using time sequence filtering to smooth the angle sequence.

Description

Technical Field

[0001] This application belongs to the field of visual measurement technology, specifically a method and system for continuous measurement of the target line of sight angle of a quadruped robot. Background Technology

[0002] With the development of robotics technology, quadruped robots, thanks to their terrain adaptability enabled by discrete foot contact, have been widely used in complex tasks such as disaster relief, field surveying, and industrial inspection. In these complex tasks, continuous measurement and stable output of the relative observation axis angle deviation towards external targets are the data foundation for centering, quantification of target relative pointing, and subsequent task planning. Typically, quadruped robots are equipped with a vision observation unit that uses image processing algorithms to obtain two-dimensional observation results of the target in the image plane. Based on the camera imaging geometry and preset imaging parameters, the two-dimensional pixel coordinates are mapped into physically quantifiable line-of-sight angle measurements or line-of-sight pointing vectors.

[0003] In existing technologies, quadruped robots often experience unstable imaging due to posture disturbances such as high-frequency vibrations during operation, and are easily affected by occlusion interference, resulting in drift or short-term loss of two-dimensional observation results, which in turn leads to low accuracy of line-of-sight angle measurement. Therefore, further improvements are needed for continuous measurement methods of the target line-of-sight angle of quadruped robots. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art; to this end, this application proposes a method and system for continuous measurement of the target line of sight angle of a quadruped robot, which is used to solve the technical problem that the quadruped robot often causes non-stable imaging due to posture disturbances such as high-frequency vibration during operation, and is easily affected by occlusion interference, resulting in drift or short-term loss of two-dimensional observation results, thus leading to low accuracy of line of sight angle measurement.

[0005] To achieve the above objectives, the first aspect of this application provides a method for continuous measurement of the target line-of-sight angle of a quadruped robot, comprising:

[0006] Acquire initial image data and real-time image data;

[0007] Initialization operations are performed on the initial image data to obtain the target template region and the target template feature vector;

[0008] Location prediction is performed based on real-time image data and target template region to obtain prediction results and prediction evaluation metrics; the prediction results include target center coordinates and target coarse boundary region; the prediction evaluation metrics include response score and APCE stability;

[0009] The prediction results are corrected based on the prediction evaluation indicators to obtain the final prediction result; the final prediction result includes the final target center coordinates and the final target boundary region;

[0010] The line-of-sight angle measurement data is obtained by measuring the target line-of-sight angle of the final prediction result.

[0011] This application utilizes the above steps to perform real-time monitoring using predictive evaluation indicators and triggers a correction mechanism when necessary; it adopts a hierarchical processing strategy to significantly reduce computational overhead; at the same time, it introduces a consistency discrimination mechanism to effectively avoid target misassociation; and it combines temporal filtering to smooth the line-of-sight angle measurement sequence, thereby improving the stability, accuracy, and temporal continuity of line-of-sight angle measurements.

[0012] Furthermore, the initialization operation on the initial image data to obtain the target template region and the target template feature vector includes:

[0013] Extract initial image data; the initial image data refers to image data containing the target.

[0014] The initial target template region is obtained by locating the target in the initial image data.

[0015] The initial target template region is normalized in terms of scale and pixel value to obtain the target template region.

[0016] The target template region is input into the feature extraction network to extract features and obtain the initial target template feature vector;

[0017] The initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, and then stored in the feature memory.

[0018] Furthermore, the location prediction based on real-time image data and the target template region to obtain prediction results and prediction evaluation metrics includes:

[0019] Extract real-time image data and target template region;

[0020] Obtain the final prediction result of the target in the previous frame image corresponding to the real-time image data; the final prediction result includes the final target center coordinates and the final target boundary region;

[0021] The search area image is obtained by cropping the real-time image data with the final target center coordinates as the center; the size of the search area image is the size corresponding to the final target boundary region of the target in the previous frame image;

[0022] The search region image and the target template region are respectively divided into blocks to obtain several corresponding image blocks, and the several image blocks are mapped to corresponding feature label vectors;

[0023] A joint label sequence is obtained by concatenating the template label sequence and the search region label sequence; the template label sequence consists of several feature label vectors corresponding to the target template region; the search region label sequence consists of several feature label vectors corresponding to the search region image.

[0024] A joint feature map is obtained by performing feature interaction on the joint label sequence;

[0025] The joint feature map is input into the prediction head to obtain a confidence response map and a rough boundary region of the target;

[0026] The target center coordinates are obtained by extracting the center coordinates from the confidence response map; the target center coordinates satisfy:

[0027] ;in, This is represented as the confidence response map corresponding to the t-th frame image. Represented as the coordinates of the target center corresponding to the t-th frame image;

[0028] The predictive evaluation index is obtained by performing a reliability assessment on the confidence response map.

[0029] Furthermore, the step of performing feature interaction on the joint label sequence to obtain a joint feature map includes:

[0030] Extract the joint marker sequence;

[0031] The joint label sequence is input into a single-stream feature extraction network, and a joint feature map is obtained by performing full-cycle feature interaction between the target template region features and the search region image features through a multi-head self-attention mechanism.

[0032] The attention calculation in the multi-head self-attention mechanism satisfies:

[0033] Where d represents the feature dimension; It is represented as a normalized exponential function; T represents the attention output; T represents the matrix transpose operation; Q, K, and V represent the query vector, key vector, and value vector obtained by linear mapping of the joint label sequence, respectively.

[0034] Furthermore, the method of obtaining predictive evaluation indicators by performing reliability assessment on the confidence response map includes:

[0035] Extracting confidence response plots ;

[0036] The maximum peak score in the confidence response graph is extracted as the response score of the confidence response graph; the response score satisfy: ;in, Represented as pixel coordinates in the confidence response map;

[0037] Calculate the average peak correlation energy corresponding to the confidence response plot. The calculation of the average peak correlation energy satisfies:

[0038] ;in, This is represented as the minimum peak score in the confidence response graph corresponding to the t-th frame image; Let N represent the response value corresponding to the i-th pixel position in the confidence response graph, and let N represent the total number of pixels in the confidence response graph.

[0039] The average peak correlation energy (APCE) of several frames within a sliding time window is obtained, and the variance of the average peak correlation energy is calculated. The variance is used as the APCE stability.

[0040] Predictive evaluation metrics are determined based on response scores and APCE stability.

[0041] This application uses real-time image data and target template regions. It crops an adaptively scaled search region image using the target center coordinates of the previous frame as the center. Then, it maps the search region and template into feature label vectors, which are then concatenated into a joint label sequence. A multi-head self-attention mechanism in a single-stream feature extraction network is used for full-cycle feature interaction, outputting a dense joint feature map. This map is then processed by a prediction head to generate a confidence response map and a rough target boundary region, and the target center coordinates are extracted. Finally, a reliability assessment of the response map yields a prediction evaluation index. Attention-driven feature interaction enhances the semantic alignment between the template and the search region, improving target localization accuracy. Simultaneously, using response score and APCE stability as dual evaluation indicators, it can identify tracking drift caused by occlusion or disturbance in real time, providing a reliable basis for subsequent trigger corrections. This suppresses cascading errors at the source, ensuring the continuity and stability of subsequent line-of-sight measurement sequences.

[0042] Furthermore, the step of correcting the prediction results based on the prediction evaluation indicators to obtain the final prediction result includes:

[0043] Extract the response score and APCE stability from the predictive evaluation metrics;

[0044] When the response score is greater than the safety score threshold and the APCE stability is less than the stability threshold, the prediction result is taken as the final prediction result; the final target boundary region in the final prediction result is input into the feature extraction network for feature extraction to obtain the initial target template feature vector corresponding to the current frame image; the initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, and it is stored in the feature memory.

[0045] Otherwise, extract the joint feature map, as well as the target center coordinates and the target coarse boundary region from the prediction results; use the target coarse boundary region as a spatial cue to drive the segmentation model to generate an initial candidate binary mask; perform masking operations on the initial candidate binary mask to obtain a candidate binary mask; the masking operations include binarization, morphological opening and closing operations and connected component analysis, and the masking operations are used to eliminate noise and retain the target main body region;

[0046] Foreground feature vectors are determined based on joint feature maps and candidate binary masks;

[0047] Extract the latest target template feature vector from the feature memory;

[0048] The final prediction result is obtained by determining the consistency between the target template feature vector and the foreground feature vector and updating the prediction result.

[0049] Furthermore, the determination of the foreground feature vector based on the joint feature map and candidate binary mask includes:

[0050] Extract the joint feature map and candidate binary masks;

[0051] The candidate binary mask is downsampled to the same resolution as the joint feature map, and the foreground feature vector of the mask region corresponding to the candidate binary mask is extracted; the foreground feature vector satisfy:

[0052] ;in, Represented as the joint feature map in the t-th frame image The eigenvalue at the location; This is represented as the mask region corresponding to the k-th candidate binary mask; This is represented as the number of pixels in the masked area.

[0053] Furthermore, by performing consistency determination on the target template feature vector and the foreground feature vector, and updating the prediction result, the final prediction result is obtained, including:

[0054] Extracting target template feature vectors With foreground feature vector ;

[0055] Calculate the consistency score between the target template feature vector and the foreground feature vector; the consistency score The cosine similarity is used for calculation, satisfying the following:

[0056] ;in, It is represented as the L2 norm of a vector;

[0057] When a consistency score is greater than the consistency threshold, the geometric center of the candidate binary mask with the consistency score greater than the consistency threshold is extracted, and the geometric center is used as the final target center coordinate; and the minimum bounding rectangle of the candidate binary mask is used as the final target boundary region.

[0058] When the consistency scores of all candidate binary masks are less than or equal to the consistency threshold, the final target center coordinates are predicted by the Kalman filter, and the final target boundary region of the current frame is determined based on the size of the final target boundary region obtained from the previous frame image and the final target center coordinates obtained from the current frame.

[0059] The Kalman filter predicts the final target center coordinates to satisfy the following conditions:

[0060] ;in, Let A be the state vector of the target in the t-th frame; A is the state transition matrix, which represents the transition relationship of the target state from the previous frame to the current frame. This is represented as the state vector corresponding to the target in the (t-1)th frame of the image; Represented as process noise, it is used to describe randomness and uncertainty.

[0061] This application extracts prediction evaluation indicators. When these indicators meet preset conditions, the prediction results are directly output and the feature memory is updated. Otherwise, a correction process is triggered: using the target's rough boundary region as a spatial cue, the segmentation model is driven to generate candidate binary masks, and the mask quality is optimized through binarization, morphological opening and closing operations, and connected component analysis. Subsequently, foreground feature vectors are extracted based on the joint feature map, and cosine similarity is calculated with the template feature vectors in the feature memory to obtain a consistency score. If the consistency score is higher than its corresponding threshold, the target position is corrected using the mask's geometric center; otherwise, the state prediction is updated using a Kalman filter, thereby dynamically maintaining the continuity of observation. A hierarchical triggering strategy is adopted to avoid continuous operation of the segmentation model and significantly reduce the computational overhead at the edge. At the same time, the consistency discrimination mechanism effectively suppresses foreground misleading errors caused by misassociation of occluded objects, and relies on historical state prediction to ensure uninterrupted output when correction fails. Ultimately, this significantly improves the accuracy and stability of line-of-sight angle measurement in high-disturbance and occluded scenarios.

[0062] Furthermore, the step of obtaining the line-of-sight angle measurement data result by measuring the target line-of-sight angle of the final prediction result includes:

[0063] Extract the final prediction results;

[0064] Extract the resolution W×H of the current frame image and the preset horizontal field of view. With vertical field of view The geometric center (W / 2, H / 2) of the current frame image is used as the center of the observation axis.

[0065] Calculate the normalized deviation of the final target center coordinates relative to the line-of-sight center. and The normalized deviation satisfies:

[0066] ; Where u and v represent the coordinates of the final target center; ∈[-1, 1], ∈[-1, 1]; and These respectively characterize the degree of deviation of the target from the center of the line of sight in the horizontal and vertical directions;

[0067] By mapping the normalized bias to angle measurements, the horizontal azimuth θ and vertical pitch ψ of the target relative to the observation line of sight are obtained.

[0068] ; Where θ represents the horizontal azimuth angle of the target relative to the observation line of sight, and ψ represents the vertical pitch angle of the target relative to the observation line of sight.

[0069] Obtain the angle measurement corresponding to the current frame image. Then, an exponential smoothing filter is applied to obtain a stable angle measurement. ;

[0070] The exponential smoothing filter operation satisfies:

[0071] ;

[0072] ;in, For smoothing coefficients, ∈(0,1);

[0073] The line-of-sight angle measurement results are determined based on stable angle measurements and predictive evaluation indicators.

[0074] This application calculates the normalized deviation based on the target center coordinates in the final prediction result, combined with image resolution and preset field-of-view parameters, using the geometric center of the image as the center of the observation axis. Then, it solves the horizontal azimuth angle of the target relative to the observation axis through a linear mapping model. Subsequently, for the angle jump caused by the quadruped robot's posture disturbance, the real-time angle sequence is subjected to exponential smoothing filtering. Finally, stable line-of-view measurement data results are encapsulated and output by combining prediction evaluation indicators. By using preset calibration parameters and a normalized mapping model, pixel coordinates are converted into angle quantities with physical measurement significance, ensuring the accuracy and interpretability of the measurement results. At the same time, temporal filtering is used to effectively suppress angle jitter caused by carrier vibration, improve the continuity and smoothness of the output sequence, and provide a reliable data foundation for upper-layer applications.

[0075] A second aspect of the present invention provides a continuous target line-of-sight angle measurement system for a quadruped robot, comprising: a data acquisition module, an initialization module, a visual monitoring module, a calibration module, and a result generation module; the data acquisition module and the initialization module are connected; the initialization module and the visual monitoring module are connected; the visual monitoring module and the calibration module are connected; and the calibration module and the result generation module are connected.

[0076] The data acquisition module acquires initial image data and real-time image data through a data acquisition device.

[0077] The initialization module performs initialization operations on the initial image data to obtain the target template region and the target template feature vector;

[0078] The visual monitoring module: performs location prediction based on real-time image data and target template area to obtain prediction results and prediction evaluation indicators;

[0079] The correction module corrects the prediction results based on the prediction evaluation indicators to obtain the final prediction result;

[0080] The result generation module obtains the line-of-sight angle measurement data by measuring the target line-of-sight angle of the final prediction result.

[0081] Compared with the prior art, the beneficial effects of this application are:

[0082] 1. This application initializes the target template region and target template feature vector by acquiring initial image data, predicts the target center coordinates, the target rough boundary region and prediction evaluation index based on real-time image data, and uses the evaluation index to trigger a correction mechanism to perform geometric correction or update the historical state prediction of the prediction results. Finally, the corrected final prediction results are solved into line-of-sight angle measurement data by combining preset imaging field parameters. The application suppresses observation drift from the source by monitoring the evaluation index and triggering correction, reduces computational overhead by adopting a hierarchical strategy, avoids target misassociation by combining consistency identification, and uses temporal filtering to smooth the angle sequence, thereby improving the stability, accuracy and continuity of the line-of-sight angle measurement.

[0083] 2. This application, based on real-time image data and target template regions, uses the target center coordinates of the previous frame as the center to crop an adaptively scaled search region image. Then, the search region and template are mapped into feature label vectors and concatenated into a joint label sequence. A multi-head self-attention mechanism in a single-stream feature extraction network is used for full-cycle feature interaction, outputting a densely structured joint feature map. This map is then used by a prediction head to generate a confidence response map and a rough target boundary region, and the target center coordinates are extracted. Finally, a reliability assessment of the response map is performed to obtain a prediction evaluation index. Attention-driven feature interaction enhances the semantic alignment between the template and the search region, improving target localization accuracy. Simultaneously, using response score and APCE stability as dual evaluation indicators, tracking drift caused by occlusion or disturbance can be identified in real time, providing a reliable basis for subsequent trigger corrections. This suppresses cascading errors from the source, ensuring the continuity and stability of subsequent line-of-sight measurement sequences.

[0084] 3. This application extracts prediction and evaluation indicators. When all prediction and evaluation indicators meet the preset conditions, the prediction results are directly output and the feature memory is updated. Otherwise, a correction process is triggered. The target's rough boundary region is used as a spatial cue to drive the segmentation model to generate candidate binary masks. The mask quality is optimized through binarization, morphological opening and closing operations, and connected component analysis. Then, the foreground feature vector is extracted based on the joint feature map, and a cosine similarity consistency score is calculated with the template feature vector in the feature memory. When the consistency score is higher than its threshold, the target position is corrected using the geometric center of the mask. Otherwise, the Kalman filter is used for prediction and updating, thereby dynamically maintaining the continuity of observation. The hierarchical triggering strategy avoids the segmentation model from running all the time, significantly reducing the computational overhead at the end. Combined with consistency identification, the foreground misleading error caused by occlusion is effectively suppressed. At the same time, when the correction fails, the historical state prediction is used to ensure uninterrupted output. Finally, the line-of-sight angle measurement is improved in terms of high accuracy and stability under high disturbance and occlusion conditions. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart of a method for continuous measurement of the target line of sight angle of a quadruped robot according to this application;

[0087] Figure 2 This is a schematic diagram of the principle of a continuous target line-of-sight angle measurement system for a quadruped robot according to this application. Detailed Implementation

[0088] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0089] Please see Figure 1 The first aspect of this application provides a method for continuously measuring the target line-of-sight angle of a quadruped robot, including:

[0090] Acquire initial image data and real-time image data;

[0091] Initialization operations are performed on the initial image data to obtain the target template region and the target template feature vector;

[0092] Location prediction is performed based on real-time image data and target template region to obtain prediction results and prediction evaluation metrics; the prediction results include target center coordinates and target rough boundary region; the prediction evaluation metrics include response score and APCE stability.

[0093] The prediction results are corrected based on the prediction evaluation indicators to obtain the final prediction results; the final prediction results include the coordinates of the center of the final target and the boundary area of ​​the final target;

[0094] The line-of-sight angle measurement data is obtained by measuring the target line-of-sight angle of the final prediction result.

[0095] In this embodiment, the initialization operation of the initial image data to obtain the target template region and the target template feature vector includes:

[0096] Extract initial image data; initial image data refers to image data containing the target.

[0097] The initial target template region is obtained by locating the target in the initial image data. In this embodiment, the initial target template region can be obtained by manually selecting boxes in the initial image data. In other embodiments, the initial target template region of the target is automatically output by a target detection algorithm.

[0098] The initial target template region is normalized in scale and pixel value to obtain the target template region. In this embodiment, scale normalization and pixel value normalization are performed to eliminate the influence of different shooting distances and lighting conditions on the subsequent feature extraction results.

[0099] The target template region is input into the feature extraction network for feature extraction to obtain the initial target template feature vector; in this embodiment, the feature extraction network adopts the visual Transformer model;

[0100] The initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, which is then stored in the feature memory. In this embodiment, the expression of the target template feature vector satisfies:

[0101] ; Represented as the initial target template feature vector; This is represented as the target template feature vector after L2 normalization. It is represented as the L2 norm of a vector.

[0102] This embodiment first extracts initial image data and calibrates the target position based on manual selection or external detection algorithms to obtain the initial target template region. Then, the region is normalized in scale and pixel value to eliminate the influence of shooting distance and illumination differences. Next, the processed template region is input into the feature extraction network to obtain the initial target template feature vector, and further enhanced in numerical stability by L2 normalization. Finally, the normalized feature vector is stored in the feature memory as a reference anchor point. Through standardization and data solidification, a high-precision and interference-resistant reference benchmark is provided for feature consistency verification in subsequent real-time tracking, effectively preventing target features from drifting due to occlusion or disturbance, thereby improving the accuracy and robustness of the line-of-sight angle measurement sequence from the source.

[0103] In this embodiment, location prediction based on real-time image data and target template region is used to obtain prediction results and prediction evaluation metrics, including:

[0104] Extract real-time image data and target template region;

[0105] The final prediction result of the target in the previous frame of the real-time image data is obtained; the final prediction result includes the final target center coordinates and the final target boundary region; in this embodiment, the real-time image data consists of several frames of images;

[0106] The search region image is obtained by cropping the real-time image data with the final target center coordinates as the center; the size of the search region image is the size corresponding to the final target boundary region in the previous frame image;

[0107] The search region image and the target template region are divided into blocks to obtain several corresponding image blocks, and these image blocks are mapped to corresponding feature label vectors.

[0108] The template label sequence and the search region label sequence are concatenated to obtain the joint label sequence; the template label sequence consists of several feature label vectors corresponding to the target template region; the search region label sequence consists of several feature label vectors corresponding to the search region image.

[0109] A joint feature map is obtained by performing feature interaction on the joint label sequence;

[0110] The joint feature map is input into the prediction head to obtain a confidence response map and a rough target boundary region. In this embodiment, the prediction head consists of four stacked Conv-BN-ReLU modules connected in series. The Conv-BN-ReLU module includes a convolutional layer, a normalization layer, and an activation layer. The prediction head has two output branches: a confidence branch and a bounding box regression branch. The confidence branch is used to output the confidence response map, and the bounding box regression branch is used to output the rough target boundary region.

[0111] The center coordinates of the target are obtained by extracting the center coordinates from the confidence response map; the center coordinates of the target satisfy:

[0112] ;in, This is represented as the confidence response map corresponding to the t-th frame image. Represented as the coordinates of the target center corresponding to the t-th frame image;

[0113] The predictive evaluation index is obtained by performing a reliability assessment on the confidence response map.

[0114] In this embodiment, the joint feature map is obtained by performing feature interaction on the joint label sequence, including:

[0115] Extract the joint marker sequence;

[0116] The joint label sequence is input into a single-stream feature extraction network, and a joint feature map is obtained by performing full-cycle feature interaction between the target template region features and the search region image features through a multi-head self-attention mechanism; in this embodiment, the single-stream feature extraction network adopts the visual Transformer model;

[0117] Attention computation in multi-head self-attention mechanisms satisfies:

[0118] Where d represents the feature dimension; It is represented as a normalized exponential function; T represents the attention output; T represents the matrix transpose operation; Q, K, and V represent the query vector, key vector, and value vector obtained by linear mapping of the joint label sequence, respectively.

[0119] In this embodiment, the target tracking model used for target localization is trained offline based on a general tracking dataset. To improve the network's generalization ability, data augmentation strategies are employed during training, including image rotation, cropping, scaling, and brightness jittering, to simulate motion blur and viewpoint disturbances caused by body vibration. The target tracking model used in this embodiment is the OSTrack model. During training, the AdamW optimizer is used, with an initial learning rate of 0.0001, a batch size of 16, and 300 training cycles. A weighted combination strategy is used for the loss function, where Focal Loss is used for target category classification, GIoU Loss is used for bounding box regression, and L1 Loss is used for target position error prediction. The total loss function L of the target tracking model satisfies:

[0120] ;in, Represented as classification loss, Represented as bounding box regression loss, This is represented as location prediction loss; and These represent the weighting coefficients for the bounding box regression loss and the location prediction loss, respectively. and All are greater than 0; the specific values ​​are set according to experience, and in this embodiment they are set to 2 and 5 respectively; the combination of the above loss functions can effectively improve the tracking accuracy of the target based on the model in dynamic targets and complex backgrounds.

[0121] In this embodiment, the predictive evaluation index is obtained by performing a reliability assessment on the confidence response map, including:

[0122] Extracting confidence response plots ;

[0123] The maximum peak score in the confidence response graph is extracted as the response score of the confidence response graph; response score satisfy: ;in, Represented as pixel coordinates in the confidence response map;

[0124] Calculate the average peak correlation energy corresponding to the confidence response plot. The calculation of the average peak correlation energy satisfies:

[0125] ;in, This is represented as the minimum peak score in the confidence response graph corresponding to the t-th frame image; Let N represent the response value corresponding to the i-th pixel position in the confidence response graph, and let N represent the total number of pixels in the confidence response graph.

[0126] The mean peak correlation energy (MAP) of several frames within a sliding time window is obtained, and the variance of the MAP is calculated. The variance is used as the APCE stability. The size of the sliding time window is set empirically, and in this embodiment it is set to 15 frames.

[0127] Predictive evaluation metrics are determined based on response scores and APCE stability.

[0128] In this embodiment, considering that during dynamic tracking, the target may be obstructed or drift due to occlusion or rapid movement, it is crucial to jointly determine the reliability of the current tracking result by using the response score and APCE stability.

[0129] In other embodiments, the average peak correlation energy corresponding to the current frame image is used as the APCE stability. The average peak correlation energy is compared with its corresponding preset threshold, and the response score and safety score threshold are combined to jointly determine whether the tracking result of the current frame image is reliable. The preset threshold corresponding to the average peak correlation energy can be set to 45.

[0130] In this embodiment, the final prediction result is obtained by correcting the prediction result based on the prediction evaluation index, including:

[0131] Extract the response score and APCE stability from the predictive evaluation metrics;

[0132] When the response score is greater than the safety score threshold and the APCE stability is less than the stability threshold, the prediction result is taken as the final prediction result. The final target boundary region in the final prediction result is then input into the feature extraction network for feature extraction to obtain the initial target template feature vector corresponding to the current frame image. The initial target template feature vector is then subjected to L2 normalization to obtain the target template feature vector, which is stored in the feature memory. In this embodiment, the feature memory contains several target template feature vectors, which are stored in the feature memory according to time order. The safety score threshold and stability threshold are set empirically, and in this embodiment, they are set to 0.6 and 25, respectively.

[0133] Otherwise, the joint feature map, along with the target center coordinates and the target coarse boundary region from the prediction results, is extracted. Using the target coarse boundary region as a spatial cue, the segmentation model is driven to generate an initial candidate binary mask. The initial candidate binary mask is then processed to obtain a candidate binary mask. The masking operations include binarization, morphological opening and closing operations, and connected component analysis. These masking operations are used to eliminate noise and preserve the target's main body region. In this embodiment, the segmentation model uses the FastSAM model.

[0134] Foreground feature vectors are determined based on joint feature maps and candidate binary masks;

[0135] Extract the latest target template feature vector from the feature memory; in this embodiment, the latest target template feature vector refers to the target template feature vector most recently stored in the feature memory.

[0136] The final prediction result is obtained by determining the consistency between the target template feature vector and the foreground feature vector and updating the prediction result.

[0137] In this embodiment, the determination of the foreground feature vector based on the joint feature map and candidate binary mask includes:

[0138] Extract the joint feature map and candidate binary masks;

[0139] The candidate binary mask is downsampled to the same resolution as the joint feature map, and the foreground feature vector of the mask region corresponding to the candidate binary mask is extracted; the foreground feature vector satisfy:

[0140] ;in, Represented as the joint feature map in the t-th frame image The eigenvalue at the location; This is represented as the mask region corresponding to the k-th candidate binary mask; This is represented as the number of pixels in the masked area.

[0141] In this embodiment, the final prediction result is obtained by determining the consistency between the target template feature vector and the foreground feature vector, and updating the prediction result. This includes:

[0142] Extracting target template feature vectors With foreground feature vector ;

[0143] Calculate the consistency score between the target template feature vector and the foreground feature vector; consistency score The cosine similarity is used for calculation, satisfying the following:

[0144] ;in, It is represented as the L2 norm of a vector;

[0145] When a consistency score is greater than the consistency threshold, the geometric center of the candidate binary mask with a consistency score greater than the consistency threshold is extracted, and the geometric center is used as the coordinate of the final target center; and the minimum bounding rectangle of the candidate binary mask is used as the final target boundary region; the consistency threshold is set according to experience, and in this embodiment it is set to 0.75;

[0146] When the consistency scores of all candidate binary masks are less than or equal to the consistency threshold, the final target center coordinates are predicted by the Kalman filter, and the final target boundary region of the current frame is determined based on the size of the final target boundary region obtained from the previous frame image and the final target center coordinates obtained from the current frame.

[0147] The Kalman filter predicts the final target center coordinates to satisfy the following conditions:

[0148] ;in, Let A be the state vector of the target in the t-th frame; A is the state transition matrix, which represents the transition relationship of the target state from the previous frame to the current frame. This is represented as the state vector corresponding to the target in the (t-1)th frame of the image; Represented as process noise, it is used to describe randomness and uncertainty.

[0149] In this embodiment, the line-of-sight angle measurement data results are obtained by measuring the target line-of-sight angle of the final prediction result, including:

[0150] Extract the final prediction results;

[0151] Extract the resolution W×H of the current frame image and the preset horizontal field of view. With vertical field of view The geometric center (W / 2, H / 2) of the current frame image is used as the center of the observation axis; in this embodiment, the preset horizontal field of view... With vertical field of view Set them to 60 and 45 respectively;

[0152] Calculate the normalized deviation of the final target center coordinates relative to the line-of-sight center. and The normalized deviation satisfies:

[0153] ; Where u and v represent the coordinates of the final target center; ∈[-1, 1], ∈[-1, 1]; and These respectively characterize the degree of deviation of the target from the center of the line of sight in the horizontal and vertical directions;

[0154] By mapping the normalized bias to angle measurements, the horizontal azimuth θ and vertical pitch ψ of the target relative to the observation line of sight are obtained.

[0155] ; Where θ represents the horizontal azimuth angle of the target relative to the observation line of sight, and ψ represents the vertical pitch angle of the target relative to the observation line of sight.

[0156] Obtain the angle measurement corresponding to the current frame image. Then, an exponential smoothing filter is applied to obtain a stable angle measurement. ;

[0157] Exponential smoothing filtering operations satisfy:

[0158] ;

[0159] ;in, For smoothing coefficients, ∈(0,1); The specific value is set based on experience, and in this embodiment it is set to 0.7;

[0160] The line-of-sight angle measurement results are determined based on stable angle measurements and predictive evaluation indicators.

[0161] This embodiment, based on the target center coordinates in the final prediction result, combined with image resolution and preset field of view parameters, uses the geometric center of the image as the observation axis reference to calculate the normalized pixel deviation, and solves it into the horizontal azimuth angle of the target relative to the observation axis through a linear mapping model. To cope with the angle jump caused by the quadruped robot's posture disturbance, an exponential smoothing filter is further applied to the real-time angle sequence. Finally, stable and reliable line-of-sight angle measurement data is encapsulated and output by combining prediction evaluation indicators. Through preset calibration parameters and a normalized mapping model, pixel coordinates are accurately converted into physically meaningful angle quantities, ensuring the accuracy and interpretability of the measurement results. At the same time, time-series filtering effectively suppresses the viewpoint jitter caused by carrier vibration, significantly improving the continuity and smoothness of the output sequence, providing high-quality data support for upper-layer applications.

[0162] Please see Figure 2 A second aspect of this application provides a continuous target line-of-sight angle measurement system for a quadruped robot, comprising: a data acquisition module, an initialization module, a visual monitoring module, a calibration module, and a result generation module; the data acquisition module and the initialization module are connected; the initialization module and the visual monitoring module are connected; the visual monitoring module and the calibration module are connected; and the calibration module and the result generation module are connected.

[0163] Data acquisition module: Acquires initial and real-time image data through data acquisition equipment; the data acquisition equipment includes several sensors, etc.

[0164] Initialization module: Performs initialization operations on the initial image data to obtain the target template region and the target template feature vector;

[0165] Visual monitoring module: Based on real-time image data and target template area, it performs location prediction to obtain prediction results and prediction evaluation indicators;

[0166] Correction module: Corrects the prediction results based on the prediction evaluation indicators to obtain the final prediction result;

[0167] Result generation module: Obtains line-of-sight angle measurement data results by measuring the target line-of-sight angle of the final prediction result.

[0168] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0169] The working principle of this application is as follows: Initialize the target template region and target template feature vector by acquiring initial image data, predict the target center coordinates, the target rough boundary region and prediction evaluation index based on real-time image data, and use the evaluation index to trigger a correction mechanism to perform geometric correction or update the historical state prediction of the prediction results. Finally, combine the preset imaging field of view parameters to solve the corrected final prediction results into line-of-sight angle measurement data. By monitoring the evaluation index and triggering correction, observation drift is suppressed from the source. A hierarchical strategy is adopted to reduce computational overhead. Consistency identification is combined to avoid target misassociation. Temporal filtering is used to smooth the angle sequence, which improves the stability, accuracy and continuity of the line-of-sight angle measurement. This avoids the problem in the prior art where quadruped robots often have unstable imaging due to posture disturbances such as high-frequency vibration during operation, and are easily affected by occlusion interference, resulting in drift or short-term loss of two-dimensional observation results, which leads to low accuracy of line-of-sight angle measurement.

[0170] The above embodiments are only used to illustrate the technical methods of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of this application without departing from the spirit and scope of the technical methods of this application.

Claims

1. A method for continuously measuring the target line-of-sight angle of a quadruped robot, characterized in that, include: Acquire initial image data and real-time image data; Initialization operations are performed on the initial image data to obtain the target template region and the target template feature vector; Location prediction is performed based on real-time image data and target template region to obtain prediction results and prediction evaluation metrics; the prediction results include target center coordinates and target coarse boundary region; the prediction evaluation metrics include response score and APCE stability; The prediction results are corrected based on the prediction evaluation indicators to obtain the final prediction result; the final prediction result includes the final target center coordinates and the final target boundary region; The line-of-sight angle measurement data is obtained by measuring the target line-of-sight angle of the final prediction result; The process of correcting the prediction results based on prediction evaluation indicators to obtain the final prediction result includes: Extract the response score and APCE stability from the predictive evaluation metrics; When the response score is greater than the safety score threshold and the APCE stability is less than the stability threshold, the prediction result is taken as the final prediction result; the final target boundary region in the final prediction result is input into the feature extraction network for feature extraction to obtain the initial target template feature vector corresponding to the current frame image; the initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, and it is stored in the feature memory. Otherwise, extract the joint feature map, as well as the target center coordinates and the target coarse boundary region from the prediction results; use the target coarse boundary region as a spatial cue to drive the segmentation model to generate an initial candidate binary mask; perform masking operations on the initial candidate binary mask to obtain a candidate binary mask; the masking operations include binarization, morphological opening and closing operations, and connected component analysis; Foreground feature vectors are determined based on joint feature maps and candidate binary masks; Extract the latest target template feature vector from the feature memory; The final prediction result is obtained by determining the consistency between the target template feature vector and the foreground feature vector and updating the prediction result.

2. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 1, characterized in that, The initialization operation on the initial image data to obtain the target template region and the target template feature vector includes: Extract initial image data; The initial target template region is obtained by locating the target in the initial image data. The initial target template region is normalized in terms of scale and pixel value to obtain the target template region. The target template region is input into the feature extraction network to extract features and obtain the initial target template feature vector; The initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, and then stored in the feature memory.

3. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 1, characterized in that, The location prediction based on real-time image data and target template region yields prediction results and prediction evaluation metrics, including: Extract real-time image data and target template regions; Obtain the final prediction result of the target in the previous frame image corresponding to the real-time image data; the final prediction result includes the final target center coordinates and the final target boundary region; The search area image is obtained by cropping the real-time image data with the final target center coordinates as the center; the size of the search area image is the size corresponding to the final target boundary region of the target in the previous frame image; The search region image and the target template region are respectively divided into blocks to obtain several corresponding image blocks, and the several image blocks are mapped to corresponding feature label vectors; A joint label sequence is obtained by concatenating the template label sequence and the search region label sequence; the template label sequence consists of several feature label vectors corresponding to the target template region; the search region label sequence consists of several feature label vectors corresponding to the search region image. A joint feature map is obtained by performing feature interaction on the joint label sequence; The joint feature map is input into the prediction head to obtain a confidence response map and a rough boundary region of the target; The target center coordinates are obtained by extracting the center coordinates from the confidence response map; the target center coordinates satisfy: ;in, This is represented as the confidence response map corresponding to the t-th frame image. Represented as the coordinates of the target center corresponding to the t-th frame image; The predictive evaluation index is obtained by performing a reliability assessment on the confidence response map.

4. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 3, characterized in that, The step of performing feature interaction on the joint label sequence to obtain a joint feature map includes: Extract the joint marker sequence; The joint label sequence is input into a single-stream feature extraction network, and a joint feature map is obtained by performing full-cycle feature interaction between the target template region features and the search region image features through a multi-head self-attention mechanism. The attention calculation in the multi-head self-attention mechanism satisfies: Where d represents the feature dimension; It is represented as a normalized exponential function; T represents the attention output; T represents the matrix transpose operation; Q, K, and V represent the query vector, key vector, and value vector obtained by linear mapping of the joint label sequence, respectively.

5. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 3, characterized in that, The method of obtaining predictive evaluation metrics by performing reliability assessment on the confidence response map includes: Extracting confidence response plots ; The maximum peak score in the confidence response graph is extracted as the response score of the confidence response graph; the response score satisfy: ;in, Represented as pixel coordinates in the confidence response map; Calculate the average peak correlation energy corresponding to the confidence response plot. The calculation of the average peak correlation energy satisfies: ;in, This is represented as the minimum peak score in the confidence response graph corresponding to the t-th frame image; Let N represent the response value corresponding to the i-th pixel position in the confidence response graph, and let N represent the total number of pixels in the confidence response graph. The average peak correlation energy (APCE) of several frames within a sliding time window is obtained, and the variance of the average peak correlation energy is calculated. The variance is used as the APCE stability. Predictive evaluation metrics are determined based on response scores and APCE stability.

6. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 1, characterized in that, The determination of the foreground feature vector based on the joint feature map and candidate binary mask includes: Extract the joint feature map and candidate binary masks; The candidate binary mask is downsampled to the same resolution as the joint feature map, and the foreground feature vector of the mask region corresponding to the candidate binary mask is extracted; the foreground feature vector satisfy: ;in, Represented as the joint feature map in the t-th frame image The eigenvalue at the location; This is represented as the mask region corresponding to the k-th candidate binary mask; This is represented as the number of pixels in the masked area.

7. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 1, characterized in that, The step of obtaining the final prediction result by performing consistency judgment on the target template feature vector and the foreground feature vector, and updating the prediction result, includes: Extracting target template feature vectors With foreground feature vector ; Calculate the consistency score between the target template feature vector and the foreground feature vector; the consistency score Calculated using cosine similarity, satisfying: ;in, It is represented as the L2 norm of a vector; When a consistency score is greater than the consistency threshold, the geometric center of the candidate binary mask with the consistency score greater than the consistency threshold is extracted, and the geometric center is used as the final target center coordinate; and the minimum bounding rectangle of the candidate binary mask is used as the final target boundary region. When the consistency scores of all candidate binary masks are less than or equal to the consistency threshold, the final target center coordinates are predicted by the Kalman filter, and the final target boundary region of the current frame is determined based on the size of the final target boundary region obtained from the previous frame image and the final target center coordinates obtained from the current frame.

8. The method for continuous measurement of the target line-of-sight angle of a quadruped robot according to claim 1, characterized in that, The process of obtaining line-of-sight angle measurement data results by measuring the target line-of-sight angle of the final prediction result includes: Extract the final prediction results; Extract the resolution W×H of the current frame image and the preset horizontal field of view. With vertical field of view The geometric center (W / 2, H / 2) of the current frame image is used as the center of the observation axis. Calculate the normalized deviation of the final target center coordinates relative to the line-of-sight center. and The normalized deviation satisfies: ; Where u and v represent the coordinates of the final target center; ∈[-1, 1], ∈[-1, 1]; and These respectively characterize the degree of deviation of the target from the center of the line of sight in the horizontal and vertical directions; By mapping the normalized bias to angle measurements, the horizontal azimuth θ and vertical pitch ψ of the target relative to the observation line of sight are obtained. ; Where θ represents the horizontal azimuth angle of the target relative to the observation axis, and ψ represents the vertical pitch angle of the target relative to the observation axis. Obtain the angle measurement corresponding to the current frame image. Then, an exponential smoothing filter is applied to obtain a stable angle measurement. ; The exponential smoothing filter operation satisfies: ； ;in, For smoothing coefficients, ∈(0,1); The line-of-sight angle measurement results are determined based on stable angle measurements and predictive evaluation indicators.

9. A continuous target line-of-sight angle measurement system for a quadruped robot, characterized in that, include: The system includes a data acquisition module, an initialization module, a visual monitoring module, a calibration module, and a result generation module; the data acquisition module is connected to the initialization module; the initialization module is connected to the visual monitoring module; the visual monitoring module is connected to the calibration module; and the calibration module is connected to the result generation module. The data acquisition module acquires initial image data and real-time image data through a data acquisition device. The initialization module performs initialization operations on the initial image data to obtain the target template region and the target template feature vector; The visual monitoring module: performs location prediction based on real-time image data and target template area to obtain prediction results and prediction evaluation indicators; The correction module corrects the prediction results based on the prediction evaluation indicators to obtain the final prediction result; The process of correcting the prediction results based on prediction evaluation indicators to obtain the final prediction result includes: Extract the response score and APCE stability from the predictive evaluation metrics; When the response score is greater than the safety score threshold and the APCE stability is less than the stability threshold, the prediction result is taken as the final prediction result; the final target boundary region in the final prediction result is input into the feature extraction network for feature extraction to obtain the initial target template feature vector corresponding to the current frame image; the initial target template feature vector is subjected to L2 normalization to obtain the target template feature vector, and it is stored in the feature memory. Otherwise, extract the joint feature map, as well as the target center coordinates and the target coarse boundary region from the prediction results; use the target coarse boundary region as a spatial cue to drive the segmentation model to generate an initial candidate binary mask; perform masking operations on the initial candidate binary mask to obtain a candidate binary mask; the masking operations include binarization, morphological opening and closing operations, and connected component analysis; Foreground feature vectors are determined based on joint feature maps and candidate binary masks; Extract the latest target template feature vector from the feature memory; The final prediction result is obtained by determining the consistency between the target template feature vector and the foreground feature vector and updating the prediction result. The result generation module obtains the line-of-sight angle measurement data by measuring the target line-of-sight angle of the final prediction result.

Images