A dual-branch quadruped robot positioning system based on video stream target recognition

Abstract

The present invention relates to the field of robot positioning, and specifically discloses a dual-branch quadruped robot positioning system based on video stream target recognition. The system includes a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The steps for performing continuous positioning operations on the quadruped robot include: synchronously collecting environmental data through an RGB-D camera, extracting the target ID and normalized geometric center coordinates through a multi-target recognition model after frame-level matching preprocessing; using a coordinate mapping algorithm to associate the RGB detection results with the depth map, combining the camera's intrinsic parameters to calculate the horizontal distance and deflection angle of the target relative to the robot, and generating motion control instructions to achieve autonomous approach; continuously optimizing the trajectory during movement, calculating the inter-frame displacement through a vector positioning algorithm, and correcting the accumulated deviation in combination with a dynamic error correction mechanism. The present invention effectively improves the real-time performance and stability of recognition and tracking, and reduces costs.

Description

Technical Field

[0001] The present invention relates to the technical field of robot units, and in particular to a dual-branch quadruped robot positioning system based on video stream target recognition. Background Art

[0002] To improve the performance of multi-target recognition in real-time video streams and enable efficient recognition and real-time tracking of multiple targets within webcams or video files, extensive research and application has been conducted. Existing technologies typically utilize video capture libraries (such as OpenCV's VideoCapture) for data acquisition and processing, and widely use common datasets such as COCO, PASCAL VOC, and WIDER FACE to train and validate recognition models. Furthermore, to enhance the model's adaptability to specific scenarios, some research uses cameras mounted on quadruped robots to collect specific environmental data and utilize depth-based algorithms to calculate the target's horizontal angle. This is further combined with deviation measurement and correction algorithms to improve recognition accuracy.

[0003] However, most current real-time video stream recognition models rely on traditional algorithms to process, identify, and track video data. This approach not only suffers from significant recognition latency, making it difficult to achieve truly real-time, efficient target recognition, but also suffers from insufficient robustness and generalization capabilities, particularly in complex situations such as changing ambient lighting, motion blur, and target occlusion. Furthermore, traditional depth information processing methods have limited ability to correct dynamic deviations during horizontal angle calculation, making it impossible to achieve effective automatic correction, reducing the accuracy and stability of practical applications.

[0004] Based on this, this study proposed a dual-branch quadruped robot positioning system based on video stream target recognition, which can effectively improve the real-time and stability of recognition and tracking. Summary of the Invention

[0005] In response to the above-mentioned problems in the prior art, the present invention provides a dual-branch quadruped robot positioning system based on video stream target recognition, which effectively improves the real-time and stability of recognition and tracking, simplifies the structure, reduces costs, and effectively constructs an efficient, accurate and stable real-time video stream multi-target recognition system.

[0006] To achieve the above objectives, the present invention proposes a dual-branch quadruped robot positioning system based on video stream target recognition. The system is composed of a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The system performs continuous positioning operations for the quadruped robot, including the following steps:

[0007] S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environment video stream, and perform frame-level matching and preprocessing on the dual-modal data;

[0008] S2. Feed the RGB video frame into the video stream multi-target recognition model. This model uses a deep learning-based target detection module and semantic alignment technology to accurately locate the target objects in the video frame, obtain the identification ID, pixel coordinate bounding box, and identification box of each target in the current frame, and extract the geometric center coordinates of each target's identification box for normalization.

[0009] S3, mapping the normalized RGB video frame coordinates to the depth video frame, performing region matching on the detection results and the synchronously acquired depth map, and calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information of the corresponding coordinates in the depth map and the intrinsic parameters of the depth camera;

[0010] S4, converting the spatial position parameters of the target object into motion control instructions for the quadruped robot, driving the quadruped robot to adjust its gait and direction of travel, and realizing autonomous approach movement toward the target object;

[0011] S5. During the movement process, the motion trajectory is continuously optimized based on the real-time updated visual data, and the position of the corresponding target in the subsequent frames is continuously located, the relative displacement of the target from the previous frame to the next frame is obtained, and the vector positioning algorithm is used to continuously position the quadruped robot on the target path. The dynamic error measurement and automatic correction algorithm are combined to determine the cumulative positioning error.

[0012] Preferably, in S2, the video stream multi-target recognition model adopts a deep learning architecture, which extracts the geometric center coordinates [x, y] of the target in each frame by intercepting the video stream of the RGB camera in real time, and locks the specific target by specifying the identification ID.

[0013] Preferably, in S3, an image coordinate mapping algorithm is used to map the normalized RGB video frame coordinates to the depth video frame, and the coordinate mapping steps are:

[0014] S311, normalizing the geometric center coordinates [x, y] of the target in the RGB video frame to obtain normalized coordinates [x1, y1];

[0015] S312: Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, completing the coordinate mapping from the RGB video frame to the depth video frame.

[0016] Preferably, the calculation formula for the normalized coordinates [x1, y1] is:

[0017] ;

[0018] ;

[0019] Where w1 represents the width of the RGB video frame, Indicates the length of the RGB video frame.

[0020] Preferably, the calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is:

[0021] ;

[0022] ;

[0023] Where w2 represents the width of the depth video frame, Indicates the length of the depth video frame.

[0024] Preferably, in S3, the step of calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information of the corresponding coordinates in the depth map and the internal parameters of the depth camera is:

[0025] S321, extracting the depth information corresponding to the target coordinates [x1', x2'] from the depth video frame ;

[0026] S322, combined with the depth camera internal parameters, the pixel coordinates of the target Convert to three-dimensional coordinates in the camera coordinate system , calculate the horizontal distance and deviation angle of the target relative to the depth camera;

[0027] S323. Obtain the relative displacement of the target between two consecutive frames through vector calculation, and continuously track and locate the target.

[0028] Preferably, the horizontal distance S and the deviation angle The calculation formula is:

[0029] ;

[0030] ;

[0031] Where, , c x is the horizontal coordinate of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y is the focal length of the RGB camera.

[0032] Preferably, the system further comprises a dynamic error correction module, and is configured to use a real-time visual backtracking algorithm and a dynamic error determination algorithm.

[0033] Preferably, the system adopts a dual-branch parallel computing architecture, in which one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

[0034] Preferably, the system uses a depth camera to directly collect spatial position information and motion state data; the video stream multi-target recognition model also combines depth information processing and dynamic deviation correction modules.

[0035] Therefore, the present invention proposes a dual-branch quadruped robot positioning system based on video stream target recognition, which has the following beneficial effects:

[0036] (1) The present invention directly extracts spatial features and target location information from video streams through a depth information algorithm, and further combines it with a real-time dynamic deviation measurement and automatic correction algorithm to improve the real-time performance and stability of recognition and tracking. Furthermore, the signal output by the depth information algorithm can directly correlate the spatial position and motion state of the target, eliminating the need for a separate and complex auxiliary sensor system. This results in a simpler structure and lower cost, enabling a more efficient, accurate, and stable real-time video stream multi-target recognition system.

[0037] (2) The present invention utilizes a dual-branch parallel computing architecture to improve real-time performance, effectively solving the problems of high cost, poor environmental adaptability, and heavy computational load in existing lidar, VIO, and GPS positioning solutions.

[0038] (3) The visual retroactive positioning system of the present invention can realize real-time and accurate retroactive positioning through depth vision in a specific indoor environment. The system is reliable and accurate, fully meeting the precision requirements in actual use, has a simple structure and low cost, and has significant social, economic and technological application value, and has good practical application prospects.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a complete flow chart of a dual-branch quadruped robot positioning system based on video stream target recognition of the present invention. DETAILED DESCRIPTION

[0041] To make the technical solutions, advantages, and purposes of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are part of the embodiments of the present invention, not all of them. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0043] This invention provides a dual-branch quadruped robot positioning system based on video stream target recognition. The system consists of a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The video stream multi-target recognition model also incorporates depth information processing and dynamic deviation correction modules.

[0044] The steps of the dual-branch quadruped robot positioning system for performing continuous positioning operations of the quadruped robot include:

[0045] S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environment video stream, and perform frame-level matching and preprocessing on the dual-modal data;

[0046] S2. Feed the RGB video frame into the video stream multi-target recognition model. This model uses a deep learning-based target detection module and semantic alignment technology to accurately locate the target objects in the video frame, obtain the identification ID, pixel coordinate bounding box, and identification box of each target in the current frame, and extract the geometric center coordinates of each target's identification box for normalization.

[0047] In S2, the video stream multi-target recognition model adopts a deep learning architecture, which captures the video stream of the RGB camera in real time, extracts the geometric center coordinates [x, y] of the target in each frame, and locks the specific target by specifying the identification ID.

[0048] S3, mapping the normalized RGB video frame coordinates to the depth video frame, performing region matching on the detection results and the synchronously acquired depth map, and calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information of the corresponding coordinates in the depth map and the intrinsic parameters of the depth camera;

[0049] In S3, the image coordinate mapping algorithm is used to map the normalized RGB video frame coordinates to the depth video frame. The coordinate mapping steps are as follows:

[0050] S311, normalizing the geometric center coordinates [x, y] of the target in the RGB video frame to obtain normalized coordinates [x1, y1];

[0051] Among them, the calculation formula of the normalized coordinate [x1, y1] is:

[0052] ;

[0053] ;

[0054] Where w1 represents the width of the RGB video frame, Indicates the length of the RGB video frame.

[0055] S312: Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, completing the coordinate mapping from the RGB video frame to the depth video frame.

[0056] The calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is:

[0057] ;

[0058] ;

[0059] Where w2 represents the width of the depth video frame, Indicates the length of the depth video frame.

[0060] In S3, the steps for calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot are as follows:

[0061] S321, extracting the depth information corresponding to the target coordinates [x1', x2'] from the depth video frame ;

[0062] S322, combined with the depth camera internal parameters, the pixel coordinates of the target Convert to three-dimensional coordinates in the camera coordinate system , calculate the horizontal distance and deviation angle of the target relative to the depth camera;

[0063] S323. Obtain the relative displacement of the target between two consecutive frames through vector calculation, and continuously track and locate the target.

[0064] Among them, the horizontal distance S and the deviation angle The calculation formula is:

[0065] ;

[0066] ;

[0067] Where, , c x is the horizontal coordinate of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y is the focal length of the RGB camera.

[0068] S4, converting the spatial position parameters of the target object into motion control instructions for the quadruped robot, driving the quadruped robot to adjust its gait and direction of travel, and realizing autonomous approach movement toward the target object;

[0069] S5. During the movement process, the motion trajectory is continuously optimized based on the real-time updated visual data, and the position of the corresponding target in the subsequent frames is continuously located, the relative displacement of the target from the previous frame to the next frame is obtained, and the vector positioning algorithm is used to continuously position the quadruped robot on the target path. The dynamic error measurement and automatic correction algorithm are combined to determine the cumulative positioning error.

[0070] The quadruped robot positioning system also includes a dynamic error correction module and is configured to implement a real-time visual backtracking algorithm and a dynamic error determination algorithm.

[0071] The system adopts a dual-branch parallel computing architecture, where one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

[0072] The system uses a depth camera to directly collect spatial position information and motion status data. It has the following technical advantages and practical effects:

[0073] (1) Accurate and reliable: After rigorous experimental verification, within the optimal ranging range of the depth camera (2-4 meters), the distance measurement error of the present invention can be accurately controlled within ±5 cm, and the angle measurement error does not exceed ±1°; in the closer (1-2 meters) and farther (4-6 meters) distance ranges, although the measurement error increases, it is still within an acceptable range. The specific error data are shown in Table 1.

[0074] Table 1 Depth camera distance and angle error measurement table

[0075] ;

[0076] (2) Efficient and Stable: This invention effectively processes motion data through a real-time visual backtracking algorithm, and combines it with a dynamic error measurement and automatic correction algorithm to significantly reduce the cumulative positioning error caused by gait slip, posture control deviation, and ambient light interference during robot movement. Specific experimental data are shown in Tables 2 to 4. The system performed stably in different path tests, with the distance error in each path controlled within the range of ±2.5 cm and the angle error not exceeding ±3°, fully demonstrating the efficiency and stability of the system of this invention.

[0077] Five different paths were designed for the experiment. Each path contained five action instructions, and the angle and distance errors were measured respectively, as shown in Tables 2-4.

[0078] Table 2 Path error measurement results

[0079] ;

[0080] Table 3 Path 2 error measurement results

[0081] ;

[0082] Table 4 Path three error measurement results

[0083] ;

[0084] (3) Simplified structure: The present invention uses a depth camera to directly and simultaneously collect spatial position information and motion status data, eliminating the need for the complex structure of multiple special sensors required in traditional technical solutions, greatly simplifying the system design, and significantly reducing the overall cost and maintenance difficulty.

[0085] (4) Economical and practical: The depth camera technology is mature and the market price is moderate, with extremely high cost performance and strong market promotion capabilities.

[0086] The main sources of error include: first, the measurement error of the depth camera itself increases with distance, affecting the accuracy of positioning calculation; second, the accumulation of gait errors may occur during the movement of the system, such as slippage, posture control deviation, etc., which lead to path deviation; third, ground friction and ambient lighting conditions (such as too dark or strong light) interfere with visual measurement, further increasing the error.

[0087] The test results show that the system is reliable and accurate, fully meets the precision requirements in actual use, has a simple structure and low cost, has significant social, economic and technological application value, and has good practical application prospects.

[0088] Example 1

[0089] like Figure 1 As shown, according to the present invention, a dual-branch quadruped robot positioning system based on video stream target recognition is provided, and its technical route is:

[0090] First, the RGB camera and depth camera on the quadruped robot synchronously capture the environmental video stream to ensure that the RGB image and depth map are strictly aligned in time and space;

[0091] The collected bimodal data is then subjected to frame-level matching and preprocessing, and the RGB images and RGB video frames are fed into a video stream multi-target recognition model. This model, based on a deep learning-based target detection module and combined with semantic alignment technology, accurately locates the target object and outputs its pixel coordinate bounding box. The detection results are then regionally matched with the simultaneously acquired depth map to extract the target object's 3D information from the depth map.

[0092] Then, the depth estimation algorithm is combined with the camera's intrinsic parameter matrix to calculate the precise distance and azimuth angle of the target object relative to the robot.

[0093] Finally, these spatial position parameters are converted into robot motion control instructions, driving the quadruped robot to adjust its gait and direction of travel, achieving autonomous approach movement toward the target object. During the movement, the motion trajectory is continuously optimized through real-time updated visual data, forming a complete closed-loop control system of perception-decision-execution.

[0094] Among them, the video multi-target recognition model uses the video file captured from the real-time video stream of the quadruped robot's front camera as the input of the video multi-target recognition model. The video multi-target recognition model can output an identification ID containing all identifiable targets (the ID is different for each target) and the identification box of the corresponding target in each frame.

[0095] The information of each identification frame includes the coordinates of the upper left corner of the corresponding identification frame in the corresponding RGB video frame [ , ], and the coordinates of the lower right corner [ , ], through the formula , , get the geometric center coordinates of the corresponding target recognition box in the RGB video frame [ , ]; through the formula , , get the normalized coordinates [ , ],in, is the width of the RGB video frame, is the length of the RGB video frame;

[0096] Then, through the formula , , the geometric center coordinates of the corresponding target recognition frame in the RGB video frame are mapped to the geometric center coordinates in the corresponding depth video frame [ ], where w2 represents the width of the depth video frame, Indicates the length of the depth video frame;

[0097] According to the coordinates in the depth map [ ]Corresponding depth information , and the internal parameters of the depth camera (focal length, optical center), the pixel coordinates of the target Convert to three-dimensional coordinates in the camera coordinate system :

[0098] ;

[0099] At this time, the horizontal distance S and the deviation angle The calculation formula is:

[0100] ;

[0101] ;

[0102] Among them, the camera internal parameters are:

[0103] ;

[0104] Thus, the positioning data of the quadruped robot from the starting point to any node on the target path is obtained, and the real three-dimensional coordinates of any node on the target path relative to the starting point (coordinate origin) can be obtained, and the quadruped robot can perform continuous positioning operations on the target path. The calculation process is:

[0105] Calculate the vector between any two points on the target path: Assume that there are two points A[ , ] and point B[ , ], calculate the vector from A to B ,use The horizontal and vertical coordinates and the above calculated horizontal distance S and horizontal angle The formula can be used to obtain the horizontal distance and horizontal angle from A to B, thereby realizing positioning from A to B, thereby realizing positioning of any two points on the target path.

[0106] Therefore, the present invention provides a dual-branch quadruped robot positioning system based on video stream target recognition, which can identify markers in the environment in real time, calculate the distance between the marker and the robot in combination with depth information, and achieve low-cost, high-robustness autonomous positioning. It uses a dual-branch parallel computing architecture to improve real-time performance, effectively solving the problems of high cost, poor environmental adaptability, and heavy computational load of existing lidar, VIO, and GPS positioning solutions, effectively improving the real-time performance and stability of recognition and tracking, simplifying the structure, reducing costs, and effectively constructing an efficient, accurate, and stable real-time video stream multi-target recognition system.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dual-branch quadruped robot positioning system based on video stream target recognition, characterized in that: The system is composed of a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The steps of the system performing continuous positioning operations on a quadruped robot include: S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environment video stream, and perform frame-level matching and preprocessing on the dual-modal data; S2. Feed the RGB video frame into the video stream multi-target recognition model. This model uses a deep learning-based target detection module and semantic alignment technology to accurately locate the target objects in the video frame, obtain the identification ID, pixel coordinate bounding box, and identification box of each target in the current frame, and extract the geometric center coordinates of each target's identification box for normalization. S3, mapping the normalized RGB video frame coordinates to the depth video frame, performing region matching on the detection results and the synchronously acquired depth map, and calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information of the corresponding coordinates in the depth map and the intrinsic parameters of the depth camera; S4, converting the spatial position parameters of the target object into motion control instructions for the quadruped robot, driving the quadruped robot to adjust its gait and direction of travel, and realizing autonomous approach movement toward the target object; S5. During the movement process, the motion trajectory is continuously optimized based on the real-time updated visual data, and the position of the corresponding target in the subsequent frames is continuously located, the relative displacement of the target from the previous frame to the next frame is obtained, and the vector positioning algorithm is used to continuously position the quadruped robot on the target path. The dynamic error measurement and automatic correction algorithm are combined to determine the cumulative positioning error.

2. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: In S2, the video stream multi-target recognition model adopts a deep learning architecture, which captures the video stream of the RGB camera in real time, extracts the geometric center coordinates [x, y] of the target in each frame, and locks the specific target by specifying the identification ID.

3. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: In S3, the image coordinate mapping algorithm is used to map the normalized RGB video frame coordinates to the depth video frame. The coordinate mapping steps are as follows: S311, normalizing the geometric center coordinates [x, y] of the target in the RGB video frame to obtain normalized coordinates [x1, y1]; S312: Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, completing the coordinate mapping from the RGB video frame to the depth video frame.

4. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 3, characterized in that: The calculation formula for the normalized coordinates [x1, y1] is: ； ； Where w1 represents the width of the RGB video frame, Indicates the length of the RGB video frame.

5. The dual-branch quadruped robot positioning system based on video stream target recognition according to claim 3 is characterized in that: The calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is: ； ； Where w2 represents the width of the depth video frame, Indicates the length of the depth video frame.

6. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: In S3, the steps for calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot are as follows: S321. Extract the depth information corresponding to the target coordinates [x1', y2'] from the depth video frame ; S322, combining the depth camera internal parameters, the pixel coordinates of the target ( ) is converted to the three-dimensional coordinates in the camera coordinate system ( , , ), calculate the horizontal distance and deviation angle of the target relative to the depth camera; S323. Obtain the relative displacement of the target between two consecutive frames through vector calculation, and continuously track and locate the target.

7. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 6, characterized in that: The calculation formula of horizontal distance S and deviation angle θ is: ； ； Where, , c x is the horizontal coordinate of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y is the focal length of the RGB camera.

8. The dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: The system further includes a dynamic error correction module configured to implement a real-time visual backtracking algorithm and a dynamic error determination algorithm.

9. The dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: The system adopts a dual-branch parallel computing architecture, in which one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

10. The dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that: The system uses a depth camera to directly collect spatial position information and motion state data; the video stream multi-target recognition model also combines depth information processing and dynamic deviation correction modules.

Images