Double-branch quadruped robot positioning system based on video stream target identification

Abstract

The invention relates to the field of robot positioning, and particularly discloses a double-branch quadruped robot positioning system based on video stream target recognition, which comprises a video stream multi-target recognition model, an image coordinate mapping algorithm and a vector positioning algorithm based on depth information. The continuous positioning operation of the quadruped robot comprises the following steps: synchronously acquiring environment data through an RGB-D camera, and extracting a target ID and a normalized geometric center coordinate by a multi-target recognition model after frame-level matching preprocessing; using a coordinate mapping algorithm to associate an RGB detection result to the depth map, calculating a horizontal distance and a deflection angle of a target relative to the robot by combining internal parameters of the camera, and generating a motion control instruction to realize autonomous approaching; the trajectory is continuously optimized in the moving process, the inter-frame displacement amount is calculated through a vector positioning algorithm, and accumulated deviation is corrected in combination with a dynamic error correction mechanism. According to the invention, the real-time performance and stability of identification and tracking are effectively improved, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of robot units, and particularly to a dual-branch quadruped robot positioning system based on video stream target recognition. Background Art

[0002] To improve the performance of multi-target recognition in real-time video streams and achieve efficient recognition and real-time tracking of multiple targets in network cameras or video files, there have been a large number of studies and applications. Existing technologies usually use video capture libraries (such as VideoCapture in OpenCV) for data acquisition and processing, and widely adopt general datasets such as COCO, PASCAL VOC, and WIDER FACE to train and verify recognition models. In addition, to enhance the adaptability of the model in special scenarios, some studies collect specific environmental data through cameras mounted on quadruped robots, and use depth information algorithms to calculate the horizontal angle of the target, and further combine deviation measurement and correction algorithms to improve the recognition accuracy.

[0003] However, most current real-time video stream recognition models rely on traditional algorithms to process, recognize, and track video data. This method not only has obvious recognition delays and is difficult to truly achieve real-time and efficient target recognition, but also has insufficient robustness and generalization ability, especially performing poorly in complex situations such as environmental light changes, motion blur, and target occlusion. In addition, traditional depth information processing methods have limited ability to correct dynamic deviations during the horizontal angle calculation process, and cannot achieve effective automatic correction, reducing the accuracy and stability of practical applications.

[0004] Based on this, this research proposes a dual-branch quadruped robot positioning system based on video stream target recognition, which can effectively improve the real-time performance and stability of recognition and tracking. Summary of the Invention

[0005] Aiming at the above problems of the existing technology, the present invention provides a dual-branch quadruped robot positioning system based on video stream target recognition, which effectively improves the real-time performance and stability of recognition and tracking, simplifies the structure, reduces the cost, and effectively constructs an efficient, accurate, and stable real-time video stream multi-target recognition system.

[0006] To achieve the above object, the present invention proposes a dual-branch quadruped robot positioning system based on video stream target recognition. The system consists of a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The steps for the system to perform continuous positioning operations on the quadruped robot include: S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environmental video stream, and perform frame-level matching and preprocessing on the dual-modal data; S2. Send the RGB video frame into the multi-object recognition model for video stream. Based on the object detection module of deep learning and combined with the semantic alignment technology, accurately locate the target objects in the video frame, obtain the recognition ID, pixel coordinate bounding box and recognition box of each target in the current frame, and extract the geometric center coordinates of the recognition box of each target for normalization processing; S3. Map the normalized RGB video frame coordinates to the depth video frame, perform regional matching on the detection result and the synchronously obtained depth map, and calculate the horizontal distance and deviation angle of the target object relative to the quadruped robot according to the depth information of the corresponding coordinates in the depth map and the internal parameters of the depth camera; S4. Convert according to the spatial position parameters of the target object into the motion control instructions of the quadruped robot, drive the quadruped robot to adjust the gait and traveling direction, and realize the autonomous approaching motion towards the target object; S5. During the movement, continuously optimize the motion trajectory according to the real-time updated visual data, continuously locate the positions of the corresponding targets in the subsequent frames, obtain the relative displacement amount of the target from the previous frame to the next frame, use the vector positioning algorithm to continuously locate the quadruped robot on the target path, and combine the dynamic error measurement and automatic correction algorithm to determine the cumulative positioning error.

[0007] Preferably, in S2, the multi-object recognition model for video stream adopts a deep learning architecture. By real-time intercepting the video stream of the RGB camera, extract the geometric center coordinates [x, y] of the target in each frame, and lock the specific target by specifying the recognition ID.

[0008] Preferably, in S3, adopt the image coordinate mapping algorithm to realize mapping the normalized RGB video frame coordinates to the depth video frame. The coordinate mapping steps are as follows: S311. Normalize the geometric center coordinates [x, y] of the target in the RGB video frame to obtain the normalized coordinates [x1, y1]; S312. Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, and complete the coordinate mapping from the RGB video frame to the depth video frame.

[0009] Preferably, the calculation formula for the normalized coordinates [x1, y1] is: ; ; In the formula, w1 represents the width of the RGB video frame, represents the length of the RGB video frame.

[0010] Preferably, the calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is: ; ; wherein, w2 represents the width of the depth video frame, represents the length of the depth video frame.

[0011] Preferably, in S3, the steps of calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot according to the depth information corresponding to the coordinates in the depth map and the internal parameters of the depth camera are as follows: S321. Extract the depth information corresponding to the target coordinates [x1', x2'] from the depth video frame ; S322. Combine the internal parameters of the depth camera to convert the pixel coordinates of the target into the three-dimensional coordinates in the camera coordinate system , and calculate the horizontal distance and deviation angle of the target relative to the depth camera; S323. Obtain the relative displacement amount of the target between two consecutive frames through vector operations for continuous tracking and positioning of the target.

[0012] Preferably, the horizontal distance S and the deviation angle are calculated as follows: ; ; wherein, , c x is the abscissa of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y are the focal lengths of the RGB camera.

[0013] Preferably, the system further includes a dynamic error correction module, and is configured to adopt a real-time vision backtracking algorithm and a dynamic error measurement algorithm.

[0014] Preferably, the system adopts a dual-branch parallel computing architecture, where one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

[0015] Preferably, the system directly acquires spatial position information and motion state data by using a depth camera; the multi-target recognition model of the video stream also combines a depth information processing and a dynamic deviation correction module.

[0016] Therefore, the present invention proposes a dual-branch quadruped robot positioning system based on video stream target recognition, and its beneficial effects are as follows: (1) The present invention directly extracts the spatial features and target position information of the video stream through a depth information algorithm, and further combines a real-time dynamic deviation measurement and automatic correction algorithm to improve the real-time performance and stability of recognition and tracking. Moreover, the signal output by the depth information algorithm can directly correlate with the spatial position and motion state of the target, without the need to separately configure a complex auxiliary sensor system, with a simpler structure and lower cost, and can implement a more efficient, accurate and stable real-time video stream multi-target recognition system.

[0017] (2) The present invention uses a dual-branch parallel computing architecture to improve real-time performance, effectively solving the problems of high cost, poor environmental adaptability, and large computational load of existing lidar, VIO, and GPS positioning solutions.

[0018] (3) The visual backtracking positioning system of the present invention realizes real-time and accurate backtracking positioning through depth vision in a specific indoor environment, and the system is reliable and accurate, fully meeting the accuracy requirements in actual use, with a simple structure and low cost, having significant social, economic and technological application values, and having good practical application prospects.

[0019] The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0020] Figure 1 is a complete flowchart of a dual-branch quadruped robot positioning system based on video stream target recognition of the present invention. Detailed Embodiments

[0021] To make the technical solutions, advantages and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be described clearly and completely below. The described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention without creative efforts fall within the protection scope of this application.

[0022] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meaning understood by those of ordinary skill in the art to which the present invention belongs.

[0023] A dual-branch quadruped robot positioning system based on video stream target recognition provided by the present invention consists of a video stream multi-target recognition model, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The video stream multi-target recognition model also combines a depth information processing and dynamic deviation correction module.

[0024] The steps for the dual-branch quadruped robot positioning system to perform continuous positioning operations on the quadruped robot include: S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environmental video stream, and perform frame-level matching and preprocessing on the bimodal data; S2. Send the RGB video frame into the video stream multi-object recognition model. This model is based on the object detection module of deep learning and combines semantic alignment technology to accurately locate the target objects in the video frame, obtain the recognition ID, pixel coordinate bounding box and recognition box of each target in the current frame, and extract the geometric center coordinates of the recognition box of each target for normalization processing; In S2, the video stream multi-object recognition model adopts a deep learning architecture. By real-time intercepting the video stream of the RGB camera, it extracts the geometric center coordinates [x, y] of the target in each frame, and locks specific targets by specifying the recognition ID.

[0025] S3. Map the normalized RGB video frame coordinates to the depth video frame, perform region matching on the detection result and the synchronously obtained depth map, and calculate the horizontal distance and deviation angle of the target object relative to the quadruped robot according to the depth information of the corresponding coordinates in the depth map and the internal parameters of the depth camera; In S3, an image coordinate mapping algorithm is used to map the normalized RGB video frame coordinates to the depth video frame. The coordinate mapping steps are as follows: S311. Normalize the geometric center coordinates [x, y] of the target in the RGB video frame to obtain the normalized coordinates [x1, y1]; Among them, the calculation formula for the normalized coordinates [x1, y1] is: ; ; In the formula, w1 represents the width of the RGB video frame, represents the length of the RGB video frame.

[0026] S312. Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, and complete the coordinate mapping from the RGB video frame to the depth video frame.

[0027] Among them, the calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is: ; ; In the formula, w2 represents the width of the depth video frame, represents the length of the depth video frame.

[0028] In S3, the steps of calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot according to the depth information corresponding to the coordinates in the depth map and the internal parameters of the depth camera are as follows: S321. Extract the depth information corresponding to the target coordinates [x1', x2'] from the depth video frame ; S322. Combine the internal parameters of the depth camera to convert the pixel coordinates of the target into the three-dimensional coordinates in the camera coordinate system , and calculate the horizontal distance and deviation angle of the target relative to the depth camera; S323. Obtain the relative displacement of the target between two consecutive frames through vector operations for continuous tracking and positioning of the target.

[0029] Among them, the horizontal distance S and the deviation angle The calculation formulas are: ; ; In the formula, , c x is the abscissa of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y are the focal lengths of the RGB camera.

[0030] S4. Convert the spatial position parameters of the target object into the motion control instructions of the quadruped robot, drive the quadruped robot to adjust the gait and traveling direction, and realize the autonomous approaching motion towards the target object; S5. During the movement, continuously optimize the motion trajectory according to the real-time updated visual data, continuously locate the position of the corresponding target in the subsequent frames, obtain the relative displacement of the target from the previous frame to the next frame, use the vector positioning algorithm to continuously locate the quadruped robot on the target path, and combine the dynamic error measurement and automatic correction algorithm to determine the cumulative positioning error.

[0031] The quadruped robot positioning system also includes a dynamic error correction module, which is configured to use the real-time visual backtracking algorithm and the dynamic error measurement algorithm.

[0032] The system adopts a dual-branch parallel computing architecture, where one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

[0033] The system directly collects the spatial position information and motion state data by using a depth camera. Specifically, it has the following technical advantages and actual effects: (1) Precise and reliable: After strict experimental verification, within the optimal ranging range (2 - 4 meters) of the depth camera, the distance measurement error of the present invention can be precisely controlled within ±5 cm, and the angle measurement error does not exceed ±1°; within closer (1 - 2 meters) and farther (4 - 6 meters) distance ranges, although the measurement error increases, it is still within an acceptable range. The specific error data is shown in Table 1.

[0034] Table 1 Depth Camera Distance and Angle Error Measurement Table ;

[0035] (2) Efficient and stable: The present invention effectively processes motion data through a real-time visual backtracking algorithm, and combines a dynamic error measurement and automatic correction algorithm, significantly reducing the cumulative positioning error caused by gait slip, attitude control deviation, and environmental light interference during robot motion. The specific experimental data is shown in Tables 2 to 4. The system performs stably in different path tests, with the distance error controlled within ±2.5 cm and the angle error not exceeding ±3° in each path, fully demonstrating the efficient and stable characteristics of the system of the present invention.

[0036] Five different sets of paths were designed for the experiment, each set of paths containing five action instructions, and the error measurements of angles and distances were carried out respectively, as shown in Tables 2 - 4.

[0037] Table 2 Error Measurement Results of Path 1 ;

[0038] Table 3 Error Measurement Results of Path 2 ;

[0039] Table 4 Error Measurement Results of Path 3 ;

[0040] (3) Simplified structure: The present invention directly collects spatial position information and motion state data simultaneously using a depth camera, eliminating the complex structure that requires additional configuration of multiple dedicated sensors in traditional technical solutions, greatly simplifying the system design, and significantly reducing the overall cost and maintenance difficulty.

[0041] (4) Economical and practical: The depth camera technology adopted is mature, the market price is moderate, and it has extremely high cost performance and strong market promotion ability.

[0042] The main sources of error mainly include: First, the measurement error of the depth camera itself increases with the increase of distance, affecting the accuracy of positioning calculation; second, cumulative gait errors may occur during the system movement, such as slip, attitude control deviation, etc., resulting in path deviation; third, ground friction and environmental light conditions (such as too dark or strong light) interfere with visual measurement, further increasing the error.

[0043] The test results show that the system is reliable and accurate, fully meeting the precision requirements in actual use. It has a simple structure and low cost, possessing significant social, economic, and technological application values, and having good practical application prospects.

[0044] Example 1: As Figure 1 shown, a dual-branch quadruped robot positioning system based on video stream object recognition provided by the present invention has the following technical route: First, the RGB camera and depth camera mounted on the quadruped robot synchronously collect the environmental video stream to ensure that the RGB image and the depth map are strictly aligned in time and space. Subsequently, frame-level matching and preprocessing are performed on the collected bimodal data. The RGB image and the RGB video frame are sent into the video stream multi-object recognition model. This model is based on the object detection module of deep learning and combines semantic alignment technology to accurately locate the target object and output its pixel coordinate bounding box. At the same time, the detection result is regionally matched with the synchronously obtained depth map to extract the three-dimensional information of the target object in the depth map. Then, through the depth estimation algorithm and combined with the camera internal parameter matrix, the accurate distance and azimuth deflection angle of the target object relative to the robot are calculated. Finally, these spatial position parameters are converted into robot motion control instructions to drive the quadruped robot to adjust its gait and traveling direction, realizing the autonomous approaching motion towards the target object. And during the movement, the motion trajectory is continuously optimized through the real-time updated visual data, forming a complete closed-loop control system of perception - decision - execution.

[0045] Among them, the video multi-object recognition model takes the video file intercepted from the real-time video stream of the front camera of the quadruped robot as the input of the video multi-object recognition model. The video multi-object recognition model can accordingly output an identification ID (this ID is different for each target) containing all recognizable targets, and the recognition box of the corresponding target in each frame.

[0046] The information of each recognition box contains the coordinates of the upper left corner , and the coordinates of the lower right corner , of the corresponding recognition box in the corresponding RGB video frame. Through the formula , , the geometric center coordinates , of the corresponding target recognition box in the RGB video frame are obtained; through the formula , , the normalized coordinates , , where is the width of the RGB video frame, is the length of the RGB video frame; Subsequently, through the formula , , the geometric center coordinates of the corresponding target recognition box in the RGB video frame are mapped to the geometric center coordinates in the corresponding depth video frame , where w2 represents the width of the depth video frame, represents the length of the depth video frame; According to the depth information corresponding to this coordinate in the depth map , and the internal parameters (focal length, optical center) of the depth camera, the pixel coordinates of the target are converted into three-dimensional coordinates in the camera coordinate system: ; At this time, the horizontal distance S and the deviation angle are calculated as: ; ; Among them, the camera internal parameters are: ;

[0047] Thus, the positioning data of any node on the path from the starting point to the target of the quadruped robot is obtained, and the real three-dimensional coordinates of any node on the target path relative to the starting point (coordinate origin) can be obtained, and continuous positioning operations of the quadruped robot for the target path are performed. The calculation process is as follows: Calculate the vector between any two points on the target path: Assume that there are two points A , and point B , on the path, calculate the vector from A to B, and use the horizontal and vertical coordinates of and the above formulas for calculating the horizontal distance S and the horizontal angle to obtain the horizontal distance and the horizontal angle from A to B, so as to realize the positioning from A to B, and thus realize the positioning of any two points on the target path.

[0048] Therefore, the present invention provides a dual-branch quadruped robot positioning system based on video stream object recognition, which can identify markers in the environment in real time, calculate the distance between the markers and the robot by combining depth information, achieve low-cost and high-robustness autonomous positioning, improve real-time performance by using a dual-branch parallel computing architecture, effectively solve the problems of high cost, poor environmental adaptability, and large computational load of existing lidar, VIO, and GPS positioning solutions, effectively improve the real-time performance and stability of recognition and tracking, simplify the structure, reduce costs, and effectively construct an efficient, accurate, and stable real-time video stream multi-object recognition system.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dual-branch quadruped robot positioning system based on video stream target recognition, characterized in that The system consists of a multi-object recognition model for video streams, an image coordinate mapping algorithm, and a vector positioning algorithm based on depth information. The steps for the quadruped robot to complete continuous positioning operations are as follows: S1. Start the RGB camera and depth camera of the quadruped robot, synchronously collect the environmental video stream, and perform frame-level matching and preprocessing on the bimodal data; S2. Send the RGB video frame into the multi-object recognition model for video streams. This model is based on the object detection module of deep learning and combines semantic alignment technology to accurately locate the target objects in the video frame, obtain the recognition ID, pixel coordinate bounding box, and recognition box of each target in the current frame, and extract the geometric center coordinates of the recognition box of each target for normalization processing; S3. Map the normalized RGB video frame coordinates to the depth video frame, perform regional matching on the detection results and the synchronously obtained depth map, and calculate the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information corresponding to the coordinates in the depth map and the internal parameters of the depth camera; S4. Convert the spatial position parameters of the target object into motion control instructions for the quadruped robot, drive the quadruped robot to adjust its gait and traveling direction, and achieve autonomous approaching motion towards the target object; S5. During the movement process, continuously optimize the motion trajectory according to the real-time updated visual data, continuously locate the positions of the corresponding targets in subsequent frames, obtain the relative displacement of the target between the previous frame and the next frame, use the vector positioning algorithm to perform continuous positioning operations on the quadruped robot on the target path, and combine the dynamic error measurement and automatic correction algorithm to determine the cumulative positioning error.

2. The dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that, In S2, the multi-object recognition model for video streams adopts a deep learning architecture. By real-time intercepting the video stream of the RGB camera, it extracts the geometric center coordinates [x, y] of the target in each frame, and locks specific targets by specifying the recognition ID.

3. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that In S3, an image coordinate mapping algorithm is used to map the normalized RGB video frame coordinates to the depth video frame. The coordinate mapping steps are as follows: S311. Normalize the geometric center coordinates [x, y] of the target in the RGB video frame to obtain the normalized coordinates [x1, y1]; S312. Multiply the normalized coordinates by the screen size of the depth camera to obtain the corresponding coordinates [x1', y2'] of the target in the depth video frame, and complete the coordinate mapping from the RGB video frame to the depth video frame.

4. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 3, characterized in that, The calculation formula for the normalized coordinates [x1, y1] is: ； ； Wherein, w1 represents the width of the RGB video frame, represents the length of the RGB video frame.

5. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 3, characterized in that, The calculation formula for the corresponding coordinates [x1', y2'] of the target in the depth video frame is: ； ； wherein, w2 represents the width of the depth video frame, represents the length of the depth video frame.

6. A dual-branch quadruped robot positioning system based on video stream object recognition according to claim 1, characterized in that, In S3, the steps for calculating the horizontal distance and deviation angle of the target object relative to the quadruped robot based on the depth information corresponding to the coordinates in the depth map and the internal parameters of the depth camera are as follows: S321. Extract the depth information corresponding to the target coordinates [x1', x2'] from the depth video frame ; S322. Combine the internal parameters of the depth camera to convert the pixel coordinates of the target into the three-dimensional coordinates in the camera coordinate system , and calculate the horizontal distance and deviation angle of the target relative to the depth camera; S323. Obtain the relative displacement of the target between two consecutive frames through vector operations for continuous tracking and positioning of the target.

7. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 6, characterized in that, The calculation formulas for the horizontal distance S and the deviation angle θ are: ； ； In the formula, , c x is the abscissa of the optical center of the RGB camera, c y is the ordinate of the optical center of the RGB camera, f x and f y are the focal lengths of the RGB camera.

8. A dual-branch quadruped robot positioning system based on video stream object recognition according to claim 1, characterized in that, The system also includes a dynamic error correction module, which is configured to use real-time visual backtracking algorithms and dynamic error measurement algorithms.

9. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that, The system adopts a dual-branch parallel computing architecture, where one branch processes the RGB video stream for target recognition, and the other branch processes the depth video stream for coordinate mapping and distance calculation.

10. A dual-branch quadruped robot positioning system based on video stream target recognition according to claim 1, characterized in that, The system directly collects spatial position information and motion state data using a depth camera; the multi-target recognition model of the video stream also incorporates a depth information processing and dynamic deviation correction module.

Images