A Vision-Based Biomimetic Quadruped Robot Personnel Perception and Following Method and System

Abstract

This invention discloses a method and system for human perception and following in a bionic quadruped robot based on visual information, relating to the field of bionic quadruped robot environmental perception technology. The method includes: acquiring target image data of the environment in which the bionic quadruped robot is located; the target image data including image information of the target to be followed; identifying the target to be followed in the target image data based on an improved multi-target recognition algorithm, and locking the target to be followed based on a multi-target tracking algorithm; and following the target using a following strategy based on image visual information. The improved multi-target recognition algorithm includes a target detection algorithm based on deep learning and a post-processing procedure for target detection based on a weighted non-maximum suppression algorithm. This invention alleviates the technical problem of low intelligence in autonomous following in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of bionic quadruped robot environmental perception technology, and in particular to a bionic quadruped robot human perception and following method and system based on visual information. Background Technology

[0002] A target-following system for a biomimetic quadruped robot can generally be divided into four parts: image acquisition, target detection and recognition, target localization, and target tracking. Target detection and tracking, as an important research direction in the field of computer vision and a crucial component of the visual perception and following system for mobile robots, has consistently attracted numerous scholars to conduct in-depth research and discussion. Target detection can be considered a component of target tracking, primarily serving as the initialization of the target's state during the tracking process.

[0003] Pedestrian detection mainly involves extracting pedestrian target features from images and classifying those features. Traditional pedestrian detection methods use sliding windows of different scales to continuously search the image, generating candidate bounding boxes for pedestrian targets. Then, features are manually extracted from these candidate boxes, and finally, a classifier is used to classify the features.

[0004] Currently, quadruped robots are not only used in simple scenarios, but also more often in complex ones, such as crowded scenes. If the target appears in an overlapping area, detection often fails, reducing the algorithm's average detection rate. This is due to the influence of NMS (Non-Maximum Suppression), which filters out boxes with low confidence that are real targets. The actual effect is easily affected by the threshold. The threshold of NMS is set manually. If the threshold is too high, false deletions will occur, and if the threshold is too low, false detections will occur, which has a significant impact on mAP. Currently, because quadruped robots have weak autonomous following capabilities, some studies use manual remote control to enable the robot to complete the target following task, but this cannot improve the robot's intelligence level. Summary of the Invention

[0005] To address the aforementioned technical problems in existing technologies, embodiments of the present invention provide a biomimetic quadruped robot human perception and following method and system based on visual information. The technical solution is as follows:

[0006] On the one hand, a method for human perception and following in a bionic quadruped robot based on visual information is provided, applied to a bionic quadruped robot; the method includes: acquiring target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed; identifying the target to be followed in the target image data based on an improved multi-target recognition algorithm, and locking the target to be followed based on a multi-target tracking algorithm; and following the target to be followed using a following strategy based on image visual information; wherein, the improved multi-target recognition algorithm includes a target detection algorithm based on deep learning and a target detection post-processing process based on a weighted nonmaximum suppression algorithm.

[0007] Optionally, acquiring target image data of the environment in which the bionic quadruped robot is located includes: acquiring image data of the bionic quadruped robot diagonally above it as the target image data based on an environmental perception sensor.

[0008] Optionally, the improved multi-target recognition algorithm includes: using a deep learning-based target detection algorithm to detect multiple bounding boxes in the target image data; classifying the multiple bounding boxes based on a preset classifier to obtain a classification score for each bounding box; selecting a first bounding box with the highest classification score and calculating its intersection-union ratio (IU / R) with the remaining bounding boxes in the multiple bounding boxes; correcting the classification scores of bounding boxes with IU / R above a preset threshold based on target weights to obtain corrected classification scores; and filtering the multiple bounding boxes using a non-maximum suppression algorithm based on the corrected classification scores; wherein the correction based on target weights includes: s k N represents the classification score of the k-th bounding box; t b represents the preset threshold; k The first identification bounding box (IoU) represents the k-th identification bounding box in the set; M represents the first identification bounding box; and IoU represents the intersection-union ratio (IoU) between the first identification bounding box and the k-th identification bounding box.

[0009] Optionally, the deep learning-based target detection algorithm includes the YOLO algorithm; the multi-target tracking algorithm includes the DeepSort algorithm.

[0010] Optionally, the image-based visual information-based following strategy includes: a pre-following phase and a formal following phase; wherein, the pre-following phase includes: determining whether there is a target to be followed within the field of vision of the bionic quadruped robot; if not, controlling the bionic quadruped robot to rotate in place until the target to be followed is found and identified; after the bionic quadruped robot identifies and locks onto the target to be followed, adjusting its pose so that the distance to the target to be followed is within the target distance and the target to be followed is at the center of the field of vision of the bionic quadruped robot; the formal following phase includes: a forward following strategy and a turning following strategy; the forward following strategy includes: determining the proximity between the bionic quadruped robot and the target to be followed based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot; controlling the forward speed of the bionic quadruped robot based on the proximity; the turning following strategy includes: determining the deviation of the target to be followed within the field of vision of the bionic quadruped robot based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot; controlling the turning angular velocity of the bionic quadruped robot based on the deviation.

[0011] Optionally, the formula for calculating the degree of proximity includes: y1 represents the ordinate of the lower left corner of the bounding box of the target to be followed; center.y represents the ordinate of the center of the target to be followed; img.y represents the height of the camera's field of view; Mov represents the proximity; the formula for calculating the forward speed includes: v x The forward speed is indicated by ; maxv represents the maximum forward speed of the bionic quadruped robot.

[0012] Optionally, the formula for calculating the degree of deviation includes: i mg.x represents the width of the camera's field of view, center.x represents the x-coordinate of the center of the target to be followed; Rot represents the degree of deviation; the formula for calculating the turning angular velocity includes: v rad The value represents the steering angular velocity; maxr represents the maximum value of the steering angular velocity.

[0013] Optionally, the method further includes: sending control commands to the motion host of the bionic quadruped robot based on the UDP protocol to control the bionic quadruped robot to follow the target.

[0014] On the other hand, a vision-based bionic quadruped robot human perception and following system is also provided, applied to a bionic quadruped robot; the system includes: an acquisition module, a target detection module, and a target tracking module; wherein, the acquisition module is used to acquire target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed; the target detection module is used to identify the target to be followed in the target image data based on an improved multi-target recognition algorithm, and to lock the target to be followed based on a multi-target tracking algorithm; the improved multi-target recognition algorithm includes a target detection post-processing procedure based on a deep learning-based target detection algorithm and a weighted non-maximum suppression algorithm; the target tracking module is used to follow the target to be followed using a following strategy based on image visual information.

[0015] On the other hand, a computer-readable storage medium is also provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the method provided in the embodiments of the present invention.

[0016] The beneficial effects of the technical solution provided by the embodiments of the present invention include at least the following: By improving the post-processing of target detection and using a weighted NMS algorithm instead of the traditional NMS algorithm, the present invention can improve the success rate of target detection in crowded scenes and lay the foundation for quadruped robots to respond quickly and flexibly in complex environments; by using deep learning algorithms to detect and follow the navigator target and designing a target following strategy based on visual information, the present invention can meet the real-time requirements of quadruped robot target following and complete the task of quadruped robots following pedestrians more intelligently. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a biomimetic quadruped robot human perception and following method based on visual information provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating a following strategy for a target to be followed, provided by an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of an image captured by a camera by a biomimetic quadruped robot provided in an embodiment of the present invention;

[0021] Figure 4 This is another schematic diagram of a biomimetic quadruped robot captured by a camera according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a biomimetic quadruped robot human perception and following system based on visual information provided in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of a biomimetic quadruped robot provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0025] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0026] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] Figure 1 This is a flowchart illustrating a visual information-based human perception and following method for a bionic quadruped robot, according to an embodiment of the present invention. This method is applied to a bionic quadruped robot. Figure 1 As shown, the method specifically includes the following steps:

[0029] Step S102: Obtain target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed.

[0030] Step S104: Based on the improved multi-target recognition algorithm, identify the target to be followed in the target image data, and lock the target to be followed based on the multi-target tracking algorithm.

[0031] Step S106: A following strategy based on image visual information is adopted to follow the target to be followed.

[0032] The improved multi-target recognition algorithm includes a target detection algorithm based on deep learning and a target detection post-processing process based on a weighted nonmaximum suppression algorithm.

[0033] Specifically, step S102 further includes: acquiring image data of the bionic quadruped robot from an oblique angle based on an environmental perception sensor as target image data. Preferably, the environmental perception sensor includes a camera.

[0034] Specifically, a camera is used as an environmental image acquisition device to obtain image data from above the quadruped robot in real time. Optionally, the camera is placed at a 30-degree angle above the quadruped robot's head to facilitate the acquisition of RGB images of pedestrian targets.

[0035] Specifically, improved multi-target recognition algorithms include:

[0036] A deep learning-based target detection algorithm is used to detect multiple bounding boxes in target image data.

[0037] Multiple bounding boxes are classified and identified based on a preset classifier, and a classification score is obtained for each bounding box.

[0038] Select the first bounding box with the highest classification score, and calculate the intersection-union ratio (IUU) with the remaining bounding boxes in the multiple bounding boxes.

[0039] The classification scores of the bounding boxes with an intersection-union ratio (IU) above a preset threshold are corrected based on the target weights to obtain the corrected classification scores.

[0040] Based on the corrected classification scores, a non-maximum suppression algorithm is used to filter multiple recognition bounding boxes.

[0041] The correction based on the target weights includes:

[0042]

[0043] s k N represents the classification score of the k-th bounding box; t Indicates a preset threshold; b k represents the k-th bounding box in the set; M represents the first bounding box; IoU represents the intersection-union ratio of the first bounding box and the k-th bounding box.

[0044] Preferably, in this embodiment of the invention, the target detection algorithm based on deep learning includes the YOLO (You Only Look Once) algorithm; the multi-target tracking algorithm includes the DeepSort algorithm.

[0045] In this embodiment of the invention, the perception layer interface of the bionic quadruped robot uses ROS topics to input camera images into the target recognition network. The recognition algorithm of the bionic quadruped robot adopts the YOLO (You Only Look Once) multi-target recognition algorithm, and its target detection post-processing process is improved: a weight-based NMS (Non-Maximum Suppression) algorithm is used instead of the traditional NMS algorithm to improve the target detection success rate in scenes with overlapping people.

[0046] Specifically, the weighted NMS algorithm first selects the bounding box M with the highest classification score and calculates its IoU (Intersection over Union) compared with other bounding boxes. For bounding boxes with an IoU greater than a set threshold, an appropriate method is used to adjust their classification scores instead of directly excluding them to avoid missed detections. This adjustment method works by reducing the classification score of bounding boxes that highly overlap with M based on the IoU, while moderately reducing the classification score of bounding boxes with less overlap. For example, when the IoU between a bounding box and M is large (e.g., 0.9), it indicates they may belong to the same target, so their scores should be reduced, and the weighted method allows for more flexible reduction. When the IoU is small (e.g., 0.7), it indicates they may be adjacent but mutually occluded different targets, so only a slight reduction in their classification scores is needed. This weighted NMS method considers the IoU when adjusting scores, thus handling overlapping targets more accurately. The following is the weighting method:

[0047]

[0048] The expression above uses s k N represents the score of the k-th detection box; t This indicates the threshold value set for the intersection-union ratio; b k Let represent the k-th candidate box in the set; M represents the highest-scoring ground truth box; and IoU represents the intersection-union ratio (IoU) between the highest-scoring ground truth box and the k-th candidate box. From the above formula, we can see that when IoU is less than the IoU threshold, the classification score remains unchanged; when IoU is greater than the IoU threshold, the larger the IoU, the smaller the weighting value, and the greater the reduction in the corrected score; conversely, the smaller the IoU, the larger the weighting value, and the less the reduction in the corrected score.

[0049] Then, the YOLO algorithm is used to process the environmental images captured by the camera to identify various objects in the images. By analyzing the multi-type detection results output by the network, objects belonging to the "Person" category are extracted, and a unique ID is assigned to each pedestrian. Subsequently, these detection results are passed to the multi-object tracking algorithm module to track pedestrians in consecutive frames of images. The multi-object tracking algorithm uses DeepSort, which enables the quadruped robot to have a certain degree of anti-occlusion capability, meaning that even if the target person is occluded by an obstacle and then reappears, the ID remains unchanged. The pedestrian ID information is organized and stored in a linked list for subsequent selection of the target individual to be followed.

[0050] Figure 2 This is a flowchart illustrating a following strategy for a target to be followed, according to an embodiment of the present invention. Figure 2 As shown, step S106 includes: a pre-following phase and a formal following phase; wherein,

[0051] The pre-follow phase includes:

[0052] Determine if there is a target to be followed within the field of vision of the bionic quadruped robot; if not, control the bionic quadruped robot to rotate in place until the target to be followed is found and identified.

[0053] After the bionic quadruped robot identifies and locks onto the target to be followed, it adjusts its pose so that the distance to the target is within the target distance and the target is in the center of the bionic quadruped robot's field of vision.

[0054] The formal following phase includes: forward following strategy and turn-following strategy;

[0055] Forward following strategies include:

[0056] Based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot, the degree of proximity between the bionic quadruped robot and the target to be followed is determined.

[0057] The forward speed of the bionic quadruped robot is controlled based on the degree of proximity.

[0058] Turning to follow strategies include:

[0059] Based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot, the degree of deviation of the target to be followed within the field of vision of the bionic quadruped robot is determined.

[0060] The turning angular velocity of the bionic quadruped robot is controlled based on the degree of deviation.

[0061] Specifically, the formulas for calculating the degree of approximation include:

[0062]

[0063] y1 represents the ordinate of the lower left corner of the bounding box of the target to be followed; center.y represents the ordinate of the center of the target to be followed; img.y represents the height of the camera's field of view; Mov represents the proximity.

[0064] The formula for calculating forward speed includes:

[0065]

[0066] v x represents the forward speed; maxv represents the maximum forward speed of the bionic quadruped robot.

[0067] Specifically, the formulas for calculating the degree of deviation include:

[0068]

[0069] img.x represents the width of the camera's field of view, center.x represents the x-coordinate of the center of the target to be followed; Rot represents the degree of deviation.

[0070] The formula for calculating the steering angular velocity includes:

[0071]

[0072] v rad represents the steering angular velocity; maxr represents the maximum value of the steering angular velocity.

[0073] Specifically, the pre-following phase is the preparation stage before the bionic quadruped robot follows a pedestrian. It mainly involves searching for, identifying, and locating the pedestrian target, as well as adjusting the quadruped robot's pose. When the quadruped robot initiates following, it first checks if there is a target within the camera's field of view. If no pedestrian target is found, the sensing host sends a UDP command to the motion host to control the quadruped robot to rotate in place at a constant angular velocity α until the target is found and identified. Once the quadruped robot identifies and locates the pedestrian, if the robot is far from the target or the target is not in the center of the quadruped robot's field of view, the quadruped robot sends control commands to automatically adjust its pose to achieve the optimal following position.

[0074] The formal following phase occurs after the pre-following phase commands are completed. The robot is in the optimal following position and performs the task of following the pedestrian, which mainly involves forward following and turning following.

[0075] Figure 3It is a schematic diagram of the image captured by the bionic quadruped robot through the camera, which can be regarded as a coordinate system. When the video stream is input into the YOLO object recognition network, the recognition frames of each object appearing in the video can be obtained, and thus the coordinate information of the lower left corner and the upper right corner of the object frame can be obtained. Therefore, for each object enclosed by the frame, their center point coordinate information (center.x, center.y) can be calculated.

[0076] In the forward following strategy provided by the embodiment of the present invention, a proportionality coefficient Mov and a threshold k are defined. In this embodiment, k is taken as 30. Mov is used to calculate the normalized horizontal position of the height of the object frame relative to the height of the camera's field of view, and it is mapped to the interval [0, 90], representing the degree of the robot approaching the object. The calculation method is as follows:

[0077]

[0078] In the above formula, y1 represents the ordinate of the lower left corner of the detected frame of the pedestrian being tracked; center.y represents the ordinate of the center of the human detection frame; img.y represents the height of the camera's field of view. The value of Mov can reflect the degree of the bionic quadruped robot approaching the tracking object in the case of linear following, and then control the forward speed of the bionic quadruped robot. When 0 < Mov < 30, it means that the target person is relatively close to the bionic quadruped robot, and the speed needs to be reduced; when 30 < Mov < 60, it means that the target person is at a moderate distance from the bionic quadruped robot, and the speed needs to be maintained; when Mov > 60, it means that the target person is relatively far from the bionic quadruped robot, and the speed needs to be increased. The relationship between the proportionality coefficient and the forward speed is as follows:

[0079]

[0080] v in the above formula x represents the running speed of the bionic quadruped robot; maxv represents the maximum forward speed of the bionic quadruped robot.

[0081] Figure 4 It is another schematic diagram of the image captured by the bionic quadruped robot through the camera. As Figure 4 shown, in the self-rotation following strategy provided by the embodiment of the present invention, an automatic steering coefficient Rot is defined. The normalized horizontal position of the center point of the object frame is calculated and mapped to the interval [-90, 90], representing the rotation direction of the robot. The calculation method is as follows:

[0082]

[0083] In the above example, `img.x` represents the width of the camera's field of view, and `center.x` represents the horizontal coordinate of the center of the pedestrian target detection box. The `Rot` parameter determines the deviation of the target from the center of the image, thereby controlling the steering of the bionic quadruped robot. When `Rot < 0`, it indicates that the target person is to the left of the robot's direction of travel, requiring a leftward angular velocity from the motion unit; when `Rot = 0`, it indicates that the target person is directly in front of the robot, in a suitable position, and no steering is needed; when `Rot > 0`, it indicates that the target person is to the right of the robot's direction of travel, requiring a rightward angular velocity from the motion unit. The relationship between `Rot` and angular velocity is as follows:

[0084]

[0085] v in the above formula rad represents the angular velocity of the biomimetic quadruped robot; maxr represents the maximum value of the angular velocity.

[0086] The method provided in this embodiment of the invention further includes: sending control commands to the motion host of the bionic quadruped robot based on the UDP protocol to control the following motion of the bionic quadruped robot to the target to be followed.

[0087] Specifically, control commands are sent to the quadruped robot's motion host using the UDP protocol, requiring data conversion during this process: converting the topic message type of the control commands to and from the UDP format. For example... Figure 5 The OpenCV library is used to display the camera's field of view on the desktop as the first-person perspective of the quadruped robot following the pedestrian, while information such as following speed and recognition status is drawn on the camera screen.

[0088] Example 2

[0089] Figure 5 This is a schematic diagram of a vision-based bionic quadruped robot human perception and following system according to an embodiment of the present invention. The system is applied to a bionic quadruped robot. As shown in the figure, the system includes: an acquisition module 10, a target detection module 20, and a target tracking module 30.

[0090] Specifically, the acquisition module 10 is used to acquire target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed.

[0091] The target detection module 20 is used to identify the target to be followed in the target image data based on the improved multi-target recognition algorithm, and to lock the target to be followed based on the multi-target tracking algorithm.

[0092] Specifically, the improved multi-target recognition algorithm includes a deep learning-based target detection algorithm and a weight-based nonmaximum suppression algorithm for target detection post-processing.

[0093] The target tracking module 30 is used to follow the target by employing a following strategy based on image visual information.

[0094] Specifically, improved multi-target recognition algorithms include:

[0095] A deep learning-based target detection algorithm is used to detect multiple bounding boxes in target image data.

[0096] Multiple bounding boxes are classified and identified based on a preset classifier, and a classification score is obtained for each bounding box.

[0097] Select the first bounding box with the highest classification score, and calculate the intersection-union ratio (IUU) with the remaining bounding boxes in the multiple bounding boxes.

[0098] The classification scores of the bounding boxes with an intersection-union ratio (IU) above a preset threshold are corrected based on the target weights to obtain the corrected classification scores.

[0099] Based on the corrected classification scores, a non-maximum suppression algorithm is used to filter multiple recognition bounding boxes.

[0100] The correction based on the target weights includes:

[0101]

[0102] s k N represents the classification score of the k-th bounding box; t Indicates a preset threshold; b k represents the k-th bounding box in the set; M represents the first bounding box; IoU represents the intersection-union ratio of the first bounding box and the k-th bounding box.

[0103] Preferably, in this embodiment of the invention, the target detection algorithm based on deep learning includes the YOLO (You Only Look Once) algorithm; the multi-target tracking algorithm includes the DeepSort algorithm.

[0104] Specifically, image-based visual information-based following strategies include: a pre-following phase and a formal following phase; among which,

[0105] The pre-follow phase includes:

[0106] Determine if there is a target to be followed within the field of vision of the bionic quadruped robot; if not, control the bionic quadruped robot to rotate in place until the target to be followed is found and identified.

[0107] After the bionic quadruped robot identifies and locks onto the target to be followed, it adjusts its pose so that the distance to the target is within the target distance and the target is in the center of the bionic quadruped robot's field of vision.

[0108] The formal following phase includes: forward following strategy and turn-following strategy;

[0109] Forward following strategies include:

[0110] Based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot, the degree of proximity between the bionic quadruped robot and the target to be followed is determined.

[0111] The forward speed of the bionic quadruped robot is controlled based on the degree of proximity.

[0112] Turning to follow strategies include:

[0113] Based on the coordinates of the target to be followed within the field of vision of the bionic quadruped robot, the degree of deviation of the target to be followed within the field of vision of the bionic quadruped robot is determined.

[0114] The turning angular velocity of the bionic quadruped robot is controlled based on the degree of deviation.

[0115] Specifically, the formulas for calculating the degree of approximation include:

[0116]

[0117] y1 represents the ordinate of the lower left corner of the bounding box of the target to be followed; center.y represents the ordinate of the center of the target to be followed; img.y represents the height of the camera's field of view; Mov represents the proximity.

[0118] The formula for calculating forward speed includes:

[0119]

[0120] v x represents the forward speed; maxv represents the maximum forward speed of the bionic quadruped robot.

[0121] Specifically, the formulas for calculating the degree of deviation include:

[0122]

[0123] img.x represents the width of the camera's field of view, center.x represents the x-coordinate of the center of the target to be followed; Rot represents the degree of deviation.

[0124] The formula for calculating the steering angular velocity includes:

[0125]

[0126] v rad represents the steering angular velocity; maxr represents the maximum value of the steering angular velocity.

[0127] Figure 6 This is a schematic diagram of a biomimetic quadruped robot provided according to an embodiment of the present invention. Figure 6 As shown, the target detection module 20 and the target tracking module 30 are both mounted on the sensing host 100 of the bionic quadruped robot; the sensing host 100 of the bionic quadruped robot is connected to the motion host 200 of the bionic quadruped robot via the UDP protocol; the acquisition module 10 includes a vision camera.

[0128] The sensor host 100 of the bionic quadruped robot is also used to send control commands to the motion host 200 of the bionic quadruped robot based on the UDP protocol, so as to control the following motion of the bionic quadruped robot towards the target.

[0129] This invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method in Embodiment 1 above.

[0130] It should be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

[0131] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0132] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A biomimetic quadruped robot human perception and following method based on visual information, characterized in that, Applied to biomimetic quadruped robots; the method includes: Acquire target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed; Based on an improved multi-target recognition algorithm, the target to be followed is identified in the target image data, and the target to be followed is locked based on a multi-target tracking algorithm; A following strategy based on image visual information is adopted to follow the target to be followed; The improved multi-target recognition algorithm includes a target detection algorithm based on deep learning and a target detection post-processing process based on a weighted nonmaximum suppression algorithm. The improved multi-target recognition algorithm includes: A deep learning-based target detection algorithm is used to detect multiple bounding boxes in the target image data. The multiple recognition bounding boxes are classified and identified based on a preset classifier to obtain a classification score for each recognition bounding box; Select the first identification bounding box with the highest classification score, and calculate the intersection-union ratio with the remaining identification bounding boxes in the plurality of identification bounding boxes; The classification scores of the recognition bounding boxes with an intersection-union ratio above a preset threshold are corrected based on the target weight to obtain the corrected classification scores. Based on the corrected classification scores, the multiple recognition bounding boxes are filtered using a non-maximum suppression algorithm. The correction based on the target weight includes: s k N represents the classification score of the k-th bounding box; t b represents the preset threshold; k The first identification bounding box (IoU) represents the k-th identification bounding box in the set; M represents the first identification bounding box; and IoU represents the intersection-union ratio (IoU) between the first identification bounding box and the k-th identification bounding box.

2. The method according to claim 1, characterized in that, Acquiring target image data of the environment in which the bionic quadruped robot is located includes: Based on the environmental perception sensor, the image data of the bionic quadruped robot diagonally above is acquired as the target image data.

3. The method according to claim 1, characterized in that, The deep learning-based target detection algorithm includes the YOLO algorithm; the multi-target tracking algorithm includes the DeepSort algorithm.

4. The method according to claim 1, characterized in that, The image-based visual information-based following strategy includes: a pre-following phase and a formal following phase; wherein, The pre-following phase includes: Determine whether there is a target to be followed within the field of vision of the bionic quadruped robot; if not, control the bionic quadruped robot to rotate in place until the target to be followed is found and identified; After the bionic quadruped robot identifies and locks onto the target to be followed, it adjusts its pose so that the distance to the target to be followed is within the target distance and the target to be followed is in the center of the bionic quadruped robot's field of vision. The formal following phase includes: forward following strategy and turning following strategy; The forward following strategy includes: Based on the coordinates of the target to be followed within the field of view of the bionic quadruped robot, the degree of proximity between the bionic quadruped robot and the target to be followed is determined; Based on the degree of proximity, the forward speed of the bionic quadruped robot is controlled; The steering follow strategy includes: Based on the coordinates of the target to be followed within the field of view of the bionic quadruped robot, the degree of deviation of the target to be followed within the field of view of the bionic quadruped robot is determined. The turning angular velocity of the bionic quadruped robot is controlled based on the degree of deviation.

5. The method according to claim 4, characterized in that, The formula for calculating the degree of proximity includes: y1 represents the ordinate of the lower left corner of the bounding box of the target to be followed; center.y represents the ordinate of the center of the target to be followed; img.y represents the height of the camera's field of view; Mov represents the proximity. The formula for calculating the forward speed includes: v x The forward speed is indicated by ; maxv represents the maximum forward speed of the bionic quadruped robot.

6. The method according to claim 4, characterized in that, The formula for calculating the degree of deviation includes: img.x represents the width of the camera's field of view, center.x represents the x-coordinate of the center of the target to be followed, and Rot represents the degree of deviation. The formula for calculating the steering angular velocity includes: v rad The value represents the steering angular velocity; maxr represents the maximum value of the steering angular velocity.

7. The method according to claim 1, characterized in that, The method further includes: sending control commands to the motion host of the bionic quadruped robot based on the UDP protocol to control the bionic quadruped robot to follow the target.

8. A biomimetic quadruped robot human perception and following system based on visual information, characterized in that, This system is applied to a biomimetic quadruped robot; it includes: an acquisition module, a target detection module, and a target tracking module; wherein, The acquisition module is used to acquire target image data of the environment in which the bionic quadruped robot is located; the target image data includes image information of the target to be followed; The target detection module is used to identify the target to be followed in the target image data based on an improved multi-target recognition algorithm, and to lock the target to be followed based on a multi-target tracking algorithm; The improved multi-target recognition algorithm includes a target detection algorithm based on deep learning and a target detection post-processing process based on a weighted nonmaximum suppression algorithm. The target tracking module is used to follow the target to be followed by employing a following strategy based on image visual information; The improved multi-target recognition algorithm includes: A deep learning-based target detection algorithm is used to detect multiple bounding boxes in the target image data. The multiple recognition bounding boxes are classified and identified based on a preset classifier to obtain a classification score for each recognition bounding box; Select the first identification bounding box with the highest classification score, and calculate the intersection-union ratio with the remaining identification bounding boxes in the plurality of identification bounding boxes; The classification scores of the recognition bounding boxes with an intersection-union ratio above a preset threshold are corrected based on the target weight to obtain the corrected classification scores. Based on the corrected classification scores, the multiple recognition bounding boxes are filtered using a non-maximum suppression algorithm. The correction based on the target weight includes: s k N represents the classification score of the k-th bounding box; t b represents the preset threshold; k The first identification bounding box (IoU) represents the k-th identification bounding box in the set; M represents the first identification bounding box; and IoU represents the intersection-union ratio (IoU) between the first identification bounding box and the k-th identification bounding box.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method of any one of claims 1-7.

Images