A Blind Path Following Method for Quadruped Robots Based on a Fast Segmentation Network

Abstract

The present invention discloses a method for a quadruped robot to follow a blind path based on a fast segmentation network, which includes the following steps: segmenting the blind path line and calculating the vision-based offset. The present invention adopts an adaptive fast segmentation strategy of macro-block assignment and local fine calculation, and only classifies and infers and calculates the mixed pixel blocks. For the pixel regions that are all background or foreground, direct assignment is performed without classification inference calculation, which greatly improves the inference speed of the model on the edge computing card without any impact on the accuracy. The present invention adopts a vision-based offset calculation algorithm. Through the binary image obtained in step A, the binary image is used as the input of step B to obtain the horizontal offset and the heading angle offset, and the horizontal offset and the heading angle offset are used as the input of the underlying PID control algorithm to control the quadruped robot, thereby effectively ensuring the relative posture of the quadruped robot and the blind path. The benchmark test offset accuracy is ±5 cm.

Description

Technical Field

[0001] The present invention relates to a method for a quadruped robot to patrol a blind path, in particular to a method for a quadruped robot to patrol a blind path based on a fast segmentation network. Background Art

[0002] Guide dogs play an important role in assisting blind people to travel. However, the training cycle of guide dogs is long, the cost is high, and there is a long-term shortage in supply. According to statistics from authoritative departments: As of 2022, there are only more than 200 guide dogs in the whole country, while the number of blind people exceeds 17 million. It takes one and a half years to train one guide dog, and the cost is as high as more than 200,000 yuan, making it difficult to promote on a social scale. This phenomenon has attracted the attention of the state and the government. Therefore, using a quadruped robot to replace a guide dog to complete the blind path patrolling method has become one of the important application scenarios of the country's new generation of artificial intelligence. Due to the complex situation of the blind path patrolling task, deep learning has become the mainstream method in the blind path patrolling task. In order to improve the accuracy of blind path patrolling, a deep model is constructed and trained in a data-driven manner. The deep model extracts features from the original image data to obtain a higher-level representation, so as to complete the established task.

[0003] At present, the structure of the deep model is too complex and usually consumes a large amount of computing resources. However, the computing power of the current mobile computing card is limited and usually difficult to bear the computing power required for the inference of the current mainstream model, resulting in inference delay and greatly increasing the safety problems of visually impaired people traveling. And too deep a network model requires a large amount of training data and consumes a large amount of human resources. So far, it is still an urgent problem to realize the blind path patrolling of a quadruped robot based on a fast segmentation network. Summary of the Invention

[0004] To solve the above problems existing in the prior art, the present invention proposes a method for a quadruped robot to patrol a blind path based on a fast segmentation network, which has a simple structure, a fast inference speed, and a low training cost.

[0005] To achieve the above object, the technical solution of the present invention is as follows: A method for a quadruped robot to patrol a blind path based on a fast segmentation network, comprising the following steps:

[0006] A. Segment the blind path line

[0007] At the boot-ready stage, the quadruped robot captures and collects the information of the surrounding blind path lines and its own relative position through the image acquisition device it carries, and adopts an adaptive fast segmentation strategy of macro-block assignment and local fine calculation to quickly segment the blind path line and obtain a binary image. The steps are as follows:

[0008] A1. Obtain scene observation information, perform per-unit-frame sampling on the collected observation information, and convert the sampled observation information to the HSV color space through the HSV conversion function. As shown in formula (1):

[0009]

[0010] Among them, r, g, and b respectively represent the red, green, and blue color channel information of the original observation, H, S, and V respectively represent hue, saturation, and brightness, that is, the 3 channel information of the HSV color space, max and min respectively represent the maximum and minimum of the pixel points in the three channels of r, g, and b, Concat represents the merge operation, and X represents the image in the HSV color space.

[0011] A2. After encoding X as the input through the convolutional network, a feature map with a height of h, a width of w, and 7 channels is obtained. As shown in formula (2):

[0012] feature = backbone(X) (2)

[0013] Among them, backbone represents the convolutional network, feature represents the feature map, the first 3 channels are defined as the class index, denoted as C = [c 1 , c 2 , c 3 , and the last 4 channels are defined as the classification feature vector, denoted as M = [ω 1 , ω 2 , ω 3 , b].[[]]END

[0014] A3. Perform the softmax function operation on C to obtain three categories, namely the background, foreground, and mixed categories. The foreground category is the blind path, and the class index is denoted as 0; the background category is the area outside the blind path, and the class index is denoted as 1; the mixed category is the intersection area of the foreground and background, and the class index is denoted as 2; mark the categories in one-hot encoding form. As shown in formula (3):

[0015]

[0016] Among them, Softmax represents the softmax function, i represents the class index, p(i) represents the probability value that this pixel point is the i-th class, and argmax(P) represents the class index corresponding to the maximum value in P.

[0017] A4. For the feature map components of the foreground category or background category, directly mark all the pixels within the receptive field area of the original HSV image X corresponding to feature as the foreground category or background category.

[0018] ​A5. For the feature map components of the mixed category, perform a dot product operation on all the pixels within the receptive field region of the original HSV image X corresponding to the feature with the classification feature vector M. The pixel points with a dot product result greater than 0 are classified as the foreground category, otherwise as the background category, to obtain a binary image. As shown in formula (4):

[0019]

[0020] where H i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the H channel of the original HSV image X, S i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the S channel of the original HSV image X, V i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the V channel of the original HSV image X, * represents the dot product operation, φ i,j is the class prediction value of the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the original HSV image X, Φ i,j is the binary representation of the classification result of the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the original HSV image X.

[0021] B. Calculate the vision-based offset

[0022] B1. Scan the horizontal midline of the binary image from left to right. The first pixel point that changes from 0 to 1 is recorded as the left anchor point, and the distance from the left anchor point to the left edge of the image is denoted as a left .

[0023] B2. Scan the horizontal midline of the binary image from right to left. The first pixel point that changes from 0 to 1 is recorded as the right anchor point, and the distance from the right anchor point to the left edge of the image is denoted as a right .

[0024] B3. Let x 1 = W / 2 - a left , x 2 = a right - W / 2, Δx = x 1 - x 2 . When Δx > 0, it indicates that the quadruped robot is biased to the right; when Δx < 0, it indicates that the quadruped robot is biased to the left; when Δx = 0, it indicates that the quadruped robot is in an ideal state.

[0025] B4. Using the left anchor point as the reference point, search upward to obtain the discrete point set of the left side line, denoted as p l , and use the least squares method to fit the mathematical quadratic equation f left (x) of the left side line of the blind path. Find the angle between the tangent line and the perpendicular bisector of f left (x) at the left anchor point, denoted as θ 1 . The expression of the above mathematical quadratic equation is as follows:

[0026] f left (x) = a l x 2 + b l x + c l (5)

[0027] where x is the abscissa of point p l a l , b l , c l are respectively the quadratic coefficient, the linear coefficient and the constant of the mathematical quadratic equation.

[0028] B5. Taking the right anchor point as the reference point, the set of discrete points of the left side line is searched upward and denoted as p r . Using the least squares method, the mathematical quadratic equation f right (x) of the right side line is fitted, and the angle between the tangent line and the perpendicular bisector of f right (x) at the right anchor point is denoted as θ 2 . The expression of the above mathematical quadratic equation is as follows:

[0029] f right (x) = a r x 2 + b r x + c r (6)

[0030] where x is the abscissa of point p r a r , b r , c r are respectively the quadratic coefficient, the linear coefficient and the constant of the mathematical quadratic equation.

[0031] B6. Let Δθ = (θ 1 + θ 2 ) / 2. When Δθ > 0, the heading angle of the quadruped robot is biased to the left; when Δθ < 0, the heading angle of the quadruped robot is biased to the right; when Δθ = 0, the heading angle of the quadruped robot is in an ideal state.

[0032] B7. Transmit the obtained Δθ and Δx to the underlying PID control algorithm to control the quadruped robot to complete the blind path following task.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention adopts an adaptive fast segmentation strategy of macro-block assignment and local fine calculation. In step A, only the mixed pixel blocks are classified and inferred, and the pixel areas that are all background or foreground are directly assigned without classification and inference calculation. This greatly improves the inference speed of the model on the edge computing card without any impact on the accuracy. The benchmark processing speed is: 15FPS on Nvidia Jetson Nano computing card and 80FPS on Nvidia Jetson NX computing card. Compared with all fine divisions, the speed is increased by 375.14% and 400.23% respectively.

[0035] 2. The present invention adopts a visual deduction offset algorithm, and uses the binarized image obtained in step A as the input of step B to obtain the horizontal offset and the heading angle offset, and uses the horizontal offset and the heading angle offset as the input of the underlying PID control algorithm to control the quadruped robot, thereby effectively ensuring the relative posture of the quadruped robot and the blind path. The benchmark test offset accuracy is ±5cm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention has attached Figure 2 Zhang, among which:

[0037] Figure 1 This is the architecture diagram of the segmentation algorithm model.

[0038] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings. Figure 2 The process shown in the figure introduces the method of quadruped robot blind road patrol based on fast segmentation network. First, the quadruped robot's built-in image acquisition device is used to collect visual observation information of the required observation target, and the observation information is pre-processed into HSV image. Figure 1 As shown, the HSV image is encoded by convolution, and all pixel blocks that are foreground or background are directly assigned values, and the mixed pixel blocks are subjected to network prediction and classification operations, and the blind road line is quickly segmented to obtain a binary image. Then, according to step B of the present invention, the horizontal deviation and the heading angle deviation are obtained, the horizontal offset is calculated by the horizontal difference, the mathematical expression of the blind road line is calculated by the least squares method, and the heading angle offset is calculated. Finally, the deviation is passed to the underlying PID algorithm to further control the quadruped robot to complete the line patrol task.

[0040] The present invention is not limited to this embodiment, and any equivalent concepts or changes within the technical scope disclosed by the present invention are included in the protection scope of the present invention.

Claims

1. A method for a quadruped robot to follow a blind path based on a fast segmentation network, characterized in that: It includes the following steps: A. Segment the blind path line In the boot-ready stage, the quadruped robot captures and collects the information of the surrounding blind path lines and its own relative position through the image acquisition device it carries, and adopts an adaptive fast segmentation strategy of macroscopic block assignment and local fine calculation to quickly segment the blind path line and obtain a binary image. The steps are as follows: A1. Obtain scene observation information, sample the collected observation information frame by frame, and convert the sampled observation information to the HSV color space through the HSV conversion function; as shown in formula (1): V = max X = Concat(H, S, V) Where r, g, and b respectively represent the red, green, and blue color channel information of the original observation, H, S, and V respectively represent hue, saturation, and brightness, that is, the 3 channel information of the HSV color space, max and min respectively represent the maximum and minimum values of the pixel points in the r, g, and b channels, Concat represents the merge operation, and X represents the image in the HSV color space; A2. After encoding X as the input through the convolutional network, a feature map with a height of h, a width of w, and 7 channels is obtained; as shown in formula (2): feature = backbone(X) (2) Among them, "backbone" represents the convolutional network, and "feature" represents the feature map. The first 3 channels are defined as the class indices, denoted as C = [c 1 , c 2 , c 3 , and the last 4 channels are defined as the classification feature vectors, denoted as M = [ω 1 , ω 2 , ω 3 , b]; A3. Perform a normalization exponential function operation on C to obtain three categories, namely background, foreground, and mixed category. The foreground category is the blind path, and the category index is recorded as 0; the background category is the area outside the blind path, and the category index is recorded as 1; the mixed category is the intersection area of the foreground and background, and the category index is recorded as 2; mark the category in the form of one-hot encoding; as shown in formula (3): Where Softmax represents the normalization exponential function, i represents the category index, p(i) represents the probability value that this pixel point belongs to the i-th category, and argmax(P) represents the category index corresponding to the maximum value in P; A4. For the feature map components of the foreground category or background category, directly mark all the pixels in the receptive field area of the original HSV image X corresponding to feature as the foreground category or background category; A5. For the feature map components of the mixed category, perform a dot product operation on all the pixels in the receptive field area of the original HSV image X corresponding to feature with the classification feature vector M respectively. The pixel points with a dot product result greater than 0 are classified as the foreground category, otherwise they are classified as the background category to obtain a binary image; as shown in formula (4): Among them, H i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the H channel of the original HSV image X. S i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the S channel of the original HSV image X. V i,j is the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the V channel of the original HSV image X. * represents the dot product operation. φ i,j is the class prediction value of the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the original HSV image X. Φ i,j is the binary representation of the classification result of the pixel at the i-th horizontal coordinate and j-th vertical coordinate on the original HSV image X; B. Calculate the vision-based offset B1. Scan the horizontal midline of the binary image from left to right. The first pixel point that changes from 0 to 1 is recorded as the left anchor point, and the distance from the left anchor point to the left edge of the image is denoted as a left ; B2. Scan the horizontal midline of the binary image from right to left, and record the first pixel point that changes from 0 to 1 as the right anchor point. Denote the distance from the right anchor point to the left edge of the image as a right ; B3. Let x 1 = W / 2 - a left , x 2 = a right - W / 2, Δx = x 1 - x 2 . When Δx > 0, it indicates that the quadruped robot is biased to the right. When Δx < 0, it indicates that the quadruped robot is biased to the left. When Δx = 0, it indicates that the quadruped robot is in an ideal state; B4. Taking the left anchor point as the reference point, search upward to obtain the set of discrete points on the left side line, denoted as p l , and using the least squares method, fit the mathematical quadratic equation f left (x) of the left side line of the blind path, and find the angle between the tangent line and the perpendicular bisector of f left (x) at the left anchor point, denoted as θ 1 ; The expression of the above mathematical quadratic equation is as follows: f left (x) = a l x 2 + b l x + c l (5) where x is the point p l abscissa, a l , b l , c l are respectively the quadratic coefficient, the linear coefficient and the constant of the mathematical quadratic equation; B5. Taking the right anchor point as the reference point, search upward for the set of discrete points on the left side line and denote it as p r , and using the least squares method, fit the mathematical quadratic equation f right (x) of the right side line, and find the angle between the tangent line of f right (x) at the right anchor point and the perpendicular bisector, denoted as θ 2 ; The expression of the above mathematical quadratic equation is as follows: f right (x) = a r x 2 + b r x + c r (6) where x is the point p r abscissa, a r , b r , c r are respectively the quadratic coefficient, the linear coefficient and the constant of the mathematical quadratic equation; B6. Let Δθ = (θ 1 + θ 2 ) / 2. When Δθ > 0, the heading angle of the quadruped robot is biased to the left; when Δθ < 0, the heading angle of the quadruped robot is biased to the right; when Δθ = 0, the heading angle of the quadruped robot is in an ideal state; B7. Transmit the obtained Δθ and Δx to the underlying PID control algorithm to control the quadruped robot to complete the blind path following task.

Images