A Human Following Method and System Based on Vision and LiDAR Perception

Abstract

The present invention provides a human following method and system based on vision and lidar perception. The method includes the following steps: A mobile robot is equipped with a camera and a lidar, and video images and lidar data are collected through the camera and the lidar respectively; Based on an object detection model, the human body is detected to obtain the coordinate information of the detection box in the image; Based on the horizontal field of view angle of the camera, a human body scanning azimuth estimation model is constructed, and according to the coordinate information of the detection box in the image, the scanning azimuth angle range where the human body is located is obtained; According to the scanning azimuth angle range where the human body is located, the human body distance is extracted from the lidar data to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located; This method combines the human body scanning azimuth estimation model with the lidar data, realizes the accurate measurement and positioning of the human body position, improves the perception ability of the mobile robot in a complex environment, and improves the accuracy and robustness of human following.

Description

Technical Field

[0001] The present invention relates to the technical field of robot perception, and in particular, to a human following method and system based on vision and lidar perception. Background Art

[0002] Robot perception and following enable a robot to perceive the surrounding environment in real time and accurately follow a target object, whether it is a person or other moving objects. With the rapid development of sensor technology and artificial intelligence, robot perception and following technology has been widely applied. Its core lies in high-precision sensors and advanced algorithms. The sensors can capture the dynamic information of the target object, and the algorithms are responsible for processing this information and adjusting the movement trajectory of the robot in real time to ensure that it closely follows the target.

[0003] A method and system for a robot to follow a human body with the publication number of CN108717553B, wherein the method includes: using the HOG method to detect the human body area on the depth map obtained by the robot; at the initialization of following, calculating and saving the apparent features of the color map corresponding to the tracked human body area; the apparent features include color features and texture features; during the following process, judging whether an intersection occurs according to the position change of the tracked human body area in consecutive frame images; after judging that an intersection occurs, controlling the robot to stop moving; at the same time, searching for a human body area similar to the apparent features of the tracked human body area saved in the subsequent frame images obtained by the robot until the human body area is found, and then controlling the robot to follow the human body area.

[0004] The existing solutions at present use the HOG method to detect the human body area on the depth map and rely on color and texture features for following. During the process of the robot following a human body, the HOG method used for human body detection on the depth map is vulnerable to environmental interference, resulting in insufficient positioning accuracy; at the same time, the color and texture feature matching method it relies on is vulnerable to factors such as illumination change and occlusion, resulting in easy errors in feature matching, and it needs to re-search and match after an intersection occurs, reducing the real-time performance and stability of following. Summary of the Invention

[0005] In view of this, the present invention proposes a human following method and system based on vision and lidar perception, which combines a human body scanning azimuth estimation model and lidar data, realizes precise measurement and positioning of the human body position, improves the perception ability of a mobile robot in a complex environment, and enhances the accuracy and robustness of human following.

[0006] The technical solution of the present invention is realized as follows: The present invention provides a human following method based on vision and lidar perception, including the following steps:

[0007] S1. The mobile robot is equipped with a camera and a lidar, and video images and lidar data are collected through the camera and the lidar respectively;

[0008] S2. Based on the collected video images, the human body is detected using an object detection model to obtain the coordinate information of the detection box in the image;

[0009] S3. A human body scanning azimuth estimation model is constructed based on the horizontal field of view angle of the camera, and according to the coordinate information of the detection box in the image, combined with the human body scanning azimuth estimation model, the scanning azimuth angle range where the human body is located is obtained;

[0010] S4. According to the scanning azimuth angle range where the human body is located, the human body distance is extracted from the lidar data to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located;

[0011] S5. Judgment and control are performed according to the distance between the human body and the mobile robot and the azimuth where the human body is located, and a motion instruction is sent to achieve the movement of the robot following the human body.

[0012] On the basis of the above technical solutions, preferably, step S2 includes:

[0013] The input is the video image collected by the mobile robot, and the YOLOv8 object detection model is used to detect the human body in the video image. The YOLOv8 object detection model outputs the detection box of the human body detected in the image. The detection box includes the coordinates of the upper left corner and the lower right corner, and the coordinate information of the detection box in the image is obtained. Let the expression of the detection box be:

[0014] [(u1, v1), (u2, v2)]

[0015] In the formula, u1 is the x coordinate of the upper left corner of the detection box, v1 is the y coordinate of the upper left corner of the detection box, u2 is the x coordinate of the lower right corner of the detection box, and v2 is the y coordinate of the lower right corner of the detection box.

[0016] On the basis of the above technical solutions, preferably, constructing the human body scanning azimuth estimation model based on the horizontal field of view angle of the camera in step S3 includes:

[0017] Let the center point of the image be the origin, the origin of the image coordinate system be (u0, v0), the point in the image coordinate system be (u, v), and the width of the video image be W, then u0 = W / 2;

[0018] Establish the mapping relationship between the point in the image coordinate system and the point in the world coordinate system, and convert the point in the image coordinate system into the point in the world coordinate system. The expression is:

[0019]

[0020] Wherein, the point in the image coordinate system is (u, v, 1), and the point in the world coordinate system after the transformation of the point in the image coordinate system is (x w , y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is (u0, v0), f represents the focal length, dx and dy respectively represent the widths of the pixels in the x direction and the y direction, and [R, t] represents the external camera parameter matrix;

[0021] Let the camera coordinate system overlap with the world coordinate system, R be the identity matrix, the translation vector t = 0, z w = z c , according to the mapping relationship between the point in the image coordinate system and the point in the world coordinate system, obtain the mapping relationship from the point in the world coordinate system to the point in the image coordinate system, and the expression is:

[0022] z w ·u = f / dx·x w + z w ·u0;

[0023] According to tan(α) = x w / z w , let u = W, substitute it into the above expression, and obtain the tangent value of the half horizontal field of view angle of the camera, and its expression is:

[0024] tan(α) = dx·W / (2·f);

[0025] According to the mapping relationship between the point in the image coordinate system and the point in the world coordinate system and the tangent value of half of the horizontal field of view angle of the camera, establish an equation of the relationship between the angle between the coordinate point and the optical axis in the front direction of the line of sight and the image coordinates, and construct a human body scanning orientation estimation model, and the expression is:

[0026] θ = arctan[(2u / W - 1)·tan(α)]

[0027] Wherein, θ is the angle between the coordinate point and the optical axis in the front direction of the line of sight, and α is half of the horizontal field of view angle of the camera.

[0028] On the basis of the above technical solutions, preferably, in step S3, according to the coordinate information of the detection frame in the image, combined with the human body scanning orientation estimation model, obtain the angle range of the scanning orientation where the human body is located, including:

[0029] According to the coordinate information of the detection frame, obtain the range of the human body detection border, and the range of the human body detection border is [u1, u2];

[0030] Substitute the range of the human body detection border into the human body scanning orientation estimation model to obtain the angle range [θ1, θ2] of the scanning orientation where the human body is located, wherein,

[0031] θ1 = arctan[(2u1 / W - 1)·tan(α)];

[0032] θ2 = arctan[(2u2 / W - 1)·tan(α)].

[0033] Based on the above technical solutions, preferably, in step S4, according to the scanning azimuth angle range where the human body is located, the distance of the human body is extracted from the lidar data to obtain the distance between the human body and the mobile robot, including:

[0034] Let the radar angular resolution be a, obtain the lidar data vector. The included angle of the lidar data vector from the first item to the Nth item corresponding to the optical axis in the clockwise direction is from 0 to 2π. Obtain the index position of the lidar data vector where the scanning azimuth angle of the human body is located, and construct an index function. The expression is:

[0035]

[0036] In the formula, k(θ) is the index position of the lidar data vector where the scanning azimuth angle of the human body is located, and round() represents the rounding function;

[0037] According to the scanning azimuth angle range [θ1, θ2] of the human body and the index function, obtain the distance vector within the corresponding range. The expression is:

[0038] D = range[k(θ1):k(θ2)]

[0039] In the formula, range[] is the distance vector measured by the lidar in the corresponding scanning angle range, and k(θ1):k(θ2) is the scanning azimuth angle range of the human body from index k(θ1) to k(θ2);

[0040] Preset a distance threshold, and obtain the minimum value of the distance greater than the distance threshold from the distance vector corresponding to the scanning azimuth angle range of the human body as the distance between the human body and the mobile robot. The expression is:

[0041] d = min(D(D > r0))

[0042] In the formula, d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

[0043] Based on the above technical solutions, preferably, in step S4, according to the scanning azimuth angle range where the human body is located, calculate the azimuth of the human body. The expression is:

[0044] θ r = (θ1 + θ2) / 2

[0045] In the formula, θr It is the orientation of the human body.

[0046] Based on the above technical solutions, preferably, in step S5, it is determined and controlled to send a motion instruction according to the distance between the human body and the mobile robot and the orientation of the human body, so as to realize the robot following the human body to move, including:

[0047] A preset control threshold distance is set. According to the orientation of the human body and the distance between the human body and the mobile robot, the distance between the human body and the mobile robot is judged against the control threshold distance. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop instruction is sent to the chassis to stop the movement of the mobile robot. Otherwise, the mobile robot is controlled to move forward following the orientation of the human body through the PID control algorithm.

[0048] In a second aspect, the present invention also provides a human body following system based on vision and lidar perception, which is implemented by using the human body following method based on vision and lidar perception as described above. The system includes:

[0049] An acquisition module, which is used for the mobile robot to carry a camera and a lidar, and respectively acquire video images and lidar data through the camera and the lidar;

[0050] An image processing module, which is used for detecting the human body based on the acquired video images by using a target detection model to obtain the coordinate information of the detection frame in the image;

[0051] A model processing module, which is used to construct a human body scanning orientation estimation model based on the horizontal field of view angle of the camera, and combine the coordinate information of the detection frame in the image with the human body scanning orientation estimation model to obtain the range of the human body scanning orientation angle;

[0052] An extraction module, which is used to extract the human body distance from the lidar data according to the range of the human body scanning orientation angle, so as to obtain the distance between the human body and the mobile robot and the orientation of the human body;

[0053] A motion control module, which is used to judge, control and send a motion instruction according to the distance between the human body and the mobile robot and the orientation of the human body, so as to realize the robot following the human body to move.

[0054] In a third aspect, the present invention also provides an electronic device, including at least one processor, at least one memory, a communication interface and a bus; wherein, the processor, the memory and the communication interface complete communication with each other through the bus; the memory stores a program for a human body following method based on vision and lidar perception that can be executed by the processor, and a program for a human body following method based on vision and lidar perception is configured to implement the human body following method based on vision and lidar perception as described above.

[0055] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a program for a human following method based on vision and lidar perception is stored. When the program for the human following method based on vision and lidar perception is executed, it implements the human following method based on vision and lidar perception as described above.

[0056] The human following method and system based on vision and lidar perception of the present invention have the following beneficial effects compared with the prior art:

[0057] (1) Through a mobile robot equipped with a camera and a lidar, video images and lidar data are collected. The YOLOv8 object detection model is used to detect a human body in the video image and obtain its coordinate information. Based on the horizontal field of view angle of the camera, a human body scanning azimuth estimation model is constructed. Combining the coordinate information of the detection frame, the scanning azimuth angle range where the human body is located is determined, and the human body distance is extracted from the lidar data. The distance and azimuth between the human body and the mobile robot are calculated. Through a preset control logic and a PID control algorithm, a motion instruction is sent to the mobile robot to make it follow the human body. The human body scanning azimuth estimation model and the lidar data are effectively combined, realizing the accurate measurement and positioning of the human body position, improving the perception ability of the mobile robot in a complex environment, and enhancing the accuracy and robustness of human body following;

[0058] (2) By parsing the coordinate of the detection frame in the image and using parameters such as the camera focal length, pixel size, and horizontal field of view angle, the points in the image coordinate system are mapped to the world coordinate system, and a human body scanning azimuth estimation model is constructed, realizing the accurate calculation of the scanning azimuth angle range where the human body is located;

[0059] (3) By constructing an index function and a preset distance threshold, the present invention can efficiently extract the lidar data of the scanning azimuth where the human body is located, avoid data traversal, significantly improve the processing efficiency, and effectively filter noise, enhancing the reliability of the measurement result. Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0061] Figure 1 It is a flowchart of the human following method based on vision and lidar perception of the present invention;

[0062] Figure 2Schematic diagram of the geometric relationship between the projected point of the target human body on the horizontal plane and its corresponding image field-of-view boundary point in the world coordinate system of the human following method based on vision and lidar perception of the present invention. Detailed implementation manners

[0063] Next, in combination with the implementation manners of the present invention, the technical solutions in the implementation manners of the present invention will be clearly and completely described. Obviously, the described implementation manners are only a part of the implementation manners of the present invention, rather than all of the implementation manners. Based on the implementation manners in the present invention, all other implementation manners obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0064] As Figure 1 shown, a human following method based on vision and lidar perception of the present invention includes the following steps:

[0065] S1, a mobile robot is equipped with a camera and a lidar, and video images and lidar data are respectively collected through the camera and the lidar.

[0066] It should be noted that a monocular camera and a single-line lidar are mounted on the mobile robot. The monocular camera is installed at an appropriate position on the robot to ensure that the field of view of the monocular camera can cover the area that the robot needs to perceive and avoid blind spots; the single-line lidar is installed on the top or side of the robot to ensure that the scanning range of the single-line lidar can cover the environment around the robot and overlap with the field of view of the monocular camera for data fusion. According to the video images collected by the monocular camera and the lidar data collected by the single-line lidar, data support is provided for subsequent target detection, human body scanning azimuth estimation, and human body distance extraction.

[0067] In order to ensure that the data collected by the camera and the lidar can be accurately corresponding, it is necessary to achieve data synchronization. By installing a time synchronization module on the robot, this time synchronization module can record the acquisition timestamps of each data packet and use these timestamps to align the data of the monocular camera and the single-line lidar in the subsequent data processing process.

[0068] S2, based on the collected video images, the human body is detected based on a target detection model to obtain the coordinate information of the detection box in the image.

[0069] Among them, step S2 includes:

[0070] Input the video images collected by the mobile robot, use the YOLOv8 target detection model to detect the human body in the video images. The YOLOv8 target detection model outputs the detection box of the detected human body in the image. The detection box includes the coordinates of the upper left corner and the lower right corner, and the coordinate information of the detection box in the image is obtained. Let the expression of the detection box be:

[0071] [(u1, v1), (u2, v2)]

[0072] In the formula, u1 is the x - coordinate of the upper - left corner of the detection box, v1 is the y - coordinate of the upper - left corner of the detection box, u2 is the x - coordinate of the lower - right corner of the detection box, and v2 is the y - coordinate of the lower - right corner of the detection box.

[0073] It should be noted that video images are obtained in real - time from the monocular camera of the mobile robot and used as the input of the YOLOv8 object - detection model. The YOLOv8 object - detection model performs forward propagation on the images and outputs the detection boxes of the detected human bodies. The detection boxes output by the YOLOv8 object - detection model include the coordinates of the upper - left and lower - right corners, and these coordinates represent the position of the human body in the image.

[0074] S3. Construct a human body scanning azimuth estimation model based on the horizontal field - of - view angle of the camera, and obtain the range of the scanning azimuth angle where the human body is located according to the coordinate information of the detection box in the image and in combination with the human body scanning azimuth estimation model.

[0075] Among them, constructing a human body scanning azimuth estimation model based on the horizontal field - of - view angle of the camera in step S3 includes:

[0076] Let the center point of the image be the origin, the origin of the image coordinate system be (u0, v0), the point in the image coordinate system be (u, v), and the width of the video image be W, then u0 = W / 2;

[0077] Establish the mapping relationship between the point in the image coordinate system and the point in the world coordinate system, and convert the point in the image coordinate system into the point in the world coordinate system. The expression is:

[0078]

[0079] In the formula, the point in the image coordinate system is (u, v, 1), the point in the world coordinate system after conversion of the point in the image coordinate system is (x w , y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is (u0, v0), f represents the focal length, dx and dy respectively represent the width of the pixel in the x - direction and y - direction, and [R, t] represents the external camera parameter matrix;

[0080] Let the camera coordinate system overlap with the world coordinate system, R be the identity matrix, the translation vector t = 0, z w = z c , and according to the mapping relationship between the point in the image coordinate system and the point in the world coordinate system, obtain the mapping relationship from the point in the world coordinate system to the point in the image coordinate system. The expression is:

[0081] z w ·u = f / dx·xw +z w ·u0;

[0082] According to tan(α) = x w / z w , let u = W, substitute it into the above expression to obtain the tangent value of the half horizontal field of view angle of the camera, and its expression is:

[0083] tan(α) = dx·W / (2·f);

[0084] According to the mapping relationship between the points in the image coordinate system and the points in the world coordinate system and the tangent value of half of the horizontal field of view angle of the camera, establish an equation of the relationship between the angle between the coordinate system point and the optical axis directly in front of the line of sight and the image coordinates, and construct a human body scanning azimuth estimation model, and its expression is:

[0085] θ = arctan[(2u / W - 1)·tan(α)]

[0086] In the formula, θ is the angle between the coordinate system point and the optical axis directly in front of the line of sight, α is half of the horizontal field of view angle of the camera, and the horizontal field of view angle of the camera is usually given by the camera manufacturer or calibrated and calculated by itself.

[0087] As Figure 2 shown, assume that the position of the camera center and the position of the radar center overlap on the x-axis and the z-axis, and the z-axis is in the direct front. In this way, the angle θ between a certain point of the person in the image and the optical axis directly in front of the line of sight is equivalent to the angle θ between the radar scanning point and the optical axis directly in front; Figure 2 shows the schematic diagram of the geometric relationship between the projection point P(x w , 0, z w ) of the target human body in the world coordinate system on the horizontal plane, where z w = d represents the depth of field distance where a certain point of the human body is located, and its corresponding image field of view boundary point Q(Q xw , 0, Q zw ). Assume that the horizontal field of view angle of the camera is known to be 2α, which can be obtained through calibration after installing the camera and the radar. Then, according to Figure 2 the geometric relationship between P and Q in

[0088] tan(θ)·f / dx = (u - u0);

[0089] tan(α)·f / dx = (W - u0).

[0090] After dividing the above two formulas, we get:

[0091] θ = arctan[(2u / W - 1)·tan(α)].

[0092] In this embodiment, according to the coordinate information of the detection frame, the range of the human body detection frame is obtained, and the range of the human body detection frame is [u1, u2];

[0093] Substitute the range of the human body detection frame into the human body scanning azimuth estimation model to obtain the range of the scanning azimuth angle [θ1, θ2] where the human body is located. Among them,

[0094] θ1 = arctan[(2u1 / W - 1)·tan(α)];

[0095] θ2 = arctan[(2u2 / W - 1)·tan(α)].

[0096] In this embodiment, by analyzing the coordinates of the detection frame in the image and using parameters such as the camera focal length, pixel size, and horizontal field of view angle, the points in the image coordinate system are mapped to the world coordinate system, realizing the accurate estimation of the range of the scanning azimuth angle where the human body is located.

[0097] S4. According to the range of the scanning azimuth angle where the human body is located, extract the distance of the human body from the lidar data to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located.

[0098] Step S4 includes:

[0099] Let the radar angle resolution be a, obtain the lidar data vector, the data vector contains N items, and the included angle of the lidar data vector corresponding to the optical axis from the first item to the Nth item in the clockwise direction is from 0 to 2π. Obtain the index position of the lidar data vector where the scanning azimuth angle where the human body is located, and construct an index function. The expression is:

[0100]

[0101] In the formula, k(θ) is the index position of the lidar data vector where the scanning azimuth angle where the human body is located, and round() represents the rounding function;

[0102] According to the range of the scanning azimuth angle [θ1, θ2] where the human body is located and the index function, obtain the distance vector within the corresponding range. The expression is:

[0103] D = range[k(θ1):k(θ2)]

[0104] In the formula, range[] is the distance vector measured by the lidar in the corresponding scanning angle range, and k(θ1):k(θ2) is the range of the scanning azimuth angle where the human body is located from index k(θ1) to k(θ2);

[0105] A preset distance threshold is used to obtain, from the distance vector corresponding to the scanning azimuth angle range where the human body is located, the distance that is greater than the distance threshold and is the minimum value as the distance between the human body and the mobile robot. The expression is:

[0106] d = min(D(D > r0))

[0107] In the formula, d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

[0108] In step S4, according to the scanning azimuth angle range where the human body is located, the azimuth where the human body is located is calculated. The expression is:

[0109] θ r = (θ1 + θ2) / 2

[0110] In the formula, θ r is the azimuth where the human body is located.

[0111] It should be noted that by constructing an index function, the index position of the scanning azimuth angle of the human body in the lidar data vector can be accurately found. According to the scanning azimuth angle range of the human body, the distance vector within the corresponding range is efficiently extracted by using the index function, avoiding traversing the entire lidar data vector and significantly improving the efficiency of data processing.

[0112] By presetting the distance threshold, noise data is effectively filtered out, improving the reliability of the measurement results. In this embodiment, by combining the human body scanning azimuth estimation model with lidar data, the accurate measurement and positioning of the human body position are realized, enhancing the perception ability of the mobile robot in a complex environment, enabling the mobile robot to more intelligently respond to various scenarios, and improving safety and reliability.

[0113] S5. According to the distance between the human body and the mobile robot and the azimuth where the human body is located, judgment and control are performed to send a motion instruction to make the robot follow the human body to move.

[0114] Among them, step S5 includes: presetting a control threshold distance, judging the distance between the human body and the mobile robot and the control threshold distance according to the azimuth where the human body is located and the distance between the human body and the mobile robot. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop instruction is sent to the chassis to stop the movement of the mobile robot; otherwise, the mobile robot is controlled to move forward and follow along the azimuth where the human body is located through the PID control algorithm.

[0115] Specifically, the control threshold distance in this embodiment is 0.8 m. This control threshold distance is used to judge the relative position relationship between the human body and the mobile robot. When the distance from the human body to the mobile robot is less than this control threshold distance, it is considered that the robot has approached the human body to a certain extent and needs to stop moving to avoid collision or interference. A stop command is sent to the chassis of the mobile robot through the control system. After receiving the stop command, the chassis performs the corresponding stop operation to make the mobile robot stop moving. If the distance from the human body to the mobile robot is greater than or equal to the control threshold distance, the motion of the mobile robot is controlled by the PID control algorithm. The PID control algorithm calculates the control amount according to the deviation between the current azimuth of the human body and the current position of the mobile robot, and sends it to the chassis of the mobile robot. The chassis adjusts the speed and direction of the mobile robot according to the received control amount to make it move forward and follow along the azimuth where the human body is located. During the process of the mobile robot following the human body, the distance from the human body to the mobile robot and the azimuth where the human body is located are continuously monitored, and the control parameters of the PID control algorithm are adjusted in real time according to the monitoring results to ensure that the mobile robot can stably and accurately follow the human body to move.

[0116] The human body azimuth and distance perception method proposed in this embodiment is significantly characterized by its computational simplicity and efficiency. The core computational burden of this method is mainly concentrated in the human body target detection stage, while the process of extracting the radar scanning azimuth and distance information is relatively simple, resulting in a low demand for computational resources by this method, thereby improving the overall system operation efficiency. Furthermore, according to the lidar, the surrounding environment can be quickly scanned, and the distance between the human body and the mobile robot and the azimuth where the human body is located can be accurately measured, enabling the mobile robot to track and follow the human body in real time and accurately.

[0117] The present invention also provides a human body following system based on vision and lidar perception, which is implemented by using the human body following method based on vision and lidar perception as described above. The system includes:

[0118] An acquisition module, which is used for the mobile robot to carry a camera and a lidar, and respectively acquire video images and lidar data through the camera and the lidar;

[0119] An image processing module, which is used to detect the human body based on the acquired video images using a target detection model to obtain the coordinate information of the detection frame in the image;

[0120] A model processing module, which is used to construct a human body scanning azimuth estimation model based on the horizontal field of view angle of the camera, and combine the coordinate information of the detection frame in the image with the human body scanning azimuth estimation model to obtain the range of the scanning azimuth angle where the human body is located;

[0121] An extraction module, configured to extract the distance of a human body from lidar data according to the scanning azimuth angle range where the human body is located, so as to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located;

[0122] A motion control module, configured to judge and control the sending of motion instructions according to the distance between the human body and the mobile robot and the azimuth where the human body is located, so as to realize the robot following the movement of the human body.

[0123] It should be noted that this system corresponds to the above-mentioned method for weighing and measuring marine engineering equipment. All implementation manners in the above method embodiments are applicable to the embodiments of this system and can achieve the same technical effects.

[0124] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0125] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described system and modules can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0126] In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in an electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0129] If the above-mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0130] In addition, it should be noted that in the system and method of the present invention, obviously, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present invention. And, the steps of performing the above series of processes can naturally be executed in chronological order according to the described order, but it is not necessary to execute them in chronological order. Some steps can be executed in parallel or independently of each other. For those of ordinary skill in the art, it is understandable that all or any steps or components of the method and device of the present invention can be implemented in any computing device (including a processor, a storage medium, etc.) or a network of computing devices in the form of hardware, firmware, software, or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.

[0131] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing system. The computing system can be a well-known general system. Therefore, the object of the present invention can also be achieved only by providing a program product containing program codes for implementing the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be noted that in the device and method of the present invention, obviously, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present invention. And, the steps of performing the above series of processes can naturally be executed in chronological order according to the described order, but it is not necessary to execute them in chronological order. Some steps can be executed in parallel or independently of each other.

[0132] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0133] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A human following method based on vision and lidar perception, characterized in that, It includes the following steps: S1. The mobile robot is equipped with a camera and a lidar, and video images and lidar data are collected through the camera and the lidar respectively; S2. Based on the collected video images, the human body is detected using a target detection model to obtain the coordinate information of the detection box in the image; S3. A human body scanning azimuth estimation model is constructed based on the horizontal field of view angle of the camera. According to the coordinate information of the detection box in the image and in combination with the human body scanning azimuth estimation model, the scanning azimuth angle range where the human body is located is obtained; The mapping relationship between the points in the image coordinate system and the points in the world coordinate system is established, and the points in the image coordinate system are converted into points in the world coordinate system; Let the camera coordinate system overlap with the world coordinate system. According to the mapping relationship between the points in the image coordinate system and the points in the world coordinate system, the mapping relationship from the points in the world coordinate system to the points in the image coordinate system and the tangent value of half of the horizontal field of view angle of the camera are obtained; According to the mapping relationship between the points in the image coordinate system and the points in the world coordinate system and the tangent value of half of the horizontal field of view angle of the camera, an equation of the relationship between the angle between the coordinate point and the optical axis in the front direction of the line of sight and the image coordinates is established to construct a human body scanning azimuth estimation model; S4. According to the scanning azimuth angle range where the human body is located, the human body distance is extracted from the lidar data to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located; The index position of the lidar data vector where the scanning azimuth angle of the human body is located is obtained, and an index function is constructed; According to the scanning azimuth angle range where the human body is located and the index function, the distance vector within the corresponding range is obtained; A preset distance threshold is set, and the minimum value with a distance greater than the distance threshold is obtained from the distance vector corresponding to the scanning azimuth angle range where the human body is located as the distance between the human body and the mobile robot; S5. According to the distance between the human body and the mobile robot and the azimuth where the human body is located, a judgment control is performed to send a motion instruction to realize the movement of the robot following the human body.

2. The human body following method based on vision and lidar perception according to claim 1, wherein Step S2 includes: The video images collected by the mobile robot are input, and the YOLOv8 target detection model is used to detect the human body in the video images. The YOLOv8 target detection model outputs the detection box of the human body detected in the image. The detection box includes the coordinates of the upper left corner and the lower right corner, and the coordinate information of the detection box in the image is obtained. Let the expression of the detection box be: [( u 1， v 1)，( u 2， v 2)] In the formula, u 1 is the x coordinate of the upper left corner of the detection box, v 1 is the y coordinate of the upper left corner of the detection box, u 2 is the x coordinate of the lower right corner of the detection box, v 2 is the y coordinate of the lower right corner of the detection box.

3. The human following method based on vision and lidar perception according to claim 2, wherein, In step S3, constructing a human body scanning azimuth estimation model based on the horizontal field of view angle of the camera includes: Let the center point of the image be the origin, and the origin of the image coordinate system be ( u 0, v 0), and the point in the image coordinate system be ( u , v ). The width of the video image is W , then u 0 = W / 2; The mapping relationship between the points in the image coordinate system and the points in the world coordinate system is established, and the points in the image coordinate system are converted into points in the world coordinate system. The expression is: ； Wherein, the point in the image coordinate system is ( u , v , 1), and the point in the world coordinate system after the conversion of the point in the image coordinate system is ( x w , y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is ( u 0, v 0), f represents the focal length, dx and dy respectively represent the widths of the pixels in the x direction and the y direction, and [R, t] represents the external camera parameter matrix; Let the camera coordinate system overlap with the world coordinate system, R be the identity matrix, and the translation vector t = 0. z w =z c , according to the mapping relationship between the points in the image coordinate system and the points in the world coordinate system, obtain the mapping relationship from the points in the world coordinate system to the points in the image coordinate system, and the expression is: z w · u = f / dx · x w + z w · u 0； According to tan(α)= x w / z w , let u = W , substitute it into the above expression to obtain the tangent value of the half horizontal field of view angle of the camera, and its expression is: tan(α)= dx · W / (2· f ); According to the mapping relationship between the points in the image coordinate system and the points in the world coordinate system and the tangent value of half of the horizontal field of view angle of the camera, an equation of the relationship between the angle between the coordinate point and the optical axis in the front direction of the line of sight and the image coordinates is established to construct a human body scanning azimuth estimation model. The expression is: θ = arctan[(2 u / W - 1)·tan(α)] In the formula, θ is the angle between the coordinate point and the optical axis in the front direction of the line of sight, and α is half of the horizontal field of view angle of the camera.

4. The human body following method based on vision and lidar perception according to claim 3, wherein In step S3, according to the coordinate information of the detection box in the image and in combination with the human body scanning azimuth estimation model, obtaining the scanning azimuth angle range where the human body is located includes: According to the coordinate information of the detection box, obtain the range of the human detection border, and the range of the human detection border is u 1, u 2]; Substitute the human body detection border range into the human body scanning azimuth estimation model to obtain the scanning azimuth angle range [θ1, θ2] where the human body is located, where, θ1 = arctan[(2 u 1 / W - 1)·tan(α)]; θ2 = arctan[(2 u 2 / W - 1)·tan(α)]。 5. The human following method based on vision and lidar perception according to claim 4, characterized in that In step S4, according to the scanning azimuth angle range where the human body is located, the human body distance is extracted from the lidar data to obtain the distance between the human body and the mobile robot, including: Let the radar angular resolution be a . Obtain the lidar data vector. The included angles of the lidar data vector from the first item to the Nth item with respect to the optical axis in the clockwise direction are from 0 to 2π. Obtain the index position of the lidar data vector at the scanning azimuth angle where the human body is located, and construct an index function. The expression is as follows: ； In the formula, k(θ) is the index position of the lidar data vector where the scanning azimuth angle of the human body is located, and round() represents the rounding function; According to the scanning azimuth angle range [θ1, θ2] where the human body is located and the index function, the distance vector within the corresponding range is obtained, and the expression is: D = range[k(θ1):k(θ2)] In the formula, range[] is the distance vector measured by the lidar in the corresponding scanning angle range, and k(θ1):k(θ2) is the scanning azimuth angle range of the human body from index k(θ1) to k(θ2); A preset distance threshold is set, and the distance greater than the distance threshold and the minimum value are obtained from the distance vector corresponding to the scanning azimuth angle range where the human body is located as the distance between the human body and the mobile robot, and the expression is: d = min(D(D > r0)) In the formula, d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

6. The human following method based on vision and lidar perception according to claim 5, characterized in that In step S4, according to the scanning azimuth angle range where the human body is located, the azimuth where the human body is located is calculated, and the expression is: θ r =(θ1 + θ2) / 2 where θ r is the orientation of the human body.

7. The method for human following based on vision and lidar perception as described in claim 6, wherein In step S5, according to the distance between the human body and the mobile robot and the azimuth where the human body is located, judgment and control are performed to send a motion instruction to realize the robot following the human body to move, including: A preset control threshold distance is set. According to the azimuth where the human body is located and the distance between the human body and the mobile robot, the distance between the human body and the mobile robot is judged against the control threshold distance. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop instruction is sent to the chassis to stop the movement of the mobile robot. Otherwise, the mobile robot is controlled to move forward following along the azimuth where the human body is located through the PID control algorithm.

8. A human following system based on vision and lidar perception, which is implemented by using the human following method based on vision and lidar perception according to any one of claims 1-7, characterized in that: The system includes: An acquisition module, which is used for the mobile robot to carry a camera and a lidar, and respectively acquire video images and lidar data through the camera and the lidar; An image processing module, which is used for detecting the human body based on the target detection model according to the acquired video images to obtain the coordinate information of the detection frame in the image; A model processing module, which is used to construct a human body scanning azimuth estimation model based on the horizontal field of view angle of the camera, and according to the coordinate information of the detection frame in the image, in combination with the human body scanning azimuth estimation model, obtain the scanning azimuth angle range where the human body is located; An extraction module, which is used to extract the human body distance from the lidar data according to the scanning azimuth angle range where the human body is located to obtain the distance between the human body and the mobile robot and the azimuth where the human body is located; A motion control module, which is used to perform judgment and control to send a motion instruction according to the distance between the human body and the mobile robot and the azimuth where the human body is located to realize the robot following the human body to move.

9. An electronic device, characterized in that, Comprising at least one processor, at least one memory, a communication interface, and a bus; wherein, the processor, the memory, and the communication interface complete communication with each other through the bus; the memory stores a program of a human body following method based on vision and lidar perception, and the program of the human body following method based on vision and lidar perception is configured to implement a human body following method based on vision and lidar perception according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program of a human body following method based on vision and lidar perception is stored on the storage medium, and when the program of the human body following method based on vision and lidar perception is executed, it implements a human body following method based on vision and lidar perception according to any one of claims 1 to 7.

Images