Human body following method and system based on vision and laser radar perception

Abstract

The invention provides a human body following method and system based on vision and laser radar perception, and the method comprises the following steps: a mobile robot carries a camera and a laser radar, and respectively collects a video image and laser radar data through the camera and the laser radar; detecting a human body based on the target detection model to obtain coordinate information of a detection frame in the image; constructing a human body scanning azimuth estimation model based on a camera horizontal field angle, and obtaining a scanning azimuth angle range of a human body according to coordinate information of a detection frame in the image; according to the scanning azimuth angle range of the human body, extracting the human body distance from the laser radar data, and obtaining the distance from the human body to the mobile robot and the azimuth of the human body; according to the method, the human body scanning orientation estimation model and the laser radar data are combined, accurate measurement and positioning of the human body position are achieved, the sensing ability of the mobile robot in a complex environment is improved, and the accuracy and robustness of human body following are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot perception technology, and in particular to a human body following method and system based on vision and laser radar perception. Background Art

[0002] Robot perception and following enables robots to perceive their surroundings in real time and accurately follow target objects, whether they are people or other moving objects. With the rapid development of sensor technology and artificial intelligence, robot perception and following technology has been widely used. Its core lies in high-precision sensors and advanced algorithms. The sensors can capture the dynamic information of the target object, while the algorithms are responsible for processing this information and adjusting the robot's motion trajectory in real time to ensure that it closely follows the target.

[0003] Publication number CN108717553B discloses a method and system for a robot to follow a human body, wherein the method comprises: using the HOG method to detect a human body region on a depth map acquired by the robot; during initialization of following, calculating and saving the apparent features of a color image corresponding to the tracked human body region; the apparent features include color features and texture features; during the following process, judging whether interlacing occurs based on the position change of the tracked human body region in consecutive frame images; after judging that interlacing occurs, controlling the robot to stop moving; and simultaneously, searching for a human body region in subsequent frame images acquired by the robot for an apparent feature similar to the saved tracked human body region until the human body region is found, and then controlling the robot to follow the human body region.

[0004] The current existing solution uses the HOG method to detect human areas on the depth map and relies on color and texture features for tracking. When the robot follows the human body, the human body detection on the depth map using the HOG method is easily affected by environmental interference, resulting in insufficient positioning accuracy. At the same time, the color and texture feature matching method it relies on is easily affected by factors such as lighting changes and occlusion, resulting in feature matching errors. In addition, after interspersed objects, the robot needs to search for and match again, which reduces the real-time and stability of the tracking. Summary of the Invention

[0005] In view of this, the present invention proposes a human body following method and system based on vision and lidar perception, which combines the human body scanning orientation estimation model with lidar data to achieve accurate measurement and positioning of the human body position, enhance the perception ability of mobile robots in complex environments, and improve the accuracy and robustness of human body following.

[0006] The technical solution of the present invention is implemented as follows: The present invention provides a human body following method based on vision and laser radar perception, comprising the following steps:

[0007] S1, the mobile robot is equipped with a camera and a lidar, which collects video images and lidar data through the camera and lidar respectively;

[0008] S2, based on the collected video image, detect the human body based on the target detection model to obtain the coordinate information of the detection box in the image;

[0009] S3, builds a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtains the scanning orientation angle range of the human body based on the coordinate information of the detection box in the image and the human body scanning orientation estimation model;

[0010] S4, extracting the distance of the human body from the lidar data according to the scanning azimuth angle range of the human body, and obtaining the distance between the human body and the mobile robot and the position of the human body;

[0011] S5, judging and controlling the sending of motion instructions based on the distance between the human body and the mobile robot and the position of the human body, so that the robot can follow the movement of the human body.

[0012] Based on the above technical solution, preferably, step S2 includes:

[0013] Input the video image captured by the mobile robot and use the YOLOv8 target detection model to detect human bodies in the video image. The YOLOv8 target detection model outputs the detection frame of the human body detected in the image. The detection frame includes the coordinates of the upper left corner and the lower right corner. The coordinate information of the detection frame in the image is obtained. Let the expression of the detection frame be:

[0014] [(u1, v1), (u2, v2)]

[0015] Where u1 is the x-coordinate of the upper left corner of the detection box, v1 is the y-coordinate of the upper left corner of the detection box, u2 is the x-coordinate of the lower right corner of the detection box, and v2 is the y-coordinate of the lower right corner of the detection box.

[0016] Based on the above technical solution, preferably, in step S3, a human body scanning orientation estimation model is constructed based on the camera's horizontal field of view angle, including:

[0017] Let the center of the image be the origin, the origin of the image coordinate system be (u0, v0), the image coordinate system point be (u, v), and the width of the video image be W, then u0 = W / 2;

[0018] Establish the mapping relationship between the image coordinate system points and the world coordinate system points, and convert the image coordinate system points into the world coordinate system points. The expression is:

[0019]

[0020] In the formula, the image coordinate system point is (u, v, 1), and the world coordinate system point after the image coordinate system point is converted is (x w ,y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is (u0, v0), f represents the focal length, dx and dy represent the width of the pixel in the x-direction and y-direction respectively, and [R, t] represents the camera extrinsic parameter matrix;

[0021] Let the camera coordinate system overlap with the world coordinate system, R is the unit matrix, the translation vector t=0, z w =z c , according to the mapping relationship between the image coordinate system point and the world coordinate system point, the mapping relationship between the world coordinate system point and the image coordinate system point is obtained. The expression is:

[0022] z w u=f / dx·x w +z w u0;

[0023] According to tan(α)=x w / z w , let u = W, substitute into the above expression, and obtain the tangent value of the camera's half horizontal field of view angle, which is expressed as:

[0024] tan(α)=dx·W / (2·f);

[0025] According to the mapping relationship between the image coordinate system point and the world coordinate system point and the half tangent value of the camera's horizontal field of view angle, the equation for the relationship between the angle between the coordinate system point and the optical axis in front of the line of sight and the image coordinate is established to construct a human body scanning orientation estimation model. The expression is:

[0026] θ=arctan[(2u / W-1)·tan(α)]

[0027] Where θ is the angle between the coordinate system point and the optical axis directly in front of the line of sight, and α is half of the camera's horizontal field of view.

[0028] On the basis of the above technical solution, preferably, in step S3, according to the coordinate information of the detection frame in the image, combined with the human body scanning orientation estimation model, obtaining the scanning orientation angle range of the human body includes:

[0029] According to the coordinate information of the detection frame, the range of the human body detection frame is obtained, and the range of the human body detection frame is [u1, u2];

[0030] Substitute the human body detection bounding box range into the human body scanning orientation estimation model to obtain the human body scanning orientation angle range [θ1, θ2], where:

[0031] θ1=arctan[(2u1 / W-1)·tan(α)];

[0032] θ2=arctan[(2u2 / W-1)·tan(α)].

[0033] Based on the above technical solution, preferably, in step S4, extracting the human body distance from the laser radar data according to the scanning azimuth angle range of the human body to obtain the distance between the human body and the mobile robot includes:

[0034] Let the radar angle resolution be a, obtain the lidar data vector, and the laser radar data vector from the first to the Nth item corresponds to the optical axis angle from 0 to 2π in clockwise order. Get the index position of the lidar data vector at the scanning azimuth angle of the human body, and construct the index function. The expression is:

[0035]

[0036] Where k(θ) is the index position of the laser radar data vector at the scanning azimuth angle of the human body, and round() represents the rounding function;

[0037] According to the scanning azimuth angle range [θ1, θ2] of the human body and the index function, the distance vector in the corresponding range is obtained. The expression is:

[0038] D=range[k(θ1):k(θ2)]

[0039] Where range[] is the distance vector measured by the lidar in the corresponding scanning angle range, k(θ1):k(θ2) is the scanning azimuth angle range of the human body from index k(θ1) to k(θ2);

[0040] A distance threshold is preset, and the minimum value whose distance is greater than the distance threshold is obtained from the distance vector corresponding to the scanning azimuth angle range of the human body as the distance between the human body and the mobile robot. The expression is:

[0041] d=min(D(D>r0))

[0042] Where d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

[0043] On the basis of the above technical solution, preferably, in step S4, the position of the human body is calculated according to the scanning azimuth angle range of the human body, and the expression is:

[0044] θ r =(θ1+θ2) / 2

[0045] Where θr The position of the human body.

[0046] Based on the above technical solution, preferably, in step S5, judging and controlling the sending of motion instructions based on the distance between the human body and the mobile robot and the position of the human body, so that the robot follows the movement of the human body, includes:

[0047] A control threshold distance is preset. According to the position of the human body and the distance between the human body and the mobile robot, the distance between the human body and the mobile robot and the control threshold distance are judged. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop command is sent to the chassis to stop the movement of the mobile robot. Otherwise, the mobile robot is controlled by the PID control algorithm to move forward and follow the position of the human body.

[0048] In a second aspect, the present invention further provides a human body following system based on vision and laser radar perception, which is implemented using the human body following method based on vision and laser radar perception as described above, and the system includes:

[0049] The acquisition module is used to carry a camera and a lidar on the mobile robot, collecting video images and lidar data through the camera and lidar respectively;

[0050] The image processing module is used to detect the human body based on the target detection model according to the collected video image and obtain the coordinate information of the detection box in the image;

[0051] The model processing module is used to build a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtain the scanning orientation angle range of the human body based on the coordinate information of the detection box in the image and the human body scanning orientation estimation model;

[0052] The extraction module is used to extract the distance of the human body from the lidar data according to the scanning azimuth angle range of the human body, and obtain the distance between the human body and the mobile robot and the position of the human body;

[0053] The motion control module judges and controls the movement instructions based on the distance between the human body and the mobile robot and the position of the human body, so that the robot can follow the movement of the human body.

[0054] In a third aspect, the present invention also provides an electronic device comprising at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores a program for a human body following method based on vision and lidar perception that can be executed by the processor, and the program for a human body following method based on vision and lidar perception is configured to implement the above-mentioned human body following method based on vision and lidar perception.

[0055] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which is stored a program for a method for following a human body based on vision and lidar perception, which, when executed, implements the method for following a human body based on vision and lidar perception as described above.

[0056] The human tracking method and system based on vision and lidar perception of the present invention have the following beneficial effects compared with the prior art:

[0057] (1) Through a mobile robot equipped with a camera and a lidar, video images and lidar data are collected, and the YOLOv8 target detection model is used to detect the human body in the video image and obtain its coordinate information. A human body scanning orientation estimation model is constructed based on the horizontal field of view of the camera. Combined with the coordinate information of the detection frame, the scanning orientation angle range of the human body is determined, and the distance of the human body is extracted from the lidar data. The distance between the human body and the mobile robot and its orientation are calculated. Through the preset control logic and PID control algorithm, motion instructions are sent to the mobile robot to make it follow the human body. The human body scanning orientation estimation model and lidar data are effectively combined to achieve accurate measurement and positioning of the human body position, improve the perception ability of the mobile robot in complex environments, and improve the accuracy and robustness of human body following.

[0058] (2) By analyzing the coordinates of the detection frame in the image and using the camera focal length, pixel size, and horizontal field of view angle parameters, the points in the image coordinate system are mapped to the world coordinate system, and a human body scanning orientation estimation model is constructed to achieve accurate calculation of the scanning orientation angle range of the human body;

[0059] (3) By constructing an index function and a preset distance threshold, the present invention can efficiently extract the lidar data of the scanning direction where the human body is located, avoid data traversal, significantly improve processing efficiency, and effectively filter noise to improve the reliability of measurement results. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 This is a flow chart of the human body following method based on vision and laser radar perception of the present invention;

[0062] Figure 2Schematic diagram of the geometric relationship between the horizontal projection point of the target human body and the boundary point of its corresponding image field of view in the world coordinate system of the human body tracking method based on vision and lidar perception of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, a human body following method based on vision and laser radar perception of the present invention includes the following steps:

[0065] S1, the mobile robot is equipped with a camera and a lidar, which collects video images and lidar data through the camera and lidar respectively.

[0066] It should be noted that the mobile robot is equipped with a monocular camera and a single-line laser radar. The monocular camera is installed at an appropriate position on the robot to ensure that the field of view of the monocular camera can cover the area that the robot needs to perceive and avoid blind spots; the single-line laser radar is installed on the top or side of the robot to ensure that the scanning range of the single-line laser radar can cover the environment around the robot and overlap with the field of view of the monocular camera to facilitate data fusion. Video images are collected based on the monocular camera, and laser radar data is collected based on the single-line laser radar to provide data support for subsequent target detection, human body scanning orientation estimation, and human body distance extraction.

[0067] In order to ensure that the data collected by the camera and lidar can accurately correspond, data synchronization is required. By installing a time synchronization module on the robot, the time synchronization module can record the acquisition timestamp of each data packet and use these timestamps to align the data of the monocular camera and single-line lidar in the subsequent data processing process.

[0068] S2: Detect the human body based on the target detection model according to the collected video image to obtain the coordinate information of the detection box in the image.

[0069] Wherein, step S2 includes:

[0070] Input the video image captured by the mobile robot and use the YOLOv8 target detection model to detect human bodies in the video image. The YOLOv8 target detection model outputs the detection frame of the human body detected in the image. The detection frame includes the coordinates of the upper left corner and the lower right corner. The coordinate information of the detection frame in the image is obtained. Let the expression of the detection frame be:

[0071] [(u1, v1), (u2, v2)]

[0072] Where u1 is the x-coordinate of the upper left corner of the detection box, v1 is the y-coordinate of the upper left corner of the detection box, u2 is the x-coordinate of the lower right corner of the detection box, and v2 is the y-coordinate of the lower right corner of the detection box.

[0073] It should be noted that video images are acquired in real time from the mobile robot's monocular camera and used as input to the YOLOv8 target detection model. The YOLOv8 target detection model performs forward propagation on the image and outputs the detection frame of the detected human body. The detection frame output by the YOLOv8 target detection model includes the coordinates of the upper left corner and the lower right corner, which represent the position of the human body in the image.

[0074] S3, builds a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtains the scanning orientation angle range of the human body based on the coordinate information of the detection box in the image and the human body scanning orientation estimation model.

[0075] Among them, in step S3, a human body scanning orientation estimation model is constructed based on the camera's horizontal field of view angle, including:

[0076] Let the center of the image be the origin, the origin of the image coordinate system be (u0, v0), the image coordinate system point be (u, v), and the width of the video image be W, then u0 = W / 2;

[0077] Establish the mapping relationship between the image coordinate system points and the world coordinate system points, and convert the image coordinate system points into the world coordinate system points. The expression is:

[0078]

[0079] In the formula, the image coordinate system point is (u, v, 1), and the world coordinate system point after the image coordinate system point is converted is (x w ,y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is (u0, v0), f represents the focal length, dx and dy represent the width of the pixel in the x-direction and y-direction respectively, and [R, t] represents the camera extrinsic parameter matrix;

[0080] Let the camera coordinate system overlap with the world coordinate system, R is the unit matrix, the translation vector t=0, z w =z c , according to the mapping relationship between the image coordinate system point and the world coordinate system point, the mapping relationship between the world coordinate system point and the image coordinate system point is obtained. The expression is:

[0081] z w u=f / dx·xw +z w u0;

[0082] According to tan(α)=x w / z w , let u = W, substitute into the above expression, and obtain the tangent value of the camera's half horizontal field of view angle, which is expressed as:

[0083] tan(α)=dx·W / (2·f);

[0084] According to the mapping relationship between the image coordinate system point and the world coordinate system point and the half tangent value of the camera's horizontal field of view angle, the equation for the relationship between the angle between the coordinate system point and the optical axis in front of the line of sight and the image coordinate is established to construct a human body scanning orientation estimation model. The expression is:

[0085] θ=arctan[(2u / W-1)·tan(α)]

[0086] Where θ is the angle between the coordinate system point and the optical axis directly in front of the line of sight, and α is half of the camera's horizontal field of view. The camera's horizontal field of view is usually provided by the camera manufacturer or calculated through self-calibration.

[0087] like Figure 2 As shown in the figure, assuming that the position of the camera center and the position of the radar center overlap on the x-axis and z-axis, and the z-axis is directly in front of the camera, the angle θ between a point on the person in the image and the optical axis directly in front of the line of sight is equivalent to the angle θ between the radar scanning point and the optical axis directly in front of the camera. Figure 2 Shows the target human body's horizontal projection point P(x w ,0,z w ), where z w =d represents the depth of field distance of a certain point on the human body, and its corresponding image field boundary point Q(Q xw ,0,Q zw ), assuming that the horizontal field of view angle of the camera is known to be 2α, which can be obtained by calibration after installing the camera and radar, and then according to Figure 2 The geometric relationship between P and Q and the mapping relationship between the image coordinate system points and the world coordinate system points are obtained:

[0088] tan(θ)·f / dx=(u-u0);

[0089] tan(α)·f / dx=(W-u0).

[0090] Dividing the above two formulas yields:

[0091] θ=arctan[(2u / W-1)·tan(α)].

[0092] In this embodiment, the human body detection frame range is obtained according to the coordinate information of the detection frame, and the human body detection frame range is [u1, u2];

[0093] Substitute the human body detection bounding box range into the human body scanning orientation estimation model to obtain the human body scanning orientation angle range [θ1, θ2], where:

[0094] θ1=arctan[(2u1 / W-1)·tan(α)];

[0095] θ2=arctan[(2u2 / W-1)·tan(α)].

[0096] In this implementation, the coordinates of the detection frame in the image are analyzed, and the points in the image coordinate system are mapped to the world coordinate system using parameters such as camera focal length, pixel size, and horizontal field of view, thereby achieving an accurate estimation of the scanning azimuth angle range of the human body.

[0097] S4, according to the scanning azimuth angle range of the human body, extract the human body distance from the lidar data to obtain the distance between the human body and the mobile robot and the human body's position.

[0098] Step S4 includes:

[0099] Let the radar angle resolution be a, obtain the lidar data vector, the data vector contains N items, the lidar data vector from the first item to the Nth item corresponds to the optical axis angle from 0 to 2π in clockwise order, obtain the lidar data vector index position where the scanning azimuth angle of the human body is located, and construct the index function, the expression is:

[0100]

[0101] Where k(θ) is the index position of the laser radar data vector at the scanning azimuth angle of the human body, and round() represents the rounding function;

[0102] According to the scanning azimuth angle range [θ1, θ2] of the human body and the index function, the distance vector in the corresponding range is obtained. The expression is:

[0103] D=range[k(θ1):k(θ2)]

[0104] Where range[] is the distance vector measured by the lidar in the corresponding scanning angle range, k(θ1):k(θ2) is the scanning azimuth angle range of the human body from index k(θ1) to k(θ2);

[0105] A distance threshold is preset, and the minimum value whose distance is greater than the distance threshold is obtained from the distance vector corresponding to the scanning azimuth angle range of the human body as the distance between the human body and the mobile robot. The expression is:

[0106] d=min(D(D>r0))

[0107] Where d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

[0108] In step S4, the position of the human body is calculated based on the scanning angle range of the human body, and the expression is:

[0109] θ r =(θ1+θ2) / 2

[0110] Where θ r The position of the human body.

[0111] It should be noted that by constructing an index function, we can accurately find the index position of the human body's scanning azimuth angle in the lidar data vector. According to the range of the human body's scanning azimuth angle, we use the index function to efficiently extract the distance vector within the corresponding range, avoiding traversal of the entire lidar data vector and significantly improving data processing efficiency.

[0112] By presetting the distance threshold, noise data is effectively filtered out, thereby improving the reliability of the measurement results. In this embodiment, the human body scanning orientation estimation model is combined with the lidar data to achieve accurate measurement and positioning of the human body position, thereby improving the perception ability of the mobile robot in complex environments, enabling the mobile robot to respond to various scenarios more intelligently, and improving safety and reliability.

[0113] S5, judging and controlling the sending of motion instructions based on the distance between the human body and the mobile robot and the position of the human body, so that the robot can follow the movement of the human body.

[0114] Among them, step S5 includes: presetting a control threshold distance, judging the distance between the human body and the mobile robot and the control threshold distance according to the position of the human body and the distance between the human body and the mobile robot. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop command is sent to the chassis to stop the movement of the mobile robot. Otherwise, the mobile robot is controlled by the PID control algorithm to move forward and follow along the position of the human body.

[0115] Specifically, the control threshold distance in this embodiment is 0.8m. The control threshold distance is used to judge the relative position relationship between the human body and the mobile robot. When the distance from the human body to the mobile robot is less than the control threshold distance, it is considered that the robot has approached the human body to a certain extent and needs to stop moving to avoid collision or interference. The control system sends a stop command to the chassis of the mobile robot. After receiving the stop command, the chassis executes the corresponding stop operation to stop the mobile robot. If the distance from the human body to the mobile robot is greater than or equal to the control threshold distance, the movement of the mobile robot is controlled by the PID control algorithm. The PID control algorithm calculates the control amount based on the deviation between the current position of the human body and the current position of the mobile robot, and sends it to the chassis of the mobile robot. The chassis adjusts the speed and direction of the mobile robot based on the received control amount, so that it moves forward and follows the position of the human body. In the process of the mobile robot following the human body, the distance from the human body to the mobile robot and the position of the human body are continuously monitored. According to the monitoring results, the control parameters of the PID control algorithm are adjusted in real time to ensure that the mobile robot can stably and accurately follow the movement of the human body.

[0116] The human body orientation and distance perception method proposed in this embodiment is significantly characterized by its computational simplicity and high efficiency. The core computational burden of this method is mainly concentrated in the human target detection stage, while the process of extracting radar scanning orientation and distance information is relatively simple, which makes the method less demanding on computing resources, thereby improving the operating efficiency of the overall system. Furthermore, this method can quickly scan the surrounding environment based on the laser radar, and accurately measure the distance between the human body and the mobile robot and the orientation of the human body, so that the mobile robot can track and follow the movement of the human body in real time and accurately.

[0117] The present invention also provides a human body following system based on vision and laser radar perception, which is implemented using the human body following method based on vision and laser radar perception as described above. The system includes:

[0118] The acquisition module is used to carry a camera and a lidar on the mobile robot, collecting video images and lidar data through the camera and lidar respectively;

[0119] The image processing module is used to detect the human body based on the target detection model according to the collected video image and obtain the coordinate information of the detection box in the image;

[0120] The model processing module is used to build a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtain the scanning orientation angle range of the human body based on the coordinate information of the detection box in the image and the human body scanning orientation estimation model;

[0121] The extraction module is used to extract the distance of the human body from the lidar data according to the scanning azimuth angle range of the human body, and obtain the distance between the human body and the mobile robot and the position of the human body;

[0122] The motion control module judges and controls the movement instructions based on the distance between the human body and the mobile robot and the position of the human body, so that the robot can follow the movement of the human body.

[0123] It should be noted that this system is a system corresponding to the above-mentioned offshore equipment weighing and measurement method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of this system and can achieve the same technical effects.

[0124] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0125] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0129] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, ROM, RAM, a magnetic disk, or an optical disk.

[0130] In addition, it should be noted that, in the system and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present invention. Moreover, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it will be understood that all or any steps or components of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or a network of computing devices in hardware, firmware, software or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.

[0131] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing system. The computing system can be a well-known general-purpose system. Therefore, the purpose of the present invention can also be achieved simply by providing a program product containing program code that implements the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Certain steps can be performed in parallel or independently of each other.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A human body following method based on vision and laser radar perception, characterized in that: The following steps are involved: S1, the mobile robot is equipped with a camera and a lidar, which collects video images and lidar data through the camera and lidar respectively; S2, detecting the human body based on the target detection model according to the collected video image, and obtaining the coordinate information of the detection box in the image; S3, building a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtaining the scanning orientation angle range of the human body in combination with the human body scanning orientation estimation model according to the coordinate information of the detection frame in the image; S4, extracting the distance of the human body from the laser radar data according to the scanning azimuth angle range of the human body, and obtaining the distance between the human body and the mobile robot and the azimuth of the human body; S5, judging and controlling the sending of motion instructions according to the distance between the human body and the mobile robot and the position of the human body, so that the robot can move along with the human body.

2. The human body following method based on vision and laser radar perception as claimed in claim 1, characterized in that: Step S2 includes: Input the video image collected by the mobile robot, use the YOLOv8 target detection model to detect human body in the video image, and the YOLOv8 target detection model outputs the detection frame of the human body detected in the image. The detection frame includes the coordinates of the upper left corner and the lower right corner. The coordinate information of the detection frame in the image is obtained, and the expression of the detection frame is: [(u1, v1), (u2, v2)] Where u1 is the x-coordinate of the upper left corner of the detection box, v1 is the y-coordinate of the upper left corner of the detection box, u2 is the x-coordinate of the lower right corner of the detection box, and v2 is the y-coordinate of the lower right corner of the detection box.

3. The human body following method based on vision and laser radar perception as claimed in claim 2, characterized in that: In step S3, a human body scanning orientation estimation model is constructed based on the camera horizontal field of view angle, including: Let the center point of the image be the origin, the origin of the image coordinate system be (u0, v0), the image coordinate system point be (u, v), and the width of the video image be W, then u0 = W / 2; Establish the mapping relationship between the image coordinate system points and the world coordinate system points, and convert the image coordinate system points into the world coordinate system points. The expression is: In the formula, the image coordinate system point is (u, v, 1), and the world coordinate system point after the image coordinate system point is converted is (x w ,y w , z w ), z c represents the depth of field in the camera coordinate system, the origin of the image coordinate system is (u0, v0), f represents the focal length, dx and dy represent the width of the pixel in the x direction and y direction respectively, and [R, t] represents the camera extrinsic parameter matrix; Let the camera coordinate system overlap with the world coordinate system, R is the unit matrix, the translation vector t = 0, z w =z c , according to the mapping relationship between the image coordinate system point and the world coordinate system point, the mapping relationship between the world coordinate system point and the image coordinate system point is obtained, and the expression is: z w ·u=f / dx·x w +z w ·u0; According to tan(α)=x w / z w , let u = W, substitute into the above expression, and obtain the tangent value of the camera's half horizontal field of view angle, which is expressed as: tan(α)=dx·W / (2·f); According to the mapping relationship between the image coordinate system point and the world coordinate system point and the half tangent value of the camera's horizontal field of view, the equation for the relationship between the angle between the coordinate system point and the optical axis directly in front of the line of sight and the image coordinates is established to construct a human body scanning orientation estimation model, which is expressed as follows: θ=arctan[(2u / W-1)·tan(α)] Where θ is the angle between the coordinate system point and the optical axis directly in front of the line of sight, and α is half of the horizontal field of view of the camera.

4. The human body following method based on vision and laser radar perception as claimed in claim 3, characterized in that: In step S3, according to the coordinate information of the detection frame in the image, combined with the human body scanning orientation estimation model, the scanning orientation angle range of the human body is obtained, including: According to the coordinate information of the detection frame, the range of the human body detection frame is obtained, and the range of the human body detection frame is [u1, u2]; Substitute the human body detection bounding box range into the human body scanning orientation estimation model to obtain the human body scanning orientation angle range [θ1, θ2], where: θ1=arctan[(2u1 / W-1)·tan(α)]; θ2=arctan[(2u2 / W-1)·tan(α)].

5. The human body following method based on vision and laser radar perception as claimed in claim 4, characterized in that: In step S4, according to the scanning azimuth angle range of the human body, the human body distance is extracted from the laser radar data to obtain the distance between the human body and the mobile robot, including: Let the radar angle resolution be a, obtain the laser radar data vector, the laser radar data vector from the first item to the Nth item in clockwise direction corresponds to the optical axis angle of 0 to 2π, obtain the laser radar data vector index position where the scanning azimuth angle of the human body is located, and construct the index function, the expression is: Where k(θ) is the index position of the laser radar data vector where the scanning azimuth angle of the human body is located, and round() represents the rounding function; According to the scanning azimuth angle range [θ1, θ2] of the human body and the index function, the distance vector in the corresponding range is obtained. The expression is: D=range[k(θ1):k(θ2)] Where range[] is the distance vector measured by the laser radar in the corresponding scanning angle range, k(θ1):k(θ2) is the scanning azimuth angle range of the human body from index k(θ1) to k(θ2); The distance threshold is preset, and the distance vector corresponding to the scanning azimuth angle range of the human body is obtained, and the distance is greater than the distance threshold, and the smallest value is taken as the distance between the human body and the mobile robot. The expression is: d=min(D(D>r0)) Where d is the distance between the human body and the mobile robot, r0 is the distance threshold, and min() is the minimum value acquisition function.

6. The human body following method based on vision and laser radar perception as claimed in claim 5, characterized in that: In step S4, the position of the human body is calculated according to the scanning position angle range of the human body, and the expression is: i r =(θ1+θ2) / 2 In the formula, θ r The position of the human body.

7. The human body following method based on vision and laser radar perception as claimed in claim 6, characterized in that: In step S5, judging and controlling the sending of motion instructions according to the distance between the human body and the mobile robot and the position of the human body, so that the robot follows the movement of the human body, includes: A control threshold distance is preset, and the distance between the human body and the mobile robot and the control threshold distance are judged according to the position of the human body and the distance between the human body and the mobile robot. If the distance between the human body and the mobile robot is less than the control threshold distance, a stop command is sent to the chassis to stop the movement of the mobile robot. Otherwise, the mobile robot is controlled by the PID control algorithm to move forward and follow along the position of the human body.

8. A human body following system based on vision and laser radar perception, implemented by the human body following method based on vision and laser radar perception as claimed in any one of claims 1 to 7, characterized in that: The system comprises: The acquisition module is used for the mobile robot to carry a camera and a laser radar, and to collect video images and laser radar data through the camera and the laser radar respectively; An image processing module is used to detect the human body based on the acquired video image and the target detection model to obtain the coordinate information of the detection frame in the image; The model processing module is used to construct a human body scanning orientation estimation model based on the camera's horizontal field of view, and obtain the scanning orientation angle range of the human body based on the coordinate information of the detection frame in the image and the human body scanning orientation estimation model; An extraction module is used to extract the distance of the human body from the laser radar data according to the scanning azimuth angle range of the human body, and obtain the distance between the human body and the mobile robot and the azimuth of the human body; The motion control module judges and controls the sending of motion instructions according to the distance between the human body and the mobile robot and the position of the human body, so that the robot can follow the movement of the human body.

9. An electronic device, characterized in that: It includes at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores a human body following method program based on vision and laser radar perception that can be executed by the processor, and the human body following method program based on vision and laser radar perception is configured to implement a human body following method based on vision and laser radar perception as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a human body following method program based on vision and laser radar perception, which, when executed, implements a human body following method based on vision and laser radar perception as claimed in any one of claims 1 to 7.

Images