Pedestrian Spatial Information Perception Method and Device Based on ZED Stereo Camera

Abstract

This invention discloses a pedestrian spatial information perception method and device based on a ZED stereo camera, mainly used for intelligent perception of pedestrian spatial position and movement speed by guide robots in public scenes such as exhibition halls. The method utilizes a ZED binocular vision camera to collect real-time data from the scene and uploads it to a cloud server. Preprocessed RGB data is input into a deployed human keypoint detection network to obtain two-dimensional information of human keypoints. A pedestrian bounding box is generated based on the two-dimensional keypoint information of the upper body's main area. Continuous tracking of multiple targets is performed on multiple consecutive frames. Three-dimensional spatial coordinates of human keypoints in the corresponding areas are obtained by combining point cloud data, and the pedestrian's spatial position and movement speed are calculated. Finally, the guide robot performs body movement control based on the acquired pedestrian spatial information to complete intelligent guide tasks such as autonomous following and obstacle avoidance, thereby increasing the flexibility of the guide robot and enhancing the interactive experience for visitors.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular to a method and apparatus for perceiving pedestrian spatial information based on a ZED stereo camera. Background Technology

[0002] Using intelligent tour guide robots to replace human guides in public places such as exhibition halls and museums can effectively save manpower. Tour guide robots need to intelligently perceive their environment, and pedestrians, as dynamic targets in the scene, have uncertainties. Therefore, intelligent perception of spatial information such as the spatial position and movement speed of pedestrians is of great significance.

[0003] Vision is a crucial way for robots to acquire external information. One approach involves first obtaining pedestrian detection bounding boxes from 2D images, and then using coordinate system transformations and depth or point cloud information to determine the pedestrian's spatial position. This method relies heavily on the accuracy of the pedestrian detection bounding boxes. However, pedestrians are flexible and their body postures constantly change. Traditional pedestrian detection bounding boxes introduce significant background noise when pedestrians' limbs change, leading to substantial errors in subsequent 2D-to-3D spatial transformations based on these bounding boxes. This further affects the estimation of the pedestrian's spatial position and movement speed. Summary of the Invention

[0004] To address the shortcomings of existing technologies and improve the accuracy of identifying pedestrian spatial location and movement speed, this invention adopts the following technical solution:

[0005] A method for perceiving pedestrian spatial information based on a stereo camera includes the following steps:

[0006] Step S1: Acquire real-time image data from the stereo camera, including RGB image data and point cloud data;

[0007] Step S2: Detect human key points through RGB image to obtain the two-dimensional key point information of pedestrians. Determine the main body area of ​​the upper body of pedestrians based on the dynamic characteristics of pedestrians. Combine the two-dimensional key point information of the main body area of ​​the upper body of pedestrians to generate pedestrian bounding boxes. Use the pedestrian bounding boxes as pedestrian detection boxes.

[0008] The generated pedestrian bounding box is expanded proportionally, and the expanded pedestrian bounding box is used as the pedestrian detection box. Since the generated pedestrian bounding box is the smallest bounding box of the pedestrian, and considering that the smallest pedestrian bounding box is only the smallest bounding box of the pedestrian skeleton, the smallest pedestrian bounding box needs to be expanded; the area ratio of the expanded pedestrian bounding box to the smallest pedestrian bounding box is 1.2.

[0009] Step S3: Based on the two-dimensional keypoint similarity features and pedestrian detection boxes, perform multi-target tracking of pedestrians in multiple consecutive frames of images;

[0010] Step S4: For continuously tracked pedestrians, obtain the pedestrian's three-dimensional key point information based on their two-dimensional key point information and point cloud data, calculate the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculate the pedestrian's moving speed based on the frame interval to generate the pedestrian's real-time spatial information relative to the stereo camera.

[0011] Further, in step S2, RGB image data is acquired, forward inference is performed using a human keypoint detection network, and a keypoint heatmap and partial association domain are output. Two-dimensional keypoints are extracted and grouped according to the keypoint heatmap and partial association domain. Two-dimensional keypoints belonging to the same pedestrian are matched to the current pedestrian to obtain the coordinates of the two-dimensional keypoints of each pedestrian in the current image.

[0012] Further, step S3 includes the following steps:

[0013] Step S3.1: Obtain pedestrian motion features based on the pedestrian detection bounding box; obtain pedestrian appearance features based on the two-dimensional keypoint similarity features;

[0014] Step S3.2: Based on the pedestrian's motion characteristics and appearance characteristics, obtain the measured state information of the pedestrian at the current time t;

[0015] Step S3.3: Associate the historical trajectory with the actual pedestrian status information at time t to obtain the ID of each pedestrian at time t; the purpose of data association is to match the detection result at the current time with the historical trajectory through appearance, geometry and other features to determine the ID of each person detected in the current frame;

[0016] Step S3.4: Update the historical trajectory using the ID of each pedestrian at time t, thereby continuously tracking the pedestrians.

[0017] Furthermore, in step S3.1, the two-dimensional key point similarity feature is calculated using the target key point similarity evaluation index OKS, and a preset threshold is used to determine whether the two-dimensional key points are related.

[0018] Furthermore, the data association in step S3.3 involves associating each frame with both motion features and appearance features, applying linear weights to obtain the final association matrix, and then using the Hungarian matching algorithm based on this association matrix to obtain the pedestrian matching results between frames.

[0019] Further, step S4 includes the following steps:

[0020] Step S4.1: Based on the main body area of ​​the pedestrian's upper body, filter the key points using the confidence level of the two-dimensional key points. Based on the filtered two-dimensional key points, obtain point cloud data to get the three-dimensional coordinate set of the pedestrian's key points.

[0021] Step S4.2: For each pedestrian target, calculate the mean of the three-dimensional coordinates based on the set of three-dimensional coordinates of its key points as the spatial position coordinates of the pedestrian target; calculate the actual distance of the pedestrian relative to the stereo camera based on the Euclidean distance; calculate the distance the pedestrian moves relative to the stereo camera in the time interval between the current time and the previous time based on the pedestrian's spatial position coordinates at the current time and the previous time; and combine the time taken to obtain the pedestrian's moving speed at the current time.

[0022] Furthermore, the formula for the pedestrian's moving speed in step S4.2 is as follows:

[0023]

[0024] Where X, Y, and Z are the three-dimensional spatial coordinates of the pedestrian, i represents the currently tracked pedestrian ID, and t represents the time of the current frame. , , This represents the spatial coordinates of the pedestrian relative to the stereo camera in the current frame. , , They represent in The spatial coordinates of the relative position of the stereo camera at that moment. m represents the frame interval, and f is the frame rate of the stereo camera.

[0025] A pedestrian spatial information sensing device based on a stereo camera includes a real-time image data acquisition module, a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module.

[0026] The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera;

[0027] The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain two-dimensional key point information of the pedestrian. Based on the dynamic characteristics of the pedestrian, it determines the main body area of ​​the upper body of the pedestrian. Combined with the two-dimensional key point information of the main body area of ​​the upper body of the pedestrian, it generates a pedestrian bounding box and uses the pedestrian bounding box as the pedestrian detection box.

[0028] The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes.

[0029] The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information combined with point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera.

[0030] A pedestrian spatial information perception method based on a ZED camera is disclosed. According to the aforementioned stereo camera-based pedestrian spatial information perception method, a ZED binocular vision camera mounted on a tour guide robot collects real-time image data and transmits it to a cloud server for pedestrian detection bounding box acquisition, multi-target tracking, and real-time spatial information generation. The RGB images undergo preprocessing, including image resizing and encoding, to improve subsequent data transmission efficiency. The preprocessed data is transmitted to the cloud server via a message middleware. The field of view of the ZED binocular vision camera is configured to match the upper body backbone area of ​​the pedestrian. Based on the pedestrian's dynamic characteristics and the field of view of the ZED binocular vision camera on the tour guide robot, the upper body backbone area of ​​the pedestrian is determined. The cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the tour guide robot. The tour guide robot then performs movement control based on the real-time spatial information to complete the tour guide task.

[0031] A pedestrian spatial information sensing device based on a ZED camera includes a cloud server and a ZED binocular vision camera mounted on a tour guide robot. The cloud server includes a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module.

[0032] The ZED binocular vision camera acquires RGB image data and point cloud data in real time and transmits them to the cloud server.

[0033] The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain the two-dimensional key point information of the pedestrian. Based on the pedestrian's dynamic characteristics, it determines the main body region of the upper body of the pedestrian. Combining the two-dimensional key point information of the main body region of the upper body of the pedestrian, it generates a pedestrian bounding box, which is used as the pedestrian detection box. The field of view of the ZED binocular vision camera is set in conjunction with the main body region of the upper body of the pedestrian. The main body region of the upper body of the pedestrian is determined based on the pedestrian's dynamic characteristics and the field of view of the ZED binocular vision camera on the guide robot.

[0034] The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes.

[0035] The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information and point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera.

[0036] The guided tour robot obtains real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, and performs movement control of its own body to complete the guided tour task.

[0037] The advantages and beneficial effects of this invention are as follows:

[0038] This invention employs a contactless ZED binocular vision camera mounted on the head of a tour guide robot to intelligently perceive pedestrians within a scene. It utilizes a lightweight human keypoint detection network deployed in the cloud to acquire two-dimensional keypoint information, effectively addressing the problem of insufficient computing power on the robot itself by fully leveraging the powerful computing and storage resources of the cloud server. Considering human motion characteristics and the tour guide robot's field of vision, the keypoints of the upper body's main trunk region are determined as the target area. This area does not include human keypoints with significant variations, such as those on the arms. The pedestrian detection box generated based on this area is more stable, and the accuracy of calculating the pedestrian's spatial position and movement speed using the three-dimensional coordinates of the pedestrian keypoints in this area is also higher. Attached Figure Description

[0039] Figure 1 This is a flowchart of the pedestrian spatial information perception method based on stereo camera vision of the present invention.

[0040] Figure 2 This is a schematic diagram illustrating the scenario-based application of pedestrian spatial information perception based on ZED vision in a face-guided robot according to an embodiment of the present invention.

[0041] Figure 3a This is a diagram of key human body parameters in an embodiment of the present invention.

[0042] Figure 3b This is a schematic diagram of generating a pedestrian bounding box based on two-dimensional key points of the upper body of a pedestrian in an embodiment of the present invention.

[0043] Figure 3c This is a schematic diagram of three-dimensional key points in an embodiment of the present invention.

[0044] Figure 4 This is a visual flowchart of pedestrian spatial information perception based on ZED vision in a real-world test scenario according to an embodiment of the present invention.

[0045] Figure 5 This is a comparison diagram of pedestrian ranging errors based on human body key points and traditional human body detection boxes in this embodiment of the invention.

[0046] Figure 6 This is a flowchart of the pedestrian spatial information perception method based on a ZED camera according to the present invention.

[0047] Figure 7 This is a schematic diagram of the pedestrian spatial information sensing device based on a stereo camera in an embodiment of the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0049] like Figure 1 As shown, the pedestrian spatial information perception method based on stereo camera vision includes the following steps:

[0050] Step S1: Acquire real-time image data from the stereo camera, including RGB image data and point cloud data;

[0051] In embodiments of the present invention, such as Figure 2 As shown, the ZED binocular vision camera is mounted on the head of the wheeled robot, approximately 1.2 meters above the ground. RGB and point cloud data from the ZED camera (left eye) are acquired at a frame rate of 30Hz, with the RGB color image resolution being 1280×720. The RGB images undergo preprocessing, including resizing and encoding, to improve subsequent data transmission efficiency. The resized size is 456×256, consistent with the input of the subsequent human keypoint detection network. The preprocessed data is then transmitted to the cloud server via a message middleware.

[0052] Step S2: Detect human key points through RGB image to obtain the two-dimensional key point information of pedestrians. Determine the main body area of ​​the upper body of pedestrians based on the dynamic characteristics of pedestrians. Combine the two-dimensional key point information of the main body area of ​​the upper body of pedestrians to generate pedestrian bounding boxes. Use the pedestrian bounding boxes as pedestrian detection boxes.

[0053] RGB image data is acquired, and a human keypoint detection network is used for forward inference to output a keypoint heatmap and a partial association domain. Two-dimensional keypoints are extracted and grouped based on the keypoint heatmap and partial association domain. Two-dimensional keypoints belonging to the same pedestrian are matched to the current pedestrian to obtain the coordinates of the two-dimensional keypoints of each pedestrian in the current image.

[0054] In this embodiment of the invention, the cloud server receives real-time data from the ZED camera collected by the robot and inputs it into the deployed human keypoint detection network for forward inference. The human keypoint detection network adopts LightWeightOpenPose, and its backbone network adopts an improved MobileNet, which can meet the real-time requirements. Through this framework, the two-dimensional keypoint information of pedestrians in the image is obtained. The network input is decoded RGB data of size [1,3,256,456]. The outputs of the forward inference network are keypoint heatmaps and Part Affinity Fields (PAFs) of size [1, 19, 32, 57] and [1, 38, 32, 57], respectively. All keypoints are extracted and grouped according to the heatmaps and PAFs. Keypoints belonging to the same pedestrian are matched to the current pedestrian to obtain the keypoint coordinates of each pedestrian in the current image. Assuming that the number of pedestrians detected in the current image is N, the output is N×[18,3], where 18 represents that the number of keypoints for each pedestrian is 18, and 3 represents the horizontal axis x, vertical axis y and confidence of the keypoint in the image coordinate system. The confidence range is 0-1.

[0055] Because pedestrians exhibit dynamic posture changes, especially with significant shifts in the limbs, and because the robot's field of vision only shows the upper half of the body when the person is close, considering both factors, and based on the pedestrian's dynamic characteristics and the robot's field of vision, the upper body's main torso area is determined as our target area. The key points within this area are: {0:NOSE, 1:NECK, 2:RIGHT_SHOULDER, 5:LEFT_SHOULDER, 8:RTGHT_HIP, 11:LEFT_HIP, 14:RIGHT_EYE, 15:LEFT_EYE, 16:RIGHT_EAR, 17:LEFT_EAR}. A pedestrian bounding box is generated based on these two-dimensional key points, as shown below. Figure 3a As shown in the "Human Body Key Points" section; based on the 2D key points of this region, the minimum bounding box is generated using OpenCV's `boundingRect` function, as follows. Figure 3b P1 in the 2D keypoint diagram is shown as a dashed line. Considering that this bounding box is only the minimum bounding box of the pedestrian skeleton, it needs to be appropriately expanded. The expanded bounding box is shown as P2, represented by a solid line. The area ratio of P2 to P1 is 1.2. This bounding box is used as the pedestrian detection box in the subsequent pedestrian tracking algorithm.

[0056] Step S3: Based on the two-dimensional keypoint similarity features and pedestrian detection boxes, perform multi-target tracking of pedestrians in multiple consecutive frames of images, including the following steps:

[0057] Step S3.1: Obtain pedestrian motion features based on the pedestrian detection box; obtain pedestrian appearance features based on the two-dimensional keypoint similarity features; calculate the similarity of the two-dimensional keypoints using the target keypoint similarity evaluation index OKS, and determine whether the two-dimensional keypoints are related by a preset threshold.

[0058] Step S3.2: Based on the pedestrian's motion characteristics and appearance characteristics, obtain the measured state information of the pedestrian at the current time t;

[0059] Step S3.3: Associate the historical trajectory with the measured pedestrian status information at time t to obtain the ID of each pedestrian at time t; the data association is to perform two associations on each frame, namely motion features and appearance features, and perform linear weighting to obtain the final association matrix. Based on this association matrix, the Hungarian matching algorithm is used to obtain the pedestrian matching results between each frame.

[0060] Step S3.4: Update the historical trajectory using the ID of each pedestrian at time t.

[0061] In this embodiment of the invention, for pedestrian targets in each frame of RGB images transmitted to the cloud, we assign them unique identity information (ID) for continuous inter-frame tracking using a multi-target tracking method. The multi-target pedestrian tracking method employs the DeepSort algorithm principle. Pedestrian motion features are obtained based on the pedestrian detection boxes generated in step S2, and pedestrian appearance features are obtained based on two-dimensional keypoint similarity features. Similarity is calculated using OKS (object keypoint similarity), and a set threshold is used to determine whether the association is successful. By fusing these two feature information, the measured state information of the pedestrian at time t can be obtained. Then, the historical trajectory is correlated with the state result at time t, and the correlation between motion features and appearance features is linearly weighted to obtain the final correlation matrix. Based on this correlation matrix, the Hungarian matching algorithm is used to obtain the matching result, thus obtaining the ID of each pedestrian target at time t. Finally, the historical trajectory is updated using the result at time t, thereby obtaining the pedestrian's identity ID.

[0062] The purpose of data association is to match the detection results at the current moment with historical trajectories using features such as appearance and geometry to determine the ID of each person detected in the current frame. The final association matrix is ​​obtained by linearly weighting two types of association: motion features and appearance features. The matching result is obtained by using the Hungarian matching algorithm based on this association matrix.

[0063] Step S4: For continuously tracked pedestrians, obtain the pedestrian's 3D keypoint information based on their 2D keypoint information and point cloud data, calculate the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculate the pedestrian's moving speed based on the frame interval to generate real-time spatial information of the pedestrian relative to the stereo camera, including the following steps:

[0064] Step S4.1: Based on the main body area of ​​the pedestrian's upper body, filter the key points using the confidence level of the two-dimensional key points. Based on the filtered two-dimensional key points, obtain point cloud data to get the three-dimensional coordinate set of the pedestrian's key points.

[0065] In this embodiment of the invention, based on the main body region of the upper body of the pedestrian determined in step 2, the key points are screened in combination with the confidence level. Key points with a confidence level greater than 0.6 participate in the subsequent calculation. Then, based on the obtained ZED point cloud data, the corresponding three-dimensional key points are retrieved to obtain the corresponding set of three-dimensional coordinates of the pedestrian key points.

[0066] The human keypoint detection network outputs a set of 2D keypoint coordinates of pedestrians in the image coordinate system. )}, where k is the number of key points, and the value of k is 0, 1, 2, 5, 8, 11, 14, 15, 16, 17. For the corresponding confidence level, such as Figure 3b The solid origin is shown in the "2D Keypoint Schematic Diagram". By combining point cloud information acquired by the ZED camera, we can obtain the corresponding 3D coordinates of the pedestrian based on the main pedestrian area and the confidence level of the 2D keypoints. The 3D coordinates corresponding to all 2D keypoints of the pedestrian are shown below. Figure 3c As shown in the "3D Key Point Diagram", the set of three-dimensional coordinates of human body key points, including the main body region, is as follows: )},like Figure 3c As shown by the triangle icon, this coordinate system is based on the left eye camera of the ZED camera and is set through the camera parameter COORDINATE_SYSTEM.LEFT_HANDED_Y_UP. The coordinate system and the directions of the XYZ axes are as follows. Figure 2 As shown;

[0067] Step S4.2: For each pedestrian target, calculate the mean of the three-dimensional coordinates based on the set of three-dimensional coordinates of its key points as the spatial position coordinates of the pedestrian target; calculate the actual distance of the pedestrian relative to the stereo camera based on the Euclidean distance; calculate the distance the pedestrian moves relative to the stereo camera in the time interval between the current time and the previous time based on the pedestrian's spatial position coordinates at the current time and the previous time; and combine the time taken to obtain the pedestrian's moving speed at the current time.

[0068] In this embodiment of the invention, the mean value of the obtained set of three-dimensional coordinates of key points on the pedestrian's body is calculated as the spatial three-dimensional coordinate position of the target pedestrian. Where N=10, representing the number of keypoints in the upper body region of each pedestrian; i represents the currently tracked pedestrian ID. Let the time of the current frame be t, then the spatial position of the pedestrian relative to the robot in the current frame is... ,exist The spatial position of the robot at any given time is Then the pedestrian's moving speed is ,in, Where f is the ZED camera frame rate, m represents the frame interval, and considering the total algorithm time, the value of m is... ;

[0069] like Figure 4 As shown, the ZED camera acquires RGB data and point cloud data, then obtains the two-dimensional coordinates of human keypoints based on the human keypoint detection algorithm, and finally combines the two-dimensional keypoint information and the corresponding point cloud information to obtain the three-dimensional coordinates of the human keypoints. To verify the effectiveness of the method of this invention, approximately 1000 frames of images were acquired. The distance from the pedestrian to the camera was calculated using a human detection bounding box based on YOLOv3 combined with point cloud information, and the distance from the pedestrian to the camera was calculated based on keypoints in the main body region of the human body according to this invention. Comparative tests and statistics were conducted, and the distance from the pedestrian to the camera was calculated using the Euclidean distance formula. Where (X, Y, Z) are the three-dimensional spatial coordinates of the pedestrian, such as Figure 5 As shown, the dark line represents the method of the present invention, and the light line represents the comparison method. It can be seen that during pedestrian movement, the method based on human body detection box will introduce greater noise as the human posture changes, while our method has better anti-interference performance and can obtain more accurate pedestrian positioning information.

[0070] A pedestrian spatial information sensing device based on a stereo camera is used to implement the pedestrian spatial information sensing method based on a stereo camera, including a real-time image data acquisition module, a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module.

[0071] The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera;

[0072] The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain two-dimensional key point information of the pedestrian. Based on the dynamic characteristics of the pedestrian, it determines the main body area of ​​the upper body of the pedestrian. Combined with the two-dimensional key point information of the main body area of ​​the upper body of the pedestrian, it generates a pedestrian bounding box and uses the pedestrian bounding box as the pedestrian detection box.

[0073] The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes.

[0074] The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information combined with point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera.

[0075] The implementation methods for this part are similar to those for the methods described above, and will not be repeated here.

[0076] like Figure 6 As shown, a pedestrian spatial information perception method based on a ZED camera is described. According to the aforementioned method, a ZED binocular vision camera mounted on a tour guide robot collects real-time image data and transmits it to a cloud server for pedestrian detection box acquisition, multi-target tracking, and real-time spatial information generation. The field of view of the ZED binocular vision camera is configured to match the upper body backbone area of ​​the pedestrian. Based on the pedestrian's dynamic characteristics and the field of view of the ZED binocular vision camera on the tour guide robot, the upper body backbone area of ​​the pedestrian is determined. The cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the tour guide robot. The tour guide robot then uses this real-time spatial information to control its own movement, completing autonomous following and obstacle avoidance tasks.

[0077] Specifically, it includes the following steps:

[0078] Step S101: Acquire real-time image data from the ZED binocular vision camera on the tour guide robot, including RGB image data and point cloud data, and transmit it to the cloud server;

[0079] Step S102: The cloud server detects key points of the human body through RGB images to obtain the two-dimensional key point information of the pedestrian. Based on the dynamic characteristics of the pedestrian, it determines the main body area of ​​the upper body of the pedestrian. Combining the two-dimensional key point information of the main body area of ​​the upper body of the pedestrian, it generates a pedestrian bounding box and uses the pedestrian bounding box as the pedestrian detection box. The field of view of the ZED binocular vision camera is set in conjunction with the main body area of ​​the upper body of the pedestrian. Based on the dynamic characteristics of the pedestrian and the field of view of the ZED binocular vision camera on the guide robot, the main body area of ​​the upper body of the pedestrian is determined.

[0080] Step S103: The cloud server performs multi-target tracking of pedestrians in multiple consecutive frames of images based on the two-dimensional key point similarity features and pedestrian detection boxes;

[0081] Step S104: For the continuously tracked pedestrian, the cloud server obtains the pedestrian's three-dimensional key point information based on the two-dimensional key point information and point cloud data, calculates the spatial position coordinates of the pedestrian relative to the ZED binocular vision camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval to generate the pedestrian's real-time spatial information relative to the ZED binocular vision camera.

[0082] Step S105: The cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the tour guide robot. The tour guide robot performs movement control of its own body according to the real-time spatial information to complete the tour guide task.

[0083] In this embodiment of the invention, the cloud stores the pedestrian ID, pedestrian spatial location, and pedestrian movement speed at the current moment in the cloud server's database, using a message queue format. The basic format is data = {'key1':'value1', 'key2': 'value2', 'key3': 'value3'}, where key1, key2, and key3 are 'p_ID', 'p_Pos3D', and 'p_Speed', representing the pedestrian ID, pedestrian 3D spatial coordinates, and pedestrian movement speed, respectively. The corresponding value is the calculated identity ID, 3D spatial coordinates, and movement speed of the specific pedestrian i. Based on the request instructions from the robot, the cloud sends the data to the robot in real time via the RocketMQ message middleware. The robot performs body movement control according to pre-set instructions, adjusts its movement speed based on the pedestrian's real-time location information, and stops moving when the distance between the pedestrian and the robot is less than the safe distance to avoid collision. The robot also adjusts its own movement speed based on the pedestrian's real-time movement speed, thereby achieving intelligent navigation tasks such as autonomous following and obstacle avoidance.

[0084] A pedestrian spatial information perception device based on a ZED camera is used to implement a pedestrian spatial information perception method based on a ZED camera. It includes a cloud server and a ZED binocular vision camera installed on a guide robot. The cloud server includes a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module.

[0085] The ZED binocular vision camera acquires RGB image data and point cloud data in real time and transmits them to the cloud server.

[0086] The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain the two-dimensional key point information of the pedestrian. Based on the pedestrian's dynamic characteristics, it determines the main body region of the upper body of the pedestrian. Combining the two-dimensional key point information of the main body region of the upper body of the pedestrian, it generates a pedestrian bounding box, which is used as the pedestrian detection box. The field of view of the ZED binocular vision camera is set in conjunction with the main body region of the upper body of the pedestrian. The main body region of the upper body of the pedestrian is determined based on the pedestrian's dynamic characteristics and the field of view of the ZED binocular vision camera on the guide robot.

[0087] The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes.

[0088] The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information and point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera.

[0089] The guided tour robot obtains real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, and performs movement control of its own body to complete the guided tour task.

[0090] The implementation methods for this part are similar to those for the methods described above, and will not be repeated here.

[0091] Corresponding to the aforementioned embodiments of the pedestrian spatial information perception method based on stereo camera vision, the present invention also provides embodiments of a pedestrian spatial information perception device based on stereo camera vision.

[0092] See Figure 7 The pedestrian spatial information perception device based on stereo camera vision provided in this embodiment of the invention includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the pedestrian spatial information perception method based on stereo camera vision in the above embodiment.

[0093] The embodiments of the pedestrian spatial information perception device based on stereo camera vision of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the pedestrian spatial information perception device based on stereo camera vision according to the present invention. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0094] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0095] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0096] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the pedestrian spatial information perception method based on stereo camera vision described in the above embodiments.

[0097] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for perceiving pedestrian spatial information based on a stereo camera, characterized in that... Includes the following steps: Step S1: Acquire real-time image data from the stereo camera, including RGB image data and point cloud data; Step S2: Detect human key points through RGB image to obtain the two-dimensional key point information of pedestrians. Determine the main body area of ​​the upper body of pedestrians based on the dynamic characteristics of pedestrians. Combine the two-dimensional key point information of the main body area of ​​the upper body of pedestrians to generate pedestrian bounding boxes. Use the pedestrian bounding boxes as pedestrian detection boxes. Step S3: Based on the two-dimensional keypoint similarity features and pedestrian detection boxes, perform multi-target tracking of pedestrians in multiple consecutive frames of images; Step S4: For continuously tracked pedestrians, obtain the pedestrian's 3D keypoint information based on their 2D keypoint information and point cloud data, calculate the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculate the pedestrian's moving speed based on the frame interval to generate real-time spatial information of the pedestrian relative to the stereo camera; specifically including the following steps: Step S4.1: Based on the main body area of ​​the pedestrian's upper body, filter the key points using the confidence level of the two-dimensional key points. Based on the filtered two-dimensional key points, obtain point cloud data to get the three-dimensional coordinate set of the pedestrian's key points. Step S4.2: For each pedestrian target, calculate the mean of the three-dimensional coordinates based on the set of three-dimensional coordinates of its key points as the spatial position coordinates of the pedestrian target; calculate the actual distance of the pedestrian relative to the stereo camera based on the Euclidean distance; calculate the distance the pedestrian moves relative to the stereo camera in the time interval between the current time and the previous time based on the pedestrian's spatial position coordinates at the current time and the previous time; and combine the time taken to obtain the pedestrian's moving speed at the current time.

2. The pedestrian spatial information perception method based on a stereo camera according to claim 1, characterized in that: In step S2, RGB image data is acquired, and a human keypoint detection network is used for forward inference to output a keypoint heatmap and a partial association domain. Two-dimensional keypoints are extracted and grouped based on the keypoint heatmap and the partial association domain. Two-dimensional keypoints belonging to the same pedestrian are matched to the current pedestrian to obtain the coordinates of the two-dimensional keypoints of each pedestrian in the current image.

3. The pedestrian spatial information perception method based on a stereo camera according to claim 1, characterized in that: Step S3 includes the following steps: Step S3.1: Obtain pedestrian motion features based on the pedestrian detection bounding box; Based on the similarity features of two-dimensional key points, the appearance features of pedestrians are obtained; Step S3.2: Based on the pedestrian's motion characteristics and appearance characteristics, obtain the measured state information of the pedestrian at the current time t; Step S3.3: Associate the historical trajectory with the actual pedestrian status information at time t to obtain the ID of each pedestrian at time t; Step S3.4: Update the historical trajectory using the ID of each pedestrian at time t.

4. The pedestrian spatial information perception method based on a stereo camera according to claim 3, characterized in that: The two-dimensional keypoint similarity feature in step S3.1 uses the target keypoint similarity evaluation index OKS for similarity calculation, and determines whether the two-dimensional keypoints are related by a preset threshold.

5. The pedestrian spatial information perception method based on a stereo camera according to claim 3, characterized in that: The data association in step S3.3 involves associating each frame with both motion features and appearance features, applying linear weights to obtain the final association matrix, and then using the Hungarian matching algorithm to obtain the pedestrian matching results between frames based on this association matrix.

6. The pedestrian spatial information perception method based on a stereo camera according to claim 1, characterized in that: The formula for pedestrian movement speed in step S4.2 is as follows: Where X, Y, and Z are the three-dimensional spatial coordinates of the pedestrian, i represents the currently tracked pedestrian ID, t represents the time of the current frame, and X... (i，t) Y (i，t) Z (i，t) This represents the spatial coordinates of the pedestrian relative to the stereo camera in the current frame. They represent in The spatial coordinates of the relative position of the stereo camera at that moment. m represents the frame interval, and f is the frame rate of the stereo camera.

7. A pedestrian spatial information sensing device based on a stereo camera, used to implement the pedestrian spatial information sensing method based on a stereo camera as described in any one of claims 1-6, comprising a real-time image data acquisition module, a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module, characterized in that: The real-time image data acquisition module acquires RGB image data and point cloud data through a stereo camera; The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain two-dimensional key point information of the pedestrian. Based on the dynamic characteristics of the pedestrian, it determines the main body area of ​​the upper body of the pedestrian. Combined with the two-dimensional key point information of the main body area of ​​the upper body of the pedestrian, it generates a pedestrian bounding box and uses the pedestrian bounding box as the pedestrian detection box. The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes. The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information combined with point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera.

8. A method for perceiving pedestrian spatial information based on a ZED camera, characterized in that... According to any one of claims 1-6, the pedestrian spatial information perception method based on a stereo camera employs a ZED binocular vision camera mounted on a tour guide robot to acquire real-time image data and transmit it to a cloud server for pedestrian detection box acquisition, multi-target tracking, and generation of real-time spatial information. The field of view of the ZED binocular vision camera is configured to match the upper body trunk area of ​​the pedestrian. Based on the pedestrian's dynamic characteristics and the field of view of the ZED binocular vision camera on the tour guide robot, the upper body trunk area of ​​the pedestrian is determined. The cloud server transmits the real-time spatial information of the pedestrian relative to the ZED binocular vision camera to the tour guide robot. The tour guide robot performs movement control based on the real-time spatial information to complete the tour guide task.

9. A pedestrian spatial information perception device based on a ZED camera, used to implement the pedestrian spatial information perception method based on a ZED camera as described in claim 8, comprising a cloud server and a ZED binocular vision camera mounted on a tour guide robot, characterized in that: The cloud server includes a pedestrian detection box acquisition module, a multi-target tracking module, and a real-time spatial information generation module; The ZED binocular vision camera acquires RGB image data and point cloud data in real time and transmits them to the cloud server. The pedestrian detection box acquisition module detects key points of the human body through RGB images to obtain two-dimensional key point information of the pedestrian. Based on the dynamic characteristics of the pedestrian, it determines the main body area of ​​the upper body of the pedestrian. Combined with the two-dimensional key point information of the main body area of ​​the upper body of the pedestrian, it generates a pedestrian bounding box and uses the pedestrian bounding box as the pedestrian detection box. The field of view of the ZED binocular vision camera is set in conjunction with the main area of ​​the upper body of the pedestrian. The main area of ​​the upper body of the pedestrian is determined based on the dynamic characteristics of the pedestrian and the field of view of the ZED binocular vision camera on the guide robot. The multi-target tracking module performs multi-target tracking of pedestrians in multiple consecutive frames of images based on two-dimensional keypoint similarity features and pedestrian detection boxes. The real-time spatial information generation module, for continuously tracked pedestrians, obtains the pedestrian's three-dimensional key point information based on the pedestrian's two-dimensional key point information and point cloud data, calculates the pedestrian's spatial position coordinates relative to the stereo camera coordinate system in the current frame, and calculates the pedestrian's moving speed based on the frame interval, thereby generating the pedestrian's real-time spatial information relative to the stereo camera. The guided tour robot obtains real-time spatial information of pedestrians relative to the ZED binocular vision camera from the cloud server, and performs movement control of its own body to complete the guided tour task.

Images