A method and system for a depth camera-based guide robot to perceive three-dimensional spatial information of a pedestrian

Abstract

A method for a guide robot based on a depth camera to perceive three-dimensional spatial information of a pedestrian, comprising: 1) a robot end acquires an RGB picture and a Depth picture of the depth camera, and transmits the preprocessed pictures to a cloud server; 2) according to a multi-target tracking method and a two-dimensional human key point detection network deployed in the cloud end, two-dimensional key point coordinates of each tracked pedestrian are acquired, three-dimensional key point coordinates of the pedestrian are acquired in combination with a mapping relationship between the Depth picture and the RGB picture; 3) according to a coordinate conversion matrix of joint calibration of the depth camera and a laser radar, three-dimensional key point coordinates of the pedestrian in a robot coordinate system are acquired, and three-dimensional spatial information of the pedestrian is output; 4) the robot performs ontology control according to the acquired three-dimensional spatial information of the pedestrian, and completes intelligent guide tasks such as autonomous awakening; and the natural interaction experience of the guide robot is improved by using a visual perception method. The application can be used for intelligent perception of three-dimensional spatial information of a pedestrian by a guide robot in a public scene such as an exhibition hall.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and more particularly to a method for a guide robot based on a depth camera to perceive three-dimensional spatial information of pedestrians. Background Technology

[0002] Guided tour robots can replace human guides in public places such as exhibition halls and museums, providing intelligent services and saving manpower. Pedestrians are the key participants in this scenario, and guided tour robots need to perceive and analyze them in real time to achieve more human-like services and improve user experience during human-computer interaction.

[0003] Vision is a crucial way for robots to acquire external information. Methods for detecting key human points in images containing pedestrians can only obtain 2D image information. However, interactions between pedestrians and robots occur in the real three-dimensional world, so it's necessary to acquire pedestrian perception information in three-dimensional space. Using a depth camera to obtain the 3D coordinates of key human points in a non-contact manner can provide reliable 3D spatial information of pedestrians for human-like interactive functions such as autonomous wake-up of navigation robots. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for a depth-camera-based navigation robot to perceive three-dimensional spatial information of pedestrians. The technical solution adopted by this invention is as follows:

[0005] A method for a depth camera-based navigation robot to perceive pedestrian 3D spatial information includes the following steps:

[0006] Step S1: The robot acquires RGB and depth images from the depth camera, performs temporal and spatial alignment, and transmits the image data, which has undergone preprocessing operations such as depth map downsampling and image encoding, to the cloud server.

[0007] Step S2 uses a cloud-deployed multi-target tracking method and a two-dimensional human key point detection method to obtain the coordinates of the two-dimensional key points of each tracked pedestrian, and combines the mapping relationship between depth images and RGB images to obtain the coordinates of the three-dimensional key points of the pedestrian.

[0008] Step S3 obtains the coordinate system transformation matrix based on the joint calibration of the depth camera and LiDAR, thereby obtaining the coordinates of the pedestrian's 3D key points in the robot coordinate system and outputting the pedestrian's 3D spatial information.

[0009] In step S4, the robot accesses the cloud server to obtain the pedestrian's three-dimensional spatial information for self-control, and completes intelligent guidance tasks such as autonomous wake-up.

[0010] Furthermore, in step S1, the temporal alignment of the RGB image and the depth image of the depth camera refers to aligning a frame of RGB image and a frame of depth image obtained at the same time using timestamps; spatial alignment is aligning the depth image to the RGB color image.

[0011] Furthermore, in step S1, the scaling factors for the depth map downsampling are kh and kw, the original depth map size is [W, H], and the downsampled size is [W / kw, H / kh]. The robot encodes the original RGB color image and the downsampled depth image it collects, and then sends them to the cloud server through the Redis message middleware.

[0012] Furthermore, step S2 includes the following steps:

[0013] S2.1 adopts a detection-based two-stage multi-target tracking method, that is, using YOLOV5+DeepSort to obtain the ID and detection box of each pedestrian target in each image in consecutive frames;

[0014] S2.2 adopts a top-down two-dimensional human keypoint detection method for pedestrian human keypoint detection. The two-dimensional keypoint detection network uses Lite-HRNet-18. The input is each pedestrian target image cropped according to the detection box in the previous step, and is scaled to a uniform size using normalization processing. The output is the two-dimensional coordinates of human keypoints [17, 3], where 17 represents the number of human keypoints, 3 represents the two-dimensional pixel coordinates x, y of the keypoint in the original RGB image and the confidence level c, and the value of c ranges from 0 to 1.

[0015] S2.3 Combine the mapping relationship between the depth image and the RGB image to obtain the coordinates of the pedestrian's 3D key points. According to the scaling factor, find the corresponding pixel coordinates (u, v) and the corresponding depth value d[u, v] of the 2D key point (x, y) in the RGB image in the depth image, where u = round(u / kw) and v = round(u / kh), round means rounding. If d does not exist, the depth point is estimated by taking the median value of the set of depth points in the rectangular area enclosed by the horizontal direction [u-pts, u+pts] and the vertical direction [v-pts, v+pts] of the point, where pts is the number of effective pixels, pts = 3;

[0016] S2.4 Obtain the coordinates of the pedestrian's 3D key points in the camera coordinate system. Calculate the 3D key points P[Xc,Yc,Zc] of the human body in the camera coordinate system based on the depth values ​​corresponding to the 2D key points (x,y) and the camera intrinsic parameters.

[0017] Zc = d[u, v]

[0018] Yc=(v-cy)×d[u,v] / fy

[0019] Xc=(u-cx)×d[u,v] / fx

[0020] Where u and v are the row and column values ​​of the two-dimensional key point (x, y) in the depth map, d[u, v] represents the depth value at the pixel coordinates, (cx, cy) is the horizontal and vertical offset of the image origin relative to the optical center imaging point, fx is the focal length in the horizontal direction, and fy is the focal length in the vertical direction.

[0021] Further, step S3 includes the following steps:

[0022] S3.1 A checkerboard calibration board is used for joint calibration of the depth camera and LiDAR. Based on the laser points of the single-line LiDAR falling on the calibration board and the acquisition of calibration images, the coordinates of these points in the laser coordinate system and the coordinates of the plane equations in the camera coordinate system are used to construct constraints on the plane and solve for the transformation matrix between the two coordinate systems, including the rotation matrix R. c2l Translation matrix T c2l ;

[0023] S3.2 Based on the transformation matrix between the camera coordinate system and the LiDAR coordinate system, obtain the 3D key point coordinates of the pedestrian in the robot coordinate system: P(Xw,Yw,Zw)=R c2l ×P(Xc, Yc, Zc)+T c2l The robot coordinate system is consistent with the lidar coordinate system;

[0024] S3.3 Obtain pedestrian 3D spatial information, including the pedestrian-to-robot spatial distance Distacne (Dis) and pedestrian height Height (Hp). Calculate the pedestrian-to-robot spatial distance based on the pedestrian's 3D keypoint information: Where X, Y, and Z represent the three-dimensional spatial coordinates of the pedestrian in the robot's coordinate system, which can be obtained through the pedestrian's three-dimensional key points: N represents the number of key points on the human body: N = 17; the pedestrian height Hp is based on the pedestrian's nose key point.

[0025] Further, step S4 includes the following steps:

[0026] The S4.1 guide robot accesses the cloud server through the message middleware to obtain real-time 3D spatial information of pedestrians. Based on the distance information between the pedestrian and the robot and the preset threshold, it determines whether to activate the guide robot. When Dis>3m, the guide robot's state machine is in a sleep state with low energy consumption; when Dis≤3m, the guide robot changes from a sleep state to an active state, thereby realizing the task of actively waking up the robot based on visual information.

[0027] S4.2 Based on the previous step, determine the current state of the guide robot. When the guide robot is in sleep mode, its head is in the initial state, with both horizontal and vertical rotation angles at zero degrees. When the guide robot is in active mode, calculate the angle between the pedestrian and the robot's head based on the 3D keypoint coordinates of the pedestrian's nose, obtaining the vertical rotation angle θ1 and horizontal rotation angle θ2 of the pedestrian relative to the robot. The robot then controls its head through the motion control unit to achieve rotation in both degrees of freedom to the specified angle. The pedestrian's height, Hp, is obtained from the 3D coordinates (Xn, Yn, Zn) of the pedestrian's nose keypoint. p =Zn + 0.20, in meters, where 0.20 is the distance from the robot chassis laser radar to the ground, H head The distance H from the robot's head rotation axis to the ground. head =1.25m, the distance from the pedestrian to the robot is Dis, then the vertical rotation angle θ1 of the robot is:

[0028] θ1=arcsin(Hp-H head ) / Dis

[0029] The horizontal rotation angle θ2 of the robot is:

[0030] θ2 = arcsin(Yn / Xn).

[0031] The present invention also includes a system for a depth camera-based navigation robot to perceive pedestrian three-dimensional spatial information, comprising:

[0032] The image preprocessing module is used to acquire RGB and depth images from the depth camera on the robot side, and perform temporal and spatial alignment. The data that has undergone preprocessing operations such as depth map downsampling and image encoding is then transmitted to the cloud server.

[0033] The pedestrian 3D key point coordinate acquisition module is used to obtain the human 2D key point coordinates of each tracked pedestrian target by using a multi-target tracking method and a 2D human key point detection method deployed in the cloud, and to obtain the pedestrian's 3D key point coordinates by combining the mapping relationship between the depth map and the RGB image.

[0034] The pedestrian 3D spatial information output module is used to obtain the coordinate system transformation matrix based on the joint calibration of the depth camera and LiDAR, thereby obtaining the coordinates of the pedestrian's 3D key points in the robot coordinate system and outputting the pedestrian's 3D spatial information.

[0035] The intelligent navigation module is used by the robot to obtain pedestrian 3D spatial information by accessing the cloud server and to control the robot itself to complete intelligent navigation tasks such as autonomous wake-up.

[0036] The present invention also includes a device for a depth camera-based tour guide robot to perceive pedestrian three-dimensional spatial information, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a method of the present invention for a depth camera-based tour guide robot to perceive pedestrian three-dimensional spatial information.

[0037] The present invention also includes a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method for a depth camera-based guide robot to perceive pedestrian three-dimensional spatial information.

[0038] The advantages and beneficial effects of this invention are as follows:

[0039] This invention employs a non-contact depth camera for intelligent perception of pedestrians within a scene; it utilizes a cloud server for data transmission and model deployment, leveraging its powerful computing and storage resources to effectively address the issue of insufficient computing power on the robot itself. By employing a two-stage method based on acquiring the coordinates of three-dimensional key points of the human body from two-dimensional key points, the complexity of the model is significantly reduced. Furthermore, by fully utilizing key point and depth information, the missing coordinates of three-dimensional key points are compensated for, thereby improving the accuracy of three-dimensional spatial perception of the human body. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention.

[0041] Figure 2 This is a schematic diagram of the method for acquiring three-dimensional key points of pedestrians based on a depth camera according to the present invention.

[0042] Figure 3 This is a schematic diagram of the coordinates of key human body points according to the present invention.

[0043] Figure 4 This is a schematic diagram of the autonomous wake-up of the tour guide robot of the present invention.

[0044] Figure 5 This is a schematic diagram of the head rotation of the tour guide robot of the present invention.

[0045] Figure 6 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0047] like Figure 1 As shown, a method for a depth camera-based navigation robot to perceive pedestrian 3D spatial information includes the following steps:

[0048] Step S1: The robot acquires RGB and depth images from the depth camera, performs temporal and spatial alignment, and transmits the image data, which has undergone preprocessing operations such as depth map downsampling and image encoding, to the cloud server.

[0049] In this embodiment of the invention, an Intel RealSense D455i depth camera is mounted on the upper chest of a wheeled guide robot, approximately 1.25m above the ground. RGB and depth images from the depth camera are acquired locally on the robot using the Jetson Xavier NX platform. The original image size is [W, H], where W = 1280 and H = 720. Since the origin of the RGB image data is the RGB camera of the RealSense depth camera, and the origin of the depth image data is the infrared camera of the RealSense depth camera, spatial alignment is required. We use the RealSense depth camera's built-in SDK to align the depth images to the color images, thus achieving a one-to-one correspondence between the RGB image coordinates and the depth image coordinates. Timestamps are used to align each frame of the RGB image with the depth image. Furthermore, due to the limited computing power of the robot itself, the acquired camera data needs to be transmitted to a cloud server. However, the depth data volume is large and transmission is affected by network factors, so downsampling is performed locally on the robot to improve transmission efficiency. The original depth map size is [W, H], and the downsampling scaling factors are kh and kw. Then the downsampling depth map size is [W / kw, H / kh]. The robot's local end encodes the preprocessed RGB image and the depth image and then sends them to the cloud server through the Redis message middleware.

[0050] Step S2 uses a cloud-deployed multi-target tracking method and a two-dimensional human key point detection method to obtain the coordinates of the two-dimensional key points of each tracked pedestrian, and combines the mapping relationship between depth images and RGB images to obtain the coordinates of the three-dimensional key points of the pedestrian.

[0051] In this embodiment of the invention, the cloud server receives real-time data from the robot, first performs decoding, and then uses the YOLOV5+DeepSort multi-target tracking method to obtain the detection box and corresponding ID of each pedestrian target in each frame of RGB image; each pedestrian target image is cropped according to the detection box and scaled to a uniform size of 256×192 using normalization processing, and input into the two-dimensional keypoint detection network deployed in the cloud. The two-dimensional keypoint detection network adopts Lite-HRNet-18, and the output is the two-dimensional coordinates of human keypoints [17, 3], where 17 represents the number of human keypoints, 3 represents the two-dimensional pixel coordinates x, y and confidence c of the keypoint in the original RGB image, and the value of c ranges from 0 to 1. The subsequent three-dimensional keypoint filtering can be performed based on the confidence c.

[0052] The 2D keypoint coordinates of the pedestrian were obtained from the RGB image. The 3D keypoint coordinates of the pedestrian were then obtained by combining the coordinate correspondence between the depth image and the RGB image. Because the depth map was downsampled to improve transmission efficiency, further processing is required during the retrieval process: Assuming the 2D keypoint coordinates in the RGB image are (x, y), the corresponding depth map coordinates are found to be (u, v) based on the scaling factor. The corresponding depth value is d[u, v], where u = round(x / kw), v = round(y / kh), and round represents rounding. If d does not exist, the depth is estimated by taking the median value of the depth points within the rectangular area enclosed by the horizontal direction [u-pts, u+pts] and the vertical direction [v-pts, v+pts], where pts is the number of effective pixels, and pts = 3.

[0053] Based on the depth values ​​corresponding to the 2D keypoints and the camera intrinsic parameters, the coordinate values ​​of the 3D human body keypoints P[Xc,Yc,Zc] in the camera coordinate system are calculated using the following formula:

[0054] Zc = d[u, v]

[0055] Yc=(v-cy)×d[u,v] / fy

[0056] Xc=(u-cx)×d[u,v] / fx

[0057] Where u and v are the row and column values ​​of the two-dimensional key point (x, y) in the depth map, d[u, v] represents the depth value at the pixel coordinates, (cx, cy) is the horizontal and vertical offset of the image origin relative to the optical center imaging point, fx is the focal length in the horizontal direction, and fy is the focal length in the vertical direction.

[0058] Step S3 obtains the coordinate system transformation matrix based on the joint calibration of the depth camera and LiDAR, thereby obtaining the coordinates of the pedestrian's 3D key points in the robot coordinate system and outputting the pedestrian's 3D spatial information.

[0059] In this embodiment of the invention, step S2 obtains the coordinates of the pedestrian's 3D key points in the camera coordinate system, which need to be further transformed to the guide robot coordinate system. The guide robot coordinate system is defined to be consistent with the LiDAR coordinate system. A checkerboard calibration board is used for joint calibration of the depth camera and LiDAR. The guide robot in this embodiment uses a wheeled chassis. The single-line LiDAR is installed 20cm above the ground. The calibration board is a checkerboard calibration board, 12×9 in size, with each grid measuring 4.5×4.5cm. The calibration board is placed 0.3m-1.5m in front of the RealSense camera and LiDAR within a field of view. The calibration board is rotated in various postures, and calibration images are acquired. Based on the laser point of the single-line LiDAR falling on the calibration board, using its coordinates in the laser coordinate system and the coordinates of the plane equation in the camera coordinate system, constraints on the point on the plane are constructed, and the transformation matrix between the two coordinate systems is solved, including the rotation matrix R and the translation matrix T. c2l In this embodiment of the invention, the values ​​of the rotation matrix and translation matrix are as follows:

[0060] R c2l =[[0.037,0.323,0.945]

[0061] [-0.995, -0.072, 0.064]

[0062] [0.089, -0.943, 0.319]]

[0063] T c2l =[0.137,0.050,0.840]

[0064] Based on the transformation matrix between the camera coordinate system and the LiDAR coordinate system, obtain the 3D keypoint coordinates P(Xw, Yw, Zw) of the pedestrian in the robot coordinate system: P(Xw, Yw, Zw) = R c2l ×P(Xc, Yc, Zc)+T c2l ,like Figure 2 As shown.

[0065] The pedestrian's 3D spatial information is obtained based on the pedestrian's 3D key points, including the spatial distance Dis from the pedestrian to the robot and the pedestrian's height Hp. The spatial distance Distance (abbreviated as Dis) from the pedestrian to the robot is calculated based on the pedestrian's 3D key point information. Where X, Y, and Z represent the three-dimensional spatial coordinates of the pedestrian in the robot's coordinate system, which can be obtained through the pedestrian's three-dimensional key points: c>0.6}, N represents the number of human keypoints: N=17; c represents the keypoint confidence level, the value of c ranges from 0-1, and the pedestrian height Hp is based on the pedestrian's nose keypoint, such as Figure 3 As shown.

[0066] In step S4, the robot accesses the cloud server to obtain pedestrian 3D spatial information and performs body control to complete intelligent guidance tasks such as autonomous wake-up.

[0067] The guide robot accesses the cloud server via a message middleware to obtain real-time 3D spatial information of pedestrians. It determines whether to activate the robot based on the distance Dis between the pedestrian and the robot. When Dis > 3m, the robot's state machine is in a sleep state, consuming less energy; when Dis ≤ 3m, the robot transitions from sleep to activation, thus achieving a vision-based active robot wake-up function. Figure 4 As shown;

[0068] like Figure 5 The diagram shows the vertical rotation angle θ1 and horizontal rotation angle θ2 of the pedestrian relative to the robot. The distance from the pedestrian to the robot, Dis, is obtained based on the pedestrian's 3D keypoint information in the robot's coordinate system (LiDAR coordinate system). The pedestrian's height, Hp, is obtained based on the 3D coordinates (Xn, Yn, Zn) of the pedestrian's nose keypoint. p =Zn + 0.20, in meters, where 0.20 is the distance from the robot chassis laser radar to the ground, H head The distance H from the robot's head rotation axis to the ground. head =1.25m, then the vertical rotation angle θ1 of the robot is:

[0069] θ1=arcsin(Hp-H head ) / Dis

[0070] The horizontal rotation angle θ2 of the robot is:

[0071] θ2=arcsin(Yn / Xn)

[0072] When the guide robot is in sleep mode, its head is in the initial state with zero horizontal and vertical rotation angles. When the guide robot is in active mode, it calculates the angle between the pedestrian and the robot's head based on the three-dimensional key point coordinates of the pedestrian's nose, and obtains the vertical rotation angle θ1 and horizontal rotation angle θ2 of the pedestrian relative to the robot. The robot then controls the head through the motion control unit to achieve rotation of the two degrees of freedom to the specified angle, thereby achieving a more human-like interactive experience.

[0073] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This paper presents a method for a depth camera-based navigation robot to perceive three-dimensional spatial information of pedestrians.

[0074] The present invention also provides Figure 6 The one shown corresponds to Figure 1 A schematic diagram of a system for a depth-camera-based navigation robot to perceive three-dimensional spatial information of pedestrians. (See diagram for example.) Figure 6 At the hardware level, this system for a depth-camera-based guide robot to perceive pedestrian 3D spatial information includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for other operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The data acquisition method described above. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0075] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are the most commonly used. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0076] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0077] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0078] For ease of description, the above apparatus is described in terms of its functions, divided into various units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0085] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0089] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0090] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for a depth-camera-based navigation robot to perceive pedestrian three-dimensional spatial information, characterized in that, Includes the following steps: Step S1: The robot acquires RGB and depth images from the depth camera, performs temporal and spatial alignment, and transmits the data after preprocessing operations such as depth map downsampling and image encoding to the cloud server. Step S2 utilizes a cloud-deployed multi-target tracking method and a 2D human keypoint detection method to obtain the coordinates of the 2D keypoints of each tracked pedestrian target. It then combines this with the mapping relationship between the depth map and the RGB image to obtain the coordinates of the pedestrian's 3D keypoints. Specifically, this includes: S2.1 The multi-target tracking method adopts a detection-based two-stage method, namely, using YOLOV5+DeepSort to obtain the ID and detection box of each pedestrian target in each image in consecutive frames; The two-dimensional key point detection described in S2.2 adopts a top-down method. The two-dimensional key point detection network uses Lite-HRNet-18. The input is each pedestrian target image cropped according to the detection box in the previous step, and it is scaled to a uniform size using normalization processing. The output is the two-dimensional coordinates of human key points. S2.3 The three-dimensional key point coordinates of the pedestrian are obtained by combining the mapping relationship between the depth map and the RGB image. The pixel coordinates (u, v) and the corresponding depth value d[u, v] of the two-dimensional key point (x, y) in the RGB image are found in the depth map according to the scaling factor, where u = round(u / kw) and v = round(u / kh). round means rounding. If d does not exist, the depth point is estimated by taking the median value of the set of depth points in the rectangular area enclosed by the horizontal direction [u-pts, u+pts] and the vertical direction [v-pts, v+pts] of the point, where pts is the number of effective pixels. S2.4 Obtain the coordinates of the pedestrian's 3D keypoints in the camera coordinate system. Calculate the 3D keypoints P[Xc, Yc, Zc] of the human body in the camera coordinate system based on the depth values ​​corresponding to the 2D keypoints (x, y) and the camera intrinsic parameters. Zc = d[u, v] Yc = (v - cy) × d[u, v] / fy Xc = (u - cx) × d[u, v] / fx Where u and v are the row and column values ​​of the two-dimensional key point (x, y) in the depth map, d[u, v] represents the depth value at the pixel coordinates, (cx, cy) is the horizontal and vertical offset of the image origin relative to the optical center imaging point, fx is the focal length in the horizontal direction, and fy is the focal length in the vertical direction. Step S3: Obtain the coordinate system transformation matrix based on the joint calibration of the depth camera and LiDAR, thereby obtaining the coordinates of the pedestrian's 3D key points in the robot coordinate system and outputting the pedestrian's 3D spatial information. Step S4: The robot obtains pedestrian 3D spatial information by accessing the cloud server and performs body control to complete intelligent guidance tasks such as autonomous wake-up, specifically including: S4.1 The guide robot accesses the cloud server through the message middleware to obtain real-time three-dimensional spatial information of pedestrians. Based on the distance information between the pedestrian and the robot and the preset threshold, it determines whether to activate the guide robot. When Dis>3m, the guide robot's state machine is in a sleep state with low energy consumption; when Dis≤3m, the guide robot changes from a sleep state to an active state, thereby realizing the task of actively waking up the robot based on visual information. S4.2 Based on the previous step, determine the current state of the guide robot. When the guide robot is in a sleep state, the robot head is in the initial state, with both horizontal and vertical rotation angles at zero degrees. When the guide robot is in an active state, calculate the angle between the pedestrian and the robot head based on the 3D keypoint coordinates of the pedestrian's nose, obtaining the vertical rotation angle θ1 and the horizontal rotation angle θ2 of the pedestrian relative to the robot. The robot controls the robot head through the motion control unit to achieve rotation in two degrees of freedom to reach the specified angle. Obtain the pedestrian's height Hp based on the 3D coordinates (Xn, Yn, Zn) of the pedestrian's nose keypoint. p =Zn+0.2, in meters, where 0.2 is the distance from the robot chassis laser radar to the ground, and H head Let Dis be the distance from the robot's head rotation axis to the ground, and Dis be the distance from the pedestrian to the robot. Then, the robot's vertical rotation angle θ1 is: θ1 = arcsin ( Hp - H head ) / Dis The horizontal rotation angle θ2 of the robot is: θ2 = arcsin (Yn / Xn).

2. The method for a depth camera-based navigation robot to perceive pedestrian three-dimensional spatial information as described in claim 1, characterized in that: Step S1 specifically includes: S1.1 In step S1, the time alignment of the RGB image and the depth image of the depth camera refers to aligning an RGB image and a depth image obtained at the same time using timestamps; spatial alignment is aligning the depth image to the RGB color image. S1.2 In step S1, the scaling factor for the depth map downsampling is kh, kw, the original depth map size is [W, H], and the downsampled size is [W / kw, H / kh]. The robot encodes the original RGB color image and the downsampled depth image, and then sends them to the cloud server through the Redis message middleware.

3. The method for a depth camera-based navigation robot to perceive pedestrian three-dimensional spatial information as described in claim 1, characterized in that: The two-dimensional coordinates of the human body key points mentioned in step S2.2 are [17, 3], where 17 represents the number of human body key points, and 3 represents the two-dimensional pixel coordinates x, y and confidence level c of the key points in the original RGB image, with c ranging from 0 to 1; the number of effective pixels mentioned in step S2.3 is pts=3.

4. The method for a guide robot to perceive pedestrian three-dimensional spatial information based on a depth camera as described in claim 1, characterized in that: Step S3 specifically includes: S3.1 The joint calibration of the depth camera and LiDAR uses a checkerboard calibration board. Based on the laser point of the single-line LiDAR falling on the calibration board and acquiring calibration images, the coordinates of the point in the laser coordinate system and the coordinates of the plane equation in the camera coordinate system are used to construct the constraints of the point on the plane and solve for the transformation matrix between the two coordinate systems, including the rotation matrix R. c2l Translation matrix T c2l ; S3.2 Based on the transformation matrix between the camera coordinate system and the LiDAR coordinate system, obtain the 3D key point coordinates of the pedestrian in the robot coordinate system: P(Xw, Yw, Zw) = R c2l ×P(Xc, Yc, Zc)+T c2l The robot coordinate system is consistent with the lidar coordinate system; S3.3 Obtain pedestrian 3D spatial information, including the spatial distance from the pedestrian to the robot (Distacne, abbreviated as Dis) and the pedestrian height (Hp, abbreviated as Hp). Calculate the spatial distance from the pedestrian to the robot based on the pedestrian's 3D keypoint information: Where X, Y, and Z represent the pedestrian's three-dimensional spatial coordinates in the robot's coordinate system, which can be obtained through the pedestrian's three-dimensional key points: N represents the number of key points on the human body: N=17; the pedestrian height Hp is based on the pedestrian's nose key point.

5. The method for a depth camera-based navigation robot to perceive pedestrian three-dimensional spatial information as described in claim 1, characterized in that: The distance H from the robot head rotation axis to the ground, as described in step S4.2 head =1.25m.

6. A system for a depth-camera-based navigation robot to perceive pedestrian three-dimensional spatial information, characterized in that, include: The image preprocessing module is used to acquire RGB and depth images from the depth camera on the robot side, and perform temporal and spatial alignment. The data that has undergone preprocessing operations such as depth map downsampling and image encoding is then transmitted to the cloud server. The pedestrian 3D keypoint coordinate acquisition module is used to obtain the 2D keypoint coordinates of each tracked pedestrian target using a cloud-deployed multi-target tracking method and a 2D human keypoint detection method. It then combines the mapping relationship between the depth map and the RGB image to obtain the pedestrian's 3D keypoint coordinates. Specifically, it includes: S2.1 The multi-target tracking method adopts a detection-based two-stage method, namely, using YOLOV5+DeepSort to obtain the ID and detection box of each pedestrian target in each image in consecutive frames; The two-dimensional key point detection described in S2.2 adopts a top-down method. The two-dimensional key point detection network uses Lite-HRNet-18. The input is each pedestrian target image cropped according to the detection box in the previous step, and it is scaled to a uniform size using normalization processing. The output is the two-dimensional coordinates of human key points. S2.3 The three-dimensional key point coordinates of the pedestrian are obtained by combining the mapping relationship between the depth map and the RGB image. The pixel coordinates (u, v) and the corresponding depth value d[u, v] of the two-dimensional key point (x, y) in the RGB image are found in the depth map according to the scaling factor, where u = round(u / kw) and v = round(u / kh). round means rounding. If d does not exist, the depth point is estimated by taking the median value of the set of depth points in the rectangular area enclosed by the horizontal direction [u-pts, u+pts] and the vertical direction [v-pts, v+pts] of the point, where pts is the number of effective pixels. S2.4 Obtain the coordinates of the pedestrian's 3D keypoints in the camera coordinate system. Calculate the 3D keypoints P[Xc, Yc, Zc] of the human body in the camera coordinate system based on the depth values ​​corresponding to the 2D keypoints (x, y) and the camera intrinsic parameters. Zc = d[u, v] Yc = (v - cy) × d[u, v] / fy Xc = (u - cx) × d[u, v] / fx Where u and v are the row and column values ​​of the two-dimensional key point (x, y) in the depth map, d[u, v] represents the depth value at the pixel coordinates, (cx, cy) is the horizontal and vertical offset of the image origin relative to the optical center imaging point, fx is the focal length in the horizontal direction, and fy is the focal length in the vertical direction. The pedestrian 3D spatial information output module is used to obtain the coordinate system transformation matrix based on the joint calibration of the depth camera and LiDAR, thereby obtaining the coordinates of the pedestrian's 3D key points in the robot coordinate system and outputting the pedestrian's 3D spatial information. The intelligent navigation module is used by the robot to obtain pedestrian 3D spatial information from a cloud server and control the robot itself to complete intelligent navigation tasks such as autonomous wake-up. Specifically, it includes: S4.1 The guide robot accesses the cloud server through the message middleware to obtain real-time three-dimensional spatial information of pedestrians. Based on the distance information between the pedestrian and the robot and the preset threshold, it determines whether to activate the guide robot. When Dis>3m, the guide robot's state machine is in a sleep state with low energy consumption; when Dis≤3m, the guide robot changes from a sleep state to an active state, thereby realizing the task of actively waking up the robot based on visual information. S4.2 Based on the previous step, determine the current state of the guide robot. When the guide robot is in a sleep state, the robot head is in the initial state, with both horizontal and vertical rotation angles at zero degrees. When the guide robot is in an active state, calculate the angle between the pedestrian and the robot head based on the 3D keypoint coordinates of the pedestrian's nose, obtaining the vertical rotation angle θ1 and the horizontal rotation angle θ2 of the pedestrian relative to the robot. The robot controls the robot head through the motion control unit to achieve rotation in two degrees of freedom to reach the specified angle. Obtain the pedestrian's height Hp based on the 3D coordinates (Xn, Yn, Zn) of the pedestrian's nose keypoint. p =Zn+0.2, in meters, where 0.2 is the distance from the robot chassis laser radar to the ground, and H head Let Dis be the distance from the robot's head rotation axis to the ground, and Dis be the distance from the pedestrian to the robot. Then, the robot's vertical rotation angle θ1 is: θ1 = arcsin ( Hp - H head ) / Dis The horizontal rotation angle θ2 of the robot is: θ2 = arcsin (Yn / Xn).

7. A device for a depth-camera-based navigation robot to perceive three-dimensional spatial information of pedestrians, characterized in that, The system includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a method for a depth camera-based guide robot to perceive pedestrian three-dimensional spatial information, as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements a method for a depth camera-based guide robot to perceive pedestrian three-dimensional spatial information, as described in any one of claims 1-5.

Images