Intelligent scene understanding method for humanoid robots based on laser point cloud

Abstract

The present invention discloses a method for intelligent scene understanding of a humanoid robot based on laser point cloud, comprising: autonomous navigation of the humanoid robot and collection of three-dimensional real scene data through a laser radar scanning system to obtain scene point cloud data; processing the obtained scene point cloud data using an adaptive radius sampling module; extracting point cloud semantic features based on RPV‑Net for the extracted valid point cloud data to obtain a 3D bounding box and semantic labels of the point cloud data; processing the point cloud data using a channel attention module, obtaining the overall features of the point cloud using an average pooling method, extracting individual features using a maximum pooling method, performing feature aggregation, and obtaining the global features of the point cloud; constructing a multimodal hybrid model of a scene understanding Bert pre-training model and an encoder‑decoder model to process the global features of the point cloud. The method uses laser radar technology to provide real-time, highly accurate three-dimensional point cloud data for a humanoid robot, and can achieve deep perception and intelligent understanding of the environment.

Description

Technical Field

[0001] The present invention relates to the field of humanoid robot perception and intelligent understanding. More specifically, the present invention relates to a humanoid robot scene intelligent understanding method based on laser point cloud. Background Art

[0002] With the rapid development of robotics technology, humanoid robots have shown broad application potential in a variety of application scenarios, such as disaster response, high-risk environment detection, daily services, etc. However, for humanoid robots to operate effectively in complex environments, highly accurate environmental perception capabilities are required. Traditional robot vision systems mainly rely on cameras or simple sensors, which face significant challenges in environments with limited vision or complex lighting conditions. Especially in dynamic environments, traditional vision systems have difficulty in obtaining high-quality environmental data in real time and accurately. In the research and development and application of humanoid robots, real-time acquisition of accurate three-dimensional environmental information and efficient scene intelligent understanding are the key to improving robot autonomy and functionality. Summary of the invention

[0003] The present invention provides a humanoid robot scene intelligent understanding method based on laser point cloud, which utilizes laser radar technology to provide real-time, highly accurate three-dimensional point cloud data for the humanoid robot and can achieve depth perception and intelligent understanding of the environment.

[0004] In order to achieve these purposes and other advantages of the present invention, a method for intelligent scene understanding of a humanoid robot based on laser point cloud is provided, comprising:

[0005] A laser radar scanning system is installed on a humanoid robot, and the humanoid robot autonomously navigates and collects three-dimensional real scene data through the laser radar scanning system to obtain scene point cloud data;

[0006] The obtained scene point cloud data is processed using an adaptive radius sampling module to extract sufficient valid point cloud data;

[0007] Perform point cloud semantic feature extraction based on RPV-Net on the extracted valid point cloud data to obtain the 3D bounding box and semantic label of the point cloud data constituting the building components in the scene;

[0008] The channel attention module is used to process the point cloud data after the semantic features of the point cloud are extracted. The average pooling method is used to obtain the overall features of the point cloud. The maximum pooling method is used to extract the discriminative individual features. The overall features and individual features are aggregated to obtain the global features of the point cloud.

[0009] A multimodal hybrid model of a scene understanding Bert pre-training model and an encoder-decoder model is constructed to process the acquired point cloud global features. The scene understanding Bert pre-training model is used to increase the diversity of training data. The encoder-decoder model is used to understand three-dimensional point cloud objects as three-dimensional ground objects and parameter information of three-dimensional ground objects, thereby improving the humanoid robot's ability to understand scenes.

[0010] Preferably, in the humanoid robot scene intelligent understanding method based on laser point cloud, the laser radar scanning system includes: a laser radar scanner, an inertial measurement unit and a PTP time soft synchronization module, wherein the laser radar scanner and the inertial measurement unit are fused in a tight coupling manner, and the PTP time soft synchronization module synchronizes the time of the laser radar, the inertial measurement unit and the humanoid robot.

[0011] Preferably, the humanoid robot scene intelligent understanding method based on laser point cloud further includes calibrating the laser radar scanning system, specifically:

[0012] Using the Allan variance analysis method to evaluate the noise characteristics and bias stability of the inertial measurement unit, and internally calibrating the inertial measurement unit;

[0013] The lidar_align calibration method is used to calibrate the external parameters of the laser radar scanner and the inertial measurement unit, and a transformation matrix between the coordinate system of the laser radar scanner and the coordinate system of the inertial measurement unit is obtained to transform the point cloud data in the laser radar scanner coordinate system into the coordinate system of the inertial measurement unit.

[0014] Preferably, the method for intelligently understanding a humanoid robot scene based on laser point cloud, after calibrating the laser radar scanning system, further comprises removing motion distortion from the scene point cloud data collected by the laser radar scanner, and the specific process is as follows:

[0015] The motion distortions that appear in the scene point cloud data include: a. If the scanning field of view of the laser radar scanner always falls on the same flat wall, when moving in a direction perpendicular to the wall, the distance from the laser radar scanner to the same point on the wall will continue to decrease, causing the point cloud data to bulge along the direction of movement of the laser radar scanner; b. When the laser radar scanner moves at an angle, the movement of the center of the scan line petals will cause the scan line petal point cloud to be stretched into an irregular shape, i.e., center distortion; c. The laser radar scanner has a fixed field of view angle and cannot cover a circle, causing the range of the field of view to scale with the change in distance, causing the size of the scan line petals to scale with the movement of the laser radar scanner, i.e., edge distortion;

[0016] The motion distortion of a and b is removed by performing motion compensation on the point cloud between frames through the inertial measurement unit. The specific calculation method is: P correct =T k,i ×P i , where t k and t k+1 The start and end time of a frame of point cloud recorded by the laser radar scanner, t i is the i-th point P in the point cloud set S of the k-th frame i The time relative to the starting time, S num is the total number of points in a frame point cloud set, T k,k+1 is the inter-frame pose transformation, T k,i P i The position transformation of the point relative to the starting time, P correct To utilize P i The point coordinates after motion distortion correction calculated by the pose transformation matrix corresponding to the point;

[0017] In view of the motion distortion of c, unreliable scanning points at the edge are filtered out by limiting the field of view angle.

[0018] Preferably, the humanoid robot scene intelligent understanding method based on laser point cloud further includes reconstructing the scene point cloud data for point cloud storage after removing motion distortion of the scene point cloud data, and the specific process is as follows:

[0019] A frame of point cloud data scanned by the laser radar scanner includes 20 scan line petals in the graphics, each scan line petal is continuous at the center, each scan line petal takes its point closest to the scan center as the starting point, and the entire petal is sequentially stored in a row of the matrix;

[0020] The spatially adjacent scan line petals are stored in adjacent rows of the matrix. Specifically, the 20 scan line petals in a frame of point cloud data are obtained by rotating 4 times with 5 scan line petals as a period. In one scanning cycle, the starting point numbers of the 5 scan line petals recorded clockwise are 0, 1000, 2000, 500, and 1500 respectively; the difference in the starting point numbers of the scan line petals in 4 scanning cycles is 2500, and the starting point numbers of adjacent scan line petals indexed clockwise in a frame of scanning point cloud are calculated through the point number relationship; the corresponding scan line petals are indexed sequentially by the obtained starting point numbers, and stored row by row in sequence to obtain a matrix with continuous spatial relationships between rows.

[0021] Preferably, the humanoid robot scene intelligent understanding method based on laser point cloud, after reconstructing the scene point cloud data for point cloud storage, further comprises: modifying the formula for extracting corner point and plane point features of each frame of point cloud data, specifically:

[0022] A. Calculate the vector difference between the point of curvature to be sought and the five points in front of it and the five points behind it to the origin of the scan, and sum them up. The length of the sum is used to reflect the curvature value of the point to be sought. The curvature value calculation formula is:

[0023]

[0024] Among them, c i is the scanning point P i The curvature value of , S is an integer set from -5 to 5 that does not include 0, and Represents the vector of the i-th point and the j-th point in the k-th frame point cloud data in the laser radar scanner coordinate system L;

[0025] B. When the curvature value of the scanning point is greater than 1, the scanning point is judged to be a corner point; when the curvature value of the scanning point is less than 0.1, the scanning point is judged to be a plane point;

[0026] C. Extract the corner points and plane points in each frame of point cloud data according to steps A and B.

[0027] Preferably, in the method for intelligent scene understanding of a humanoid robot based on laser point cloud, the method uses an adaptive radius sampling module to process the obtained scene point cloud data, specifically: firstly, the Euclidean distance between the sampling center point and all points in the scene point cloud data is calculated, and the points exceeding a specific threshold R are marked as Q, and the remaining points retain the original value; then, the points within the spherical sampling radius are arranged in ascending order of distance, and the nearest w points are extracted as sampling points; if the sampling point contains a Q value, indicating that the number of valid points within the spherical sampling radius is less than w, then the sampling radius needs to be gradually increased until the condition of at least w valid points is met, so as to ensure that there are enough valid points for feature learning;

[0028] The extraction of point cloud semantic features based on RPV-Net is performed on the extracted valid point cloud data, specifically: input point cloud P∈R N×3 , that is, the original point cloud data contains three dimensions, where N represents the number of point clouds; seeds are obtained through the semantic segmentation network; the pillar-based idea is used to extract the boundary voxel features of the entire pillar, a vote is performed on the obtained pillar, and then a voting & sampling &

[0029] In the Grouping operation, each seed generates a vote independently through the voting module; then the votes are grouped into h groups and processed by the Proposal module to generate the final Proposal, and finally output the 3D bounding box and semantic label.

[0030] Preferably, the humanoid robot scene intelligent understanding method based on laser point cloud processes the point cloud data after the point cloud semantic features are extracted by using a channel attention module, and the specific process is as follows:

[0031] The feature of point cloud data can be expressed as F = R BxNx1xC , where B, N, and C represent the batch size, the number of point clouds, and the number of feature channels, respectively;

[0032] The average pooling method and the maximum pooling method are used along the dimension of the number of feature channels C. The average pooling method is used to obtain the overall features of the point cloud, and the maximum pooling method is used to extract more distinguishing individual features. The overall features and individual features are then aggregated to generate two different point cloud global features F. avg and F max .

[0033] Preferably, the method for intelligent scene understanding of a humanoid robot based on laser point cloud, wherein the diversity of training data is increased by using the scene understanding Bert pre-training model, is specifically:

[0034] The scene understanding BERT pre-training model expands the data set for the generative adversarial network (GAN) model and increases the diversity of training data;

[0035] In the standard generative adversarial network GAN model, it includes a generator G and a discriminator D;

[0036] By adjusting the parameter settings of the Generative Adversarial Network (GAN) model, training samples are set for the scene point cloud dataset;

[0037] The goal of the generator G is to generate fake data that is as close to the real data as possible, and the goal of the discriminator D is to distinguish between the real data and the fake data generated by the generator G, adding a balancing coefficient γ to adjust the weight between the two.

[0038] Preferably, the encoder-decoder model processing process constructed by the humanoid robot scene intelligent understanding method based on laser point cloud is specifically as follows:

[0039] The point cloud encoder transforms the point cloud P∈R n*d As input, where n is the number of input point clouds and d is the feature dimension of each input point;

[0040] The output of the point cloud encoder is a point feature sequence X = (x1, x2, ..., x m) ∈R m*c , where m is the number of output point features and c is the output point feature dimension;

[0041] Through the multi-layer perceptron MLP, the point feature X is mapped to the point label Y = (y1, y2, ..., ym) ∈R k*c′ , where c′ is the dimension of the point label;

[0042] 3D-MELL is a decoder-only interface that accepts a token sequence consisting of text and point tokens, i.e., a mixed token sequence, which is represented by Z = (z1, z2, ..., z k) ∈R k*c′ , where k is the total number of tokens;

[0043] Using the self-attention mechanism, 3D-MELL is able to understand the contextual relationships between different types of tokens, enabling it to generate responses based on text and point cloud inputs; the output of 3D-MELL is a sequence of predicted tags Z <i =(z1,z2,…,z i-1 ), the prediction of the i-th token is conditioned on all previous tokens.

[0044] The present invention has at least the following beneficial effects:

[0045] First, in order to solve the problem of uneven density of point cloud data, the present invention uses an adaptive radius sampling module to process the obtained scene point cloud data to extract sufficient valid point cloud data, thereby ensuring that there are enough points for feature learning and improving the humanoid robot's ability to understand complex scenes and the segmentation accuracy.

[0046] Second, the present invention designs a channel attention module to address the problems of slow processing speed and feature loss caused by the large amount of environmental data, so as to solve the problem of avoiding the loss of individual features and overall features while reducing the number of parameters.

[0047] Third, the laser radar scanner used in the present invention is a solid-state laser radar scanner. In view of the small viewing angle, petal-shaped scanning trajectory and non-repetitive scanning method of the solid-state laser radar scanner, the motion distortion is removed, the point cloud of the solid-state laser radar scanner is reconstructed, the spatial relationship between the scanning lines is established, and the feature extraction formula is corrected, thereby improving the extraction efficiency and accuracy, effectively improving the situation of insufficient feature points and incorrect selection caused by the laser radar scanner and the environment, and solving the problem of point cloud stratification and aggravated degradation.

[0048] Fourth, use the multimodal hybrid model of the scene understanding Bert pre-training model and the encoder-decoder model to improve the humanoid robot's ability to understand complex scenes.

[0049] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a process flow of a humanoid robot scene intelligent understanding method based on laser point cloud according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the PTP time synchronization principle described in an embodiment of the present invention;

[0052] Figure 3 This is a flow chart of the laser radar scanner and IMU external parameter calibration according to an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of the distortion of the laser radar scanner perpendicular to the wall in an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the central distortion when the laser radar scanner has an angle deviation in an embodiment of the present invention;

[0055] Figure 6 Schematic diagram of edge distortion caused by a small field of view angle of a laser radar scanner in an embodiment of the present invention;

[0056] Figure 7 A schematic diagram of a petal-shaped point cloud scanned by a laser scanning radar having a certain curvature in an embodiment of the present invention;

[0057] Figure 8 : is a corresponding relationship diagram between a point number and a standard curvature difference of the point under standard conditions in an embodiment of the present invention;

[0058] Fig. 9 A distance correction value variation diagram in which the difference between the curvature of the corresponding point at different distances and the curvature of the corresponding point under standard conditions is calculated as the distance correction value in an embodiment of the present invention;

[0059] Fig.10 This is a schematic diagram of the sampling method using Pointnet++;

[0060] Fig.11 Schematic diagram of the adaptive radius sampling method of the present invention used in an embodiment of the present invention;

[0061] Fig.12 Schematic diagram of point cloud semantic segmentation in an embodiment of the present invention;

[0062] Fig.13 Schematic diagram of a channel attention module in an embodiment of the present invention;

[0063] Fig.14 Schematic diagram of the encoder-decoder model processing process in an embodiment of the present invention;

[0064] Fig.15 is a 3D bounding box estimation image of the overall point cloud image acquired by the laser radar system in the example of the present invention;

[0065] Fig.16 A 3D bounding box estimation image of the local extraction image obtained in the example of the present invention;

[0066] Fig.17 It is a 3D bounding box estimation image of the accurate local extraction image obtained in the example of the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0068] It should be understood that the terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other elements or combinations thereof.

[0069] like Figure 1 As shown, an embodiment of the present invention provides a method for intelligent scene understanding of a humanoid robot based on laser point cloud, comprising the following steps:

[0070] S100, assembling a laser radar scanning system on a humanoid robot, the humanoid robot autonomously navigates and collects three-dimensional real scene data through the laser radar scanning system to obtain scene point cloud data.

[0071] Among them, in the ROS operating system of the humanoid robot, a move_base navigation planning framework is added, and an mbot_navigation random autonomous navigation function module is built, which realizes the function of the humanoid robot from traveling on a fixed route to random autonomous navigation to collect three-dimensional real-scene data.

[0072] The laser radar scanning system includes: a laser radar scanner, an inertial measurement unit and a PTP time soft synchronization module, wherein the laser radar scanner and the inertial measurement unit are integrated through a tight coupling manner, and the PTP time soft synchronization module synchronizes the time of the laser radar, the inertial measurement unit and the humanoid robot.

[0073] Specifically, the LiDAR scanner is a solid-state LiDAR scanner, which is a scanning sensor that obtains the geometric shape of the target by measuring the distance. It is mainly composed of three parts: a laser transmitter, a detector, and an information processing system. It has strong anti-interference capabilities and can work in harsh environments. The working principle of the LiDAR scanner is to emit visible and near-infrared laser waves to the target. After the emitted laser reaches the object to be measured, it will be reflected back to the LiDAR and received by the receiver. By recording the time from the laser beam being emitted to the reflection to the reception, the distance of the target to be measured from the scanner and the intensity of the laser beam reflected back are calculated in combination with the speed of light, thereby obtaining the geometric shape of the target to be measured. Specific measurement methods include triangulation ranging, time of flight (ToF) and phase difference method.

[0074] An inertial measurement unit (IMU) is a sensor used to measure the physical quantities of an object, including linear acceleration, angular velocity, and magnetic field direction. It is mainly composed of an accelerometer, a gyroscope, and a magnetometer. The working principle of an IMU is to determine the motion state of the instrument by measuring acceleration and angular velocity. The accelerometer can measure linear acceleration along the X, Y, and Z axes, the gyroscope can measure angular velocity around the X, Y, and Z axes, and the magnetometer can measure the direction and intensity of the geomagnetic field to calibrate the drift of the gyroscope, thereby obtaining the moving direction and rotation state of the device. There is a deviation between the physical quantity measured by the IMU and the actual physical quantity, which can be mainly divided into two categories: systematic errors such as bias and random errors such as noise, so the existence of errors must be taken into account during the calculation process.

[0075] The basic calculation formula of IMU is as follows:

[0076]

[0077] in is the measurement value of the accelerometer, a true is the true acceleration, b a is the bias of the accelerometer, n a is the noise in the accelerometer measurement process, is the measurement value of the gyroscope, ω true is the true angular velocity, b g is the bias of the gyroscope, n g is the noise in the gyroscope measurement process.

[0078] In practical applications, in order to ensure measurement accuracy, such errors need to be compensated. Compensation methods usually include two types of calibration and filtering algorithms. Calibration can be performed when the device is stationary or in motion to determine the magnitude of the error, and the error value obtained by calibration is subtracted from the subsequent measurement. Filtering algorithms (such as Kalman filtering) estimate and compensate for bias in real time during motion to reduce the impact of noise.

[0079] In order to obtain a high-precision point cloud map, multiple sensors need to be fused and used to obtain multi-source data for mutual constraint correction. There are two ways of sensor fusion: loose coupling and tight coupling. The fusion method used in the embodiment of the present invention is tight coupling. Tight coupling is the joint construction of observation equations and motion equations for motion state estimation by the laser radar scanner and IMU. The principle is to put the data collected by the laser radar scanner and IMU into the same state vector, jointly optimize the rotation matrix, translation matrix, speed, and IMU zero bias, and only the state vector needs to be optimized. The error term containing the laser radar measurement residual and the IMU measurement residual is constructed using the laser radar scanner and IMU data to achieve the purpose of simultaneously optimizing the posture, speed and deviation of the tightly coupled system. The tightly coupled system has a compact architecture, highly integrated sensors, mutual constraints between sensor models, more full information utilization, and can achieve higher positioning and mapping accuracy and efficiency. The core issue of multi-sensor fusion operation is multi-sensor data fusion. Data fusion mainly includes two aspects, namely, time synchronization of multi-sensor data and coordinate system alignment.

[0080] Among them, for time synchronization, the embodiment of the present invention adopts a PTP time soft synchronization module, specifically:

[0081] The laser radar scanner used in the embodiment of the present invention is a solid-state laser radar. Therefore, in the absence of GPS and PPS hardware signals, PTPv2 can be used for time synchronization between the laser radar scanner and other devices. This method requires minimal external hardware support and only requires a master clock device in the entire network. That is, the clock of the solid-state laser radar scanner is synchronized to the host used. The synchronization process follows the IEEE1588v2.0PTP protocol, using the delay request-response mechanism in the protocol. Livox Mid-40 acts as a slave and performs PTP time synchronization with the master port (the host's clock device).

[0082] The principle of PTP synchronization is that periodic time information is sent from the master port, and the slave port receives the information with a timestamp. By calculating the time delay and time difference between the information being sent and received, the local time is adjusted to keep the time of the slave device consistent with that of the master device. The principle diagram is as follows Figure 2 shown.

[0083] The transmission path delay can be expressed as:

[0084]

[0085] The time offset can be expressed as:

[0086]

[0087] The synchronization process is as follows:

[0088] (1) The master port sends a Sync message to the slave port and records the time T1;

[0089] (2) After receiving the Sync message from the port, record the receiving time T2;

[0090] (3) After the master port sends the Sync message, it immediately sends a Follow_Up message with a T1 value to the slave port;

[0091] (4) The slave port sends a Delay_Req delay request message to the master port and records the sending time T3;

[0092] (5) After receiving the Delay_Req message, the master port records the receiving time T4;

[0093] (6) The master port sends a Delay_Resp delay request response message carrying T4 to the slave port.

[0094] S110, calibrating the laser radar scanning system, specifically:

[0095] S111. Use the Allan variance analysis method to evaluate the noise characteristics and bias stability of the inertial measurement unit, and perform internal calibration on the inertial measurement unit.

[0096] Among them, the errors of IMU are mainly Gaussian white noise and bias (bias random walk). The purpose of calibrating the internal parameters of IMU is to improve the accuracy of positioning, navigation and attitude estimation.

[0097] Allen variance is a time domain analysis method that can separate and identify different sources of noise from time series data.

[59] , Allen variance σ 2 (τ) is a function of the observation time interval τ.

[0098]

[0099] Where N is the number of data points, x i is the i-th data point, and τ is the sampling time interval.

[0100] S112. Calibrate the external parameters of the laser radar scanner and the inertial measurement unit using the lidar_align calibration method, and obtain a transformation matrix between the coordinate system of the laser radar scanner and the coordinate system of the inertial measurement unit to convert the point cloud data in the laser radar scanner coordinate system to the coordinate system of the inertial measurement unit.

[0101] Among them, the alignment of the coordinate system is to convert the point cloud data to the coordinate system of the IMU sensor for subsequent data processing. Therefore, the alignment of the coordinate system requires obtaining the relative position relationship between the coordinate systems of the two sensors, so it is necessary to perform external parameter calibration of the laser radar and the IMU to obtain the conversion matrix between the two coordinate systems. The essence of external parameter calibration is to solve the problem of data optimization, that is, the point cloud data obtained by the laser radar scanner is converted to the IMU coordinate system, and the point clouds of the two frames before and after can overlap to the maximum extent. In this case, what is the value of the external parameter? The calibration method adopted in the embodiment of the present invention is lidar_align, and its core principle is to use a random sampling method, that is, to randomly sample the original point cloud (most of the point clouds sampled by default are repeated points), and to find the nearest neighbor point in the next frame of the point cloud through the kdtree method for distance calculation. During the calculation process, K neighbors will produce errors. When the conversion matrix is ​​not optimal, the error will increase. The error is optimized by the nonlinear optimization library nlopt, so that the K neighbor error is approximately 0, thereby solving the rotation matrix and translation matrix at this time.

[0102] The specific calibration process is as follows: Figure 3 As shown, it should be noted that the calibration data should include a large amount of rotation and translation during the recording process, and the data volume should be controlled within 2G, so that the calibration result has a higher accuracy.

[0103] S120, after calibrating the laser radar scanning system, further comprising removing motion distortion of the scene point cloud data collected by the laser radar scanner, the specific process is as follows:

[0104] The motion distortions that appear in the scene point cloud data include: a. If the scanning field of view of the LiDAR scanner always falls on the same flat wall, when moving in a direction perpendicular to the wall, the distance from the LiDAR scanner to the same point on the wall will continue to decrease, causing the point cloud data to bulge along the direction of movement of the LiDAR scanner, such as Figure 4 As shown in Figure 2; b. When the laser radar scanner moves at an angle, the movement of the scan line petal center will cause the scan line petal point cloud to be stretched into an irregular shape, that is, center distortion, such as Figure 5 As shown; c. The field of view of the laser radar scanner is fixed and cannot cover a circle, resulting in the range of the field of view scaling with the change of distance, so that the size of the scan line petals scales with the movement of the laser radar scanner, that is, edge distortion, as shown in Figure 6;

[0105] The motion distortion of a and b is removed by motion compensation of the inter-frame point cloud by the inertial measurement unit. The acquisition frequency of IMU is 100HZ, and the laser radar point cloud with a frequency of 10HZ can be interpolated. Specific calculation method: P correct =Tk,i ×P i , where t k and t k+1 The start and end time of a frame of point cloud recorded by the laser radar scanner, t i is the i-th point P in the point cloud set S of the k-th frame i The time relative to the starting time, S num is the total number of points in a frame point cloud set, T k,k+1 is the inter-frame pose transformation, T k,i P i The position transformation of the point relative to the starting time, P correct To utilize P i The point coordinates after motion distortion correction are calculated by the pose transformation matrix corresponding to the point. It should be noted that when calculating t i When calculating T, it is assumed that the solid-state laser radar scanner emits lasers at uniform time intervals and the internal prism rotates at a uniform speed. k,i , assuming that the lidar scanner maintains a uniform motion within one frame time, the pose change is also uniform within one frame time.

[0106] The above motion distortion removal method can be used to roughly correct the point cloud in the same frame to the starting time of the current frame, eliminating the influence of motion distortion on the point cloud features.

[0107] In view of the motion distortion of c, unreliable scanning points at the edge are filtered out by limiting the field of view angle.

[0108] It should be noted that the scanning frequency of the solid-state laser radar scanner selected in the embodiment of the present invention is 10Hz, and the scanning time of a frame of point cloud is 0.1s. Ideally, the same frame of point cloud should be collected at the same time, but in fact, a frame of point cloud is collected evenly within 0.1s. While collecting data, the laser radar scanner itself is also in motion. When the last point of the current frame point cloud is collected, the laser radar scanner has a different scanning position from the first point of the current frame. However, the timestamps of all scanning points in a frame are the time when the point cloud collection starts, and it is considered that the point cloud of a frame is collected at the beginning. This causes the movement of the laser radar scanner itself to affect the scanned point cloud, generating motion distortion of the point cloud. The more violent the laser radar movement within a frame, the more serious the motion distortion caused.

[0109] S130, after removing the motion distortion of the scene point cloud data, it also includes reconstructing the scene point cloud data for point cloud storage, and the specific process is as follows:

[0110] A frame of point cloud data scanned by the laser radar scanner includes 20 scan line petals in the graphics, each scan line petal is continuous at the center, each scan line petal takes its point closest to the scan center as the starting point, and the entire petal is sequentially stored in a row of the matrix;

[0111] The spatially adjacent scan line petals are stored in adjacent rows of the matrix. Specifically, the 20 scan line petals in a frame of point cloud data are obtained by rotating 4 times with 5 scan line petals as a period. In one scanning cycle, the starting point numbers of the 5 scan line petals recorded clockwise are 0, 1000, 2000, 500, and 1500 respectively; the difference in the starting point numbers of the scan line petals in 4 scanning cycles is 2500, and the starting point numbers of adjacent scan line petals indexed clockwise in a frame of scanning point cloud are calculated through the point number relationship; the corresponding scan line petals are indexed sequentially by the obtained starting point numbers, and stored row by row in sequence to obtain a matrix with continuous spatial relationships between rows.

[0112] It should be noted that a frame of point cloud after removing motion distortion is generally stored in a matrix in the order of scanning, so that the algorithm can quickly index and call the point cloud by row or column later. The solid-state laser radar scanning point cloud used in the embodiment of the present invention can also be stored in such a matrix, but due to the non-repetitive scanning method of the solid-state laser radar, its scanning point cloud does not have obvious row and column characteristics in space, resulting in the row and column characteristics of its storage matrix having no physical meaning, and its rows and columns cannot be used to index and construct relationships of point clouds in space. Therefore, it is necessary to reconstruct the point cloud of the solid-state laser radar to establish the spatial relationship between the scan lines.

[0113] A frame of scanning point cloud of solid-state laser radar is composed of 20 petals in graphics, and each petal is continuous at the center. The scanning point cloud of a single petal can be stored in the matrix row by row. Each scan line petal takes its closest point to the scanning center as the starting point, and the entire petal is stored in a row of the matrix in sequence. If there are empty points on the scan line petals that are replaced by the origin coordinates without return values, they will be removed to ensure that the same scan line in the matrix is ​​continuous in space. Since the petals with continuous point numbers on the scan line are not adjacent in space, there is no spatial relationship between adjacent rows of the matrix stored in point number order, and the petals that are adjacent in space need to be stored in adjacent rows of the matrix.

[0114] After the point cloud spatial relationship matrix is ​​established according to step S130, the matrix row-column relationship can be used to quickly index the same feature in adjacent rows and columns during subsequent feature point extraction and feature matching, without traversing all feature points, reducing the amount of calculation and speeding up the algorithm. The method of removing empty points and storing in the order of adjacent petals will cause the points in adjacent rows and columns in the matrix to be discontinuous in point number. Although the matrix stored according to this method will lose its original corresponding point number continuous features, it provides a more efficient and faster spatial indexing method than the point number indexing method, and reconstructs the spatial features between points.

[0115] S140, after reconstructing the scene point cloud data for point cloud storage, further comprising: modifying the formula for extracting corner point and plane point features of each frame of point cloud data, specifically:

[0116] A. Calculate the vector difference between the point of curvature to be sought and the five points in front of it and the five points behind it to the origin of the scan, and sum them up. The length of the sum is used to reflect the curvature value of the point to be sought. The curvature value calculation formula is:

[0117]

[0118] Among them, c i is the scanning point P i The curvature value of , S is an integer set from -5 to 5 that does not include 0, and Represents the vector of the i-th point and the j-th point in the k-th frame point cloud data in the laser radar scanner coordinate system L;

[0119] B. When the curvature value of the scanning point is greater than 1, the scanning point is judged to be a corner point; when the curvature value of the scanning point is less than 0.1, the scanning point is judged to be a plane point;

[0120] C. Extract the corner points and plane points in each frame of point cloud data according to steps A and B.

[0121] It should be noted that the LEGO-LoopLOAM algorithm proposed in the embodiment of the present invention extracts corner points and plane points in each frame of the point cloud by calculating the curvature of each point as a judgment condition. The method calculates the vector difference between the desired curvature point and the five points before and after it to the scanning origin and sums them up, and uses the length of the sum to reflect the curvature of the desired point. The method defines that the curvature value is greater than 1 as a corner point, and the curvature value is less than 0.1 as a plane point. Let the spatial coordinates of the desired curvature point be P i (x i ,y i ,z i ), the curvature calculation formula is as follows:

[0122]

[0123] where c i Point P i The curvature of , S is a set of integers from -5 to 5 that does not include 0, and Represents the vector of the i-th point and the j-th point in the k-th frame point cloud data in the laser radar scanner coordinate system L.

[0124] The method of replacing the point curvature with this calculated value cleverly reduces the amount of calculation of the actual curvature calculation method, is more efficient, and has good application effects. However, it requires that the points on the scan line have no height difference and can be losslessly projected on the plane under the radar coordinate system. The non-repetitive petal-shaped scan line of the solid-state laser radar scanner does not meet this requirement and is not suitable for this method. Therefore, the feature extraction formula is modified according to the scanning characteristics of the solid-state laser radar scanner.

[0125] The point cloud of a single scan line of a mechanical laser radar scanner is arranged in a straight line along the XOY plane. The curvature calculated according to the original formula is the curvature of the point on the XOY plane. However, the point cloud of a single frame scanned by a solid-state laser radar scanner is in the shape of a petal. The arrangement of the point cloud on the plane is not straight, but has a certain curvature. Figure 7 The inherent curvature of the scan line will interfere with the calculation results of the feature point extraction formula, resulting in misjudgment of diagonal feature points and plane feature points.

[0126] Since the scanning line of the solid-state LiDAR scanner has its own curvature, each point on the scanning line has a corresponding curvature even on a plane, which causes the curvature value calculated by the solid-state LiDAR scanning point cloud using the above formula to change discontinuously and inconsistent with the actual curvature change. In actual experiments, sometimes a plane feature point is judged as an angle feature point, so it is necessary to eliminate the influence of the inherent curvature of the scanning point.

[0127] First, it is necessary to determine the difference between the curvature of each point on the solid-state scanner scanning line and the curvature of the straight scanning line under standard conditions. This study defines the standard condition as the scanning head is 10 meters away from the vertical plane and the scanning direction is perpendicular to the plane. The calculated value of the point curvature of the scanned point cloud under this condition is used as the standard curvature difference c between the solid-state laser radar scanning line and the straight scanning line. stdif . Under standard conditions, the petals of each scan line in the scan point cloud are basically similar, with 500 points on each scan petal, and a single petal is basically symmetrical on both sides of the line connecting the scan center point and the farthest end point. Therefore, it is only necessary to calculate the standard curvature difference of 250 points on half a petal to replace the standard curvature difference of all points on the scan line under standard conditions with the corresponding points. Number each point from the point closest to the scan center point on a single scan line to the farthest end point of the petal, with corresponding numbers from 0 to 249. Under standard conditions, the correspondence between the point number and the standard curvature difference of the point is as follows: Figure 8As shown. The change of standard curvature with point number can be approximated by the following expression:

[0128] c srdif =5.28e —7 ×Index 2 +0.002

[0129] The standard curvature difference of the scanning point is not a fixed value. Its size is mainly affected by two factors: the X-axis angle of the line connecting the scanning point and the scanning origin and the distance from the scanning point to the scanning origin. Therefore, it is necessary to construct a dynamic correspondence between the angle and distance based on the standard curvature difference to compensate for the dynamic curvature difference of the scanner at different distances and different postures.

[0130] The inherent curvature of a point on a single petal of a single-frame point cloud will change with the change of the X-axis angle θ between the scanning point and the scanning origin, so it is necessary to determine the corresponding relationship between the angle and the standard curvature difference. By calculating the corresponding relationship between the angle and the point number, the corresponding relationship between the standard curvature difference and the change of the angle can be derived. The data recorded by the lidar point cloud only have the three-dimensional coordinates and reflection intensity of the spatial point, which limits the available known quantities calculated in the subsequent expressions to only three-dimensional coordinates. The calculation formula for the angle θ at any point on the scan line is:

[0131]

[0132] Calculate the angle corresponding to the point number under standard conditions, and the fitting linear expression is as follows:

[0133] θ=1.9e —4 ×Index+0.54

[0134] The corresponding expression of the standard curvature difference and the X-axis angle between the scanning point and the scanning origin can be obtained by combining them, and then converted into the corresponding expression of the standard curvature difference and the spatial coordinates of the point. The specific formula is as follows:

[0135]

[0136] The curvature calculation formula used in the present invention is essentially a method of reflecting the change in curvature size through the difference in distance between local point clouds. When the scanning distance becomes longer, the distance difference between adjacent points and the scanning center will also increase, causing the corresponding curvature values ​​to increase synchronously. The same feature will have different curvatures calculated at different scanning distances, affecting feature matching and loop detection accuracy. Therefore, it is necessary to eliminate the influence of distance on curvature as much as possible, and its influence is mainly reflected in the fact that the calculated curvature value will increase with the increase of scanning distance.

[0137] In order to obtain the distance correction equation corresponding to the curvature change, the present invention collects point clouds for the same wall every 2 meters, with a collection distance range of 10-60 meters, and a total of 25 collections. The curvature of the points with the same point number on a single petal in the different distance scan lines collected each time is calculated. i The difference between the curvature of the corresponding point and the curvature of the corresponding point under standard conditions is taken as the distance correction value. Then the distance correction value changes as follows Fig. 9 The corresponding expression is:

[0138] c dis =0.016x i +0.006

[0139] Combining the above formulas, the dynamic correction value of curvature can be obtained as follows:

[0140]

[0141] The corrected curvature calculation formula of the present invention is as follows:

[0142]

[0143] S200 , using an adaptive radius sampling module to process the obtained scene point cloud data to extract sufficient valid point cloud data.

[0144] The specific process of the adaptive radius sampling module processing the obtained scene point cloud data is as follows: first, the Euclidean distance between the sampling center point and all points in the scene point cloud data is calculated, and the points exceeding the specific threshold R are marked as Q, and the remaining points retain the original value; then the points within the spherical sampling radius are arranged in ascending order by distance, and the nearest w points are extracted as sampling points; if the sampling point contains a Q value, it means that the number of valid points within the spherical sampling radius is less than w, then the sampling radius needs to be gradually increased until the condition of at least w valid points is met, to ensure that there are enough valid points for feature learning, thereby improving the humanoid robot model's ability to understand complex scenes and segmentation accuracy. Fig.10 and Fig.11 As shown, Fig.10 It adopts the sampling method of Pointnet++. Fig.11 The adaptive radius sampling method of the present invention is adopted. The green balls represent the point cloud data within the initial radius, the yellow balls represent the point cloud data copied by selecting the maximum value, and the red balls and blue balls represent the learned point cloud features with the initial radius.

[0145] In addition, the present invention also extracts local features by locally sampling the point cloud and defining a range, selecting points within the range as local features to capture the fine geometric structure of a small neighborhood. Each local point will undergo a multi-layer perceptron (MLP) operation and local maximum pooling to obtain local global features. These local features are regrouped and processed on a larger scale to produce feature expressions on a wider scale. This process is repeated until the features of the entire point cloud data are obtained, thereby enhancing the model's understanding and representation capabilities of the overall point cloud structure.

[0146] S300, performing point cloud semantic feature extraction based on RPV-Net on the extracted valid point cloud data, and obtaining 3D bounding boxes and semantic labels of the point cloud data constituting building components in the scene.

[0147] The extraction of point cloud semantic features based on RPV-Net is specifically as follows: Input point cloud P∈R N×3 , that is, the original point cloud data containing three dimensions; randomly sample through the semantic segmentation backbone network and obtain features with point depth information after processing in the feature aggregation module, cluster into M point subsets according to the point features, and then vote for each point subset to obtain votes, use the pillar-based BEV method to estimate and classify the shape of the bounding box, and redefine seeds, that is, obtain seeds through the semantic segmentation network; use the pillar-based idea to extract the boundary voxel features of the entire pillar (Boundary voxel), vote on the obtained pillar, and then perform a Vote&Sampling&Grouping operation on the obtained point cloud features. Each seed seed independently generates a vote Vote through the voting module; then the votes are grouped into h Groups and processed by the Proposal module to generate the final Proposal, and finally output the 3D bounding box (3D bounding box) and semantic labels (Sematiclabels). Fig.12 As shown, given an input point cloud with XYZ coordinates, votes are generated from the point cloud and grouped into M clusters. For each of the M voting cluster centers, f grids are divided from them. The representative points in the divided grids imply the possible areas of the objects. Then, the seed points around the representative points are revisited and the features of the surrounding seed points are aggregated to the representative points. The proposal refinement and classification module fuses and processes the aggregated voting features and the revisited seed features to generate refined representative points and semantic categories of objects, which can be easily converted into 3D object bounding boxes.

[0148] S400. Use the channel attention module to process the point cloud data after the semantic features of the point cloud are extracted, use the average pooling method to obtain the overall features of the point cloud, use the maximum pooling method to extract the distinctive individual features, aggregate the overall features and individual features, and obtain the global features of the point cloud.

[0149] The specific processing process is as follows:

[0150] After obtaining the high-dimensional features of three-dimensional point cloud data, the present invention focuses on exploring the impact of each feature channel on the overall point cloud information and implements a channel attention mechanism for point cloud data. This mechanism highlights the information of key channels by analyzing the information association of each channel of the point cloud feature. First, extract information that can reflect the global features of the point cloud, and based on this information, learn to obtain the attention weights of each channel, and then obtain the point cloud features processed by the channel attention module. The channel attention module is as follows: Fig.13 shown.

[0151] The feature of point cloud data can be expressed as F = R BxNx1xC , where B, N, and C represent the batch size, the number of point clouds, and the number of feature channels, respectively. The present invention performs a pooling operation along the dimension of the number of feature channels C to generate a point cloud channel attention mechanism Mc∈R BxNx1xC The output feature Fc is obtained by matrix cross product of the point cloud feature F and Mc, that is, the present invention performs pooling along the dimension of the number of feature channels C, and the formula can be expressed as: Fc = F × Mc(F).

[0152] In order to generate the point cloud channel attention mechanism Mc, the present invention adopts two methods, average pooling and maximum pooling, along the dimension of the number of feature channels N. Average pooling is used to obtain the overall features of the point cloud, while maximum pooling helps to extract more distinguishing features. These two pooling methods perform feature aggregation on the point cloud data input feature F to generate two different point cloud global features F avg and F max . In addition, the present invention uses a multi-layer perceptron (MLP) to extract point cloud features, and uses a shared parameter MLP to train the features. By first reducing the coefficient to k, and then restoring the size of the point cloud channel C, combined with the nonlinear activation function Sigmoid, the required channel attention mechanism weight is obtained. The point cloud channel attention mechanism formula can be expressed as: Mc = s{MLP(Avg(F))+MLP(Max(F))}, where s represents the Sigmoid function.

[0153] S500. Construct a multimodal hybrid model of a scene understanding Bert pre-training model and an encoder-decoder model, process the acquired point cloud global features, increase the diversity of training data through the scene understanding Bert pre-training model, and understand the three-dimensional point cloud object into a three-dimensional ground object and parameter information of the three-dimensional ground object through the encoder-decoder model, thereby improving the humanoid robot's ability to understand the scene.

[0154] S510: Increasing the diversity of training data through the scenario understanding Bert pre-training model, specifically:

[0155] S511, the scene understanding BERT pre-training model helps the generative adversarial network GAN model to expand the data set and increase the diversity of training data;

[0156] S512. In the standard generative adversarial network GAN model, it includes a generator G and a discriminator D;

[0157] S513, setting training samples for the scene point cloud data set by adjusting parameter settings of the generative adversarial network GAN model;

[0158] S514. The goal of the generator G is to generate fake data that is as close to the real data as possible. The goal of the discriminator D is to distinguish the real data from the fake data generated by the generator G, and a balance coefficient γ is added to adjust the weight between the two.

[0159] It should be noted that the scene understanding Bert pre-training model can perform noise reduction on the extracted global features of the point cloud, expand the types of data sets, increase the diversity of training data, and enhance the sample set. The output of the scene understanding Bert pre-training model is the data input of the generative adversarial network GAN. In the standard generative adversarial network, it mainly includes the generator G and the discriminator D. The scene understanding Bert pre-training model helps the generative adversarial network GAN model to learn a wider data distribution, thereby improving the generalization ability of the generative adversarial network GAN model.

[0160]

[0161] Among them, V(D,G) is the entire optimization objective function, E represents the expectation, x is the real data, and z is from a certain prior distribution p z The sampled noise, G(z) is the data generated by the generator from the noise, and γ is a dynamically adjusted parameter that is adjusted according to the performance of the generator and the discriminator during training to maintain a balance between the two. If the log(1-D(G(z)) value is low, the loss of the generator can be enhanced by increasing the γ value. The accuracy of the discriminator is evaluated after each epoch and γ is adjusted accordingly.

[0162] S520, the encoder-decoder model processing process is specifically as follows:

[0163] S521, point cloud encoder transforms point cloud P∈R n*d As input, n is the number of input point clouds and d is the feature dimension of each input point; the output of the point cloud encoder is a point feature sequence X = (x1, x2, ..., x m) ∈R m*c , where m is the number of output point features and c is the output point feature dimension;

[0164] S522, through the multi-layer perceptron MLP, the point feature X is mapped to the point mark Y = (y1, y2, ..., y m) ∈R k*c′ , where c′ is the dimension of the point label;

[0165] S523, 3D-MELL is a decoder-only interface that accepts a token sequence consisting of text and point tokens, i.e., a mixed token sequence, which is represented by Z = (z1, z2, ..., z k) ∈R k*c′ , where k is the total number of tokens; using the self-attention mechanism, 3D-MELL is able to understand the contextual relationship between different types of tokens, enabling it to generate responses based on text and point cloud inputs; the output of 3D-MELL is a series of predicted tags Z <i =(z1,z2,…,z i-1 ), the prediction of the i-th token is conditioned on all previous tokens.

[0166] in, Fig.14 Schematic diagram of the encoder-decoder model processing process. The point cloud encoder includes a self-attention mechanism module, a feedforward neural network module, and a geometric information perception adapter module.

[0167] The encoder-decoder model constructed can understand the intelligently extracted 3D point cloud objects into 3D ground objects such as beams, columns, brackets, plaques, pavements, flower beds, trees, etc., as well as their size, age, material, etc. Therefore, the multimodal hybrid model can intelligently recognize point clouds, images, text and other information.

[0168] In summary, through the above embodiments, humanoid robots will have a high degree of environmental perception and intelligent understanding capabilities, enabling them to work more effectively and autonomously in various application scenarios. This will not only promote the development of humanoid robot technology, but also contribute to the construction of a future intelligent society.

[0169] Let's take an example to illustrate this.

[0170] The laser radar scanning system is installed behind the eyes of the robot head. The laser radar scanning system is used to collect the three-dimensional information of the components to be assembled and obtain high-precision three-dimensional scene point cloud data. The adaptive radius sampling module is used to process the obtained scene point cloud data and extract the point cloud semantic features based on RPV-Net for the extracted valid point cloud data to obtain the 3D bounding box and semantic label of the point cloud data constituting the building components in the scene. Figure 15 to Figure 17 shown. Fig.15 It is the 3D bounding box estimation map of the whole point cloud map obtained by the LiDAR system; Fig.16 The 3D bounding box estimation map of the acquired local extraction map; Fig.17 The 3D bounding box estimation map of the local extraction map is obtained accurately. Then it is combined with a large language model for understanding and communication.

[0171] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0172] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A humanoid robot scene intelligent understanding method based on laser point cloud, characterized in that: include: A laser radar scanning system is installed on a humanoid robot, and the humanoid robot autonomously navigates and collects three-dimensional real scene data through the laser radar scanning system to obtain scene point cloud data; The obtained scene point cloud data is processed using an adaptive radius sampling module to extract sufficient valid point cloud data; Perform point cloud semantic feature extraction based on RPV-Net on the extracted valid point cloud data to obtain the 3D bounding box and semantic label of the point cloud data constituting the building components in the scene; The channel attention module is used to process the point cloud data after the semantic features of the point cloud are extracted. The average pooling method is used to obtain the overall features of the point cloud. The maximum pooling method is used to extract the discriminative individual features. The overall features and individual features are aggregated to obtain the global features of the point cloud. A multimodal hybrid model of a scene understanding Bert pre-training model and an encoder-decoder model is constructed to process the acquired point cloud global features, increase the diversity of training data through the scene understanding Bert pre-training model, and understand the three-dimensional point cloud object into a three-dimensional ground object and parameter information of the three-dimensional ground object through the encoder-decoder model, thereby improving the humanoid robot's ability to understand the scene; The adaptive radius sampling module is used to process the obtained scene point cloud data, specifically: firstly, the Euclidean distance between the sampling center point and all points in the scene point cloud data is calculated, and the distance between the sampling center point and all points in the scene point cloud data exceeds a specific threshold. R The points are marked as Q, and the other points retain their original values; then the points within the spherical sampling radius are arranged in ascending order of distance, and the nearest w points are extracted as sampling points; If the sampling point contains a Q value, it means that the number of valid points within the spherical sampling radius is less than w. In this case, the sampling radius needs to be gradually increased until the condition of at least w valid points is met to ensure that there are enough valid points for feature learning.

2. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 1, characterized in that: The laser radar scanning system includes: a laser radar scanner, an inertial measurement unit and a PTP time soft synchronization module, wherein the laser radar scanner and the inertial measurement unit are integrated through a tight coupling manner, and the PTP time soft synchronization module synchronizes the time of the laser radar, the inertial measurement unit and the humanoid robot.

3. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 2, characterized in that: The laser radar scanning system is also calibrated, specifically: Using the Allan variance analysis method to evaluate the noise characteristics and bias stability of the inertial measurement unit, and internally calibrating the inertial measurement unit; The lidar_align calibration method is used to calibrate the external parameters of the laser radar scanner and the inertial measurement unit, and a transformation matrix between the coordinate system of the laser radar scanner and the coordinate system of the inertial measurement unit is obtained to transform the point cloud data in the laser radar scanner coordinate system into the coordinate system of the inertial measurement unit.

4. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 3, characterized in that: After the laser radar scanning system is calibrated, the motion distortion of the scene point cloud data collected by the laser radar scanner is removed. The specific process is as follows: The motion distortions that appear in the scene point cloud data include: a. If the scanning field of view of the laser radar scanner always falls on the same flat wall, when moving in a direction perpendicular to the wall, the distance from the laser radar scanner to the same point on the wall will continue to decrease, causing the point cloud data to bulge along the direction of movement of the laser radar scanner; b. When the laser radar scanner moves at an angle, the movement of the center of the scan line petals will cause the scan line petal point cloud to be stretched into an irregular shape, i.e., center distortion; c. The laser radar scanner has a fixed field of view angle and cannot cover a circle, causing the range of the field of view to scale with the change in distance, causing the size of the scan line petals to scale with the movement of the laser radar scanner, i.e., edge distortion; The motion distortion of a and b is removed by performing motion compensation on the point cloud between frames through the inertial measurement unit. The specific calculation method is: , , ,in, and The start and end time of a frame of point cloud recorded by the laser radar scanner. For the k Frame point cloud collection S Neidi i point Relative to the start time, is the total number of points in a frame point cloud set, is the inter-frame pose transformation, for The position transformation of the point relative to the starting time, To use The point coordinates after motion distortion correction calculated by the pose transformation matrix corresponding to the point; In view of the motion distortion of c, unreliable scanning points at the edge are filtered out by limiting the field of view angle.

5. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 4, characterized in that: After removing the motion distortion of the scene point cloud data, the scene point cloud data is also reconstructed and stored in the point cloud. The specific process is as follows: A frame of point cloud data scanned by the laser radar scanner includes 20 scan line petals in the graphics, each scan line petal is continuous at the center, each scan line petal takes its point closest to the scan center as the starting point, and the entire petal is sequentially stored in a row of the matrix; The spatially adjacent scan line petals are stored in adjacent rows of the matrix. Specifically, the 20 scan line petals in a frame of point cloud data are obtained by rotating 4 times with 5 scan line petals as a period. In one scanning cycle, the starting point numbers of the 5 scan line petals recorded clockwise are 0, 1000, 2000, 500, and 1500 respectively; the difference in the starting point numbers of the scan line petals in 4 scanning cycles is 2500, and the starting point numbers of adjacent scan line petals indexed clockwise in a frame of scanning point cloud are calculated through the point number relationship; the corresponding scan line petals are indexed sequentially by the obtained starting point numbers, and stored row by row in sequence to obtain a matrix with continuous spatial relationships between rows.

6. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 5, characterized in that: After reconstructing the scene point cloud data for point cloud storage, it also includes: correcting the formula for extracting the corner point and plane point features of each frame of point cloud data, specifically: A. Calculate the vector difference between the point of curvature to be sought and the five points in front of it and the five points behind it to the origin of the scan, and sum them up. The length of the sum is used to reflect the curvature value of the point to be sought. The curvature value calculation formula is: , in, Scan point The curvature value of , S is an integer set from -5 to 5 that does not include 0, and Represented in the LiDAR scanner coordinate system L Next k Frame point cloud data i point, j The vector of a point; B. When the curvature value of the scanning point is greater than 1, the scanning point is judged to be a corner point; when the curvature value of the scanning point is less than 0.1, the scanning point is judged to be a plane point; C. Extract the corner points and plane points in each frame of point cloud data according to steps A and B.

7. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 1, characterized in that: The extracted valid point cloud data is subjected to point cloud semantic feature extraction based on RPV-Net, specifically: input point cloud , that is, the original point cloud data contains three dimensions, among which, N Indicates the number of point clouds; seeds are obtained through the semantic segmentation network; the pillar-based idea is used to extract the boundary voxel features of the entire pillar, a vote is taken on the obtained pillar, and then a voting, sampling and grouping operation is performed on the obtained point cloud features. Each seed generates a vote independently through the voting module; the votes are then grouped into h groups and processed by the proposal module to generate the final proposal, and finally the 3D bounding box and semantic label are output.

8. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 1, characterized in that: The point cloud data after point cloud semantic feature extraction is processed by using the channel attention module. The specific process is as follows: The characteristics of point cloud data can be expressed as F = R B xN x 1 xC ,in B , N , C Represent the batch size, the number of point clouds, and the number of feature channels respectively; Number of channels along the feature C The average pooling method and the maximum pooling method are used in the dimension. The average pooling method is used to obtain the overall features of the point cloud, and the maximum pooling method is used to extract more distinguishing individual features. Then, the overall features and individual features are aggregated to generate two different point cloud global features. F avg and F max .

9. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 1, characterized in that: The diversity of training data is increased by using the scene understanding BERT pre-training model, specifically: The scene understanding BERT pre-training model expands the data set for the generative adversarial network (GAN) model and increases the diversity of training data; In the standard generative adversarial network GAN model, it includes generator 𝐺 and discriminator 𝐷; By adjusting the parameter settings of the Generative Adversarial Network (GAN) model, training samples are set for the scene point cloud dataset; The goal of the generator G is to generate fake data that is as close to the real data as possible, and the goal of the discriminator D is to distinguish the real data from the fake data generated by the generator G, adding a balancing coefficient Adjust the weight between the two.

10. The method for intelligent scene understanding of a humanoid robot based on laser point cloud according to claim 9, characterized in that: The processing process of the constructed encoder-decoder model is as follows: The point cloud encoder converts the point cloud As input, n is the number of input point clouds, d is the feature dimension of each input point; The output of the point cloud encoder is a sequence of point features ,in m is the number of output point features, c is the output point feature dimension; Through the multi-layer perceptron MLP, the point feature X is mapped to the point label , where c′ is the dimension of the point label; 3D-MELL is a decoder-only interface that accepts a token sequence consisting of text and point tokens, i.e., a mixed token sequence, which is represented as ,in k is the total number of tokens; Using the self-attention mechanism, 3D-MELL is able to understand the contextual relationships between different types of tokens, enabling it to generate responses based on text and point cloud inputs; The output of 3D-MELL is a series of predicted labels , No. The prediction of a token is conditioned on all previous tokens.

Images