A point cloud data processing method, system, device and medium of a humanoid robot

Abstract

The application discloses a point cloud data processing method, system, device and medium of a humanoid robot. The method comprises the following steps: acquiring a motion state parameter of a robot and initial point cloud data collected; performing dynamic sampling processing on the robot according to the motion state parameter and the initial point cloud data, and generating a projection image; performing multi-channel feature extraction and dynamic weight fusion processing on the projection image, and obtaining an encoded feature map; performing multi-scale residual coding processing on the encoded feature map according to an information theory constraint, and obtaining encoded data; and performing reconstruction processing on the encoded data according to the motion state parameter, and obtaining reconstructed point cloud data. The embodiment of the application can improve the processing efficiency of point cloud data and can be widely applied to the technical field of intelligent robots.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to a point cloud data processing method, system, device and medium for humanoid robots. Background Technology

[0002] When humanoid robots move in complex environments, they generate a large amount of point cloud data through sensors such as LiDAR. While point cloud data processing methods exist, these methods are not tailored to the application scenarios of humanoid robots, resulting in poor dynamic adaptability and a lack of task relevance. Consequently, they still cannot meet the dynamic operational needs of humanoid robots in terms of real-time processing and reconstruction accuracy.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to provide a method, system, device, and medium for processing point cloud data of a humanoid robot, which can improve the efficiency of point cloud data processing for humanoid robots.

[0005] To achieve the above objectives, one aspect of this application proposes a point cloud processing method for a humanoid robot, the method comprising:

[0006] Acquire the robot's motion state parameters and the collected initial point cloud data;

[0007] The robot is dynamically sampled based on the motion state parameters and initial point cloud data to generate a projection image;

[0008] Multi-channel feature extraction and dynamic weight fusion processing are performed on the projected image to obtain an encoded feature map;

[0009] The encoded feature map is subjected to multi-scale residual coding based on information theory constraints to obtain encoded data.

[0010] The encoded data is reconstructed based on the motion state parameters to obtain reconstructed point cloud data.

[0011] In some embodiments, the step of dynamically sampling the robot based on the motion state parameters and initial point cloud data to generate a projected image includes:

[0012] The pose transformation matrix and motion velocity are obtained based on the motion state parameters.

[0013] Geometric feature analysis is performed on the initial point cloud data to obtain curvature and normal vector;

[0014] The sampling magnification is calculated based on the curvature, the normal vector, and the motion speed.

[0015] The robot's sampling region is predicted based on the pose transformation matrix to obtain the predicted sampling region;

[0016] The predicted sampling region is sampled according to the sampling magnification to obtain the projected image.

[0017] In some embodiments, the step of performing multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map includes:

[0018] The projected image is subjected to geometric information extraction processing to obtain a geometric feature map;

[0019] The projected image is subjected to motion state extraction processing to obtain a motion feature map;

[0020] The initial point cloud data is processed by spatiotemporal attention weight calculation to obtain dynamic weights;

[0021] The geometric feature map and the motion feature map are fused according to the dynamic weights to obtain the encoded feature map.

[0022] In some embodiments, the process of calculating spatiotemporal attention weights on the initial point cloud data to obtain dynamic weights includes:

[0023] The initial point cloud data is subjected to point cloud feature extraction processing to obtain point cloud features;

[0024] The point cloud features are subjected to feature mapping processing to obtain query, key, and value vectors;

[0025] The initial point cloud data is subjected to relative motion velocity calculation and mapping processing to obtain the velocity modulation factor;

[0026] Spatial attention weights and temporal attention weights are calculated based on the query, the key, and the value vector.

[0027] The spatial attention weight and the temporal attention weight are jointly calculated based on the velocity modulation factor to obtain the dynamic weight.

[0028] In some embodiments, the step of performing multi-scale residual coding processing on the coded feature map according to information theory constraints to obtain coded data includes:

[0029] The information theory constraints are constructed based on the robot's processing tasks;

[0030] The encoded feature map is subjected to global structure preservation processing to obtain the base layer feature data;

[0031] The encoded feature map is subjected to local feature capture processing to obtain residual layer feature data;

[0032] The base layer feature data and the residual layer feature data are entropy encoded according to the information theory constraints to obtain the encoded data.

[0033] In some embodiments, the step of performing local feature capture processing on the encoded feature map to obtain residual layer feature data includes:

[0034] The encoded feature map is layered according to the voxel grid to obtain multi-layer point cloud data;

[0035] The residual layer feature data is obtained by performing local feature aggregation and skip connection processing on the multi-layer point cloud data.

[0036] In some embodiments, the step of reconstructing the encoded data based on the motion state parameters to obtain reconstructed point cloud data includes:

[0037] The encoded data is subjected to multi-scale decoding to obtain multi-scale point cloud data;

[0038] Based on the motion state parameters, the multi-scale point cloud data is subjected to coordinate system alignment processing to obtain preprocessed data;

[0039] The preprocessed data is subjected to spatiotemporal fusion and geometric optimization to obtain the reconstructed point cloud data.

[0040] To achieve the above objectives, another aspect of this application proposes a point cloud data processing system for a humanoid robot, the system comprising:

[0041] The data acquisition module is used to acquire the robot's motion state parameters and the collected initial point cloud data;

[0042] The dynamic sampling module is used to perform dynamic sampling processing on the robot based on the motion state parameters and initial point cloud data to generate a projection image;

[0043] The feature fusion module is used to perform multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map;

[0044] The point cloud encoding module is used to perform multi-scale residual encoding processing on the encoded feature map according to information theory constraints to obtain encoded data;

[0045] The point cloud reconstruction module is used to reconstruct the encoded data based on the motion state parameters to obtain reconstructed point cloud data.

[0046] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0047] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0048] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0049] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, device, and medium for processing point cloud data of a humanoid robot. This solution generates a projection image by dynamically sampling the robot based on motion state parameters and point cloud data. It can dynamically adjust the sampling strategy and encoding parameters by analyzing the local geometric characteristics of the point cloud and the robot's motion state, providing a data foundation for subsequent point cloud encoding compression and reconstruction processing. Furthermore, this solution obtains an encoded feature map by performing multi-channel feature extraction and dynamic weight fusion processing on the projection image. It can capture the spatiotemporal correlation of point cloud sequences through a self-attention mechanism, improving the feature correlation in dynamic scenes. This solution obtains encoded data by performing multi-scale residual encoding processing on the encoded feature map according to information theory constraints. It can perform multi-scale residual encoding on the point cloud, allowing each layer of point cloud data to be independently optimized in conjunction with pose information, improving the hierarchical control of bitrate-distortion. In addition, this solution, combined with information theory-constrained encoding, improves the data transmission efficiency related to navigation tasks. Attached Figure Description

[0050] Figure 1 This is a flowchart of a point cloud data processing method for a humanoid robot provided in an embodiment of this application;

[0051] Figure 2 This is a schematic diagram of a spatiotemporal attention module structure provided in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the structure of a point cloud data processing system for a humanoid robot provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0055] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0056] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] When humanoid robots move in complex environments, the point cloud data generated by sensors such as LiDAR is enormous (typically >1MB per frame), posing significant challenges to data transmission, storage, and real-time processing. Existing point cloud data processing methods neglect the local geometric characteristics of the point cloud, leading to large errors in the reconstruction of complex surfaces and insufficient preservation of geometric features. Furthermore, fixed sampling strategies cannot adapt to changes in the field of vision during humanoid robot movement; critical area information is severely lost during rapid rotation, resulting in poor dynamic adaptability. Additionally, this method does not optimize encoding to meet the humanoid robot's task requirements (such as grasping and navigation), resulting in a high proportion of invalid data transmission and a lack of task relevance.

[0059] In view of this, this application provides a method, system, device, and medium for processing point cloud data of a humanoid robot. This solution generates a projection image by dynamically sampling the robot based on motion state parameters and point cloud data. It can dynamically adjust the sampling strategy and encoding parameters by analyzing the local geometric characteristics of the point cloud and the robot's motion state, providing a data foundation for subsequent point cloud encoding compression and reconstruction. Furthermore, this solution obtains an encoded feature map by performing multi-channel feature extraction and dynamic weight fusion processing on the projection image. It can capture the spatiotemporal correlation of point cloud sequences through a self-attention mechanism, improving feature correlation in dynamic scenes. This solution obtains encoded data by performing multi-scale residual encoding processing on the encoded feature map according to information theory constraints. This allows for multi-scale residual encoding of the point cloud, enabling each layer of point cloud data to be independently optimized in conjunction with pose information, improving hierarchical control of bitrate-distortion. In addition, this solution, combined with information theory-constrained encoding, improves the data transmission efficiency related to navigation tasks.

[0060] This application provides a point cloud data processing method for a humanoid robot, relating to the field of intelligent robot technology. This method can be applied to humanoid robots, servers, or software running on either a humanoid robot or a server. In some embodiments, the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a point cloud data processing method for a humanoid robot, but is not limited to the above forms.

[0061] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0062] Figure 1 This is an optional flowchart of a point cloud data processing method for a humanoid robot provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0063] Step S101: Obtain the robot's motion state parameters and the collected initial point cloud data;

[0064] Step S102: Perform dynamic sampling processing on the robot based on the motion state parameters and initial point cloud data to generate a projection image;

[0065] Step S103: Perform multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map;

[0066] Step S104: Perform multi-scale residual coding on the encoded feature map according to information theory constraints to obtain encoded data;

[0067] Step S105: Reconstruct the encoded data according to the motion state parameters to obtain reconstructed point cloud data.

[0068] Steps S101 to S105 of this embodiment involve acquiring the robot's motion state parameters and initial point cloud data. The motion state parameters may include pose transformation and motion speed. By performing local geometric characteristic analysis on the initial point cloud data, the point cloud curvature and normal vector can be calculated. Then, based on the motion state parameters and the initial point cloud data, the robot undergoes dynamic sampling processing to generate a projection image. This embodiment dynamically adjusts the sampling strategy based on the point cloud curvature and the humanoid robot's motion speed to generate an asymmetric projection image. This increases the sampling density in regions with high curvature or fast movement, ensuring the preservation of key features. This embodiment performs multi-channel feature extraction and dynamic weight fusion processing on the projection image to obtain an encoded feature map. A self-attention mechanism can be used to capture the spatiotemporal correlation of the point cloud sequence, improving feature correlation in dynamic scenes. This application embodiment performs multi-scale residual coding on the coded feature map according to information theory constraints to obtain coded data. It can decompose the point cloud into a base layer and multiple detail layers. Each layer is independently optimized in combination with pose information to achieve hierarchical control of bit rate and distortion. Furthermore, it introduces task-related information bottleneck constraints and derives the optimal coding strategy through Lagrange optimization to ensure that the reconstructed point cloud retains the minimum amount of information required for the task, thereby improving the task adaptability of the coding.

[0069] In some embodiments, the step of dynamically sampling the robot based on the motion state parameters and initial point cloud data to generate a projected image includes:

[0070] The pose transformation matrix and motion velocity are obtained based on the motion state parameters.

[0071] Geometric feature analysis is performed on the initial point cloud data to obtain curvature and normal vector;

[0072] The sampling magnification is calculated based on the curvature, the normal vector, and the motion speed.

[0073] The robot's sampling region is predicted based on the pose transformation matrix to obtain the predicted sampling region;

[0074] The predicted sampling region is sampled according to the sampling magnification to obtain the projected image.

[0075] In this embodiment, the pose transformation matrix of the humanoid robot can be calculated using joint angle sensor data and a kinematic model. Specifically, encoders are installed on each driven joint of the humanoid robot (such as the hip, knee, and ankle joints) to output joint angles in real time. A second-order Butterworth low-pass filter is used to preprocess the data to construct the joint transformation matrix. Based on the forward kinematic model, the global pose matrix from the base to the end effector can be obtained as the product of the joint transformation matrices. This embodiment can also optimize pose estimation based on sensor fusion. A state-space model is constructed by collecting three-axis angular velocities and accelerations, and the pose transformation matrix is ​​updated in real time based on the state-space model. This embodiment performs geometric feature analysis on the initial point cloud data. Specifically, the local covariance matrix of each point cloud is calculated, and the point cloud curvature and normal vector are calculated based on the eigenvalues ​​of the covariance matrix. This embodiment calculates the sampling rate using curvature, normal vector, and motion velocity. The normal vector reflects the local surface orientation of the point cloud, and its interaction with the robot's motion direction can be used to dynamically adjust the sampling strategy. By defining the robot's linear velocity direction vector, the angle between the normal vector and the motion direction can be calculated. A smaller angle (the closer the surface orientation is to the motion direction) indicates that the robot is about to approach the surface, and the sampling density needs to be increased. The formula for calculating the angle between the normal vector and the motion direction is as follows:

[0076] θ nv =arccos(|n(p)·v d |);

[0077] In the formula, θ nv Let n(p) represent the included angle, and n(p) represent the normal vector. d This represents the direction vector of the robot's linear velocity.

[0078] Introducing the normal vector influence factor, the expression for which is shown below:

[0079]

[0080] In the formula, λ n =0.8 is the adjustment coefficient. When θ nv When →0, γ n →1.8, sampling density can be increased by 80%.

[0081] In this embodiment, the sampling ratio is calculated by integrating curvature k(p), motion velocity ‖v‖, and normal vector influence factor. The formula for calculating the sampling ratio is as follows:

[0082] k(p) = k0·(1+λ) K k(p))·(1+λ v ||v||)·γ n ;

[0083] In the formula, k(p) represents the sampling rate, k0 is the base rate, and λ κ ,λ v This is the adjustment coefficient.

[0084] This application embodiment also predicts changes in the robot's field of view through a pose transformation matrix. Based on the current pose and movement speed, it predicts the pose at the next moment. By dividing the predicted field of view into N sector regions, the weight of each region can be calculated based on the Jacobian matrix of the pose transformation, enabling the allocation of sampling rates according to weights. Based on the sampling rate, the predicted sampling region is sampled and processed by data acquisition and perception from the humanoid robot's sensors to obtain a projected image.

[0085] In some embodiments, the step of performing multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map includes:

[0086] The projected image is subjected to geometric information extraction processing to obtain a geometric feature map;

[0087] The projected image is subjected to motion state extraction processing to obtain a motion feature map;

[0088] The initial point cloud data is processed by spatiotemporal attention weight calculation to obtain dynamic weights;

[0089] The geometric feature map and the motion feature map are fused according to the dynamic weights to obtain the encoded feature map.

[0090] In this embodiment, geometric information extraction and motion state extraction are performed on the projected image to obtain geometric feature maps and motion feature maps. Then, spatiotemporal attention weights are calculated on the initial point cloud data based on the spatiotemporal attention module to obtain dynamic weights. Based on the calculated dynamic weights, feature fusion processing is performed on the geometric feature maps and motion feature maps to obtain coded feature maps.

[0091] In some embodiments, the process of calculating spatiotemporal attention weights on the initial point cloud data to obtain dynamic weights includes:

[0092] The initial point cloud data is subjected to point cloud feature extraction processing to obtain point cloud features;

[0093] The point cloud features are subjected to feature mapping processing to obtain query, key, and value vectors;

[0094] The initial point cloud data is subjected to relative motion velocity calculation and mapping processing to obtain the velocity modulation factor;

[0095] Spatial attention weights and temporal attention weights are calculated based on the query, the key, and the value vector.

[0096] The spatial attention weight and the temporal attention weight are jointly calculated based on the velocity modulation factor to obtain the dynamic weight.

[0097] In the embodiments of this application, please refer to Figure 2 Feature vectors can be generated from the initial input point cloud data using a multilayer perceptron (MLP) and convolution operations to obtain point cloud features. These point cloud features are then mapped to query, key, and value vectors. In this embodiment, the relative motion velocity of the initial point cloud data is calculated. Specifically, the global motion velocity of the point cloud can be calculated based on the pose transformation matrix. Then, the relative velocity matrix between points is calculated, and a hyperbolic tangent function is used to perform a nonlinear mapping on the relative velocity to obtain a velocity modulation factor. The expression for the velocity modulation factor is as follows:

[0098]

[0099] Wherein, γ(v t,i,j ) represents the speed modulation factor, v t,i,j v is the relative velocity. mean Let σ be the mean velocity. v This represents the standard deviation of the velocity.

[0100] Spatial attention weights and temporal attention weights can be calculated from the query, key, and value vectors. The expression for calculating the spatial attention weight is as follows:

[0101]

[0102] In the formula, Represents spatial attention weights. Indicates a query, K t,j Indicates key, This is the scale normalization factor.

[0103] The expression for calculating the temporal attention weights is as follows:

[0104]

[0105] In the formula, Indicates time attention weights. Indicates a query, K r,i Indicates key, This is the scale normalization factor.

[0106] Finally, the spatial attention weights and temporal attention weights are jointly calculated based on the velocity modulation factor to obtain the dynamic weights. The expression for the joint calculation formula is shown below:

[0107]

[0108] This application embodiment improves the feature correlation in dynamic scenes by fusing point cloud features with attention weights.

[0109] In some embodiments, the step of performing multi-scale residual coding processing on the coded feature map according to information theory constraints to obtain coded data includes:

[0110] The information theory constraints are constructed based on the robot's processing tasks;

[0111] The encoded feature map is subjected to global structure preservation processing to obtain the base layer feature data;

[0112] The encoded feature map is subjected to local feature capture processing to obtain residual layer feature data;

[0113] The base layer feature data and the residual layer feature data are entropy encoded according to the information theory constraints to obtain the encoded data.

[0114] In this embodiment of the application, the expression for the information-theoretic constraint is as follows:

[0115] min q(y|x) I(X;Y)stI(Y;Z)≥I0;

[0116] In the formula, X represents the original point cloud, Y represents the encoding result, Z represents the reconstructed point cloud, and I0 represents the minimum amount of information to be retained for the task. This embodiment also employs low-sampling-rate encoding to capture the global structure of the encoded feature map to obtain basic layer feature data, and performs local feature capture processing by calculating the residuals of the encoded feature map to obtain residual layer feature data, thus preserving local features. This embodiment combines pose information to independently encode each layer, uses pose guidance for spatiotemporal alignment of the encoded feature map, performs motion compensation through inter-frame prediction error calculation, and finally uses arithmetic coding compression, which improves the encoding efficiency of geometric features and pose.

[0117] In some embodiments, the step of performing local feature capture processing on the encoded feature map to obtain residual layer feature data includes:

[0118] The encoded feature map is layered according to the voxel grid to obtain multi-layer point cloud data;

[0119] The residual layer feature data is obtained by performing local feature aggregation and skip connection processing on the multi-layer point cloud data.

[0120] In this embodiment, multi-layer point clouds are generated using different voxel sizes. A base layer preserves the global structure, a detail layer captures mesoscale details, and a detail layer extracts high-frequency details, resulting in multi-layer point cloud data. Then, local feature aggregation and skip connections are performed on the multi-layer point cloud data. The voxel size can be adjusted according to the robot's motion speed. Multi-layer features are aggregated using a spatiotemporal attention mechanism, and each layer's features retain detailed information through skip connections. A curvature histogram is calculated within each local neighborhood and concatenated with the feature vector, enhancing the semantic expression of the geometric structure.

[0121] In some embodiments, the step of reconstructing the encoded data based on the motion state parameters to obtain reconstructed point cloud data includes:

[0122] The encoded data is subjected to multi-scale decoding to obtain multi-scale point cloud data;

[0123] Based on the motion state parameters, the multi-scale point cloud data is subjected to coordinate system alignment processing to obtain preprocessed data;

[0124] The preprocessed data is subjected to spatiotemporal fusion and geometric optimization to obtain the reconstructed point cloud data.

[0125] In this embodiment, a multi-scale point cloud representation is generated by hierarchically decoding the encoded sequence, and the multi-frame point cloud is aligned to a unified coordinate system using the robot pose transformation matrix. The multi-frame point cloud is then fused using spatiotemporal attention weights to generate a reconstructed point cloud. Finally, the continuity of the reconstructed point cloud is optimized based on the local geometric characteristics of the point cloud to obtain the reconstructed point cloud data.

[0126] The following is a detailed description and explanation of the solutions in the embodiments of the present invention, using specific application examples:

[0127] This application applies to the field of intelligent robot technology and is applicable to point cloud data processing scenarios for humanoid robots. By analyzing the local geometric characteristics of point clouds and the motion state of the humanoid robot, it achieves efficient compression and high-precision reconstruction of point cloud data. This application analyzes the local geometric characteristics of point cloud data, calculates point cloud curvature and normal vectors, and obtains the current motion state parameters of the humanoid robot, including pose transformation and motion speed. Based on the point cloud curvature and the humanoid robot's motion speed, the sampling strategy is dynamically adjusted to generate asymmetric projection images. Sampling density is increased in regions with high curvature or fast motion to ensure the preservation of key features. Furthermore, a spatiotemporal attention module is constructed to capture the spatiotemporal correlation of point cloud sequences through a self-attention mechanism, improving feature correlation in dynamic scenes. The point cloud is decomposed into a base layer and multiple detail layers, with each layer independently optimized based on pose information, achieving hierarchical control of bitrate and distortion. This application introduces task-related information bottleneck constraints and derives the optimal encoding strategy through Lagrange optimization to ensure that the reconstructed point cloud retains the minimum amount of information required for the task, improving the task adaptability of the encoding.

[0128] Please see Figure 3 This application also provides a point cloud data processing system for a humanoid robot, which can implement the above-described method. The system includes:

[0129] Data acquisition module 301 is used to acquire the robot's motion state parameters and the collected initial point cloud data;

[0130] The dynamic sampling module 302 is used to perform dynamic sampling processing on the robot based on the motion state parameters and initial point cloud data to generate a projection image;

[0131] The feature fusion module 303 is used to perform multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map;

[0132] Point cloud encoding module 304 is used to perform multi-scale residual encoding processing on the encoded feature map according to information theory constraints to obtain encoded data;

[0133] The point cloud reconstruction module 305 is used to reconstruct the encoded data according to the motion state parameters to obtain reconstructed point cloud data.

[0134] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0135] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0136] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0138] The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0139] The memory 402 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 using the methods described in the embodiments of this application.

[0140] Input / output interface 403 is used to implement information input and output;

[0141] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0142] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0143] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0144] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0145] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0146] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0147] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] This application provides a method, system, device, and medium for processing point cloud data of a humanoid robot. This solution generates a projection image by dynamically sampling the robot based on motion state parameters and point cloud data. It can dynamically adjust the sampling strategy and encoding parameters by analyzing the local geometric characteristics of the point cloud and the robot's motion state, providing a data foundation for subsequent point cloud encoding compression and reconstruction. Furthermore, this solution obtains an encoded feature map by performing multi-channel feature extraction and dynamic weight fusion processing on the projection image. It can capture the spatiotemporal correlation of point cloud sequences through a self-attention mechanism, improving feature correlation in dynamic scenes. This solution obtains encoded data by performing multi-scale residual encoding processing on the encoded feature map according to information theory constraints. This allows for multi-scale residual encoding of the point cloud, enabling each layer of point cloud data to be independently optimized in conjunction with pose information, improving hierarchical control of bitrate-distortion. In addition, this solution, combined with information theory-constrained encoding, improves the data transmission efficiency related to navigation tasks.

[0150] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0151] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0152] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0154] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0155] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0156] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0157] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing point cloud data of a humanoid robot, characterized in that, The method includes the following steps: Acquire the robot's motion state parameters and the collected initial point cloud data; The robot is dynamically sampled based on the motion state parameters and the initial point cloud data to generate a projection image; Multi-channel feature extraction and dynamic weight fusion processing are performed on the projected image to obtain an encoded feature map; The encoded feature map is subjected to multi-scale residual coding based on information theory constraints to obtain encoded data. The encoded data is reconstructed based on the motion state parameters to obtain reconstructed point cloud data. The step of dynamically sampling the robot based on the motion state parameters and initial point cloud data to generate a projected image includes: The pose transformation matrix and motion velocity are obtained based on the motion state parameters. Geometric feature analysis is performed on the initial point cloud data to obtain curvature and normal vector; The sampling magnification is calculated based on the curvature, the normal vector, and the motion speed. The robot's sampling region is predicted based on the pose transformation matrix to obtain the predicted sampling region; The predicted sampling region is sampled according to the sampling magnification to obtain the projected image.

2. The method according to claim 1, characterized in that, The step of performing multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map includes: The projected image is subjected to geometric information extraction processing to obtain a geometric feature map; The projected image is subjected to motion state extraction processing to obtain a motion feature map; The initial point cloud data is processed by spatiotemporal attention weight calculation to obtain dynamic weights; The geometric feature map and the motion feature map are fused according to the dynamic weights to obtain the encoded feature map.

3. The method according to claim 2, characterized in that, The process of calculating spatiotemporal attention weights on the initial point cloud data to obtain dynamic weights includes: The initial point cloud data is subjected to point cloud feature extraction processing to obtain point cloud features; The point cloud features are subjected to feature mapping processing to obtain query, key, and value vectors; The initial point cloud data is subjected to relative motion velocity calculation and mapping processing to obtain the velocity modulation factor; Spatial attention weights and temporal attention weights are calculated based on the query, the key, and the value vector. The spatial attention weight and the temporal attention weight are jointly calculated based on the velocity modulation factor to obtain the dynamic weight.

4. The method according to claim 1, characterized in that, The step of performing multi-scale residual coding processing on the encoded feature map according to information theory constraints to obtain encoded data includes: The information theory constraints are constructed based on the robot's processing tasks; The encoded feature map is subjected to global structure preservation processing to obtain the base layer feature data; The encoded feature map is subjected to local feature capture processing to obtain residual layer feature data; The base layer feature data and the residual layer feature data are entropy encoded according to the information theory constraints to obtain the encoded data.

5. The method according to claim 4, characterized in that, The step of performing local feature capture processing on the encoded feature map to obtain residual layer feature data includes: The encoded feature map is layered according to the voxel grid to obtain multi-layer point cloud data; The residual layer feature data is obtained by performing local feature aggregation and skip connection processing on the multi-layer point cloud data.

6. The method according to any one of claims 1 to 5, characterized in that, The step of reconstructing the encoded data based on the motion state parameters to obtain reconstructed point cloud data includes: The encoded data is subjected to multi-scale decoding to obtain multi-scale point cloud data; Based on the motion state parameters, the multi-scale point cloud data is subjected to coordinate system alignment processing to obtain preprocessed data; The preprocessed data is subjected to spatiotemporal fusion and geometric optimization to obtain the reconstructed point cloud data.

7. A point cloud data processing system for a humanoid robot, characterized in that, The system includes: The data acquisition module is used to acquire the robot's motion state parameters and the collected initial point cloud data; The dynamic sampling module is used to perform dynamic sampling processing on the robot based on the motion state parameters and initial point cloud data to generate a projection image; The feature fusion module is used to perform multi-channel feature extraction and dynamic weight fusion processing on the projected image to obtain an encoded feature map; The point cloud encoding module is used to perform multi-scale residual encoding processing on the encoded feature map according to information theory constraints to obtain encoded data; The point cloud reconstruction module is used to reconstruct the encoded data based on the motion state parameters to obtain reconstructed point cloud data. The dynamic sampling module is used to perform dynamic sampling processing on the robot based on the motion state parameters and initial point cloud data to generate a projected image, including: The pose transformation matrix and motion velocity are obtained based on the motion state parameters. Geometric feature analysis is performed on the initial point cloud data to obtain curvature and normal vector; The sampling magnification is calculated based on the curvature, the normal vector, and the motion speed. The robot's sampling region is predicted based on the pose transformation matrix to obtain the predicted sampling region; The predicted sampling region is sampled according to the sampling magnification to obtain the projected image.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images