Terrain segmentation method, device, equipment and medium
By combining lidar data and text guidance methods, the problems of existing terrain recognition methods overfitting and label set restrictions in diverse scenarios are solved, and better generalization capabilities and flexible applications are achieved.
Patent Information
- Application Number
- CN202411831406.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-25
Smart Images
Figure CN119888215A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of terrain recognition technology, and in particular to a terrain segmentation method, device, equipment and medium based on the combination of text guidance and radar data. Background Art
[0002] The existing method of terrain recognition for quadruped robots mainly uses one or more sensor data such as cameras, LiDAR, infrared sensors, force sensors, etc. to perform classification or semantic segmentation through deep learning methods. The most studied terrain recognition method is to perform semantic segmentation on image data through convolutional neural network methods.
[0003] In the existing technology, when training terrain segmentation models, a single dataset is mainly used for training. However, a single dataset is mainly for two or three scenes. The model is prone to overfitting to a specific dataset or scene, resulting in poor model reasoning when facing scenes that are significantly different from the training dataset. Some studies also choose to merge existing datasets, but the labeling categories and division standards of existing datasets are not completely consistent, so merging datasets will consume considerable manpower and material resources.
[0004] In addition, most studies can only determine a limited and fixed set of labels when performing terrain segmentation based on the data set used during model training. However, the terrain faced in real scenarios may not belong to the labels in the fixed label set, making it difficult to directly apply the trained model to real scenarios. Summary of the invention
[0005] In view of the above problems, the present invention provides a terrain segmentation method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. The method can use more terrain segmentation data sets to train the model at zero cost. By increasing the data diversity, the overfitting of a specific data set is reduced to improve its generalization ability on unknown data. At the same time, the limitation of a limited label set is solved, so that the model can be more conveniently applied to various real-world scenarios.
[0006] The present invention provides the following scheme:
[0007] A terrain segmentation method, comprising:
[0008] Obtain a three-dimensional point cloud of the target ground area collected by the laser radar;
[0009] Projecting the three-dimensional point cloud to obtain a two-dimensional range image;
[0010] The two-dimensional range image is processed by a visual encoder to obtain a high-dimensional feature vector of the image, so that the high-dimensional feature vector of the image is upsampled by a decoder to obtain a preliminary terrain segmentation result of the two-dimensional point cloud;
[0011] The text data is converted into a text high-dimensional feature vector by using a text encoder, and the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to represent the type of the target ground area;
[0012] Performing a dot product between the two-dimensional point cloud preliminary terrain segmentation result and the text high-dimensional feature vector to obtain a two-dimensional point cloud terrain segmentation result;
[0013] The two-dimensional point cloud terrain segmentation result is three-dimensionally reconstructed to obtain a terrain segmentation result.
[0014] Preferably: the three-dimensional point cloud is mapped onto a two-dimensional plane, and the distance information of each point is converted into a pixel value of an image so as to obtain the two-dimensional distance image.
[0015] Preferably: the visual encoder includes an improved ViT architecture, the improved ViT architecture includes a CNN feature extraction network, the CNN feature extraction network is used to extract visual marker features from the two-dimensional range image, so that the visual encoder processes the visual marker features to obtain a high-dimensional feature vector of the image.
[0016] Preferably: the text encoder includes a Transformer sequence model.
[0017] Preferably, the pre-training parameters of the Transformer sequence model use any one of CLIP training parameters and ViT training parameters.
[0018] Preferably, the tags include at least snow, lawn, gravel, asphalt and sand.
[0019] Preferably: the laser radar is arranged on a quadruped robot.
[0020] A terrain segmentation device, used to perform the above terrain segmentation method, the device comprising:
[0021] A three-dimensional point cloud acquisition unit, used to acquire a three-dimensional point cloud of a target ground area collected by a laser radar;
[0022] A two-dimensional projection unit, used for projecting the three-dimensional point cloud to obtain a two-dimensional range image;
[0023] A preliminary terrain segmentation result acquisition unit is used to process the two-dimensional range image using a visual encoder to obtain an image high-dimensional feature vector, so as to upsample the image high-dimensional feature vector through a decoder to obtain a two-dimensional point cloud preliminary terrain segmentation result;
[0024] A text feature conversion unit, used to convert text data into a text high-dimensional feature vector using a text encoder, wherein the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to indicate the type of the target ground area;
[0025] A dot product unit, used for performing a dot product between the preliminary terrain segmentation result of the two-dimensional point cloud and the high-dimensional feature vector of the text to obtain a terrain segmentation result of the two-dimensional point cloud;
[0026] The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on the two-dimensional point cloud terrain segmentation result to obtain a terrain segmentation result.
[0027] A terrain segmentation device, the device comprising a processor and a memory:
[0028] The memory is used to store program code and transmit the program code to the processor;
[0029] The processor is used to execute the above terrain segmentation method according to the instructions in the program code.
[0030] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned terrain segmentation method.
[0031] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0032] The embodiments of the present application provide a terrain segmentation method, device, equipment and medium, which adds a text-guided branch on the basis of traditional radar data semantic segmentation for terrain recognition. Without the need to spend manpower and material resources to merge existing data sets, all public data sets can be used to train the model, reducing the overfitting of the model to specific data sets and improving the generalization ability of the model in real-world scenarios. The model can learn the alignment relationship between point clouds and text (labels) for terrain segmentation. It has the ability to identify terrain types that have not appeared in the training set. When applied in actual scenarios, different label inputs can be selected according to different scenarios, free from the limitations of limited label sets in traditional methods.
[0033] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 is a flow chart of a terrain segmentation method provided by an embodiment of the present invention;
[0036] Figure 2 is a schematic diagram of a terrain segmentation process of three-dimensional point cloud data provided by an embodiment of the present invention;
[0037] Figure 3 is a schematic diagram of a terrain segmentation device provided by an embodiment of the present invention;
[0038] Figure 4 It is a schematic diagram of a terrain segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0040] See also Figure 1 , which is a terrain segmentation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:
[0041] S101: Acquire a three-dimensional point cloud of a target ground area collected by a laser radar;
[0042] S102: Project the three-dimensional point cloud to obtain a two-dimensional distance image; in specific implementation, the embodiment of the present application can provide mapping the three-dimensional point cloud onto a two-dimensional plane, and the distance information of each point is converted into a pixel value of the image to obtain the two-dimensional distance image.
[0043] S103: Use a visual encoder to process the two-dimensional range image to obtain a high-dimensional feature vector of the image, so as to upsample the high-dimensional feature vector of the image through a decoder to obtain a preliminary terrain segmentation result of a two-dimensional point cloud; in specific implementation, an embodiment of the present application may provide that the visual encoder includes an improved ViT architecture, the improved ViT architecture includes a CNN feature extraction network, and the CNN feature extraction network is used to extract visual marker features from the two-dimensional range image, so that the visual encoder processes the visual marker features to obtain a high-dimensional feature vector of the image.
[0044] S104: Use a text encoder to convert text data into a text high-dimensional feature vector, and the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to represent the type of the target ground area; in specific implementation, the embodiment of the present application can provide that the text encoder includes a Transformer sequence model. Furthermore, the pre-training parameters of the Transformer sequence model use any one of the CLIP training parameters and the ViT training parameters. The labels include at least snow, lawn, gravel, asphalt and sand.
[0045] S105: performing a dot product between the two-dimensional point cloud preliminary terrain segmentation result and the text high-dimensional feature vector to obtain a two-dimensional point cloud terrain segmentation result;
[0046] S106: Perform three-dimensional reconstruction on the two-dimensional point cloud terrain segmentation result to obtain a terrain segmentation result.
[0047] This method can use more terrain segmentation datasets to train the model at zero cost. By increasing data diversity, it can reduce overfitting of specific datasets and improve its generalization ability on unknown data. At the same time, it solves the limitation of limited label sets, making the model more convenient to apply to various real-world scenarios.
[0048] The terrain segmentation method provided in this application adds a text-guided branch based on the traditional semantic segmentation of radar data. All public terrain segmentation datasets are implemented at zero cost to train the model. The model learns the alignment relationship between point clouds and text (labels) for terrain segmentation. When applied in actual scenarios, different label inputs can be selected according to different scenarios, free from the limitation of limited label sets in traditional methods.
[0049] It is understandable that the method provided in the embodiment of the present application can be used in various scenarios that require terrain segmentation. For example, in one implementation, the embodiment of the present application can provide that the laser radar is set on a quadruped robot. By setting the laser radar on the quadruped robot, you can try to collect three-dimensional point cloud data of the target area in front of the quadruped robot, and finally obtain accurate terrain recognition results after processing by this method. By combining text guidance and radar data, the quadruped robot can be unaffected by a limited label set and light shadows when performing terrain recognition in an actual environment.
[0050] The following is a detailed introduction to the terrain segmentation method provided in this application, taking its application on a quadruped robot as an example.
[0051] This application chooses to use Laser Radar (LiDAR) point cloud for terrain segmentation. First, LiDAR provides ultra-high resolution, can be accurate to the centimeter level, and generate high-quality three-dimensional point cloud data, which is suitable for the measurement of complex terrain and slender structures. Secondly, LiDAR has all-weather working capabilities, can operate stably in low light and bad weather conditions, and is not affected by color and light, ensuring ranging accuracy. Based on the above advantages of LiDAR, it is a good choice whether to use radar alone for terrain segmentation or as a beneficial supplement to terrain recognition in images in multimodal methods.
[0052] ViT (Vision Transformer) successfully applied Transformer to the image field, while Transformer had already dominated the text processing field before. So Transformer has opened up the processing of two modal data, text and image (two-dimensional data), and on this basis, multimodal pre-training models such as CLIP for image and text information have emerged.
[0053] Based on the development of the above technologies, this application proposes a collaborative training method for point cloud and text (label) data, which realizes zero-cost multi-dataset training and a terrain segmentation method that is not restricted by a limited label set by learning the alignment relationship between the two.
[0054] Next, each part of the main structure of the quadruped robot terrain segmentation using three-dimensional point cloud data provided by this application is described in detail. Figure 2 shown.
[0055] The overall structure of the present invention is based on Contrastive Language-Image Pre-training (CLIP). CLIP uses a large amount of paired data of images and texts for training, so that the model learns the alignment relationship between images and texts. In the prediction stage, the traditional fixed category label paradigm can be broken, and image categories that may not be seen during training can be predicted based on arbitrary text labels. Therefore, the terrain segmentation model provided by this application draws on the ideas therein, and performs terrain segmentation by learning the alignment relationship between point cloud and text (label) data. The following is a detailed description of each step in terrain segmentation.
[0056] (1) Data processing of 3D point cloud.
[0057] First, the 3D point cloud is projected to obtain a 2D representation of the point cloud. Then, we extract visual markers using CNN from this 2D distance map and input them into a Transformer encoder to obtain high-dimensional features. These features are then decoded through a lightweight network to obtain preliminary segmentation results for the 2D point cloud.
[0058] ①Two-dimensional projection.
[0059] In LiDAR point cloud processing, this step is mainly to convert 3D point cloud data into 2D range images. Specifically, the 3D point cloud data is mapped to a 2D plane, and the distance information of each point is converted into the pixel value of the image. This representation is not only compact and efficient, but also retains important spatial information.
[0060] ②Visual encoder.
[0061] The visual encoder provided in this application is implemented by improving the architecture of Vision Transformer (ViT). The implementation of ViT is that the image is first divided into small blocks, which are linearly embedded into a high-dimensional space to generate a sequence of visual markers; then, this marker sequence is directly processed by the Transformer architecture to output its high-dimensional features.
[0062] The ViT architecture is improved in two aspects: Since the processing of ViT basically does not use the inductive bias of the image, in order to make full use of the inductive bias, the embodiment of the present application first uses CNN to extract features from the image, and then uses the obtained feature map to generate a visual tag sequence, in order to obtain better results. CNN here refers to a convolutional neural network that can extract features from images.
[0063] Since the visual task of the embodiment of the present application is semantic segmentation, a Decoder needs to be added to upsample the low-level feature information extracted by the Encoder to restore the original resolution of the input image and generate pixel-level classification results. Typically, the decoder is implemented by a series of structures, such as interpolation, deconvolution, and depooling.
[0064] (2) Text-guided branch processing.
[0065] The processing of text (label) is done by the Text Encoder, which is responsible for converting it into a high-dimensional feature vector. This feature vector can capture the semantic information of the label and can be compared and calculated with the image feature vector generated by the Vision Encoder in the same vector space. Figure 1 As shown in Figure 1, inputting N labels into the text encoder will generate a high-dimensional feature vector for each label, where the dimension of the feature is C. The structure of the text encoder is relatively uniform, and a large number of studies have shown that the multi-layer Transformer sequence model structure is more suitable for extracting text features.
[0066] (3) Integration of point cloud and text (label) features.
[0067] The dot product of the two-dimensional point cloud preliminary segmentation result obtained from the point cloud data and the high-dimensional features extracted from the labels is performed on the feature dimension to obtain the terrain segmentation result of the two-dimensional point cloud. It can be seen from the figure that the dimensions of the features extracted by the visual encoder and the label encoder are the same, so no matter how many labels there are and what the labels are, it will not affect the dot product of the two.
[0068] (4) Three-dimensional reconstruction.
[0069] 3D refiner converts pixel-level features in 2D space into point-level predictions in 3D space. It is usually necessary to use K-nearest neighbors (K-NN) or conditional random fields (CRFs) to correct errors in the conversion from 2D to 3D (for example, multiple 3D images are projected onto the same 2D pixel). Of course, you can also use KPConv proposed by Hugues Thomas et al. to achieve 3D mapping using a network approach.
[0070] Next, the specific training and reasoning process of the model for terrain segmentation of three-dimensional point cloud data proposed in the embodiment of the present application is described:
[0071] (1) Training process
[0072] First, the pre-trained parameters of the text encoder come from CLIP, and the pre-trained parameters of the Transformer model in the visual encoder can choose to use the training parameters of CLIP, ViT, etc. For the model parameters of the text encoder, according to existing research results, keeping the parameters frozen throughout the training can better maintain the excellent features trained by the CLIP model, which is beneficial to the final model reasoning accuracy. For the model parameters of the visual encoder, all semantic segmentation datasets (such as Freiburg Forest, RUGD, RELLIS-3D, GTOS, etc.) that are not exactly the same in terms of camera settings, lighting, weather conditions, and annotation labels can be used to fine-tune the model parameters in the visual encoder. When fine-tuning the model for different datasets, you only need to change the input of the text encoder of the text-guided branch according to the label category of each dataset.
[0073] The specific process of adjusting the model parameters is to input the 3D point cloud data and text (label) into the model to obtain the terrain segmentation result of the 3D point cloud and the actual segmentation result provided by the data set to calculate the loss function, and then adjust the model parameters through back propagation to finally obtain the optimized model. During the training process, the text encoder and the visual encoder work together to learn the alignment relationship between the point cloud and the text, so that the text feature vectors that match the image content are closer in the vector space, and the irrelevant ones are farther away, thereby achieving cross-modal semantic alignment.
[0074] (2) Reasoning process
[0075] Since the model training process learns the alignment relationship between text and image, the labels input during inference are not limited to the labels input during training. When the model is applied to the actual environment for inference, you only need to input the expected labels to the text encoder according to the current environment (such as snow, lawn, gravel, asphalt, sand, etc.). The number and order of labels have no effect on the final result.
[0076] In summary, the terrain segmentation method provided by this application adds a text-guided branch on the basis of traditional radar data semantic segmentation for terrain recognition. Without the need to spend manpower and material resources to merge existing data sets, all public data sets can be used to train the model, reducing the overfitting of the model to specific data sets and improving the generalization ability of the model in real-world scenarios. The model can learn the alignment relationship between point clouds and text (labels) for terrain segmentation. It has the ability to identify terrain types that have not appeared in the training set. When applied in actual scenarios, different label inputs can be selected according to different scenarios, free from the limitations of limited label sets in traditional methods.
[0077] See also Figure 3 , the embodiment of the present application may also provide a terrain segmentation device, such as Figure 3As shown, for executing the above-mentioned terrain segmentation method, the device may include:
[0078] The three-dimensional point cloud acquisition unit 301 is used to acquire the three-dimensional point cloud of the target ground area collected by the laser radar;
[0079] A two-dimensional projection unit 302, used for projecting the three-dimensional point cloud to obtain a two-dimensional range image;
[0080] The preliminary terrain segmentation result acquisition unit 303 is used to process the two-dimensional range image using a visual encoder to obtain an image high-dimensional feature vector, so as to upsample the image high-dimensional feature vector through a decoder to obtain a two-dimensional point cloud preliminary terrain segmentation result;
[0081] A text feature conversion unit 304 is used to convert text data into a text high-dimensional feature vector using a text encoder, wherein the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to indicate the type of the target ground area;
[0082] A dot product unit 305 is used to perform a dot product between the preliminary terrain segmentation result of the two-dimensional point cloud and the high-dimensional feature vector of the text to obtain a terrain segmentation result of the two-dimensional point cloud;
[0083] The three-dimensional reconstruction unit 306 is used to perform three-dimensional reconstruction on the two-dimensional point cloud terrain segmentation result to obtain a terrain segmentation result.
[0084] The embodiment of the present application may also provide a terrain segmentation device, the device comprising a processor and a memory:
[0085] The memory is used to store program code and transmit the program code to the processor;
[0086] The processor is used to execute the steps of the above-mentioned terrain segmentation method according to the instructions in the program code.
[0087] like Figure 4 As shown, a terrain segmentation device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.
[0088] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.
[0089] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the terrain segmentation method.
[0090] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:
[0091] Obtain a three-dimensional point cloud of the target ground area collected by the laser radar;
[0092] Projecting the three-dimensional point cloud to obtain a two-dimensional range image;
[0093] The two-dimensional range image is processed by a visual encoder to obtain a high-dimensional feature vector of the image, so that the high-dimensional feature vector of the image is upsampled by a decoder to obtain a preliminary terrain segmentation result of the two-dimensional point cloud;
[0094] The text data is converted into a text high-dimensional feature vector by using a text encoder, and the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to represent the type of the target ground area;
[0095] Performing a dot product between the two-dimensional point cloud preliminary terrain segmentation result and the text high-dimensional feature vector to obtain a two-dimensional point cloud terrain segmentation result;
[0096] The two-dimensional point cloud terrain segmentation result is three-dimensionally reconstructed to obtain a terrain segmentation result.
[0097] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.
[0098] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.
[0099] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.
[0100] Of course, it should be noted that Figure 4 The structure shown does not constitute a limitation on the terrain segmentation device in the embodiment of the present application. In actual applications, the terrain segmentation device may include Figure 4More or fewer components than shown, or combinations of certain components.
[0101] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned terrain segmentation method.
[0102] It should be noted that, in the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.
[0103] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0104] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A terrain segmentation method, characterized in that: include: Obtain a three-dimensional point cloud of the target ground area collected by the laser radar; Projecting the three-dimensional point cloud to obtain a two-dimensional range image; The two-dimensional range image is processed by a visual encoder to obtain a high-dimensional feature vector of the image, so that the high-dimensional feature vector of the image is upsampled by a decoder to obtain a preliminary terrain segmentation result of the two-dimensional point cloud; The text data is converted into a text high-dimensional feature vector by using a text encoder, and the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to represent the type of the target ground area; Performing a dot product between the two-dimensional point cloud preliminary terrain segmentation result and the text high-dimensional feature vector to obtain a two-dimensional point cloud terrain segmentation result; The two-dimensional point cloud terrain segmentation result is three-dimensionally reconstructed to obtain a terrain segmentation result.
2. The terrain segmentation method according to claim 1, characterized in that: The three-dimensional point cloud is mapped onto a two-dimensional plane, and the distance information of each point is converted into a pixel value of an image to obtain the two-dimensional distance image.
3. The terrain segmentation method according to claim 1, characterized in that: The visual encoder includes an improved ViT architecture, which includes a CNN feature extraction network. The CNN feature extraction network is used to extract visual marker features from the two-dimensional range image, so that the visual encoder processes the visual marker features to obtain a high-dimensional feature vector of the image.
4. The terrain segmentation method according to claim 1, characterized in that: The text encoder includes a Transformer sequence model.
5. The terrain segmentation method according to claim 4, characterized in that: The pre-training parameters of the Transformer sequence model use any one of CLIP training parameters and ViT training parameters.
6. The terrain segmentation method according to claim 1, characterized in that: The labels include at least snow, grass, gravel, asphalt and sand.
7. The terrain segmentation method according to claim 1, characterized in that: The laser radar is arranged on the quadruped robot.
8. A terrain segmentation device, characterized in that: Used to execute the terrain segmentation method according to any one of claims 1 to 7, the device comprises: A three-dimensional point cloud acquisition unit, used to acquire a three-dimensional point cloud of a target ground area collected by a laser radar; A two-dimensional projection unit, used for projecting the three-dimensional point cloud to obtain a two-dimensional range image; A preliminary terrain segmentation result acquisition unit is used to process the two-dimensional range image using a visual encoder to obtain an image high-dimensional feature vector, so as to upsample the image high-dimensional feature vector through a decoder to obtain a two-dimensional point cloud preliminary terrain segmentation result; A text feature conversion unit, used to convert text data into a text high-dimensional feature vector using a text encoder, wherein the text high-dimensional feature vector and the image high-dimensional feature vector can be compared and calculated in the same vector space; the text high-dimensional feature vector is used to capture the semantic information of each label in the text, and the label is used to indicate the type of the target ground area; A dot product unit, used for performing a dot product between the preliminary terrain segmentation result of the two-dimensional point cloud and the high-dimensional feature vector of the text to obtain a terrain segmentation result of the two-dimensional point cloud; The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on the two-dimensional point cloud terrain segmentation result to obtain a terrain segmentation result.
9. A terrain segmentation device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the terrain segmentation method according to any one of claims 1 to 7 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the terrain segmentation method according to any one of claims 1 to 7.