Point cloud network training method, point cloud data processing method and device

Abstract

The embodiments of the present application provide a point cloud network training method, a point cloud data processing method and a device, which relate to the fields of unmanned driving and intelligent transportation technology. The method uses a point cloud network to obtain three-dimensional point cloud features of lidar point cloud data, uses an image encoder to obtain two-dimensional image features of multi-view images, uses a visual basic model to mask and divide the multi-view images to obtain multiple object regions, uses a text encoder to generate text features corresponding to each object region, performs image-text-point cloud contrast learning based on text features, two-dimensional region pixel features corresponding to each object region, and three-dimensional region point cloud features, generates semantic region contrast loss values, and trains the point cloud network. Multimodal data is used for joint training, and region-level text features are used to achieve fine-grained cross-modal feature alignment. The self-supervised contrast learning framework based on image-text-point cloud is used to enhance the representation ability of the point cloud network and improve the accuracy of downstream three-dimensional point cloud understanding.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving and intelligent transportation technology, and in particular to a point cloud network training method, a point cloud data processing method and a device. Background Art

[0002] LiDAR point cloud-based 3D scene understanding is widely used in scenarios such as autonomous driving and robotic navigation. It uses algorithms or models to extract features from input 3D point clouds and, based on the extracted results, to provide semantic understanding of the 3D scene content. LiDAR point cloud-based 3D scene understanding models are typically trained using supervised learning based on manually annotated point clouds. However, the high cost of annotating point cloud data limits the applicability of 3D scene understanding.

[0003] Related technologies use self-supervised learning methods based on images and point clouds to train 3D point cloud networks through point-by-point comparisons of pixels and points in the point cloud. However, this point-level learning approach is susceptible to restricted receptive fields, limited contextual information, and potential noise or erroneous pixel semantics, resulting in low 3D feature extraction accuracy and impacting the precision of 3D scene understanding. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a point cloud network training method, a point cloud data processing method and a device to improve the accuracy of the point cloud network in extracting features from lidar point cloud data.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a point cloud network training method, which is performed by a self-supervised training system. The self-supervised training system includes at least an image encoder, a text encoder, a visual base model, and the point cloud network. Except for the point cloud network, other network parameters are frozen. The method includes:

[0006] Acquire laser radar point cloud data and corresponding multi-view images, input the laser radar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features;

[0007] Inputting the multi-view image into the visual base model for mask segmentation to obtain at least one object region, and using the text encoder to generate text features corresponding to each object region;

[0008] Associating the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, and dividing the point-pixel association result according to the object area to obtain a two-dimensional regional pixel feature and a three-dimensional regional point cloud feature corresponding to each object area;

[0009] Image-text-point cloud comparative learning is performed according to the text features, the two-dimensional region pixel features, and the three-dimensional region point cloud features to generate a semantic region contrast loss value, and the parameters of the point cloud network are adjusted based on the semantic region contrast loss value until the trained point cloud network is obtained.

[0010] In some embodiments, the multi-view image includes a scene image corresponding to at least one view, and inputting the multi-view image into the visual base model for mask segmentation to obtain at least one object region includes:

[0011] Inputting the scene images into the visual basic model respectively for mask division to obtain a mask image corresponding to each scene image, wherein the mask image includes at least one mask area;

[0012] Region division is performed in each of the scene images based on the mask region to obtain at least one object region corresponding to each of the scene images.

[0013] In some embodiments, the self-supervised training system further includes an image-to-text base model, and generating text features corresponding to each object region using the text encoder includes:

[0014] For each of the scene images, inputting the corresponding object region into the image-to-text basic model to generate text, thereby obtaining a guide language description corresponding to each of the object regions;

[0015] The guide language description is input into the text encoder for feature extraction to obtain the corresponding text features.

[0016] In some embodiments, performing image-text-point cloud contrast learning based on the text features, the two-dimensional region pixel features, and the three-dimensional region point cloud features to generate a semantic region contrast loss value includes:

[0017] Performing average pooling on the two-dimensional regional pixel features to obtain regional point features, and performing average pooling on the three-dimensional regional point cloud features to obtain regional pixel features;

[0018] The semantic region contrast loss value is calculated according to the region point feature, the region pixel feature and the text feature.

[0019] In some embodiments, performing comparative learning based on the regional point features, the regional pixel features, and the text features to calculate the semantic region contrast loss value includes:

[0020] For each of the object regions, calculating an indication parameter between the region and other object regions according to the region pixel features and the text features, and determining a plurality of negative samples from the other object regions based on the indication parameters;

[0021] A regional loss value is calculated based on the negative sample, the regional point feature corresponding to the object region, and the regional pixel feature, and the semantic region contrast loss value is obtained according to an average of the regional loss values.

[0022] In some embodiments, calculating the indication parameter between the region and the other object regions based on the region pixel features and the text features includes:

[0023] Selecting comparison areas one by one from the other object areas, and obtaining the regional pixel features of the comparison areas as comparison pixel features;

[0024] Calculating a first text similarity between the text feature and the region pixel feature, and calculating a second text similarity between the text feature and the comparison pixel feature;

[0025] A difference between the first text similarity and the second text similarity is obtained, and if the difference is greater than or equal to a preset reference value, the indication parameter is set to 1 to indicate that the comparison area is a negative sample of the object area.

[0026] In some embodiments, calculating the regional loss value based on the negative sample, the regional point feature corresponding to the object region, and the regional pixel feature includes:

[0027] For each of the object regions, calculating a first semantic similarity between the region point feature and the region pixel feature;

[0028] Obtaining the regional pixel feature of each of the negative samples as a sample pixel feature, calculating a sample semantic similarity between the regional point feature and each of the sample pixel features, and calculating a sum of the sample semantic similarities as a second semantic similarity;

[0029] The sum of the first semantic similarity and the second semantic similarity is obtained as an intermediate reference value, and the region loss value is obtained according to a quotient of the first semantic similarity and the intermediate reference value.

[0030] In some embodiments, the self-supervised training system further includes a two-dimensional mapping layer and a three-dimensional mapping layer to be trained. Before dividing the two-dimensional image features according to the object regions to obtain the two-dimensional region pixel features corresponding to each object region, the method further includes:

[0031] Inputting the two-dimensional image features into the two-dimensional mapping layer to perform channel dimension mapping, and updating the two-dimensional image features;

[0032] The three-dimensional point cloud features are input into the three-dimensional mapping layer for channel dimension mapping, and the three-dimensional point cloud features are updated. The channel dimensions of the updated three-dimensional point cloud features and the two-dimensional image features are consistent.

[0033] To achieve the above objectives, a second aspect of an embodiment of the present application provides a three-dimensional point cloud data processing method, comprising:

[0034] Get initial point cloud data;

[0035] The initial point cloud number is input into a point cloud network for feature extraction to obtain point cloud feature data, and the point cloud network is trained by the point cloud network training method described in any one of the first aspects.

[0036] To achieve the above objectives, a third aspect of an embodiment of the present application provides a point cloud network training device, which is executed by a self-supervised training system. The self-supervised training system includes at least an image encoder, a text encoder, a visual base model, and the point cloud network. Except for the point cloud network, other network parameters are frozen. The device includes:

[0037] Initial feature extraction module: used to obtain lidar point cloud data and corresponding multi-view images, input the lidar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features;

[0038] A text description generation module is configured to input the multi-view image into the visual base model for mask segmentation to obtain at least one object region, and generate text features corresponding to each object region using the text encoder;

[0039] A mapping association module is configured to associate the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, and to divide the point-pixel association result according to the object area to obtain a two-dimensional regional pixel feature and a three-dimensional regional point cloud feature corresponding to each object area;

[0040] Contrastive learning module: used to perform image-text-point cloud contrast learning based on the text features, the two-dimensional region pixel features and the three-dimensional region point cloud features, generate semantic region contrast loss values, and adjust the parameters of the point cloud network based on the semantic region contrast loss values ​​until the trained point cloud network is obtained.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first or second aspect above when executing the computer program.

[0042] To achieve the above-mentioned purpose, the fifth aspect of the embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in the first or second aspect above.

[0043] The point cloud network training method, point cloud data processing method, and device proposed in the embodiments of the present application obtain laser radar point cloud data and corresponding multi-view images, input the laser radar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features, input the multi-view images into the visual basic model for mask division to obtain at least one object region, and use a text encoder to generate text features corresponding to each object region, associate the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, divide the point-pixel association result according to the object region, obtain two-dimensional region pixel features and three-dimensional region point cloud features corresponding to each object region, perform image-text-point cloud comparative learning based on the text features, two-dimensional region pixel features, and three-dimensional region point cloud features, generate a semantic region contrast loss value, and adjust the parameters of the point cloud network based on the semantic region contrast loss value until a trained point cloud network is obtained. The embodiments of the present application introduce multimodal joint training and use region-level text features to achieve cross-modal alignment. Based on the self-supervised contrastive learning framework of image-text-point cloud, the representation ability of the point cloud network is enhanced, and local semantic consistency is strengthened during the training process: by shortening the cross-modal feature distance of the same semantic area and pushing away the features of different semantic areas, fine-grained cross-modal feature alignment is achieved, avoiding global feature blurring, and significantly improving the feature extraction accuracy of 3D point clouds, thereby improving the accuracy of downstream 3D point cloud understanding. Without relying on manually annotated point cloud data, low-cost and high-efficiency 3D space understanding is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the training process of the self-supervised training system provided in the embodiment of the present application.

[0045] Figure 2 This is a flowchart of the point cloud network training method provided in an embodiment of the present application.

[0046] Figure 3This is a flowchart provided by an embodiment of the present application for inputting a multi-view image into a visual basic model for mask division to obtain at least one object area.

[0047] Figure 4 It is a schematic diagram of the object area provided in an embodiment of the present application.

[0048] Figure 5 This is a flowchart of using a text encoder to generate text features corresponding to each object area provided by an embodiment of the present application.

[0049] Figure 6 This is a flowchart of channel dimension mapping of two-dimensional image features and three-dimensional point cloud features provided by an embodiment of the present application.

[0050] Figure 7 This is a flowchart provided in an embodiment of the present application for performing image-text-point cloud comparison learning based on text features, two-dimensional region pixel features, and three-dimensional region point cloud features to generate semantic region comparison loss values.

[0051] Figure 8 This is a flowchart provided in an embodiment of the present application for performing comparative learning based on regional point features, regional pixel features, and text features to calculate the semantic region contrast loss value.

[0052] Figure 9 This is a flowchart provided by an embodiment of the present application for calculating indication parameters between regions and other object regions based on regional pixel features and text features.

[0053] Figure 10 This is a flowchart of calculating the regional loss value based on negative samples, regional point features corresponding to the object area, and regional pixel features provided in an embodiment of the present application.

[0054] Figure 11 This is a structural block diagram of a point cloud network training device provided in another embodiment of the present application.

[0055] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] First, let’s analyze some of the terms used in this application:

[0060] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0061] LiDAR point cloud-based 3D scene understanding is widely used in scenarios such as autonomous driving and robotic navigation. It uses algorithms or models to extract features from input 3D point clouds and, based on the extracted results, to provide semantic understanding of the 3D scene content. LiDAR point cloud-based 3D scene understanding models are typically trained using supervised learning based on manually annotated point clouds. However, the high cost of annotating point cloud data limits the applicability of 3D scene understanding.

[0062] Related technologies use self-supervised learning methods based on images and point clouds to train 3D point cloud networks through point-by-point comparisons of pixels and points in the point cloud. However, this point-level learning approach is susceptible to restricted receptive fields, limited contextual information, and potential noise or erroneous pixel semantics, resulting in low 3D feature extraction accuracy and impacting the precision of 3D scene understanding.

[0063] Based on this, the embodiments of the present application provide a point cloud network training method, a point cloud data processing method and a device, introduce multimodal joint training, and use regional text features to achieve cross-modal alignment. Based on the image-text-point cloud self-supervised contrastive learning framework, the representation ability of the point cloud network is enhanced, and local semantic consistency is enhanced during the training process: by shortening the cross-modal feature distance of the same semantic region and pushing away the features of different semantic regions, fine-grained cross-modal feature alignment is achieved, avoiding global feature fuzzification, significantly improving the feature extraction accuracy of the three-dimensional point cloud, and thus improving the accuracy of downstream three-dimensional point cloud understanding, without relying on manually annotated point cloud data, achieving low-cost and high-efficiency three-dimensional space understanding.

[0064] The embodiments of the present application provide a point cloud network training method, a point cloud data processing method and an apparatus, which are specifically illustrated by the following embodiments. First, the point cloud network training method in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making.

[0066] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] The point cloud network training method provided in the embodiment of the present application relates to the field of unmanned driving and intelligent transportation technology. The point cloud network training method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be a computer program running in a terminal or a server. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports point cloud network training, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The point cloud network training method can be executed by the terminal or the server, or by the terminal and the server in collaboration.

[0068] In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch. Furthermore, the terminal can also be an intelligent in-vehicle device. This intelligent in-vehicle device applies the point cloud network training method of this embodiment to provide related services and enhance the driving experience. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). The terminal and server can be connected via Bluetooth, Universal Serial Bus (USB), or a network connection, etc., which is not limited in this embodiment.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0070] First, the self-supervised training system provided by the embodiment of the present application is described. Figure 1 , Figure 1 Schematic diagram of the training process of the self-supervised training system provided in the embodiment of the present application.

[0071] The self-supervised training system includes an image encoder, a text encoder, a visual base model, an image-to-text base model, a 2D mapping layer, a 3D mapping layer, and a point cloud network to be trained. Except for the point cloud network and 3D mapping layer, the model parameters corresponding to the remaining networks are frozen (frozen parameters are indicated by bold solid lines in the figure). This means that during training, only the point cloud network and 3D mapping layer are fine-tuned (the parameters to be trained are indicated by bold dashed lines in the figure), while the other network parameters remain unchanged. This reduces the computational overhead of training and improves the training efficiency of the point cloud network. It also prevents the loss function from non-converging during joint training.

[0072] In one embodiment, the dual encoders in the multimodal model (Contrastive Language-Image Pre-training, CLIP) serve as image and text encoders. The image encoder is the CLIP image encoder, which uses the Vision Transformer (ViT) or ResNet architecture to map input image data to feature vectors. The text encoder is the CLIP text encoder, which uses the Transformer architecture to map input text data to feature vectors.

[0073] In one embodiment, the visual foundational model is the Segment Anything Model (SAM), which performs mask segmentation on input image data and outputs category-independent mask information corresponding to the image data. Furthermore, the image-to-text foundational model is the BLIP-2 model, which generates corresponding text descriptions based on input image data.

[0074] In one embodiment, both the two-dimensional mapping layer and the three-dimensional mapping layer can be MLP network layers, which are used to align the channel dimensions of the input data. The two-dimensional mapping layer is a network layer with frozen parameters, and the three-dimensional mapping layer is a network layer to be trained.

[0075] The point cloud network training method in the embodiment of the present application is described below in conjunction with the above-mentioned self-supervised training system.

[0076] Figure 2 This is an optional flowchart of the point cloud network training method provided in the embodiment of the present application. Figure 2 The method may include but is not limited to steps 110 to 140. It is also understood that this embodiment is Figure 2 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0077] Step 110: Obtain lidar point cloud data and corresponding multi-view images, input the lidar point cloud data into a point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into an image encoder for feature extraction to obtain two-dimensional image features.

[0078] In one embodiment, referring to Figure 1 LiDAR point cloud data is three-dimensional spatial data obtained by scanning the target scene with a LiDAR. The LiDAR emits laser pulses into the target scene and receives the reflected signals generated by the target scene. The distance and orientation of objects in the target scene are measured based on the reflected signals, and LiDAR point cloud data consisting of a set of discrete points is generated. Each point in the LiDAR point cloud data usually contains information such as three-dimensional coordinates and reflection intensity. Multi-view images are a set of two-dimensional images obtained by photographing the target scene from different perspectives. In other words, multi-view images include scene images corresponding to multiple perspectives. These scene images can be collected by multiple fixed cameras (such as vehicle-mounted surround view systems) or mobile cameras (such as drone aerial photography), which can provide rich visual information, including color, texture, and lighting changes.

[0079] Next, refer to Figure 1 , input the lidar point cloud data into the point cloud network Perform feature extraction to obtain three-dimensional point cloud features , where N and D represent the number and feature dimension of point cloud features respectively. In addition, the multi-view images are input into the image encoder Perform feature extraction to obtain the two-dimensional image features corresponding to each scene image , where h, w, and C represent the size and feature dimension of the image features, respectively.

[0080] Step 120: Input the multi-view image into the visual basic model for mask segmentation to obtain at least one object region, and use a text encoder to generate text features corresponding to each object region.

[0081] In one embodiment, considering that direct modal alignment of two-dimensional image features and three-dimensional point cloud features may lack sufficient receptive field and cannot perceive the context, thus limiting the accuracy of representation learning, the embodiment of the present application performs regional-level semantic feature extraction on multi-view images to guide the alignment process of multi-modal data, enhance the ability of point cloud representation learning, and thus improve the accuracy of downstream three-dimensional point cloud understanding. Figure 3 , Figure 3 This is a flowchart of inputting a multi-view image into a visual base model for mask segmentation to obtain at least one object area, provided by an embodiment of the present application, specifically including the following steps:

[0082] Step 310: Input the scene images into the visual basic model respectively for mask division to obtain a mask image corresponding to each scene image.

[0083] In one embodiment, for each scene image, it is input into the visual basic model for mask division to obtain a mask image corresponding to each scene image, wherein possible objects in the scene image are located using their corresponding masks. Therefore, the mask image includes a mask area corresponding to each possible object, and the mask area is used to identify the range occupied by the corresponding object in the scene image.

[0084] Step 320: Perform region division in each scene image based on the mask region to obtain at least one object region corresponding to each scene image.

[0085] In one embodiment, since the object may be irregular, the mask area may be irregular. In order to better understand the context information, the embodiment of the present application expands each mask area into a rectangular box. The rectangular box can be the minimum rectangle containing the corresponding object, or it can be a slightly expanded rectangle of the minimum rectangle. This embodiment does not limit this.

[0086] After obtaining the rectangular area corresponding to each mask area, the corresponding scene image is intercepted using the rectangular area, and the local area of ​​the scene image corresponding to the rectangular area is used as the object area. It can be understood that according to this interception method, each object area contains the corresponding object and part of the background information of the object.

[0087] In one embodiment, to avoid data redundancy, a minimum mask area can be set. Only masked areas larger than this area are required to generate object regions, thereby preventing the generation of overly localized or meaningless data. Furthermore, descriptions that are irrelevant or meaningless to the pre-trained dataset categories, such as "cloudy sky," can be filtered out to generate meaningful and descriptive data, thereby enhancing the regional information for each category.

[0088] In one embodiment, referring to Figure 4 , Figure 4 It is a schematic diagram of the object area provided in an embodiment of the present application. Figure 4 The figure illustrates a scene image that has been masked using a visual foundational model. The resulting mask image includes multiple mask regions, such as a car mask, a traffic cone mask, a pedestrian mask, and a fire hydrant mask. Each mask region is then expanded to obtain a corresponding rectangular frame. This rectangular frame is then used to locate the object region within the scene image. Each mask region contains both the object and its background. For example, the object region corresponding to a traffic cone contains both the cone and its ground background.

[0089] Through the above process, multiple object regions corresponding to each scene image are obtained. It is understandable that due to the difference in viewing angles, the shape and size of the same object in different scene images may be different, and the corresponding object region is also related to the corresponding viewing angle.

[0090] Next, for each object region, determine its corresponding semantic information. Figure 5 , Figure 5 This is a flowchart of using a text encoder to generate text features corresponding to each object area provided by an embodiment of the present application, specifically including the following steps:

[0091] Step 510: For each scene image, the corresponding object region is input into the image-to-text basic model for text generation to obtain a guiding language description corresponding to each object region.

[0092] In one embodiment, for each scene image, all object regions thereof are analyzed to obtain the semantic information contained therein. Figure 1, the object region is fed into the image-based model for content analysis, and a text prompt is used to guide the image-based model to generate a guiding language description based on the object in the object region and its corresponding background. For example, the text prompt can be: "Question: What is the content of the image? Answer:". Figure 4 ,The guided language descriptions generated for different object regions can be: “the rear end of a yellow car on the road”, “a close-up of a truck with large tires”, “a large excavator parked in front of a person”, “a traffic cone placed on the ground”, etc.

[0093] It can be seen that the embodiment of the present application explores the potential of text modality by guiding language description to capture meaningful semantic information, and can generate real and specific region-level text descriptions for each scene image, such as positional relationships, color attributes, etc., to provide more fine-grained and comprehensive reference information for the subsequent comparative learning process, thereby enhancing the self-supervised representation learning capability.

[0094] Step 520: Input the guidance language description into a text encoder for feature extraction to obtain corresponding text features.

[0095] In one embodiment, each object area in the above process has a corresponding guide language description, refer to Figure 1 , input the guide language description into the text encoder for feature extraction, and obtain the corresponding text features , where B and L represent the number of categories in the pre-training dataset and the feature dimension of the text respectively.

[0096] Next, the multimodal data alignment process is performed based on the text features. Before this, the channel dimension mapping of the two-dimensional image features and the three-dimensional point cloud features is also required. In one embodiment, referring to Figure 6 , Figure 6 This is a flow chart of channel dimension mapping of two-dimensional image features and three-dimensional point cloud features provided by an embodiment of the present application, which specifically includes the following steps:

[0097] Step 610: Input the two-dimensional image features into the two-dimensional mapping layer for channel dimension mapping, and update the two-dimensional image features.

[0098] Step 620: Input the 3D point cloud features into the 3D mapping layer for channel dimension mapping, and update the 3D point cloud features.

[0099] In one embodiment, referring to Figure 1 , the two-dimensional image features are updated through the two-dimensional mapping layer to obtain the updated two-dimensional image features , where L represents the updated channel dimension, which is consistent with the feature dimension of the text, and the 3D image features are updated through the 3D mapping layer to obtain the updated 3D point cloud features ,visible, 、 and The channel dimensions of are consistent, which can realize the subsequent comparative learning process.

[0100] Step 130: Associating the two-dimensional image features and the three-dimensional point cloud features according to the preset spatial projection relationship to obtain a point-pixel association result, dividing the point-pixel association result according to the object area to obtain the two-dimensional regional pixel features and three-dimensional regional point cloud features corresponding to each object area.

[0101] In one embodiment, since both the lidar point cloud data and the multi-view images are obtained for the target scene, a preset spatial projection relationship is obtained based on the external parameter data (rotation matrix and translation vector) and internal parameter data (camera focal length, distortion, etc.) of the camera-lidar. For each scene image, the point in the lidar point cloud data is projected into the scene image to obtain the pixel corresponding to the point.

[0102] In one embodiment, a point in the lidar point cloud data has a corresponding point feature in the corresponding 3D point cloud feature, and a pixel has a pixel feature in the corresponding 2D image feature. Therefore, based on the above projection relationship, the 2D image feature and the 3D point cloud feature can be associated to obtain a point-pixel association result, which is expressed as: ,in, Represents the point feature corresponding to the i-th point, represents the pixel feature corresponding to the i-th point in one of the scene images, and K represents the number of successfully paired points.

[0103] In one embodiment, since the above process has divided the object area corresponding to each scene image, for a scene image, the point-pixel association results can be divided according to its object area, and the pixel features corresponding to the pixels in the object area in the point-pixel association results are constructed into two-dimensional regional pixel features, and then the corresponding three-dimensional regional point cloud features are determined based on the two-dimensional regional pixel features.

[0104] According to the above process, multiple groups of two-dimensional region pixel features and corresponding three-dimensional region point cloud features corresponding to each viewing angle are obtained.

[0105] Step 140: Perform image-text-point cloud comparative learning based on text features, two-dimensional region pixel features, and three-dimensional region point cloud features to generate semantic region contrast loss values, and adjust the parameters of the point cloud network based on the semantic region contrast loss values ​​until a trained point cloud network is obtained.

[0106] In one embodiment, referring to Figure 7 , Figure 7This is a flowchart of performing image-text-point cloud contrast learning based on text features, two-dimensional region pixel features, and three-dimensional region point cloud features to generate semantic region contrast loss values, provided by an embodiment of the present application. Specifically, the flowchart includes the following steps:

[0107] Step 710: Perform average pooling on the two-dimensional region pixel features to obtain regional point features, and perform average pooling on the three-dimensional region point cloud features to obtain regional pixel features.

[0108] In one embodiment, different from point-level contrast learning in related art, the present embodiment implements a region-level solution to promote information-rich feature learning. Specifically, for each object region, the two-dimensional region pixel features are average pooled to obtain the regional point features. , and average pooling the three-dimensional regional point cloud features to obtain regional pixel features . Among them, average pooling can be understood as the operation of calculating the average value. In addition, the embodiment of the present application can represent all regional point features as , all regional pixel features are expressed as , where M represents the total number of object regions in all views.

[0109] Step 720: Calculate the semantic region contrast loss value based on the region point features, region pixel features, and text features.

[0110] In one embodiment, during the contrastive learning process, since different parts of the same object in the target scene may be marked as different mask areas, these different parts of the same object with the same semantics may be regarded as negative samples, thereby pushing away these erroneous negative samples and making it difficult to learn. Therefore, when calculating the loss function, the embodiment of the present application needs to use regional semantic information to filter these false negative samples, effectively enhance the perception of contextual information, alleviate the self-semantic conflict problem at the regional level, and force the consistency of regional representation to improve the accuracy of contrastive learning. Figure 8 , Figure 8 This is a flowchart of performing comparative learning based on regional point features, regional pixel features, and text features to calculate the semantic region contrast loss value provided by an embodiment of the present application, specifically including the following steps:

[0111] Step 810: For each object region, calculate an indication parameter between the region and other object regions based on the region pixel features and text features, and determine multiple negative samples from the other object regions based on the indication parameters.

[0112] In one embodiment, it is necessary to determine a corresponding negative sample for each object region. In the process of determining the negative sample, it is necessary to use the indicator parameter to determine whether other object regions outside the object region can be used as its negative sample. Figure 9 , Figure 9 This is a flowchart of calculating indication parameters between regions and other object regions based on regional pixel features and text features, provided by an embodiment of the present application, specifically including the following steps:

[0113] Step 910: Select comparison areas one by one from other object areas, and obtain regional pixel features of the comparison areas as comparison pixel features.

[0114] In one embodiment, assuming that the negative sample of object region i needs to be determined, a comparison region is selected one by one from other object regions, assuming that it is object region j. At this time, the text feature of object region i is , the regional pixel features are , the regional pixel features of object region j .

[0115] Step 920: Calculate a first text similarity between the text feature and the region pixel feature, and calculate a second text similarity between the text feature and the comparison pixel feature.

[0116] In one embodiment, the first text similarity is the cosine similarity between the text feature and the region pixel feature, expressed as , the second text similarity is the cosine similarity between the text feature and the contrast pixel feature, expressed as ,

[0117] Step 930: Obtain a difference between the first text similarity and the second text similarity. If the difference is greater than or equal to a preset reference value, set an indication parameter to one.

[0118] In one embodiment, the difference is expressed as: , the preset reference value is ,if , then the parameter If it is 0, it means that the object region j is not a negative sample of the object region i. Otherwise, the parameter is 1, indicating that object region j is a negative sample of object region i.

[0119] In this way, one or more negative samples corresponding to each object region are obtained.

[0120] Step 820: Calculate the regional loss value based on the negative sample, the regional point features and the regional pixel features corresponding to the object area, and obtain the semantic region contrast loss value according to the average of the regional loss values.

[0121] In one embodiment, referring to Figure 10 , Figure 10 This is a flowchart of calculating the regional loss value based on the negative sample, the regional point features corresponding to the object area, and the regional pixel features provided in an embodiment of the present application, which specifically includes the following steps:

[0122] Step 1010: For each object region, calculate a first semantic similarity between the region point features and the region pixel features.

[0123] In one embodiment, taking the object region i as an example, its region point features are , the regional pixel features are , so the first semantic similarity is expressed as: ,in, It is a preset temperature parameter used to control the sharpness of the distribution of the first semantic similarity and can be set according to actual conditions.

[0124] Step 1020: Obtain the regional pixel feature of each negative sample as the sample pixel feature, calculate the sample semantic similarity between the regional point feature and each sample pixel feature, and calculate the sum of the sample semantic similarities as the second semantic similarity.

[0125] In one embodiment, for a negative sample, its region pixel feature is referred to as a sample pixel feature. Therefore, for an object region i, when the negative sample is an object region j, the sample semantic similarity is expressed as: , here we introduce the indicator parameter , if it is 0, the calculated value of the sample semantic similarity is 0. Next, the semantic similarities of all samples in the object region i are summed up, and the obtained second semantic similarity is expressed as: .

[0126] Step 1030: Obtain the sum of the first semantic similarity and the second semantic similarity as an intermediate reference value, and obtain a region loss value according to the quotient of the first semantic similarity and the intermediate reference value.

[0127] In one embodiment, the area loss value is expressed as:

[0128]

[0129] Next, the semantic region contrast loss value is obtained based on the average of the region loss values , expressed as:

[0130]

[0131] After obtaining the semantic region contrast loss value, the semantic region contrast loss value is minimized and the parameters of the point cloud network and the three-dimensional mapping layer are adjusted until the trained point cloud network and the three-dimensional mapping layer are obtained.

[0132] The technical solution provided by the embodiment of the present application obtains laser radar point cloud data and corresponding multi-view images, inputs the laser radar point cloud data into a point cloud network for feature extraction to obtain three-dimensional point cloud features, inputs the multi-view images into an image encoder for feature extraction to obtain two-dimensional image features, inputs the multi-view images into a visual basic model for mask division to obtain at least one object region, and uses a text encoder to generate text features corresponding to each object region, associates the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, divides the point-pixel association result according to the object region, obtains two-dimensional region pixel features and three-dimensional region point cloud features corresponding to each object region, performs image-text-point cloud comparative learning based on the text features, two-dimensional region pixel features, and three-dimensional region point cloud features, generates a semantic region contrast loss value, and adjusts the parameters of the point cloud network based on the semantic region contrast loss value until a trained point cloud network is obtained. The embodiment of the present application introduces multimodal joint training and uses region-level text features to achieve cross-modal alignment. Based on the self-supervised contrastive learning framework of image-text-point cloud, the representation ability of the point cloud network is enhanced, and local semantic consistency is strengthened during the training process: by shortening the cross-modal feature distance of the same semantic area and pushing away the features of different semantic areas, fine-grained cross-modal feature alignment is achieved, avoiding global feature blurring, and significantly improving the feature extraction accuracy of 3D point clouds, thereby improving the accuracy of downstream 3D point cloud understanding. Without relying on manually annotated point cloud data, low-cost and high-efficiency 3D space understanding is achieved.

[0133] In one embodiment, a method for processing three-dimensional point cloud data is also provided. The specific steps include: obtaining initial point cloud data, inputting the initial point cloud data into a point cloud network for feature extraction, and obtaining point cloud feature data. The point cloud network is trained using the point cloud network training method provided in any of the above embodiments. The point cloud feature data can be used in downstream three-dimensional spatial processing according to actual needs.

[0134] The present application also provides a point cloud network training device, which can implement the above-mentioned point cloud network training method. Figure 11 , the device comprises:

[0135] Initial feature extraction module 1110: used to obtain lidar point cloud data and corresponding multi-view images, input the lidar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features.

[0136] The text description generation module 1120 is used to input the multi-view image into the visual basic model for mask segmentation to obtain at least one object area, and use the text encoder to generate text features corresponding to each object area.

[0137] Mapping association module 1130: used to associate two-dimensional image features and three-dimensional point cloud features according to a preset spatial projection relationship to obtain point-pixel association results, divide the point-pixel association results according to object areas, and obtain two-dimensional regional pixel features and three-dimensional regional point cloud features corresponding to each object area.

[0138] Contrastive learning module 1140: used to perform image-text-point cloud contrast learning based on text features, two-dimensional region pixel features and three-dimensional region point cloud features, generate semantic region contrast loss values, and adjust the parameters of the point cloud network based on the semantic region contrast loss values ​​until a trained point cloud network is obtained.

[0139] The specific implementation of the point cloud network training device of this embodiment is basically the same as the specific implementation of the above-mentioned point cloud network training method, and will not be repeated here.

[0140] An embodiment of the present application further provides an electronic device, including:

[0141] at least one memory;

[0142] at least one processor;

[0143] at least one program;

[0144] The program is stored in the memory, and the processor executes at least one of the programs to implement the point cloud network training method or the three-dimensional point cloud data processing method described above. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0145] See also Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0146] The processor 1201 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0147] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called by the processor 1201 to execute the point cloud network training method or the three-dimensional point cloud data processing method of the embodiments of this application.

[0148] Input / output interface 1203, used to implement information input and output;

[0149] Communication interface 1204, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0150] Bus 1205 , which transmits information between various components of the device (e.g., processor 1201 , memory 1202 , input / output interface 1203 , and communication interface 1204 );

[0151] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0152] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned point cloud network training method or three-dimensional point cloud data processing method.

[0153] The memory, as a non-transient storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] The point cloud network training method, point cloud data processing method, and device proposed in the embodiments of the present application obtain laser radar point cloud data and corresponding multi-view images, input the laser radar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features, input the multi-view images into the visual basic model for mask division to obtain at least one object region, and use a text encoder to generate text features corresponding to each object region, associate the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, divide the point-pixel association result according to the object region, obtain two-dimensional region pixel features and three-dimensional region point cloud features corresponding to each object region, perform image-text-point cloud comparative learning based on the text features, two-dimensional region pixel features, and three-dimensional region point cloud features, generate a semantic region contrast loss value, and adjust the parameters of the point cloud network based on the semantic region contrast loss value until a trained point cloud network is obtained. The embodiments of the present application introduce multimodal joint training and use region-level text features to achieve cross-modal alignment. Based on the self-supervised contrastive learning framework of image-text-point cloud, the representation ability of the point cloud network is enhanced, and local semantic consistency is strengthened during the training process: by shortening the cross-modal feature distance of the same semantic area and pushing away the features of different semantic areas, fine-grained cross-modal feature alignment is achieved, avoiding global feature blurring, and significantly improving the feature extraction accuracy of 3D point clouds, thereby improving the accuracy of downstream 3D point cloud understanding. Without relying on manually annotated point cloud data, low-cost and high-efficiency 3D space understanding is achieved.

[0155] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0156] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0158] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0159] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "more" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0165] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A point cloud network training method, characterized in that: The method is performed by a self-supervised training system, wherein the self-supervised training system includes at least an image encoder, a text encoder, a visual base model, and the point cloud network. Except for the point cloud network, other network parameters are frozen. The method includes: Acquire laser radar point cloud data and corresponding multi-view images, input the laser radar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features; Inputting the multi-view image into the visual base model for mask segmentation to obtain at least one object region, and using the text encoder to generate text features corresponding to each object region; Associating the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, and dividing the point-pixel association result according to the object area to obtain a two-dimensional regional pixel feature and a three-dimensional regional point cloud feature corresponding to each object area; Average pooling is performed on the two-dimensional regional pixel features to obtain regional point features, and average pooling is performed on the three-dimensional regional point cloud features to obtain regional pixel features. For each of the object regions, comparison regions are selected one by one from the other object regions, and the regional pixel features of the comparison regions are obtained as comparison pixel features. A first text similarity between the text feature and the regional pixel feature is calculated, and a second text similarity between the text feature and the comparison pixel feature is calculated. A difference between the first text similarity and the second text similarity is obtained. If the difference is greater than or equal to a preset reference value, an indication parameter is set to one to indicate that the comparison region is a negative sample of the object region. Based on the indication parameter, multiple negative samples are determined from the other object regions. A regional loss value is calculated based on the negative samples, the regional point features corresponding to the object region, and the regional pixel features. A semantic region contrast loss value is obtained according to the average of the region loss values. Parameters of the point cloud network are adjusted based on the semantic region contrast loss value until the trained point cloud network is obtained.

2. The point cloud network training method according to claim 1, characterized in that: The multi-view image includes a scene image corresponding to at least one view, and inputting the multi-view image into the visual basic model for mask segmentation to obtain at least one object area includes: Inputting the scene images into the visual basic model respectively for mask division to obtain a mask image corresponding to each scene image, wherein the mask image includes at least one mask area; Region division is performed in each of the scene images based on the mask region to obtain at least one object region corresponding to each of the scene images.

3. The point cloud network training method according to claim 2, characterized in that: The self-supervised training system further includes a graph-to-text basic model, and the generating of text features corresponding to each object region by using the text encoder includes: For each of the scene images, inputting the corresponding object region into the image-to-text basic model to generate text, thereby obtaining a guide language description corresponding to each of the object regions; The guide language description is input into the text encoder for feature extraction to obtain the corresponding text features.

4. The point cloud network training method according to claim 1, characterized in that: The calculating of the regional loss value based on the negative sample, the regional point feature corresponding to the object region, and the regional pixel feature includes: For each of the object regions, calculating a first semantic similarity between the region point feature and the region pixel feature; Obtaining the regional pixel feature of each of the negative samples as a sample pixel feature, calculating a sample semantic similarity between the regional point feature and each of the sample pixel features, and calculating a sum of the sample semantic similarities as a second semantic similarity; The sum of the first semantic similarity and the second semantic similarity is obtained as an intermediate reference value, and the region loss value is obtained according to a quotient of the first semantic similarity and the intermediate reference value.

5. The point cloud network training method according to any one of claims 1 to 4, characterized in that: The self-supervised training system further includes a two-dimensional mapping layer and a three-dimensional mapping layer to be trained. Before dividing the two-dimensional image features according to the object regions to obtain the two-dimensional region pixel features corresponding to each of the object regions, the method further includes: Inputting the two-dimensional image features into the two-dimensional mapping layer to perform channel dimension mapping, and updating the two-dimensional image features; The three-dimensional point cloud features are input into the three-dimensional mapping layer for channel dimension mapping, and the three-dimensional point cloud features are updated. The channel dimensions of the updated three-dimensional point cloud features and the two-dimensional image features are consistent.

6. A three-dimensional point cloud data processing method, characterized in that: include: Get initial point cloud data; The initial point cloud number is input into a point cloud network for feature extraction to obtain point cloud feature data, and the point cloud network is trained by the point cloud network training method according to any one of claims 1 to 5.

7. A point cloud network training device, characterized in that: The self-supervised training system is executed by a self-supervised training system, which includes at least an image encoder, a text encoder, a visual base model, and the point cloud network. Except for the point cloud network, other network parameters are frozen. The device includes: Initial feature extraction module: used to obtain lidar point cloud data and corresponding multi-view images, input the lidar point cloud data into the point cloud network for feature extraction to obtain three-dimensional point cloud features, and input the multi-view images into the image encoder for feature extraction to obtain two-dimensional image features; A text description generation module is configured to input the multi-view image into the visual base model for mask segmentation to obtain at least one object region, and generate text features corresponding to each object region using the text encoder; A mapping association module is configured to associate the two-dimensional image features with the three-dimensional point cloud features according to a preset spatial projection relationship to obtain a point-pixel association result, and to divide the point-pixel association result according to the object area to obtain a two-dimensional regional pixel feature and a three-dimensional regional point cloud feature corresponding to each object area; Contrastive learning module: used to perform average pooling on the two-dimensional regional pixel features to obtain regional point features, and to perform average pooling on the three-dimensional regional point cloud features to obtain regional pixel features. For each object region, a contrast region is selected one by one from the other object regions, and the regional pixel features of the contrast region are obtained as contrast pixel features. A first text similarity between the text feature and the regional pixel feature is calculated, and a second text similarity between the text feature and the contrast pixel feature is calculated. The difference between the first text similarity and the second text similarity is obtained. If the difference is greater than or equal to a preset reference value, an indication parameter is set to one to indicate that the contrast region is a negative sample of the object region. Based on the indication parameter, multiple negative samples are determined from the other object regions. A regional loss value is calculated based on the negative samples, the regional point features corresponding to the object region, and the regional pixel features. A semantic region contrast loss value is obtained according to the average of the region loss values. The parameters of the point cloud network are adjusted based on the semantic region contrast loss value until the trained point cloud network is obtained.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the point cloud network training method described in any one of claims 1 to 5, or the three-dimensional point cloud data processing method described in claim 6.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the point cloud network training method according to any one of claims 1 to 5 or the three-dimensional point cloud data processing method according to claim 6 is implemented.

Images